o ?߱ih`@spddlmZddlZddlZddlZddlmZmZmZm Z ddl Z ddl Z ddl m ZddlZ ddlZddlZddlmZddlm Z mZmZmZddlmZzddlmZWn eycdZYnwerxddlmZdd lm Z dd l!m"Z"Gd d d Z#Gd ddZ$Gddde$Z%Gddde$Z&Gddde$Z'Gddde$Z(Gddde$Z)Gddde$Z*dS)) annotationsN) TYPE_CHECKINGAnyCallableOptional) instantiate) distributedlogmisc wandb_util)get_local_tensor_if_DTensor)parallel_state)Config)ImaginaireModel)ImaginaireTrainerc@s$eZdZdZddd Zdd dZdS) CallBackGroupaA class for hosting a collection of callback objects. It is used to execute callback functions of multiple callback objects with the same method name. When callbackgroup.func(args) is executed, internally it loops through the objects in self._callbacks and runs self._callbacks[0].func(args), self._callbacks[1].func(args), etc. The method name and arguments should match. Attributes: _callbacks (list[Callback]): List of callback objects. configrtrainerrreturnNonecCsg|_|jj}|rft|tst|tjjr&tj dt ddddt |D}| D]=\}}d|vr>t d|d|q*t d |d |t|}t|tsYJ|d ||_||_|j|q*d Sd S) zInitializes the list of callback objects. Args: config (Config): The config object for the Imaginaire codebase. trainer (ImaginaireTrainer): The main trainer. zdThe 'config.trainer.callbacks' parameter should be a dict instead of a list. Please update your code stacklevelcSi|] \}}d||qS) callback_).0ivrrX/data/cameron/vidgen/cosmos-predict2.5/cosmos_predict2/_src/imaginaire/utils/callback.py Jz*CallBackGroup.__init__.._target_z Callback z) is missing the '_target_' field. SKip zInstantiating callback z: z is not a valid callback.N) _callbacksr callbacks isinstancelist omegaconf listconfig ListConfigwarningswarnDeprecationWarning enumerateitemsr criticalrCallbackrappend)selfrrZcallback_configsZ callback_nameZcurrent_callback_cfg _callbackrrr__init__9s0 zCallBackGroup.__init__ method_namestrrcsdfdd }|S)zLoops through the callback objects to call the corresponding callback function. Args: method_name (str): Callback method name. rrcsBjD]}t|s Jt|}t|sJ||i|}qdSN)r#hasattrgetattrcallable)argskwargscallbackmethod_r5r2rrmulti_callback_wrapper_s   z9CallBackGroup.__getattr__..multi_callback_wrapperNrrr)r2r5rArr@r __getattr__XszCallBackGroup.__getattr__N)rrrrrr)r5r6rr)__name__ __module__ __qualname____doc__r4rCrrrrr.s rc@steZdZdZd^d_ddZd`daddZd`dbddZ d`dbddZd`dcddZd`dcddZ d`ddd d!Z d`ded"d#Z d`dcd$d%Z d`dcd&d'Z dfd(d)Zdfd*d+Z d`dgd2d3Z d`dhd4d5Z d`did8d9Z d`did:d;Z d`djd>d?Z d`dbd@dAZ d`didBdCZd`dadDdEZdkdFdGZ dldmdJdKZdndNdOZd`dadPdQZd`dadRdSZdodpdVdWZdndXdYZd`dadZd[Zdfd\d]Z dS)qr0zThe base class for all callbacks. All callbacks should inherit from this class and adhere to the established method names and signatures. NrOptional['Config']rOptional['ImaginaireTrainer']cCs(|dus|durtjdtdd~~dS)aInitializes a Callback object. Args: config (Optional[Config]): The configuration object for the Imaginaire codebase, if available. trainer (Optional[ImaginaireTrainer]): The main trainer handling the training loop, if available. Notes: The config and trainer parameters are optional to maintain backward compatibility. In future releases, these parameters will be removed. Upon using these parameters, a deprecation warning will be issued. NzThe 'config' and 'trainer' parameters are deprecated and will be removed in a future release. Please update your code to create Callback instances without these parameters.rr)r*r+r,)r2rrrrrr4os zCallback.__init__rmodelr iterationintrrcCdSr7rr2rJrKrrron_train_startzCallback.on_train_startdatadict[str, torch.Tensor]cCdS)a` Called before the training step, for each batch. This is paired with on_training_step_end() but note that when using gradient accumulation, while on_training_step_end() is only called when the optimizer is updated, this function is called for every batch. Use on_training_step_batch_start and on_training_step_batch_end if you need callbacks that are called for every batch, albeit with the same iteration number. FIXME - should this either be deprecated, or called only when a new training step is started after having updated the optimizer? Nrr2rJrQrKrrron_training_step_start zCallback.on_training_step_startcCrS)a Called before the training step, for each batch, similarly to on_training_step_start(). This function is paired with on_training_step_batch_end(), and both functions are called for every batch even when using gradient accumulation. Note that the iteration is only updated when the optimizer is updated, and therefore it may be the same for multiple invocations. NrrTrrron_training_step_batch_startz%Callback.on_training_step_batch_startcCrMr7rr2rKrrron_before_forwardrPzCallback.on_before_forwardcCrMr7rrYrrron_after_forwardrPzCallback.on_after_forward model_ddp#distributed.DistributedDataParallelloss torch.TensorcCrMr7rr2r\r^rKrrron_before_backwardzCallback.on_before_backwardcCrMr7rr2r\rKrrron_after_backwardrPzCallback.on_after_backwardcCrMr7rrYrrron_before_dataloadingrPzCallback.on_before_dataloadingcCrMr7rrYrrron_after_dataloadingrPzCallback.on_after_dataloadingcCrMr7rr2rrron_optimizer_init_startrPz Callback.on_optimizer_init_startcCrMr7rrgrrron_optimizer_init_endrPzCallback.on_optimizer_init_end optimizertorch.optim.Optimizer scheduler$torch.optim.lr_scheduler.LRScheduler grad_scalertorch.amp.GradScalercCrMr7rr2r\rjrlrnrKrrron_before_optimizer_steprXz!Callback.on_before_optimizer_stepcCrMr7rr2r\rjrlrKrrron_before_zero_gradszCallback.on_before_zero_grad data_batch output_batchcCrS)a/ Called at the end of a training step for every batch even when using gradient accumulation. This is paired with on_training_step_batch_start(). Note that the iteration is only updated when the optimizer is updated, and therefore it may be the same for multiple batches. Nrr2rJrtrur^rKrrron_training_step_batch_ends z#Callback.on_training_step_batch_endcCrS)a} Called at the end of a training step, but note that when using gradient accumulation, this is only called when the optimizer is updated, and the iteration incremented, whereas on_training_step_start is called every time. Use on_training_step_batch_start and on_training_step_batch_end if you need callbacks that are called for every batch. Nrrvrrron_training_step_endszCallback.on_training_step_enddataloader_valtorch.utils.data.DataLoadercCrMr7rr2rJryrKrrron_validation_startrbzCallback.on_validation_startcCrMr7rrTrrron_validation_step_startrbz!Callback.on_validation_step_startcCrMr7rrvrrron_validation_step_endrXzCallback.on_validation_step_endcCrMr7rrNrrron_validation_endrPzCallback.on_validation_endcCrMr7r)r2rJrrron_load_checkpoint_startrPz!Callback.on_load_checkpoint_startcheckpoint_path Optional[str]cCrMr7r)r2rJrKrrrron_load_checkpoint_endrbzCallback.on_load_checkpoint_end state_dict dict[Any]cCrS)z Called when checkpoint loading is about to start, but after on_save_checkpoint_start(). FIXME - why do we need this callback, can't we just use on_save_checkpoint_start()? Nrr2rJrrrron_load_checkpoint szCallback.on_load_checkpointcCrS)zB Called when checkpoint saving is about to start. NrrNrrron_save_checkpoint_startsz!Callback.on_save_checkpoint_startcCrS)a Called when the synchronous part of checkpointing is finished, this function can be used along with on_save_checkpoint_start() to measure the exposed (synchronous) checkpoint time. Note that for asynchronous checkpoint, the checkpoint may still be ongoing, so this function does not mean the checkpoint is finished for the asynchronous case, use on_save_checkpoint_success() for that. NrrNrrron_save_checkpoint_endrXzCallback.on_save_checkpoint_end elapsed_timefloatcCrS)a Called when checkpoint saving is fully finished, and succeeded. Not called if checkpoint failed. For synchronous checkpoint, it is called at the same time as on_save_checkpoint_end(), but for asynchronous checkpoint, it is called after the asynchronous part has also finished. For checkpointers with out-of-process checkpointing, this function is called as soon as the notification is received from the checkpointer process, which may not be immediately after the checkpoint has completed but later on. Therefore, if you need to measure the full checkpoint duration for the asynchronous part, use the elapsed_time parameter, do not measure it directly as this would be a significant overestimate. Nr)r2rKrrrron_save_checkpoint_success rVz#Callback.on_save_checkpoint_successcCrMr7rrrrron_save_checkpoint,rPzCallback.on_save_checkpointcCrMr7rrNrrr on_train_end/rPzCallback.on_train_endcCrMr7rrgrrr on_app_end2rPzCallback.on_app_end)NN)rrHrrIrrJrrKrLrrrJrrQrRrKrLrrrKrLrrr\r]r^r_rKrLrrr\r]rKrLrrrB r\r]rjrkrlrmrnrorKrLrr r\r]rjrkrlrmrKrLrr rJrrtrRrurRr^r_rKrLrrrJrryrzrKrLrr)rJrrr)rN)rJrrKrLrrrr)rJrrrrr)rr)rKrLrrrr)!rDrErFrGr4rOrUrWrZr[rardrerfrhrirqrsrwrxr|r}r~rrrrrrrrrrrrrrr0isR                    r0c@s*eZdZdZddd d Z ddddZdS)EMAModelCallbackz2The callback class for tracking EMA model weights.rrJrrKrLrrcCsH|jjjrt|dsJd|jjtjd|_dSt|dr"JddS)Nemaz.EMA should be initialized from ImaginaireModeldtypez#There should be no EMA initialized.)rrenabledr8totorchfloat32rNrrrrO9s zEMAModelCallback.on_train_startrtrRrur^r_cCs |jjjr|j||dSdSr7)rrrZupdate_averagervrrrrxBs z%EMAModelCallback.on_training_step_endNrrr)rDrErFrGrOrxrrrrr6s  rc@seZdZdZejddd d Zej ddddZej dd ddZej ddddZ ejddddZ ejddddZ dS)!ProgressBarCallbackWThe callback class for visualizing the training/validation progress bar in the console.rrJrrKrLrrcCstj|jjj|dd|_dS)NTraining)initialdesc)tqdmtrangerrmax_iter train_pbarrNrrrrORsz"ProgressBarCallback.on_train_startrtrRrur^r_cC|jdSr7)rupdatervrrrrxV z(ProgressBarCallback.on_training_step_endryrzcCsX|jjjdur |jjj}nt|}|dur|dks Jd|tj|dddd|_dS)Nrz)Invalid number of validation iterations: Z ValidatingF)rpositionleave)rr max_val_iterlenrrval_pbar)r2rJryrKnum_iterrrrr|as  z'ProgressBarCallback.on_validation_startcCrr7)rrrvrrrr~lrz*ProgressBarCallback.on_validation_step_endcCrr7)rcloserNrrrrwz%ProgressBarCallback.on_validation_endcCs|jj|jdSr7)r checkpointerfinalizerrrNrrrr{s z ProgressBarCallback.on_train_endNrrrr) rDrErFrGr rank0_onlyrOrxr|r~rrrrrrrOs"   rc@sHeZdZdZejddd d Zejddd dZej ddddZdS)IterationLoggerCallbackrrrJrrKrLrrcCst|_d|_dS)Nr)timestart_iteration_timeelapsed_iteration_timerNrrrrOs  z&IterationLoggerCallback.on_train_startrQrRcCst|_dSr7)rrrTrrrrUrz.IterationLoggerCallback.on_training_step_startrtrur^r_c Csl|jt|j7_||jjjdkr4|j|jjj}td|d|dd|dd|_dSdS)Nrz Iteration: z, average iter time: Z2fz , total loss 4f) rrrrr logging_iterr infoitem)r2rJrtrur^rKZavg_timerrrrxs  $ z,IterationLoggerCallback.on_training_step_endNrrrr) rDrErFrGrrrOrUrxrrrrrsrc@sleZdZdZd'd(d d Z d'd)ddZ d'd*ddZ d'd+ddZ d'd*d d!Zd'd(d"d#Z d'd(d$d%Z d&S), WandBCallbackaPThe callback class for logging to Weights and Biases (W&B). By default, WandBCallback logs the following training stats to W&B every config.trainer.logging_iter: - iteration: The current iteration number (useful for visualizing the training progress over time). - train/loss: The computed overall loss in the training batch. - optim/lr: The current learning rate. - timer/*: The averaged timing results of each code block recorded by trainer.training_timer. For validation, WandBCallback logs: - val/loss: The computed overall loss in the validation dataset. rrJrrKrLrrcCstj|j|ddS)N)rJ)r Z init_wandbrrNrrrrOszWandBCallback.on_train_startr\r]rjrkrlrmrnrocCsV||jjjdkr'tr)tjd|di|dtjd|i|ddSdSdS)Nrzoptim/lrstepzoptim/grad_scale) rrrris_rank0wandbr get_last_lr get_scalerprrrrqsz&WandBCallback.on_before_optimizer_steprtrRrur^r_cCsz||jjjdkr;|jj}tr3tjdd| D|dtjd|i|dtjd|i|d|jj dSdS)NrcSr)ztimer/r)rkeyvaluerrrr r!z6WandBCallback.on_training_step_end..rz train/lossrK) rrrtraining_timercompute_average_resultsrrrr r.reset)r2rJrtrur^rKZ timer_resultsrrrrxs z"WandBCallback.on_training_step_endryrzcCs*tggtjdddtjdddd|_dS)Ngcuda)devicer) data_batchesZoutput_batchesr^ sample_size)dictrtensor _val_cacher{rrrr|s    z!WandBCallback.on_validation_startcCs6t|}|jd||7<|jd|7<dS)Nr^r)r get_data_batch_sizer)r2rJrtrur^rK batch_sizerrrr~s z$WandBCallback.on_validation_step_endcCstj|jdtjjdtj|jdtjjd|jd|jd}tr?t d|d|dt jd|i|ddSdS) Nr^)oprzValidation loss (iteration z): rzval/lossr) dist all_reducerReduceOpSUMrrrr rr)r2rJrKr^rrrrszWandBCallback.on_validation_endcCs tdSr7)rfinishrNrrrrs zWandBCallback.on_train_endNrrrrr) rDrErFrGrOrqrxr|r~rrrrrrrs      rc@sNeZdZdZd!dd Zd"d#ddZd"d$ddZ d"d$ddZ d"d%ddZd S)&LowPrecisionCallbackz2The callback class handling low precision trainingrrrr update_iterrLcCs ||_dSr7)r)r2rrrrrrr4s zLowPrecisionCallback.__init__rrJrrKrrcCsJ|jtjkrtdtj|_n|jtjtj tj fvsJd|j|_ dS)Nz4Using fp32. We should disable master weights update.z4LowPrecisionCallback must use a low precision dtype.) precisionrrr r/sysmaxsizerbfloat16float16halfprecision_typerNrrrrOs    z#LowPrecisionCallback.on_train_startrQrRcCB|D]\}}t|tjrt||r|j|jd||<qdSNrr.r%rTensoris_floating_pointrrr2rJrQrKkrrrrrUs z+LowPrecisionCallback.on_training_step_startcCrrrrrrrr} s z-LowPrecisionCallback.on_validation_step_startr\r]rjrkrlrmc Cs||jdkrDt|ddrFgg}}t|j|jD]"\}}t|d|dD]\} } |t| j|t| jq&qt ||dSdSdS)Nrmaster_weightsFparams) rr9zip param_groupsZparam_groups_masterr1r rQr_foreach_copy_) r2r\rjrlrKrZ master_paramsgroupZ group_masterpZp_masterrrrrss  z(LowPrecisionCallback.on_before_zero_gradN)rrrrrrLrrrr) rDrErFrGr4rOrUr}rsrrrrrs     rcseZdZdZ   d+d,fd d Zd-d.ddZd-d.ddZ d-d/ddZd-d0ddZ d-d1d#d$Z d-d2d%d&Z d-d.d'd(Z d-d.d)d*Z Z S)3 NVTXCallbackz%The callback for creating NVTX rangesFN synchronizeboolrrHrrIcst||||_dSr7)superr4r)r2rrr __class__rrr4$s zNVTXCallback.__init__rrKrLrrcC"|jrtjtjjddS)Nforwardrrrnvtx range_pushrYrrrrZ-s zNVTXCallback.on_before_forwardcC |jrtjtjjdSr7rrrr range_poprYrrrr[2 zNVTXCallback.on_after_forwardr\r]r^r_cCr)Nbackwardrr`rrrra7s zNVTXCallback.on_before_backwardcCrr7rrcrrrrd>rzNVTXCallback.on_after_backwardrjrkrlrmrnrocCr)Noptimizer_steprrprrrrqCs z%NVTXCallback.on_before_optimizer_stepcCrr7rrrrrrrsOs z NVTXCallback.on_before_zero_gradcCstjjddS)N dataloading)rrrrrYrrrreZsz"NVTXCallback.on_before_dataloadingcCstjjdSr7)rrrrrYrrrrf]sz!NVTXCallback.on_after_dataloading)FNN)rrrrHrrIrrrrrr)rDrErFrGr4rZr[rardrqrsrerf __classcell__rrrrr!s"    r)+ __future__rrrr*typingrrrrr'rtorch.distributedrrtorch.utils.datarr+cosmos_predict2._src.imaginaire.lazy_configr%cosmos_predict2._src.imaginaire.utilsr r r *cosmos_predict2._src.imaginaire.utils.miscr megatron.corer ImportError&cosmos_predict2._src.imaginaire.configr%cosmos_predict2._src.imaginaire.modelr'cosmos_predict2._src.imaginaire.trainerrrr0rrrrrrrrrrs@        ;N2 Q/