o ?߱i@sddlmZddlmZddlmZddlZddlmZm Z m Z ddl m Z ddl mZddlmZmZmZmZmZdd lmZdd lmZmZd d ZedddZdddZdS)) annotations)contextmanager)partialN)CheckpointImplapply_activation_checkpointingcheckpoint_wrapper)init_device_mesh)FullyShardedDataParallel) _post_forward_post_forward_reshard _pre_forward_pre_forward_unshard_root_pre_forward) _p_assert) distributedlogcs6tdtttjd}fdd}t|||ddS)zZapply activation checkpointing to model returns None as model is updated directly z---> applying fdsp activation checkpointing...)checkpoint_implcs&d}D] }t||rd}|Sq|S)NFT) isinstance) submoduleresultZ block_clslist_block_cls[/data/cameron/vidgen/cosmos-predict2.5/cosmos_predict2/_src/imaginaire/utils/fsdp_helper.pycheck_fn4s z*apply_fsdp_checkpointing..check_fn)checkpoint_wrapper_fnrN)rcriticalrrr NO_REENTRANTr)modelrZnon_reentrant_wrapperrrrrapply_fsdp_checkpointing)s  rrtorch.nn.Modulec cst|t}|r_trJd|j}dgtdd}}tjjd4t ||||\}}d}t ||t |j ||\}}|rPt |jj|jkd|jd|jjWdn1sZwYzdVW|ruddi}t||t|||dSdS|rddi}t||t|||ww) Nz1FSDP context should be entered with grad disabledr)dummyz,FullyShardedDataParallel.possible_fsdp_scopez5Expected `FlatParameter` to be on the compute device z but got output)rFSDPtorchis_grad_enabled_handledictautogradprofilerrecord_functionrr r _fsdp_wrapped_moduler flat_paramdevicecompute_devicer r )renabledhandleargskwargsunusedr"rrrpossible_fsdp_scope?sJ  r5cCst}|dur t|d}t||}|dur||}|pd}||dkr/td|d|d|||dkr?td|dt|||fd d }|durPtd td |d ||S)a Initializes a device mesh for use with Hybrid Sharding strategy in FSDP (HSDP) training. This function requires explicit sizes for replica and sharding groups to accommodate models whose GPU fit is unknown, providing flexibility in distributed training setups. Args: replica_group_size (int): The size of each replica group. Must be provided to ensure the model fits within the available resources. sharding_group_size (int): The size of each sharding group that the model can fit. Must be provided to ensure the correct distribution of model parameters. device (str, optional): The device to use (e.g., "cuda:0"). If None, defaults to "cuda" with the local rank as the device index. Returns: A device mesh object compatible with FSDP. Raises: ValueError: If replica_group_size or sharding_group_size are not provided, or if the world size is not evenly divisible by the sharding group size. RuntimeError: If a valid device mesh cannot be created. Usage: If your model fits on 4 GPUS, and you have 3 nodes of 8 GPUs, then: Sharding_Group_Size = 4 Replica_Groups_Size = (24 total gpus, 4 per sharding group) = 6 Replica Groups >>> device_mesh = initialize_device_mesh(replica_group_size, sharding_group_size) >>> sharded_model = FSDP(model, device_mesh=device_mesh, ...) Ncudarz World size z0 is not evenly divisible by sharding group size .zVThe calculated number of replica groups is not evenly divisible by replica_group_size ) replicateshard)mesh_dim_namesz%Failed to create a valid device mesh.z0Device mesh initialized with replica group size z and sharding group size )rget_world_sizemin ValueErrorr RuntimeErrorrr)Zreplica_group_sizesharding_group_sizer. world_size device_meshrrrhsdp_device_meshas4     rC)rr )NNN) __future__r contextlibr functoolsrr%;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrrrtorch.distributed.device_meshrtorch.distributed.fsdpr r$%torch.distributed.fsdp._runtime_utilsr r r r rtorch.distributed.utilsr%cosmos_predict2._src.imaginaire.utilsrrrr5rCrrrrs       !