o ?߱iq@s,ddlZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z ddl mZzddlmZWneyEdZedYnwddlmZmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl(m*Z*dZ+dZ, ddl-m.Z.e.dgdZ/Gdddej0Z1Gddde1Z2ddZ3de/fdd Z4Gd!d"d"ej0Z5Gd#d$d$ej6Z7Gd%d&d&e!Z8Gd'd(d(ej0Z9Gd)d*d*e9Z:Gd+d,d,e9Z;e:e;d-ZGd2d3d3ejj0Z?Gd4d5d5e'Z@dS)6N)Optional) rearrangerepeat)MinimalA2AAttnOp) attention)apply_rotary_embzflash_attn is not installed.) ProcessGroupget_process_group_ranks) fully_shard)checkpoint_wrapper)_IncompatibleKeys) transforms)DotProductAttention)log)split_inputs_cp)WeightTrainingStat)CheckpointMode) SACConfigi) namedtuple VideoSizeTHWcsleZdZfddZdefddZddZedd Zd e j d e j fd d Z de j fddZ ddZZS)VideoPositionEmbcstd|_dSN)super__init__ _cp_groupself __class__X/data/cameron/vidgen/cosmos-predict2.5/cosmos_predict2/_src/predict2/networks/wan2pt1.pyrBs  zVideoPositionEmb.__init__ process_groupcCs ||_dSrr)r!r&r$r$r%enable_context_parallelF z(VideoPositionEmb.enable_context_parallelcCs d|_dSrr'r r$r$r%disable_context_parallelIr)z)VideoPositionEmb.disable_context_parallelcCdS)Nr$r r$r$r%seq_dimLzVideoPositionEmb.seq_dim x_B_T_H_W_Creturnc CsV|j}|jdur!t|j}t|}|\}}}}} |||||| f}||} || S)z With CP, the function assume that the input tensor is already split. It delegates the embedding generation to generate_embeddings function. N)shaperr lengenerate_embeddings_split_for_context_parallel) r!r/ B_T_H_W_Ccp_rankscp_sizeBrrrC embeddingsr$r$r%forwardPs    zVideoPositionEmb.forwardr5cCstr)NotImplementedError)r!r5r$r$r%r3_sz$VideoPositionEmb.generate_embeddingscCs |jdurt||j|jd}|S)N)xr-cp_group)rrr-)r!r:r$r$r%r4bs z,VideoPositionEmb._split_for_context_parallel)__name__ __module__ __qualname__rrr(r*propertyr-torchTensorr;Sizer3r4 __classcell__r$r$r"r%rAs  rcseZdZ   ddededededededeffd d Zdd dZ ddejde ede ede efddZ e ddZ Z S)VideoRopePosition3DEmb?head_dimlen_hlen_wlen_th_extrapolation_ratiow_extrapolation_ratiot_extrapolation_ratioc st||_||_||_|}|dd} | } |d| } || | | ks6Jd|d| d| d| | |_| |_|| | d|_|| | d|_|| | d|_ d|_ dS)Nz bad dim: z != z + F) rrmax_hmax_wmax_t_dim_h_dim_t h_ntk_factor w_ntk_factor t_ntk_factor_is_initialized) r!rIrJrKrLrMrNrOdimdim_hdim_wdim_tr"r$r%ris   0 zVideoRopePosition3DEmb.__init__r0NcCs|jrdS|j}|j}tt|j|j|j |_ td|dd|d ||_ td|dd|d ||_ d|_dS)NrrQT)rZrUrVrCarangemaxrRrSrTfloatcudaseqdim_spatial_rangedim_temporal_range)r!r\r^r$r$r%cache_parameterss"(( z'VideoRopePosition3DEmb.cache_parametersr5rWrXrYc CsP||dur |n|j}|dur|n|j}|dur|n|j}d|}d|}d|}d||j}d||j} d||j} |\} } } }}| |jkrQ||jksdJd| d|d|jd|jd t |j d| |}t |j d|| }t |j d| | }tj t |d | |d t |d | |d t |d | | dgdd}t |dS)a Generate embeddings for the given input size. Args: B_T_H_W_C (torch.Size): Input tensor size (Batch, Time, Height, Width, Channels). fps (Optional[torch.Tensor], optional): Frames per second. Defaults to None. h_ntk_factor (Optional[float], optional): Height NTK factor. If None, uses self.h_ntk_factor. w_ntk_factor (Optional[float], optional): Width NTK factor. If None, uses self.w_ntk_factor. t_ntk_factor (Optional[float], optional): Time NTK factor. If None, uses self.t_ntk_factor. Returns: Not specified in the original code snippet. Ng@rHzInput dimensions (H=z, W=z') exceed the maximum dimensions (max_h=z, max_w=)zt d -> t h w d)hwzh d -> t h w d)trizw d -> t h w d)rjrhr[zt h w d -> (t h w) 1 1 d)rfrWrXrYrdrerRrSrCouterrccatrrra)r!r5rWrXrYh_thetaw_thetat_thetah_spatial_freqsw_spatial_freqstemporal_freqsr8rrr_Zfreqs_hZfreqs_wZfreqs_tZ freqs_T_H_W_Dr$r$r%r3s2  z*VideoRopePosition3DEmb.generate_embeddingscCr+)Nrr$r r$r$r%r-r.zVideoRopePosition3DEmb.seq_dim)rHrHrH)r0N)NNN)r?r@rAintrarrfrCrErr3rBr-rFr$r$r"r%rGhsD  4rGc Csn|ddksJ|d}|tj}t|tdt||| }tjt |t |gdd}|S)NrQri'r,rl) typerCfloat64rmpowr_todivrncossin)r[positionhalfZsinusoidr=r$r$r%sinusoidal_embedding_1ds  (r video_sizecCs|j\}}}}|\}}} ||| } || ksJd|||d}t|tj} t|tj} t|tj| | ddd} | |jS)a Optimized version of rope_apply using flash_attention's rotary embedding implementation. This version processes the entire batch at once for efficiency. Args: x (Tensor): Input tensor with shape [batch_size, seq_len, n_heads, head_dim] video_size (VideoSize): Video dimensions with shape [T, H, W] freqs (Tensor): Complex frequencies with shape [max_seq_len, head_dim // 2] Returns: Tensor: Rotary-embedded tensor with same shape as input z&Sequence length must be equal to T*H*WrQTF) interleavedinplace) r1viewrCr|rzfloat32r}flash_apply_rotary_embdtype)r=rfreqs batch_sizeseq_lenn_headsrIrrrZ curr_seq_lenr|r}rotatedr$r$r% rope_applys   rcs6eZdZd fdd ZddZddZdd ZZS) WanRMSNormh㈵>cs,t||_||_tt||_dSr) rrr[epsnn ParameterrConesweight)r!r[rr"r$r%rs zWanRMSNorm.__init__cCs|jjddS)NrH)rdatafill_r r$r$r%reset_parametersszWanRMSNorm.reset_parameterscCs||||jSz> Args: x(Tensor): Shape [B, L, C] )_normratype_asrr!r=r$r$r%r;szWanRMSNorm.forwardcCs$|t|djddd|jS)NrQrkT)r[keepdim)rCrsqrtrymeanrrr$r$r%rs$zWanRMSNorm._norm)r)r?r@rArrr;rrFr$r$r"r%rs rcs*eZdZdfdd ZfddZZS) WanLayerNormư>Fcstj|||ddS)N)elementwise_affiner)rr)r!r[rrr"r$r%r szWanLayerNorm.__init__cs t|Srrr;rr"r$r%r;s zWanLayerNorm.forward)rF)r?r@rArr;rFr$r$r"r%r srcs*eZdZ ddeeffdd ZZS) SelfAttnOpNrcst|||Srr)r! q_B_L_H_D k_B_L_H_D v_B_L_H_Drr"r$r%r;szSelfAttnOp.forwardr)r?r@rArrr;rFr$r$r"r%rs rcsFeZdZ     dfdd Zdd Zd efd d Zd dZZS)WanSelfAttentionrkrkTrp2ptransformer_enginecs||dksJt||_||_|||_||_||_||_||_||_||_ t |||_ t |||_ t |||_t |||_|rNt||dnt |_|r[t||dnt |_|j dkrut|j|j|jdddd|_dS|j dkrt|_dSJd |j ) Nrrrbshdno_masknum_gqa_groupsattention_dropout qkv_formatattn_mask_type minimal_a2aF!Unreckognized attention backend: )rrr[ num_headsrI window_sizeqk_normr cp_comm_typeattention_backendrLinearqkvorIdentitynorm_qnorm_krattn_oprr!r[rrrrrrr"r$r%r$s:     zWanSelfAttention.__init__cCsdt|j}tjjj|jj|dtjjj|j j|dtjjj|j j|dtjjj|j j|d|jj j |j j j |j j j |j j j |jr_|j|jdSdSNrHstd)mathsqrtr[rCrinit trunc_normal_rrrrrbiasrzero_rrrrr!rr$r$r% init_weightsPs zWanSelfAttention.init_weightsrc sg|jddjjR\fdd}||\}}}t|||t|||||}|d}|}|S)z Args: x(Tensor): Shape [B, L, num_heads, C / num_heads] seq_lens(Tensor): Shape [B] video_size(VideoSize): Shape [T, H, W] freqs(Tensor): Rope freqs, shape [1024, C / num_heads / 2] NrQcsX|}|}|}|||fSr)rrrrrr)r=rrrbdnsr!r$r%qkv_fnks z(WanSelfAttention.forward..qkv_fn)r1rrIrrflattenr) r!r=seq_lensrrrrrrr$rr%r;`s(   zWanSelfAttention.forwardcCsR|jdkr|jj||||jddS|jdkr!|j|||dSJd|j)Nr)rrFr)rrset_context_parallel_groupr)r!r&ranksstreamr$r$r%rzs  z+WanSelfAttention.set_context_parallel_grouprTrrr) r?r@rArrrr;rrFr$r$r"r%r#s,rc@seZdZddZdS)WanT2VCrossAttentionc Cs|d|j|j}}}||||d||}||||d||}|||d||} | ||| d}| d}| |}|S) Args: x(Tensor): Shape [B, L1, C] context(Tensor): Shape [B, L2, C] context_lens(Tensor): Shape [B] rrkNrQ) sizerrIrrrrrrrrr) r!r=context context_lensrrrrrrr$r$r%r;s  zWanT2VCrossAttention.forwardN)r?r@rAr;r$r$r$r%rs rcs<eZdZ     d fdd Zfdd Zd d ZZS) WanI2VCrossAttentionrTrrrc st|||||||t|||_t|||_|r"t||dnt|_|j dkr.cross_attn_ffn) rrCrrrrchunkrrrar) r!r=rrrrrrrrr$r r%r;s4 zWanAttentionBlock.forward)rTFrrr) r?r@rAstrrrrr;rFr$r$r"r%rs # rc.eZdZdfdd ZddZddZZS) Headrcslt||_||_||_||_t||}t|||_ t |||_ t tdd||d|_dS)Nr,rQr)rrr[out_dim patch_sizerrprodrnormrrheadrrCrr)r!r[r r rr"r$r%rBs  "z Head.__init__cCsV|jdt|j}tjjj|j |dtjjj|j j |d|j j j dSr)r rrrr[rCrrrrr rrrrrr$r$r%rQs zHead.init_weightscCs|jtjksJtjdtjd(|j|djddd}|| |d|d|d}Wd|S1s;wY|S)zc Args: x(Tensor): Shape [B, L1, C] e(Tensor): Shape [B, C] rbrr,rQrlrN) rrCrrrr unsqueezerr r )r!r=rr$r$r%r;Ys& z Head.forward)rrr$r$r"r%rAsrcr) MLPProjFc slttjtj|tj||tjtj||tj||_|r4t t dt d|_ dSdS)Nr,) rrrCrr LayerNormrrprojrzeros%FIRST_LAST_FRAME_CONTEXT_TOKEN_NUMBERemb_pos)r!in_dimr  flf_pos_embr"r$r%rgs     zMLPProj.__init__cCsV|jd|jd|jd|jdt|dr)|jjdSdS)Nrr,rrr)rrhasattrrrrr r$r$r%rts zMLPProj.init_weightscCs@t|dr|j\}}}|dd||}||j}||}|S)NrrkrQ)rr1rrr)r! image_embedsbsrrZclip_extra_context_tokensr$r$r%r;}s    zMLPProj.forward)Frr$r$r"r%rfs  rcseZdZdZdddddddd ddd d d d d deddfdedededeffdd Z      d,dee j fddZ ddZ ddZ dd Zd-d!eefd"d#Zed$d%Zded&ejfd'd(Zd.d)effd*d+ ZZS)/WanModelzR Wan diffusion backbone supporting both text-to-video and image-to-video. t2v)r,rQrQrii i rTrFrrconcat_padding_mask sac_configrrc st|dvs J|_|_|_|___|_|_ | _ _ | _ _ ___|_|__jrH|dn|}t||d|d|d_tt|tjddt_tt|tt_tttd_|dkrd nd t f d d t| D_t | |_!dkrȈddksJ}t"|d d dd_#|dks|dkrt$d|dkd_%&'|jdS)a Initialize the diffusion model backbone. Args: model_type (`str`, *optional*, defaults to 't2v'): Model variant - 't2v' (text-to-video) or 'i2v' (image-to-video) or 'flf2v' (first-last-frame-to-video) patch_size (`tuple`, *optional*, defaults to (1, 2, 2)): 3D patch dimensions for video embedding (t_patch, h_patch, w_patch) text_len (`int`, *optional*, defaults to 512): Fixed length for text embeddings in_dim (`int`, *optional*, defaults to 16): Input video channels (C_in) dim (`int`, *optional*, defaults to 2048): Hidden dimension of the transformer ffn_dim (`int`, *optional*, defaults to 8192): Intermediate dimension in feed-forward network freq_dim (`int`, *optional*, defaults to 256): Dimension for sinusoidal time embeddings text_dim (`int`, *optional*, defaults to 4096): Input dimension for text embeddings out_dim (`int`, *optional*, defaults to 16): Output video channels (C_out) num_heads (`int`, *optional*, defaults to 16): Number of attention heads num_layers (`int`, *optional*, defaults to 32): Number of transformer blocks window_size (`tuple`, *optional*, defaults to (-1, -1)): Window size for local attention (-1 indicates global attention) qk_norm (`bool`, *optional*, defaults to True): Enable query/key normalization cross_attn_norm (`bool`, *optional*, defaults to False): Enable cross-attention normalization eps (`float`, *optional*, defaults to 1e-6): Epsilon value for normalization layers concat_padding_mask (`bool`, *optional*, defaults to False): Enable concat padding mask cp_comm_type (str, *optional*, defaults to 'p2p'): CP communication type passed to TE. attention_backend (str, defaults to 'transformer_engine', options are: ['transformer_engine', 'minimal_a2a']) Backend used for attention )ri2vflf2vr,rrQrrrPrrrc s(g|]}t j qSr$)rr).0ru rrrr[rrrrr!rr$r% s z%WanModel.__init__..r)rIrJrKrLr"r#r)rN)(rr model_typer text_lenrr[rfreq_dimtext_dimr r num_layersrrrrr rZattention_backedrrpatch_embeddingrrtext_embeddingSiLUtime_embeddingtime_projection ModuleListrangeblocksrr rGrope_position_embeddingrimg_embrenable_selective_checkpoint)r!r(r r)rr[rr*r+r rr,rrrrr r!rrrr"r%r%rsX @ &($  zWanModel.__init__N padding_maskc  Ks|jddks J|dddf} ~ |jdks|jdkr&|dur$|dus&J|dur3tj||gdd}|jr^tjj|t|jddtj j d}tj|| d dd|jd ddgdd}t |d |jd|jd|jd d } || } t| jd| jd | jd d } t | d}tjdd|Dtjd}|}||ksJtjdtjd-|t|j| }||dd|jf}|jtjkr|jtjksJWdn1swYd}| |}|dur|!|}tj"||gdd}t#||| |$| ||d} t%|j&D]\}}| dur || vr |r q ||fi| }q |'||}| \}}}t |d|jd|jd|jd ||||j(d }|S)aW Forward pass through the diffusion model Args: x_B_C_T_H_W (Tensor): Input video tensor with shape [B, C_in, T, H, W] t (Tensor): Diffusion timesteps tensor of shape [B] context (List[Tensor]): List of text embeddings each with shape [L, C] seq_len (`int`): Maximum sequence length for positional encoding frame_cond_crossattn_emb_B_L_D (Tensor, *optional*): CLIP image features for image-to-video mode or first-last-frame-to-video mode y_B_C_T_H_W (Tensor, *optional*): Conditional video inputs for image-to-video mode, shape [B, C_in, T, H, W] Returns: Tensor: Denoised video tensor with shape [B, C_out, T, H / 8, W / 8] r,Nrr"r#rl) interpolationrQz0b c (t kt) (h kh) (w kw) -> b t h w (c kt kh kw))ktkhkwrrzb t h w d -> b (t h w) dcSsg|]}|dqS)r)r)r$ur$r$r%r&Usz$WanModel.forward..rrbrP)rrrrrrz2b (t h w) (nt nh nw d) -> b d (t nt) (h nh) (w nw))ntnhnwrjrhrir))r1r(rCrnr r functionalresizelistInterpolationModeNEARESTrrrr r-rtensorlongr`itemrrrr0rr*rar1 unflattenr[rr.r6concatdictr5 enumerater4r r )r! x_B_C_T_H_W timesteps_B_T crossattn_embrframe_cond_crossattn_emb_B_L_D y_B_C_T_H_Wr8Z is_uncondZ slg_layerskwargst_B x_B_T_H_W_DrZx_B_L_DrZe_B_DZe0_B_6_Dr context_B_L_DZ context_clip block_idxblockrjrhrir$r$r%r;s~""          zWanModel.forwardcCs|D]}t|tjrtj|j|jdurtj|jq|j D]}| q#|j tj|j j dtj|j j|jD]}t|tjrdtjj|jdd|jdurdtj|jqG|jD]}t|tjrtjj|jdd|jdurtj|jqj|jD]}t|tjrtjj|jdd|jdurtj|jqtj|j j j|j j jdurtj|j j jdSdS)zJ Initialize model parameters using Xavier initialization. Nr,g{Gz?r)modules isinstancerrrxavier_uniform_rrzeros_r4rr r-rr.normal_r0r1)r!mrXr$r$r%rsB            zWanModel.init_weightscCsft|jD] \}}t||ddqt|j|ddt|j|ddt|j|ddt|j|dddS)NT)meshreshard_after_forwardF)rMr4r r r.r0r-)r!r_irXr$r$r%r s zWanModel.fully_shardcCs8|j|jD]}|jjddtjdqd|_dS)Nr&rrF) r5r*r4rrrCrbStream_is_context_parallel_enabled)r!rXr$r$r%r*s   z!WanModel.disable_context_parallelr&cCsD|jj|dt|}|jD]}|jj||tjdqd|_ dS)N)r&rbT) r5r(r r4rrrCrbrcrd)r!r&r6rXr$r$r%r(s  z WanModel.enable_context_parallelcCs|jSr)rdr r$r$r%is_context_parallel_enabledsz$WanModel.is_context_parallel_enabledr4c Cs|jtjkr td|jd|jdt||}|D]"\}}t ||jdkrCtd|t ||dd}| ||q!| dt |j |dddS) Nz!Enable selective checkpoint with z , for every z blocks. Total blocks: rz&Enable selective checkpoint for block F) context_fnpreserve_rng_stater ) moderNONErinfoevery_n_blocksr2get_context_fnnamed_childrenrvptd_checkpoint_wrapperregister_moduler )r!r!r4 _context_fnblock_idrXr$r$r%r7s0  z$WanModel.enable_selective_checkpointr0c si}|D]\}}d|vrtd|dq|||<q|}tj|d|d\}}|durDtdd|Ds9Jtd d|DsDJt||S) N _extra_statez Skipping key z; introduced by TransformerEngine for FP8 in the checkpoint.F)strictassignTcs|]}d|vVqdSrrNr$r$rr$r$r% z+WanModel.load_state_dict..csrurvr$rwr$r$r%rxry)itemsrwarningrload_state_dictallr ) r! state_dictrsrtZfiltered_state_dictrr missing_keysunexpected_keysr"r$r%r|s  zWanModel.load_state_dict)NNNNFNr)TF)r?r@rA__doc__rboolrrrrCrDr;rr r*rr(rBrerr2r7r r|rFr$r$r"r%rs` s+  r)ArtypingrrCZ torch.amprtorch.nnreinopsrr-cosmos_predict2._src.predict2.networks.a2a_cpr0cosmos_predict2._src.predict2.networks.attentionrflash_attn.layers.rotaryrr ImportErrorprinttorch.distributedrr "torch.distributed._composable.fsdpr ;torch.distributed.algorithms._checkpoint.checkpoint_wrapperr rntorch.nn.modules.moduler torchvisionr $transformer_engine.pytorch.attentionr%cosmos_predict2._src.imaginaire.utilsr6cosmos_predict2._src.imaginaire.utils.context_parallelr:cosmos_predict2._src.predict2.networks.model_weights_statsrFcosmos_predict2._src.predict2.networks.selective_activation_checkpointrrrr collectionsrrModulerrGrrrrrrrrrrrrrrr$r$r$r%s^                  'b    `KW%