o ݱi!@sddlZddlZddlZddlmZddlmZGdddejZ GdddejZ GdddejZ Gd d d ejZ Gd d d ejZ Gd ddejZdddZdddZddZdddZddZddZdS) N) rearrangecs$eZdZfddZddZZS)SinusoidalPosEmbcst||_dS)N)super__init__dim)selfr __class__?/data/cameron/vidgen/unified-world-model/models/common/utils.pyr s  zSinusoidalPosEmb.__init__cCsv|jd}td|d}ttj||jd| }|dddf|dddf}tj|| fdd}|S)N')devicer) rmathlogtorchexparangercatsincos)rxhalf_dimembr r r forwards  zSinusoidalPosEmb.forward)__name__ __module__ __qualname__rr __classcell__r r rr r s rcs*eZdZdZdfdd ZddZZS) GaussianFourierEmbz-Gaussian Fourier embeddings for noise levels.TcsHt||_||_||_t|jd|j}tj||jd|_ dS)Nr ) requires_grad) rrrscale trainablerrandnnn ParameterW)rrr%r&ZW_initrr r rs zGaussianFourierEmb.__init__cCsF|dddf|jdddfdtj}tj||gddS)Nr rr)r*nppirrrr)rrx_projr r r r$s,zGaussianFourierEmb.forward)r#Trrr __doc__rrr!r r rr r"s r"c0eZdZdZ   d fdd ZddZZS) PatchEmbed2Dz Convert image to patch embeddingr#cs*ttj||||f||fd|_dSN in_channels out_channels kernel_sizestride)rrr(Conv2dproj)r patch_size num_chans embed_dimrr r r,s  zPatchEmbed2D.__init__cC||ddd}|SNr rr;flatten transposerrr r r r:zPatchEmbed2D.forward)r#r2r3r.r r rr r1)r1cr0) PatchEmbed3D Convert video to patch embeddingr r#r#r2r3cs"ttj||||d|_dSr4)rrr(Conv3dr;)r patch_shaper=r>rr r rDs  zPatchEmbed3D.__init__cCr?r@rArDr r r rRrEzPatchEmbed3D.forward)rIr2r3r.r r rr rGArFrGcs8eZdZdZdddgddgffdd Zd d ZZS) MultiLayerPatchEmbed3DrHr2ir3)r rM)r rNc sztt}|}tt||D]#\}\}}|tj||||d|t|dkr2|t |}qtj ||_ dS)Nr5r) rrlist enumeratezipappendr(rJlenGELU Sequentialr;) rr= embed_dimsZ patch_shapeslayersin_chansirKZ out_chansrr r r\s zMultiLayerPatchEmbed3D.__init__cCr?r@rArDr r r rsrEzMultiLayerPatchEmbed3D.forwardr.r r rr rLYsrLcs2eZdZdZ    d fdd Zdd ZZS) MultiviewPatchEmbed3Dz4Convert video with multiple views to patch embeddingrIrr2r3cs2ttj|||||||d|_||_dS)N)r6r7r8r9groups)rrr(rJr; num_views)rrKr\r=r>rr r r}s  zMultiviewPatchEmbed3D.__init__cCs(t|d}||}t|d|jd}|S)NzB V C T H W -> B (V C) T H WzB (V D) t h w -> B (V t h w) D)V)rr;r\rDr r r rs  zMultiviewPatchEmbed3D.forward)rIrr2r3r.r r rr rZzsrZFr c stt|}||dks J||ddksJdd|D}tj|ddi}fdd|D}tj|dd }|S) arCreate n-dimensional rotary positional embeddings. Args: dim: an int of the embedding dimension. grid_shape: a sequence of int of the length along each axis. base: the base from which to calculate the rotation angles. Returns: pos_embed: a tensor of shape (grid_shape[0]*...*grid_shape[N-1], dim) of positional embeddings. rr cSsg|] }t|qSr )rrfloat.0lengthr r r z'get_nd_rotary_embed..indexingijcsg|] }t|qSr )get_1d_rotary_embedrB)r`Z axis_gridaxis_dimbaser r rbsrr)rSrmeshgridr) r grid_shape cls_tokenrinum_axis axis_ticks axis_gridsZ axis_thetasthetasr rgr get_nd_rotary_embeds  rqcCsJ|ddksJd|td|d|}t||}|dd}|S)a)Create 1D rotary positional embeddings from a grid of positions. Args: dim: the output dimension for each position. pos: a tensor of size (seq_len,) of positions to be encoded. Returns: thetas: a tensor of size (seq_len, dim) of rotary positional embeddings. r r?r)rrr^outerrepeat)rposrirpr r r rfs    rfcCs^|jdd|jddksJ|jddd\}}tj| |gdd}||||S)a&Rotates the input tensors by the positional embeddings. Args: x: a tensor of shape (..., seq_len, dim). thetas: a tensor of shape (..., seq_len, dim) of positional embeddings. Returns: x: a tensor of shape (..., seq_len, dim) of the rotated input tensors. Nr rr)shapechunkrrrr)rrpx1x2Z x_rotate_halfr r r apply_rotary_embeds r{cst|}tt|d|ddd|D}tj|ddi}fdd|D}tj|dd}|d d d |f}|rJtjtd|f|gd d}|S) a2Create n-dimensional sinusoidal positional embeddings. Args: dim: an int of the embedding dimension. grid_shape: a sequence of int of the length along each axis. Returns: pos_embed: an array of shape (grid_shape[0]*...*grid_shape[N-1], dim) of positional embeddings. r cSsg|] }tj|tdqS)dtype)r+rr^r_r r r rbrcz+get_nd_sinusoidal_embed..rdrecsg|] }t|dqS)r)get_1d_sinusoidal_embedreshape)r`axisrhr r rbsrrNr)rSintr+ceilrj concatenatezeros)rrkrlrmrnroZ axis_embeds pos_embedr rr get_nd_sinusoidal_embeds  rcCsr|ddksJtj|dtd}||d}dd|}td||}t|}t|}tj||gdd }|S) a!Create 1D sinusoidal positional embeddings from a grid of positions. Args: dim: the output dimension for each position. pos: a list of size (seq_len,) of positions to be encoded. Returns: emb: an array of size (seq_len, dim) of positional embeddings. r rr|g@rrr zm,d->mdrr)r+rr^einsumrrr)rruomegaoutemb_sinemb_cosrr r r r~s     r~cCst|tjr)tjj|jdddt|tjr%|jdur'tj|jddSdSdSt|tjrI|j rKtj|jd|jdurMtj|jddSdSdSdS)Ngg{Gz?)meanstdrrr) isinstancer(Linearinitnormal_weightbias constant_ LayerNormelementwise_affine)mr r r init_weights s  r)Fr )r )F)rnumpyr+rtorch.nnr(einopsrModulerr"r1rGrLrZrqrfr{rr~rr r r r s"  !