o 㮬i(@sddlZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddlm mZddlmZddlmZddlmZddlmZmZmZmZmZmZdd lmZdd l m!Z!e"e#Z$Gd d d e j%Z&Gd dde j%Z'Gddde j(e'Z)Gddde j%Z*Gddde j%Z+Gddde'Z,Gddde j%Z-Gddde j%Z.Gddde j%Z/Gddde j%Z0Gdd d e j%Z1dS)!N)abstractmethod)IterableListOptionalTupleUnion) rearrange) checkpoint)SpatialTransformer) avg_pool_ndconv_ndlinear normalizationtimestep_embedding zero_module)SpatialVideoTransformer)existsc sPeZdZdZ d dedededeeffdd Zd ejd ejfd d Z Z S)AttentionPool2dzS Adapted from CLIP: https://github.com/openai/CLIP/blob/main/clip/model.py N spacial_dim embed_dimnum_heads_channels output_dimcspttt||dd|d|_td|d|d|_td||p'|d|_ |||_ t |j |_ dS)Ng?r ) super__init__nn Parameterthrandnpositional_embeddingr qkv_projc_proj num_heads QKVAttention attention)selfrrrr __class__R/data/cameron/vidgen/generative-models/sgm/modules/diffusionmodules/openaimodel.pyrs  zAttentionPool2d.__init__xreturncCs|j\}}}|||d}tj|jddd|gdd}||jdddddf|j}||}| |}| |}|dddddfS)NT)dimkeepdimr/r) shapereshapercatmeanr!todtyper"r&r#)r'r,bc_r*r*r+forward+s $   zAttentionPool2d.forwardN) __name__ __module__ __qualname____doc__intrrrTensorr; __classcell__r*r*r(r+rs rc@s*eZdZdZedejdejfddZdS) TimestepBlockzT Any module where forward() takes timestep embeddings as a second argument. r,embcCsdS)zJ Apply the module to `x` given `emb` timestep embeddings. Nr*r'r,rEr*r*r+r;;szTimestepBlock.forwardN)r=r>r?r@rrrBr;r*r*r*r+rD6srDc@seZdZdZ         ddejdejdeejdeejdeejdeejd eejd eed eed eed eefddZ dS)TimestepEmbedSequentialzt A sequential module that passes timestep embeddings to the children that support it as an extra input. Nr,rEcontextcamimage_only_indicator cond_view cond_motion time_contextnum_video_frames time_stepnamec  Csddlm} m} ddlm}m}m}|D]t}|}t|||fr-|||||| |||| | }qt||frA|||||| |||| | }qt|trO||||| |}qt|| r\|||| |}qt|| ri|||| |}qt|t ryt|| sy|||}qt|t r|||}q||}q|S)Nr ) VideoResBlockPostHocResBlockWithTime)BasicTransformerTimeMixBlock'PostHocSpatialTransformerWithTimeMixing0PostHocSpatialTransformerWithTimeMixingAndMotion) $modules.diffusionmodules.video_modelrQrRmodules.spacetime_attentionrSrTrU isinstancerrDr )r'r,rErHrIrJrKrLrMrNrOrPrQrRrSrTrUlayermoduler*r*r+r;Hsv         zTimestepEmbedSequential.forward) NNNNNNNNN) r=r>r?r@rrBrrAstrr;r*r*r*r+rGBsD     rGcsjeZdZdZ      ddeded ed eed ed ed edeffdd Zdej dej fddZ Z S)UpsampleaA An upsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determining if a convolution is applied. :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then upsampling occurs in the inner-two dimensions. rNrFr channelsuse_convdims out_channelspaddingthird_up kernel_size scale_factorc sVt||_|p ||_||_||_||_||_|r)t||j|j||d|_ dSdS)Nra) rrr]r`r^r_rbrdr conv) r'r]r^r_r`rarbrcrdr(r*r+rs   zUpsample.__init__r,r-cCs|jd|jks J|jdkr4|jsdn|j}tj|||jd|jd|j|jd|jfdd}n tj||jdd}|jrE||}|S)Nrr rnearest)mode)rdri) r2r]r_rbrdF interpolater^rf)r'r,t_factorr*r*r+r;s    zUpsample.forward)rNrFr r r=r>r?r@rAboolrrrrBr;rCr*r*r(r+r\s4  r\c s^eZdZdZ    ddededed eed ed ef fd d Zdej dej fddZ Z S) DownsampleaD A downsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determining if a convolution is applied. :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then downsampling occurs in the inner-two dimensions. rNrFr]r^r_r`ra third_downc st||_|p ||_||_||_|dkrdn|sdnd}|rXtd|dtd|jd|jd |d ||dkrItd |t||j|jd||d |_ dS|j|jks`Jt |||d |_ dS)Nr r)rrr)rrrz!Building a Downsample layer with z dims.z --> settings are: in-chn: z , out-chn: z, kernel-size: 3, stride: z , padding: z& --> Downsampling third axis (time): )stridera)rcrq) rrr]r`r^r_logpyinfor opr )r'r]r^r_r`rarprqr(r*r+rs6    zDownsample.__init__r,r-cCs|jd|jks J||S)Nr)r2r]rt)r'r,r*r*r+r;s zDownsample.forward)rNrFrmr*r*r(r+ros( #rocseZdZdZ          ddededed eed ed ed ed edededededeffdd Zde j de j de j fddZ de j de j de j fddZ Z S)ResBlocka A residual block that can optionally change the number of channels. :param channels: the number of input channels. :param emb_channels: the number of timestep embedding channels. :param dropout: the rate of dropout. :param out_channels: if specified, the number of out channels. :param use_conv: if True and out_channels is specified, use a spatial convolution instead of a smaller 1x1 convolution to change the channels in the skip connection. :param dims: determines if the signal is 1D, 2D, or 3D. :param use_checkpoint: if True, use gradient checkpointing on this module. :param up: if True, use this block for upsampling. :param down: if True, use this block for downsampling. NFrr r] emb_channelsdropoutr`r^use_scale_shift_normr_use_checkpointupdownrcexchange_temb_dims skip_t_embc st||_||_||_|p||_||_||_||_| |_ t | t r,dd| D}n| d}t t|t t|||j| |d|_| pG| |_| rZt|d||_t|d||_n| rkt|d||_t|d||_nt |_|_| |_|r}d|jn|j|_|jrtd|jj|jrJd|_d|_ n t t t||j|_t t|jt t j |dt!t||j|j| |d|_"|j|krt |_#dS|rt|||j| |d|_#dSt|||jd|_#dS) NcSsg|]}|dqS)rr*).0kr*r*r+ /sz%ResBlock.__init__..rreFzSkipping timestep embedding in )pr)$rrr]rvrwr`r^ryrxr|rXrr SequentialrSiLUr in_layersupdownr\h_updx_updroIdentityr}emb_out_channelsrrrsr)r= emb_layersrDropoutr out_layersskip_connection)r'r]rvrwr`r^rxr_ryrzr{rcr|r}rar(r*r+rsz         zResBlock.__init__r,rEr-cCs |jr t|j||S|||S)a Apply the block to a Tensor, conditioned on a timestep embedding. :param x: an [N x C x ...] Tensor of features. :param emb: an [N x emb_channels] Tensor of timestep embeddings. :return: an [N x C x ...] Tensor of outputs. )ryr _forwardrFr*r*r+r;ns zResBlock.forwardc Csn|dur:|jd|jdkr:|jd|jddkr:|jd|jd}|j|jd|g|jddRjdd}|jr]|jdd|jd}}||}||}||}||}n||}|jrkt |}n | | |j }|jd|jdkr|jd|jddkr|jd|jd}|j|jd|g|jddRjdd}n1|jd|jddkr|j |jd|jddd}n|ddj|jdg|jddR}t|jt|jkr|d}t|jt|jks|jr|jd|jdd}} tj|ddd\} } ||d| | }| |}n|jr't|d}||}||}|||S)Nrrr1r.).Nrzb t c ... -> b c t ...)r2r3r5rrrrr}r zeros_likertyper7repeat_interleaveexpandlenrxrchunkr|rr) r'r,rEnin_restin_convhemb_outout_normout_restscaleshiftr*r*r+rzsB4,     . *   zResBlock._forward) NFFrFFFr FF)r=r>r?r@rAfloatrrnrrrBr;rrCr*r*r(r+rusR     Z$ ruc sleZdZdZ    ddedededed ef fd d Zd ejd ejfddZ d ejd ejfddZ Z S)AttentionBlocka An attention block that allows spatial positions to attend to each other. Originally ported from here, but adapted to the N-d case. https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/models/unet.py#L66. rr.Fr]r$num_head_channelsryuse_new_attention_ordercst||_|dkr||_n||dks Jd|d||||_||_t||_td||dd|_|r@t |j|_ nt |j|_ t td||d|_ dS)Nr.rzq,k,v channels z' is not divisible by num_head_channels rr )rrr]r$ryrnormr qkvr%r&QKVAttentionLegacyrproj_out)r'r]r$rryrr(r*r+rs    zAttentionBlock.__init__r,r-cKs t|j|Sr<)r r)r'r,kwargsr*r*r+r; zAttentionBlock.forwardcCsV|j^}}}|||d}|||}||}||}||j||g|RS)Nr.)r2r3rrr&r)r'r,r8r9spatialrrr*r*r+rs   zAttentionBlock._forward)rr.FF) r=r>r?r@rArnrrrBr;rrCr*r*r(r+rs& rc<eZdZdZdeffdd ZdejdejfddZZ S) rzh A module which performs QKV attention. Matches legacy QKVAttention + input/ouput heads shaping n_headsct||_dSr<rrrr'rr(r*r+r  zQKVAttentionLegacy.__init__rr-c Cs|j\}}}|d|jdksJ|d|j}|||j|d|j|dd\}}}dtt|} td|| || } tj| dd | j } td| |} | |d|S)z Apply QKV attention. :param qkv: an [N x (H * 3 * C) x T] tensor of Qs, Ks, and Vs. :return: an [N x (H * C) x T] tensor after attention. r rrr1 bct,bcs->btsr. bts,bcs->bct) r2rr3splitmathsqrtreinsumsoftmaxrrr7 r'rbswidthlengthchqrvrweightar*r*r+r;s (zQKVAttentionLegacy.forward r=r>r?r@rArrrBr;rCr*r*r(r+rrcr) r%zP A module which performs QKV attention and splits in a different order. rcrr<rrr(r*r+rrzQKVAttention.__init__rr-c Cs|j\}}}|d|jdksJ|d|j}|jddd\}}}dtt|} td|| ||j|||| ||j||} tj| dd | j } td| | ||j||} | |d|S)z Apply QKV attention. :param qkv: an [N x (3 * H * C) x T] tensor of Qs, Ks, and Vs. :return: an [N x (H * C) x T] tensor after attention. r rrr1rr.r) r2rrrrrrviewrrrr7r3rr*r*r+r;s zQKVAttention.forwardrr*r*r(r+r%rr%cs8eZdZdeffdd ZdejdejfddZZS)Timestepr/crr<)rrr/)r'r/r(r*r+rrzTimestep.__init__tr-cCs t||jSr<)rr/)r'rr*r*r+r;rzTimestep.forward) r=r>r?rArrrBr;rCr*r*r(r+rsrc4seZdZdZ                  d-d ed ed ededededeeefde dede eee fde dededede de dede ede ee de eede d e d!e d"e d#e ef2fd$d% Z   d.d&e jd'e e jd(e e jd)e e jd*e jf d+d,ZZS)/ UNetModela The full UNet model with attention and timestep embedding. :param in_channels: channels in the input Tensor. :param model_channels: base channel count for the model. :param out_channels: channels in the output Tensor. :param num_res_blocks: number of residual blocks per downsample. :param attention_resolutions: a collection of downsample rates at which attention will take place. May be a set, list, or tuple. For example, if this contains 4, then at 4x downsampling, attention will be used. :param dropout: the dropout probability. :param channel_mult: channel multiplier for each level of the UNet. :param conv_resample: if True, use learned convolutions for upsampling and downsampling. :param dims: determines if the signal is 1D, 2D, or 3D. :param num_classes: if specified (as an int), then this model will be class-conditional with `num_classes` classes. :param use_checkpoint: use gradient checkpointing to reduce memory usage. :param num_heads: the number of attention heads in each attention layer. :param num_heads_channels: if specified, ignore num_heads and instead use a fixed channel width per attention head. :param num_heads_upsample: works with num_heads to set a different number of heads for upsampling. Deprecated. :param use_scale_shift_norm: use a FiLM-like conditioning mechanism. :param resblock_updown: use residual blocks for up/downsampling. :param use_new_attention_order: use a different attention pattern for potentially increased efficiency. rrrgTrNFr.rr in_channelsmodel_channelsr`num_res_blocksattention_resolutionsrw channel_mult conv_resampler_ num_classesryr$rnum_heads_upsamplerxresblock_updowntransformer_depth context_dimdisable_self_attentionsnum_attention_blocksdisable_middle_self_attndisable_middle_transformeruse_linear_in_transformerspatial_transformer_attn_typeadm_in_channelsc(st|dkr | }| dkr| dksJd| dkr#| dks#Jd|_|_|_t|tr8t||g}|d}t|trJt||g_nt|t|krVt d|_|durgt|t|ksgJdurttjksvJt t fddt tsJt dd|d|_|_|_|_| _| _| _| _|_|d }tt||tt||_jdur)tjtrt| |_nRjd krt d td |_n@jd krtt |tt||tt||_n#jdkr'|dusJttt||tt||_nt t!t"t#| ||dd dg_$|_%|g}|}d }t&|D]\}} t j|D]m}!t'|||| || | |dg}"| |}||vr| dkru|| }#n|| } | }#|durt(|r||}$nd}$t(r|!|kr|")t*|| |#||||$||| d j$)t"|"j%|7_%|)|qR|t|d kr|}%j$)t"|rt'||||%| | |ddnt+||| |%d|%}|)||d9}j%|7_%qG| dkr|| }#n|| } | }#t"t'||||| | |d|s't*|| |#|||||| d nt,j-t'|||| | |d_.j%|7_%t!g_/t0t&|dddD]\}} t j|d D]}&|1}'t'||'|||| | | |dg}"|| }||vr| dkr|| }#n|| } | }#t(|r||}$nd}$t(r|&|kr|")t*|| |#||||$||| d |r|&j|kr|}%|")|rt'||||%| | |ddnt2||| |%d|d}j/)t"|"j%|7_%q\qOtt3|tt4t#| ||dd d_5dS)Nr.z3Either num_heads or num_head_channels has to be setzprovide num_res_blocks either as an int (globally constant) or as a list/tuple (per-level) with the same length as channel_multcsj||kSr<)r)irr'r*r+wsz$UNetModel.__init__..z7Constructor of UNetModel received num_attention_blocks=z;. This option has LESS priority than attention_resolutions zz, i.e., in cases where num_attention_blocks[i] > 0 but 2**i not in attention_resolutions, attention will still not be set.rg continuousz'setting up linear c_adm embedding layerrtimestep sequentialr re)r`r_ryrxF)depthrdisable_self_attn use_linear attn_typeryT)r`r_ryrxr{)r_r`r)r_ryrx)r`r_ryrxrz)6rrrrr`rXrArr ValueErrorallmaprangerrrsrrwrrrryr$rrrrrr time_embed Embedding label_embLinearr ModuleListrGr input_blocks _feature_size enumeraterurappendr rorr middle_block output_blockslistpopr\rrout)(r'rrr`rrrwrrr_rryr$rrrxrrrrrrrrrrtransformer_depth_middletime_embed_diminput_block_chansrdslevelmultnrlayersdim_head disabled_saout_chrichr(rr+r5s                                 ? zUNetModel.__init__r, timestepsrHyr-c Ks|du|jduks Jdg}t||jdd}||}|jdur4|jd|jdks-J|||}|} |jD] } | | ||} || q9|| ||} |j D]} t j | | gdd} | | ||} qQ| |j} || S)aQ Apply the model to an input batch. :param x: an [N x C x ...] Tensor of inputs. :param timesteps: a 1-D batch of timesteps. :param context: conditioning plugged in via crossattn :param y: an [N] Tensor of labels, if class-conditional. :return: an [N x C x ...] Tensor of outputs. Nzr?r@rArrrrrnrr[rrrBr;rCr*r*r(r+rs$        ?r)2loggingrabcrtypingrrrrrtorchrtorch.nnrtorch.nn.functional functionalrjeinopsrtorch.utils.checkpointr modules.attentionr modules.diffusionmodules.utilr r rrrrmodules.video_attentionrutilr getLoggerr=rrModulerrDrrGr\rorurrr%rrr*r*r*r+s4           [52!0