ܱi]ddlmZddlmZmZmZmZddlZddlm Z ddl m Z m Z ddl mZddlmZmZddlmZmZmZddlmZmZdd lmZdd lmZmZmZeje Z!eGd d eZ"Gd dee eZ#dS)) dataclass)DictOptionalTupleUnionN) ConfigMixinregister_to_config)UNet2DConditionLoadersMixin) BaseOutputlogging)CROSS_ATTENTION_PROCESSORSAttentionProcessor AttnProcessor)TimestepEmbedding Timesteps) ModelMixin)UNetMidBlockSpatioTemporalget_down_block get_up_blockc,eZdZUdZdZejed<dS)!UNetSpatioTemporalConditionOutputa The output of [`UNetSpatioTemporalConditionModel`]. Args: sample (`torch.Tensor` of shape `(batch_size, num_frames, num_channels, height, width)`): The hidden states output conditioned on `encoder_hidden_states` input. Output of last layer of model. Nsample)__name__ __module__ __qualname____doc__rtorchTensor__annotations__H/data/cameron/vidgen/Ctrl-World/models/unet_spatio_temporal_condition.pyrrs0 FELr!rc&eZdZdZdZe d.deedededee dee deededede eeefde eeefde eeeeefde eeefdeffd Z e de e effdZd e ee e efffd!Zd"Zd/d$eed%eddfd&Z d0d(ejd)e ejeefd*ejd+ejd,ede eeff d-ZxZS)1 UNetSpatioTemporalConditionModela A conditional Spatio-Temporal UNet model that takes a noisy video frames, conditional state, and a timestep and returns a sample shaped output. This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Parameters: sample_size (`int` or `Tuple[int, int]`, *optional*, defaults to `None`): Height and width of input/output sample. in_channels (`int`, *optional*, defaults to 8): Number of channels in the input sample. out_channels (`int`, *optional*, defaults to 4): Number of channels in the output. down_block_types (`Tuple[str]`, *optional*, defaults to `("CrossAttnDownBlockSpatioTemporal", "CrossAttnDownBlockSpatioTemporal", "CrossAttnDownBlockSpatioTemporal", "DownBlockSpatioTemporal")`): The tuple of downsample blocks to use. up_block_types (`Tuple[str]`, *optional*, defaults to `("UpBlockSpatioTemporal", "CrossAttnUpBlockSpatioTemporal", "CrossAttnUpBlockSpatioTemporal", "CrossAttnUpBlockSpatioTemporal")`): The tuple of upsample blocks to use. block_out_channels (`Tuple[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): The tuple of output channels for each block. addition_time_embed_dim: (`int`, defaults to 256): Dimension to to encode the additional time ids. projection_class_embeddings_input_dim (`int`, defaults to 768): The dimension of the projection of encoded `added_time_ids`. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. cross_attention_dim (`int` or `Tuple[int]`, *optional*, defaults to 1280): The dimension of the cross attention features. transformer_layers_per_block (`int`, `Tuple[int]`, or `Tuple[Tuple]` , *optional*, defaults to 1): The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for [`~models.unets.unet_3d_blocks.CrossAttnDownBlockSpatioTemporal`], [`~models.unets.unet_3d_blocks.CrossAttnUpBlockSpatioTemporal`], [`~models.unets.unet_3d_blocks.UNetMidBlockSpatioTemporal`]. num_attention_heads (`int`, `Tuple[int]`, defaults to `(5, 10, 10, 20)`): The number of attention heads. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. TN CrossAttnDownBlockSpatioTemporalr(r(DownBlockSpatioTemporalUpBlockSpatioTemporalCrossAttnUpBlockSpatioTemporalr,r,i@ir. r7 sample_size in_channels out_channelsdown_block_typesup_block_typesblock_out_channelsaddition_time_embed_dim%projection_class_embeddings_input_dimlayers_per_blockcross_attention_dimtransformer_layers_per_blocknum_attention_heads num_framesc t||_t|t|krt d|d|dt|t|krt d|d|dt | t s6t| t|krt d| d|dt | tr6t| t|krt d| d|dt | t s6t| t|krt d| d|dtj ||d d d |_ |d d z}t|d dd |_ |d }t|||_t|dd |_t|||_tjg|_tjg|_t | t r| ft|z} t | t r| ft|z} t | t r| gt|z} t | t r| gt|z} |}|d }t)|D]r\}}|}||}|t|d z k}t+|| || ||||| d| || |d }|j|st/|d|| d| d| d|_d |_tt5|}tt5| }tt5| }tt5| }tt5| }|d }t)|D]\}}|t|d z k}|}||}|t7|d zt|d z }|sd}|xjd z c_nd}t9|||d z|||||||d|||||d }|j||}tj|d dd|_tj|_ tj |d |d d |_!dS)Nz\Must provide the same number of `down_block_types` as `up_block_types`. `down_block_types`: z. `up_block_types`: .zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: zdMust provide the same number of `cross_attention_dim` as `down_block_types`. `cross_attention_dim`: z^Must provide the same number of `layers_per_block` as `down_block_types`. `layers_per_block`: rr3) kernel_sizepaddingr&T)downscale_freq_shiftgh㈵>silu) num_layersrCr:r; temb_channelsadd_downsample resnet_epsrBrD resnet_act_fn)rNrCrBrDF) rMrCr:r;prev_output_channelrN add_upsamplerPresolution_idxrBrDrQ ) num_channels num_groupseps)"super__init__r9len ValueError isinstanceintlistnnConv2dconv_inr time_projrtime_embedding add_time_proj add_embedding ModuleList down_blocks up_blocks enumeraterappendr mid_blocknum_upsamplersreversedminr GroupNorm conv_norm_outSiLUconv_actconv_out)!selfr9r:r;r<r=r>r?r@rArBrCrDrEtime_embed_dimtimestep_input_dimblocks_time_embed_dimoutput_channelidown_block_type input_channelis_final_block down_blockreversed_block_out_channelsreversed_num_attention_headsreversed_layers_per_blockreversed_cross_attention_dim%reversed_transformer_layers_per_block up_block_typerSrTup_block __class__s! r"r[z)UNetSpatioTemporalConditionModel.__init__Fs6 &  C$7$7 7 7goggVdggg  ! " "c*:&;&; ; ;svHss`psss -s33 [\]]]./TVdee=,,r** )3 / / Q#6"83?O;P;P"P  )3 / / Q#6"83?O;P;P"P  & , , J 01C8H4I4II  2C 8 8 b,H+ICP`LaLa+a ( .,A."+,<"="= 0 0 A*M/2N#&8"9"9A"==N'+A.-I!-L)+3#11$7$:$7$:$   J   # #J / / / /4 r "/)Eb)I 3B 7 3B 7     '+84F+G+G&H&H#'+H5H,I,I'J'J$$(2B)C)C$D$D!'+H5H,I,I'J'J$04X>Z5[5[0\0\-4Q7 ). 9 9 1 1 A}#&8"9"9A"==N"0 8;N7AE3GYCZCZ]^C^8_8_`M" %# ##q(###$ #4Q7!;-RST-U)+$73) $@$C$@$C$H N ! !( + + +"0   \7I!7LY[aefff   q !      r!returnci}dtdtjjdttt fffd |D]\}}||||S)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namemodule processorsct|dr|||d<|D]\}}|d||||S)N get_processor .processorrG)hasattrrnamed_children)rrrsub_namechildfn_recursive_add_processorss r"rzUUNetSpatioTemporalConditionModel.attn_processors..fn_recursive_add_processorss v// I282F2F2H2H d.../#)#8#8#:#: U U%++t,@,@h,@,@%TTTT r!)strrraModulerrr)rvrrrrs @r"attn_processorsz0UNetSpatioTemporalConditionModel.attn_processorss   HO S"445      !//11 B BLD& ' 'fj A A A Ar! processorc zt|j}t|tr9t||kr&t dt|d|d|ddt dtjj ffd | D]\}}|||dS) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrc,t|drVt|ts||n+|||d|D]\}}|d|||dS)N set_processorrrG)rr^dictrpopr)rrrrrfn_recursive_attn_processors r"rzXUNetSpatioTemporalConditionModel.set_attn_processor..fn_recursive_attn_processor)sv// M!)T22M((3333(($7J7J7J)K)KLLL#)#8#8#:#: T T%++t,@,@h,@,@%SSSS T Tr!N) r\rkeysr^rr]rrrarr)rvrcountrrrs @r"set_attn_processorz3UNetSpatioTemporalConditionModel.set_attn_processorsD(--//00 i & & 3y>>U+B+BmQTU^Q_Q_mm05mmRWmmm   Tc T58? T T T T T T!//11 A ALD& ' 'fi @ @ @ @ A Ar!c 4td|jDrt}nCt dt t |j||dS)ze Disables custom attention processors and sets the default attention implementation. c32K|]}|jtvVdS)N)rr ).0procs r" zNUNetSpatioTemporalConditionModel.set_default_attn_processor..:s*fft~!;;ffffffr!zOCannot call `set_default_attn_processor` when attention processors are of type N)allrvaluesrr]nextiterr)rvrs r"set_default_attn_processorz;UNetSpatioTemporalConditionModel.set_default_attn_processor6s ffH\HcHcHeHefff f f %IINbfgklpmAmHmHmJmJhKhKcLcLNN   *****r!r chunk_sizedimc|dvrtd||pd}dtjjdtdtffd |D]}|||dS) aX Sets the attention processor to use [feed forward chunking](https://huggingface.co/blog/reformer#2-chunked-feed-forward-layers). Parameters: chunk_size (`int`, *optional*): The chunk size of the feed-forward layers. If not specified, will run feed-forward layer individually over each tensor of dim=`dim`. dim (`int`, *optional*, defaults to `0`): The dimension over which the feed-forward computation should be chunked. Choose between dim=0 (batch) or dim=1 (sequence length). )rr3z-Make sure to set `dim` to either 0 or 1, not r3rrrct|dr||||D]}|||dS)Nset_chunk_feed_forward)rr)rrchildren)rrrrfn_recursive_feed_forwards r"rz[UNetSpatioTemporalConditionModel.enable_forward_chunking..fn_recursive_feed_forwardWsov788 N---MMM** B B))%SAAAA B Br!N)r]rrarr_r)rvrrrrs @r"enable_forward_chunkingz8UNetSpatioTemporalConditionModel.enable_forward_chunkingDs f  RSRRSS S _1  Beho B3 BUX B B B B B Bmmoo ? ?F % %fj# > > > > ? ?r!Frtimestepencoder_hidden_statesadded_time_ids return_dictc d|jzd}d}tfd|jddDrtdd}|} t j| s|jjdk} |jjd k} t|tr| s| r t j n t j } n| s| r t j n t j} t j| g| |j } n8t!| jd kr | d|j} |jdd\} }| | } || }||j }||}||}|| d f}||j}||}||z}|d d}||d |jd |z}|s||d }n%|| |zd |jd }||}t j| ||j|j }|f}|jD]B}t=|dr|jr|||||\}}n||||\}}||z }C| ||||}tC|j"D]\}}|t!|j"dz k}|t!|j# d}|dt!|j# }|s|r|d jdd}t=|dr|jr|||||||}||||||}|$|}|%|}|&|}|j| |g|jddR}|s|fStO|S)a The [`UNetSpatioTemporalConditionModel`] forward method. Args: sample (`torch.Tensor`): The noisy input tensor with the following shape `(batch, num_frames, channel, height, width)`. timestep (`torch.Tensor` or `float` or `int`): The number of timesteps to denoise an input. encoder_hidden_states (`torch.Tensor`): The encoder hidden states with shape `(batch, sequence_length, cross_attention_dim)`. added_time_ids: (`torch.Tensor`): The additional time ids with shape `(batch, num_additional_ids)`. These are encoded with sinusoidal embeddings and added to the time embeddings. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.unet_slatio_temporal.UNetSpatioTemporalConditionOutput`] instead of a plain tuple. Returns: [`~models.unet_slatio_temporal.UNetSpatioTemporalConditionOutput`] or `tuple`: If `return_dict` is True, an [`~models.unet_slatio_temporal.UNetSpatioTemporalConditionOutput`] is returned, otherwise a `tuple` is returned where the first element is the sample tensor. r1FNc3*K|] }|zdkVdS)rNr )rsdefault_overall_up_factors r"rz;UNetSpatioTemporalConditionModel.forward..s.MMaq,,1MMMMMMr!z9Forward upsample size to force interpolation output size.Tmpsnpu)dtypedevicer)rrRr3)r output_size)rhas_cross_attention) hidden_statestembrimage_only_indicator)rrr)rrres_hidden_states_tupler upsample_sizer)rrrrr)r)(rnanyshapeloggerinfor is_tensorrtyper^floatfloat32float64int32int64tensorr\toexpandrdrrerfflattenreshapergrepeat_interleaverczerosrirrrmrkrjresnetsrrrtrur)rvrrrrrframe_level_condforward_upsample_sizer timestepsis_mpsis_npur batch_sizerEt_embemb time_embedsaug_embrdown_block_res_samplesdownsample_block res_samplesr{upsample_blockr~rs @r"forwardz(UNetSpatioTemporalConditionModel.forwardas B%&t':$:!!& MMMM6<;LMMM M M ) KKS T T T$( ! y)) :]'50F]'50F(E** K*0NFN (.J&J u{ i[fmTTTII  ! !Q & &!$**6=99I"(bqb!1 J$$Z00 y)) v|,,!!%(((()?)?)A)ABB !)):r*:;; !nnSY// $$[11Gm1%%##JA39Q?? DTDh &6&6"(*?)= '''# '7&6"()='''# #k 1 " " "7!5    "+4>!:!:   A~#dn"5"5"99N0#n6L2M2M1M1O1OPK%;CY?Z?Z>ZX6 AE2Dd3PbKbFc2c,d A A A AD + + +??(3-?S?Y]????F!i@i@ i@ eS01i@ %| i@  i@  i@ 0%7 8i@i@i@i@i@i@i@i@r!r$)$ dataclassesrtypingrrrrrtorch.nnradiffusers.configuration_utilsrr diffusers.loadersr diffusers.utilsr r $diffusers.models.attention_processorr rrdiffusers.models.embeddingsrrdiffusers.models.modeling_utilsr%diffusers.models.unets.unet_3d_blocksrrr get_loggerrrrr$r r!r"rs!!!!!!//////////// IIIIIIII999999////////nnnnnnnnnnDDDDDDDD666666jjjjjjjjjj  H % %            j@j@j@j@j@z;@[j@j@j@j@j@r!