o 2AiV@sddlmZddlmZmZmZmZddlZddlm Z ddl m Z m Z ddl mZddlmZmZddlmZmZmZddlmZmZdd lmZdd lmZmZmZdd lm Z e!e"Z#eGd d d eZ$Gdddee ee Z%dS)) dataclass)DictOptionalTupleUnionN) ConfigMixinregister_to_config)UNet2DConditionLoadersMixin) BaseOutputlogging)CROSS_ATTENTION_PROCESSORSAttentionProcessor AttnProcessor)TimestepEmbedding Timesteps) ModelMixin)get_down_block get_up_blockUNetMidBlockSpatioTemporal)PeftAdapterMixinc@s eZdZUdZdZejed<dS)!UNetSpatioTemporalConditionOutputa! The output of [`UNetSpatioTemporalConditionModel`]. Args: sample (`torch.FloatTensor` of shape `(batch_size, num_frames, num_channels, height, width)`): The hidden states output conditioned on `encoder_hidden_states` input. Output of last layer of model. Nsample)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrG/data/cameron/vidgen/svd_motion_lora/Motion-LoRA/svd/models/svd_unet.pyrs rcsfeZdZdZdZe        d6deedededee dee deededede eeefde eeefde eeeeefde eeefdeffdd Z e de e effd d!Zd"e ee e efffd#d$Zd%d&Zd7d(d)Zd8d+eed,eddfd-d.Z d9d/ejd0e ejeefd1ejd2ejd3ede eeff d4d5ZZS): UNetSpatioTemporalConditionModela A conditional Spatio-Temporal UNet model that takes a noisy video frames, conditional state, and a timestep and returns a sample shaped output. This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Parameters: sample_size (`int` or `Tuple[int, int]`, *optional*, defaults to `None`): Height and width of input/output sample. in_channels (`int`, *optional*, defaults to 8): Number of channels in the input sample. out_channels (`int`, *optional*, defaults to 4): Number of channels in the output. down_block_types (`Tuple[str]`, *optional*, defaults to `("CrossAttnDownBlockSpatioTemporal", "CrossAttnDownBlockSpatioTemporal", "CrossAttnDownBlockSpatioTemporal", "DownBlockSpatioTemporal")`): The tuple of downsample blocks to use. up_block_types (`Tuple[str]`, *optional*, defaults to `("UpBlockSpatioTemporal", "CrossAttnUpBlockSpatioTemporal", "CrossAttnUpBlockSpatioTemporal", "CrossAttnUpBlockSpatioTemporal")`): The tuple of upsample blocks to use. block_out_channels (`Tuple[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): The tuple of output channels for each block. addition_time_embed_dim: (`int`, defaults to 256): Dimension to to encode the additional time ids. projection_class_embeddings_input_dim (`int`, defaults to 768): The dimension of the projection of encoded `added_time_ids`. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. cross_attention_dim (`int` or `Tuple[int]`, *optional*, defaults to 1280): The dimension of the cross attention features. transformer_layers_per_block (`int`, `Tuple[int]`, or `Tuple[Tuple]` , *optional*, defaults to 1): The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for [`~models.unet_3d_blocks.CrossAttnDownBlockSpatioTemporal`], [`~models.unet_3d_blocks.CrossAttnUpBlockSpatioTemporal`], [`~models.unet_3d_blocks.UNetMidBlockSpatioTemporal`]. num_attention_heads (`int`, `Tuple[int]`, defaults to `(5, 10, 10, 20)`): The number of attention heads. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. TN CrossAttnDownBlockSpatioTemporalr%r%DownBlockSpatioTemporalUpBlockSpatioTemporalCrossAttnUpBlockSpatioTemporalr)r)i@ir+ r4 sample_size in_channels out_channelsdown_block_typesup_block_typesblock_out_channelsaddition_time_embed_dim%projection_class_embeddings_input_dimlayers_per_blockcross_attention_dimtransformer_layers_per_blocknum_attention_heads num_framesc st||_t|t|krtd|d|dt|t|kr.td|d|dt| tsFt| t|krFtd| d|dt| tr^t| t|kr^td| d|dt| tsvt| t|krvtd| d|dtj ||d d d d |_ |d d }t |d dd d|_ |d }t |||_t |dd d|_t |||_tg|_tg|_t| tr| ft|} t| tr| ft|} t| tr| gt|} t| tr| gt|} |}|d }t|D]0\}}|}||}|t|d k}t|| || ||||| d| || |dd }|j|qt|d|| d| d| dd|_d |_tt|}tt| }tt| }tt| }tt| }|d }t|D]R\}}|t|d k}|}||}|t|d t|d }|sd}|jd 7_nd}t|||d |||||||d|||||dd }|j||}q\tj|d ddd|_t|_ tj |d |d d d |_!dS)Nz\Must provide the same number of `down_block_types` as `up_block_types`. `down_block_types`: z. `up_block_types`: .zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: zdMust provide the same number of `cross_attention_dim` as `down_block_types`. `cross_attention_dim`: z^Must provide the same number of `layers_per_block` as `down_block_types`. `layers_per_block`: rr0) kernel_sizepaddingr#T)downscale_freq_shiftgh㈵>silu) num_layersr@r7r8 temb_channelsadd_downsample resnet_epsr?rA resnet_act_fn)rJr@r?rAF) rIr@r7r8prev_output_channelrJ add_upsamplerLresolution_idxr?rArM ) num_channels num_groupseps)"super__init__r6len ValueError isinstanceintlistnnConv2dconv_inr time_projrtime_embedding add_time_proj add_embedding ModuleList down_blocks up_blocks enumeraterappendr mid_blocknum_upsamplersreversedminr GroupNorm conv_norm_outSiLUconv_actconv_out) selfr6r7r8r9r:r;r<r=r>r?r@rArBtime_embed_dimtimestep_input_dimblocks_time_embed_dimoutput_channelidown_block_type input_channelis_final_block down_blockreversed_block_out_channelsreversed_num_attention_headsreversed_layers_per_blockreversed_cross_attention_dim%reversed_transformer_layers_per_block up_block_typerOrPup_block __class__rr rWFs                    z)UNetSpatioTemporalConditionModel.__init__returncsLi}dtdtjjdtttfffdd |D] \}}|||q|S)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namemodule processorscsLt|dr|jdd||d<|D]\}}|d|||q|S)N get_processorT)return_deprecated_lora .processorrC)hasattrrnamed_children)rrrsub_namechildfn_recursive_add_processorsrr rs zUUNetSpatioTemporalConditionModel.attn_processors..fn_recursive_add_processors)strrr]Modulerr r)rrrrrrrr attn_processorss  z0UNetSpatioTemporalConditionModel.attn_processors processorcst|j}t|tr"t||kr"tdt|d|d|ddtdtjj ffdd | D] \}}|||q3d S) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrcsbt|drt|ts||n |||d|D]\}}|d|||qdS)N set_processorrrC)rrZdictrpopr)rrrrrfn_recursive_attn_processorrr r)s   zXUNetSpatioTemporalConditionModel.set_attn_processor..fn_recursive_attn_processorN) rXrkeysrZrrYrrr]rr)rrrcountrrrrr set_attn_processors   z3UNetSpatioTemporalConditionModel.set_attn_processorcCsJtdd|jDrt}ntdtt|j||dS)ze Disables custom attention processors and sets the default attention implementation. css|]}|jtvVqdS)N)rr ).0procrrr :szNUNetSpatioTemporalConditionModel.set_default_attn_processor..zOCannot call `set_default_attn_processor` when attention processors are of type N)allrvaluesrrYnextiterr)rrrrrr set_default_attn_processor6s z;UNetSpatioTemporalConditionModel.set_default_attn_processorFcCst|dr ||_dSdS)Ngradient_checkpointing)rr)rrrvaluerrr _set_gradient_checkpointingCs  z.fn_recursive_feed_forwardN)rYrr]rr[r)rrrrrrrr enable_forward_chunkingHs  z8UNetSpatioTemporalConditionModel.enable_forward_chunkingrtimestepencoder_hidden_statesadded_time_ids return_dictcCs`|}t|s.|jjdk}t|tr|rtjntj}n|r tjntj }tj |g||jd}nt |j dkr=|d |j}|j dd\} } || }||} | j |jd} || } ||} | | df} | | j} || }| |} |dd}| j| dd } |j| dd }||}tj| | |j|jd}|f}|jD]"}t|d r|jr||| ||d \}}n ||| |d \}}||7}q|j|| ||d }t|jD]2\}}|t |j d}|dt |j }t|d r|jr||| |||d }q||| ||d}q| |}|!|}|"|}|j| | g|j ddR}|s+|fSt#|dS)a The [`UNetSpatioTemporalConditionModel`] forward method. Args: sample (`torch.FloatTensor`): The noisy input tensor with the following shape `(batch, num_frames, channel, height, width)`. timestep (`torch.FloatTensor` or `float` or `int`): The number of timesteps to denoise an input. encoder_hidden_states (`torch.FloatTensor`): The encoder hidden states with shape `(batch, sequence_length, cross_attention_dim)`. added_time_ids: (`torch.FloatTensor`): The additional time ids with shape `(batch, num_additional_ids)`. These are encoded with sinusoidal embeddings and added to the time embeddings. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.unet_slatio_temporal.UNetSpatioTemporalConditionOutput`] instead of a plain tuple. Returns: [`~models.unet_slatio_temporal.UNetSpatioTemporalConditionOutput`] or `tuple`: If `return_dict` is True, an [`~models.unet_slatio_temporal.UNetSpatioTemporalConditionOutput`] is returned, otherwise a `tuple` is returned where the first element is the sample tensor. mps)dtypedevicerNr.)rrNr0)rhas_cross_attention) hidden_statestembrimage_only_indicator)rrr)rrres_hidden_states_tuplerr)rrrr)r)$r is_tensorrtyperZfloatfloat32float64int32int64tensorrXshapetoexpandr`rrarbflattenreshapercrepeat_interleaver_zerosrerrrirgrfresnetsrnrprqr)rrrrrrr timestepsis_mpsr batch_sizerBt_embemb time_embedsaug_embrdown_block_res_samplesdownsample_block res_samplesrwupsample_blockrrr forwardes                 z(UNetSpatioTemporalConditionModel.forward) Nr"r#r$r'r*r,r-r.r/r0r1r5)F)Nr)T)rrrr _supports_gradient_checkpointingrrr[rrrrWpropertyrr rrrrrrrTensorrboolrr __classcell__rrrr r!!s" 2" # r!)& dataclassesrtypingrrrrrtorch.nnr]Zdiffusers.configuration_utilsrrZdiffusers.loadersr diffusers.utilsr r Z$diffusers.models.attention_processorr r rZdiffusers.models.embeddingsrrZdiffusers.models.modeling_utilsrZsvd.models.svd_unet_blocksrrrZ LoRA.peftr get_loggerrloggerrr!rrrr s