ܱiddlmZmZmZmZmZddlZddlmZm Z ddl Z ddlZddl m Z ddl mZddlmZddlmZddlmZmZdd lmZdd lmZdd ejfd ZGdde ZdZGddeZGddeZdS))CallableDictListOptionalUnionN) rearrangerepeat)TextToVideoSDPipeline)StableVideoDiffusionPipeline)TextToVideoSDPipelineOutput)"StableVideoDiffusionPipelineOutput)LoraLoaderMixinTextualInversionLoaderMixin) randn_tensor)_resize_with_antialiasingnpvideoc|j\}}}}}g}t|D]K} || dddd} || |} || L|S)Nr)shaperangepermute postprocessappend) r processor output_type batch_sizechannels num_framesheightwidthoutputs batch_idx batch_vid batch_outputs =/data/cameron/vidgen/Ctrl-World/models/pipeline_ctrl_world.pysvd_tensor2vidr)s7 1`. Higher guidance scale encourages to generate videos that are closely linked to the text `prompt`, usually at the expense of lower video quality. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the video generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). eta (`float`, *optional*, defaults to 0.0): Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to [`schedulers.DDIMScheduler`], will be ignored for others. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.FloatTensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for video generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random `generator`. Latents should be of shape `(batch_size, num_channel, num_frames, height, width)`. prompt_embeds (`torch.FloatTensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.FloatTensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. output_type (`str`, *optional*, defaults to `"np"`): The output format of the generate video. Choose between `torch.FloatTensor` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion.TextToVideoSDPipelineOutput`] instead of a plain tuple. callback (`Callable`, *optional*): A function that will be called every `callback_steps` steps during inference. The function will be called with the following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`. callback_steps (`int`, *optional*, defaults to 1): The frequency at which the `callback` function will be called. If not specified, the callback will be called at every step. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.cross_attention](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/cross_attention.py). Examples: Returns: [`~pipelines.stable_diffusion.TextToVideoSDPipelineOutput`] or `tuple`: [`~pipelines.stable_diffusion.TextToVideoSDPipelineOutput`] if `return_dict` is True, otherwise a `tuple. When returning a tuple, the first element is a list with the generated frames. rNr?scale) prompt_embedsnegative_prompt_embeds lora_scaledevicetotalr)encoder_hidden_statescross_attention_kwargscondition_latentmaskmotionrptfinal_offload_hookframes)%unetconfig sample_sizevae_scale_factor check_inputs isinstancestrlistlenrr<get_encode_prompt scheduler set_timesteps timestepsprepare_extra_step_kwargsordertorchcat progress_bar enumeratescale_model_inputtensorsamplechunkrreshapestep prev_sampleupdatedecode_latents tensor2vidhasattrrFoffloadr )*selfpromptr"r#r!r1guidance_scalenegative_promptr2 generatorlatentsr8r9rr3callbackr4r@rArBrVrCnum_images_per_promptrr<do_classifier_free_guidancetext_encoder_lora_scaleextra_step_kwargsnum_warmup_stepsuncondition_latentr[itlatent_model_input noise_prednoise_pred_uncondnoise_pred_textbszchannelrH video_tensorrs* r(__call__zLatentToVideoPipeline.__call__!s |O49+7$:OOM)58MM !  FE>?MSi    *VS"9"9 JJ  Jvt$<$< VJJ&,Q/J'5s&:#:P9[ " & &w 5 5 5ae ++   ! ' '#9.,    $$%8$HHH  0II"%i.. !::9cJJy>>,?$.BV,VV-PkB59&8:J%KLLLrB   %8  9 9$ 0\!),,# 0# 01A\%iUYy1}%=%=%=bi"%)^%E%EFXZ[%\%\"%"\&@@@F!YY&*7+A%5!'/l9C9I9I!9L9L6%!2^YjGj5k!kJ7>m3WfeV!//!Q1a88@@vwX]_eff'//1aA>>FFsV|U\^cekll .$.-j!WZZHYZZf"$'*223PVWW__`acdfgijlmnnI***A9I/I/IqSTuX\XfXlNlpqNqNq '')))+N0Ba0G0G Aw///G# 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0L**733 $   EE|,,E 4- . . .43J3V  # + + - - - $7# #*%8888sGM''M+.M+)NNNr-r.r/Nr0NNNNrTNrNNNNN) __name__ __module__ __qualname__rYno_gradintfloatboolrr*r(r,r, sU]__#% $  $ -H9H9  H9 ! H9H9H9"#H9H9H9_H9H9H9r*r,ct||jz }|dkrtd|jd|d|dd|zzS)zNAppends dimensions to the end of a tensor until it has target_dims dimensions.rz input has z dims but target_dims is z, which is less).)N)ndim ValueError)x target_dimsdims_to_appends r( _append_dimsrsR 16)Ncafcc{cccddd Vg. . //r*c$deZdZejddddddddd dd ddd dd gd dddddfdededeedededededededeedeedeeej e ej fd eej dee dee eeegdfde e def"dZdS) CtrlWorldDiffusionPipeline@Nr6@{Gz?rpilrnTFr"r#r!r1min_guidance_scalemax_guidance_scalefpsmotion_bucket_idnoise_aug_strengthdecode_chunk_sizenum_videos_per_promptrmrcallback_on_step_end"callback_on_step_end_tensor_inputsr3c  |p|jjj|jz}|p|jjj|jz}||n|jjj}| | n|} |jj}|dk}|}|jd}|r*tj|}tj ||g}|j j tj ko|j jj }|jddkr|j|||}t!|j||j|j }|r%|j tj|||| |} | |j } |r%|j tj nH||j jjz } |rtj | gd z} | |j } |L|j\}!}"}#}$}%||"z}&| d d |&d d d } |r d | ddd|"f<n,| d d |d d d } || | | |j || |}'|'|}'|j|| |jj}(|jjj})||| z||)|||j ||| }tj|||d}*|*||j }*|*|| zd }*t=|*|j}*|*|_ tC|(||jj"zz }+tC|(|_#|A|j\}!},}#}$}%tIj|d d|}|rtj |gd zn|}||rtj |gd zn|}|%|5}-tM|(D]\}.}/|rtj |gd zn|}0|j'|0|/}0|tj ||0gd }0tj |0| gd }0||.dkrtj |0|gd}0|0|jj }0||jj }||0|/||'d|d}1||1ddddddd|$d|%f}1||1dd|"dddddddf}1|r(|1(d \}2}3|2|j)|3|2z zz}1|j*|1|/|j+}|Bi}4|D]}5tY|5|4|5<|||.|/|4}6|6-d|}|.tC|(d z ks|.d z|+kr*|.d z|jj"zdkr|-. dddn #1swxYwY|dksu|r%|j tj ||j j }|/||| }7ta|7|j|}7n|}7|1|s|7|fSte|7S)  The call function to the pipeline for generation. Args: image (`PIL.Image.Image` or `List[PIL.Image.Image]` or `torch.FloatTensor`): Image or images to guide image generation. If you provide a tensor, it needs to be compatible with [`CLIPImageProcessor`](https://huggingface.co/lambdalabs/sd-image-variations-diffusers/blob/main/feature_extractor/preprocessor_config.json). height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): The height in pixels of the generated image. width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): The width in pixels of the generated image. num_frames (`int`, *optional*): The number of video frames to generate. Defaults to 14 for `stable-video-diffusion-img2vid` and to 25 for `stable-video-diffusion-img2vid-xt` num_inference_steps (`int`, *optional*, defaults to 25): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. This parameter is modulated by `strength`. min_guidance_scale (`float`, *optional*, defaults to 1.0): The minimum guidance scale. Used for the classifier free guidance with first frame. max_guidance_scale (`float`, *optional*, defaults to 3.0): The maximum guidance scale. Used for the classifier free guidance with last frame. fps (`int`, *optional*, defaults to 7): Frames per second. The rate at which the generated images shall be exported to a video after generation. Note that Stable Diffusion Video's UNet was micro-conditioned on fps-1 during training. motion_bucket_id (`int`, *optional*, defaults to 127): The motion bucket ID. Used as conditioning for the generation. The higher the number the more motion will be in the video. noise_aug_strength (`int`, *optional*, defaults to 0.02): The amount of noise added to the init image, the higher it is the less the video will look like the init image. Increase it for more motion. decode_chunk_size (`int`, *optional*): The number of frames to decode at a time. The higher the chunk size, the higher the temporal consistency between frames, but also the higher the memory consumption. By default, the decoder will decode all frames at once for maximal quality. Reduce `decode_chunk_size` to reduce memory usage. num_videos_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.FloatTensor`, *optional*): Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random `generator`. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generated image. Choose between `PIL.Image` or `np.array`. callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] instead of a plain tuple. Returns: [`~pipelines.stable_diffusion.StableVideoDiffusionPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.stable_diffusion.StableVideoDiffusionPipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list of list with the generated frames. Examples: ```py from diffusers import StableVideoDiffusionPipeline from diffusers.utils import load_image, export_to_video pipe = StableVideoDiffusionPipeline.from_pretrained("stabilityai/stable-video-diffusion-img2vid-xt", torch_dtype=torch.float16, variant="fp16") pipe.to("cuda") image = load_image("https://lh3.googleusercontent.com/y-iFOHfLTwkuQSUegpwDdgKmOjRSTvPxat63dQLB25xkTs4lhIbRUFeNBWZzYf370g=s1200") image = image.resize((1024, 576)) frames = pipe(image, num_frames=25, decode_chunk_size=8).frames[0] export_to_video(frames, "generated.mp4", fps=7) ``` Nr6rrr"r#rmr<dtyperrrr0r;zb l c h w -> b (f l) (n c) h w)nfr=dimF)r?added_time_idsr3frame_level_condrnlatentrrG)3rIrJrKrLr!r<rrY zeros_likerZvaerfloat16 force_upcastvideo_processor preprocessrtofloat32_encode_vae_imagescaling_factor unsqueezer _get_add_time_idsrTrUrV in_channelsprepare_latentslinspacerr_guidance_scalerQrX_num_timestepseinopsr[r\r]r`rkrbrclocalspoprdrer)maybe_free_model_hooksr )8riimagetextr"r#r!r1rrrrrrrrmrnrrrr3rB cond_wristhistoryr his_cond_zeror<rqimage_embeddingsrnegative_image_embeddingsneeds_upcastingnoise image_latentsBnum_hisCHWnum_frames_allrrVnum_channels_latentsrkrtFr[rvrwrxryrznoise_pred_condcallback_kwargskcallback_outputsrHs8 r(rz#CtrlWorldDiffusionPipeline.__call__sRO49+7$:OOM)58MM#-#9ZZty?O?Z 1B1N--T^!'93&>#  %+A. & X(-(89I(J(J %$y*CEU)VWW (.EM9Zdho>Z ;r?a  (33E&PU3VVE  %,^c^ijjjE 1 %- 000 225&BWYtuuM),,-=-CDDM 1 %- 000!$(/"@@M* =!& =/!*; < < ),,-=-CDDM  ")- Aw1a''1N)33A66==aQRTUWXYYM 0,/ aaaj))33A66==aQPQSTUUM//     "  ! '  (**622 $$%8$HHHN,  $y/;&& . .     "      (:>,?$.BV,VV!)nn  !"=LAaAqz3SWX[efffJ6QaJ<>222WaJ  2MZei A ...SZG   %8  9 9F *\!),,E *E *1A\%iUYy1}%=%=%=bi"%)^%E%EFXZ[%\%\"&).G=O3PVW)X)X)X&&+Y0BM/RXY%Z%Z%Z")add).4F 3SYZ)[)[)[&&8%:%:49?%K%K"#3#6#6ty#G#G "YY&*:#1 %%5 ' )!+AAAqqq2A2rrM!:J&",AAAwxxAAAqqq,@!AJ/q9C9I9I!9L9L6%!2T5HO^oLo5p!pJ .--j!WEEQ'3&(O?99-3XXa[**';';D!Q'X'X$.229gFFGI***A9I/I/IqSTuX\XfXlNlpqNqNq '')))KE *F *F *F *F *F *F *F *F *F *F *F *F *F *F *F *Ph&& 1 %- 000jj00G((*>OPPF#FD,@kZZZFFF ##%%% "'> !1@@@@sH X**X.1X.rrrrYrrrrr Generatorr FloatTensorrOrrrrrr*r(rrsU]__ $(#%$'$' #"&+//0MQ/3%*KO9B  3{A{A {A  {A SM {A!{A"{A"{A{A{A {A$C={A (}{AE%/43H"HIJ{A %+,!{A"c]#{A$'xc40@$0F'GH%{A&-1I'{A(){A{A{A_{A{A{Ar*rc$deZdZejddddddddddd dd ddd dd gd dddfdededeedededededededeedeedeeej e ej fd eej dee dee eeegdfde e def"dZdS) TextStableVideoDiffusionPipelineNrrrr6rrrrrrrnTrr"r#r!r1rrrrrrrrmrrrr3c ^ |p|jjj|jz}|p|jjj|jz}||n|jjj}| | n|} ||||t |tjjrd}n2t |trt|}n |j d}|j }| dk}|dkr| ||||}ni|dkr|rtj||g}|}nH| ||||}|rtj||g}tj||gd}|jjjdk}|rtj|gd z}| dz } |j||| }t'|j ||j|j }|| |zz}|jjtjko|jjj}|r%|jtj |_|||||}||j}|dd|ddd}n|rtj|gd z}|r%|jtj || | | |j|||} | |} |j || |jj!}!|jjj}"|"||z||"|||j||| }tj#|| |d}#|#||j}#|#||zd}#tI|#|j%}#|#|_&t|!||jj'zz }$t|!|_(|)|5}%tU|!D]i\}&}'|rtj|gd zn|}(|j+|(|'}(|rtj||(|gd }(ntj|(|gd }(||(|'|| dd})|r(|),d \}*}+|*|j-|+|*z zz})|j.|)|'|j/}|Bi},|D]}-ta|-|,|-<|||&|'|,}.|.1d|}|&t|!dz ks|&dz|$kr*|&dz|jj'zdkr|%2k dddn #1swxYwY|dksV|r%|jtj |3||| }/ti|/|j|}/n|}/|5|s|/Stm|/S)rNrrr6rrr rrrrr;r=F)r?rr3rnrrrG)7rIrJrKrLr!rMrNPILImagerPrQr_execution_device _encode_imagerYrZrrrrr<rrrrrrrrr rrTrUrVrrrrrrXrr[r\r]r`rkrbrcrrrdrer)rr )0rirr8r9r"r#r!r1rrrrrrrrmrnrrrr3rBcondition_typerArr<rqr motion_maskrrrrrVrrkrtr[rvrwrxryrzrrrrrHs0 r(rz)TextStableVideoDiffusionPipeline.__call__3s*PO49+7$:OOM)58MM#-#9ZZty?O?Z 1B1N--T^ %/// eSY_ - - (JJ t $ $ (UJJQJ''93&># 7 " "#11%AVXstt   V # #* S % +A=*Q R R ,  #11%AVXstt * S % +A=*Q R R $y*:M)JPQRRR i&2a7 & '9dVAX&&DAg$//fE/RRU[IelZ_Zefff*U22(.EM9Zdho>Z  - HKKemK , , ,  # 225&BWYtuuM),,-=-CDDM -66q99@@JPQSTVWXX  * E#(9.>-?!-C#D#D   - HKKemK , , ,//     "  ! '  (**622 $$%8$HHHN,  $y/;&& . .     "      (:>,?$.BV,VV!)nn   %8  9 9# *\!),," *" *1A\%iUYy1}%=%=%=bi"%)^%E%EFXZ[%\%\"b).D:LN^3_ef)g)g)g&&).4FHX3Y_`)a)a)a&!YY&*:#1 % '   /q9C9I9I!9L9L6%!2T5HO^oLo5p!pJ.--j!WEEQ'3&(O?99-3XXa[**';';D!Q'X'X$.229gFFGI***A9I/I/IqSTuX\XfXlNlpqNqNq '')))E" *# *# *# *# *# *# *# *# *# *# *# *# *# *# *# *Jh&& 1 %- 000((*>OPPF#FD,@kZZZFFF ##%%% M1@@@@sE;VV"%V"rrr*r(rr2sU]__!%$(#%$'$' #"&+//0MQ/3%*KO9B   1EAEA  EA  EASMEA!EA"EA"EAEAEA EA$C=EA (}EA E%/43H"HIJ!EA"%+,#EA$c]%EA&'xc40@$0F'GH'EA(-1I)EA*+EAEAEA_EAEAEAr*r)r)typingrrrrrrYrrr r diffusersr &models.pipeline_stable_video_diffusionr Hdiffusers.pipelines.text_to_video_synthesis.pipeline_text_to_video_synthr Jdiffusers.pipelines.stable_video_diffusion.pipeline_stable_video_diffusionr diffusers.loadersrrdiffusers.utils.torch_utilsrrTensorr)r,rrrrr*r(rs88888888888888 $$$$$$$$ ,+++++OOOOOOqpppppyyyyyyJJJJJJJJ444444pppppp  %,    J9J9J9J9J91J9J9J9X000}A}A}A}A}A!=}A}A}A~ GAGAGAGAGA'CGAGAGAGAGAr*