ܱit ddlZddlmZddlmZmZmZmZmZddl Z ddl Z ddl Z ddlmZmZddlmZddlmZddlmZddlmZdd lmZmZmZmZdd lmZm Z dd l!m"Z"dd l#m$Z$er ddl%m&cm'Z(d Z)ndZ)ej*e+Z,dZ-dZ. d!dee/deee0e j1fdeee/deee2fdZ3eGddeZ4Gdde$Z5d"dZ6dZ7dZ8de/fdZ9d Z:dS)#N) dataclass)CallableDictListOptionalUnion)CLIPImageProcessorCLIPVisionModelWithProjection)PipelineImageInput)AutoencoderKLTemporalDecoder) UNetSpatioTemporalConditionModel)EulerDiscreteScheduler) BaseOutputis_torch_xla_availableloggingreplace_example_docstring)is_compiled_module randn_tensor)VideoProcessor)DiffusionPipelineTFa Examples: ```py >>> from diffusers import StableVideoDiffusionPipeline >>> from diffusers.utils import load_image, export_to_video >>> pipe = StableVideoDiffusionPipeline.from_pretrained( ... "stabilityai/stable-video-diffusion-img2vid-xt", torch_dtype=torch.float16, variant="fp16" ... ) >>> pipe.to("cuda") >>> image = load_image( ... "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/svd-docstring-example.jpeg" ... ) >>> image = image.resize((1024, 576)) >>> frames = pipe(image, num_frames=25, decode_chunk_size=8).frames[0] >>> export_to_video(frames, "generated.mp4", fps=7) ``` ct||jz }|dkrtd|jd|d|dd|zzS)zNAppends dimensions to the end of a tensor until it has target_dims dimensions.rz input has z dims but target_dims is z, which is less).N)ndim ValueError)x target_dimsdims_to_appends I/data/cameron/vidgen/Ctrl-World/models/pipeline_stable_video_diffusion.py _append_dimsrFsR 16)Ncafcc{cccddd Vg. . //num_inference_stepsdevice timestepssigmasc f||td|dttj|jjv}|std|jd|jd ||d||j}t|}n|dttj|jjv}|std|jd|jd ||d ||j}t|}n|j|fd |i||j}||fS) a Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. Args: scheduler (`SchedulerMixin`): The scheduler to get timesteps from. num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` must be `None`. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`List[int]`, *optional*): Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, `num_inference_steps` and `sigmas` must be `None`. sigmas (`List[float]`, *optional*): Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, `num_inference_steps` and `timesteps` must be `None`. Returns: `Tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the second element is the number of inference steps. NzYOnly one of `timesteps` or `sigmas` can be passed. Please choose one to set custom valuesr#zThe current scheduler class zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)r#r"r$zv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)r$r"r") rsetinspect signature set_timesteps parameterskeys __class__r#len) schedulerr!r"r#r$kwargsaccepts_timesteps accept_sigmass rretrieve_timestepsr3Os>!3tuuu'3w/@AX/Y/Y/d/i/i/k/k+l+ll  by/Bbbb  M)FMMfMMM' !)nn   C(9):Q(R(R(](b(b(d(d$e$ee  `y/B```  GvfGGGGG' !)nn  3MMFMfMMM' ) ))r cpeZdZUdZeeeejjej e j fe d<dS)"StableVideoDiffusionPipelineOutputaB Output class for Stable Video Diffusion pipeline. Args: frames (`[List[List[PIL.Image.Image]]`, `np.ndarray`, `torch.Tensor`]): List of denoised PIL images of length `batch_size` or numpy array or torch tensor of shape `(batch_size, num_frames, height, width, num_channels)`. framesN) __name__ __module__ __qualname____doc__rrPILImagenpndarraytorchTensor__annotations__r&r rr5r5sH $tCIO,-rz5<G HHHHHHr r5c)eZdZdZdZdgZdededede de f fd Z d e d e eejfd ed edejf dZd ejd e eejfd ed efdZdedededejded ed efdZd9dejdedefdZdZ d:dedededededejd e eejfd ejdeejfd!Zed"Zed#Z ed$Z!ej"e#e$d%d&dd'dd(d)d*d+d,dd-ddd.ddgd/fd e e%j&j&e'e%j&j&ejfdededeed0ed1ee'ed2ed3ededededeed eed ee eje'ejfdeejd4eed5ee(eee)gdfd6e'ed7ef&d8Z*xZ+S);StableVideoDiffusionPipelineae Pipeline to generate video from an input image using Stable Video Diffusion. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.). Args: vae ([`AutoencoderKLTemporalDecoder`]): Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations. image_encoder ([`~transformers.CLIPVisionModelWithProjection`]): Frozen CLIP image-encoder ([laion/CLIP-ViT-H-14-laion2B-s32B-b79K](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K)). unet ([`UNetSpatioTemporalConditionModel`]): A `UNetSpatioTemporalConditionModel` to denoise the encoded image latents. scheduler ([`EulerDiscreteScheduler`]): A scheduler to be used in combination with `unet` to denoise the encoded image latents. feature_extractor ([`~transformers.CLIPImageProcessor`]): A `CLIPImageProcessor` to extract features from generated images. zimage_encoder->unet->vaelatentsvae image_encoderunetr/feature_extractorc*t||||||t|ddr$dt |jjjdz znd|_td|j|_ dS)N)rErFrGr/rHrET) do_resizevae_scale_factor) super__init__register_modulesgetattrr.rEconfigblock_out_channelsrNrvideo_processor)selfrErFrGr/rHr-s rrPz%StableVideoDiffusionPipeline.__init__s  '/    W^^bdikoVpVp wc$(/*L&M&MPQ&Q R Rvw-tOdeeer imager"num_videos_per_promptdo_classifier_free_guidancereturnct|jj}t |t jsT|j|}|j |}|dzdz }t|d}|dzdz }| |dddddj }| ||}||j}|d }|j\}}} |d |d }|||z|d }|r*t j|} t j| |g}|S) N@?)r^TFpt)images do_normalizedo_center_croprM do_rescalereturn_tensorsr"dtyperK)nextrFr+rf isinstancer?r@rU pil_to_numpy numpy_to_pt_resize_with_antialiasingrH pixel_valuesto image_embeds unsqueezeshaperepeatview zeros_likecat) rVrWr"rXrYrfimage_embeddingsbs_embedseq_len_negative_image_embeddingss r _encode_imagez*StableVideoDiffusionPipeline._encode_imagesT'224455;%.. ((55e<7I$(7S7Si+33Y]YaYi)S1B>1R1R1]1b1b1d1d-e-eeq'-*,=>> ! !A#A,=(=$=>DQGMM! <.; l+#DHOGA4E0E,E$FXX-XX_E MM% 6q))) JBabb1ABBBJJ1aQRTUWXYY r c0t|tjsSt|tjjs4t|t st dt||dzdks |dzdkrt d|d|ddS)Nz``image` has to be of type `torch.Tensor` or `PIL.Image.Image` or `List[PIL.Image.Image]` but is rLrz7`height` and `width` have to be divisible by 8 but are z and .)rir?r@r;r<listrtype)rVrWheightwidths r check_inputsz)StableVideoDiffusionPipeline.check_inputsDs5%,// ucio66 ud++  "KK""  A:??eai1nnlW]lldilllmm m-nr Nnum_channels_latentsrr generatorc D|||dz||jz||jzf} t|tr6t||kr#t dt|d|d| t | |||} n| |} | |jjz} | S)NrJz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.rr"rf) rNrirr.rrrnr/init_noise_sigma) rVrrrrrrfr"rrDrqs rprepare_latentsz,StableVideoDiffusionPipeline.prepare_latentsRs   A % d+ + T* *   i & & 3y>>Z+G+Gi#i..ii&iii  ?"5IfTYZZZGGjj((GDN;;r c|jSr)_guidance_scalerVs rguidance_scalez+StableVideoDiffusionPipeline.guidance_scalets ##r ct|jttfr |jdkS|jdkS)NrK)ririntrmaxrs rrYz8StableVideoDiffusionPipeline.do_classifier_free_guidance{sC d)C< 8 8 +&* *"&&((1,,r c|jSr)_num_timestepsrs r num_timestepsz*StableVideoDiffusionPipeline.num_timestepss ""r i@ir]g@g{Gz?rKpilTr!r$min_guidance_scalemax_guidance_scale output_typecallback_on_step_end"callback_on_step_end_tensor_inputs return_dictc $ |p|jjj|jz}|p|jjj|jz}||n|jjj}| | n|} ||||t |tjjrd}n2t |trt|}n |j d}|j }||_ |||| |j}| dz } |j||||}t'|j |||j}|| |zz}|jjt,jko|jjj}|r%|jt,j|||| |j}||j}|r%|jt,j|dd|ddd}|| | | |j|| |j}||}t=|j||d|\}}|jjj }|!|| z|||||j||| }t-j"|||d}|||j}||| zd}tG||j$}||_ t|||jj%zz }t||_&|'|5}tQ|D]q\} }!|jrt-j)|gd zn|}"|j*|"|!}"t-j)|"|gd }"||"|!||d d}#|jr(|#+d \}$}%|$|j,|%|$z zz}#|j-|#|!|j.}|Bi}&|D]}'t_|'|&|'<||| |!|&}(|(0d |}| t|dz ks| dz|kr*| dz|jj%zdkr|1tdrtgj4s dddn #1swxYwY|dks[|r%|jt,j|5||| })|j6|)|})n|})|7|s|)Stq|)S)a The call function to the pipeline for generation. Args: image (`PIL.Image.Image` or `List[PIL.Image.Image]` or `torch.Tensor`): Image(s) to guide image generation. If you provide a tensor, the expected value range is between `[0, 1]`. height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): The height in pixels of the generated image. width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): The width in pixels of the generated image. num_frames (`int`, *optional*): The number of video frames to generate. Defaults to `self.unet.config.num_frames` (14 for `stable-video-diffusion-img2vid` and to 25 for `stable-video-diffusion-img2vid-xt`). num_inference_steps (`int`, *optional*, defaults to 25): The number of denoising steps. More denoising steps usually lead to a higher quality video at the expense of slower inference. This parameter is modulated by `strength`. sigmas (`List[float]`, *optional*): Custom sigmas to use for the denoising process with schedulers which support a `sigmas` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. min_guidance_scale (`float`, *optional*, defaults to 1.0): The minimum guidance scale. Used for the classifier free guidance with first frame. max_guidance_scale (`float`, *optional*, defaults to 3.0): The maximum guidance scale. Used for the classifier free guidance with last frame. fps (`int`, *optional*, defaults to 7): Frames per second. The rate at which the generated images shall be exported to a video after generation. Note that Stable Diffusion Video's UNet was micro-conditioned on fps-1 during training. motion_bucket_id (`int`, *optional*, defaults to 127): Used for conditioning the amount of motion for the generation. The higher the number the more motion will be in the video. noise_aug_strength (`float`, *optional*, defaults to 0.02): The amount of noise added to the init image, the higher it is the less the video will look like the init image. Increase it for more motion. decode_chunk_size (`int`, *optional*): The number of frames to decode at a time. Higher chunk size leads to better temporal consistency at the expense of more memory usage. By default, the decoder decodes all frames at once for maximal quality. For lower memory usage, reduce `decode_chunk_size`. num_videos_per_prompt (`int`, *optional*, defaults to 1): The number of videos to generate per prompt. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for video generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random `generator`. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generated image. Choose between `pil`, `np` or `pt`. callback_on_step_end (`Callable`, *optional*): A function that is called at the end of each denoising step during inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.stable_diffusion.StableVideoDiffusionPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.stable_diffusion.StableVideoDiffusionPipelineOutput`] is returned, otherwise a `tuple` of (`List[List[PIL.Image.Image]]` or `np.ndarray` or `torch.Tensor`) is returned. NrKr)rrrr)r"rXrY)totalrJrF)encoder_hidden_statesadded_time_idsrrDlatent)videor)r6)9rGrS sample_sizerNrrrir;r<rr.rq_execution_devicerr{rYrU preprocessrnrrfrEr?float16 force_upcastfloat32rrprrrr3r/ in_channelsrlinspacerrorderr progress_bar enumerateruscale_model_inputchunkrstep prev_samplelocalspopupdate XLA_AVAILABLExm mark_steprpostprocess_videomaybe_free_model_hooksr5)*rVrWrrrr!r$rrrrrrrXrrDrrrrrr"rvnoiseneeds_upcastingrrr#rrnum_warmup_stepsrrtlatent_model_input noise_prednoise_pred_uncondnoise_pred_condcallback_kwargskcallback_outputsr6s* r__call__z%StableVideoDiffusionPipeline.__call__sK@O49+7$:OOM)58MM#-#9ZZty?O?Z 1B1N--T^ %/// eSY_ - - (JJ t $ $ (UJJQJ' 2 --eV=RTXTtuuAg$//fE/RRUUV\]]U[IfTYT_```*U22(.EM9Zdho>Z  - HKKemK , , ,.. "7(,(H /  &(()9)?@@   - HKKemK , , ,&//2299!ZAqQQ //     "  !  ,  (**622*>,?$.BV,VV!)nn   %8  9 9& #\!),,% #% #1AEAa%nUYy1}%=%=%=gn"%)^%E%EFXZ[%\%\"&+Y0BM/RXY%Z%Z%Z""YY&*:#1 % '   3q9C9I9I!9L9L6%!2T5HO^oLo5p!pJ.--j!WEEQ'3&(O?99-3XXa[**';';D!Q'X'X$.229gFFGI***A9I/I/IqSTuX\XfXlNlpqNqNq ''))) #LNNNK% #& #& #& #& #& #& #& #& #& #& #& #& #& #& #& #Ph&& 1 %- 000((*>OPPF);;&Va;bbFFF ##%%% M1@@@@s,FS<<TT)rr),r7r8r9r:model_cpu_offload_seq_callback_tensor_inputsr r r rr rPr rstrr?r"rboolr@r{rrrfrrr GeneratorrrpropertyrrYrno_gradrEXAMPLE_DOC_STRINGr;r<rrrr __classcell__)r-s@rrCrCs(7(kf )f5f/ f * f . ffffff(. !. c5<'(.  # . &* .  . . . . `|c5<'( #  &* . "  {    #&*8elX[: n n n0+/    "     { c5<'( ? %,'    D$$X$ --X- ##X#U]__122$(#%(,$'$' #$(+//0MQ*.%*KO9B  )rArASY_d39?&;U\IJrArA rA SM rA ! rAe%rA"rA"rArArA"rA$C=rA (}rAE%/43H"HIJrA %,'!rA"c]#rA$'xc40@$0F'GH%rA&-1I'rA()rArArA32_rArArArArAr rCbicubiccX|jdd\}}||dz ||dz f}t|ddz dz dt|ddz dz df}ttd|dzdttd|dzdf}|dd zdkr|ddz|df}|dd zdkr|d|ddzf}t|||}tjj|||| } | S) NrrKr]r\gMbP?g@rrJ)sizer align_corners)rqrr_gaussian_blur2dr?nn functional interpolate) inputr interpolationrhwfactorsr$ksoutputs rrlrl~s= ;rss DAq47{AQK(G WQZ#  $e,, WQZ#  $e,,F S6!9$a ( ( ) )3s7VAY3F/J/J+K+K KB 1 a UQY1  1 a UBqEAI  UB / /E X , ,UMan , o oF Mr c$t|dkrt|d|D}dt|zdgz}tt|D].}||dz }|dz}||z }||d|zdz<||d|zdz</|S)zCompute padding tuple.rJcg|]}|dz S)rKr&).0rs r z$_compute_padding..s+++!A+++r rrK)r.AssertionErrorr) kernel_sizecomputed out_paddingr computed_tmp pad_frontpad_rears r_compute_paddingrs ;![)))++{+++Hc+&&&!,K 3{## $ $**!a%)  A% )+!* AEAI!) AEAI r c|j\}}}}|ddddf|j|j}|d|dd}|jdd\}}t ||g} t jj || d}| dd||}| d| d| d| d}t jj ||| ddd } | ||||} | S) N.rergrreflect)rrKr)groupspaddingstride)rqrnr"rfexpandrr?rrpadrrsrconv2d) rkernelbcr r  tmp_kernelrr padding_shaperouts r _filter2dr)s<JAq!Q4 %(( EK(PPJ""2q"b11J$RSS)MFE/@@M H  # #E=y # I IE##B6599J JJr:??1--uzz"~~uzz"~~ N NEX ' 'z*//RSBTBT^_hi ' j jF ++aAq ! !C Jr window_sizect|trtj|gg}|jd}tj||j|j|dzz |d}|dzdkr|dz}tj | d d| dzz }|| ddz S) NrrerJrgg?r\T)keepdim) rirr?rrqaranger"rfr exppowsum)r*sigmarrgausss r _gaussianr3s%( ugY''QJ k%,ek J J J[\]M] ]eefprtuuAQ! G IquuSzzkQ3%78 9 9E 599R9.. ..r c(t|trtj|g|j}n||j}t |dt |d}}|jd}t||dddf |d}t||dddf |d}t||ddddf}t||d} | S)NrrrK.).N) ritupler?rrfrnrrqr3rsr)) rrr1kykxbskernel_xkernel_yout_xr(s rrrs%, eWEK888u{++ Q #k!n"5"5B QBU111a4[--b!4455HU111a4[--b!4455H eXc4l3 4 4E E8I. / /C Jr )NNNN)rT);r( dataclassesrtypingrrrrrnumpyr= PIL.Imager;r? transformersr r diffusers.image_processorr diffusers.modelsr %models.unet_spatio_temporal_conditionr diffusers.schedulersrdiffusers.utilsrrrrdiffusers.utils.torch_utilsrrdiffusers.video_processorr"diffusers.pipelines.pipeline_utilsrtorch_xla.core.xla_modelcore xla_modelrr get_loggerr7loggerrrrrr"rr3r5rCrlrr)r3rr&r rrNs!!!!!!88888888888888 JJJJJJJJ888888:99999RRRRRR777777bbbbbbbbbbbbHHHHHHHH444444@@@@@@)))))))))MMM  H % %,000*.15%)$( 8*8*!#8* U3 ,- .8*S " 8* T%[ ! 8*8*8*8*v  I I I I I I I  IaAaAaAaAaA#4aAaAaAL:.. /3 / / / /     r