o 1Ai@spddlZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl m Z ddlmZddlmZddlmZddl mZdd lmZdd lmZdd lmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(dd l)m*Z*m+Z+e#rdd l,m-Z-ddl.m/Z/m0Z0e"rddl1m2Z2m3Z3m4Z4e$5e6Z7dZ8dZ9dZ:dZ;dZGddde>Z?dS)N)Path)CallableDictListOptionalUnion) model_info)HF_HUB_OFFLINE)validate_hf_hub_args)version)nn) __version__)_LOW_CPU_MEM_USAGE_DEFAULT)USE_PEFT_BACKEND_get_model_fileconvert_state_dict_to_diffusersconvert_state_dict_to_peftconvert_unet_state_dict_to_peftdelete_adapter_layersget_adapter_nameget_peft_kwargsis_accelerate_availableis_transformers_availableloggingrecurse_remove_peft_layersscale_lora_layersset_adapter_layers!set_weights_and_activate_adapters) _convert_kohya_lora_to_diffusers"_maybe_map_sgm_blocks_to_diffusers)PreTrainedModel)text_encoder_attn_modulestext_encoder_mlp_modules)AlignDevicesHook CpuOffloadremove_hook_from_module text_encoderunet transformerzpytorch_lora_weights.binz pytorch_lora_weights.safetensorszYou are using an old version of LoRA backend. This will be deprecated in the next releases in favor of PEFT make sure to install the latest PEFT and transformers packages in the future.c@seZdZdZeZeZeZ dZ dOde e e e ejfffddZeede e e e ejfffddZe dPd d Zed dZe dQddZe     dRddZe dQddZedefddZddZe       dSde e ejfde e e ejj ejffde e ejj fde e ejj fd e!d!e d"e"d#e!fd$d%Z#e$d&e e ejfde d e!d!e d"e"d#e!f d'd(Z%d)d*Z&    dTd+e!d,e!d-ed.e!d/e'e(e f d0d1Z)dUd2e!d3e!fd4d5Z*  dVd/e e(e e fd6e'd7d8e(efd9d:Z+dOd6e'd7fd;d<Z,dOd6e'd7fd=d>Z- dOd/e e(e e fd?e'e(efd@dAZ.dBdCZ/dDdEZ0d/e e(e e ffdFdGZ1de(e fdHdIZ2de e e(e ffdJdKZ3d/e(e dLe ej4e e5fddfdMdNZ6dS)WLoraLoaderMixinz Load LoRA layers into [`UNet2DConditionModel`] and [`CLIPTextModel`](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPTextModel). rN%pretrained_model_name_or_path_or_dictc Kststdt|tr|}|j|fi|\}}tdd|D}|s+td|dt }|j ||t |ds@t ||j n|j|||d|j||t |dsWt ||jn|j|j|||d d S) a_ Load LoRA weights specified in `pretrained_model_name_or_path_or_dict` into `self.unet` and `self.text_encoder`. All kwargs are forwarded to `self.lora_state_dict`. See [`~loaders.LoraLoaderMixin.lora_state_dict`] for more details on how the state dict is loaded. See [`~loaders.LoraLoaderMixin.load_lora_into_unet`] for more details on how the state dict is loaded into `self.unet`. See [`~loaders.LoraLoaderMixin.load_lora_into_text_encoder`] for more details on how the state dict is loaded into `self.text_encoder`. Parameters: pretrained_model_name_or_path_or_dict (`str` or `os.PathLike` or `dict`): See [`~loaders.LoraLoaderMixin.lora_state_dict`]. kwargs (`dict`, *optional*): See [`~loaders.LoraLoaderMixin.lora_state_dict`]. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. )PEFT backend is required for this method.cs|]}d|vVqdSloraN.0keyr/r/=/data/cameron/vidgen/svd_motion_lora/Motion-LoRA/LoRA/lora.py tz4LoraLoaderMixin.load_lora_weights..Invalid LoRA checkpoint.low_cpu_mem_usager')network_alphasr'r7 adapter_name _pipeliner&)r8r& lora_scaler7r9r:N)r ValueError isinstancedictcopylora_state_dictallkeyspoprload_lora_into_unethasattrgetattr unet_namer'load_lora_into_text_encodertext_encoder_namer&r;)selfr*r9kwargs state_dictr8is_correct_formatr7r/r/r3load_lora_weightsPs8   z!LoraLoaderMixin.load_lora_weightsc Ks|dd}|dd}|dd}|dd}|dd}|dd}|d d} |d d} |d d} |d d} |d d} d}| durLd} d}ddd}d}t|ts| r^| dusg| dur| drz'| durt|j|d|d} t|| pyt||||||| | |d }tjj |dd}Wnt tj fy}z |s|d}WYd}~nd}~ww|dur| dur|j|d|d} t|| pt ||||||| | |d }tj |dd}n|}d}tdd|Dr| durt|| }t|\}}||fS)a Return state dict for lora weights and the network alphas. We support loading A1111 formatted LoRA checkpoints in a limited capacity. This function is experimental and might change in the future. Parameters: pretrained_model_name_or_path_or_dict (`str` or `os.PathLike` or `dict`): Can be either: - A string, the *model id* (for example `google/ddpm-celebahq-256`) of a pretrained model hosted on the Hub. - A path to a *directory* (for example `./my_model_directory`) containing the model weights saved with [`ModelMixin.save_pretrained`]. - A [torch state dict](https://pytorch.org/tutorials/beginner/saving_loading_models.html#what-is-a-state-dict). cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory where a downloaded pretrained model configuration is cached if the standard cache is not used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download (`bool`, *optional*, defaults to `False`): Whether or not to resume downloading the model weights and configuration files. If set to `False`, any incompletely downloaded files are deleted. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, for example, `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only (`bool`, *optional*, defaults to `False`): Whether to only load local model weights and configuration files or not. If set to `True`, the model won't be downloaded from the Hub. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, the token generated from `diffusers-cli login` (stored in `~/.huggingface`) is used. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, a commit id, or any identifier allowed by Git. subfolder (`str`, *optional*, defaults to `""`): The subfolder location of a model file within a larger model repository on the Hub or locally. low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`): Speed up model loading only loading the pretrained weights and not initializing the weights. This also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model. Only supported for PyTorch >= 1.9.0. If you are using an older version of PyTorch, setting this argument to `True` will raise an error. mirror (`str`, *optional*): Mirror source to resolve accessibility issues if you're downloading a model in China. We do not guarantee the timeliness or safety of the source, and you should refer to the mirror site for more information. cache_dirNforce_downloadFresume_downloadproxieslocal_files_onlytokenrevision subfolder weight_name unet_configuse_safetensorsTattn_procs_weightspytorch) file_type framework .safetensors)file_extensionrS) weights_namerOrPrQrRrSrTrUrV user_agentcpu)devicez.bin) map_locationcss8|]}|dp|dp|dp|dVqdS)Zlora_te_Z lora_unet_Z lora_te1_Z lora_te2_N) startswithr1kr/r/r3r4"s  z2LoraLoaderMixin.lora_state_dict..)rCr=r>endswith_best_guess_weight_namerLORA_WEIGHT_NAME_SAFE safetensorstorch load_fileIOErrorSafetensorErrorLORA_WEIGHT_NAMEloadrArBrr)clsr*rKrOrPrQrRrSrTrUrVrWrXrY allow_picklera model_filerLer8r/r/r3r@s A                 zLoraLoaderMixin.lora_state_dictr^Fcs |strtdg}tj|rdStj|r%fddt|D}nt|j}fdd|D}t |dkr;dShdt t fdd|}t d d |Dr]t t d d|}nt d d |Drot t d d|}t |dkrtdd|d|d}|S)Nz>When using the offline mode, you must specify a `weight_name`.csg|] }|r|qSr/)rhr1fr_r/r3 ?s  z;LoraLoaderMixin._best_guess_weight_name..csg|] }|jr|jqSr/) rfilenamerhrvrxr/r3ryDr> optimizer checkpoint schedulercstfddDS)Nc3s|]}|vVqdSNr/)r1 substringxr/r3r4Mr5zLLoraLoaderMixin._best_guess_weight_name....)rAr)unallowed_substringsrr3Msz9LoraLoaderMixin._best_guess_weight_name..cs|]}|tVqdSrrhrprvr/r/r3r4Pz:LoraLoaderMixin._best_guess_weight_name..cS |tSrrrr/r/r3rQ csrrrhrjrvr/r/r3r4RrcSrrrrr/r/r3rSrz9Provided path contains more than one weights file in the z~ format. Either specify `weight_name` in `load_lora_weights` or make sure there's only one `.safetensors` or `.bin` file in .) r r<ospathisfileisdirlistdirrsiblingslenlistfilterany)rrr*r_rSZtargeted_filesZ files_in_reporWr/)r_rr3ri3s6      z'LoraLoaderMixin._best_guess_weight_namecCsxd}d}|dur8|jD]*\}}t|tjr7t|dr7|s$t|jt}|s,t|jt}t dt ||dq ||fS)ar Optionally removes offloading in case the pipeline has been already sequentially offloaded to CPU. Args: _pipeline (`DiffusionPipeline`): The pipeline to disable offloading for. Returns: tuple: A tuple indicating if `is_model_cpu_offload` or `is_sequential_cpu_offload` is True. FN_hf_hookzAccelerate hooks detected. Since you have called `load_lora_weights()`, the previous hooks will be first removed. Then the LoRA parameters will be loaded and the hooks will be applied again.)recurse) componentsitemsr=r ModulerErr$r#loggerinfor%)rrr:is_model_cpu_offloadis_sequential_cpu_offload_ componentr/r/r3_optionally_disable_offloading\s    z.LoraLoaderMixin._optionally_disable_offloadingcststdddlm}m}m} |dur|nt}t|} t fdd| Drdt dj dfd d | Dfd d | D}|durcfd d |Dfdd | D}n tsmd} t | t|dkr|t|divrtd|dt|}|durt|}i} | D]\} }d| vr|jd| | <qt| ||dd}|di|}|durt|}|\}}||||d| |||}|durt|dd}|rt d|d|r|n|r||j||||ddS)aa This will load the LoRA layers specified in `state_dict` into `unet`. Parameters: state_dict (`dict`): A standard state dict containing the lora layer parameters. The keys can either be indexed directly into the unet or prefixed with an additional `unet` which can be used to distinguish between text encoder lora layers. network_alphas (`Dict[str, float]`): See `LoRALinearLayer` for more details. unet (`UNet2DConditionModel`): The UNet model to load the LoRA layers into. low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`): Speed up model loading only loading the pretrained weights and not initializing the weights. This also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model. Only supported for PyTorch >= 1.9.0. If you are using an older version of PyTorch, setting this argument to `True` will raise an error. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. r+r LoraConfiginject_adapter_in_modelset_peft_model_state_dictNc3s(|]}|jp|jVqdSr)rerGrIr0rrr/r3r4s&z6LoraLoaderMixin.load_lora_into_unet..Loading rcg|] }|jr|qSr/rerGrfrr/r3ryz7LoraLoaderMixin.load_lora_into_unet..c.i|]\}}|vr|jdd|qSrreplacerGr1rgv)rr unet_keysr/r3 s.z7LoraLoaderMixin.load_lora_into_unet..crr/rrfrr/r3ryrc.i|]\}}|vr|jdd|qSrrr alpha_keysrrr/r3r(a+You have saved the LoRA weights using the old format. To convert the old LoRA weights to the new format, you can first load them in a dictionary and then create a new dictionary like the following: `new_state_dict = {f'unet.{module_name}': params for module_name, params in old_state_dict.items()}`. peft_config Adapter name z? already in use in the Unet - please select a new adapter name.lora_BrTis_unetr9unexpected_keysXLoading adapter weights from state_dict led to unexpected keys not found in the model: . )r8r7r:r/)rr<peftrrrrrrBrArrrGrwarningrrFrshaperrrenable_model_cpu_offloadenable_sequential_cpu_offloadload_attn_procs)rrrLr8r'r7r9r:rrrrB warn_messagerankr2vallora_config_kwargs lora_configrrincompatible_keysrr/)rrrrr3rD{sj        z#LoraLoaderMixin.load_lora_into_unet?c ststdddlm} |dur|nt}t|} dur!jntfdd| Drfdd| Dfd d | D} t | dkr t d d i} t | } t| } t|D]\} }| d }| |jd| |<qbtdd| D}|rt|D] \} }| d}| d}| |jd| |<| |jd| |<q|dur‡fdd|Dfdd | D}t| || dd}| di|}|durt|}|\}}|j|| |dt||d|j|j|jd|r|dS|r |dSdSdSdS)aE This will load the LoRA layers specified in `state_dict` into `text_encoder` Parameters: state_dict (`dict`): A standard state dict containing the lora layer parameters. The key should be prefixed with an additional `text_encoder` to distinguish between unet lora layers. network_alphas (`Dict[str, float]`): See `LoRALinearLayer` for more details. text_encoder (`CLIPTextModel`): The text encoder model to load the LoRA layers into. prefix (`str`): Expected prefix of the `text_encoder` in the `state_dict`. lora_scale (`float`): How much to scale the output of the lora linear layer before it is added with the output of the regular lora layer. low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`): Speed up model loading only loading the pretrained weights and not initializing the weights. This also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model. Only supported for PyTorch >= 1.9.0. If you are using an older version of PyTorch, setting this argument to `True` will raise an error. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. r+r)rNc3s|]}j|vVqdSr)rIr0rr/r3r4rz>LoraLoaderMixin.load_lora_into_text_encoder..cs,g|]}|r|ddkr|qSrrresplitrfprefixr/r3rys,z?LoraLoaderMixin.load_lora_into_text_encoder..cs,i|]\}}|vr|dd|qSrrr)rtext_encoder_keysr/r3r&z?LoraLoaderMixin.load_lora_into_text_encoder..rrz.out_proj.lora_B.weightrcsr,)z.mlp.Nr/r0r/r/r3r4-r5z.fc1.lora_B.weightz.fc2.lora_B.weightcs,g|]}|r|ddkr|qSrrrfrr/r3ry7s cs,i|]\}}|vr|dd|qSrrr)rrr/r3r:rFr)r9adapter_state_dictr)weight)rcdtyper/)rr<rrrrrBrIrrrrrrrr!rr"rrr load_adapterrtorcrrr)rrrLr8r&rr;r7r9r:rrBZtext_encoder_lora_state_dictrnamerZrank_keyZ patch_mlpZ rank_key_fc1Z rank_key_fc2rrrrr/)rrrrrr3rHsh%           6z+LoraLoaderMixin.load_lora_into_text_encodercsddlm}m}m} |dur|nt}t|} fdd| Dfdd|D}|durHfdd|Dfd d|D}t|dkr|t |d ivr`t d |d i} |D]\} } d | vru| j d| | <qft | ||}|di|}|durt |}|\}}||||d| |||}|durt |dd}|rtd|d|r|dS|r|dSdSdS)ah This will load the LoRA layers specified in `state_dict` into `transformer`. Parameters: state_dict (`dict`): A standard state dict containing the lora layer parameters. The keys can either be indexed directly into the unet or prefixed with an additional `unet` which can be used to distinguish between text encoder lora layers. network_alphas (`Dict[str, float]`): See `LoRALinearLayer` for more details. unet (`UNet2DConditionModel`): The UNet model to load the LoRA layers into. low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`): Speed up model loading only loading the pretrained weights and not initializing the weights. This also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model. Only supported for PyTorch >= 1.9.0. If you are using an older version of PyTorch, setting this argument to `True` will raise an error. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. rrNcrr/retransformer_namerfrr/r3ryzrz>LoraLoaderMixin.load_lora_into_transformer..crrrrr)rrtransformer_keysr/r3r{rz>LoraLoaderMixin.load_lora_into_transformer..crr/rrfrr/r3ryrcrrrrrr/r3rrrrzF already in use in the transformer - please select a new adapter name.rrrrrrr/)rrrrrrrBrrrFr<rrrrrrrr)rrrLr8r(r7r9r:rrrrBrr2rrrrrrrr/)rrrrr3load_lora_into_transformer[sV         %z*LoraLoaderMixin.load_lora_into_transformerreturncCst|dr|jSdS)N _lora_scaler)rErrJr/r/r3r;szLoraLoaderMixin.lora_scalecCsxt}t|dr||jt|jdddur|j`d|j_t|dr8||jt|jdddur:|j`d|j_dSdSdS)Nr&rtext_encoder_2)rrEr&rFr_hf_peft_config_loadedr)rJZ remove_methodr/r/r3!_remove_text_encoder_monkey_patchs     z1LoraLoaderMixin._remove_text_encoder_monkey_patchTsave_directoryunet_lora_layerstext_encoder_lora_layerstransformer_lora_layersis_main_processrW save_functionsafe_serializationc Cszi} dd} |s|s|std|r| | ||j|r&| | ||j|r0| | |d|j| |||||ddS) Save the LoRA parameters corresponding to the UNet and text encoder. Arguments: save_directory (`str` or `os.PathLike`): Directory to save LoRA parameters to. Will be created if it doesn't exist. unet_lora_layers (`Dict[str, torch.nn.Module]` or `Dict[str, torch.Tensor]`): State dict of the LoRA layers corresponding to the `unet`. text_encoder_lora_layers (`Dict[str, torch.nn.Module]` or `Dict[str, torch.Tensor]`): State dict of the LoRA layers corresponding to the `text_encoder`. Must explicitly pass the text encoder LoRA state dict because it comes from 🤗 Transformers. is_main_process (`bool`, *optional*, defaults to `True`): Whether the process calling this is the main process or not. Useful during distributed training and you need to call this function on all processes. In this case, set `is_main_process=True` only on the main process to avoid race conditions. save_function (`Callable`): The function to use to save the state dictionary. Useful during distributed training when you need to replace `torch.save` with another method. Can be configured with the environment variable `DIFFUSERS_SAVE_MODE`. safe_serialization (`bool`, *optional*, defaults to `True`): Whether to save the model using `safetensors` or the traditional PyTorch way with `pickle`. c4t|tjjr |n|}fdd|D}|S)Nc i|] \}}d||qSrr/r1 module_nameparamrr/r3r zKLoraLoaderMixin.save_lora_weights..pack_weights..r=rlr rrLrlayersrZlayers_weightsZlayers_state_dictr/rr3 pack_weightsz7LoraLoaderMixin.save_lora_weights..pack_weightszkYou must pass at least one of `unet_lora_layers`, `text_encoder_lora_layers`, or `transformer_lora_layers`.r(rLrrrWrrN)r<updaterGrIwrite_lora_layers) rrrrrrrrWrrrLrr/r/r3save_lora_weightss("  z!LoraLoaderMixin.save_lora_weightsrLcCstj|rtd|ddS|dur|rdd}ntj}tj|dd|dur1|r/t}nt }t || }|||t d|dS)NzProvided path (z#) should be a directory, not a filecSstjj||ddidS)Nformatpt)metadata)rkrl save_file)weightsfilenamer/r/r3rsz8LoraLoaderMixin.write_lora_layers..save_functionT)exist_okzModel weights saved in ) rrrrerrorrlsavemakedirsrjrpras_posixr)rLrrrWrr save_pathr/r/r3rs   z!LoraLoaderMixin.write_lora_layerscCst|ds t||jn|j}ts3tttdkrt d| D]\}}t|dr1| dq#n t |t|dr>|` |dS)z Unloads the LoRA parameters. Examples: ```python >>> # Assuming `pipeline` is already loaded with the LoRA parameters. >>> pipeline.unload_lora_weights() >>> ... ``` r'z0.23zYou are using `unload_lora_weights` to disable and unload lora weights. If you want to iteratively enable and disable adapter weights,you can use `pipe.enable_lora()` or `pipe.disable_lora()`. After installing the latest version of PEFT.set_lora_layerNr)rErFrGr'rr parser rr named_modulesr rrr)rJr'rmoduler/r/r3unload_lora_weights&s     z#LoraLoaderMixin.unload_lora_weights fuse_unetfuse_text_encoderr; safe_fusing adapter_namescsddlm|s |r|jd7_|jdkrtd|r3t|ds(t||jn|j}|j |||ddfd d }|rZt|d rJ||j |||d t|dr\||j |||d d Sd Sd S)a Fuses the LoRA parameters into the original parameters of the corresponding blocks. This is an experimental API. Args: fuse_unet (`bool`, defaults to `True`): Whether to fuse the UNet LoRA parameters. fuse_text_encoder (`bool`, defaults to `True`): Whether to fuse the text encoder LoRA parameters. If the text encoder wasn't monkey-patched with the LoRA parameters then it won't have any effect. lora_scale (`float`, defaults to 1.0): Controls how much to influence the outputs with the LoRA parameters. safe_fusing (`bool`, defaults to `False`): Whether to check fused weights for NaN values before fusing and if values are NaN not fusing them. adapter_names (`List[str]`, *optional*): Adapter names to be used for fusing. If nothing is passed, all active adapters will be fused. Example: ```py from diffusers import DiffusionPipeline import torch pipeline = DiffusionPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16 ).to("cuda") pipeline.load_lora_weights("nerijs/pixel-art-xl", weight_name="pixel-art-xl.safetensors", adapter_name="pixel") pipeline.fuse_lora(lora_scale=0.7) ``` rBaseTunerLayerrzThe current API is supported for operating with a single LoRA file. You are trying to load and fuse more than one LoRA which is not well-supported.r')rrrFNcsd|i}|D]6}t|r>|dkr||tt|jj}d|vr*||d<n d|vr6|dur6td|jdi|qdS)N safe_mergerrzThe `adapter_names` argument is not supported with your PEFT version. Please upgrade to the latest version of PEFT. `pip install -U peft`r/) modulesr= scale_layerrinspect signaturemerge parametersr<)r&r;rr merge_kwargsrsupported_merge_kwargsrr/r3fuse_text_encoder_lora}s    z9LoraLoaderMixin.fuse_lora..fuse_text_encoder_lorar&)rr)rFN) peft.tuners.tuners_utilsrnum_fused_lorasrrrErFrGr' fuse_lorar&r)rJrrr;rrr'r r/rr3r#Fs$ *   zLoraLoaderMixin.fuse_lora unfuse_unetunfuse_text_encodercsddlmt|dst||jn|j}|r&|D] }t|r%|qfdd}|rBt|dr8||j t|drB||j |j d8_ d S) af Reverses the effect of [`pipe.fuse_lora()`](https://huggingface.co/docs/diffusers/main/en/api/loaders#diffusers.loaders.LoraLoaderMixin.fuse_lora). This is an experimental API. Args: unfuse_unet (`bool`, defaults to `True`): Whether to unfuse the UNet LoRA parameters. unfuse_text_encoder (`bool`, defaults to `True`): Whether to unfuse the text encoder LoRA parameters. If the text encoder wasn't monkey-patched with the LoRA parameters then it won't have any effect. rrr'cs$|D] }t|r|qdSr)rr=unmerge)r&rrr/r3unfuse_text_encoder_loras  z=LoraLoaderMixin.unfuse_lora..unfuse_text_encoder_lorar&rrN) r!rrErFrGr'rr=r&r&rr")rJr$r%r'rr'r/rr3 unfuse_loras        zLoraLoaderMixin.unfuse_lorar&r text_encoder_weightscCsbtstddd}t|tr|gn|}|||}|p t|dd}|dur)tdt|||dS)a4 Sets the adapter layers for the text encoder. Args: adapter_names (`List[str]` or `str`): The names of the adapters to use. text_encoder (`torch.nn.Module`, *optional*): The text encoder module to set the adapter layers for. If `None`, it will try to get the `text_encoder` attribute. text_encoder_weights (`List[float]`, *optional*): The weights to use for the text encoder. If `None`, the weights are set to `1.0` for all the adapters. r+cSsX|dur dgt|}nt|tr|g}t|t|kr*tdt|dt||S)NrzLength of adapter names z+ is not equal to the length of the weights )rr=floatr<)rrr/r/r3process_weightss zFLoraLoaderMixin.set_adapters_for_text_encoder..process_weightsr&NzrThe pipeline does not have a default `pipe.text_encoder` class. Please make sure to pass a `text_encoder` instead.)rr<r=strrFr)rJrr&r)r+r/r/r3set_adapters_for_text_encoders z-LoraLoaderMixin.set_adapters_for_text_encodercCs<tstd|p t|dd}|durtdt|dddS)a Disables the LoRA layers for the text encoder. Args: text_encoder (`torch.nn.Module`, *optional*): The text encoder module to disable the LoRA layers for. If `None`, it will try to get the `text_encoder` attribute. r+r&NText Encoder not found.Fenabled)rr<rFrrJr&r/r/r3disable_lora_for_text_encoders  z-LoraLoaderMixin.disable_lora_for_text_encodercCs>tstd|p t|dd}|durtdt|jdddS)a Enables the LoRA layers for the text encoder. Args: text_encoder (`torch.nn.Module`, *optional*): The text encoder module to enable the LoRA layers for. If `None`, it will try to get the `text_encoder` attribute. r+r&Nr.Tr/)rr<rFrr&r1r/r/r3enable_lora_for_text_encoders  z,LoraLoaderMixin.enable_lora_for_text_encoderadapter_weightscCsdt|ds t||jn|j}|||t|dr!|||j|t|dr0|||j|dSdS)Nr'r&r)rErFrGr' set_adaptersr-r&r)rJrr4r'r/r/r3r5 s   zLoraLoaderMixin.set_adapterscCdtstdt|dst||jn|j}|t|dr#||jt|dr0||j dSdSNr+r'r&r) rr<rErFrGr' disable_lorar2r&rrJr'r/r/r3r8   zLoraLoaderMixin.disable_loracCr6r7) rr<rErFrGr' enable_lorar3r&rr9r/r/r3r;&r:zLoraLoaderMixin.enable_loracCs|tstdt|tr|g}t|dst||jn|j}|||D]}t|dr0t |j |t|dr;t |j |q#dS)a Args: Deletes the LoRA layers of `adapter_name` for the unet and text-encoder(s). adapter_names (`Union[List[str], str]`): The names of the adapter to delete. Can be a single string or a list of strings r+r'r&rN) rr<r=r,rErFrGr'delete_adaptersrr&r)rJrr'r9r/r/r3r<4s      zLoraLoaderMixin.delete_adapterscCs`tstdddlm}g}t|dst||jn|j}|D] }t ||r-|j }|Sq |S)a Gets the list of the current active adapters. Example: ```python from diffusers import DiffusionPipeline pipeline = DiffusionPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", ).to("cuda") pipeline.load_lora_weights("CiroN2022/toy-face", weight_name="toy_face_sdxl.safetensors", adapter_name="toy") pipeline.get_active_adapters() ``` iPEFT backend is required for this method. Please install the latest version of PEFT `pip install -U peft`rrr') rr<r!rrErFrGr'rr=active_adapters)rJrr>r'rr/r/r3get_active_adaptersLs   z#LoraLoaderMixin.get_active_adapterscCststdi}t|drt|jdrt|jj|d<t|dr2t|jdr2t|jj|d<t|ds=t||j n|j }t||j rVt|drVt|j j||j <|S)zR Gets the current list of all available adapters in the pipeline. r=r&rrr') rr<rEr&rrrBrrFrGr')rJr5r'r/r/r3get_list_adapterslsz!LoraLoaderMixin.get_list_adaptersrcc Cststdddlm}t|dst||jn|j}|D]}t ||r:|D]}|j | ||j | |q'qg}t|drH| |jt|drS| |j|D]#}|D]} t | |rw|D]}| j | || j | |qdq[qUdS)a Moves the LoRAs listed in `adapter_names` to a target device. Useful for offloading the LoRA to the CPU in case you want to load multiple adapters and free some GPU memory. Args: adapter_names (`List[str]`): List of adapters to send device to. device (`Union[torch.device, str, int]`): Device to send the adapters to. Can be either a torch device, a str or an integer. r+rrr'r&rN)rr<r!rrErFrGr'rr=lora_Arrappendr&r) rJrrcrr'Z unet_moduler9Zmodules_to_processr&Ztext_encoder_moduler/r/r3set_lora_devices0          zLoraLoaderMixin.set_lora_devicer)r^F)NNN)NrNNNNNNTNNT)TTrFN)TT)NN)7__name__ __module__ __qualname____doc__TEXT_ENCODER_NAMErI UNET_NAMErGTRANSFORMER_NAMErr"rr,rrlTensorrN classmethodr r@rirrDrHrpropertyr*r;rrPathLiker rboolrr staticmethodrrrrr#r(r-r2r3r5r8r;r<r?r@rcintrCr/r/r/r3r)Es >$ (  k s R  @  !"  R) *   *r)c@seZdZdZ ddeeeeejffde efddZ e       ddeee j fd eeeejjejffd eeeejjejffd eeeejjejffd ed ededefddZddZdS) StableDiffusionXLLoraLoaderMixinz\This class overrides `LoraLoaderMixin` with LoRA loading/saving code that's specific to SDXLNr*r9c Kststdt|tr|}|j|fd|jji|\}}tdd| D}|s/td|j |||j||ddd| D}t |d krW|j |||jd |j||d d d| D}t |d krv|j |||jd |j||d dSdS)a_ Load LoRA weights specified in `pretrained_model_name_or_path_or_dict` into `self.unet` and `self.text_encoder`. All kwargs are forwarded to `self.lora_state_dict`. See [`~loaders.LoraLoaderMixin.lora_state_dict`] for more details on how the state dict is loaded. See [`~loaders.LoraLoaderMixin.load_lora_into_unet`] for more details on how the state dict is loaded into `self.unet`. See [`~loaders.LoraLoaderMixin.load_lora_into_text_encoder`] for more details on how the state dict is loaded into `self.text_encoder`. Parameters: pretrained_model_name_or_path_or_dict (`str` or `os.PathLike` or `dict`): See [`~loaders.LoraLoaderMixin.lora_state_dict`]. adapter_name (`str`, *optional*): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. kwargs (`dict`, *optional*): See [`~loaders.LoraLoaderMixin.lora_state_dict`]. r+rXcsr,r-r/r0r/r/r3r4r5zEStableDiffusionXLLoraLoaderMixin.load_lora_weights..r6)r8r'r9r:cSi|] \}}d|vr||qS)z text_encoder.r/rr/r/r3rr{zFStableDiffusionXLLoraLoaderMixin.load_lora_weights..rr&)r8r&rr;r9r:cSrT)ztext_encoder_2.r/rr/r/r3rr{rN)rr<r=r>r?r@r'configrArBrDrrrHr&r;r) rJr*r9rKrLr8rMtext_encoder_state_dictZtext_encoder_2_state_dictr/r/r3rNsP      z2StableDiffusionXLLoraLoaderMixin.load_lora_weightsTrrrtext_encoder_2_lora_layersrrWrrc Csvi} dd} |s|s|std|r| | |d|r.|r.| | |d| | |d|j| |||||ddS) rcr)Ncrrr/rrr/r3r#rz\StableDiffusionXLLoraLoaderMixin.save_lora_weights..pack_weights..rrr/rr3r!rzHStableDiffusionXLLoraLoaderMixin.save_lora_weights..pack_weightszmYou must pass at least one of `unet_lora_layers`, `text_encoder_lora_layers` or `text_encoder_2_lora_layers`.r'r&rrN)r<rr) rrrrrrWrrWrrrLrr/r/r3rs&"  z2StableDiffusionXLLoraLoaderMixin.save_lora_weightscCs\t|jt|jdddur|j`d|j_t|jt|jdddur,|j`d|j_dSdS)Nr)rr&rFrrrrr/r/r3r;s   zBStableDiffusionXLLoraLoaderMixin._remove_text_encoder_monkey_patchrrD)rErFrGrHrr,rrlrLrrNrMrrOr rrPrrrr/r/r/r3rSsD M  =rS)@rrpathlibrtypingrrrrrrkrlhuggingface_hubrZhuggingface_hub.constantsr huggingface_hub.utilsr packagingr r diffusersr Zdiffusers.models.modeling_utilsrdiffusers.utilsrrrrrrrrrrrrrrrZ'diffusers.loaders.lora_conversion_utilsrr transformersr Zdiffusers.models.lorar!r"Zaccelerate.hooksr#r$r% get_loggerrErrIrJrKrprjZLORA_DEPRECATION_MESSAGEr)rSr/r/r/r3sJ         D  o