o 1Aiz@sLddlmZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z m Z mZddlZddlmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZe e!Z"e ddZ#Gdddej$eZ%GdddeZ&d3ddZ'd4d5ddZ(d6d$d%Z)d7d8d*d+Z*d9d:d-d.Z+d;d1d2Z,dS)<) annotationsN)ABCabstractmethod)contextmanager)AnyOptionalUnion)AlignDevicesHook)named_module_tensorsoffload_state_dict)nn)PreTrainedModel)Conv1D)INCLUDE_LINEAR_LAYERS_SHORTHAND) PeftConfig)ModulesToSaveWrapper_get_submodulesccsLg}|D]#\}}|dvrqt|dr*t|jtr*|jjr*|j|||qd}t|drat|jdrat|jjtra|jjjrat d|jjj vrW|jjj jj}|jj|jd}dV|D] }|j|t gqf|rdd t|jD|jj_ t d|jjj vrt||jjj |jj|jt gdSdS) a A utility for modifying a module containing one or more tuners and a base layer, any of which are offloaded to the CPU or disk. Moves a module's sub-modules to the execution device before some action is performed, after that the base layer state dictionary is re-assigned (if that layer was offloaded to the disk) and finally the parameters are offloaded. If the module has no offloaded sub-modules, this function does nothing. Args: layer ('torch.nn.Module'): layer with tuners to be merged ) base_layer_hf_hookFrmetaTNcSsi|] \}}||dqS)cpu)to).0nameparamrD/data/cameron/vidgen/svd_motion_lora/Motion-LoRA/LoRA/tuner_utils.py Rsz onload_layer..) named_moduleshasattr isinstancerr offload pre_forwardappendrtorchdeviceoriginal_devicesvalues weights_mapdataset save_folder post_forwardtensorr r )layeroffloaded_modulesrmodulebase_layer_offloadoffload_folderrrr onload_layer(s@      r3cseZdZdZd7fdd Zed8d d Zd9ddZed:ddZ d;ddZ edd%d&Z d?d(d)Zd*d+Zd@d,d-ZdAdBd1d2Zd3d4ZdCd5d6ZZS)D BaseTunera A base tuner model that provides the common methods and attributes for all tuners that are injectable into a torch.nn.Module For adding a new Tuner class, one needs to overwrite the following methods: - **_prepare_adapter_config**: A private method to eventually prepare the adapter config, for example in case the field `target_modules` is missing. - **_create_and_replace**: A private method to create and replace the target module with the adapter module. - **_check_target_module_exists**: A private helper method to check if the passed module's key name matches any of the target modules in the adapter_config. The easiest is to check what is done in the `peft.tuners.lora.LoraModel` class. Attributes: model (`torch.nn.Module`): The model to which the adapter tuner layers will be attached. forward (`Callable`): The forward method of the model. peft_config (`Union[`PeftConfig`, dict[str, PeftConfig]]`): The adapter configuration object, it should be a dictionary of `str` to `PeftConfig` objects. One can also pass a PeftConfig object and a new adapter will be created with the default name `adapter` or create a new dictionary with a key `adapter_name` and a value of that peft config. config (`dict[str, Any]`): The model configuration object, it should be a dictionary of `str` to `Any` objects. targeted_module_names (`list[str]`): The list of module names that were actually adapted. Can be useful to inspect if you want to quickly double-check that the `config.target_modules` where specified correctly. peft_config(Union[PeftConfig, dict[str, PeftConfig]] adapter_namestrreturnNonecst||_g|_t|dst|tr||in||_nt dt|tr-||j|<n|j |||_ | |j||j|j_dS)Nr5zAlready found a `peft_config` attribute in the model. This will lead to having multiple adapters in the model. Make sure to know what you are doing!) super__init__modeltargeted_module_namesr r!rr5loggerinfoupdateactive_adapterinject_adapter)selfr=r5r7 __class__rrr<~s     zBaseTuner.__init__ list[str]cCt|jtr |jgS|jSNr!rBr8rDrrractive_adapters zBaseTuner.active_adaptersargsrkwargscOs|jj|i|SrI)r=forward)rDrNrOrrrrPszBaseTuner.forwardr model_configdictcCdS)a A private method to eventually prepare the adapter config. For transformers based models, if `peft_config.target_modules` is None, we can automatically infer the target modules from the `TRANSFORMERS_MODELS_TO_XXX_TARGET_MODULES_MAPPING`. This method can be further refactored in the future to automatically infer it for all tuner models. Check out `peft.tuner.lora.LoraModel._prepare_adapter_config` for an example. Args: peft_config (`PeftConfig`): The adapter config. model_config (`dict`): The transformers model config, that config should contain the `model_type` key. Nr)rDr5rQrrr_prepare_adapter_configsz!BaseTuner._prepare_adapter_configr= nn.ModulecCrS)a_ A private method to modify the model structure before adapter is applied. See `peft.tuner.lora.LoraModel._prepare_model` for an example. Args: peft_config (`PeftConfig`): The prepared adapter config. model (`nn.Module`): The model that is going to be adapted. Nr)rDr5r=rrr_prepare_model zBaseTuner._prepare_modelkeyboolcCrS)aq A helper private method to check if the passed module's key name matches any of the target modules in the `peft_config.target_modules` list. If it does, return `True`, else return `False`. Args: peft_config (`PeftConfig`): The adapter config. key (`str`): The module's key name. Nr)r5rXrrr_check_target_module_existsrWz%BaseTuner._check_target_module_existstarget target_nameparent current_keycCrS)a Inplace replacement of the target module with the adapter layer. This method needs to be overridden by all the tuner classes. Check `peft.tuners.lora.LoraModel._create_and_replace` for an example. Args: peft_config (`PeftConfig`): The adapter config. adapter_name (`str`): The adapter name. target (`nn.Module`): The target module. target_name (`str`): The target module's name. parent (`nn.Module`): The parent module. current_key (`str`): The key of the current target being adapted. Nr)rDr5r7r[r\r]r^rrr_create_and_replaceszBaseTuner._create_and_replacecCrS)a) A helper method to mark only the adapter layers as trainable (i.e. module.requires_grad = False) This needs to be overridden for all tuner classes to match the correct key names. Check `peft.tuners.lora.LoraModel._mark_only_adapters_as_trainable` for an example. Nr)rDr=rrr _mark_only_adapters_as_trainablesz*BaseTuner._mark_only_adapters_as_trainableconfigcCrS)z A helper method to check the config when a new adapter is being added. Raise a ValueError if there is something wrong with the config or if it conflicts with existing adapters. Nr)rDrarrr_check_new_adapter_configsz#BaseTuner._check_new_adapter_configcCrS)zHelper method to check whether the adapter can be merged. Raise a ValueError if it is not possible to merge the adapter with the given configuration. NrrKrrr_check_merge_allowedszBaseTuner._check_merge_allowedc s|j|}||t|dddu}d}t|dddi}t|dr%|}|||}|||d}dd |D}t||}|D]S|rtt fd d |j Drtt |\} } } t | t slt | |} t| | | n| |d }qC||s{qC|jd }t |\} } } |j||| | | d qC|std|jd|||j|jr|D] \} }|| vrd|_q|rt|dst|j |_ dS|j t|j dSdS)a Creates adapter layers and replaces the target modules with the adapter layers. This method is called under the hood by `peft.mapping.get_peft_model` if a non-prompt tuning adapter class is passed. The corresponding PEFT config is directly retrieved from the `peft_config` attribute of the BaseTuner class. Args: model (`nn.Module`): The model to be tuned. adapter_name (`str`): The adapter name. modules_to_saveNFra model_typecustomto_dictcSsg|] \}}d|vr|qS)temporal_transformer_blockrrrX_rrr +z,BaseTuner.inject_adapter..c3s|] }|VqdSrIendswith)rmodule_to_saverXrr 1  z+BaseTuner.inject_adapter..T)r^zTarget modules zL not found in the base model. Please check the target modules and try again.)r5rbgetattrr rgrTrVr _maybe_include_all_linear_layersanyrdrr!rsetattrrArZr>r$r_ ValueErrortarget_modulesr`inference_modenamed_parameters requires_gradset)rDr=r7r5_check_for_modules_to_save_has_modules_to_saverQis_target_modules_in_base_modelkey_listr]r[r\ new_modulenprrprrC sX              zBaseTuner.inject_adapterN adapter_namesOptional[list[str]]c CsZ||jD]!}t|tr*t||j|dWdn1s%wYq dS)a This method merges the adapter layers into the base model. Merging adapters can lead to a speed up of the forward pass. A copy of the adapter weights is still kept in memory, which is required to unmerge the adapters. In order to merge the adapter weights without keeping them in memory, please call `merge_and_unload`. Args: safe_merge (`bool`, *optional*): If `True`, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`list[str]`, *optional*): The list of adapter names that should be merged. If `None`, all active adapters will be merged. Defaults to `None`. )rN)rcr=modulesr!BaseTunerLayerr3merge)rDrr0rrr merge_adapter[s  zBaseTuner.merge_adapterc CsN|jD]}t|tr$t| |Wdn1swYqdS)zU This method unmerges all merged adapter layers from the base model. N)r=rr!rr3unmerge)rDr0rrrunmerge_adapterrs   zBaseTuner.unmerge_adaptercs@|pj}tfdd|D}|rt|dkrtddSdS)Nc3s|] }j|jVqdSrI)r5rd)radapterrKrrrq}rrz.BaseTuner._unloading_checks..z?Cannot unload multiple adapters that specify `modules_to_save`.)rLrulenrw)rDradapters_to_consideris_modules_to_save_availablerrKr_unloading_checks{s  zBaseTuner._unloading_checks)r5r6r7r8r9r:r9rG)rNrrOr)r5rrQrRr9r)r5rr=rU)r5rrXr8r9rY)r5rr7r8r[rUr\r8r]rUr^r8r9r:)r=rU)rarr9r:)r=rUr7r8rI)rrr9r:)rr)__name__ __module__ __qualname____doc__r<propertyrLrPrrTrVrZr_r`rbrcrCrrr __classcell__rrrErr4\s*!          P r4c@seZdZUdZdZdZded<dZded<dZded <d Z d ed <gZ d ed<d3ddZ e d4ddZ e d4ddZd5d6ddZd7ddZe d8d d!Ze d8d"d#Ze d9d%d&Ze d'd(Zd:d*d+Zd;d,d-ZdraK A tuner layer mixin that provides the common methods and attributes for all tuners. Args: is_pluggable (`bool`, *optional*): Whether the adapter layer can be plugged to any pytorch module active_adapters (Union[List[`str`], `str`], *optional*): The name of the active adapter. Nrz tuple[str]adapter_layer_namesother_param_namesFrY_disable_adaptersdefaultstr | list[str]_active_adapterrGmerged_adaptersr9rUcCs"|}t|dr|j}t|ds|S)z (Recursively) get the base_layer. This is necessary for the case that the tuner layer wraps another tuner layer. r)r rrDrrrrget_base_layers   zBaseTunerLayer.get_base_layer torch.TensorcCs&|}t|dr|j}|S|j}|S)Nqweight)rr rweight)rDrrrrrrs  zBaseTunerLayer.weightcCs|}|jSrI)rbiasrrrrrszBaseTunerLayer.bias safe_mergerrr:cCtrINotImplementedError)rDrrrrrrzBaseTunerLayer.mergecCrrIrrKrrrrrzBaseTunerLayer.unmergecCs t|jSrI)rYrrKrrrmergeds zBaseTunerLayer.mergedcC|jSrI)rrKrrrdisable_adapterszBaseTunerLayer.disable_adaptersr8cCrrI)rrKrrrrBrzBaseTunerLayer.active_adaptercCrHrIrJrKrrrrLrMzBaseTunerLayer.active_adaptersenabledcCsD|r ||jd|_dS|jD] }t||}|dqd|_dS)zToggle the enabling and disabling of adapters Takes care of setting the requires_grad flag for the adapter weights. Args: enabled (bool): True to enable adapters, False to disable adapters FTN) set_adapterrLrrrsrequires_grad_)rDr layer_namer.rrrenable_adapterss      zBaseTunerLayer.enable_adapterscCs`t|tr|g}|jD]}t||}|D]\}}||vr$|dq|dqq ||_dS)aSet the active adapter(s). Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is not desired, use the following code. ```py >>> for name, param in model_peft.named_parameters(): ... if ...: # some check on name (ex. if 'lora' in name) ... param.requires_grad = False ``` Args: adapter_name (`str` or `List[str]`): Name of the adapter(s) to be activated. TFN)r!r8rrsitemsrr)rDrr module_dictrXr.rrrrs      zBaseTunerLayer.set_adaptercCsBt}|j|jD]}t||}t|dr||q t|S)z3Return a sorted list of all available adapter nameskeys)r|rrrsr rArsorted)rDrrattrrrr_all_available_adapter_names s  z+BaseTunerLayer._all_available_adapter_namesr7cCs|j|jD]}|t||vrt|||=q||jvrV|jdd}|||r0||dS|}|s=|gdS|d}td|d|d||ddSdS)a Delete an adapter from the layer This should be called on all adapter layers, or else we will get an inconsistent state. This method will also set a new active adapter if the deleted adapter was an active adapter. It is important that the new adapter is chosen in a deterministic way, so that the same adapter is chosen on all layers. Args: adapter_name (`str`): The name of the adapter to delete NrzAdapter z< was active which is now deleted. Setting active adapter to .) rrrsrLremoverrwarningswarn)rDr7rrLremaining_adaptersnew_active_adapterrrrdelete_adapters(    zBaseTunerLayer.delete_adapter)r9rU)r9r)FN)rrYrrr9r:)r9r:)r9rY)r9r8)rrYr9r:)rrr9r:r)r7r8r9r:)rrrrrBr__annotations__rrrrrrrrrrrrrLrrrrrrrrrs6                   rrXr8r9bool | re.Match[str] | Nonecs.t|jtrt|j}|S|jvrd}|Stfdd|jD}t|dd}t|dd}|duo@t|tr?t|dknd}|r|rd}|dusQt|dkrXt d}nt|tr`|gn|}|D]}t d |d }|durvnqd|durd }|St | d }t|t r||k}|S||v}|S) aA helper method to check if the passed module's key name matches any of the target modules in the adapter_config. Args: config (`LoraConfig` | `LycorisConfig`): A config to match target modules from key (`str`): A key to search any matches in config Returns: `bool` | `re.Match[str]` | `None`: True of match object if key matches any target modules from config, False or None if no match found Tc3s |] }d|VqdS)rNrm)r target_keyrprrrqNsz-check_target_module_exists..layers_to_transformNlayers_patternrz.*\.[^.]*\.(\d+)\.z.*\.z \.(\d+)\.Fr) r!rxr8re fullmatchrurslistrmatchintgroup)rarXtarget_module_found layer_indexesris_using_layer_indexes layer_indexpatternrrprcheck_target_module_exists=s> #     rrtunerr7rRcCs`|j|}dd|jD}ggd}|D]}|||r&|d|q|d|q|S)zw A helper function to inspect the set of matched and unmatched modules for a PEFT model and the given adapter. cSsg|]\}}|qSrrrirrrrktsz+inspect_matched_modules..)matched unmatchedrr)r5r=rrZr$)rr7rarrrXrrrinspect_matched_modulesos   rr5rr=rUcst|jtr |jtks|St|tstdtdtjj t f}t }| D]\}}t||r?| ddd}||q)|durZfdd| Dd }||h8}||_|S) z Helper function to update `target_modules` to all linear/Conv1D layers if provided as 'all-linear'. Adapted from the QLoRA repository: https://github.com/artidoro/qlora/blob/main/qlora.py z:Only instances of PreTrainedModel support `target_modules=`rrNcsg|] \}}|ur|qSrr)rrr0 output_embrrrkrlz4_maybe_include_all_linear_layers..r)r!rxr8lowerrr rwr%r Linearrr|rrsplitaddget_output_embeddings)r5r=linear_classeslinear_module_namesrr0nameslast_module_namerrrrt~s*       rtr0rrrGcsl|dur|j}|jr4t|jfdd|D}|r/tdd|jdd|d|Std|S) z Helper function to check which adapters should be merged. Only return those adapters that are not already merged. Give a warning if some or all of the adapters are already merged. Ncsg|]}|vr|qSrr)rrrrrrksz+check_adapters_to_merge..z'Already following adapters were merged ,z#. You are now additionally merging rz/All adapters are already merged, nothing to do.)rLrr|rrrjoin)r0rrrrcheck_adapters_to_merges  rFcCs>t|}ddd}|r|D] \}}||||q|S)zClone a module in a pytorch model. Clones a module of a model, optionally sharing all the parameters between the original and the clone. Simplifies reusing a module when manipulating the architecture of a model. srcrUdstcSs&|jddD] \}}|||qdS)NF)recurse)rzregister_parameter)rrrrrrr_share_weightssz$clone_module.._share_weightsN)rrUrrU)copydeepcopyr get_submodule)r0 share_weightsclonerr submodulerrr clone_modules  r layer_maplist[tuple[int, int]]c Cs^t|dr |j}t|dst|dr|j}d}d}t|dr$d}|j}nt|dr6t|jdr6d}|jj}n t|dr@d }|j}|rHt|tj sLt d g}|D],\}}t ||D]"}t |}| t||d d |d D] } t| drz|| _qpqYqPt |}|dkr||_n|dkr||j_n |d kr||_nt dt|jdrt ||j_dSdS)a~Replicate layers in a transfomer model with weight sharing. This function looks for a module list attribute at model[(.model)*].layers and replicates the layers in the module list according to the layer map. For example the map `[[0, 4], [2, 5]]` will take the set of layers `[0, 1, 2, 3, 4]` and replace them with a module list containing `[0, 1, 2, 3, 2, 3, 4]`. r=bertNlayersllamaencoderr.hfalconzlCould not locate the layers attribute in the model. Expected Llama, Bert or Falcon compatible architectures.T)rr layer_idxz@Unexpected model type, need to handle post-processing of layers.num_hidden_layers)r r=rrrr.rr!r ModuleListrwrangerr$rrrrar) r=rrer new_layersstartendi current_idxrrrrreplicate_layerssT          r)rXr8r9r)r)rr4r7r8r9rR)r5rr=rUr9rrI)r0rrrr9rG)F)r0rU)r=rUrr)- __future__rrloggingrrabcrr contextlibrtypingrrrr%Zaccelerate.hooksr accelerate.utilsr r r transformersr Ztransformers.pytorch_utilsr peft.utilsr peft.configrrr getLoggerrr?r3Moduler4rrrrtrrrrrrrs>          3* : 2  $