o 1Aiȇ@sxddlmZddlZddlZddlZddlZddlmZddlm Z m Z ddl m Z ddl mZmZddlmZddlmZmZddlZdd lmZdd lmZdd lmZmZdd lmZmZmZmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5m6Z6m7Z7ddl8m9Z9ddl:m;Z;ddZ/data/cameron/vidgen/svd_motion_lora/Motion-LoRA/LoRA/model.py_adapter_names_pre_forward_hook8sr.csBeZdZUdZdZded<dNfdd ZdOd d Zed dZ dPddZ ddZ ddZ dQddZ eddZdRfdd ZdSdTd"d#ZdUdVd&d'ZdNd(d)ZdNd*d+ZdWd.d/Zed0d1Zd2d3Zed4d5Z $   6dXdYd;d<Z = 6 6 $ 6 6 >dZd[dAdBZ 6 $ 6d\dCdDZdEdFZd]dGdHZ 6d^d_dJdKZd`dLdMZZ S)a LoraModelaR Creates Low Rank Adapter (LoRA) model from a pretrained transformers model. The method is described in detail in https://arxiv.org/abs/2106.09685. Args: model ([`torch.nn.Module`]): The model to be adapted. config ([`LoraConfig`]): The configuration of the Lora model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. Returns: `torch.nn.Module`: The Lora model. Example: ```py >>> from transformers import AutoModelForSeq2SeqLM >>> from peft import LoraModel, LoraConfig >>> config = LoraConfig( ... task_type="SEQ_2_SEQ_LM", ... r=8, ... lora_alpha=32, ... target_modules=["q", "v"], ... lora_dropout=0.01, ... ) >>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>> lora_model = LoraModel(model, config, "default") ``` ```py >>> import torch >>> import transformers >>> from peft import LoraConfig, PeftModel, get_peft_model, prepare_model_for_kbit_training >>> rank = ... >>> target_modules = ["q_proj", "k_proj", "v_proj", "out_proj", "fc_in", "fc_out", "wte"] >>> config = LoraConfig( ... r=4, lora_alpha=16, target_modules=target_modules, lora_dropout=0.1, bias="none", task_type="CAUSAL_LM" ... ) >>> quantization_config = transformers.BitsAndBytesConfig(load_in_8bit=True) >>> tokenizer = transformers.AutoTokenizer.from_pretrained( ... "kakaobrain/kogpt", ... revision="KoGPT6B-ryan1.5b-float16", # or float32 version: revision=KoGPT6B-ryan1.5b ... bos_token="[BOS]", ... eos_token="[EOS]", ... unk_token="[UNK]", ... pad_token="[PAD]", ... mask_token="[MASK]", ... ) >>> model = transformers.GPTJForCausalLM.from_pretrained( ... "kakaobrain/kogpt", ... revision="KoGPT6B-ryan1.5b-float16", # or float32 version: revision=KoGPT6B-ryan1.5b ... pad_token_id=tokenizer.eos_token_id, ... use_cache=False, ... device_map={"": rank}, ... torch_dtype=torch.float16, ... quantization_config=quantization_config, ... ) >>> model = prepare_model_for_kbit_training(model) >>> lora_model = get_peft_model(model, config) ``` **Attributes**: - **model** ([`~transformers.PreTrainedModel`]) -- The model to be adapted. - **peft_config** ([`LoraConfig`]): The configuration of the Lora model. lora_strprefixreturnNonecst|||dSN)super__init__)selfmodelconfig adapter_name __class__r)r-r7szLoraModel.__init__r:r cCs2t|jdkr|jdkrt|jjddSdS)z A helper method to check the config when a new adapter is being added. Raise a ValueError if there is something wrong with the config or if it conflicts with existing adapters. r&nonezf supports only 1 adapter with bias. When using multiple adapters, set bias to 'none' for all adapters.N)len peft_configbias ValueErrorr=__name__)r8r:r)r)r-_check_new_adapter_configs   z#LoraModel._check_new_adapter_configcCs t||Sr5)r) lora_configkeyr)r)r-_check_target_module_existss z%LoraModel._check_target_module_existsr@r9 nn.ModulecCs|jr t||jdSdS)a A private method to modify the model structure before adapter is applied. Args: peft_config (`PeftConfig`): The prepared adapter config. model (`nn.Module`): The model that is going to be adapted. N)layer_replicationr)r8r@r9r)r)r-_prepare_models zLoraModel._prepare_modelc sPdurtdtt|j|j}ttfdd|}|j||j } |j||j } | | |j |j |j |j|jt|jddt|jddd } gd} | D]} t|j| d }|durf|| | d <qRd d lm}t|trt||s|j|| | |j |j |j|jd dS|j|||fi| }||jkr|d|||||dS)NzCurrent Key shouldn't be `None`cstd|dS)Nz.*\.$)rematch)rF current_keyr)r-sz/LoraModel._create_and_replace..is_loaded_in_8bitFis_loaded_in_4bit) r lora_alpha lora_dropoutfan_in_fan_outinit_lora_weights use_rslorause_doraloaded_in_8bitloaded_in_4bit)gptqaqlmawq)method_quantization_configr) AdaLoraLayer)rTrUrWrXrY)rBlistr rank_patternkeys alpha_patternnextfiltergetrSrTrUrVrWrXrYgetattrr9rpeft.tuners.adalorara isinstancer# update_layer_create_new_moduleactive_adapterrequires_grad__replace_module)r8rEr;r* target_nameparentrO pattern_keystarget_name_keyrSalphar, quant_methods quant_methodquantization_configra new_moduler)rNr-_create_and_replacesJ        zLoraModel._create_and_replacecCst|||t|dr|j}t|ds |j|_t|dr |j|_t|dddur>t|dr3|j|j_n|j|_||jj| D]\}}|j |vsOd|vr`t|drW|j n|j}||jqBdS)N base_layerrAstateranknumqweight) setattrhasattrr{weightrArir|todevice named_modulesr2r~)r8rr child_namerychildnamemodulerr)r)r-rps$       zLoraModel._replace_modulecCs|D] \}}|j|vrd|_q|jD]H}|j|j}|dkr!q|dkr6|D] \}}d|vr4d|_q)q|dkrU|D]}t|trSt |drS|jdurSd|j_q>qt d|ddS) NFr>allrAT lora_onlyzRequested bias: z, is not implemented.) named_parametersr2 requires_gradactive_adaptersr@rAmodulesrkr#rNotImplementedError)r8r9nprnrAmr)r)r- _mark_only_adapters_as_trainables,    z*LoraModel._mark_only_adapters_as_trainablec Ksg}trddlm}||trddlm}|||ttt t t gd}|D]}|||fd|i|}|dur?nq,|durLt d|d|S)Nr)dispatch_bnb_8bit)dispatch_bnb_4bitrEzTarget module z is not supported. Currently, only the following modules are supported: `torch.nn.Linear`, `torch.nn.Embedding`, `torch.nn.Conv2d`, `transformers.pytorch_utils.Conv1D`.) rZpeft.tuners.lora.bnbrappendrrextendrrr!r%r$rB) rEr;r*r, dispatchersrrry dispatcherr)r)r-rms&     zLoraModel._create_new_modulercs.zt|WStyt|j|YSw)z1Forward missing attributes to the wrapped module.)r6 __getattr__AttributeErrorrir9)r8rr<r)r-r4s  zLoraModel.__getattr__F inferenceboolcCsFi}|jD]\}}ddt|D}|rd|d<q|||<|S)NcSs&i|]\}}|t|tr|jn|qSr))rkrvalue).0kvr)r)r- >s&z5LoraModel.get_peft_config_as_dict..Tinference_mode)r@itemsr)r8r config_dictrFrr:r)r)r-get_peft_config_as_dict;sz!LoraModel.get_peft_config_as_dictTenabledcCs,|jD]}t|ttfr||qdSr5)r9rrkrrenable_adapters)r8rrr)r)r-_set_adapter_layersDs  zLoraModel._set_adapter_layerscCs|jdddS)zyEnable all adapters. Call this if you have previously disabled all adapters and want to re-enable them. TrN)rr8r)r)r-enable_adapter_layersIszLoraModel.enable_adapter_layerscCsF|jD]}|j|j}|dkrd|d}t|q|jdddS)zDisable all adapters. When disabling all adapters, the model output corresponds to the output of the base model. r>z>Careful, disabling adapter layers with bias configured to be 'zP' does not produce the same output as the the base model would without adaption.FrN)rr@rAwarningswarnr)r8rnvalmsgr)r)r-disable_adapter_layersPs    z LoraModel.disable_adapter_layersr;str | list[str]cCsF|jD]}t|tr|jrtd|||q||_ dS)aSet the active adapter(s). Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is not desired, use the following code. ```py >>> for name, param in model_peft.named_parameters(): ... if ...: # some check on name (ex. if 'lora' in name) ... param.requires_grad = False ``` Args: adapter_name (`str` or `list[str]`): Name of the adapter(s) to be activated. zJAdapter cannot be set when the model is merged. Unmerging the model first.N) r9rrkr#mergedrrunmerge set_adapterrn)r8r;rr)r)r-r_s    zLoraModel.set_adaptercos|dd}|durdVdS|jrtdg}|D]}t|tr6tt|d}|j|dd}| |qdV|D]}| qCannot merge LORA layers when base model layers are replicated)rir9rBr@rhrr)r)r-_check_merge_alloweds  zLoraModel._check_merge_allowedcCs4|jdur|dtvrtdtt|d|_|S)N model_typez0Please specify `target_modules` in `peft_config`)target_modulesrrBset)r@ model_configr)r)r-_prepare_adapter_configs   z!LoraModel._prepare_adapter_configN progressbar safe_merger(Optional[list[str]]c s|rfddjD}d|rdndd}t|| |dD]g}z tj|\}} } Wn ty:Yq$wt| Dt| drY|rN| j||d  || | | n#t | t r|| j | j} t| drv|rr| j||d | } t|| | Wdn1swYq$jS) Ncg|] \}}j|vr|qSr)r2rrF_rr)r- z:LoraModel._unload_and_optionally_merge..z Unloading z and merging r9)disabledescr{)rr()rr9rr rrrrmergerpget_base_layerrkrmodules_to_savernr) r8rrrr(key_listrrFrrr*rqryr)rr-_unload_and_optionally_merges4       z&LoraModel._unload_and_optionally_mergesvdtotalmajority_sign_methodLiteral['total', 'frequency']c s|tjvr dS|D]} | tjvr td| dq t|dkr)dn|}fdd|D} |dvrItt| dkrDtd | d } n|d krRt| } n|d r^|p\t| } ntd |fdd|D}|swtd|tt|dkrtd|d t krd fdd|D}n|d tkrt t j fdd|D}n td|d dtj|d | | |dj|<j|tj|fddjD}|D]}tj|\}}}t|tr||jvr|j|j}|j|j}n||jvr|j|}|j|}nq|jd|_|jd|_|d krgg}}t||D]>\} }| |jvrG|j| j}|j| j}n| |jvrX|j| }|j| }nq0||j||j| ||jq0t|d krztdt j!|d d}t j!|dd}||jd|j"d ddf<||jddd|j"df<q|dvrj#|||| |||| | |||d \|_|_q|dvrӈ$||||| | \|_|_qdS)a This method adds a new adapter by merging the given adapters with the given weights. When using the `cat` combination_type you should be aware that rank of the resulting adapter will be equal to the sum of all adapters ranks. So it's possible that the mixed adapter may become too big and result in OOM errors. Args: adapters (`list`): List of adapter names to be merged. weights (`list`): List of weights for each adapter. adapter_name (`str`): Name of the new adapter. combination_type (`str`): The merging type can be one of [`svd`, `linear`, `cat`, `ties`, `ties_svd`, `dare_ties`, `dare_linear`, `dare_ties_svd`, `dare_linear_svd`, `magnitude_prune`, `magnitude_prune_svd`]. When using the `cat` combination_type, the rank of the resulting adapter is equal to the sum of all adapters ranks (the mixed adapter may be too big and result in OOM errors). svd_rank (`int`, *optional*): Rank of output adapter for svd. If None provided, will use max rank of merging adapters. svd_clamp (`float`, *optional*): A quantile threshold for clamping SVD decomposition output. If None is provided, do not perform clamping. Defaults to None. svd_full_matrices (`bool`, *optional*): Controls whether to compute the full or reduced SVD, and consequently, the shape of the returned tensors U and Vh. Defaults to True. svd_driver (`str`, *optional*): Name of the cuSOLVER method to be used. This keyword argument only works when merging on CUDA. Can be one of [None, `gesvd`, `gesvdj`, `gesvda`]. For more info please refer to `torch.linalg.svd` documentation. Defaults to None. density (`float`, *optional*): Value between 0 and 1. 0 means all values are pruned and 1 means no values are pruned. Should be used with [`ties`, `ties_svd`, `dare_ties`, `dare_linear`, `dare_ties_svd`, `dare_linear_svd`, `magnintude_prune`, `magnitude_prune_svd`] majority_sign_method (`str`): The method, should be one of ["total", "frequency"], to use to get the magnitude of the sign values. Should be used with [`ties`, `ties_svd`, `dare_ties`, `dare_ties_svd`] NAdapter  does not existr&linearcsg|]}j|jqSr))r@rSradapterrr)r-rsz2LoraModel.add_weighted_adapter..)rrrrrzkAll adapters must have the same r value when using combination_type linear, ties, dare_ties or dare_linear.rcatrzInvalid combination_type: csg|] }tj|jqSr))typer@rrrr)r-rsz'Found no adapter matching the names in zall adapter configs should follow the same target modules type. Combining adapters with `target_modules` type being a mix of list/set and string is not supported.|c3s$|] }dj|jdVqdS)()Nr@rrrr)r- s"z1LoraModel.add_weighted_adapter..c3s|] }j|jVqdSr5rrrr)r-r!sz Invalid type z found in target_modules)rSrTrcrr)rrrr)r-r1rgz9No matching LoRAs found. Please raise an issue on GitHub.)dim)rties_svddare_linear_svd dare_ties_svdmagnitude_prune_svd full_matricesdriver)rrrrr)%rbr@rdrBr?rsumendswithmaxr1joinroperatoror_ TypeErrorrinject_adapterr9rrrrkr#lora_Arlora_Blora_embedding_Alora_embedding_Bdataziprscalingtorchrshape1_svd_generalized_task_arithmetic_weighted_adapter-_generalized_task_arithmetic_weighted_adapter)r8adaptersweightsr;combination_typesvd_rank svd_clampsvd_full_matrices svd_driverdensityrradapters_ranksnew_ranktarget_module_typesnew_target_modulesrrFrr* target_lora_A target_lora_Bloras_Aloras_Brcurrent_adapter_lora_Acurrent_adapter_lora_Br)rr-add_weighted_adapters5                           zLoraModel.add_weighted_adapterc sDg} g}tfdd|D}t||D]\}}|jvs"|jvr1| |||j|qt| dkr.rz9No matching LoRAs found. Please raise an issue on Github.csg|]}|qSr))get_delta_weightrrr)r-rzOLoraModel._svd_generalized_task_arithmetic_weighted_adapter..rrrrrz*Invalid value passed to combination type: )r&r&r&) start_dimrVr)#anyrrrrrr?rBrtensorrrrrrrrrkr"rsizeflattensqueezerrVTlinalgrdiagrquantileclampreshaperr)r8rrrr r*rrr rr$rrvalid_adapters valid_weights is_embeddingrr delta_weightconv2d conv2d_1x1USVhdisthi_vallow_valr)rr-rns\         z;LoraModel._svd_generalized_task_arithmetic_weighted_adaptercsrg}g}g} t||D]@\} } | |jvr!|j| j} |j| j} n| |jvr1|j| } |j| } nq |t| |j | || j | | j q t | |dj}|| g}|djt|D]I\}}|dkrtt||||<qd|dkrt||||||<qd|dkrt|||||<qd|dkrt||||||<qd|dkrt|||||<qdtdfdd |D}|S) NrrrrrrzInvalid combination typecsg|]}|qSr))r)rdeltadtyper)r-rrzKLoraModel._generalized_task_arithmetic_weighted_adapter..)rrrrrrrmathsqrtrrrrrrr4 enumeraterrrrrrB)r8rrrr*r rr' lora_A_deltas lora_B_deltasrrrr lora_deltasi task_tensorsr)r3r-rs>        z7LoraModel._generalized_task_arithmetic_weighted_adaptercs|tjvrtd|dj|=fddjD}d}|D] }tj|\}}}t|trE| ||durE|j dd}q%|pIg_ dS)z Deletes an existing adapter. Args: adapter_name (str): Name of the adapter to be deleted. rrcrr)rrrr)r-rrz,LoraModel.delete_adapter..N) rbr@rdrBr9rrrkr#delete_adapterrrn)r8r;r new_adapterrFrr*r)rr-r=s  zLoraModel.delete_adaptertorch.nn.ModulecCs|j|||dS)aG This method merges the LoRa layers into the base model. This is needed if someone wants to use the base model as a standalone model. Args: progressbar (`bool`): whether to show a progressbar indicating the unload and merge process safe_merge (`bool`): whether to activate the safe merging check to check if there is any potential Nan in the adapter weights adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. Example: ```py >>> from transformers import AutoModelForCausalLM >>> from peft import PeftModel >>> base_model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b") >>> peft_model_id = "smangrul/falcon-40B-int4-peft-lora-sfttrainer-sample" >>> model = PeftModel.from_pretrained(base_model, peft_model_id) >>> merged_model = model.merge_and_unload() ``` )rrr(r)r8rrr(r)r)r-merge_and_unloadszLoraModel.merge_and_unloadcCs |jddS)z Gets back the base model by removing all the lora modules without merging. This gives back the original base model. F)rr@rr)r)r-unloads zLoraModel.unload)r3r4)r:r r3r4)r@r r9rH)r9rHr3r4)rr1)F)rr)T)rrr3r4)r;rr3r4)TFFN)rrrrr(r)rNNTNNr)rrr3r4)NTN)r;r1r3r4)FFN)rrrrr(rr3r?)r3r?)!rC __module__ __qualname____doc__r2__annotations__r7rD staticmethodrGrJrzrprrmrrrrrrrrrrrrrrr=rArB __classcell__r)r)r<r-r/>s^  F    8  !        ' 6 D ,  r/)> __future__rr5rrLr contextlibr dataclassesrrenumr functoolsrr itertoolsr typingr r rr r peft.import_utilsrrpeft.tuners.tuners_utilsrrrr peft.utilsrrrrrpeft.utils.merge_utilsrrrrrZpeft.tuners.lora.aqlmrZpeft.tuners.lora.awqrZpeft.tuners.lora.configr Zpeft.tuners.lora.gptqr!peft.tuners.lora.layerr"r#r$Zpeft.tuners.lora.tp_layerr%Z tuner_utilsr'r.r/r)r)r)r-s6