qi/ ddlZddlmZddlZddlZddlZddlZddlmZm Z ddl Z ddl m Z ddl m Z ddl mZddlZddlmZddlZddlmZddlZddlZddlmZddlmZddlmcmZ ej!d Z"e d ej#ej$zej%z Z&Gd d ej'Z(dZ)dZ*e j+e j,Gddee&Z-e j.e&dfZ/de)e*dde j0dzde-de1dee2de3e4e4fde-f dZ5ej,dGddej6Z7ej,Gd d!e j8ej6Z9ej#ddd"d#ej:e2zd$e;ej%e;ej#zd%ej<dzd&ej=j>dzde j?f d'Z@dS)(N)Sequence)GenericTypeVar)nnx)struct) traverse_util) pi0_pytorch) image_toolsopenpiArrayT)boundceZdZdZdZdZdZdS) ModelTypezSupported model types.pi0pi0_fastpi05N)__name__ __module__ __qualname____doc__PI0PI0_FASTPI054/home/robot-lab/Pi0.5_yam/src/openpi/models/model.pyrrs#  CH DDDrr) base_0_rgbleft_wrist_0_rgbright_wrist_0_rgb)r ceZdZUdZeeejedffe d<eeej edffe d<ejedfe d<dZ ej ed fdze d <dZ ej ed fdze d <dZej ed fdze d <dZej ed fdze d <edejeddfdZdejefdZdS) ObservationzHolds observations, i.e., inputs to the model. See `Observation.from_dict` to see the expected dictionary form. This is the format that should be produced by the data transforms. z*b h w cimagesz*b image_masksz*b sstateNz*b ltokenized_prompttokenized_prompt_mask token_ar_masktoken_loss_maskdatareturnzObservation[ArrayT]c d|vd|vkrtd|dD]}|d|jtjkr>|d|tjdz dzdz |d|<at |d|drt|d|jtjkrS|d|tj d d d d dz dzdz |d|<||d|d |d| d| d| d| dS)zsThis method defines the mapping between unstructured data (i.e., nested dict) to the structured Observation format.r&r'zEtokenized_prompt and tokenized_prompt_mask must be provided together.imagego@@?dtyper image_maskr%r(r)r#r$r%r&r'r(r)) ValueErrorr0npuint8astypefloat32hasattrtorchtopermuteget)clsr*keys r from_dictzObservation.from_dictms $ &,Ct,K L Ldee e= r rCG}S!'2833%)']3%7%>%>rz%J%JU%RUX%X[^%^W c""gs+W55 r$w-:L:RV[Va:a:a%)']3%7%:%:5=%I%I%Q%QRSUVXY[\%]%]`e%ehk%knq%qW c"s=\*w-!XX&899"&((+B"C"C((?33 HH%677    rctj|}|d|d<|d|d<|S)z)Convert the Observation to a nested dict.r#r-r$r4) dataclassesasdictpop)selfresults rto_dictzObservation.to_dictsB#D)) **X..w%zz-88| r)rrrrdictstratFloatr __annotations__Boolr&Intr'r(r) classmethodPyTreerBrIrrrr"r"QsX bhvz122 3333c2764<001111 8FFN ####7;bfVV^,t3:::<@2766>2T9@@@ 48M26&&.)D07776:ORWVV^,t3::: RYv. 3H   [ *6*rr"z*b ah adF)train image_keysimage_resolutionrng observationrSrTrUr+c t||js'td|dt |j|jjdd}i}|D]d}|j|}|jdd|krBtd|d|jddd |tj |g|R}|r|d z d z}g} d |vrr|jdd\} } | tj t| d zt| d ztj| | tjdgz } | tjddd gz } t"j||jd} t#jtj| | |}|d zdz }|||<fi} |D]Q}||jvr$t/j|t.j| |</t/j|j|| |<Rt7|| |j|j|j|j|jS)zPreprocess the observations by performing image augmentations (if train=True), resizing (if necessary), and filling in a default image mask (if necessary). z#images dict missing keys: expected z, got Nr2r1zResizing image z from z to r.g?wristgffffff?)g333333?g?) brightnesscontrast saturationrr/)r0r5) setissubsetr#r6listr%shapeloggerinfor resize_with_padaugmax RandomCropintResizeRotate ColorJitterjaxrandomsplitvmapChainr$jnponesboolasarrayr"r&r'r(r))rVrWrSrTrU batch_shape out_imagesrAr- transformsheightwidthsub_rngs out_maskss rpreprocess_observationr}s z?? # #K$6 7 7mkzkkQUVaVhQiQikklll#)#2#.KJ  "3' ;qs / / / KK]#]]U[15E]]K[]] ^ ^ ^/I8HIIIE  &CK#%EJc!! % AaC 0 %c%$,&7&7Vd]9K9KLLM%00M'** "cCCPPP Jz''U[^<>)E**%  W(889K9K9M9MvVVF  %*<*<*>*>FY]lqrrrr ""6***y5)))r weight_pathctd|tj|j}t j|||S)Nztrain_config: )config)rdrer PI0Pytorchr safetensorsr< load_model)rG train_configrrs r load_pytorchzBaseModelConfig.load_pytorchsQ 3\33444&l.@AAA$$UK888 rr2 batch_sizercdS)zOReturns the input specification for the model. Values are jax.ShapeDtypeStruct.Nr)rGrs r inputs_speczBaseModelConfig.inputs_specrrcv||\}}tjd|S)Nrc@tj|j|jSNrrrsrcr0xs rz*BaseModelConfig.fake_obs..chqw&@&@rrrmtreemap)rGrobservation_spec_s rfake_obszBaseModelConfig.fake_obss9"..*.EE!x||@@BRSSSrcv||\}}tjd|S)Nrc@tj|j|jSrrrs rrz*BaseModelConfig.fake_act..rrr)rGrr action_specs rfake_actzBaseModelConfig.fake_acts6))Z)@@;x||@@+NNNrN)r2)rrrrrirNpropertyabcabstractmethodrrrL KeyArrayLikerParamsrtrrKrtupler"Actionsrrrrrrrrs OOO IX ;"/;k;;;;FJ***29*d*k****c  /0^^^^U;;O5P^^^^TT3T{TTTTOO3OwOOOOOOrrceZdZUdZeed<eed<eed<ejdddej de d e d e d ej ejd ff d Zejdej de d e fdZdS)rzBase class for all model implementations. Specific models should inherit from this class. They should call super().__init__() to initialize the shared attributes (action_dim, action_horizon, and max_token_len). rrrF)rSrVrWactionsrSr+z*b ahcdSrr)rGrVrWrrSs r compute_losszBaseModel.compute_losss '*crc dSrr)rGrVrWkwargss rsample_actionszBaseModel.sample_actionsscfcfrN)rrrrrirNrrrLrr"rrtrMArrayrrrrrrrsOOO *** _*!* *  * "(G# $**** f"/f fZaffffffrr) restore_typer0sharding params_pathrr0rc t|ds&tj|n|}t juromt jt j d}t j |t j tj 5}||}d|di}||tj|t jfd|d}dddn #1swxYwYt)j|} t-d| Drd| D} t)j| S) aRestores unstructured params PyTree from a checkpoint. This works with checkpoints saved with `save_state` during openpi training (see `training/checkpoints.py`) as well as pre-trained checkpoints released for openpi. Args: params_path: The local path to the checkpoint directory. restore_type: The type to restore the params as. Can be set to `np.ndarray` to load the params as a numpy array. dtype: The dtype to restore all params as. If not provided, will use the original dtype from the checkpoint. sharding: The sharding to use for the params. If not provided, the params will be replicated across all devices. Returns: The restored params. zgs://Nrrc2tjS)N)rrr0)rArrayRestoreArgs)rr0rrs rrz restore_params..Bsc2HS_glmmmr)item restore_argsc3.K|]}|ddkVdS)rYvalueNr).0kps r z!restore_params..Js+ 3 32b6W  3 3 3 3 3 3rc*i|]\}}|dd|S)NrYr)rrvs r z"restore_params..Ks&CCCeb!r#2#wCCCr)rK startswithpathlibPathresolvermrrMeshdevices NamedSharding PartitionSpecrPyTreeCheckpointermetadatarestoreargs PyTreeRestorerrr flatten_dictallitemsunflatten_dict) rrr0rmeshckptrrrr flat_paramss ``` rrestore_paramsrs*>A=M=M=X=XY`=a=ar',{++33555grKsy  X%5|  77<--dCL4N4N4P4PQQ  ! ! U>>+..(8,-  H " " X\\mmmmmmos #                      ,V44K 3 3{ 3 3 333DCC{/@/@/B/BCCC  ' 4 44sA>E((E,/E,)Arcollections.abcrrDenumloggingrtypingrrrgflaxrrrrm jax.numpynumpyrrr7orbax.checkpoint checkpointrrr<openpi.models_pytorchr openpi.sharedr openpi.shared.array_typingshared array_typingrL getLoggerrdrTensorndarrayr Enumr IMAGE_KEYSIMAGE_RESOLUTION typecheck dataclassr"rMrrrtrKrrir}ABCrModulerrtyper0rShardingrrrrrrs $$$$$$ ########   ------%%%%%%'''''''''  8 $ $ U\!9BJ!F G G G  D55555'&/555t (6:% & *(8 @@@ 4 @@  @  @ CHo @@@@@Fd###/O/O/O/O/Ocg/O/O$#/Od ggggg CGggg48;y"-1 .5.5.5#.5rz"T#)_4.5 9t  .5 l#d* .5 Y .5.5.5.5.5.5r