g#bddlmZddlmZddlmZmZmZmZddl Z ddl m Z ddl m Z m Z mZmZmZddlmZdd lmZdd lmZdd lmZmZmZeGd d ZeddddddZ ddZee j<de j>f ddZ Gdde jBZ"y)) annotations) dataclass)MappingOptionalTupleUnionN)nn)ComposeConvertImageDtypeLambda NormalizeToTensor)MultiresConvDecoder)DepthProEncoder) FOVNetwork)VIT_CONFIG_DICT ViTPreset create_vitcZeZdZUdZded<ded<ded<dZded <dZd ed <d Zd ed<y)DepthProConfigzConfiguration for DepthPro.rpatch_encoder_presetimage_encoder_presetintdecoder_featuresNz Optional[str]checkpoint_urizOptional[ViTPreset]fov_encoder_presetTbool use_fov_head)__name__ __module__ __qualname____doc____annotations__rrr@/home/cameronsmith/repos/ml-depth-pro/src/depth_pro/depth_pro.pyrrs7%####$(NM(.2+2L$r&r dinov2l16_384z./checkpoints/depth_pro.ptT)rrrrrrcd|tvrt|}t|d}||fStd|d)zCreate and load a backbone model given a config. Args: ---- preset: A backbone preset to load pre-defind configs. Returns: ------- A Torch module and the associated config. F)presetuse_pretrainedzPreset z not found.)rrKeyError)r+configmodels r'create_backbone_modelr01sD  (&? &= 455r&cpuc t|j\}}t|j\}}d}|jr%|jt|j\}}|j }|j } t|||| |j} t|jgt| jz|j} t| | d|j|j} |tjk(r| jt!t#t%fdt'gdgdt)|g} |j*tj,|j*d }| j/|d \}}t1|d k7rt3d||Dcgc] }d|vs| }}t1|d k7rt3d|| | fScc}w)aCreate a DepthPro model and load weights from `config.checkpoint_uri`. Args: ---- config: The configuration for the DPT model architecture. device: The optional Torch device to load the model onto, default runs on "cpu". precision: The optional precision used for the model, default is FP32. Returns: ------- The Torch DepthPro model and associated Transform. )r+N) dims_encoder patch_encoder image_encoderhook_block_idsr)r3 dim_decoder) r)encoderdecoder last_dimsr fov_encoderc&|jS)N)to)xdevices r'z-create_model_and_transforms..sQTT&\r&)?rBrBr1) map_locationT) state_dictstrictrz.Found unexpected keys when loading monodepth: fc_normz)Keys are missing when loading monodepth: )r0rrrrencoder_feature_dimsencoder_feature_layer_idsrrrlistr3DepthPror>torchhalfr rr r r rloadload_state_dictlenr-)r.r@ precisionr4patch_encoder_configr5_r<r3r6r9r:r/ transformrD missing_keysunexpected_keyskeys ` r'create_model_and_transformsrWHs$+@**+'M'-**M1K v88D.f6O6OP Q'<) % o  1 $@@QR  (4Ly7KL L |  !F|nUV V )  Ms : G*G*ceZdZdZ d dfd ZeddZd dZej d d dZ xZ S) rJzDepthPro network.c Xt|||_||_|j}t j t j||dzdddt j|dz|dzddddt j|dz|ddddt jdt j|d|ddddt j|_ |jdjjjd|rt|| |_y y ) amInitialize DepthPro. Args: ---- encoder: The DepthProEncoder backbone. decoder: The MultiresConvDecoder decoder. last_dims: The dimension for the last convolution layers. use_fov_head: Whether to use the field-of-view head. fov_encoder: A separate encoder for the field of view. r) kernel_sizestridepaddingrT) in_channels out_channelsr\r]r^bias) num_featuresr<N)super__init__r9r:r7r SequentialConv2dConvTranspose2dReLUheadradatafill_rfov)selfr9r:r;rr<r7 __class__s r'rezDepthPro.__init__s&   )) MM II-1QPQ    '1,(A-   IIq !    GGDM IIilIaLaST U GGI+  2 ! $$Q' !{ TDH r&c.|jjS)z.Return the internal image size of the network.)r9img_size)rns r'rqzDepthPro.img_sizes||$$$r&cH|j\}}}}||jk(r||jk(sJ|j|}|j|\}}|j |}d} t |dr*|j j||j} || fS)aDecode by projection and fusion of multi-resolution encodings. Args: ---- x (torch.Tensor): Input image. Returns: ------- The canonical inverse depth map [m] and the optional estimated field of view [deg]. Nrm) shaperqr9r:rjhasattrrmforwarddetach) rnr?rRHW encodingsfeatures features_0canonical_inverse_depthfov_degs r'ruzDepthPro.forwardsWW 1aDMM!a4==&88LLO #||I6*"&))H"5 4 hh&&q**;*;*=>G&//r&c t|jdk(r|jd}|j\}}}}||jk7xs||jk7}|r9tj j ||j|jf|d}|j|\}} |Nd|ztjdtj| jtjzz }|||z z} |j}|r%tj j | ||f|d} dtj| dd z } | j|d S) aInfer depth and fov for a given image. If the image is not at network resolution, it is resized to 1536x1536 and the estimated depth is resized to the original image resolution. Note: if the focal length is given, the estimated value is ignored and the provided focal length is use to generate the metric depth values. Args: ---- x (torch.Tensor): Input image f_px (torch.Tensor): Optional focal length in pixels corresponding to `x`. interpolation_mode (str): Interpolation function for downsampling/upsampling. Returns: ------- Tensor dictionary (torch.Tensor): depth [m], focallength [pixels]. r[rF)sizemode align_cornersrBg?g-C6?g@)minmax)depthfocallength_px)rOrs unsqueezerqr functional interpolaterurKtandeg2radr>floatsqueezeclamp) rnr?f_pxinterpolation_moderRrwrxresizer|r} inverse_depthrs r'inferzDepthPro.infersD2 qww<1  AAWW 1admm#9qDMM'9  ))mmT]]3'# *A,0<<?( <7UYYsU]]7::ekk;R-S'STTD/1t8< ||~ MM55QF1CSX6Mekk-TsCC]]_"  r&)TN) r9rr:rr;ztuple[int, int]rrr<zOptional[nn.Module])returnr)r? torch.Tensorrz+Tuple[torch.Tensor, Optional[torch.Tensor]])Nbilinear)r?rrz$Optional[Union[float, torch.Tensor]]rzMapping[str, torch.Tensor]) r r!r"r#repropertyrqrurKno_gradr __classcell__)ros@r'rJrJs"+/ 6U 6U%6U# 6U  6U ) 6Up%%02U]]_6:% 6 6 36 $ 6 6 r&rJ)r+rrzTuple[nn.Module, ViTPreset])r.rr@z torch.devicerPz torch.dtyperzTuple[DepthPro, Compose])# __future__r dataclassesrtypingrrrrrKr torchvision.transformsr r r r rnetwork.decoderrnetwork.encoderr network.fovrnetwork.vit_factoryrrrrDEFAULT_MONODEPTH_CONFIG_DICTr0r@float32rWModulerJr%r&r'rs #!22 1,#GG     !/((/& !  0;'5<<."]]O O OO OdP ryyP r&