o ݱii!@sddlmZddlZddlZddlmZddlZddlmZddl m Z dddZ dej d eej ge fd eej gej fd ej fd d ZdddZdddZddZGdddej ZGdddej ZGdddej ZdS))CallableN) rearrange NormalizeTcCstgdgd|dS)z8 Construct an ImageNet normalization transform. )g ףp= ?gv/?gCl?)gZd;O?gy&1?g?)meanstdinplacer)rr @/data/cameron/vidgen/unified-world-model/models/common/vision.pyget_imagenet_norm sr root_module predicatefuncreturnc s|r||Sfdd|jddD}|D]>^}}|}t|dkr,|d|}t|tjr9|t|}nt||}||}t|tjrO||t|<qt |||qfdd|jddD}t|dkskJ|S)a Recursively replace submodules that satisfy a given predicate. Args: root_module (nn.Module): The root module to process. predicate (Callable[[nn.Module], bool]): A function that takes a module as input and returns True if the module should be replaced. func (Callable[[nn.Module], nn.Module]): A function that takes a module as input and returns a new module to replace it. **kwargs: Additional keyword arguments to be passed to the ResNet model constructor. c"g|] \}}|r|dqS.split.0kmr r r ( z&replace_submodules..T)remove_duplicaterrcrrrrrr r r<r) named_moduleslen get_submodulejoin isinstancenn Sequentialintgetattrsetattr) r r r module_listparentr parent_moduleZ src_moduleZ tgt_moduler rr replace_submoduless*        r*cKsNttj|}|dd|i|}t|jj||_|r%t|ddddd}|S)a* Construct a ResNet model with a custom output embedding dimension and optional batch norm replacement. Args: name (str): The name of the ResNet architecture to use (e.g., "resnet18", "resnet34", "resnet50"). embed_dim (int): The dimension of the output embedding. weights (Optional[str]): Pre-trained weights to load (e.g., "IMAGENET1K_V1"). If None, no pre-trained weights are used. replace_batch_norm (bool, optional): If True, replaces `nn.BatchNorm2d` layers with `nn.GroupNorm` layers. Default is True. weightscSs t|tjSN)r!r" BatchNorm2dxr r r Us zget_resnet..cSstj|jd|jdS)N) num_groups num_channels)r" GroupNorm num_featuresr.r r r r0Vs)r r rNr )r% torchvisionmodelsr"Linearfc in_featuresr*)name embed_dimr+Zreplace_batch_normkwargsrresnetr r r get_resnetEs r?cKs0ttj|}|dd|i|}td||_|S)a Construct a Vision Transformer (ViT) model with a custom output embedding dimension. Args: name (str): The name of the ViT architecture to use (e.g., "vit_b_16", "vit_b_32", "vit_l_16", "vit_l_32", "vit_h_14"). embed_dim (int): The dimension of the output embedding. weights (Optional[str]): Pre-trained weights to load (e.g., "IMAGENET1K_V1"). If None, no pre-trained weights are used. **kwargs: Additional keyword arguments to be passed to the ViT model constructor. r+Nr )r%r6r7r"r8heads)r;r<r+r=rvitr r r get_vit^s rCcKs(tj dddi|}td||_|S)a Construct a pretrained CLIP encoder with a custom output embedding dimension. Args: embed_dim (int): The dimension of the output embedding. **kwargs: Additional keyword arguments to be passed to the timm model creation function. ,hf_hub:timm/vit_base_patch32_clip_224.openai pretrainedTr@N)rD)timm create_modelr"r8head)r<r=clipr r r get_clipnsrJc:eZdZdZdedeffdd ZdejfddZZ S) ResNetImageEncodera Multi-view image encoder using a ResNet backbone. The input is expected to be a tensor with shape (B, V, C, T, H, W), where: - B is the batch size, - V is the number of views, - C is the number of channels, - T is the number of frames, - H and W are the image height and width, respectively. The encoder reshapes the input to treat each view and frame as an individual image, extracts features using the ResNet model, and then concatenates the features across all views and frames. Args: num_views (int): Number of camera views in the input. embed_dim (int): Dimension of the output embedding features. num_viewsr<c,t||_t|_td|dd|_dS)Nresnet18 IMAGENET1K_V1r+)super__init__rMr normr?modelselfrMr< __class__r r rS zResNetImageEncoder.__init__imgscCs@|jdd\}}t|d}|||}t|d||d}|S)Nb v c t h w -> (b v t) c h w(b v t) c -> b (v t c)bv)shaperrUrT)rWr[BVfeatsr r r forwards  zResNetImageEncoder.forward __name__ __module__ __qualname____doc__r$rStorchTensorrf __classcell__r r rXr rL}srLcrK) ViTImageEncoderz Multi-view image encoder using a Vision Transformer (ViT) backbone. Args: num_views (int): Number of camera views in the input. embed_dim (int): Dimension of the output embedding features. rMr<crN)Nvit_b_32rPrQ)rRrSrMr rTrCrUrVrXr r rSrZzViTImageEncoder.__init__r[cCs|jdd\}}t|d}||}|j|}|jj|jddd}tj||gdd}|j |}|j |dddf}t|d||d}|S) Nr\r]rdimr^r_) rbrrTrU_process_input class_tokenexpandrlcatencoderrArWr[rcrdr/Zbatch_cls_tokenrer r r rfs    zViTImageEncoder.forwardrgr r rXr rosrocs>eZdZdZdededeffdd Zdejfdd ZZ S) ViTImagePatchEncodera; Multi-view image patch encoder using a Vision Transformer (ViT) backbone with learnable positional embeddings. Args: num_views (int): Number of camera views in the input. num_frames (int): Number of frames per view. embed_dim (int): Dimension of the output embedding features. rM num_framesr<cslt||_t|_td|dd|_tjt d||d|dd|_ tjt d||d|dd|_ dS)NrprPrQrrT) requires_grad) rRrSrMr rTrCrUr" Parameterrlzeros pos_shift pos_scale)rWrMr|r<rXr r rSs  zViTImagePatchEncoder.__init__r[cCs|jdd\}}t|d}||}|j|}|jj|jddd}tj||gdd}|j |}|j |}t|d||d}|d|j |j }| ddS) Nr\r]rrqrrrsz(b v t) n c -> b (v t) n cr_)rbrrTrUrurvrwrlrxryrArrflattenrzr r r rfs      zViTImagePatchEncoder.forwardrgr r rXr r{s r{)T)NTr,)typingrrFrltorch.nnr"r6einopsrZtorchvision.transformsrr Moduleboolr*r?rCrJrLror{r r r r s.       1 !"