o ! j@sdZddlZddlZddlZddlZdD] ZeeZeeje<qddejd_ ddejd_ d dejd_ ddl Z ddl mZdd lmZd Zd Zd ZdZdZdZdZGdddejZedkre e jrndndZeeZe dde!DZ"e#de"de $ddeeeZ%e & ee%Z'Wdn1swYe'(D]>\Z)Z*e+e*dre#de)de,e*j-qe.e*e/re#de)de0e*de*re+e*ddre#d e,e*dj-qej1dkr e#d!e j2d"d#d$dSdSdS)%uzDA3-based volume model with factored KV-attention. Architecture (Cameron 2026-05-18 spec): - DA3 backbone + DPT head; aux head's final 1×1 conv replaced to output KEY_DIM (+1 conf). - Per-pixel feature F ∈ R^(B × KEY_DIM × H × W) — "value/query" stream. - Learnable height embeddings h_emb ∈ R^(N_HEIGHT_BINS × H_DIM). - Learnable time embeddings t_emb ∈ R^(N_WINDOW × T_DIM). - Key per (t, z): key(t, z) = concat(t_emb[t], h_emb[z]) ∈ R^(T_DIM + H_DIM = KEY_DIM). - Volume logits via bilinear scoring: l(b, t, z, u, v) = F(b, :, u, v) · key(t, z) / sqrt(KEY_DIM). We replace the original libero (B, N_WINDOW * N_HEIGHT_BINS, H, W) dense head with the factored KV decomposition: parameter-efficient, structural inductive bias that height/time are categorical attributes that share representation across spatial locations. forward(rgb) returns: volume_logits: (B, N_WINDOW, N_HEIGHT_BINS, h_out, w_out) pred_depth: (B, H, W) — for distillation against frozen DA3 depth dino_feats: list of intermediate DINO features (for PCA viz) pixel_feats: (B, KEY_DIM, h_out, w_out) — for debugging / viz N)depth_anything_3.utils.export!depth_anything_3.utils.pose_aligncOdSNakrr0/data/cameron/para/libero/model_da3_volume_v2.pyr rcOrrrrrrr r r rcOrrrrrrr r r )DepthAnything3z/data/cameron/da3_large_weightsi 0c sVeZdZeeeeeedfde de de de de de f fdd Z d d Z d d Z ZS)DA3VolumeModelN weights_pathn_window n_height_binskey_dimtime_dim height_dimc s<t||kr ||ksJ||_||_||_||_||_t|}|j j |_ |j j |_ ~|dur?t t |j dgd|_nt ||_|j jjd} | d} tj| j|ddddd} tj| jtjj| jdd| | d<t|||_t|||_tjj|jjd d d d tjj|jjd d d d t||_dS) NZ out_layers) r)Z kernel_sizeZstrideZpaddingg?)stdg?gɿg?)rrb)super__init__rrrrrr Zfrom_pretrainedZmodelbackboneheadlistgetattrdino_feat_layersZscratchZoutput_conv2_auxnnZConv2dZ in_channelsinitZzeros_ZbiasZnormal_weightZ Embeddingh_embt_embZ trunc_normal_Z LayerNorm pixel_norm) selfrrrrrrr&ZfullZ last_aux_seqZold_convZnew_conv __class__rr r!/s2     zDA3VolumeModel.__init__cCs0|j}|j}|jjd}|jjd}||S)uv2: SUM of full-dim time and height embeddings (instead of concat halves). Returns keys: (n_window, n_height_bins, key_dim). key(t, z) = t_emb[t] + h_emb[z] (both ∈ R^key_dim). rr)rrr+r) unsqueezer*)r-TZZt_eZh_errr _build_keysYs zDA3VolumeModel._build_keyscCs"|d}tjr tjntj}tj|jj|d|j |d|j dd\}}Wdn1s0wY|j d|j d}}tj|jjdd |j |||d d }Wdn1s]wY|d } | d krv| ddd d f} n| dkr| ddd f} n tdt| j |d} | d kr| ddd fd ddd} n| dkr| d ddd} n tdt| j | j \} }}}|| d dddd ddd}|}dt|j}td|||}g}|D]}t|ttfr||d q||q|| | |dS)z rgb: (B, 3, 504, 504) in [0, 1].r) device_typeZdtypeNZsaddle_balanced)Z cam_tokenZexport_feat_layersZref_view_strategyrF)r4enabledr)Zpatch_start_idxdepthrzunexpected depth shape rayzunexpected ray shape g?zbchw, tzc -> btzhw) volume_logits pred_depth pixel_feats dino_feats)r0torchcudaZis_bf16_supportedZbfloat16Zfloat16Zautocastdevicetyper"r&shaper#dim RuntimeErrortupleZpermuter,r3mathZsqrtrZeinsum isinstancer$append)r-rgbxZautocast_dtypeZfeatsZ_auxHWZhead_outr7r=r9r>BZCfZHfZWfZf_normkeysZscaler<r?Z layer_featsrrr forwarddsP      " zDA3VolumeModel.forward)__name__ __module__ __qualname__DA3_WEIGHTS_DEFAULTN_WINDOW N_HEIGHT_BINSKEY_DIMTIME_DIM HEIGHT_DIMstrintr!r3rQ __classcell__rrr.r r.s$* r__main__rAZcpuccs|] }|jr|VqdSr)Z requires_gradZnumel).0prrr sraz Trainable: ,r;r:rDz z: z: list()z first: zpeak: geAz.2fz GB)3__doc__systypesosrHn ModuleTypemmodulesZexportZalign_poses_umeyamaZbatch_align_poses_umeyamar@Ztorch.nnr'Zdepth_anything_3.apir rUZ DA3_INPUTrVrWrXrYrZZModulerrRrBrAZ is_availabletoevalsumZ parametersZn_tprintrandrKZno_gradoutitemsr vhasattrrGrDrIr$lenrCZmax_memory_allocatedrrrr sL    n    *