o ÁB jÇ<ã@sˆdZddlZddlZddlZddlmZddlmmZej   dd¡Z ej   dd¡Z dZ dZd Zd Zd Zd Zd Zdd„Zdd„ZGdd„dejƒZedkrÂe ej ¡rWdnd¡Zeddd e¡Zedd„e ¡DƒƒZ edd„e ¡DƒƒZ!e"de d›de!d›ƒd\Z#Z$e %e#e$d d d ¡ e¡Z&e 'e#e$d!¡ e¡d Z(e )¡ ee&e(ƒZ*Wdƒn1sµwYe"d"e*j+ƒdSdS)#uÚAutoregressive transformer policy with high-res patches + relative-to-EEF positional embeddings. Architecture (per Cameron's spec, 2026-05-16): Per-frame patches (frozen DINOv3 ViT-S/16, 28x28 = 784 patches, D=384) + 1 learnable EEF token per timestep History context H frames (default 8) Three summed positional embeddings on every token: (1) temporal PE — which timestep (0..H-1) (2) absolute 2D PE — patch position (or EEF position) in [0,1]^2 (3) relative-to-EEF PE — (xy - EEF(t-1).xy), the anchor inductive bias for free-space motion Causal mask along time; within a frame all tokens are bidirectional. Readout: take the EEF_query token at t = H-1, MLP -> logits over a 56x56 spatial grid Loss: cross-entropy on grid cell containing next-EEF GT. First-pass scope: xy-only output (no height / rotation / gripper — bolt on later as parallel readout queries). Teacher-force past EEF coords from data; predict next-step (single step) EEF cell. éNÚ DINO_REPO_DIRz;/Users/cameronsmith/Projects/robotics_testing/random/dinov3ÚDINO_WEIGHTS_PATHzt/Users/cameronsmith/Projects/robotics_testing/random/dinov3/weights/dinov3_vits16plus_pretrain_lvd1689m-4057cbaa.pthééé8i€ééiÀcCsV|d}t tj|||dt d¡| ¡}| d¡|}tj| ¡| ¡gddS)zKpositions: (...,) float in any range. dim must be even. returns (..., dim).é©ÚdeviceÚdtypegˆÃ@éÿÿÿÿ©Údim) ÚtorchÚexpÚarangeÚmathÚlogÚ unsqueezeÚcatÚsinÚcos)Ú positionsrr r ÚhalfZfreqsÚangles©rú1/data/cameron/para/libero/model_autoregressive.pyÚ _sincos_pe_1d's&rcCs>|d}t|d|||ƒ}t|d|||ƒ}tj||gddS)z=xy: (..., 2). dim must be divisible by 4. returns (..., dim).r ).r).ér r)rrr)Úxyrr r rÚpxÚpyrrrÚ _sincos_pe_2d/sr#csLeZdZeeeeeedf‡fdd„ Z dd„Z dd„Z dd „Z d d „Z ‡ZS) ÚARTransformerPolicyTc sètƒ ¡||_||_||_||_t|_||j|_|jd|_ t d  |¡ƒt j jtddtd|_|rD|j ¡D]}d|_q9|j ¡|jj|_|j|ksZJd|›d|j›d ƒ‚||_t t  |¡d ¡|_t t  |¡d ¡|_t t  |¡d ¡|_t t d|¡t ¡t ||¡¡|_ tj!||d |d d ddd} tj"| |d|_#t t $|¡t ||¡t ¡t |||¡¡|_%t  &|j¡d|j} t  &|j¡d|j} t j'| | dd\} } |j(dt j)| | gdd *|j d¡dddS)Nr z&Loading DINOv3 backbone (frozen={})...Zdinov3_vits16plusÚlocal)ÚsourceÚweightsFz d_model (z) must equal DINO embed_dim (u?) — otherwise add a linear projection before the transformer.g{®Gáz”?rgÚgeluT)Úd_modelÚnheadÚdim_feedforwardÚdropoutÚ activationÚ batch_firstÚ norm_first)Ú num_layersgà?Úij)ÚindexingÚ patch_xy_01r r)Ú persistent)+ÚsuperÚ__init__Ú target_sizeÚ history_lenÚ grid_sizer)ÚDINO_PATCH_SIZEZ patch_sizeZpatches_per_sideÚ n_patchesÚprintÚformatrÚhubÚloadrrÚdinoÚ parametersÚ requires_gradÚevalÚ embed_dimÚfreeze_backboneÚnnÚ ParameterÚrandnÚ eef_tokenÚtype_embed_patchÚtype_embed_eefÚ SequentialÚLinearÚGELUÚ rel_pe_mlpÚTransformerEncoderLayerÚTransformerEncoderÚ transformerÚ LayerNormÚreadoutrÚmeshgridÚregister_bufferÚstackÚreshape)Úselfr7r8r9r)Zn_headsZn_layersrEÚpÚlayerÚysÚxsÚgrid_yÚgrid_x©Ú __class__rrr68s`   ü   ÿÿ þý ýzARTransformerPolicy.__init__c Csä|jrˆt ¡u|j |¡\}\}}|jjD]}|jjr$|jj||dnd}|||ƒ}q|jjr^|j |dd…d|jj d…f¡}|j  |dd…|jj dd…f¡}tj ||gdd}n|j  |¡}|dd…|jj dd…f} Wdƒ|   ¡S1swY|   ¡S|j |¡\}\}}|jjD]}|jjr¤|jj||dnd}|||ƒ}q–|jjrÞ|j |dd…d|jj d…f¡}|j  |dd…|jj dd…f¡}tj ||gdd}n|j  |¡}|dd…|jj dd…fS)z8x: (N, 3, H, W). returns patch tokens (N, n_patches, D).)ÚHÚWNrr) rErÚno_gradr@Zprepare_tokens_with_masksÚblocksZ rope_embedZuntie_cls_and_patch_normsZcls_normZn_storage_tokensÚnormrÚdetach) rYÚxÚtokensZH_pZW_pZblkZ rope_sincosZcls_nZpat_nÚpatchesrrrÚ _dino_patches…s4   $$  õ ô   $$ z!ARTransformerPolicy._dino_patchescCsD|j}|j}|j}|j}tj|||d}t||||ƒ} t|j  |¡|||ƒ} |dd…ddd…f} |j  |¡  d¡  ||d¡} | |   d¡d} |  | ¡}t||||ƒ}||   d¡d}|  |¡}|   d¡|}g}t |ƒD]+}|| |}|dd…|f|dd…|f| |  d¡}| tj||gdd¡qotj|ddS) ušBuild the summed positional embedding for every token in the sequence. Sequence layout per batch element: for t = 0..H-1: patch tokens (Np) — at frame t EEF token (1) — at frame t (query when t == H-1) Total tokens per frame: Np + 1; total length: H * (Np + 1). history_eef_xy_01: (B, H, 2) in [0,1] image coords. The EEF token at the LAST frame is the prediction query; we pass the previous frame's EEF as that token's positional anchor (since the current EEF is unknown at prediction time, by spec "Relative-to-current-EEF uses EEF(t-1) as anchor"). r Nr rr rg@r)r r r)r;rrrr#r3ÚtorÚexpandrOÚrangeÚappendr)rYÚBrbZhistory_eef_xy_01r r ÚDÚNpZtime_idxZtime_peZ patch_abs_peZ anchor_xy_01Zpatch_xyZ patch_rel_xyZ patch_rel_peZ eef_abs_peZ eef_rel_xyZ eef_rel_peZpatch_pe_per_frameÚfullÚtZp_peZe_perrrÚ_positional_embeddings¤s*    .z*ARTransformerPolicy._positional_embeddingsc Csl|j}|d}||}tj|||d}| tjtj|||tjdddtdƒ¡|j|ddj|dd}|S)aCausal across time: token in frame t may NOT attend to tokens in frame > t. Within a frame, all tokens (patches + eef) attend bidirectionally. Returns additive mask (T, T) with 0 / -inf entries, suitable for nn.TransformerEncoder. r)r r )Údiagonalz-infrr) r;rÚzerosÚ masked_fill_ÚtriuÚonesÚboolÚfloatÚrepeat_interleave)rYrbr rrZ per_frameÚTÚblockÚmaskrrrÚ_build_causal_time_maskãs*z+ARTransformerPolicy._build_causal_time_maskc Csž|j\}}}}}||jksJd|j›d|›ƒ‚||jkr"||jks4Jd|j›d|j›d|›d|›ƒ‚| |||||¡}| |¡} |  |||j|j¡} | |j} |j|j } |  dd|j¡  |||j¡} g} t |ƒD] } t j | dd…| f| dd…| | d…fgdd}|  |¡qlt j | dd}|t|jƒ}| |||¡}||}| ||j¡}|j||d}||jdd}|dd…|dd…f}| |¡}|S) uð Args: history_imgs: (B, H, 3, 448, 448) — past H frames (most recent at index H-1). history_eef_xy: (B, H, 2) — past H EEF pixel coords in image-pixel space (the last entry is the current frame's EEF; teacher-forced). Returns: logits: (B, grid_size*grid_size) — categorical over 56x56 spatial grid for the predicted next-step EEF cell. z Expected H=z, got z Expected rhrNr)r€)Úshaper8r7Úviewrkr;r)rJrIrKrmrnrrror|rurr rRrT)rYZ history_imgsZhistory_eef_xyrprbÚCZHiZWirhrjZ eef_protoZ eef_tokensZ frame_blocksrtrÚseqZhistory_eef_01Úper€ÚoutZ query_idxZ query_featÚlogitsrrrÚforwardõs2 ÿ    2  zARTransformerPolicy.forward)Ú__name__Ú __module__Ú __qualname__Ú IMAGE_SIZEÚ HISTORY_LENÚ GRID_SIZEÚ TRANSFORMER_DÚ TRANSFORMER_HÚ TRANSFORMER_Lr6rkrurr‰Ú __classcell__rrr`rr$7søM?r$Ú__main__ÚcudaÚcpuT)r8rEccs|] }|jr| ¡VqdS©N)rBÚnumel©Ú.0rZrrrÚ +s€r›ccs|]}| ¡VqdSr—)r˜r™rrrr›,s€zTrainable params: ú,z / )r rér rˆ),Ú__doc__ÚosrrÚtorch.nnrFZtorch.nn.functionalÚ functionalÚFÚenvironÚgetrrr:rŽrrr‘r’rrr#ÚModuler$rŠr r•Ú is_availablerlÚmodelÚsumrAÚn_trainÚn_totalr<rprbrHÚimgsÚrandÚeefrdrˆr‚rrrrÚsB r  ÿô