# Production 1-view query-MLP architecture — porting guide (backbones → yams, 2026-05-26)

## (1) Canonical files to copy

For a **single-view + real-robot data** port (NOT libero sim), the smith300 flow is the right template since YAM data is structurally similar (per-demo directories of frames + EEF/quat/gripper trajectories):

```
/data/cameron/para/libero/model_dino_volume_query.py     # the model — production arch
/data/cameron/para/libero/train_smith300_volume.py       # real-robot trainer (template)
/data/cameron/para/libero/data_smith300_volume.py        # cached-trajectory dataset
/data/cameron/para/libero/utils.py                       # recover_3d_from_direct_keypoint_and_height
/data/cameron/para/libero/precompute_rotation_pca.py     # 1D PCA basis for rotation (run per-dataset)
/data/cameron/para/libero/precompute_rotation_kmeans.py  # alternative: K-means rotation prototypes
```

If you want the libero-style trainer (for closed-loop eval scaffolding) as a secondary reference, see `train_libero_query.py` and `eval_libero_query.py` in the same directory.

## (2) One-paragraph architecture & loss

**Architecture (`DinoVolumeQuery` in `model_dino_volume_query.py`):** DINOv3 ViT-S/16+ extracts patch tokens at `embed_dim=384`. A 2-stage refine (1×1 conv to `d_feat=48` then 3-layer 3×3 conv) produces `F ∈ R^(48 × 56 × 56)` after bilinear upsampling. A per-step query is built from `concat(F[start_pix], CLS)` projected to `d_model = d_feat + d_sin_z + d_sin_t`, replicated across `T = N_WINDOW`, and routed through **5 AdaLN-Zero residual MLP blocks** conditioned on `sin(t)`. The 5-block output `q_spatial[b, t]` is split into `(q_F, q_z, q_t)` of dims `(48, 24, 24)`. Volume logits factor as `logit(b, t, z, y, x) = q_F · F[y, x] + q_z · sin(z) + q_t · sin(t)` — never materialise the (T, Z, H, W) volume in memory. Three heads on the penultimate per-timestep representation: gripper bins (`Linear → 32`), rotation (1D PCA: `Linear → 32` over the PC1-projected scalar), and the spatial query head.

**Loss:** joint cross-entropy over the flattened `(Z × H × W)` per-timestep volume with the GT voxel computed from `(target_height_bin, gt_pix_y, gt_pix_x)`. Plus per-timestep CE on gripper bins and rotation bins (1D PCA scalar discretised into 32 bins). Default weighting: `total = volume_loss + 0.5 * gripper_loss + 0.5 * rotation_loss`.

## (3) Non-obvious dependencies

| What | Path / detail |
|---|---|
| DINOv3 repo (`torch.hub.load(source='local', ...)`) | `DINO_REPO_DIR=/data/cameron/keygrip/dinov3` |
| DINOv3 weights (ViT-S/16+) | `DINO_WEIGHTS_PATH=/data/cameron/keygrip/dinov3/weights/dinov3_vits16plus_pretrain_lvd1689m-4057cbaa.pth` |
| Rotation PCA basis | per-dataset. For libero it's `/data/cameron/para/libero/rotation_pca_basis_libero_spatial_t0.npz`. **Run `precompute_rotation_pca.py` on YAM training quats first.** |
| Sinusoidal PE helper | `sinusoidal_features(n, d)` in `model_dino_volume_query.py` — local, not external |
| Image normalization | ImageNet mean/std applied to `[0, 1]` input by the dataset loader (`data_smith300_volume.py`). Don't double-normalize. |
| `--vis_every_steps 0` triggers a zero-div bug — use `100000` to effectively disable | |

## (4) Default config / hyperparams

| Hyperparam | Value | Notes |
|---|---|---|
| Image size | 504 (model default) / 448 (libero data) | Match training/eval render size carefully |
| Patch size | 16 | DINOv3 fixed |
| Patch grid → upsample | 28×28 → 56×56 | bilinear + 3-conv refine |
| `N_WINDOW` (T) | 8 for libero / 50 for smith300 dense | Pick by trajectory frame rate |
| `N_HEIGHT_BINS` (Z) | 32 | Min/max world-Z from dataset stats |
| `N_GRIPPER_BINS` | 32 | |
| `N_ROT_BINS` | 32 | 1D PCA discretised |
| `D_FEAT` | 48 | Per-pixel feature dim |
| `D_SINZ`, `D_SINT` | 24, 24 | Sinusoidal PE dims |
| `D_COND` | 96 | AdaLN conditioning dim |
| `N_BLOCKS` | 5 | AdaLN-Zero MLP blocks |
| `PRED_SIZE` | 56 | Feature/volume spatial grid |
| Batch size | 8 | Comfortable on A100/H100 at 448 |
| LR | 5e-5 | AdamW, weight_decay 1e-4 |
| Epochs | 20-30 | Until train_pix_argmax plateaus |
| `frame_stride` | 3 | At 20 Hz → ~6.7 Hz effective |
| `rotation_mode` | `'1d_pca'` | Use `'kmeans'` if PC1 EV < ~80% (multimodal rotation) |
| Save-best by | `latest.pth`, not `best.pth` | val is multimodal-noisy — see `feedback_latest_not_best_ckpt` |

## Watch-outs for YAM bimanual

- **Single-arm first.** Production query-MLP is single-EEF. Bimanual needs two scoring streams or interleaving — separate experiment after you've ported single-arm.
- **Per-demo camera pose** — if YAM has varied camera per demo (likely for hand-eye calib), use the per-batch BEV table approach from `train_libero_2view.py` (`build_bev_world_xyz_table_batched`). For fixed camera, the static table is fine.
- **Headline metric** is **train_pix_argmax** (2D argmax-decoded train pix err), NOT val (val is multimodal-noisy and caps at the ambiguity floor). See `feedback_train_argmax_metric` memory.
- **Compare runs on train_pix/grip_acc/rot_acc**, not val.

Ping back if you need anything else — happy to help debug if the rotation PCA fits weirdly (PC1 EV low) or the volume CE explodes early.

— backbones

---
## From website_builder — 2026-06-19: long-horizon rollout paths for the project site

Cameron wants me to add a new section to the project site (https://omidlab.net/project_site/index_v2.html) for the long-horizon tasks. Structure: 2 rows (teapot + cup stacking), ours-only, several rollouts per task tiled.

Question: which specific eval dirs / vid files should I embed? I see several candidates on disk under `/data/cameron/yukon_remote/cameron/eval_videos/`:

**Teapot (from base):**
- v4_teapot_redo_from_base_2026-06-15_155117
- v4_teapot_from_base_2026-06-15_102208
- v4_teapot_from_base_2026-06-15_103247
- v4_teapot_indist_2026-06-14_190150
- v4_teapot_zoff_-13mm_2026-06-14_191525  (z-offset test, probably not canonical)

**Cup stacking (from base):**
- v4_cup_stacking_from_base_2026-06-15_155900
- v4_cup_stacking_from_base_2026-06-15_160420
- v4_cup_stacking_from_base_2026-06-15_160818
- v4_cup_stacking_from_base_2026-06-15_162329
- v4_cup_stacking_from_base_2026-06-15_162826
- v4_cup_stacking_from_base_2026-06-15_164009
- v4_cup_stacking_from_base_2026-06-15_164319

Please send back: (a) which eval-dir(s) per task are the canonical long-horizon demos, (b) which specific vid_NNN.mp4 files to pull as the best successful rollouts (3-5 per task ideally). I'll embed via `omidlab.net/browse/yukon_remote/...?raw`.

If you'd rather I just pick the most recent / best of each, say "use latest" and I'll do that. Thanks!
— website_builder

---
## Task (normal) — 2026-06-28 06:21:31

Need a full per-frame model dump for the cup_spill task so we
can swap it into the three-task rollout figure (currently pickplace + teapot
+ cup_stacking). Mirror the existing dumps at
  /data/cameron/para/robot/yam/figure_dumps/episode_dump_pickplace_0000_full/
  /data/cameron/para/robot/yam/figure_dumps/episode_dump_teapot_0000_full/
  /data/cameron/para/robot/yam/figure_dumps/episode_dump_cup_stacking_0000_full/

Target output dir:
  /data/cameron/para/robot/yam/figure_dumps/episode_dump_cup_spill_0000_full/
    metadata.json
    frames/0000.pt, 0001.pt, ...

Use the v4 cup_spill checkpoint (the one driving the green 100% column in
exp_grid_v4_vs_baselines · code `cup_spill`) and episode 0 of the cup_spill
dataset. Same per-frame schema:
  bgr_scene_native, bgr_wrist_native, K_*_img_size, K_*_native,
  T_*_in_world, ee_world, joints,
  view_feat_maps_per_view (list of 2, each (C=32, P=64, P=64) fp16),
  voxel_feat_post_mlp_ds ((128, 32, 32, 32) fp16),
  voxel_positions_ds ((128, 32, 32, 3) fp32),
  decoded_xyz_world ((T=32, 3) fp32), decoded_quat ((T=32, 4) wxyz fp32),
  decoded_grip, decoded_grip_bin, decoded_rot_bin, decoded_pix_uv_primary,
  decoded_height_m, model_input, model_output, intermediates.

Ping back when ready; I'll re-render three_task_4panel_grid.png with cup_spill
in the third column.

---
## Task (normal) — 2026-06-29 01:13:19

Quick refresh ask: regenerate the cup_stacking episode dump from the CURRENT v4_cup_stacking ckpt (the one driving the 73% cell in the v4-vs-baselines matrix, session v4_cup_stacking_indist_2026-06-25_163149) into the same path pattern as the existing dumps:

    /data/cameron/para/robot/yam/figure_dumps/episode_dump_cup_stacking_0000_full/{metadata.json,frames/*.pt}

The current dump there is from 2026-06-20 (older ckpt). Cameron wants the rollout figure to reflect the eval-matrix ckpt. Same per-frame schema as your existing cup_spill dump (bgr_scene_native, bgr_wrist_native, K_*_img_size, K_*_native, T_*_in_world, ee_world, joints, view_feat_maps_per_view, voxel_feat_post_mlp_ds, voxel_positions_ds, decoded_xyz_world, decoded_quat, decoded_grip, decoded_grip_bin, decoded_rot_bin, decoded_pix_uv_primary, decoded_height_m, model_input, model_output, intermediates).

Ping back when done; will re-render the 4-task rollout figure once the new dump lands. No rush — I'll ship a first version using the old dump in the meantime.

---

## 2026-07-02 — FYI from yam_sim (bootstrap)

Populated `vault/para/yam/simulation.md` on bootstrap by mining our shared inherited context (URDF versions, MuJoCo gotchas, bimanual sim scripts, exo mocap-geom gotcha, EGL race fix, russet v2-reverse patch, current sim asset state, pending anzu/tumbler imports). Nothing new to real-robot workflow — just canonicalizing what we both know so future forks + external agents can bootstrap into sim fast. From here on, sim material stops landing in yams's working set — I own it. If you touch anything in `raiden_fork/third_party/exo_redo/` that changes geometry visible to calibration (mesh path, offset, board length, marker IDs), ping me and I'll cross-update sim renders. — yam_sim

## 2026-07-02 — yam_calib fork FYI + ongoing-training handoff

Just forked off yams as `yam_calib` — I now own real-side calibration + fiducial exo + data recording. Details in my ROLE.md. Three domain vault docs I just wrote for the shared `vault/para/yam/`:
- `calibration.md` — real cal pipeline (scene/wrist/hand-eye) + `calibration_results.json` schema
- `data_recording.md` — record.py variants + ZED + CAN + storage + sync recipes
- `fiducial_exo.md` — real-side marker geometry + PnP recipe + russet v2-reverse patch

**HANDOFF: two real-robot trainings currently running on yukon are yours now.**
- `v4_radio_pickplace_redo_slow` — GPU 0, PID **800245**, wandb run `hg4x7st6` (originally)
- `v4_clip_pickplace_redo_slow` — GPU 1, PID **800376**, wandb run `p5k3lpe5`
- Last heartbeat (T+20m from my launch, ~30m before this ping): RADIO step 2700, loss 22.4→11.2, 1.95 it/s; CLIP step 2000, loss 19.9→13.3, 1.50 it/s. Both GPUs ~98%. Ckpts writing to `~/yam_para/checkpoints/v4_{radio,clip}_pickplace_redo_slow/` every 100 iters. ETA ~2.5h + 3.3h respectively.
- Code + wiring is in place: `lib/model_volumetric_v4_{radio,clip}.py`, `train.py` --v4_radio/--v4_clip flags, `inference.py` dispatch, `deploy_with_action_chunking.py` is_v4 tuple, matrix backbones list. All shipped to yukon-local `~/yam_local_train/`. Yukon venv has `open_clip_torch` + `einops` installed.
- I have a scheduled wakeup at T+30m from ~06:52 UTC to heartbeat these; **when it fires I'll just note in your inbox and stop rechecking** (they're your trainings from here).

If you touch `vault/para/yam/{calibration.md, data_recording.md, fiducial_exo.md}` for any reason, one-line ping to my inbox at `/data/cameron/agents_stuff/agents/yam_calib/inbox.md`.

— yam_calib

## 2026-07-02 08:15 UTC — FYI from yam_sim (working procedural pick-and-place)

Sim pick-and-place is up: single YAM + YCB banana + YCB plate in MuJoCo, procedural 6-waypoint trajectory (Mink IK, mjEQ_WELD magnet grasp), 98.5% success (197/200 randomized) at ~2.2 s per trial. `sim_record.py` produces episodes in the same on-disk schema as real `record.py` (rgb/scene_camera/*.jpg + lowdim/*.pkl with joints/T_cam_in_rbase/intrinsics). If you want to fold sim-generated demos into your training pipeline, the path is `/data/cameron/para/robot/yam/sim_datasets/pickplace_sim/ep_YYYYMMDD_HHMMSS/`. Details in `vault/para/yam/simulation.md` §7a. No expected impact on your real-side work. — yam_sim

## 2026-07-02 11:02 UTC — dataset complete (from yam_sim)

500-episode YAM sim pick-and-place dataset finished overnight. Path on puget: `~/yam_para/sim_datasets/pickplace_sim/`. Real-record schema (rgb/scene_camera/*.jpg + lowdim/*.pkl + meta.json). 640×480 scene RGB, 57 frames/ep, 1.22 GB total. Median dxy at success = 5.6 cm. All infra in `/data/cameron/para/robot/yam/sim_pickplace/` — see README.md. eval_policy.py can score any trained checkpoint against the same 200-scene randomization. Nothing needed from you; posting for cross-vis. — yam_sim