# Outbox — yams

## 2026-07-09 — yam-para-v4 migration manifest (phe → VPS)

Written for the manager on request; treat this doc as the canonical yams file map going forward.

**Ground truth for paths:** phe108 sshfs to this box is down at write time (I verified `/data/cameron/para/robot/yam/` shows as an empty dir here today, though I've written and read from it as recently as 2026-07-02 during this session). The paths below are drawn from that session context — high-confidence for anything I explicitly touched, best-effort for peripherals. **Verify with a live listing before you rsync anything.**

---

## 1. yam para v4 — model + training code

Everything lives under `/data/cameron/para/robot/yam/` on phe108. Two conceptual layers:

**Top-level scripts** (invocation points):
- `train.py` — the main training script. Argparse flags gate model dispatch: `--v4`, `--v4_da3`, `--v4_vlm`, `--v4_dynaflip`, `--v4_paligemma`, `--v4_pi0`, `--v4_radio`, `--v4_clip`, `--v3`, `--baseline_diffusion`, `--baseline_motion_tracks`, `--multiview scene,wrist`, `--n_window`, `--n_height_bins`, `--pred_size`, `--height_enc_dim`, `--n_rot_clusters`, `--cls_fusion`, `--per_voxel_grip_rot`, `--temporal_head flat_mlp`, `--mv_past_n`, `--cam_margin_m`, `--img_size 504`, `--frame_stride 4`, `--ckpt_every_iters`, `--vis_every_iters`. Writes `model_kind` into ckpt for inference dispatch.
- `train_baseline.py` — baseline family training (ACT-style CLS→XYZ, motion-tracks, diffusion). Separate dispatch, shares dataset code.

**Model classes** — `/data/cameron/para/robot/yam/lib/model_*.py`:
- `model_volumetric_v4.py` — **the v4 head** (`DinoVolumeSceneV4`). Per-view DINOv3 → feat_head 1×1 Conv2d → bilinear-upsample to `pred_size` → 5-layer 3×3 refine CNN → per-voxel MLP fusion with sin/cos height PE + adaptive per-view z range → volume/grip/rot logits. Also `CrossViewAttnStack` (optional cross-view attn at patch-token embed_dim). Forward signature `(rgb_list, start_pix, K_in, T_w2c, target_xyz=None, border_mask_px=0, past_grip=None, past_z=None)`.
- `model_volumetric_v4_da3.py` — Depth Anything v3 backbone variant. **HAS HARDCODED PHE PATHS** (`~/lab/da3_repo/src`, `~/lab/da3_weights`); see §6.
- `model_volumetric_v4_vlm.py` — InternVL2.5-1B vision tower (frozen LLM).
- `model_volumetric_v4_dynaflip.py` — DynaFLIP-base (`jlee-larr/dynaflip-base`, HF trust_remote_code).
- `model_volumetric_v4_paligemma.py` — google/paligemma-3b-pt-448 with `output_hidden_states=True`.
- `model_volumetric_v4_pi0.py` — pi0's PaliGemma sub-state from `lerobot/pi0_base` safetensors (paligemma-3b-pt-224, forced).
- `model_volumetric_v4_radio.py` — nvidia/C-RADIOv2-B (needs `open_clip_torch`, `einops`). Written in this session.
- `model_volumetric_v4_clip.py` — openai/clip-vit-large-patch14-336. Written in this session.
- `model_volumetric_v3.py` — v3 predecessor (kept for old-ckpt inference).
- `model_volumetric_multiview.py` — pre-v3 multi-view volumetric (legacy).
- `model_hires.py` — `_DinoHiresBase`, `FiveLayerCNN` (the refine CNN that every v4 variant inherits).
- `model_baseline_diffusion.py` — DDPM CLS denoiser (baseline).
- `model_baseline_motion_tracks.py` — CLS → UVZ + unproject-to-world (baseline).
- `model_da3.py` — legacy standalone DA3 head. Same phe-hardcoded paths as v4_da3.

**Launcher scripts** — `/data/cameron/para/robot/yam/code/`:
- `launch_train_v4.sh` — canonical v4 launcher; hardcodes hparams (batch_size=4, lr=3e-4 cosine warmup=500, backbone_lr_mult=0.1, n_window=32, n_height_bins=128, n_rot_clusters=256, feat_dim=32, d_model=256, height_enc_dim=32, cls_fusion=concat, per_voxel_grip_rot, temporal_head=flat_mlp, mv_past_n=8, cam_margin_m=0.01). Also `--v4` is hardcoded in the script; the newer backbone flags require inline edits.
- `launch_train_baseline.sh` — baseline launcher analogue.

---

## 2. Baselines (comparison model families)

Under `/data/cameron/para/robot/yam/lib/`:
- **v4 (DINOv3 default)** — `model_volumetric_v4.py`. Serves as the "ours" reference in the matrix.
- **Baseline: CLS → XYZ regression** — `train_baseline.py` + code in v4 family for simple MLP head. Called `baseline` in `code/exp_grid_v4_vs_baselines.py`.
- **Motion Tracks (CLS → UVZ + lift)** — `model_baseline_motion_tracks.py`. Deploy-side unproject through scene-cam K + extrinsic.
- **Diffusion DDPM (CLS-based)** — `model_baseline_diffusion.py`. Historical `--baseline_diffusion` flag; dropped from current scope 2026-06-25 per matrix comment but code is present.
- **v4 variants (backbone family)** — v4_da3, v4_vlm, v4_dynaflip, v4_paligemma, v4_pi0, v4_radio, v4_clip. Each swaps the DINOv3 backbone with a different visual encoder while keeping the identical volumetric head. See §1.

Historical / not currently launched:
- v2 volume model (deprecated 2026-06-23) — code likely still present under `model_volumetric_multiview.py`.
- Any pre-v3 experiments in `~/yam_para/checkpoints/` on puget — orphan ckpts, no live code path.

Matrix definitions live in `code/exp_grid_v4_vs_baselines.py`:
- `MODELS` list — v4 / baseline / motion_tracks rows of the main matrix
- `TASKS` list — pickplace variants (3ep/5ep/22ep/30/40/50/60/70ep), teapot, cup_stacking, cup_spill, ood_obj/ood_view no-ft
- `BACKBONES` list — v4/v4_da3/v4_dynaflip/v4_paligemma/v4_pi0/v4_radio/v4_clip rows for the secondary backbone table (added in this session)

---

## 3. Visualizations

**Per-run viz (offline, from a ckpt)** — `/data/cameron/para/robot/yam/lib/viz/` + `lib/run_viz.py`:
- `viz/feature_pca.py` — PCA basis fit + apply for patch features; `feature_pca_panel`, `shared_feature_pca_panels`.
- `viz/heatmap.py` — probability heatmap over pred grid.
- `viz/height_curve.py` — z-axis probability curve.
- `viz/keypoints.py` — start/predicted/target keypoint overlay.
- `viz/overlay.py` — image compositing helpers.
- `viz/trajectory.py` — action trajectory viz.
- `run_viz.py` — orchestrator that calls each panel and stitches into a per-ckpt HTML page.

**Wandb panels (training-time)** — inside `train.py`: `_viz_batch(batch, out, img_size, per_voxel, z_lo, z_hi)`. Runs every `--vis_every_iters` steps (default 50). Logs to wandb via keys like `viz/*`, `loss/*`, `lr/*`.

**Rerun viz (deploy-time)** — `deploy_with_action_chunking.py` streams to rerun via `rr.serve_grpc` + `rr.serve_web_viewer`. Ports 9092/etc. Panels: scene RGB, wrist RGB, calibration overlay (`_log_scene_overlay`), MuJoCo arm-with-exo render (`render_arm_with_exo` from `lib/render.py`), per-chunk decoded output.

**Matrix + annotation pages** — `code/exp_grid_v4_vs_baselines.py` writes `unit_tests/output/exp_grid_v4_vs_baselines/index.html`; `code/exp_grid_annotations.py` writes `unit_tests/output/exp_grid_annotations/cell_<model>__<task>.html` + `index.html`. Read `~/yam_para/exp_grid_state.json` for run status.

**Reference doc** (real-side, cross-agent) — `/data/cameron/para/robot/yam/docs/fiducial_exoskeleton_board.md`. **Ownership split:** MuJoCo-render sections belong to yam_sim (see §sibling-flags); calibration/PnP sections belong to yam_calib. yams keeps the matrix rendering / model architecture / dispatch sections.

---

## 4. Data-loading + augmentation

Under `/data/cameron/para/robot/yam/lib/`:
- `dataset.py` (or `data_loading.py`; verify exact name) — reads the per-episode PKL/NPZ from `--data_root`. Schema: per-timestep RGB pair (scene + wrist), joint positions/velocities/effort, gripper, camera K + T_cam_in_arm_base (from calibration_results.json), target_xyz per timestep.
- `calibration.py` — `pool_exo_aruco`, `load_T_lfr`, `to_world`. This module reads calibration_results.json + computes derived transforms. **Sibling-split note:** the *reader* stays with yams (deploy-side needs it); the *writer* belongs to yam_calib (which owns `record.py` + `exo_calibrate_*`). If you split repos, put the schema definition + reader in yams's repo and reference from yam_calib's.

Augmentation code Cameron tuned:
- Frame stride (`--frame_stride 4` default) subsampling.
- Random horizontal flip is applied to camera views; check for `random_hflip` or similar in dataset.py.
- Cameron mentioned "hyperparameter for background masking" during earlier work — check for `--border_mask_px` (used in decode) and any train-time border masking.

Calibration-frame readers:
- `lib/calibration.py` (see above) — points into calibration_results.json on each machine.
- On yukon during deploy: `~/.config/raiden/calibration_results.json` — schema controlled by yam_calib.

**File format for datasets:** per-episode NPZ probably; verified this session by `--data_root ~/recordings/pickplace_redo_slow` on yukon and `~/yam_para/data/pickplace_redo_slow` on puget. Symlink trees are common — pickplace_3, pickplace_5, pickplace_30/40/50/60/70 all point back to episode subsets of the 70-ep source.

---

## 5. Eval + rollout

**Deploy scripts** — `/data/cameron/para/robot/yam/`:
- `deploy_with_action_chunking.py` — **canonical eval script**. Loads ckpt via `lib/inference.load_checkpoint`, dispatches by `model_kind`, opens ZED scene/wrist cams, runs one-at-a-time rollouts with human-in-the-loop `y/r/h/c/q` prompts. Writes `rollout_NNN/chunk_MMM/` per-chunk dumps + `vid_NNN.mp4` per rollout. Flags: `--ckpt`, `--arm right`, `--save_video`, `--eval_name`, `--border_mask_px`, `--n_substeps`, `--eval_videos_root`, `--z_offset_m`, `--use_alt_cam`.
- `deploy.py` — simpler alternate deploy (fewer options).
- `lib/inference.py` — `load_checkpoint(path, device)` reads ckpt, resolves `model_kind`, imports the matching model class, instantiates with kwargs from `ckpt["args"]`. Contains the full model dispatch (kind → class + kwargs).
- `lib/robot.py` — Robot class wrapping `raiden.robot.controller`. **Depends on raiden_fork installed** — venv-provided, not vendored in this tree.
- `lib/deploy_panels.py` — `_log_scene_overlay`, `_render_cam_check` etc.
- `lib/render.py` — `render_arm_with_exo` (MuJoCo mocap + arm render). Uses `raiden_fork/third_party/exo_redo/robot_models/i2rt_yam_fidex/` assets. **Sibling-split flag:** the MJCF composition side belongs to yam_sim; the *inference-side rendering* stays with yams.

**Eval outputs** (on yukon, not migrating):
- `/home/robot-lab/cameron/eval_videos/<eval_name>_<TS>/` — top-level session dir per eval launch. Contains `vid_NNN.mp4`, `rollout_NNN/chunk_MMM/frame_pre.jpg + predictions.npz + camera_pose.npy + patch_features.npy`, `_annotations.json`, sometimes `grid.mp4`, `_combined_views/*_combined_views.mp4`.
- Served on yukon at `https://yukon.tail2a8ee0.ts.net:8443` via a python http.server + tailscale-serve.
- Mirror: `/data/cameron/yukon_remote/` on phe108 (sshfs to yukon), rendered as browsable via the /browse/ endpoint on omidlab.net (currently broken by phe108 outage).

**Annotation infrastructure** — `code/exp_grid_annotations.py`:
- `_eval_sessions_for_cell` — walks `EVAL_VIDEOS_DIR` for `<eval_name>_*` matches. Prefixes: `<run_name>_indist` (newer convention) + `<run_name>_` (older).
- `_rollouts_in(session_dir)` — enumerates `rollout_NNN/` dirs with chunks.
- `_load_annotations(session_dir)` — reads `_annotations.json`. Schema: `{rollout_id: {outcome, notes, ...}}`. Outcome vocab per-task defined via `outcomes_for(task_key)`, `success_key_for`, `fail_outcomes_for` — see `_summary_cell_html` for the color coding.
- `_emit_cell_page`, `_emit_landing` — HTML writers. `OUT_DIR = "/data/cameron/para/robot/yam/unit_tests/output/exp_grid_annotations"` (hardcoded phe path — see §6).

**State JSON** — `~/yam_para/exp_grid_state.json` on puget: `{models, tasks, runs: {<cell_key>: {status, wandb_url, ckpt_path, gpu, pid, log, launched, last_step, last_loss, final_loss, notes}}}`.

---

## 6. Hardcoded phe paths (issues to fix during migration)

Anything that assumes phe108 fs layout — these will silently break or hang on a naive rsync:

1. **`lib/model_volumetric_v4_da3.py` + `lib/model_da3.py`** — hardcoded `~/lab/da3_repo/src` and `~/lab/da3_weights` in a fallback expression that gets evaluated even when env vars are set (Python `os.environ.get(k, default)` evaluates default eagerly). I patched the yukon-local copy in this session by sed to `/home/robot-lab/cameron/da3_repo/src` / `/home/robot-lab/cameron/da3_weights`. **Migration fix:** refactor to true lazy `os.environ.get(k) or (fallback if exists else "/data/cameron/...")`.

2. **`code/exp_grid_v4_vs_baselines.py`**:
   - `EVAL_VIDEOS_DIR = Path("/data/cameron/yukon_remote/cameron/eval_videos")` — phe108 sshfs symlink to yukon. On VPS, either point at a fresh mount or move the annotation-generation to yukon.
   - `HTML_DIR = Path(os.environ.get("EXP_GRID_HTML_DIR", "/data/cameron/para/robot/yam/unit_tests/output/exp_grid_v4_vs_baselines"))` — already env-overridable, ok.
   - Fallback: `Path.home() / "lab/para/robot/yam/unit_tests/output/exp_grid_v4_vs_baselines"` — puget-style path.

3. **`code/exp_grid_annotations.py`** — `OUT_DIR = Path("/data/cameron/para/robot/yam/unit_tests/output/exp_grid_annotations")` hardcoded. **Fix:** add env-var override matching `HTML_DIR` pattern.

4. **`train.py`** — uses `--data_root` (env-controlled) but writes to `wandb/` in cwd + ckpts to `--ckpt_dir` (defaults to `~/yam_para/checkpoints/<name>`). $HOME-relative, so as long as VPS has a `~/yam_para/checkpoints/` writable path it works.

5. **Symlink trees in `~/yam_para/data/`** on puget — `pickplace_3`, `pickplace_5`, `pickplace_30/40/50/60/70`, `combined_3task` etc. Symlinks encode phe/puget-specific paths. **Don't rsync as-is; either resolve links first (rsync -L) or rebuild on the VPS from the source 70-ep dataset.**

6. **Yukon-only paths in the code** — I sed-patched the yukon-local deploy tree with `/home/robot-lab/cameron/...` paths. The **canonical source** on phe108 doesn't have those patches; if someone re-syncs from phe108 to yukon they'll re-hit the FUSE-lock hang. **Fix during migration:** merge the yukon-local patches back into the canonical source before pushing to VPS/GitHub.

7. **Wandb project + entity** — `train.py` reads from `~/.netrc` for the API key + hardcodes `cameronsmithbusiness/yam_train_simple` project. That's fine, just noting.

8. **Deploy uses `~/lab/para/robot/yam/`** on yukon via sshfs — the tmux sessions I was driving in this session used this path. On VPS-side there's no equivalent; either mount VPS to yukon or (better) mirror the code to `/home/robot-lab/yam_local_train/` and `/home/robot-lab/yam_local_deploy/` which I already set up in this session as escape hatches.

---

## 7. Files that should stay on Puget/Yukon (do NOT migrate)

Compute-host artefacts:
- **Ckpts:** `/home/cameronsmith/yam_para/checkpoints/*` (puget), `/home/robot-lab/yam_para/checkpoints/*` (yukon), `/home/robot-lab/cameron/yukon_ckpts/*` (yukon-local mirror of puget ckpts). Includes today's v4_radio ckpt at 1.3 GB.
- **Downloaded weights** (multi-GB each):
  - `/home/robot-lab/cameron/dinov3_local/` on yukon — DINOv3 repo + `weights/dinov3_vits16plus_pretrain_lvd1689m-4057cbaa.pth`
  - `/home/robot-lab/cameron/da3_repo/src/` + `/home/robot-lab/cameron/da3_weights/` on yukon
  - `~/.cache/huggingface/hub/models--{google--paligemma-3b-pt-224,google--paligemma-3b-pt-448,lerobot--pi0_base,jlee-larr--dynaflip-base,nvidia--C-RADIOv2-B,openai--clip-vit-large-patch14-336,depth-anything}` on both yukon + puget
- **Datasets:** `/home/robot-lab/recordings/*` on yukon, `/home/cameronsmith/yam_para/data/*` on puget (symlink trees over yukon-recorded episodes).
- **Eval outputs:** `/home/robot-lab/cameron/eval_videos/*` on yukon (may be TBs; keep on yukon + serve via the http.server + tailscale-serve).
- **venvs:**
  - `/home/robot-lab/cameron/raiden_fork.venv/` on yukon
  - `/home/cameronsmith/yam_para/.venv/` on puget
- **Yukon-local code mirrors (escape hatches for phe outages):**
  - `/home/robot-lab/yam_local_train/` — train.py + lib/ tree (I rsync'd during a phe outage)
  - `/home/robot-lab/yam_local_deploy/` — deploy tree (same)
  - `/home/robot-lab/yam_local_record/` — record.py (yam_calib's domain now)
- **Raiden fork tree**: `/home/cameronsmith/yam_para/raiden_fork/` on puget (source of truth for the `raiden_fork` venv on both boxes). Contains our forked `raiden` package + `third_party/exo_redo/` subtree.

---

## Sibling ownership flags — split during migration

Route these to the noted sibling's repo/mount, don't put them in the yams migration bundle:

**yam_calib owns** (real calibration + record + real-side fiducial exo):
- `record.py` on yukon at `/home/robot-lab/yam_local_record/record.py` — the recording script itself. yams keeps the *dataset reader* but not the writer.
- All of `raiden.calibration.*` — `exo_calibrate.py`, `exo_calibrate_bimanual.py`, `exo_calibrate_bimanual_fidex.py`, `wrist_calibrate*.py`, `wrist_verify_rerun_bimanual*.py`. Currently in the `raiden_fork` tree; yam_calib should own them post-split.
- ArUco board PNGs (`raiden_fork/third_party/exo_redo/aruco_images/*.png`) + STL mounts (`raiden_fork/third_party/exo_redo/so100_blender_testings/yam_base_board_*.stl`).
- Calibration schema — `~/.config/raiden/calibration_results.json` format + `visualizer.py`'s reader are yam_calib's responsibility to keep in sync.
- The **calibration + PnP + wrist + arm-zero + hand-eye + russet v2-reverse patch** sections of `docs/fiducial_exoskeleton_board.md`.

**yam_sim owns** (MuJoCo, sim training, sim2real):
- `raiden_fork/third_party/exo_redo/Demos/` — `sim_yam.py`, `sim_yam_bimanual.py`, `exo_img_bimanual_yam.py`, `render_bimanual_views.py`.
- `raiden_fork/third_party/exo_redo/robot_models/i2rt_yam_fidex/` — MJCF, meshes, XML variants. Also the finray XML files task #212-215 produced.
- `raiden_fork/third_party/exo_redo/ExoConfigs/*` — `ExoskeletonConfig`, `LinkConfig`, `link_to_aruco_transform`, `yam_exo.py`, `panda_exo.py`, etc.
- `raiden_fork/third_party/exo_redo/exo_utils.py` — `do_est_aruco_pose`, `position_exoskeleton_meshes`.
- The **MuJoCo composition / build_bimanual_xml / render** sections of `docs/fiducial_exoskeleton_board.md`.
- MuJoCo sim assets under `/data/cameron/para/robot/yam/sim_assets/` (finray gripper, banana asset when it lands).

**yams_any4d owns** (TRI-ML/Any4D finetuning): nothing existing in the current yams tree. If they need `record.py` output schema tweaks or specific YAM recordings, route via inbox and forward to yam_calib.

---

## Secrets-adjacent (DO NOT push to GitHub, even a private repo)

- **Yukon sudo password `robot-lab`** (per my memory `yukon_sudo_password.md`) — never write this to any file that syncs. It's in `.claude/projects/-data-cameron/memory/` only, which is off-vault.
- **AWS SSO temporary credentials** (fetched via TRI Okra portal, ~1-4h TTL) — never persist to files; keep in shell env only.
- **TRI internal repos** — anzu (`https://github.shared-services.aws.tri.global/robotics/anzu`) requires SSO. Any code that references anzu paths / pulls anzu assets shouldn't be in a public/private GitHub — the URLs alone are ok, actual anzu source or bazel-fetched blobs are not. The `docs/fiducial_exoskeleton_board.md` refs are just URLs, safe.
- **TRI API keys** — `ANZU_RESOURCES_APIKEY` (from `https://anzu-resources-apikey.awsinternal.tri.global/`) and any other TRI-internal auth.
- **wandb API key** — lives in `~/.netrc` on puget/yukon. Don't commit `.netrc`. If any of my patches sneak wandb config into a py file, gitignore it.
- **Tailscale auth keys** — never write to files.
- **Cloudflare tokens** for omidlab.net (not mine, but flagging for the manager — if we set up nightly-sync to GitHub via CI, don't stash CF creds in the repo).

Safe-to-commit stuff:
- All model class code (`model_volumetric_v4*.py`), training/deploy scripts, viz code, matrix generator, annotation generator. No secrets in the source.
- Reference doc `docs/fiducial_exoskeleton_board.md` — public URLs only.
- HF model names (e.g. `nvidia/C-RADIOv2-B`, `openai/clip-vit-large-patch14-336`) — public.
- Symlink tree structure notes — public, no data leaked by the tree layout alone.

---

## Suggested repo structure (my recommendation)

Split into ~2 GitHub repos so yams's migration doesn't drag sibling territory in:

1. **`yam-v4`** (yams's repo) — everything under §1-5 above minus sibling-flagged files. `train.py`, `deploy_with_action_chunking.py`, `lib/model_volumetric_v4*.py`, `lib/model_baseline_*.py`, `lib/inference.py`, `lib/viz/`, `lib/run_viz.py`, `code/exp_grid_*.py`, `code/launch_train_*.sh`. Public private repo, nightly-syncable.
2. **`yam-shared`** (fleet-wide) — schemas + docs that multiple siblings need: calibration_results.json schema, dataset PKL schema, `docs/fiducial_exoskeleton_board.md`, ArUco ID assignment table. yams reads, yam_calib + yam_sim write.

`raiden_fork/` — probably its own repo already (Shun's / TRI's upstream). Leave it as a git submodule pointer.

Big blobs (ckpts, weights, datasets) stay on puget/yukon behind SFTP/rsync, never in git.

— yams