# YAM agent — read this first after a compaction

The minimum context-restoration checklist for any agent picking up YAM
work after a context compaction (or as a fresh agent on the YAM stack).

**Reading order**: this file → `FAQ.md` → whichever topic file the
current task touches. Don't dump everything; the FAQ + this file +
the right topic page is usually enough.

## What this stack does, in one paragraph

PARA "place mug" on the TRI bimanual YAM rig. DINOv3 ViT-S/16+
backbone, a **factored YX × Z × T volume head** with per-pixel CE on
the predicted future EE trajectory, K-means quaternion centroids
(K=64) for rotation, a 32-bin gripper head. Train on puget (GPU);
deploy on russet (has a 4090 too — direct raiden, no chiral). Scene
camera is HD1080 ZED 2i with an aruco board on the right-arm base;
**live exo-aruco recal is run every session** (no JSON load except
the static `T_lfr`).

## Canonical paths (memorize these)

| Concern | Path |
|---|---|
| YAM code (single source of truth) | `/data/cameron/para/robot/yam/` |
| Mounted on puget + russet as | `~/lab/para/robot/yam/` (sshfs of above) |
| Vault YAM slice | `/data/cameron/vault/para/robot/yam/` |
| Agent vault slice | `/data/cameron/vault/fleet/agents/yams/` |
| Training data + ckpts + eval_videos | `~/yam_para/{data,checkpoints,eval_videos}/` on **puget** |
| DINOv3 weights | `~/yam_para/dinov3/weights/` on **puget** (russet sees via `~/cameron/puget/`) |
| Russet venv | `~/cameron/raiden_fork.venv/` |
| Puget venv | `~/yam_para/.venv/` |
| Lab tailscale | `100.74.71.38` (`phe108-yuewang-01`, user `cameronsmith`) |
| Puget tailscale | `100.104.232.94` (alias `dev`) |
| Russet tailscale | `100.123.182.78` (alias `robot-lab`, password `robot-lab`) |

## Rules that override everything

1. **Edit on lab, never on rigs.** Both rigs sshfs-mount `/data/cameron`
   at `~/lab/`. Use `Read`/`Edit`/`Write` directly on the lab path; the
   rigs see updates instantly. Never `scp` code or edit via ssh heredocs.
   (See `feedback_edit_in_puget_yam_para.md`, but the pre-rehaul
   "edit in puget" rule is now obsolete — single source = lab.)
2. **Lab is for scripts, not large files.** Tailscale → DERP relay,
   ~25 MB/min. Anything > 100 MB stays on puget. If you find yourself
   rsyncing > 100 MB to lab, stop.
3. **Fail loud — no defensive try/except.** Identity fallbacks, `dict.get`
   on required keys, "log a warning and continue" — all banned. The full
   list of acceptable try/except is short; see `GUIDELINES.md`.
4. **Code Hygiene**: ≤300 lines/file, one responsibility per file, thin
   launchers + `lib/` modules, shared viz between train + deploy + viz
   scripts, delete dead code.
5. **Live exo recal at every session start**, never load extrinsics from
   JSON. `T_lfr` (= `T_left_from_right`) is the only static frame.
6. **127.0.0.1 not `localhost`** in any SSH tunnel through russet.
7. **Poll russet launches for 30 s** (10s/20s/30s) before walking away —
   ~90% of failures (ZED wedge, CAN dropout, raiden import) land in that
   window.
8. **Update vault after non-trivial work**: `tasks.md`, `overview.md`,
   `memory.md` (or `known-bugs.md` for fingerprinted failures).

## Minimum reading set (don't skip any)

| File | Why |
|---|---|
| `vault/para/robot/yam/FAQ.md` | **Read first.** Copy-paste commands for every task; common-error → fix table |
| `vault/para/robot/yam/INDEX.md` | Implemented vs TODO files; canonical vs deprecated |
| `vault/para/robot/yam/overview.md` | Current SOTA + tree layout + what's running |
| `vault/para/robot/yam/memory.md` | Frames (`T_lfr` gotcha), HD1080 patch, mujoco warp, chiral fragility, yellow-button table |
| `vault/para/robot/yam/known-bugs.md` | 14 fingerprinted failure modes + their recovery |
| `vault/fleet/agents/yams/memory.md` | Agent-level operating style (terse responses, edit-on-lab rule, watch russet 30 s) |
| `vault/tri/overview.md` | SSH chain + machine roster |
| `/data/cameron/agents_stuff/shared/GUIDELINES.md` | Fail-loud, Code Hygiene (300-line rule), vault protocol |

## Topic pages (read on demand)

| File | When |
|---|---|
| `control.md` | Touching `Robot` (russet-direct), `PolicyClient` (chiral remote), or any joint-command path |
| `calibration.md` | Touching exo aruco, `T_lfr`, wrist verify, scene-cam HD1080 patch |
| `recording.md` | Touching `record.py` / `record_skip_wrist.py` (legacy); episode layout |
| `conversion.md` | SVO2 → PNG via `rd convert`; PKL schema; hot-patch for legacy `_scene_in_lbase` |
| `training.md` | Architecture details for the pre-refactor `train_yam_single.py`; wandb scalars; xattn block |
| `inference.md` | Old fusion math (`sum`/`max`/`scene_only_decode`); chiral-based deploy topology |
| `rendering.md` | MuJoCo cam_fovy + principal-point warp + K-resolution scaling |
| `rerun-panels.md` | `LiveRerunSession` pattern + per-panel content + SSH tunnel ports |

## Code map (after the rehaul, single source of truth)

`/data/cameron/para/robot/yam/` (mount as `~/lab/para/robot/yam/`):

```
lib/
├── robot.py           ← Robot (raiden-direct); .move_to(target14, vel_rad_s)
├── smooth_move.py     ← smooth_move + MoveResult; parallel-thread per arm
├── policy_client.py   ← PolicyClient (chiral remote) — TODO/skeleton
├── calibration.py     ← solve_exo_aruco, pool_exo_aruco, load_T_lfr, to_world
├── render.py          ← render_arm_with_exo(joints7, T_cam_in_rbase, K, W, H)
├── teleop.py          ← start_teleop / stop_teleop
├── rerun_helpers.py   ← init_web, to_jpeg
├── rerun_session.py   ← LiveRerunSession (canonical scaffold)
├── ik.py              ← ee_chain_to_joints (mink + PostureTask)
├── inference.py       ← load_checkpoint, decode, decode_baseline, compute_workspace_mask
├── dataset.py         ← YamScene (FK + K-means rot + bin precompute)
├── model.py           ← DinoVolumeScene + all_losses
├── model_baseline.py  ← DinoCLSXYZBaseline + baseline_losses
├── data_io.py         ← TODO (PKL helpers)
├── video_recorder.py  ← VideoRecorder.{append,rotate,stop} — deploy.py --save_video
├── deploy_panels.py   ← log_{volume,baseline}_panels — deploy.py rerun panels
└── viz/
    ├── keypoints.py      ← rainbow KP + polyline (GT white-ring vs pred-filled)
    ├── heatmap.py        ← marginal_heatmap_{panel,grid}
    ├── feature_pca.py    ← DINOv3 PCA → RGB; fit_pca_basis + apply_pca_basis (two-step API for video)
    ├── overlay.py        ← compose_{render,mask}_over_live
    ├── trajectory.py     ← project_world_to_image
    └── bin_grids.py      ← per-(T, bin) Z/grip/rot heatmaps

calibrate.py             ← live exo calib + overlay (canonical LiveRerunSession example)
record.py                ← episode recorder (yellow-toggle state machine)
train.py                 ← scene-only DinoVolumeScene loop (puget GPU 0)
train_baseline.py        ← CLS→XYZ regression baseline (puget GPU 1)
deploy.py                ← live inference on russet (auto-detects model_kind via ckpt;
                            --save_video writes per-rollout MP4s)
vis_dataset.py           ← per-episode start-frame + robot mask + trajectory
vis_dataset_grid.py      ← 4×5 PNG + MP4 grid of all 20 training start frames
vis_eval_grid.py         ← per-session eval-video grid MP4 (one per ~/cameron/eval_videos/<session>/)
vis_paper_features.py    ← paper figure: 3×N rows of (RGB, shared-PCA, KP) across N episodes
vis_paper_features_within_episode.py ← same 3×N rows but N frames within one episode
vis_paper_features_video.py ← animated version: --n_episodes 1 (RGB|PCA|KP strip) or 10 (3×10 grid)
vis_results.py           ← snapshot N frames of one episode w/ wandb-style panels
vis_ik_debug.py          ← offline IK verification: per-T mujoco overlay
bin/
├── yam-mount            ← sshfs ~/lab (idempotent; -u to unmount)
└── yam-rd-serve         ← rd serve respawn loop on russet
unit_tests/
├── calibration_test.py  ← offline aruco + render + composite PNG
└── control_speed_test.py ← Robot.move_to timing on a 2-pose .npz
```

Legacy / pre-rehaul also lives here (kept for refactor reference):
`code/` (deploy_yam_2view.py, train_yam_single.py, model_yam_single.py,
data_yam_single.py, hot_patch_scene_to_lbase.py), `yam_control/`
(record_skip_wrist.py), `raiden_fork/` (TRI tooling source + meshes).

## Conventions to internalize before touching anything

- **World frame = right_arm_base** (new convention, 2026-06-04). Fresh
  PKLs from `record.py` store the scene extrinsic as `T_cam_in_rbase`
  directly and mark themselves with `_scene_in_rbase=True`. No T_lfr
  loading — calibration is live every session via exo-aruco
  (`pool_exo_aruco` in `LiveRerunSession.lock_exo_pose`). The
  `calibration_results.json` carries `world_frame: "right_arm_base"`
  to make this explicit downstream.
- **Legacy convention: world = left_arm_base.** Old PKLs (e.g.
  `place_mug_saucer_single_proper`) store `T_cam_in_world` where world
  is left_arm_base, plus a `T_left_from_right` key and an
  `_scene_in_lbase=True` marker. The trained ckpts on those datasets
  (`place_mug_saucer_v0`, `place_mug_saucer_baseline_v0`) assume that
  world frame. Downstream code (dataset.py, deploy.py, vis_* scripts)
  still uses the legacy path; needs a follow-up to support both
  cleanly. See task #115.
- **The misnamed JSON key**: raiden's `calibration_results.json` writes
  the key `right_base_to_left_base` but stores its **inverse** (= true
  T_lfr = left_from_right). Only matters when loading legacy data —
  fresh data has no T_lfr at all.
- **IK targets must be in right_arm_base**, not world. With the new
  convention, world IS right_arm_base, so `ee_chain_to_joints` no
  longer needs a T_lfr (pass `T_lfr=None` or omit). With legacy data,
  pass `T_lfr=T_lfr` so the conversion happens.
- **Volume-head dot products are RAW** (no `F.normalize`, no `log_temperature`).
  Normalization caps logits in [-1, 1] and the per-T softmax over ~524k
  voxels can never sharpen — heatmap stays flat. Loss plateaus ~11 nats
  vs. ~6 raw. See `known-bugs.md` #11.
- **Cell-center pixel coords** for unprojection: `(bin + 0.5) * cell_size`,
  not top-left. Matches pre-refactor `unproject_voxel_to_world`.
- **Workspace mask** is applied **before** the volume argmax at decode
  time; ~92k of 100k voxels valid for healthy extrinsics, ~30k = bad.
- **IK chain keeps partial-q on threshold-miss**; do NOT fall back to seed.
  At 1e-4 thresholds even ~5–10 mm solves nominally "fail" — throwing
  them away cascades to "stay at home" across the chunk. See `known-bugs.md` #12.
- **Robot vs chiral methods**: russet-direct `Robot` has `.move_to(...)`.
  Chiral `PolicyClient` has `.smooth_move_to(...)`. Don't conflate (this
  bit us 2026-06-03). All live robot code in launchers should wrap the
  main loop in `try: … finally: go_home(); robot.shutdown()` so a crash
  doesn't drop the arm under gravity.
- **K-means rot centroids ride INSIDE the checkpoint** under
  `dataset_cfg.rot_centroids`. No external `.npz` cache. Refit at every
  dataset init is deterministic (seed=0) so we never have two-copies-
  that-disagree.

## Failure-mode fingerprints (quick lookup)

| Symptom | Cause | Recovery |
|---|---|---|
| ~3 s pause between chunk substeps | chiral lock contention from obs stream | lower obs hz to 5; do NOT stop_obs_stream mid-chunk (bug #1) |
| `rd serve` dies on every deploy quit | client disconnect → estop → shutdown (intentional) | use `bin/yam-rd-serve` respawn wrapper |
| ZED 2i lockup / `MOTION SENSORS NOT DETECTED` | wedge after Killed/OOM | unplug 30 s OR reboot — only physical recovers |
| `cv2.aruco` has no `detectMarkers` | missing contrib | `uv pip install --force-reinstall opencv-contrib-python==4.13.0.92` |
| Workspace mask ~30k voxels (was ~92k) | extrinsic in right_arm_base, not world | apply `T_lfr` (hot-patch PKLs or `to_world(...)` live) |
| Overlay arm half-size | K not scaled to render res | scale K to (Wr, Hr) before `cam_fovy` |
| Overlay arm offset ~10 px | missing principal-point warp | `cv2.warpAffine` by `(cx − W/2, cy − H/2)` |
| `success: True` missing from calib JSON | converter silently drops extrinsics | re-write JSON, re-run `rd convert` for affected episodes |
| Volume head flat heatmap, loss ~11 nats | L2-normalized dot products | raw scores; no `F.normalize`, no temperature |
| IK chain collapses to all-home | seed fallback on threshold-miss | keep partial q (bug #12) |
| `'Robot' object has no attribute 'smooth_move_to'` | conflating chiral API with local Robot | use `.move_to(...)`; wrap in `try/finally + go_home` |

Full table: `vault/para/robot/yam/known-bugs.md` (bugs 1-14, this list
should grow as new fingerprints land).

## What to do BEFORE making changes

1. **`git status`** in both `/data/cameron/vault/` and
   `/data/cameron/para/`. Commits are local — Cameron pushes manually.
2. **Confirm mounts on rigs** if your work touches them:
   `ssh dev 'ls ~/lab/para/robot/yam/ | head'` and same for `robot-lab`.
3. **Confirm `rd serve`** if you're about to deploy:
   `ssh robot-lab 'nc -z 127.0.0.1 8765 && echo ok || echo dead'`.
4. **Pin canonical script names** — if you can't tell from the launcher's
   first 50 lines what's current, check `INDEX.md`.

## What to do AFTER non-trivial work

1. Update `vault/para/robot/yam/tasks.md` (mark done, add follow-ups).
2. Update `overview.md` if SOTA / current run / blocker changed.
3. Append a dated entry to `memory.md` if you learned a non-obvious
   convention; or `known-bugs.md` if you fingerprinted a new failure.
4. `git commit` in `vault/` and (if touched) `para/`. Don't push.
5. If your work pollutes auto-memory, drop relevant feedback entries to
   `~/.claude/projects/-data-cameron/memory/` so future sessions inherit.

## How to drive the rigs (tmux send-keys, not direct ssh)

Cameron keeps long-lived shells open in tmux windows on the lab server.
**The canonical way an agent drives puget or russet is via
`tmux send-keys` to those windows** — not by opening a fresh `ssh dev` /
`ssh robot-lab` from your own bash. The tmux pane already has the right
venv activated and the right cwd; reuse it.

| Window | Host | Typical use |
|---|---|---|
| `agents:5` | puget (`dev`) | training launches, dataset rsync, wandb tail |
| `agents:6` | russet (`robot-lab`) | `rd serve`, `deploy.py`, `record.py`, `calibrate.py` |

### Read what's on a pane

```bash
tmux capture-pane -t agents:6 -p | tail -20
```

(Use `-S -200` for more scrollback.)

### Send a command

```bash
tmux send-keys -t agents:6 'python ~/lab/para/robot/yam/deploy.py --ckpt …' Enter
```

After sending: **poll every 10 s for 30 s** (agent-level rule, see
`vault/fleet/agents/yams/memory.md`). Most failures (ZED wedge, CAN
dropout, raiden import error, missing-file) surface in that window:

```bash
for i in 1 2 3; do
  sleep 10
  out=$(tmux capture-pane -t agents:6 -p | tail -10)
  echo "$out" | grep -qE "Traceback|fail to communicate|MOTION SENSORS|RuntimeError|FileNotFoundError" && { echo FAIL; break; }
  echo "$out" | grep -qE "locked\\.|teleop is live|deploy prompt|\\[y\\]=deploy" && { echo OK; break; }
done
```

If you see a fresh `Traceback`, surface it to Cameron with the relevant
lines. If you see the deploy/record prompt, report ready.

### Don't open a parallel ssh session

If you `ssh robot-lab` from your bash and launch a long-running process
there, its output goes to your shell (gone when this turn ends),
Cameron can't see it in his pane, and you've fragmented state. Always
use the existing pane.

### Gotchas

- **`capture-pane -p` shows only the visible pane**, not full scrollback.
  Right after a launch, the OLD traceback may still occupy the visible
  area while the NEW process's startup output is below the cut-line.
  Use `tmux capture-pane -t agents:6 -p -S -200 | grep -E "loaded ckpt|locked|Traceback"` to confirm a fresh launch landed.
- **Long send-keys may need a second `Enter`** (GUIDELINES paste-confirm
  note). If your command sits on the prompt unexecuted, send
  `tmux send-keys -t agents:6 "" Enter` once more.
- **Don't send blank Enters while a script is at an input() prompt** —
  they're consumed by the script's prompt loop, not the shell. Confirm
  the pane state with `capture-pane -S -200` before sending Enter.

### Interactive prompts (y/q/h/r/c on `deploy.py`, yellow on `record.py`)

These are driven by Cameron's keyboard — don't try to send `y` via
`tmux send-keys` unless he asks you to. Your job is to launch the
script, confirm it reached the prompt, and report.

## Where to get unstuck

- **What's running right now?** `tmux ls`, then `tmux capture-pane -t
  agents:<n> -p | tail` for the active panes (5 = puget, 6 = russet).
- **Last known good state?** `vault/para/robot/yam/overview.md` is
  updated after non-trivial changes.
- **Why does the code do X?** Grep the relevant `lib/<module>.py` for
  docstrings — they encode reasoning, not just behavior.
- **Cameron prefers**: terse + opinionated answers. Reference vault by
  path (`per vault/para/robot/yam/memory.md…`). Don't ask permission
  between obvious chain steps; surface decisions only.