# YAM — canonical scripts INDEX

What's canonical, what's deprecated, and where it lives. Update whenever
the canonical changes.

## Post-rehaul layout — LIVE (2026-06-02): `/data/cameron/para/robot/yam/`

Mounted as `~/lab/para/robot/yam/` on puget + russet (see
`vault/tri/overview.md` for the mount setup).

### Implemented modules

| File | Status | Purpose |
|---|---|---|
| `lib/robot.py` | implemented | `Robot` class (raiden-direct on-russet). `move_to(target14, vel)` blocks until done. Context-manager safe. |
| `lib/smooth_move.py` | implemented | `smooth_move(robot, target, vel)` algorithm; `MoveResult` with planned vs actual timing |
| `lib/calibration.py` | implemented | `solve_exo_aruco`, `pool_exo_aruco`, `load_T_lfr`, `to_world`, `average_transforms` |
| `lib/render.py` | implemented | `render_arm_with_exo(joints7, T_cam_in_rbase, K, W, H) → (rgb, mask)`. Uses `MUJOCO_GL=egl`. |
| `lib/viz/overlay.py` | implemented | `compose_render_over_live` + `compose_mask_over_live` (red silhouette variant) |
| `lib/rerun_helpers.py` | implemented | `init_web(name, port)` + `to_jpeg(bgr, target_w, quality)` |
| `lib/rerun_session.py` | implemented | **`LiveRerunSession`** — canonical scaffold for live rerun scripts (camera + teleop + rerun + JPEG log + yellow exit). See `rerun-panels.md`. |
| `lib/teleop.py` | implemented | `start_teleop(arm)` / `stop_teleop(rc)` |
| `lib/policy_client.py` | TODO | Chiral remote variant of Robot (for inference) |
| `lib/data_io.py` | TODO | PKL load + hot-patch helpers |
| `lib/viz/keypoints.py` | implemented | `keypoints_overlay(bgr, kp_2d, in_frustum)` — rainbow points + polyline with fade for out-of-frustum |
| `lib/viz/trajectory.py` | implemented | `project_world_to_image(points_world, K, T_world_to_cam, image_wh)` — generic projection + in-frustum mask |
| `lib/viz/feature_pca.py` | implemented | `feature_pca_panel(patch_features, out_wh)` — first-3 PCs → RGB |
| `lib/viz/heatmap.py` | implemented | `marginal_heatmap_panel(img_bgr, logits_tzhw, timestep)` — jet-colored softmax-over-Z blended |
| `lib/model.py` | implemented | `DinoVolumeScene` (scene-only, factored KV head) + `volume_ce_loss(logits, target_yx, target_z)` |
| `lib/model_da3.py` | implemented | `DA3VolumeScene` — third variant. Depth-Anything-3 backbone (`/data/cameron/da3_weights/`) + camera token (replaces CLS) + parallel feat DPT branch (deep-copy of depth branch, final 1×1 conv re-targeted to `feat_dim`). Same factored YX×Z×T scoring + AdaLN-Zero heads as `DinoVolumeScene`. Defaults to `n_height_bins=128`, `img_size=504` (DA3 requires `H % 14 == 0`), `feat_dim=32`. Train with `--da3`; inference dispatch via `model_kind=da3_volume_scene`. |
| `lib/dataset.py` | implemented | `YamScene` — reads `record.py` output; FK on saved joints → target_yx + target_z |
| `train.py` | implemented | Minimal scene-only loop: wandb logging, ckpt every 100 iters, viz panels every 50. |
| `train_baseline.py` | implemented | CLS-only → global XYZ regression baseline (no spatial reasoning). Same grip + rot heads. Viz uses cam pose to project predicted 3D back to image. |
| `lib/model_baseline.py` | implemented | `DinoCLSXYZBaseline` + `baseline_losses(MSE + 2× CE)`. |
| `deploy.py` | implemented | Live inference on russet (4090): forward + decode + mink-IK + `Robot.move_to` per chunk step. y/q/h/r/c text interface. Same rerun panels as training viz (heatmap with + without RGB, pred kp, PCA, mujoco@current overlay). |
| `lib/ik.py` | implemented | `ee_chain_to_joints` — seeded mink-IK over a (T, 3) + (T, 4) trajectory. |
| `lib/inference.py` (extended) | implemented | Adds `unproject_pixel_to_world(pix, height, K, T_cam_in_world)`; `decode` now returns `xyz_world` when K + extrinsic provided. |
| `vis_results.py` | implemented | Snapshot predictions on N evenly-spaced frames of one episode → unit_tests/output/fig_results_vis_scratch/. |
| `vis_ik_debug.py` | implemented | Offline IK debug: forward + decode + IK + per-T mujoco render of IK'd joints over the start frame + predicted-keypoint overlay. Convergence error printed per timestep. Output → `unit_tests/output/ik_debug/<ep>/f<frame>/`. |
| `lib/inference.py:compute_workspace_mask` | implemented | (Z, P, P) bool — voxels in front of cam AND in front of both arm bases. Applied in `decode` before argmax. |
| `vis_dataset.py` | implemented | Start-frame + robot mask + EE-trajectory polyline per episode. Output → `unit_tests/output/dataset_viz/<name>/NNNN.png`. |
| `vis_dataset_grid.py` | implemented | 4×5 PNG grid of all 20 training start frames + synchronized 4×5 video (`dataset_vis_vid.mp4`). Output → `unit_tests/output/dataset_grid/`. |
| `vis_dataset_grid_kp.py` | implemented | Auto-sized PNG grid (default ≤6 cols) of all episode start frames, each with the FK'd EE world trajectory projected through the scene-cam and drawn as a rainbow polyline. The at-a-glance "show me what's in this dataset" view — preferred over `vis_dataset.py` (per-episode) when reviewing a fresh dataset before training. `--data_root <path>`. Output → `unit_tests/output/dataset_grid_kp/<name>/grid_start_frames_kp.png`. |
| `vis_eval_grid.py` | implemented | Per-eval-session video grid: combines all `vid_*.mp4` rollouts in `~/cameron/eval_videos/<session>/` into one auto-sized grid MP4 named `<session>_eval_grid_vid.mp4`. Filters out rollouts with `< --min_frames` frames (default 10) so empty/1-frame writes don't pollute the grid. |
| `vis_paper_features.py` | implemented | Paper figure: 3 rows × N cols of (RGB / shared-PCA features / predicted keypoints), one start-frame each from N=10 evenly-spaced training episodes. Single SVD across all 10 maps so colors are comparable. Output → `unit_tests/output/paper_features/grid_rgb_pca_kp.png`. |
| `vis_paper_features_within_episode.py` | implemented | Same 3×10 figure but columns are 10 frames uniformly sampled from start→end of one episode. Output → `unit_tests/output/paper_features/grid_rgb_pca_kp_within_ep_<NNNN>.png`. |
| `vis_paper_features_video.py` | implemented | Animated version. `--n_episodes 1` (default): horizontal `RGB|PCA|KP` strip per frame, video over one episode. `--n_episodes 10`: 3×10 grid synced across 10 episodes (pad with last frame). Shared PCA basis fit on `--pca_n_samples` (default 10) frames per episode. Output → `unit_tests/output/paper_features/video_{within_ep_<ep>,<N>_episodes}.mp4`. |
| `lib/video_recorder.py` | implemented | `VideoRecorder.{append,rotate,stop}` — sparse per-substep MP4 writer used by `deploy.py --save_video`. Auto-rotates to `vid_NNN.mp4` on `recorder.rotate()` (called on h-press in deploy). |
| `lib/deploy_panels.py` | implemented | `log_volume_panels()` + `log_baseline_panels()` for deploy.py rerun panels. Both call the same `lib/viz/*` primitives that train.py/train_baseline.py use → wandb panels and rerun panels stay visually identical. |
| `lib/viz/feature_pca.py` (extended) | implemented | Three APIs: `feature_pca_panel` (per-image), `shared_feature_pca_panels` (one basis across N maps, returns panels), and the **two-step `fit_pca_basis` → `PCABasis` → `apply_pca_basis`** for video pipelines (basis fit on sample frames, then applied per-frame). |
| `lib/viz/bin_grids.py` | implemented | `bin_grid_panel(logits_tk, target=None)` — per-(T, bin) heatmap with optional GT outline. Used by train.py + train_baseline.py + deploy.py (z/grip/rot panels). |
| `unit_tests/control_speed_test.py` | implemented | Chain `Robot.move_to` over 2-pose `.npz`; report wall-clock vs planned vs achieved rad/s |
| `unit_tests/calibration_test.py` | implemented | Offline aruco solve + render + composite PNG to `unit_tests/output/` |
| `calibrate.py` | implemented | Live: pool 10 aruco frames, render at latest joints, log overlay to rerun. **Canonical `LiveRerunSession` example.** |
| `record.py` | implemented | Episode recording on top of `LiveRerunSession`: 15Hz, scene 960×540 JPEG q90 + wrist 1280×720 JPEG q90 + lowdim PKL per frame; writes `calibration_results.json` with `success=True`; yellow button → finalize + exit. |
| `bin/yam-mount` | implemented | sshfs `/data/cameron` → `~/lab` (idempotent) |
| `bin/yam-rd-serve` | implemented | `rd serve` in respawn loop |

### TODO launchers (skeletons exist)

| Launcher | Replaces | Purpose |
|---|---|---|
<!-- record.py implemented above; the row below is removed -->
| ~~`record.py`~~ — implemented | (was `yam_control/record_skip_wrist.py`) | — see above |
| `convert.py` | direct `rd convert` invocation | SVO2 → PNG + lowdim PKL with archival JSON |
| `train.py` | `code/train_yam_single.py` | Train DinoVolumeQueryBimanual with optional wrist + xattn |
| `deploy.py` | `code/deploy_yam.py` + `code/deploy_yam_2view.py` | Live inference with scene+wrist fusion |

### `lib/` modules (target)

| Module | Responsibilities |
|---|---|
| `lib/calibration.py` | `T_lfr` load, exo aruco solve, pool-until-N detections, scene-cam K |
| `lib/mujoco_render.py` | YAM_BASE_ONLY_CONFIG renderer, principal-point warp, K scaling, GL context |
| `lib/data_io.py` | PKL load/save, per-frame `T_wrist_in_world`, `_scene_in_lbase` marker |
| `lib/chiral_helpers.py` | Safe obs-stream patterns (don't pause mid-loop), latest_obs read |
| `lib/viz.py` | Shared rerun panels: keypoints overlay, marginal heatmap, PCA |
| `lib/fusion.py` | Lift-project decode, sum/max fusion, scene_only |
| `lib/model.py` | `DinoVolumeQueryBimanual` (currently `model_yam_single.py`) |

## Current (pre-rehaul) lab paths

Single source of truth = puget `~/yam_para/`. Russet/yukon sshfs-mount it.

| File | Path on puget |
|---|---|
| Single-arm recorder | `~/yam_para/yam_control/record_skip_wrist.py` |
| Two-view deploy | `~/yam_para/code/deploy_yam_2view.py` |
| Trainer | `~/yam_para/code/train_yam_single.py` |
| Model | `~/yam_para/code/model_yam_single.py` |
| Dataset | `~/yam_para/code/data_yam_single.py` |
| Viz (start-frame + KP) | `~/yam_para/code/viz_start_frame_keypoints.py` |
| Hot-patch (one-shot) | `~/yam_para/code/hot_patch_scene_to_lbase.py` |
| raiden_fork | `~/yam_para/raiden_fork/` |
| DINOv3 backbone | `~/yam_para/dinov3/` |
| Weights | `~/yam_para/dinov3/weights/dinov3_vits16plus_pretrain_lvd1689m-4057cbaa.pth` |
| Checkpoints | `~/yam_para/checkpoints/yam_wrist_xattn_v1/latest.pth` |
| Training PKLs | `~/yam_para/data/place_mug_wrist_cam/` |
| Eval videos | `~/yam_para/eval_videos/` |

Auxiliary on russet: `~/cameron/yam_control/` is the sshfs mount of
puget's `~/yam_para/yam_control/`.

## Evaluation reports

See **`evaluation.md`** for the rollout-annotation hub, the 8-outcome
taxonomy + colors, the bar-chart reading guide, and the per-experiment
prefix mapping. Current canonical report:
[volume_based_vs_baseline_2026-06-10](https://omidlab.net/browse/para/robot/yam/unit_tests/output/eval_rollout_pages/volume_based_vs_baseline_2026-06-10/index.html).

Hub: <https://omidlab.net/browse/para/robot/yam/unit_tests/output/eval_rollout_pages/index.html>

Regenerate the latest report (after new annotations land or new eval
sessions get recorded):

```bash
cd /data/cameron/para/robot/yam
python code/eval_rollout_pages_volume_based.py
```

## Deprecated (do not re-resurrect)

- `deploy_yam.py` — single-cam variant, superseded by `deploy_yam_2view.py`.
  Kept only as a fallback reference. Move to `archive/` after rehaul.
- Stock raiden `record` for single-arm — wrist-cam workflow breaks without
  `record_skip_wrist.py`. Single-arm always uses the custom recorder.
- Old single-arm config files predating wrist-cam — `archive/` on rehaul.

## Verifying which version is canonical

When in doubt: read the launcher's first ~50 lines (it documents intent +
flags). The launcher imports from `lib/<module>.py`; the modules are the
durable layer.