# YAM — conventions, gotchas, and code pointers

Append-only. Detailed pointers to canonical implementations. Higher-level
"what kind of agent am I" notes live in
`vault/fleet/agents/yams/memory.md`; the cross-cutting PARA conventions
live in `vault/para/memory.md`.

## Frames / transforms

### World frame convention — switched 2026-06-04

**New (post-2026-06-04, fresh recordings)**: **world = right_arm_base.**
No T_lfr loading; live exo aruco solves `T_cam_in_rbase` and that's
stored directly as the scene extrinsic. PKLs carry
`_scene_in_rbase=True` + `calibration_results.json` carries
`world_frame: "right_arm_base"`. `record.py` no longer takes a
`--t_lfr_path` flag. See `yam_agent_on_startup.md` "Conventions".

Driving reason: T_lfr is per-rig calibration that drifted between
russet's saved value and the actual physical state. Cameron's standing
rule is "never use pre-calibrated poses" — every session lives off
the live exo solve, no static frames.

**Legacy (pre-2026-06-04, `place_mug_saucer_single_proper` and
earlier)**: world = left_arm_base; PKLs carry `T_left_from_right` +
`_scene_in_lbase=True`. Old ckpts (`place_mug_saucer_v0`,
`place_mug_saucer_baseline_v0`) assume that frame. Downstream code
(dataset.py, deploy.py, vis_*) still threads T_lfr through — a
follow-up will support both conventions cleanly (vault task #115).

### DA3VolumeScene — Depth-Anything-3 backbone variant (2026-06-05)

Third model variant alongside `DinoVolumeScene` (factorized DINOv3) and
`DinoVolumeScenePerVoxel` (didn't generalize as well). Lives in
`lib/model_da3.py`. Same factored YX×Z×T scoring + AdaLN-Zero stack as
the DINOv3 factorized model — only the backbone + per-pixel head + global
token differ:

- **Backbone**: DA3-SMALL (`/data/cameron/da3_weights/`, HF
  `depth-anything/da3-small`). ViT-S/14 + cat_token (output is
  `2 × embed_dim = 768`-d per token). Geometry-pretrained on monocular
  depth + camera pose.
- **Global token = DA3 camera token**, NOT CLS. DA3 wasn't trained with
  CLS as a meaningful signal; the cam token at `feats[-1][1]` is what
  DA3's own pose decoder consumes. We feed it through a fresh LayerNorm
  + the existing `input_proj` (concat with `eef_feat` sampled at
  `start_pix`, exactly like CLS in `DinoVolumeScene`).
- **Per-pixel features = third DPT branch**, deep-copied from the depth
  branch and re-targeted. We share the trunk (`head.norm`, `head.projects`,
  `head.resize_layers`, `head.scratch.layer{1..4}_rn`) by reference,
  deep-copy `refinenet{1..4}` and `output_conv1`, and replace the final
  1×1 conv of `output_conv2` (`32 → output_dim=2`) with
  `Conv2d(32, feat_dim, 1)`. Depth and ray branches keep their original
  parameters but aren't called.
- **Defaults** vs DinoVolumeScene: `n_height_bins=128` (vs 32; 4× finer
  height resolution, ~2 mm/bin on the in-dist data — see auto z_range
  printout); `feat_dim=32` (vs 48); `img_size=504` (DA3 requires
  `H % 14 == 0`; 405 fails the patch-embed assertion). `pred_size=128`
  unchanged (we bilinear-up the refinenet1 output directly to 128 to
  skip DA3's native 504-res output — saves ~16× on the final convs).
- **Fragility note**: DA3 imports require sys.path injection of
  `/data/cameron/da3_repo/src` and stubbing of
  `depth_anything_3.utils.{export,pose_align}` to avoid optional deps
  (GLB writers, scipy pose-alignment). All of this is contained in
  `lib/model_da3.py` module-level setup — single source of monkey-patching.

CLI: `--da3` flag in `train.py` (mutex with `--per_voxel`). Inference
dispatches via `model_kind == "da3_volume_scene"`. Saves the same
ckpt payload shape as DinoVolumeScene, so the `decode()` path works
unchanged.

Deps installed on yukon (`raiden_fork.venv`): `addict`, `omegaconf`,
`safetensors`, `einops`, `huggingface_hub`, `scikit-learn`, `wandb`.

First in-dist run: `da3_yukon_pickplace_indist_v1` on yukon GPU 0,
batch 16 → ~9 it/s (28.4M trainable params; DA3-small + DPT + PARA
heads). Wandb: `cameronsmithbusiness/yam_train_simple/runs/o3v8k78c`.

### Always normalize bin ranges from the dataset (2026-06-05)

Convention: **per-dataset bin ranges, never hardcoded.** Mirrors how
rotation K-means centroids are already data-derived. Concretely:

- **`z_range` (height bins)**: auto-derive `(z_min, z_max)` from the
  dataset's EE z observations (with small padding, default ±1cm).
  `lib/dataset.py:YamScene` does this when `z_range=None` (the new
  default) and stores the chosen range in `dataset_cfg["z_range"]` so
  inference/viz pick it up. `train.py --z_lo/--z_hi` still take explicit
  overrides for reproducibility but default to None → auto.
- **K-means rot centroids**: already auto-fit on the dataset's
  quaternions (deterministic seed=0). Saved in
  `dataset_cfg["rot_centroids"]`.
- **Finetune (`--init_from`)**: MUST inherit both `z_range` AND
  `rot_centroids` from the source ckpt — otherwise the volume head's
  z-bin semantics and the rot_head's centroid IDs scramble. `train.py`
  does this automatically when `--init_from` is set.

Pre-2026-06-05 ckpts were trained with the hardcoded `(0.0, 0.4)` m
range. They wasted ~6 of 32 bins on empty 0.31–0.4 m space — not catastrophic
but a chunk of head capacity. Future runs should leave `--z_lo/--z_hi`
unset and let YamScene fit the range from data.

If you ever want to compare two ckpts trained on different datasets,
they likely have **different z_range and rot_centroids**, so direct
weight comparisons / averaging are not meaningful (the volume and
rot heads' output spaces differ).

## `T_lfr = T_left_from_right` (legacy — misnamed JSON key)

For legacy data only: `raiden`'s `calibration_results.json` writes the
key `right_base_to_left_base` but its content is **the inverse** of
what the name suggests — it actually maps left → right. When loading
legacy JSON directly:

```python
T_lfr = np.linalg.inv(json_data["right_base_to_left_base"])
```

Legacy PKLs store the (correct, non-inverted) T_lfr under the
`T_left_from_right` key — load from there for any legacy data.

See: `feedback_raiden_bimanual_transform_misnamed.md` in auto-memory.
Hot-patch script for existing PKLs: `code/hot_patch_scene_to_lbase.py`
(idempotent via `fd['_scene_in_lbase'] = True` marker).

### Per-frame wrist-in-world

The training PKLs store `T_wrist_in_world` per-frame (not derived from
joint state on the fly). Dataset reads it from each frame's lowdim
when projecting scene predictions into the wrist image.

## Scene cam HD1080 — two-file patch

The ZED 2i scene cam needs HD1080; the wrist ZED Minis stay HD720.
Two name-conditional patches required in `raiden_fork`:

```python
# raiden/cameras/zed.py:57  AND  raiden/server.py:1213
params.camera_resolution = (
    sl.RESOLUTION.HD1080 if name == "scene_camera"
    else sl.RESOLUTION.HD720
)
```

Both must be patched — `cameras/zed.py` is used by the recorder,
`server.py` is used by `rd serve`. Patch one and not the other →
silent resolution mismatch.

## MuJoCo principal-point warp (critical for exo+arm overlay)

MuJoCo's `cam_fovy` renderer assumes principal point = image center
(cx = W/2, cy = H/2). Real ZED 2i intrinsics shift cx, cy by tens of
pixels. After rendering, apply a translation warp:

```python
M = np.float32([[1, 0, cx - W / 2], [0, 1, cy - H / 2]])
rendered = cv2.warpAffine(rendered, M, (W, H))
```

Compose with the segmentation mask for overlay. See:
`feedback_mujoco_principal_point_warp.md` in auto-memory.

## Render K must match render resolution

For the exo+arm render at `(DW, DH)` (e.g. 1280×720), scale the
HD1080-native K accordingly:

```python
K[0, 0] *= DW / W_native; K[0, 2] *= DW / W_native
K[1, 1] *= DH / H_native; K[1, 2] *= DH / H_native
```

Bug fingerprint when omitted: fovy looks wrong (~35° instead of ~52°),
arm overlay clearly miniature against the real image.

Heuristic for `W_native`: `cx ≈ 960` → HD1080; `cx ≈ 640` → HD720.

## Texturedir for the exo aruco PNG

The MuJoCo XML for `YAM_BASE_ONLY_CONFIG` references the aruco board
PNG by a relative path. Without injecting `texturedir`, the renderer
fails to find it unless the cwd happens to be right. Patch the
`<compiler>` tag in the XML string:

```python
xml = xml.replace("<compiler ", f'<compiler texturedir="{_EXO_DIR}/" ')
```

## opencv-contrib-python required

`cv2.aruco.detectMarkers` lives in opencv-**contrib**, not the
mainstream wheel. Both russet and puget need `opencv-contrib-python`.
If you see `module 'cv2' has no attribute 'aruco'` after a fresh
install, force-reinstall:

```bash
uv pip install --force-reinstall opencv-contrib-python==4.13.0.92
```

## chiral protocol — single-lock fragility

`raiden/chiral/client.py` serializes send+recv on a single
`asyncio.Lock`. **Do not** `env.stop_obs_stream()` and then immediately
call other RPCs — cancelling the obs task between its `send` and `recv`
leaves an orphan obs frame on the wire, which the next `smooth_move_to`
recv consumes (returning `{}`), and the real smooth_move response then
poisons the next obs decode (`KeyError('cameras')`). See
`known-bugs.md` for the full failure mode.

## `rd serve` dies on client disconnect

The server interprets every client disconnect as an emergency stop and
shuts itself down after homing the arms. **Restart `rd serve` between
every deploy session.** No workaround on the server side — this is
intentional safety behavior in raiden.

## ZED 2i recovery — physical unplug only

Once a ZED 2i locks up (typically after a `Killed` process while
streaming HD1080), software resets do not recover it. Only:
1. Unplug for 30+ seconds, OR
2. Reboot the host.

If you must keep deploying, failover to a different scene cam via
yukon. See `feedback_polling_cadence.md` (don't sleep 30s polling, the
cam either comes back fast or never).

## Yellow leader button mapping

| Context | Button | Action |
|---|---|---|
| `rd record` | Left pedal / leader top button | Start or stop recording |
| `rd record` | Middle pedal / top leader button | Mark current episode as **Success** |
| `rd record` | Right pedal / bottom leader button | Mark current episode as **Failure** |
| `rd calibrate-wrist` | Yellow #1 | FREEZE 1: stop teleop; start accumulating aruco detections |
| `rd calibrate-wrist` | Yellow #2 | UNFREEZE: resume teleop (data locked from #1) |
| `rd calibrate-wrist` | Yellow #3 | FREEZE 2: stop teleop; snapshot wrist+scene+state |
| `rd calibrate-wrist` | Yellow #4 | UNFREEZE: resume teleop to safe end pose |
| `rd calibrate-wrist` | Yellow #5 | END: solve `T_ee_cam` from FREEZE 1 history |
| `rd wrist-verify` | Yellow | END + return home |
| `rd capture-wrist-pose` | Yellow | CAPTURE joint state + exit |

Source: `raiden_fork/raiden/calibration/wrist_calibrate.py:4-13`,
`raiden_fork/raiden/cli.py:871-873`, `raiden_fork/raiden/recorder.py:594-597`.

## Workspace mask

`DinoVolumeQueryBimanual` builds a workspace mask of valid 3D voxels.
~92k / 100k valid when the scene cam extrinsic is right. **A grossly
wrong extrinsic** (e.g. forgot to apply `T_lfr`) shows up as ~30k or
fewer valid voxels — fast sanity check at deploy startup.

## Edit-in-puget rule (until rehaul completes)

Single source of truth for now: edit YAM scripts in puget's
`~/yam_para/{code,raiden_fork,yam_control}/`. russet and yukon mount via
sshfs. Editing on russet directly = lost-on-next-mount silent overwrite.
Post-rehaul, single source moves to `/data/cameron/para/robot/yam/`.

See: `feedback_edit_in_puget_yam_para.md` in auto-memory.

## SSH tunnels — 127.0.0.1 not "localhost"

When setting up SSH `LocalForward` *into* russet,
**use `127.0.0.1`**, not `localhost`. russet's nsswitch resolution of
"localhost" fails. Pattern that works:

```
LocalForward 9094 127.0.0.1:9094
LocalForward 9095 127.0.0.1:9095
```

See: `feedback_russet_ssh_tunnel_use_127.md` in auto-memory.

## Deploy ergonomics

- Use `--use_train_cam` when live recal is wrong (drift > a few cm /
  rotation > ~5°). It loads the cam extrinsic from the first PKL of
  training data.
- Camera-drift warning in deploy startup is a strong signal that recal
  failed — investigate before proceeding (don't just override).
- Pool live recal to `min_detections=8` (not the older 3 s window).
- `--no_render` skips the cam_check render; useful for headless / fast
  spin-up.
- `--save_video` is heavy — only enable for "save for paper" runs;
  default off.
- Eval video naming: `<eval_name>__<timestamp>.mp4` under
  `~/yam_para/eval_videos/` (puget) → mounted into russet → browsable
  at `omidlab.net/browse/yam_remote/cameron/puget/eval_videos/`.

## Aug-token (out-of-frustum) — wrist abstain pattern

When the predicted scene pixel lifts to a 3D point that projects outside
the wrist image (or behind the camera), there is no patch token to attend
to. Instead of zeroing or masking, we add a learnable "out-of-frustum"
token to the wrist logits and let the model emit it as an abstain class.

```python
self.F_oof_token = nn.Parameter(torch.randn(d_feat) * 0.02)
...
score_yx_w = torch.einsum('btac, bchw -> btahw', q_F, F_wrist)
score_oof = torch.einsum('btac, c -> bta', q_F, self.F_oof_token)
wrist_logits = torch.cat(
    [score_yx_w.reshape(B, T, A, Hw * Ww), score_oof.unsqueeze(-1)],
    dim=-1,
)
# CE target = abstain_idx = Pw * Pw  when GT off-image.
```

Origin: borrowed verbatim from Backbones agent (do not reimplement —
reuse).