# YAM — FAQ + how-to cheatsheet

The recall layer: copy-paste commands + the most common error/fix
pairs. For deeper rationale, see the sibling `*.md` files.

> **Truly fresh agent?** Read
> [`yam_agent_on_startup.md`](yam_agent_on_startup.md) first — that's
> the minimum context-restoration checklist (paths, conventions,
> reading order). This FAQ assumes you already know what `T_lfr` is
> and where the code lives.

## Quickstart

| Thing | Where | One-liner |
|---|---|---|
| Code (canonical) | `/data/cameron/para/robot/yam/` on lab | edit directly with `Read`/`Edit`/`Write` |
| Code (mounted) | `~/lab/para/robot/yam/` on puget + russet | runs see lab updates instantly |
| Data, checkpoints, eval videos | `~/yam_para/{data,checkpoints,eval_videos}/` on puget | stays local for IO |
| DINOv3 weights | `~/yam_para/dinov3/weights/` on puget | reference via `DINO_WEIGHTS_PATH` env |
| Vault (this doc + siblings) | `/data/cameron/vault/para/robot/yam/` on lab | committed; pull/push via git |
| rd serve (on russet) | `~/lab/para/robot/yam/bin/yam-rd-serve` | respawn loop; one Ctrl-C kills serve, two breaks the loop |
| Mount lab on a rig | `~/lab/para/robot/yam/bin/yam-mount` | idempotent; `-u` to unmount |
| Reset all CAN | sudo `ip link set <iface> up type can bitrate 1000000` for each | password: `robot-lab` on russet |

## How do I run a live rerun session?

**Always use `LiveRerunSession`** — see `rerun-panels.md` for the full
table; this section is the copy-paste recipe.

**Run the canonical example** (live exo calibration, teleop, render
overlay on russet):

```bash
ssh robot-lab
source ~/cameron/raiden_fork.venv/bin/activate
python ~/lab/para/robot/yam/calibrate.py
# flags: --arm right --n_detections 10 --render_scale 1.0 --render_every 2 --rerun_port 9092
```

Tunnel from your Mac to view rerun:

```bash
ssh -L 9092:127.0.0.1:9092 -L 9093:127.0.0.1:9093 robot-lab
# then open http://localhost:9092 in your browser
# (127.0.0.1 LITERAL — `localhost` fails on russet's nsswitch)
```

Yellow leader top button = home + clean exit.

**Write a new live rerun script** — skeleton (put in
`/data/cameron/para/robot/yam/<your_script>.py` or `unit_tests/`):

```python
import sys, cv2
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))  # if in unit_tests/

from lib.rerun_session import LiveRerunSession
from lib.render import render_arm_with_exo
from lib.viz.overlay import compose_render_over_live

with LiveRerunSession("my_script", teleop_arm="right") as s:
    T_lock = s.lock_exo_pose(min_detections=10)  # optional; skip if no calib needed
    W, H = s.WH
    cache = {"render_bgr": None, "mask": None}

    def step(idx, bgr):
        if idx % 2 == 0 or cache["render_bgr"] is None:
            joints7 = s.follower_joints7()
            rgb, mask = render_arm_with_exo(joints7, T_lock, s.K, W, H)
            cache["render_bgr"] = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
            cache["mask"] = mask
        s.log("scene/rgb", bgr)
        s.log("scene/composite",
              compose_render_over_live(bgr, cache["render_bgr"], cache["mask"], 0.5))

    s.loop(step)   # blocks until yellow button / Ctrl-C
```

What the class bakes in (do not override unless you know why — each was
debugged the hard way on 2026-06-02):

- JPEG-encode all logs (`to_jpeg(target_w=960, quality=75)`) — raw HD1080
  at 15 fps swamps the gRPC channel.
- Single ZED `grab()` per iteration (no drain — drain *blocks* per call).
- `rr.set_time("step", sequence=frame_idx)` per iter — without it the
  web viewer freezes on frame 0.
- `rr.serve_grpc` + `rr.serve_web_viewer` (NOT `rr.serve_web` — doesn't
  exist in installed SDK).
- `MUJOCO_GL=egl` set before mujoco is imported (headless puget).
- Yellow leader top button polled at 20 Hz for clean exit.

## How do I record an episode?

```bash
ssh robot-lab
source ~/cameron/raiden_fork.venv/bin/activate
python ~/lab/para/robot/yam/record.py --name testing
# flags: --arm right --scene_save_wh 960 540 --wrist_save_wh 1280 720 --jpeg_quality 90
```

Yellow leader top button **toggles** recording:
- Script starts in ARMED state. Nothing is saved.
- 1st yellow press → start a new episode (next NNNN under
  `~/recordings/<name>/`), create dir, write `calibration_results.json`,
  begin saving frames.
- 2nd yellow press → stop, finalize, increment completed counter,
  return to ARMED.
- Repeat for as many episodes as you want; **Ctrl-C exits the session**.

**Wrist cam is off by default.** Add `--wrist` to also open + save the
wrist camera (lowdim PKL then includes wrist intrinsics + `wrist_save_wh`).

Rerun panels (drag into layout once, stays sticky):
- `scene/rgb`, `scene/composite`, `wrist/rgb` — live + overlay.
- `status` — text panel: `dataset / completed episodes / state /
  current episode / frames this episode`.

What it does on startup:
1. Open scene (HD1080) + wrist (HD720) ZEDs, start right-arm teleop, init rerun.
2. Pool 10 exo-aruco frames → lock scene cam pose for the whole session.
3. Wait in ARMED state for yellow.

Saved per frame (15 Hz): scene 960×540 JPEG q90, wrist 1280×720 JPEG q90,
lowdim PKL (joints, intrinsics @ save res, world-frame extrinsic, T_lfr,
timestamp). One `calibration_results.json` per episode with `success: True`.

Storage: ~135 KB/frame → ~60 MB / 30 s episode → ~30 GB / 500 episodes.

Output episode dir layout (matches the post-rd-convert PKL convention):

```
~/recordings/place_mug/0000/
├── calibration_results.json
├── rgb/
│   ├── scene_camera/0000000000.jpg, 0000000001.jpg, …
│   └── right_wrist_camera/0000000000.jpg, …
└── lowdim/0000000000.pkl, 0000000001.pkl, …
```

After recording, rsync to puget for training:

```bash
rsync -avh ~/recordings/place_mug/ dev:~/yam_para/data/place_mug/
```

## How do I visualize what the training input/output looks like?

To preview the **trajectory targets** the model will see for a given
`(stride, window_len)` setting — start-frame + robot mask + the
N keypoints sampled at `[start + 1*stride, …, start + window_len*stride]`:

```bash
ssh robot-lab
source ~/cameron/raiden_fork.venv/bin/activate
MUJOCO_GL=egl python ~/lab/para/robot/yam/vis_dataset.py \
    --name testing --window_len 16 --stride 8
```

Output filename convention is **`<ep>_training_<stride>_<window_len>.png`**:

```
unit_tests/output/dataset_viz/<name>/<ep>_training_<stride>_<window_len>.png
```

So you can drop a dataset + a (stride, window_len) pair and immediately
see what the model will be asked to predict. Browse:

```
https://omidlab.net/browse/yam_remote/lab/para/robot/yam/unit_tests/output/dataset_viz/<name>/
```

Match the `--frame_stride` and `--n_window` you pass to `train.py` to
preview the targets the trainer will use.

## How do I visualize a recorded dataset?

```bash
ssh robot-lab
source ~/cameron/raiden_fork.venv/bin/activate
MUJOCO_GL=egl python ~/lab/para/robot/yam/vis_dataset.py --name testing
# flags: --episodes 0000 0001 0002   (default: all)  --n_keypoints 30
```

Per episode: start-frame scene image + red robot mask (mujoco render at
start-frame joints) + rainbow EE-trajectory polyline (FK on every
saved frame's joints, sub-sampled to 30 evenly-spaced points, projected
to scene cam).

Output:
`/data/cameron/para/robot/yam/unit_tests/output/dataset_viz/<name>/NNNN.png`
→ browsable at
`https://omidlab.net/browse/yam_remote/lab/para/robot/yam/unit_tests/output/dataset_viz/<name>/`.

The viz functions used (`lib/viz/keypoints.py`,
`lib/viz/trajectory.py`, `lib/viz/overlay.py`) are the **same code
that training (wandb panels) + inference (rerun panels) call** — single
source of truth for the visual primitives.

## How do I train a model?

Scene-only DINOv3 volume model (no wrist, no xattn) — minimal MVP.
Reuses `lib/viz/{feature_pca,heatmap,keypoints,overlay,trajectory}.py`
so train + inference + dataset-viz all draw the same panels.

```bash
ssh dev
source ~/yam_para/.venv/bin/activate
cd ~/lab/para/robot/yam
CUDA_VISIBLE_DEVICES=0 nohup python train.py \
    --name <run_name> \
    --data_root ~/yam_para/data/<dataset> \
    > /tmp/train_<run_name>.log 2>&1 &
echo $! > /tmp/train_<run_name>.pid
```

Defaults: batch **40** (≈ 21 GB on a 24 GB GPU; lower if OOM),
lr 3e-4, IMG_SIZE 504, PRED_SIZE 56, N_WINDOW 8, N_HEIGHT_BINS 32,
feat_dim 48. Ckpts every 100 iters → atomic write to
`~/yam_para/checkpoints/<run_name>/latest.pth`. Viz panels every
200 iters → wandb (`yam_train_simple` project).

| Stage | Resolution |
|---|---|
| Recorded JPEG (source) | 960 × 540 |
| Model input | 405 × 405 (square; squish-resized) |
| DINOv3 patch grid | 45 × 45 |
| Bilinear upsample + 2-layer residual MLP | → 128 × 128 |
| Per-pixel feature `F` | 128 × 128 × 48 |
| Volume / heatmap logits | (B, T=16, Z=32, 128, 128) |
| Pred grid | 128 × 128 = `--pred_size` |

Architecture (single-arm, scene-only — no wrist, no xattn):

```
rgb (405) → DINOv3 → patch (45×45×384) + cls (384)
                              ↓ 1×1 conv → bilinear up → 2-layer residual MLP
              F_scene (128×128×48)        cls (384)
                ↑ sample at start_pix
              eef_feat (48)
                ⊕ cls → input_proj (d_model=256)
              q_in → 5× AdaLN-Zero blocks (time-conditioned sin/cos)
              penult (B, T, 256)
                ├── q_head      → (q_F, q_z, q_t)
                ├── grip_head   → (B, T, 32) bin logits
                └── rot_head    → (B, T, 64) K-means quaternion logits
              volume = q_F · F_scene + q_z · z_sin + q_t · t_sin
              losses: CE_volume (Z·P·P) + CE_grip + CE_rot
```

K-means rotation centroids (K=64) are **always saved inside every
checkpoint** as `ck["dataset_cfg"]["rot_centroids"]` (shape (64, 4)).
No external npz cache — refit at every dataset init is deterministic
(seed=0, ~30 s on ~1k quats), so we never have two copies that can
disagree. To resume training without refitting, pass `rot_centroids=`
from a loaded ckpt to `YamScene(...)`.

### Don't normalize the volume-head dot products

The bilinear scores (`q_F · F_scene`, `q_z · z_sin`, `q_t · t_sin`) use
**raw dot products** — no `F.normalize`, no temperature. With L2 normalization
all per-T softmaxes are bounded in [-1, 1] and can never sharpen as
training progresses; the heatmap stays flat over time and loss plateaus
high (~11 nats vs. the ~6 we get raw). Time conditioning via AdaLN-Zero
is fine and identical to the pre-refactor code; **the peakiness comes
from letting the dot-product magnitudes grow freely**. Caught
2026-06-02 in a minimal-v1 leftover.

The squish on input means pred-grid → source-image mapping uses
separate axis scales (`sx = W/P`, `sy = H/P`) — see `_viz_batch` for
the keypoint overlay.

Throughput target: ~2 it/s once the JPEG cache is warm.

Data convention: training reads what `record.py` writes
(`rgb/scene_camera/*.jpg` + `lowdim/*.pkl`). Each PKL must have
`intrinsics["scene_camera"]`, `extrinsics["scene_camera"]` (in world
= left_arm_base, `_scene_in_lbase=True`), and `T_left_from_right`.

Always rsync data russet → puget first:

```bash
ssh dev 'rsync -ah robot-lab@10.110.22.11:/home/robot-lab/recordings/<dataset>/ \
  ~/yam_para/data/<dataset>/'
```

Poll the run: `tail -f /tmp/train_<run_name>.log` or watch the wandb
URL printed at startup. Per agent rule, poll at 20 s / 1 min / 10 min
after launch for errors before walking away.

## How do I make paper figures?

Three scripts under `~/lab/para/robot/yam/`, all sharing the same
`lib/viz/*` primitives so colors and layouts stay consistent. Run on
**puget** (model + data live there).

**Across-episode comparison** (10 cols of evenly-spaced training episodes,
3 rows of RGB / shared-PCA features / predicted keypoints):

```bash
ssh dev
source ~/yam_para/.venv/bin/activate
python ~/lab/para/robot/yam/vis_paper_features.py \
    --ckpt ~/yam_para/checkpoints/<run>/latest.pth
# → unit_tests/output/paper_features/grid_rgb_pca_kp.png   (1600×480)
```

**Within-episode comparison** (10 cols are 10 frames sampled start→end of one episode):

```bash
python ~/lab/para/robot/yam/vis_paper_features_within_episode.py \
    --ckpt ~/yam_para/checkpoints/<run>/latest.pth --episode 0000
# → unit_tests/output/paper_features/grid_rgb_pca_kp_within_ep_0000.png
```

**Animated single episode** (MP4 — horizontal `RGB|PCA|KP` strip, every frame):

```bash
python ~/lab/para/robot/yam/vis_paper_features_video.py \
    --ckpt ~/yam_para/checkpoints/<run>/latest.pth --episode 0000
# → unit_tests/output/paper_features/video_within_ep_0000.mp4   (1800×600, ~20s @ 15fps)
```

**Animated 10 episodes in parallel** (MP4 — 3×10 grid synced across eps,
pad shorter eps with last frame):

```bash
python ~/lab/para/robot/yam/vis_paper_features_video.py \
    --ckpt ~/yam_para/checkpoints/<run>/latest.pth --n_episodes 10
# → unit_tests/output/paper_features/video_10_episodes.mp4   (1800×540)
```

All four use a **shared PCA basis** (`lib/viz/feature_pca.py`): single SVD
across all maps in the figure so the color scheme is comparable across
columns/episodes. For video, the basis is fit once on `--pca_n_samples`
(default 10) uniformly-spaced frames per episode and applied to every
frame thereafter.

Browse:
```
https://omidlab.net/browse/yam_remote/lab/para/robot/yam/unit_tests/output/paper_features/
```

## How do I make dataset-overview figures?

```bash
ssh dev
source ~/yam_para/.venv/bin/activate
python ~/lab/para/robot/yam/vis_dataset_grid.py
# → unit_tests/output/dataset_grid/grid_start_frames.png   (1200×540)
# → unit_tests/output/dataset_grid/dataset_vis_vid.mp4     (4×5 video grid)
```

Defaults: 4 rows × 5 cols of episode 0 frames from `place_mug_saucer_single_proper`. Pass `--data_root` for a different dataset.

## How do I save eval videos during deploy?

`deploy.py` has a sparse per-substep recorder (writes one frame after each
`robot.move_to` completes + the start frame of each rollout):

```bash
python ~/lab/para/robot/yam/deploy.py \
    --ckpt <volume_or_baseline_ckpt>/latest.pth \
    --save_video --eval_name ours_in_dist_pickplace
# Output: ~/cameron/eval_videos/ours_in_dist_pickplace_<YYYY-MM-DD_HHMMSS>/vid_001.mp4, vid_002.mp4, …
```

Pressing `h` rotates to the next vid file (one MP4 per rollout). The
on-screen `── rollout N ──` counter increments each time.

After a session, combine all rollouts into a grid MP4 for at-a-glance review:

```bash
ssh robot-lab
source ~/cameron/raiden_fork.venv/bin/activate
python ~/lab/para/robot/yam/vis_eval_grid.py
# Writes <session>_eval_grid_vid.mp4 in each session dir; auto-sized grid; skips <10-frame rollouts.
```

**Eval-video storage path** is on **russet** (`~/cameron/eval_videos/`,
russet-local since the puget sshfs mount at `~/cameron/puget/` is
read-only). Rsync to puget for archival:

```bash
ssh dev
rsync -avh robot-lab@russet:/home/robot-lab/cameron/eval_videos/ \
    ~/yam_para/backup_pickplace_eval_vids/
```

## How do I run live inference on the rig?

Live model-driven inference on **russet** (it has a 4090 — no chiral
roundtrip needed; direct raiden via the same `Robot.move_to` wrapper
that `control_speed_test.py` exercises).

```bash
ssh robot-lab
source ~/cameron/raiden_fork.venv/bin/activate
python ~/lab/para/robot/yam/deploy.py \
    --ckpt ~/cameron/puget/checkpoints/<run>/latest.pth \
    [--device cpu|cuda]   # auto: cuda if available, else cpu
```

### Each iteration

1. Grab a fresh scene RGB + read the current 7-DoF right-arm joints.
2. FK + project current EE → `start_pix` (model input).
3. Forward model → `volume_logits + grip_logits + rot_logits`.
4. Decode each head (argmax) + ray-unproject pixel @ predicted-height
   plane → world (x, y, z); K-means centroid lookup → quaternion;
   gripper bin → value (via `lib.inference.decode`).
5. **mink-IK** per future timestep (seeded chain) → `(T, 7)` joint
   commands (`lib.ik.ee_chain_to_joints`).
6. Log rerun panels (port 9092):
   - `scene/rgb` — live scene image
   - `scene/composite` — mujoco arm @ current joints, alpha-blended (calib check)
   - `scene/mask_overlay` — red silhouette over live image
   - `pred/kp` — rainbow filled pred keypoints on 405-input view (no GT shown — inference)
   - `pred/heatmap_grid` — per-T marginal heatmap with RGB underneath
   - `pred/heatmap_grid_raw` — per-T marginal heatmap, pure jet (no RGB)
   - `pred/pca_dino_mlp` — PCA of refined features
   - `status` — text panel with pred summary
7. Prompt `y/q/h/r/c`:
   - **y** — solve IK chain + run `Robot.move_to` per command in sequence
   - **r** — re-query (grab + forward again on fresh obs)
   - **h** — home (move to startup-time joint state)
   - **c** — re-pool aruco + relock the cam pose
   - **q** — home + quit

### Setup gotchas (one-time on russet)

- `pip install torchvision` (via `~/.local/bin/uv pip install torchvision`
  — the venv lacks pip itself; uv handles the install).
- DINOv3 weights are mounted from puget via `~/cameron/puget/dinov3/weights/`.
  `deploy.py` auto-sets `DINO_WEIGHTS_PATH` to that path if it exists.
- **Per-rig torch matrix** — the venv ships with torch 2.12.0+cu130
  but the right reinstall depends on the rig's GPU + driver. **Verify
  with a real cuda matmul** (`x @ x.T`), not just
  `torch.cuda.is_available()` — the latter returns True even when no
  compatible kernels are present.

  | Rig | GPU | sm | Driver | Right torch |
  |---|---|---|---|---|
  | russet | RTX 4090 (Ada) | sm_89 | 570 (CUDA ≤ 12.x) | **2.6.0+cu126** — stock cu130 silently falls back to CPU |
  | yukon | RTX PRO 6000 (Blackwell) | sm_120 | 595 (CUDA 13.2) | **2.12.0+cu130** (stock) — cu126 errors `no kernel image available for execution on the device` |

  Install commands:

  ```bash
  # russet
  ~/.local/bin/uv pip install --reinstall "torch==2.6.0" "torchvision==0.21.0" \
      --index-url https://download.pytorch.org/whl/cu126

  # yukon (default after fresh venv install; reinstall if you wiped it)
  ~/.local/bin/uv pip install --reinstall "torch==2.12.0" torchvision \
      --index-url https://download.pytorch.org/whl/cu130
  ```

  Verify with:
  ```python
  import torch
  x = torch.randn(64, 64, device="cuda"); y = x @ x.T
  print(torch.__version__, torch.cuda.get_device_capability(0), float(y.sum()))
  ```
  If `y.sum()` returns a number, the kernel ran. If it raises
  `no kernel image is available for execution on the device`, the
  wheels don't match the GPU's compute capability.

  After this fix, `--device` defaults to cuda; forward ≈ 50-100 ms.

## How do I run inference offline (unpack saved predictions)?

Every checkpoint saved by `train.py` carries the full decoding payload:
the K-means quaternion centroids (K, 4), gripper bin range, height bin
range, all hyper-params. Use `lib/inference.py`:

```python
from lib.inference import load_checkpoint, decode

ck = load_checkpoint("~/yam_para/checkpoints/<run>/latest.pth", device="cuda")
pred = decode(ck["model"], rgb, start_pix, ck["cfg"])
# pred["pix_uv"]   (T, 2)  pixel coords in 405-input space
# pred["height_m"] (T,)    EE Z in meters (world = left_arm_base)
# pred["grip"]     (T,)    gripper value in original units
# pred["quat"]     (T, 4)  EE quaternion (w, x, y, z) from K-means centroid
```

No external file needed — centroids ride with the checkpoint.

## How do I run the unit tests?

| Test | Command | Notes |
|---|---|---|
| Control speed | `python ~/lab/para/robot/yam/unit_tests/control_speed_test.py` (on russet, raiden venv) | Loads `recordings/test_record_*.npz`, chains `Robot.move_to`, reports planned-vs-actual rad/s |
| Calibration | `MUJOCO_GL=egl python ~/lab/para/robot/yam/unit_tests/calibration_test.py [--frame N]` (on puget, yam_para venv) | Offline aruco + render + composite PNG to `unit_tests/output/`; browsable at `https://omidlab.net/browse/yam_remote/lab/para/robot/yam/unit_tests/output/` |

## How do I bring up the rig from scratch?

```bash
# 1. Reset CAN buses (russet, sudo)
ssh robot-lab
echo robot-lab | sudo -S bash -c \
  'for i in can_leader_l can_follower_l can_follower_r can_leader_r; do
     ip link set $i down; ip link set $i up type can bitrate 1000000;
   done'
ip -br link show type can     # all 4 should be UP

# 2. Start rd serve with auto-respawn (russet, raiden venv)
source ~/cameron/raiden_fork.venv/bin/activate
~/lab/para/robot/yam/bin/yam-rd-serve     # respawn loop; survives client estops

# 3. Run deploy / calibrate / training from puget or russet as needed.
```

If a rig is brand-new (lab mount not set up):

```bash
# from lab: ensure the bin script exists
cat /data/cameron/para/robot/yam/bin/yam-mount
# from the rig (puget or russet):
mkdir -p ~/lab && sshfs cameronsmith@100.74.71.38:/data/cameron ~/lab \
  -o reconnect,ServerAliveInterval=15,follow_symlinks,cache=yes,kernel_cache
# tailscale SSH must already be approved for that rig
```

## What common errors look like, and the fix

| Symptom | Cause | Fix |
|---|---|---|
| `re_grpc_server: Sender has been blocked …` warnings, web viewer freezes | Logging raw HD1080 RGB — too much bandwidth | Use `to_jpeg` (auto in `LiveRerunSession.log`) |
| Web viewer stuck on frame 0 | Missing `rr.set_time` | Use `LiveRerunSession.loop` (sets timeline automatically) |
| `module 'rerun' has no attribute 'serve_web'` | Wrong API; SDK uses two-call pattern | Use `lib.rerun_helpers.init_web` (calls `serve_grpc` + `serve_web_viewer`) |
| `Missing or unavailable CAN interfaces: can_leader_r` | CAN bus is DOWN | Reset CAN (see above) |
| `fail to communicate with the motor N on … can_leader_r` (single motor drops mid-run) | Hardware-side bus fault on that arm's CAN chain | Power-cycle the arm; wiggle CAN connectors; reseat USB-CAN adapter |
| `CAMERA MOTION SENSORS NOT DETECTED` (ZED 2i open fails) | ZED 2i wedge state after Ctrl-C or `Killed` | Wait 10-30 s and retry; if persists, physically unplug 30 s |
| `cv2.aruco has no attribute detectMarkers` | Missing opencv-contrib | `uv pip install --force-reinstall opencv-contrib-python==4.13.0.92` |
| Workspace mask collapses from ~92k → ~30k voxels | `T_lfr` not applied to scene extrinsic | Hot-patch PKLs (see `conversion.md`); for live, `to_world(T_cam_in_rbase, T_lfr)` |
| Aruco solve gives nonsense pose | Forgot the `link_to_aruco_transform` final compose | Use `lib.calibration.solve_exo_aruco` (chains do_est → T_link_in_cam → invert) |
| `rd serve` died mid-deploy | Client disconnect triggers safety estop; intentional | Use `yam-rd-serve` respawn wrapper |

## Where do I commit?

- `/data/cameron/vault/` — local git, **no remote configured**. Just
  `git commit` here; no push. Cameron syncs via Obsidian Git on his Mac.
- `/data/cameron/agents_stuff/` — github remote (`cameronosmith/agents_stuff`).
  Commit locally; `git push` needs Cameron's GitHub token.
- `/data/cameron/para/` — github remote (`cameronosmith/para`). Same:
  commit locally, Cameron pushes.

After any non-trivial change: update the relevant `vault/.../tasks.md`,
`overview.md`, and append a dated note to `memory.md` (project) or
`vault/fleet/agents/yams/memory.md` (agent-level).

## After launching anything on russet — poll for 30 s

Don't fire-and-forget russet launches. Most failures (ZED wedge, CAN
dropout, raiden import error, missing file) surface within the first
~30 s. Watch the pane:

```bash
tmux send-keys -t agents:6 'python …' Enter
for i in 1 2 3; do
  sleep 10
  out=$(tmux capture-pane -t agents:6 -p | tail -8)
  echo "$out" | grep -qE "Traceback|fail to communicate|MOTION SENSORS|RuntimeError|FileNotFoundError" && { echo "FAIL"; break; }
  echo "$out" | grep -qE "teleop is live|locked\\." && { echo "OK"; break; }
done
```

After 30 s of clean output: treat as healthy.

## I just came back — what's running?

Quick orient:

```bash
tmux ls                           # see active panes
tmux capture-pane -t agents:5 -p | tail   # puget pane (training / deploy)
tmux capture-pane -t agents:6 -p | tail   # russet pane (rd serve / calibrate)
ip -br link show type can         # (on russet) CAN health
ls ~/yam_para/checkpoints/        # (on puget) current runs
```

Then read this file's "Quickstart" + `overview.md` to confirm the
current SOTA + active workstream.

## See also

- [`rerun-panels.md`](rerun-panels.md) — full `LiveRerunSession` pattern table
- [`memory.md`](memory.md) — technical conventions + code pointers
- [`known-bugs.md`](known-bugs.md) — recovery procedures for each failure mode
- [`overview.md`](overview.md) — current state + tree layout
- [`INDEX.md`](INDEX.md) — implemented vs TODO files
- [`control.md`](control.md) — robot control + chiral + teleop
- [`calibration.md`](calibration.md) — exo aruco solve + T_lfr
- [`rendering.md`](rendering.md) — mujoco render + principal-point warp
- [`recording.md`](recording.md) — data collection procedure