# YAM — rerun panels + canonical streaming pattern

Two parts:
1. **`LiveRerunSession`** — the canonical scaffold for any live rerun
   script (calibration, recording overlay, inference overlay, testing).
2. **Panels** logged by deploy / training viz / calibrate.

## Canonical pattern: `LiveRerunSession`

All live rerun scripts use the proven scaffold at
`/data/cameron/para/robot/yam/lib/rerun_session.py`. It bakes in every
debugging milestone from the calibrate.py development (2026-06-02):

| Setting | Default | Why |
|---|---|---|
| ZED resolution | HD1080 | matches training data |
| ZED grab | single `cam.grab()` | ZED keeps only the freshest frame; drain loops *block* per grab and ruin latency |
| Rerun transport | `serve_grpc` + `serve_web_viewer` | matches raiden's working pattern |
| Image encoding | JPEG via `to_jpeg(target_w=960, quality=75)` | raw HD1080 RGB at 15 fps swamps the gRPC channel (~280 MB/s); JPEG is ~50× lighter; without it `re_grpc_server` channel-backlog warnings freeze the viewer |
| Rerun timeline | `rr.set_time("step", sequence=frame_idx)` per iter | without this the web viewer freezes on frame 0 |
| Teleop exit | yellow leader top button, polled at 20 Hz | `rc.check_button_press() is not None` |
| Render rate | render every other frame, cache between | render is the dominant cost; live image still logs every frame |
| Render resolution | full live res by default | half-res + upscale also supported via `--render_scale` |
| MUJOCO_GL | egl (headless) | set in render.py + rerun_session.py before mujoco import |

Skeleton (see `calibrate.py` for the full canonical example):

```python
from lib.rerun_session import LiveRerunSession

with LiveRerunSession("my_session", teleop_arm="right") as s:
    T_lock = s.lock_exo_pose(min_detections=10)
    W, H = s.WH

    cache = {"render_bgr": None, "mask": None}
    def step(idx, bgr):
        if idx % 2 == 0 or cache["render_bgr"] is None:
            joints7 = s.follower_joints7()
            rgb, mask = render_arm_with_exo(joints7, T_lock, s.K, W, H)
            cache["render_bgr"] = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
            cache["mask"] = mask
        s.log("scene/rgb", bgr)                                     # JPEG-encoded auto
        s.log("scene/composite",
              compose_render_over_live(bgr, cache["render_bgr"], cache["mask"], 0.5))

    s.loop(step)         # blocks until yellow button or Ctrl-C
```

The class' public surface (see `lib/rerun_session.py` for full docstrings):

| Method | Returns / Use |
|---|---|
| `__enter__` / `__exit__` | open ZED, start teleop (if `teleop_arm`), init rerun, then tear down |
| `s.grab()` | latest scene BGR (single grab; ZED's buffer holds latest) |
| `s.follower_joints7()` | right or left follower 7-DoF; raises if no teleop |
| `s.yellow_pressed()` | True on yellow leader top button (False if no teleop) |
| `s.lock_exo_pose(min_detections, max_wait_s)` | pool aruco frames → averaged `T_cam_in_right_arm_base` |
| `s.log(path, bgr)` | JPEG-encode + log at current timeline step |
| `s.loop(callback)` | bg thread runs `callback(idx, bgr)`; main thread polls exit |
| `s.K`, `s.WH` | scene intrinsics + (W, H) at the live res |

**Use this class for any new live rerun script.** Recording, deploy
inference, calibration tests — all share the same scaffold. Don't
re-implement the ZED grab + teleop + rerun init dance from scratch.

## Panels

What we log into rerun during deploy (and the equivalents during
training viz). Includes panel paths, what they show, and the SSH tunnel
setup.

## Tunnel from your Mac

```
ssh -L 9094:127.0.0.1:9094 -L 9095:127.0.0.1:9095 dev
```

**Use `127.0.0.1` literally** — `localhost` fails to resolve on
russet's nsswitch when the tunnel is forwarded onward. Same rule for
russet-side tunnels.

Then open `http://localhost:9094` in your browser. The deploy script
prints the full URL on startup, including the `?url=rerun%2B...` query
fragment that auto-connects.

Default ports:

| Service | Port | Notes |
|---|---|---|
| rerun web (UI) | 9094 | 9092 is the rerun-stock default; often taken by another user on puget |
| rerun grpc | 9095 | The actual data stream proxied by the web UI |

## Panels logged during deploy

| Path | Content |
|---|---|
| `right_scene_camera/rgb` | Live scene RGB with predicted trajectory: 30 rainbow points + connecting line strip |
| `right_wrist_camera/rgb` | Live wrist RGB with the *projected* scene trajectory overlaid (uses live `T_world_to_wrist`) |
| `cam_check/rgb` | Exo+arm mujoco render composited 50/50 over the live scene frame — visual calib sanity (off when `--no_render`) |
| `cam_check/mask` | Just the mujoco segmentation mask (debugging the overlay) |
| `features/dino_pca_scene` | First-3-PC PCA of scene patch features → RGB |
| `features/dino_pca_wrist` | First-3-PC PCA of wrist patch features |
| `bins/scene_heatmap` | Per-(t) marginal YX heatmap from the factorized volume head |
| `bins/wrist_heatmap` | Same for wrist; an extra bar shows abstain probability mass |
| `bins/keypoints_scene` | 56×56 grid of scene log-probs as a heatmap |
| `bins/keypoints_wrist` | Same for wrist |
| `scalars/...` | Per-step scalars: pix, z_bin, gripper, rot_bin, abstain_flag |

The trajectory points come from each predicted timestep `t` in the
chunk; the line strip connects consecutive `t`. Color = rainbow by `t`
so you can see the temporal ordering.

## Panel layout

Rerun caches panel layout in the browser tab. After adding new panels
(or fixing log paths), **refresh the tab** — the old layout will hide
the new panels.

## Wrist-cam projection

For the `right_wrist_camera/rgb` overlay:

1. Sample the predicted 3D point `p_world` per timestep.
2. Project via the live `T_world_to_wrist` (= inverse of per-frame
   `T_wrist_in_world` from chiral obs).
3. Project pixel via `K_wrist`.
4. Log as 2D point on `right_wrist_camera/rgb`.

Out-of-frustum (projected x or y outside `[0, W)` / `[0, H)`) — clip and
mark with a different color (typically gray) so abstain frames are
visually distinct.

## During training (wandb)

The same panel-construction code is shared between deploy (rerun) and
training (wandb). See `training.md` for the wandb panel names — they
mirror the rerun paths but flattened (`viz/scene_keypoints` ←
`bins/keypoints_scene`).

## Common bugs

| Symptom | Fix |
|---|---|
| All panels empty | Check rerun-grpc port (`9095`); may have collided with another user — set `--vis_port` |
| Wrist RGB panel missing but scene works | Confirm `--use_wrist 1` and the obs has the wrist cam key |
| PCA panel is grayscale-noise | Forgot to z-score the patch features before PCA |
| Heatmap is uniformly bright | softmax stays under abstain (wrist) — check `wrist/pred_abstain_frac` |
| Cam_check overlay misaligned by ~10 px | Principal-point warp missing — see `rendering.md` |
| Cam_check overlay tiny | K not scaled to render res — see `rendering.md` |

## Disabling panels (for performance)

Most overhead is the wrist-projection panel and the cam_check render.
Quickest minimal session: `--no_render --use_wrist 0`. With wrist on +
no_render, the run is still ~5-10% slower than scene-only — usually
fine.