# AR-view training data — visualizations + episode annotations

Everything a downstream training agent needs to understand the AR-view (feetech UMI)
datasets: the raw layout, the episode start/end annotations, and the rendered
overlay visualization that ships alongside for eyeballing quality.

## What's here vs. what's referenced

| topic | this doc | see also |
|---|---|---|
| raw dataset layout, `state/NNNNNN.npz` schema, `meta.json` intrinsics | brief pointer | [`DATASET_FORMAT.md`](./DATASET_FORMAT.md) |
| episode start/end annotations + how they're derived | **full spec** | — |
| how the MuJoCo mesh overlays are rendered (Layer API, board loop-closure, camera math) | brief pointer | [`RENDER_OVERLAY.md`](./RENDER_OVERLAY.md) |
| what the *specific* per-frame composite in these datasets shows | **full spec** | this doc |
| robot XML (UMI gripper mesh, calib board) | brief pointer | [`RENDER_OVERLAY.md`](./RENDER_OVERLAY.md), meshes in `mj_umi_v2/` and `calib_board_model/` |

## Dataset locations

- **Raw data (source of truth)** — puget: `~/ar_datasets/<task>/` (rsync'd from the Mac). Layout in [`DATASET_FORMAT.md`](./DATASET_FORMAT.md).
- **Fleet sshfs mount** — omid-fleet: `/data/cameron/puget_ar_datasets/<task>/` (same tree, network-mounted).
- **Baked overlay composites (viewer + this doc's viz)** — omid-fleet: `/data/cameron/ar_renders/<task>/frames/f000000.jpg …`.
- **Web viewer** — the whole thing is browsable + scrubbable at `https://omidlab.net/data_viewer` under the **AR-view (feetech UMI)** tab.

## Episode start/end annotations

### Where they live

Alongside the frames the viewer serves:
```
/data/cameron/ar_renders/<task>/frames/episodes.json
```

For tasks that haven't been re-baked to the overlay composites yet, the same file lives at:
```
/data/cameron/puget_ar_datasets/<task>/scene_wrist/episodes.json
```

The scrubber's annotation editor is the source of truth; the JSON is written atomically by the viewer's `POST /api/data_viewer/annotations` endpoint. Manual edits are safe as long as the schema is preserved.

### Schema

```jsonc
{
  "version": 1,
  "source_path": "ar_renders/testing_cube_in_bowl/frames",   // browse-relative
  "total_frames": 272,
  "fps": 4.0,
  "episodes": [
    {
      "id": "ep_auto_mrjt0q63_0",     // opaque string, unique within the file
      "start": 28,                     // inclusive frame index
      "end": 59,                       // inclusive frame index (>= start, < total_frames)
      "label": "",                     // free-text; empty by default
      "notes": "",                     // free-text
      "keyframes": [                   // optional sparse annotations inside the episode
        { "id": "kf_...", "frame": 42, "label": "grasp" }
      ]
    }
    // ...
  ],
  "updated_at": "2026-07-13T22:48:11+00:00"
}
```

Episodes are sorted by `start` on write. Keyframes are sorted by `frame` and constrained to `start ≤ frame ≤ end`; any keyframe outside the range is dropped when the episode range is edited.

### How the boundaries were determined

Two-stage: **auto-detect from gripper threshold → hand-correction as needed.**

1. **Gripper-based auto-detect** (the scrubber's "Gripper auto-detect" panel).
   - Reads `gripper.json` (sibling of the frames dir; see below).
   - Finds every frame where `gripper_rad` crosses a user-chosen threshold (default **−0.8 rad** on this rig; more negative = jaw more closed).
   - Assigns alternating **Start / End** roles from a chosen base phase (`Start-first` by default → first crossing = episode start, second = episode end, third = next start, …).
   - Pairs consecutive `(Start, End)` into episodes. An open trailing `Start` becomes an episode ending at `total_frames − 1`; an unmatched trailing `End` is dropped.

2. **Manual overrides** (viewer UI):
   - Clicking any boundary tick flips its role **and cascades the flip through every subsequent boundary** to preserve alternation from the corrected point onward. This handles the "the auto-detected S₃ was actually an End" case in one click.
   - Each episode card has directly editable `start` / `end` frame inputs plus **set start / set end** buttons that snap to the playhead. Use these when the gripper didn't cleanly cross the threshold at the true event (`ar_testing_cube_in_bowl` has a couple of episodes that needed this).
   - Sparse keyframes (`K` in the scrubber) mark events *inside* an episode (grasp, release, etc.).

Downstream: `episodes[*].start` and `episodes[*].end` are the ground-truth demonstration segments. Frames outside any episode are "between demos" (approach/retract/idle).

## The overlay composite visualization

Each frame in `/data/cameron/ar_renders/<task>/frames/f<N>.jpg` is a **1500×289 horizontal 3-panel composite** produced by `/data/cameron/claude_feetech_controller/ar_view/render_dataset_frame.py`.

### Panels (left → right)

```
┌────────────────────────────────┬────────────────────────────────┬─────────────────────────────────┐
│                                │                                │                                 │
│  1. SCENE VIEW                 │  2. WRIST VIEW                 │  3. FUTURE-30 (scene view)      │
│                                │                                │                                 │
│  real scene image              │  real wrist image              │  real scene image               │
│  + calib board mesh  (cyan)    │  + calib board mesh  (cyan)    │  + next-30 UMI keypoints        │
│  + UMI gripper mesh  (magenta) │  + UMI gripper mesh  (magenta, │    (yellow→orange polyline)     │
│  + gripper jaw posed from      │    chained via constant        │  + rotation axes triads at each │
│    state.gripper_rad           │    wrist_cam→umi transform)    │    keypoint  (X=red Y=green     │
│                                │                                │    Z=blue)                      │
│                                │                                │                                 │
└────────────────────────────────┴────────────────────────────────┴─────────────────────────────────┘
```

Text label on each panel shows the frame index and visibility flags (`sv/wv/uv` = scene / wrist / UMI board seen this frame). `grip=<rad>` on the scene panel is the calibrated jaw angle from `state.gripper_rad`.

### Semantics

- **Cyan silhouette** = the calibration board mesh, rendered from that camera's pose (world = calib board `GridBoard` frame). If the outline lands on top of the real board, PnP + intrinsics are healthy.
- **Magenta silhouette** = the UMI gripper mesh, rendered with `data.qpos[jaw] = state.gripper_rad`. On the scene panel it's placed at `umi_pose(f)`; on the wrist panel it's placed at the constant `T_wrist_cam→umi` (locked in — see below).
- **Future keypoints** = the origin (translation) of `umi_pose(f+1..f+30)` projected into panel 1's scene camera. Color fades from yellow at the current frame toward orange at the horizon. A **dim ring** around a keypoint means `umi_visible=False` at that frame (`umi_pose` is reusing the last observed value; not a real measurement).
- **Rotation triads** at each future keypoint are the world-frame X/Y/Z axes of `umi_pose(f)` drawn 2 cm long. Line width and brightness fade with horizon distance so "now" is bright and "30 frames from now" is dim.

### `wrist_cam → umi` calibration (frozen)

The wrist camera is rigidly mounted on the UMI gripper, so the transform between them is constant. This isn't stored in the raw dataset. It was recovered post-hoc by:

1. Filtering every frame where `scene_visible ∧ wrist_visible ∧ umi_visible` (188 frames in `testing_cube_in_bowl`).
2. At each: `T_wrist_cam→umi(f) = inv(umi_pose(f)) @ wrist_cam_pose(f)`.
3. MAD-filtering translation outliers (kept 180/188).
4. Chordal-averaging rotations via SVD; mean translation.

Residual scatter across the 180 frames: **translation 3.3 mm, rotation 0.56°.** Persisted at `/data/cameron/claude_feetech_controller/ar_view/wrist_umi_calibration.json`. Delete + rerun with `--force-calib` to re-derive (e.g. after re-mounting the wrist camera).

### `gripper.json` sidecar

Also at `/data/cameron/ar_renders/<task>/gripper.json`. This is what powers the auto-detect UI and any downstream policy that wants gripper state without opening npzs one at a time.

```json
{
  "num_frames": 272,
  "gripper_rad": [-1.356, -1.350, ..., -1.727],   // one per frame; null if the servo read failed
  "umi_visible": [true, true, ..., true]           // one per frame; True means UMI board observed this frame
}
```

## MuJoCo details

For the mesh-overlay mechanics, board loop-closure, camera conventions, and where the model XMLs live — read [`RENDER_OVERLAY.md`](./RENDER_OVERLAY.md). What this pipeline adds on top of what that doc describes:

- Skips the PnP round-trip. `render_dataset_frame.py` uses the poses already stored in `state/NNNNNN.npz`, so the sim camera is driven by `layer.T_board_w @ (composed real poses)` instead of `layer.T_board_w @ inv(solve_pose(...))`.
- Hides the sim's ground/checker plane by setting `rgba[3]=0` on every plane-type geom in the loaded model — the alpha compositor keys off of black pixels for the "no sim here" mask.
- Poses the UMI `jaw` joint qpos from `state.gripper_rad` before rendering.
- Adds the future-trajectory + rotation-triads projection (both use the same `scene_cam_pose` and `K_save` from `meta.json`).
- Multi-frame chordal averaging of `wrist_cam → umi` (see above).

## Reproducing the composites

Prereqs on omid-fleet (already installed):
- `MUJOCO_GL=osmesa` — headless renderer (`libEGL/libOSMesa` from apt).
- `/data/cameron/para/.agents/reports/.venv/bin/python` — venv with `mujoco 3.10`, `opencv-python-headless`, numpy, PIL.

Single frame → 4 individual PNGs + a 2×2 grid:
```
MUJOCO_GL=osmesa /data/cameron/para/.agents/reports/.venv/bin/python \
  /data/cameron/claude_feetech_controller/ar_view/render_dataset_frame.py \
  --dataset-root /data/cameron/puget_ar_datasets/<task> \
  --frame 28 \
  --out /data/cameron/ar_renders/<task>
```

Batch (all frames) → one horizontal-stack JPEG per frame, viewer-ready:
```
MUJOCO_GL=osmesa /data/cameron/para/.agents/reports/.venv/bin/python \
  /data/cameron/claude_feetech_controller/ar_view/render_dataset_frame.py \
  --dataset-root /data/cameron/puget_ar_datasets/<task> \
  --frames all \
  --out /data/cameron/ar_renders/<task>/frames \
  --name-fmt "f{frame:06d}.jpg"
```

Then wire it into the viewer by (a) copying the source dataset's `episodes.json` and `gripper.json` into the new render tree and (b) editing `/data/cameron/para/.agents/reports/data_viewer/datasets.json` to point the task's `path` at the new frames dir.

Bake speed is ~1.7 s/frame on the fleet (dominated by loading 30 future-window npzs per frame).

## What to train on

Given the above, a typical training pipeline consumes:

1. `state/NNNNNN.npz` for the ground-truth per-frame poses + gripper state.
2. `meta.json` for the pinhole intrinsics (use `K_save` since it's matched to the 500 px `scene/` and `wrist/` JPEGs; **not** `K_native`).
3. `episodes.json` for demonstration segmentation.
4. The `scene/` and `wrist/` JPEGs directly for image input — **don't** use the rendered composites as image input, those are for humans to eyeball. The composites still work as scrubber previews.
5. The frozen `T_wrist_cam→umi` transform if you want a stable wrist-cam pose even on frames where the wrist board was momentarily lost (compose `umi_pose(f) @ inv(T_wrist_cam→umi)` — mind the direction of the composition).