# YAM — evaluation How the rollout-annotation reports work, what experiments live in them, and what we read off the bar charts. Updated whenever a new report subdir lands under `/data/cameron/para/robot/yam/unit_tests/output/eval_rollout_pages/`. ## Report hub - **URL**: - **On disk**: `/data/cameron/para/robot/yam/unit_tests/output/eval_rollout_pages/` - `index.html` — hub, auto-rebuilt by the latest report generator. - `/index.html` — one self-contained report per experiment. - `/charts/*.png` — bar charts embedded by the report. The hub is rebuilt at the bottom of `code/eval_rollout_pages_volume_based.py` (or `code/eval_rollout_pages.py` for the older data-scaling sweep). Both scan the same disk layout — adding a new report is *write a generator script that drops `/index.html`, then re-run the latest hub rebuilder*. No central REPORTS list to keep in sync. ## Annotation flow Each eval-session directory under `yukon:~/cameron/eval_videos/` carries: - `rollout_NNN/chunk_MMM/frame_pre.jpg` — input image at decision moment - `rollout_NNN/chunk_MMM/predictions.npz` — decoded xyz/quat/grip/bins + grip/rot logits - `rollout_NNN/chunk_MMM/patch_features.npy` — primary-view DINO patches (fp16) - `rollout_NNN/chunk_MMM/camera_pose.npy` — T_cam_in_rbase (4×4) - `rollout_NNN/chunk_MMM/substep_NN.jpg` — frame after each executed substep - `vid_NNN.mp4` — the full rollout video (one MP4 per `h` press) - `_annotations.json` — outcome labels keyed by `rollout_NNN`, written by the `/api/annotate` POST endpoint on `serve.py`. Annotations are POSTed from the per-report HTML's dropdown and persist across page reloads (the `rehydrate()` fix from when annotations were getting clobbered by auto-refresh). ### Outcome taxonomy (8 categories) | key | label | meaning | |---|---|---| | `pick_and_place` | pick and place | Full success. | | `pick_wrong_orientation_place` | pick + wrong-orientation place | Picked but tipped over or wrong roll. | | `pick_wrong_position_place` | pick + wrong-position place | Picked but placed off-target. | | `pick_and_drop` | pick and drop | Picked, dropped mid-transit (gripper opened early). | | `slipped_pick_touched` | slipped pick / touched | Contact but no grasp. | | `slight_miss` | hover / slight miss | Within a few cm of target, no contact. | | `no_touch` | medium miss | Stayed in workspace, wrong area. | | `wrong_direction` | wrong direction / way off | No real attempt at the target. | Colors are fixed across reports (`OUTCOME_COLORS` in `code/eval_rollout_pages.py`) so the bar charts read consistently: green = success, ambers / orange = partials, red = miss, grey = small miss, purple = touched-only. ## Active reports ### `volume_based_vs_baseline_2026-06-10` (current canonical) - **Models compared**: 2 - `volume_based` — `DinoVolumeSceneVolumetricMultiView` with cross-view fusion, per-voxel grip/rot (flat MLP temporal head), past_n=8 (grip, z) proprio conditioning. - `baseline` — `DinoCLSXYZBaselineMultiView` with the same DINOv3 backbone + DA3-style cross-view attention + concat-CLS, but flat XYZ regression head + CE grip/rot. - **Conditions**: 3 - `in_dist` — cup at training-distribution positions. - `ood_objpos` — cup at OOD positions, BEFORE the 5-ep finetune. - `ood_finetuned` — same OOD positions, AFTER the 2k-iter finetune from a 5-ep OOD recording. - **Training data**: `pickplace_redo_slow` (22 deliberate-release teleop episodes recorded 2026-06-10; the corrective re-record after we found zero release-event supervision in the earlier 50-ep cup dataset). - **Finetune data**: `5_ep_ood_finetune_collection` (5 episodes). - **Report**: - **Eval-dir prefix mapping** (matches the `--eval_name` flag passed to `deploy_with_action_chunking.py`): | model | condition | eval-dir prefix | |---|---|---| | volume_based | in_dist | `pickplace_redo_slow_pastn8_indist_*` | | volume_based | ood_objpos | `pickplace_redo_slow_pastn8_ood_objpos_*` | | volume_based | ood_finetuned | `pickplace_redo_slow_pastn8_finetuned_ood_*` | | baseline | in_dist | `baseline_in_dist_*` | | baseline | ood_objpos | `baseline_ood_*` | | baseline | ood_finetuned | `baseline_finetuned_ood_*` (queued) | To regenerate after new annotations land: ```bash cd /data/cameron/para/robot/yam python code/eval_rollout_pages_volume_based.py ``` ### `data_scaling_flat_mlp_2026-06-08` (archived) The earlier (20/40/60/80/100ep) × (ours flat-MLP / baseline) × (in_dist / ood_obj) sweep on `cup_move_100`. Generated by `code/eval_rollout_pages.py`. Conclusion: flat-MLP dominated baseline at small subsets; the gap closed at 100ep. Finetune-on-5ep-OOD rows were excluded because 5 episodes wasn't enough to anchor a finetune in that older recipe (no past_n proprio, no deliberate-release recordings). ## Bar-chart reading guide Every per-condition chart shows stacked outcome counts per model, with `n=…` above each stack = total annotated rollouts in that cell. Same outcome colors across all reports. When reading these: 1. **Compare *stack height* not *bar height***. The `n=` totals are rarely identical across models (annotators get tired); always check `n` before drawing a conclusion. If `n` differs by >2× between bars, normalize before comparing. 2. **Green-share** (`pick_and_place / n`) is the headline success rate. 3. **Amber-share** (`pick_wrong_orientation_place + pick_wrong_position_place + pick_and_drop`) is the "model did the right intent but failed execution" share — usually rotation, IK, or release-timing trouble. Cameron's deliberate-release re-record specifically targeted shrinking the `pick_and_drop` slice. 4. **Purple-share** (`slipped_pick_touched`) is interesting because the model gets credit for spatial accuracy — it reached the cup — but missed the grasp. Often a wrist-rotation issue. 5. **Red / dark-red share** (`no_touch + wrong_direction`) is the catastrophic failure share. On OOD, this is the share the 5-ep-OOD-finetune is supposed to claw back. Cross-report observations to log (write here as new reports land): - *2026-06-10 — pending — fill in after annotation pass.* ## Adding a new report 1. Drop a generator at `code/eval_rollout_pages_.py`. Easiest path: copy the latest one and adjust the `MODEL_KINDS`, `CONDITIONS`, `PREFIX` map, `MODEL_BULLETS`, `DATA_DESC`, and `EVAL_DESC`. 2. Run it once: `python code/eval_rollout_pages_.py`. The script writes `/index.html` + per-condition charts and rebuilds the hub `index.html`. 3. Commit the script (the generated HTML + PNGs are not committed — they live under `unit_tests/output/eval_rollout_pages/`, which is served via the standard browse mount). 4. Add a section in this file under `## Active reports` documenting the model bullets + prefix mapping + URL.