---
date: 2026-06-24
phase: 4
headline: Eval batch running, collection ramping for the data-efficiency curve
---

## What we did
- **Eval batch firing** — hard eval week underway after Tue Jun 23 was eaten by social/onboarding day.
- **Data collection ramping** — collectors scaling up under the new script; episode count growing from the existing 20.
- **Sergey's data-efficiency-curve ask landed mid-day**: train in-dist models at 20 / 30 / 40 / 50 / 60 episodes for the data-eff plot. Reframes today's collection from "side-quest pipeline derisk" to **paper-figure dependency** — the curve doesn't land before Fri Jun 26 unless 40+ new episodes get collected this week.
- **Planning decisions surfaced (open)**:
  - Subset construction: nested growth (20 ⊂ 30 ⊂ 40 ⊂ 50 ⊂ 60) vs random subsets × N seeds. Nested is the v1-this-week answer; error bars are a v2 pass.
  - Eval matrix scope: Option A (in-dist only for the 5 data-eff models, OOD stays on the 60-ep model) vs Option B (full matrix across all 5). A is the cleaner paper story; B is the stronger possible claim if numbers land.
- **Fiducial boards for bimanual** — captured as a walk-during-eval-runs side errand; ChArUco preferred for multi-cam extrinsics on the bimanual setup.

## Reflection
- The day reordered itself once Sergey's ask landed — collection script ceased being onboarding hygiene and became the critical path. Catching that reframe in real time (vs realizing on Thursday) is the kind of triage that earns back the slack you didn't know you needed.
- Tue Jun 23 going social rather than evals isn't free — eval week was already tight at 4 days, now it's 3. The Friday deadline now has zero buffer for re-runs.

## Next steps
- Today (Jun 25): continue eval batch, push collector throughput, start the first new training (30-ep) the moment 30 episodes exist.
- Decide Option A vs B on the eval matrix before kicking off the data-eff trainings — affects total wall-clock.
- Fri Jun 26: evaluate the data-eff series + compile Fig 4.