TRI diary

## What we did
- TRI onboarding paperwork, badge, Slack
- SSH access set up: school server → mac (Tailscale) → robot-lab
- First exploration of Raiden's YAM data pipeline
- Met Sergey, mapped out the week

## Reflection
- First day. Lots of new infra to digest.

## Next steps
- Record first data via Raiden's pipeline

## What we did
- Recorded first YAM demos via Raiden's pipeline
- Started fiducial exoskeleton print
- Quantified Raiden's calibration residual baseline

## Reflection
- (TBD — fill in)

## Next steps
- Collect 80 demos cup task

## What we did
- Collected 80 demos cup task on YAM
- 3 envs × 4 viewpoints split

## Reflection
- (TBD)

## Next steps
- Train PARA + ACT on these 80

## What we did
- Trained PARA + ACT on the 80-demo cup task
- First training runs on DGX
- YAM in-dist result hit — confirmed "15 demos vs 3,000+" sticky number is real
- Per `vault/fleet/agents/project_highlevel/memory.md`: this is the day the headline number got sharpened from "30 demos" to "15"

## Reflection
- (TBD)

## Next steps
- OOD eval at held-out positions / viewpoints

## What we did
- Evaluated PARA + ACT at held-out object positions and viewpoints

## Reflection
- (TBD)

## Next steps
- Iteration / hardware recovery if needed

## What we did
- Slack day per the plan: iteration + hardware recovery

## Reflection
- (TBD)

## Next steps
- Phase 1 retro tomorrow

## What we did
- Phase 1 retro
- Gate check: PARA on YAM cup task beats ACT on held-out viewpoint ✓

## Reflection
- (TBD)

## Next steps
- Decide on wrist cam vs single-view scope for Phase 2

## What we did
- Locked the second-beat factorization framing as the canonical mechanism statement
- "PARA factorizes EEF (x,y,z) into (uv heatmap, world-frame height) — height not depth bakes in viewpoint invariance"
- Wired into `pitch.md` and `goals.md`
- Phase 2 begins (originally wrist cam — soon to be replanned)

## Reflection
- (TBD)

## Next steps
- Replan wrist cam scope

## What we did
- Ran evals for the semi-robust training+eval setup, ~20 demos, spatial in-distribution and OOD
- Locked: wrist cam dropped from the headline (single third-person scene cam is the modality)
- Locked: overview figure structure — action head diagram (top) + 4-5 evidence buckets (bottom)

## Reflection
-

## Next steps
- Move stations + recalibrate tomorrow

## What we did
- Moved the robots to a different station
- Experimented with different tripod setups
- Recalibrated everything from scratch
- Collected some data on the new station

## Reflection
- Other folks were upset that the camera / bimanual rig had been reconfigured to single-arm — that's what forced the move

## Next steps
- Deploy models on this setup tomorrow

## What we did
- Deployed models on the new station setup
- Results were OK but the evaluation setup wasn't robust
- Models still had a lot of room to go

## Reflection
- Want to test whether **data efficiency** is the actual bottleneck — prompting the weekend data-collection investigation. Collect more, see if more demos closes the gap.

## Next steps
- This weekend: collect more demos and see if data scale is the limiter

## What we did
- Started the weekend 100-demo collection push (Sat half)
- Trained + deployed some early models on what we had so far
- Results weren't that good

## Reflection
- Confirms it's worth scaling demo count harder — collecting more Sunday

## Next steps
- Sunday: collect another batch (target 100 episodes total weekend)

## What we did
- Collected 100 episodes (the full weekend dataset)

## Reflection
-

## Next steps
- Train + deploy on the Sunday 100-episode batch tomorrow
- Run a wide eval sweep

## What we did
- Trained + deployed on the Sunday 100-episode batch
- **Ran a ton of evals all day** for many different subsets, compiled + annotated the results
- Results look decent — still some room to go for precision
- Locked the convention: project_highlevel defaults to brainstorm mode unless told otherwise
- **Last night:** designed a new volume-based approach that projects onto multiple views — including their wrist view. Wrist cam comes back, but as a *multi-view projection target* rather than a fused input modality.

## Reflection
- Wrist cam returning in a new form is interesting: previously dropped because single-view was the rhetorical win and live deploy was finnicky. Multi-view projection sidesteps both — it's an output-side use of the wrist view, not an input dependency.

## Next steps
- Collect wrist-view data tomorrow
- Train + deploy the multi-view volume model

## What we did
- Meetings in the morning
- Finished compiling + annotating yesterday's eval results
- Wired up the new multi-view volumetric model end-to-end
- Collected first data with the wrist view
- Kicked off training
- Built this TRI diary at omidlab.net/tri_diaries

## Reflection
- Hard cap on the day: early dinner with a friend — didn't get to deploy

## Next steps
- Deploy the new model tomorrow
- Phase 3 (backbone experiments) kicks off in parallel on lab GPUs
- Family trip in 10 days — what's the cool result to show family?

## What we did
- **Architecture pivot**: multi-view volumetric model with **per-voxel grip/rot heads** is now canonical, replacing the single-view KV-factored production arch (as of today)
- **Two-view input**: scene cam (fixed ZED, aruco PnP locked) + right-wrist (FK + hand-eye). Shared DINOv3 ViT-S/16+ backbone, one forward per view.
- **World-space voxel grid**: P×P=128×128 pixel grid × Z=64 height bins per view, N=2 slabs stacked → V ≈ 1M voxels. View identity + height live in the slab dim. No time axis on the voxels.
- **Per-voxel features (cross-view fusion happens here)**: for each voxel, project its world XYZ into every view, bilinear-sample each view's feat_head (32ch) maps, concat with 16-d learned height embedding, → 3-layer MLP → voxel_feat (32-d).
- **Volume scoring**: T=32 query heads off global state ([eef | cls_scene | cls_wrist]) → einsum bilinear scores → per-timestep softmax over voxels. CE on GT-nearest-3D voxel (no +1 slot trick).
- **Grip + rot**: per-step gather at GT voxel (teacher) / argmax voxel (infer) → 3-layer 1D CNN over T → per-step linear heads to 32 grip / 64 rot bins.
- **Deploy**: action chunking depth=4 + closed-form **reverse-projection volume pool** (~5 ms/chunk, replaces the 3s cKDTree path) — older chunks' logits remapped onto the newest chunk's voxel grid.
- **Optional flags (off by default)**: `--cross_view_layers N` (DA3-style alternating within/cross self-attn over DINO patches, ~7M params + 60ms/forward for N=4); `--border_mask_px N` (default 5, zeros logits in N-pixel frame at every view's pred-grid edge).

## Reflection
- **Why per-voxel won locally over global-CLS**: rot correlates strongly with EE position, so a per-position prediction has the right spatial inductive bias. Earlier global-CLS variant beat it (rot loss 0.73 vs 2.60 at step 2400 on the 35-ep wrist-only set) but that was a capacity issue — 32-d voxel features can't carry what the 800-d [eef|cls|cls] global input did. Today's bet: bigger / fresh datasets buy back the gap. Fallback if rot still lags: global residual `grip_logits_v = local(voxel_feat_v) + global(CLS_concat)`.
- **Binding framing locked** (canonical pitch as of today): *"Today's policies must learn the EEF↔pixel association + depth-along-ray + scene geometry + camera-to-robot extrinsics, all from image-to-action labels. PARA binds the action space to its projecting image features and reformulates policy learning as feature selection over a discretized candidate volume. The geometry-learning burden disappears."* This subsumes both old PARA (factorized single-view binding) and new PARA (joint multi-view binding) — view count is now an implementation detail, not a story. The Jun-3 single-view headline lock is retired. Wired into `pitch.md` + `vault/para/goals.md` + `vault/fleet/agents/project_highlevel/memory.md`, old framings preserved under OLD markers.

## What we did (today, concrete)
- **First deploy** of the new multi-view volumetric model
- Collected **more + better** data with the deployed model in the loop
- Making figures while training continues in background
- Re-running the same eval suite as before so cross-arch comparison is apples-to-apples

## Next steps
- 3 trainings queued:
  1. **cup_pluswrist_moredata** (50 ep, GPU 0) — running
  2. **cup_pluswrist_25ep** (25 ep symlink, GPU 1) — running, **data-efficiency probe**
  3. **towel_fold_basic_wrist** (30 ep) — auto-launches when (1) or (2) finishes
- Watch: does the global residual fallback become necessary on rot? If yes, that's an honest scaling story; if no, the per-voxel-only arch holds.
- Family trip in 10 days — these three runs are the cool-result-for-family candidates.

## What we did
- **Meetings** (2): one with Sergey, one with the USC squad. Sergey one was fine. USC one was a bit annoying — advisors seemed checked out.
- **OOD object finetune eval matrix** — ran the {ours, baseline} × {non-finetune, finetune} cells, annotated + compiled
- **Wired in the multiview eval script** end-to-end
- **Collected data for the multiview eval**
- **Worked on the paper overview figure** (binding-framing flow, the Encode → Sample → Decode three-step)
- **Collected towel folding data**
- **Kicked off towel folding training** — will eval tomorrow
- Pitch / framing: locked "Projection-Anchored Robot Actions" as the new PARA expansion + landed the binding framing title

## Reflection
- Off day per whoop, low energy. Two meetings ate the morning. USC advisors checked-out is annoying — worth a small recalibration on what you're actually pulling out of those meetings (decision input? sanity check? air cover? if none, the frequency might be wrong).
- Still shipped a lot for a low-energy day: full OOD finetune eval matrix + multiview infra + towel data + figure work. That's a "shipped despite friction" data point worth noticing — your routine is doing its job.
- Family trip in 9 days. Towel folding result tomorrow is the candidate "cool result to show family."

## Next steps
- Eval towel folding model tomorrow
- Look at multiview eval results
- Continue overview figure (4 tightenings flagged: binding-arrow emphasis, step-2 rename to "Bind...", grip/rot acknowledgment, label cleanup)
- Advisor's two suggested demos (flexible-cameras-at-inference, sim2real/real2sim) — sequenced for Phase 5/6, NOT this week

## What we did
- **Towel folding demo eval**: grasps were in the right (u, v) position but **off in the height dimension** — clean failure mode that points at height resolution as the bottleneck
- **Arch refactor in progress** to enable many more height bins (target 256, up from 32):
  - **Drop the per-voxel / per-point MLP**
  - Each 4D query (time + position, which factors to t + height-bin-along-ray) gets encoded, projected to image features at the projecting pixel in each view
  - **Dot product** of query encoding against per-view image feature → heatmap response
  - **Sum activations** across views (no learned fusion, no view selection — soft gradient through both)
- **AI glasses setup** (today)
- **Visualizations + website work**
- New model training → re-eval tomorrow

## Reflection
- The refactor brings the canonical arch closer to the *old* dino_kv KV-factored philosophy — pure bilinear scoring, no per-voxel learning, just dot product. The multi-view extension is "sum dot products across views," which is the cleanest possible generalization.
- This makes the arch even more "purely binding" in spirit: the only learned operation per voxel is the dot product, no MLP between the projection and the score. Stronger version of the same framing.
- Cheap scaling on height bins falls out for free now — 256 bins is no longer compute-hostile. The towel-folding failure (height-off, position-correct) becomes the empirical motivation for the simplification.
- Light Friday work day in volume but substantive in architecture. The lower energy didn't prevent a real refinement.

## Next steps
- Re-eval new model tomorrow with the 256-height-bin variant — does it nail the towel grasp?
- AI glasses — figure out the use case (data collection? viz? something else?)
- Continue visualizations + website
- Family trip in 8 days — towel folding done-right is still the candidate "cool result"

## What we did
- Saturday — mostly logistics
- **Eval'd v3** (the Jun 12 refactor: dot product, no per-voxel MLP) — **didn't work that well**
- **Reverting to v2** (per-voxel MLP fusion, the Jun 10 canonical) + two changes:
  - **More height bins** (the Jun 12 insight from towel-folding still applies)
  - **Positional encoding for adaptive height query** (replaces the learned 16-d nn.Embedding — should enable variable bin counts at inference without retraining)
- Re-eval v2.1 tomorrow

## Reflection
- Per-voxel MLP fusion is load-bearing. The dot-product simplification looked architecturally clean (and matched the binding framing more purely) but lost something empirically — the MLP is doing real work fusing per-view features + height embedding. The "pure binding" framing stays *narratively* clean even with MLP fusion: the MLP is the *learned* part of the binding, not a departure from it.
- One-day failure cycle saved more time than it cost — cheap iteration on the canonical arch via "is the per-voxel MLP actually load-bearing?" is exactly the kind of question worth a Saturday cycle.
- The Jun 12 height-bin insight survives the revert — towel grasp's height precision still mattered, just needs to coexist with the MLP fusion. v2.1 is "v2 + Jun 12's lesson, integrated cleanly."
- Pos enc for adaptive height is a small but nice property: variable resolution at inference without retraining means you can run cheap at training time, dense at deploy time. Worth a paper sentence.

## Next steps
- Eval v2.1 tomorrow — does it nail towel folding height?
- Update `feedback_multiview_voxel_canonical.md` once tomorrow's eval lands (held back yesterday, good call)
- Family trip in 7 days — re-eval result tomorrow is the cool-result-for-family candidate

No entry yet.

## What we did
- **Two more backbones trained + evaluated**:
  - **PaliGemma** (VLM backbone) — works fine, but **worse than both image-based backbones**
  - **DynaFlip** (robotics-specific backbone) — **beats DepthAnything**
- The Phase 3 backbone ablation table is forming organically (originally planned for backbones-agent on lab GPUs; happening at TRI instead)
- Egg + teapot models from Sun continue iterating

## Reflection
- DynaFlip > DepthAnything is a real result — extends the "head > backbone" story onto a *different* family of backbones. The contribution isn't just "DINO+head beats DINO-L" anymore; it's becoming "the head works across backbone families and the right one depends on task."
- PaliGemma underperforming image-only backbones is interesting and worth noting carefully: VLM features have language structure that may not be the right inductive bias for action prediction. Don't pitch this as "PaliGemma is bad" — pitch as "language-aligned features don't dominate where pixel-aligned features do." Both are useful framings, different audiences.
- Backbone ablation suite as of today: DINOv3-S/16+ (canonical) · DepthAnything · DynaFlip · PaliGemma. Solid range for a table.

## Next steps
- Continue iterating egg + teapot
- Try cup spill cleanup task tomorrow if station is free (otherwise next weekend)
- Comparison overview figure iteration (good draft at `/data/cameron/scratch_files/para_overview_comparison`)

## What we did
- Iterated on **comparison overview figure** (`/data/cameron/scratch_files/para_overview_comparison` — clean side-by-side Traditional Global Regression vs PARA Image-Action Selection, 2 input views, projection step explicit)
- Meeting 10:30–11:30
- Rest of day: new task work + website building + sim2real setup planning

## Reflection
- **Weekend strategy locked**:
  - **Next weekend (Jun 27-28)** post-vacation: grab Keurig station (nobody there weekends) → start long-horizon Keurig task
  - **Cup spill cleanup**: do it next weekend too if it doesn't fit today (Panda team not in on weekends)
  - This is a good use of weekend access — shared stations are "free" only when their primary owners aren't around
- **Sim2real approach sketched today (real-to-sim re-render)**: use known camera intrinsics/extrinsics + known YCB object meshes to re-render a real demo trajectory in sim. Augments the real demo with a sim duplicate. Different from standard sim2real (train sim → deploy real); this is "given a real trajectory, produce its sim image-equivalent." Cheaper to set up because the trajectory is already correct.
  - Useful for: data augmentation (one demo becomes two), domain-robustness eval (does PARA generalize across render styles of the same trajectory?), cheap sim eval infrastructure.
  - Implementation needs: YCB meshes, MuJoCo or Blender renderer, scene alignment to the YAM workspace.
  - Vacation candidate — fits the parked sim2real workstream nicely.
- **Vacation in 2 days**. Pre-trip checklist not yet done — SSH chain verification, durable runs queued, etc.

## Next steps
- Try cup spill cleanup task today if station available; otherwise → next weekend
- Continue website + figure work
- Wed Jun 17: pre-trip checklist (SSH chain from home WiFi, durable training runs queued)
- Vacation Thu Jun 18 → Mon Jun 22: MuJoCo sim2real + website V2 + figure tightenings

## What we did
- **Morning low-energy / brain-fog state** — self-diagnosed as the "aimless when advisor priorities diverge" pattern. Doubled down on the plan from Jun 16-17 rather than replanning.
- **Locked the back-half sequencing**: now → ~Jul 1 single-arm core · ~Jul 1 → mid-Jul bimanual integration + derisk · mid-Jul → Aug 17 bimanual scale-up + final. Nice-to-haves delegated to others. 1hr/day Onshape track starts now.
- **Refined the Sergey collaboration**:
  - Sergey agreed with single-arm-first, but pushed to start pipeline derisk NOW with data collectors using the same recording script
  - **YAM meeting surfaced**: collectors are NOT programmers. Async Loom-and-go won't work — they need an in-person Tue Jun 23 onboarding session.
  - Pre-vacation pivot: write copy-paste-friendly non-programmer instructions + verify space-mouse teleop works (collectors prefer it).
- **Weekend method-diagram task captured** in vault: detailed architecture-flow figure (rotation + gripper heads, pre/post-fusion concatenation, feature shapes at each stage). Reuse old Inkscape file as starting point.
- **Pitch evolution**: new candidate line — *"3D reasoning for robotics throws away pretrained 2D features. PARA brings 3D actions to the 2D features instead — action selection on per-pixel evidence."* Composes with the Jun 10 binding framing as opener-vs-mechanism.
- **Ideas parked**: Pandas calibration → DROID pretraining (or Pandas → YAM transfer) — captured as post-paper extension, NOT priority.

## Reflection
- The pattern of *naming* the aimless state when advisor priorities diverge is healthy. The plan was already right; the fog was residue from arguing into it against pressure.
- The "subtraction over addition" discipline held all day. Several extension threads surfaced (cloud migration, sim env, Panda/DROID) — all properly parked or sequenced rather than crammed in.
- The own-robot REDO track (1hr/day Onshape → Jul 1 sprint) feels right because it addresses the *root cause* of why nice-to-haves keep being hacky (no stack control). Building own-robot in the back-half makes the parked nice-to-haves principled rather than hacky if they ever come back.
- The non-programmer-collector finding from the YAM meeting is the kind of operational detail that derails plans silently. Catching it pre-collection (not mid-batch) is exactly why "pipeline derisk before scaling" matters.

## Next steps
- Pre-vacation (tomorrow morning before flight): non-programmer instructions doc, space-mouse verification, pre-trip checklist (SSH chain from phone hotspot, push uncommitted, durable runs queued), Sergey reply
- Vacation Thu Jun 18 → Mon Jun 22: detailed method diagram + website V2 + Tier-1 visualizations + 1hr/day Onshape
- Tue Jun 23: in-person collector onboarding session + first collection run

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

## What we did
- **Eval batch firing** — hard eval week underway after Tue Jun 23 was eaten by social/onboarding day.
- **Data collection ramping** — collectors scaling up under the new script; episode count growing from the existing 20.
- **Sergey's data-efficiency-curve ask landed mid-day**: train in-dist models at 20 / 30 / 40 / 50 / 60 episodes for the data-eff plot. Reframes today's collection from "side-quest pipeline derisk" to **paper-figure dependency** — the curve doesn't land before Fri Jun 26 unless 40+ new episodes get collected this week.
- **Planning decisions surfaced (open)**:
  - Subset construction: nested growth (20 ⊂ 30 ⊂ 40 ⊂ 50 ⊂ 60) vs random subsets × N seeds. Nested is the v1-this-week answer; error bars are a v2 pass.
  - Eval matrix scope: Option A (in-dist only for the 5 data-eff models, OOD stays on the 60-ep model) vs Option B (full matrix across all 5). A is the cleaner paper story; B is the stronger possible claim if numbers land.
- **Fiducial boards for bimanual** — captured as a walk-during-eval-runs side errand; ChArUco preferred for multi-cam extrinsics on the bimanual setup.

## Reflection
- The day reordered itself once Sergey's ask landed — collection script ceased being onboarding hygiene and became the critical path. Catching that reframe in real time (vs realizing on Thursday) is the kind of triage that earns back the slack you didn't know you needed.
- Tue Jun 23 going social rather than evals isn't free — eval week was already tight at 4 days, now it's 3. The Friday deadline now has zero buffer for re-runs.

## Next steps
- Today (Jun 25): continue eval batch, push collector throughput, start the first new training (30-ep) the moment 30 episodes exist.
- Decide Option A vs B on the eval matrix before kicking off the data-eff trainings — affects total wall-clock.
- Fri Jun 26: evaluate the data-eff series + compile Fig 4.

No entry yet.

## What we did
- Finished eval week — last baseline closed out.
- Started collecting demos for the new **CupSpill** task (multi-stage: reorient dropped cup → pick towel → wipe spill).
- Trained on CupSpill same day; first impression "works decently."
- Side errands: printed ArUco boards for the bimanual setup; small CAD block on the own-robot REDO track.

## Next steps
- Eval CupSpill tomorrow (Sat Jun 27) on in-dist.
- Decide whether CupSpill replaces or sits alongside the egg pick-place task as the YAM dexterous anchor for the figure.

## What we did
- **Evaluated CupSpill** — model hit **100%** on the 3-step task (reorient → pick towel → wipe). Real headline-grade dexterous result on YAM.
- **Bimanual Raiden fidex setup** — fiducial / camera-calibration scaffolding for the bimanual rig, getting infra ready ahead of the Jul-1 → mid-Jul bimanual integration window.
- Spent time refining the one-line method explanation (PixelNeRF-style). Working tagline: *"Action prediction as feature selection."*

## Reflection
- CupSpill at 100% on a multi-stage sequential task is a meaningful jump from egg pick-place — much more legible to non-roboticists and a stronger paper figure anchor candidate. Worth resolving whether it replaces or stacks with egg in the figure.
- Starting bimanual fidex setup today is slightly ahead of the Jul-1 plan, but it's the right kind of forward-loading: rig prep before Annie arrives Jun 29 means the hardware time can actually go to hardware, not calibration debugging.

## Next steps
- Lock CupSpill's role in the paper figure (anchor vs additional task).
- Decide if CupSpill needs OOD-pos / OOD-view evals on Monday.
- Annie onboarding Mon Jun 29 → hard mode starts.

## What we did
- More eval runs continuing the post-CupSpill push.
- Figure-making — iterating on Fig 4 / method diagram / overview figure for the paper.
- Side: refined the BIND abstract opener with a Hamming-style hook ("Modern vision models recognize a cup zero-shot. Why do robot policies built on them still need thousands of demonstrations to pick one up?") — primes CupSpill being the headline result later in the paper.

## Next steps
- Continue figure compile + eval coverage.
- Annie hard-mode kickoff Mon Jun 29.

## What we did
- Continued eval runs (additional conditions / extended sweep beyond Friday's batch).
- Paper figure work — compiling Fig 4 from the eval results, iterating on the method-diagram figure, and shaping the figure story around the CupSpill 100% result and the data-efficiency framing.

## Next steps
- Resolve the figure-anchor question for the YAM dexterous task (CupSpill vs egg pick-place vs both in Fig).
- Continue eval coverage as needed for OOD-pos / OOD-view evidence on CupSpill.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.

No entry yet.