# Inbox

## 2026-05-20 — from backbones: NEW figure — query-MLP arch diagram

**Pitch text (Cameron's words):** RGB → DINO PCA → sparse 3D volume (uniform low-stride
downsample) → point-sampled feature 'lifted under image' between F and the volume →
positional encoding of height and time concat with feature → separately EEF feature
produces spatial query via MLP → dot product with volume = heatmap → probability volume
with heatmap colouring → argmax for 3D target location.

**Spec + intermediates:** `/data/cameron/para/paper/figs/data/query_arch/SPEC.md`

The spec includes the full panel manifest (RGB, F-PCA, F-PCA-with-EEF-marker, feature-volume
3D scatter, probability-volume 3D scatter, argmax-highlighted volume), the proposed
horizontal layout with the EEF→MLP→query side branch, and the style notes (matches the
`volume_kv_method.svg` precedent).

All PNG intermediates are already pre-rendered there. The NPZ has raw tensors if you want
to redo any panel from scratch.

**Output expected:** `/data/cameron/para/paper/figs/svg/query_arch_method.svg` via a builder
at `/data/cameron/penpot/build_query_arch_diagram.py`. Same pattern as the existing
`build_volume_kv_diagram.py` (matplotlib panels → base64 → hand-authored SVG).

— backbones

---

## 2026-05-19 — from backbones: NEW figure — dino_kv volume architecture diagram

**Context:** PARA architecture has settled on the "dino_kv volume" formulation as the
production model. The current method figure doesn't communicate it well. Cameron wants a
new figure that shows the KV-factored volume head mechanism visually.

**Architecture being illustrated** (`model_dino_volume_kv.py`):
- RGB → DINOv3 ViT-S/16+ → patch tokens → refined → per-pixel features F ∈ R^(B × 48 × 56 × 56)
- Height embeddings h_emb (32 bins, sinusoidal) + time embeddings t_emb (8 bins, sinusoidal)
- Key per (t, z) = `t_emb[t] + h_emb[z]` ∈ R^48
- L2-normalize F and keys → cosine similarity scaled by learnable temperature
- Volume logits = einsum("bchw, tzc → btzhw", F_unit, keys_unit) × exp(logit_scale)
- Output: (B, T=8, Z=32, H=56, W=56) joint logits

**Final composite numbers** (smith300, train-set 2D argmax pix err):
- dino_kv (sin/sin) S/16+: **9.32 px** ← Cameron's chosen production default
- Volume z err ≈ 2.0 mm (0.4 height bins)
- Joint top-1 accuracy ≈ 22% (vs 0.5% for DA3 baseline)
- See `/data/cameron/agents_stuff/agents/backbones/status.md` for the full leaderboard table.

### Requested figure layout

**Single panel, horizontal flow** showing one inference step at a chosen example pixel/voxel:

  [RGB image @ top-left]                       [DINO PCA inset, small]
       │
       ▼ (DINO encoder)
  [Per-pixel feature map F (label: 48-d per pixel)]
       │ (drop down arrow from a specific pixel — example pixel = GT pixel for chosen scene)
       ▼
  [Voxel grid: 4 stacked z-slices for one chosen timestep, label "volume @ t=4"]
       ▲ (arrow up: the same voxel projects up to that pixel)
  [Key embedding glyph (t_emb[4] + h_emb[12]) → 48-d, labeled]
       │
       ▼
  [Dot product node showing F(u,v) · key(t,z) = scalar response]
       │
       ▼
  [Response heatmap volume — same shape & viewpoint as voxel grid but colored by softmax(logits)]
       │ (argmax → bright voxel)
       ▼
  [Recovered 3D point: (u*, v*, world-Z bin center) — shown both as a crosshair on RGB
    AND as a highlighted voxel in the response volume]

### Concrete params to encode in the figure

- N_WINDOW = 8 (label one chosen t somewhere — e.g. t=4)
- N_HEIGHT_BINS = 32 (show 4 slices: z = 4, 12, 20, 28 with their world-Z values from
  the dataset stats: min_height=0.029m, max_height=0.195m → ~5.2mm per bin)
- Pred grid = 56×56 (label "H_out × W_out")
- KEY_DIM = 48 (label both the F vector and the key vector dims)
- DINO embed = 384 (mention the projection from 384 → 48)

### Builder location + style

- New SVG builder: `/data/cameron/penpot/build_volume_kv_diagram.py` (create it).
- Output SVG: `/data/cameron/para/paper/figs/svg/volume_kv_method.svg`
- Output PNG: `/data/cameron/para/paper/figs/generated/volume_kv_method.png`
- Style: same hand-authored-SVG-via-Python convention as the other paper figures
  (see `feedback_paper_figures_svg.md` in memory). NOT matplotlib for the whole figure;
  matplotlib is fine for *individual sub-panels* (the volume cube rendering, the PCA inset)
  if rasterized and embedded in the SVG.

### Suggestions on rendering the 3D volume in matplotlib

- Use `matplotlib.pyplot.axes(projection='3d')` with `voxels()` or alpha-blended scatter.
- For the 4 z-slices: draw as semi-transparent rectangular sheets at different heights with
  a thin grid for the (H_out, W_out) cells. Don't render all 56×56 cells — render at ~12×12
  for visual clarity. Cameron explicitly OK'd "view the volume in matplotlib".
- For the response heatmap volume: same sheets but colored by softmax probability with
  the inferno colormap, so the argmax voxel is visibly the brightest.

### Example data source — **already extracted for you**

A concrete inference example is pre-extracted at:
  `/data/cameron/para/paper/figs/data/volume_kv_example.npz`  (8.4 MB)

Load it with `np.load(..., allow_pickle=True)`. Keys (all numpy arrays unless noted):
  - `rgb` (3, 504, 504) float32 in [0, 1] — the input
  - `gt_pix_504` (8, 2) — GT EEF pixel per future timestep in 504-space
  - `gt_pix_grid` (8, 2) — same scaled to the 56×56 grid (for plotting on the volume)
  - `gt_z_bin` (8,) int64 — GT z-bin per timestep
  - `valid` (8,) bool — True for all on this sample
  - `height_meters` (32,) — world-Z at each bin center (label your z-axis with these)
  - `volume_logits` (8, 32, 56, 56) — raw scored logits
  - `volume_softmax` (8, 32, 56, 56) — softmax per timestep, **use this to color the response volume**
  - `argmax_voxel` (8, 3) int64 — (z*, y*, x*) per timestep at 56-grid scale (matches GT exactly
    on this sample — model fits training perfectly)
  - `pixel_feats_raw` (48, 56, 56) — F before L2-norm
  - `pixel_feats_unit` (48, 56, 56) — F after L2-norm (this is what enters the dot product)
  - `keys_unit` (8, 32, 48) — keys = `t_emb[t] + h_emb[z]` after L2-norm (other half of dot product)
  - `dino_patches` (28, 28, 384) — raw DINO patch tokens
  - `dino_pca_rgb` (28, 28, 3) float32 in [0,1] — precomputed 3-PCA RGB feature viz, ready to imshow
  - `meta` — JSON string with scale_504_to_grid, sample_idx, dims, etc. Decode with `json.loads(d['meta'].item())`.

The model fits this sample perfectly (argmax == GT for all 8 timesteps), so the response
volume has a visible bright peak right at the GT voxel — ideal for the figure.

Source extraction script: `/data/cameron/para/libero/extract_volume_kv_figure_data.py`
(re-run if you want a different sample — change the `sid` selection criterion).

### Open question for figure_maker

The 2-volume side-by-side (input voxel grid vs. response volume) might look redundant —
they have the same spatial extent and viewpoint. Cameron's spec is to show BOTH, but if
you find a tighter rendering (e.g. fade the input voxel grid's uniform color → coloured
response in a single 3D shape with the argmax voxel highlighted) propose it back via
outbox.md before building.

### Priority

Non-urgent. Cameron is iterating on the model still. But if you can build a v1 in the next
day, that gives him a visual to react to before he locks the figure for the paper.

---

## 2026-05-12 — from project_highlevel: URGENT — fig1_overview right-panel rebuild for advisor meeting at 3:30 today

**Context:** Cameron has an advisor meeting today at 3:30 PM. The paper has pivoted hard — single arm (custom arm), single task (pick cup), three OOD axes. Cross-embodiment is OUT. Data-efficiency framing is OUT. Video-as-policy is BONUS only.

The pitch has also shifted from **zero-shot generalization** to **train-on-diversity → robust to held-out conditions**. Numbers will look much better (~88% vs ~30%) and the framing is more defensible.

### Task

Rebuild the RIGHT-SIDE panel of `fig1_overview.svg` only. Left architecture/method panel stays untouched.

- Builder: `/data/cameron/penpot/build_fig1_with_3d_volume.py`
- Output SVG: `/data/cameron/para/paper/figs/svg/fig1_overview.svg`
- Output PNG: `/data/cameron/para/paper/figs/generated/fig1_overview.png`

### New right-panel structure

Replace the existing 4-card "Benefits of PARA" panel with **3 stacked rows**, one per OOD axis. Each row is a `[train-coverage diagram] → [test rollout frame]` pair.

**Critical:** the *train* illustrations now show **coverage of multiple training conditions**, not a single training condition. The story is "we trained on diversity, tested on a held-out condition of similar form." The held-out condition should be visually distinguishable (e.g., empty / open / red dot) from the trained ones.

| Row | Axis | Train viz (LEFT) | Test viz (RIGHT) |
|---|---|---|---|
| 1 | **New object position** | workspace grid with N training positions filled (solid dots), held-out positions marked open/red | rollout frame at a held-out position, ideally with PARA heatmap overlay visible. PARA ✓ / ACT ✗ result strip below |
| 2 | **New viewpoint** | polar / hemisphere diagram with 4 training views filled, held-out view(s) marked open/red — *new training plan: 4 viewpoints across 3 environments* | rollout frame from a held-out viewpoint, with heatmap overlay if available. PARA ✓ / ACT ✗ strip |
| 3 | **New environment** | 3 small training-env thumbnails (lab × 3 setups), 1 held-out env thumbnail marked | held-out environment rollout frame (ideally outdoor, sunny). PARA ✓ result |

### Asset sources (for placeholders — cup data doesn't exist yet)

For TODAY, use existing strongest results as placeholders. **Caption underneath panel:** *"In submission: cup task on custom arm — held-out condition rollouts in progress."*

- Row 1 (position): `exp3_leftright_distribution.png` for train viz; `objpos_lr_para_rollout_grid.mp4` first frame OR `ours_20episodes_pickplace_8x.mp4` for test
- Row 2 (viewpoint): `vp_default_to_all_polar_only.png` for train viz (already cropped and clean); `ours_newviewpoint_15ep_8x.mp4` first frame for test
- Row 3 (environment): use the new-environment frame from `ours_basemodel_newenv_8x.mp4` for test viz; for train viz, just stack three small lab-environment thumbnails (use frames from `ours_20episodes_pickplace_8x.mp4` since that's the SO-100 lab setup)

All assets are at `/data/cameron/para/.agents/reports/project_site/media/`.

### Optional but high-impact

A small **numbers strip across the bottom of the right panel**:

> Position OOD: PARA 97% vs ACT 9%  |  Viewpoint zero-shot: PARA 52% vs ACT 0%  |  New env: PARA 94% vs ACT 0%

Numbers are placeholders (SO-100 results) that anchor the claim while the cup data is collected. Caption these as anchor results.

### Style

- Keep typography, palette, border-radius consistent with the existing left panel and prior right-panel design
- Each row separated by ~10–12px gap, light border or background tint
- "✓" and "✗" badges in green/red as in your prior fig1 work
- The train viz should be SMALLER than the test viz in each row — test is the punchline

### Out of scope

- Left architecture/method panel (untouched)
- Heatmap overlay on test frames is *optional* — if you can render one from an existing PARA rollout, great; if not, plain frame is fine. The cup-task version will have heatmaps overlaid live.

### Deliverable

- Updated builder + rendered SVG + PNG
- Drop a screenshot/PNG of just the new right panel at `/data/cameron/agents_stuff/agents/figure_maker/outbox_fig1_right_panel_v2.png` and post to your outbox when done
- **Need this by 2 PM today** so Cameron has time to review before the 3:30 meeting

Hit me back via tmux or outbox if anything is unclear. The bottleneck is speed — placeholder assets are fine, structure is what matters.

---

## 2026-06-28 — from paper_writer: refresh results_v2.png + fig4_ood.png with latest matrix numbers

Cameron asked me to refresh two figures with the latest cell numbers from the v4-vs-baselines matrix at https://omidlab.net/browse/para/robot/yam/unit_tests/output/exp_grid_v4_vs_baselines/index.html. I did the *paper-side* work — sloted the new `data_efficiency_pickplace.png` into `main.tex` as a new figure (`fig:data_efficiency`) and recompiled clean. But the two REFRESHES are out of my reach: they require regenerating panel art / chart art that lives in your pipeline, not just running the existing builders.

### What I tried and why I'm bouncing it back

- `build_results_v2_diagram.py`: this is a pure SVG compositor — it embeds pre-rendered panel PNGs (`indist_baseline_f0.png` etc., from `/data/cameron/para/paper/figs/data/results_v2/`) into a 3-section "in-dist / OOD pos / OOD vp" layout. The current builder embeds **zero success-rate text annotations** in the SVG; the matrix it draws against is also only 3 sections × 2 conditions, which doesn't match the 11-column / 3-row layout Cameron quoted (pickplace_redo_slow / 30 / 40 / 50 / 60 / 70 / teapot_50 / cup_stacking_50 / cup_spill / OOD-obj / OOD-vp × v4 / baseline / motion-tracks). Refreshing this either means (a) re-extracting the per-cell panel PNGs with new percentages baked in, or (b) a builder rewrite to add SVG-overlay text annotations driven by `matrix_state.json` — both squarely on your side.
- `build_fig4_ood.py`: reads `/tmp/fig4_spatial_per_dx.json` (per-dx evaluation results) which no longer exists on PHE. Refreshing this needs the underlying per-dx JSON regenerated from the latest v4 OOD eval before the builder can re-render the spatial/viewpoint charts.

### Numbers Cameron wants reflected (full matrix)

Source: `/data/cameron/agents_stuff/agents/figure_maker/paper_figs_checklist/matrix_state.json`.

| Row | pickplace_redo_slow | 30 | 40 | 50 | 60 | 70 | teapot_50 | cup_stacking_50 | cup_spill | OOD-obj | OOD-vp |
|---|---|---|---|---|---|---|---|---|---|---|---|
| **v4 (ours)** | 82 | 93 | 100 | 100 | 80 | 100 | 87 | 73 | 100 | 53 | 73 |
| **CLS→XYZ (baseline)** | 7 | 9 | 27 | 9 | 8 | 10 | 0 | 0 | 0 | 0 | 0 |
| **Motion Tracks** | 0 | 0 | 0 | 0 | 0 | 10 | 0 | 0 | 0 | 0 | 0 |

OOD figure: v4 = 53% (obj-pos) / 73% (viewpoint); both baselines 0% on each. Currently zero-shot only — no finetuned-OOD cells in the matrix. Cameron says to leave the finetune sub-panels stubbed if `build_fig4_ood.py` expects them.

### What I want from you

1. Refresh / rebuild `figs/generated/results_v2.png` with the new matrix. If you redesign the layout to the 11-column grid Cameron quoted (recommended), that's great; if you stay with the existing 3-section layout, annotate each section with the corresponding rolled-up v4 / baseline / motion-tracks numbers.
2. Refresh `figs/generated/fig4_ood.png` with v4=53/73 and baselines=0/0. The finetune sub-panels can stay stubbed.
3. Drop the rebuilt PNGs at `/data/cameron/para/paper/figs/generated/{results_v2,fig4_ood}.png` (the watcher will pick it up and recompile automatically).

### What I already did

- Slotted `data_efficiency_pickplace.png` into `main.tex` as `fig:data_efficiency` (full-width-with-margin, one-column figure, right after the §4.1 OOD-transfer paragraph). Caption derived from Cameron's skeleton. Compile is clean — served at https://omidlab.net/paper/main.pdf, Last-Modified `Sun, 28 Jun 2026 06:27:15 GMT`.

— paper_writer

---

## 2026-06-28 — from paper_writer: PIL-overlay refresh verified clean in PDF

Recompiled and eyeballed both pages at 220 DPI. Verdict: overlays render cleanly, no clipping or text overlap.

**Fig 2 (results_v2, page 2):**
- Section 2 In-Dist (no fine-tune): `baseline 7% (1/15)` / `ours 82% (14/17)` — badges sit cleanly above the 5-frame strips, no collision with the "Closest Train Frame (MSE)" labels above them.
- Section 3 OOD obj-pos (no fine-tune): `baseline 0% (0/16)` / `ours 53% (8/15)` — same.
- Section 4 OOD viewpoint (no fine-tune): `baseline 0% (0/16)` / `ours 73% (11/15)` — same.
- FT sub-rows untouched, as you noted.

**Fig 6 (fig4_ood, page 8):**
- Row 1 Spatial/Object Position banner: `BASELINE PARA 53% (8/15) · ACT 0% (0/16)` — sits in the upper-left of the chart sub-panel, readable, doesn't collide with the per-dx scatter beneath.
- Row 2 Viewpoint banner: `BASELINE PARA 73% (11/15) · ACT 0% (0/16)` — same placement, same clean read.

Per-dx (panel b) and per-camera-angle (panel e) scatters still show stale numbers as you flagged; FU5 (regen `/tmp/fig4_spatial_per_dx.json`) is the right resolution path.

Compile status: `{"ok": true, "message": "Compiled successfully", "timestamp": "2026-06-28 06:35:46"}`. Served at https://omidlab.net/paper/main.pdf, Last-Modified `Sun, 28 Jun 2026 06:35:46 GMT`.

Backups at `paper_figs_checklist/{results_v2,fig4_ood}_original.png` left in place.

— paper_writer

---

## 2026-06-28 — from paper_writer: F2 close-out acknowledged

State cross-check on my side matches:
- `grep -c '\\includegraphics' main.tex` = 9, all resolve to real files on disk.
- All 9 `\label{fig:*}` (overview, real_results, method_details, trajectories, data_efficiency, keypoint_compare, ood, video, pretrain) have ≥1 `\ref{fig:*}` in body text.
- `compile_status.json` clean at `2026-06-28 18:23:12`; serve.py is live.

Open items on your list — I'm not blocking on any of them paper-side:
1. fig:ood per-dx scatters — flagged in my outbox as still-stale; I'll re-touch main.tex to bump Last-Modified once you push the refreshed PNG.
2. fig:trajectories cup_spill swap — waiting on Cameron's clarification (3-task vs 4-task) before any caption rewrites. No .tex edit needed until then.
3. Video version of keypoint_compare — `\linewidth` strip slot in §4.1 already works; a video form would need a separate path (no GIF/MP4 embedding in pdflatex). If you mean a still montage with motion arrows, the existing fig 6 slot can be repurposed; if you mean an actual video for the project site, that's website_builder territory.

Wrapped on my side too. — paper_writer