# V4 — production volumetric head for PARA (paper version)

Authoritative design doc for `lib/model_volumetric_v4.py`. Mirrors the
module docstring but adds figures, the version-history table, and the
backbone-experiment hooks. Update this **and** the module docstring
together when the architecture changes.

## TL;DR

V4 is **v2 + sin/cos PE + per-view adaptive z range**, with no other
architectural changes. v2's per-voxel MLP fusion across cross-projected
view features is preserved exactly — v4 only swaps height conditioning
and the per-view voxel grid spacing.

## Pipeline

```
                          ┌── view 0 (scene) ──┐
   rgb[k] ─────► DINOv3 ─►│ patch + CLS        ├──► per-view feat (B,F,P,P)
                          └────────────────────┘
                                                  ▼
                  ┌──── adaptive z range per view ────┐
                  │  z_hi_k = min(z_hi, cam_z_k − m)  │
                  │  bin_centers_k = (B, Z) per view  │
                  │  PE_z_k = sin/cos PE (B, Z, h_enc)│
                  └────────────────┬──────────────────┘
                                   ▼
       per-view voxel grid (B, Z, P, P, 3) via closed-form unproject
                                   ▼
        concat over views → (B, NZ=N·Z, P, P, 3)         ╮
                                                          │
   for each view k: bilinear-sample F_k at projected (u,v)│
       → (B, V, feat_dim), concatenate over N             │
                                                          │
   concat per-view PE_z_k along NZ → (B, V, h_enc)        │ Stage 5
                                                          │ per-voxel
   voxel_in = concat(all_view_feat, h_emb)                │ MLP fusion
   voxel_feat = MLP_3layer(voxel_in)        (B, V, F)     ╯

                                   ▼
   q = q_head(global_mlp([eef_feat, cls_concat, past_enc]))   (B, T, F)
   volume_logits = q · voxel_feat                              (B, T, V)
                                   ▼
                       per-voxel grip/rot head
                       (4-layer MLP at GT/argmax voxel)
                                   ▼
                       loss = CE on volume + CE on grip + CE on rot
```

## Why sin/cos PE instead of learned `height_emb`

| | learned `height_emb` (v2) | sin/cos PE (v4) |
|---|---|---|
| Init informativeness | random — must learn every bin | exact closed-form from step 0 |
| Cross-view consistency | per-bin index, no physical meaning | normalised by GLOBAL span → same basis at same physical Z |
| Generalises across z_range | no (needed `--refit_z_range`) | yes (PE is a function of metres) |
| Param count | Z × h_enc | 0 |
| Inductive bias for smooth z | none | Fourier basis is naturally smooth |

The "cross-view consistency" row is the biggest practical win. v2's
learned embedding meant the model had to discover SEPARATELY that "bin 17
in view 0" and "bin 9 in view 1" refer to the same physical height. v4's
PE makes that identity explicit at init.

## Why adaptive per-view z range

The wrist cam rides the EE. With v2's fixed `(z_lo, z_hi) = (0, 0.4)` m
and 32 bins/view:

- Wrist cam is typically at 0.15–0.20 m above the table during grasping.
- The wrist's frustum cannot see anything above its own camera.
- 25 of the wrist's 32 bins were physically unreachable from that view.
- The 7 useful bins had ~6 cm width each.

V4 caps `z_hi_wrist = cam_z_wrist − cam_margin_m` per frame, so all 128
wrist bins land in the reachable range. At cam_z = 0.20 m, that's ~1.5 mm
per bin. Combined with the shared PE basis this delivers sub-millimetre
height inference without a decoder change.

Scene cam is high (z ≈ 1.0 m for the YAM scene cam) and static, so
`min(z_hi=0.4, 1.0 − 0.01) = 0.4` — scene range unchanged. Sanity-check
in code: scene cam should always be above z_hi global.

## Hyperparameters (paper config)

| Parameter | Value | Notes |
|---|---|---|
| Backbone | DINOv3 ViT-S/16+ | 30 M params, frozen at low LR |
| `img_size` | 504 | DINOv3 ViT-S/16+ native multiple of 16 |
| `pred_size` | 64 | per-view feature-map resolution after bilinear-up |
| `feat_dim` | 32 | per-pixel feature dim |
| `n_height_bins` | 128 | per view (NZ = 256 total at N=2) |
| `height_enc_dim` | 16 | PE_z dim (= 2 × 8 octaves) |
| `n_window` | 32 | action chunk length |
| `frame_stride` | 4 | per-chunk subsample |
| `cam_margin_m` | 0.01 | wrist voxel grid stays 1 cm below cam |
| `batch_size` | 4 | fits 24 GB RTX 3090 |
| `mv_past_n` | 8 | past-trajectory conditioning depth |
| `lr` / `backbone_lr_mult` | 3e-4 / 0.1 | head 3e-4, DINO 3e-5 |
| `lr_warmup` / `lr_schedule` | 500 / cosine | |
| `max_iters` | 20000 | v4 benefits from 2× v2's budget |

## Voxel-count parity with v2

| | n_z/view | P | V = NZ · P² | uniform CE baseline |
|---|---|---|---|---|
| v2 | 32 | 128 | 1,048,576 | log(V · T=32) ≈ 17.33 |
| v4 | 128 | 64 | 1,048,576 | log(V · T=32) ≈ 17.33 |

Loss numbers are directly comparable.

## Version history

| Version | File | Status | Key idea | Why |
|---|---|---|---|---|
| v1 | `lib/model.py` | shipping | single-view DinoVolumeScene + factored YX × Z × T head | original PARA |
| v2 | `lib/model_volumetric_multiview.py` | shipping | multi-view, per-voxel MLP fusion across cross-projected views, learned height_emb | first multi-view |
| v3 | `lib/model_volumetric_v3.py` | **abandoned** | factored sum-of-views (product of experts in log space), no per-voxel MLP | Sacrificed cross-view nonlinear fusion for cheap LSE. Real-rollout regression vs v2. Documented for completeness; do not deploy. |
| v4 | `lib/model_volumetric_v4.py` | **production** | v2 + sin/cos PE + adaptive per-view z range | Height precision was the missing piece in v2. v4 keeps v2's MLP fusion intact. |

## Backbone-experiment hooks

v4 is the head; the backbone is swappable. The "backbone experiments
track" replicates v4 with three backbone choices for the paper:

| Variant | Backbone | File | Status |
|---|---|---|---|
| v4 | DINOv3 ViT-S/16+ (HF, frozen at low LR) | `lib/model_volumetric_v4.py` | shipping |
| v4_da3 | Depth-Anything-3 ViT (multi-view ViT with built-in cross-attn + camera tokens, NO DPT head) | `lib/model_volumetric_v4_da3.py` | new |
| v4_vlm | InternVL 2.5-1B vision tower (vision tokens trainable, LLM frozen, text prompt = dataset name) | `lib/model_volumetric_v4_vlm.py` | new |

The voxel-fusion head + global MLP + per-voxel grip/rot are byte-identical
across the three. Only the backbone differs. Same hyperparameters where
the backbone allows.

## Files this touches at change-time

- `lib/model_volumetric_v4.py` — the model class + V4 helpers
- `lib/V4_DESIGN.md` — this doc
- `lib/model_volumetric_multiview.py` — `multiview_losses` (v4 reuses)
- `lib/inference.py` — `kind == "dino_volume_scene_v4"` dispatch
- `deploy_with_action_chunking.py` — `is_v4` detection + same is_multiview path
- `train.py` — `--v4` CLI flag + model construction + ckpt model_kind