# Any4D — YAM finetune bootstrap doc

**Owner:** yams_any4d (forked from yams on 2026-07-08)
**Status:** SSO-blocked on repo clone (2026-07-08). Public-info draft below.

---

## What Any4D is (best current understanding, unverified against private code)

`TRI-ML/Any4D` is SAML-SSO-gated on GitHub, so the private codebase is not yet visible from the VPS. The closest public analog appears to be the ICLR 2026 paper **"Geometry-aware 4D Video Generation for Robot Manipulation"** (Stanford + Toyota Research Institute):

- arXiv: https://arxiv.org/abs/2507.01099
- Project page: https://robot4dgen.github.io/
- Public code (external mirror, not TRI-org): https://github.com/lzylucy/4dgen
- Data + ckpts (Stanford real lab): https://real.stanford.edu/4dgen
- HF weights: https://huggingface.co/Zeyi

If `TRI-ML/Any4D` **is** the same architecture (or a superset), then the mapping below applies. **Verify against the private repo as soon as SSO access lands.**

### Architecture (per the public paper)

| Layer | Component |
|---|---|
| Backbone | **Stable Video Diffusion (SVD)** U-Net, latent-space |
| Input encoders | Separate VAEs for **RGB** and **pointmaps** (Xie et al. depth conditioning) |
| Multi-view fusion | **12 added cross-attention layers** between per-view U-Net decoder branches |
| Conditioning inputs | Historical RGB-D from **two camera views** (h=10 frames repeated) + gripper binary mask (SAM2 in real-world) |
| No explicit inputs | Camera pose, action sequence, language |
| Output | (h′) future RGB frames + pointmap frames, per view |

**Downstream use:** actions are extracted post-hoc via pose tracking on the generated video — the model itself is action-free.

### Training data format (paper)

- **Resolution:** RGB + depth at **256 × 320**, downsampled to 32 × 40 in VAE latent
- **Cameras per demo:** 16 camera poses recorded; train sample 12, hold out 4 for eval
- **Simulation tasks (Drake / LBM):** `StoreCerealBoxUnderShelf` (25 demos), `PutSpatulaOnTable` (22), `PlaceAppleFromBowlIntoBin` (20)
- **Real-world set:** 5 demos of `PutSpatulaOnTable`, used for sim2real transfer
- **Hardware envelope (paper):** single RTX 3090 sufficient for eval; training scale not published

### Evaluation metrics (paper)

- Fréchet Video Distance (FVD) for RGB
- AbsRel + δ₁ for depth accuracy
- Cross-view mask IoU (gripper) as consistency check
- Downstream: robot manipulation success rate on the same task

---

## Format-mismatch delta (YAM → Any4D input schema)

### What YAM recordings currently ship

From inherited yams context, our YAM `record.py` per-rollout dump (yukon `~/recordings/<task>/<ep>/`):

| Field | Value |
|---|---|
| Scene RGB cam | ZED HD720 (1280×720) — recently HD1080 debated, memory says record still at 720p by default |
| Wrist RGB cam | ZED HD720 (1280×720) |
| Ego RGB cam | (optional in bimanual sessions) |
| Depth | ZED SDK depth per frame, saved as .npy / .mp4 depending on recorder version |
| Cameras per demo | 2–3 (scene + wrist, sometimes + ego) — **NOT 16 like paper** |
| Actions | 7-DoF joint targets + gripper state, per timestep |
| Language | Task name string per session (no per-frame captioning) |
| ArUco boards | Fiducial exo bases, IDs 217-234 (see [[fiducial_exo]]) |
| Rollout length | Variable, tens to hundreds of frames at 30 fps (rerun) |

### Deltas we will need to bridge

1. **Camera count:** paper wants 12 train views per demo, we have 2 (scene+wrist). Options: (a) treat 2-view sessions as the entire supervised signal — retrain the cross-attention stack with `n_views=2`; (b) synthetically render extra views from calibrated depth (would need proper extrinsics — `yam_calib` domain); (c) collect a new multi-view dataset (much bigger project).
2. **Resolution:** paper 256×320, we record 1280×720 → downsample to 256×320 at data-loader time (cheap; can also try higher, since we have room on 3090 latents).
3. **Depth format:** paper expects pointmaps (Xie et al.). We currently store per-pixel depth. **Need a preprocessing step** to convert depth + intrinsics → pointmaps in the format Any4D's VAE expects.
4. **Gripper mask:** paper uses SAM2. We may need to run SAM2 once per rollout on wrist views to generate the mask sequence — that's a one-time preprocessing pass.
5. **Sequence length:** h=10 historical, h' future prediction. Compatible with our recordings; just a windowing choice at data-loader time.
6. **Task label / conditioning:** paper is task-conditioned via which demos went into training, not via language token per-frame. We can either (a) train one Any4D per YAM task (pickplace, teapot, cup_stacking, etc.), or (b) mix all tasks together and skip explicit conditioning.
7. **Action-conditioning:** paper is action-free (extracts actions post-hoc). If the private `TRI-ML/Any4D` adds action-conditioning as an input, we already have YAM joint targets ready to feed.

### What we DON'T know until SSO unblocks

- Exact data-loader API surface for `TRI-ML/Any4D` — dataset class signature, expected on-disk layout
- Whether `TRI-ML/Any4D` differs from the public `4dgen` code (added components, different backbone, different conditioning)
- What checkpoints ship with the private repo — SVD-derived, or from-scratch?
- Whether the training loop supports the RTX 3090 budget or needs bigger cards

---

## Compute + storage plan

Per 2026-07-06 storage policy: **code on VPS, big files on compute host.**

- Code: `/data/cameron/repos/Any4D/` on VPS (blocked until SSO)
- Ckpts + eval videos: on **Puget** (2× RTX 3090, currently UP). Suggested paths:
  - `/home/cameronsmith/yam_para/checkpoints/any4d_<task>/`
  - `/home/cameronsmith/yam_para/any4d_eval_videos/<run_name>/`
- Training data: on **Yukon** (`/home/robot-lab/recordings/<task>/`) — rsync task-specific slice to Puget at run start
- wandb project: `yam_any4d` (new — proposed; ping Cameron)

---

## Key open questions (answered in outbox.md; pending Cameron)

1. **Repo access.** `TRI-ML/Any4D` requires SAML SSO. My key `id_ed25519.pub` on VPS is authenticated as `cameronosmith` per `ssh -T git@github.com`, but the TRI-ML org needs SSO authorization for it. Please authorize the key (browser flow in GitHub settings) OR paste a TRI-ML-scoped PAT with SSO grant.
2. **Which YAM task to finetune on first?** Pickplace (most data, ~22 eps) is the easiest; teapot / cup_stacking are more visually distinct and would tell us more about generalization. Recommendation: pickplace_redo_slow as the smoke test, teapot as the first "real" run.
3. **Metric priority.** Is the paper-figure endpoint (a) FVD + depth quality on held-out held-out YAM rollouts, or (b) downstream manipulation success via post-hoc action extraction? Different training + eval loops.
4. **Camera count.** Do you want us to (a) live with n_views=2 (scene+wrist), (b) synthesize extra views from calibrated depth (needs yam_calib coord), or (c) plan a multi-view data collection push?
5. **wandb project + naming.** New `yam_any4d` project, or fold into existing `yam_train_simple`?

---

## Related vault docs

- [[data_recording]] — YAM rollout / calibration schema
- [[calibration]] — real-side extrinsics (needed for depth→pointmap)
- [[fiducial_exo]] — real-side ArUco boards
- [[simulation]] — sim-derived data option

---

## 2026-07-09 — FIRST VIDEO GENERATED ✅ (updated after repo access + successful run)

**Correction to the public-info draft above:** `TRI-ML/Any4D` is NOT the SVD-based public `4dgen` — it's **Any4D v2**, built on NVIDIA **cosmos-predict2** (2B/14B DiT world models) + imaginaire trainer + AnyData webshard loader. Entries/streams abstraction, multi-view DiT (`custom/any4d/a4d_network.py`), tasks realized by masking entry subsets.

**YAM specifics:**
- YAM data already webbed on S3: `s3://tri-ml-sandbox-16011-us-west-2-datasets/cv_webbed/videos/YAMxdof/split_all__ILA_32_str3_res-1x512/` (bimanual XDOF teleop, 5 cams recorded / 4 used: top_left, top_right, left_wrist, right_wrist; 30fps webbed at stride 3 → 10 fps; labels rgb+language+action joint_raw)
- YAM-finetuned ckpt (Basile's): `s3://…/any4d/a4d2/sm/06-30-09-28_s52h_yam4_r480_4n/model/iter_000044000_005632000.pt` (4.19 GB, 4-view, 480res)
- Exp config: `custom/experiment/basile/rwm4_yamxdof.py`; dataset yaml `custom/config/anydata/web/debug/yamxdof_ila4_41.yaml`
- First generated video: https://omidlab.net/browse/agents_stuff/agents/yams_any4d/first_yam_video.mp4?raw

**Working env: russet** (`~/miniconda3/envs/a4d`, RTX 4090) — full runbook + 7 required fixes in `agents_stuff/agents/yams_any4d/outbox.md` (2026-07-09 entry). Key gotchas: repo needs py3.12 f-string syntax (we patched 7 lines to stay on 3.11 for AnyData); T5-11B doesn't fit on 24 GB alongside the DiT → `text_encoder_path='disable'` (zero text embeddings, valid); `/any4d` local_root hardcode → russet exp variant; ALWAYS `MAX_JOBS=8` on compiles (uncapped build crashed yukon).

**Yukon caveat:** yukon GPUs are RTX PRO 6000 Blackwell (sm_120), NOT H100 — repo-pinned torch 2.6 has no kernels for them. Needs torch 2.11+cu128 (env `a4d-bw` half-built there). Prefer russet (sm_89 works with pinned stack) until the repo moves to newer torch.

## 2026-07-09 (later) — OUR raiden data through Any4D ✅

Full raiden→AnyData→Any4D pipeline working; videos at agents_stuff/agents/yams_any4d/raiden_yam_video{1,2}.mp4 (omidlab ?raw URLs). Converter: `AnyData/anydata/converters/raiden_yam.py`. Recipe + runbook + 6 webbing gotchas: see yams_any4d outbox 2026-07-09 entry. Rerunning on any other raiden task = change one path in the download step + YAMraiden.yaml.

## 2026-07-09 (final) — LoRA finetune loop CLOSED ✅

Full cycle proven on russet: our raiden data → LoRA (r16, 2000 iters, ~35 min, 19 GB VRAM) → finetuned eval videos. wandb cameronsmithbusiness/yam_any4d. Ckpt: raiden run dir `model/iter_*.pt`. All gotchas + runbook in yams_any4d outbox (2026-07-09). Next scale-up: more episodes (recording push) + yukon 96GB pair when revived (T5 on, bigger batch, maybe full FT).

## 2026-07-10 — pickplace_70 unconditional LoRA ✅ (yukon Blackwell pair)

Switched from raiden flip_pink_cup to the 70-ep single-arm cup pick-and-place, unconditional (task_probs world_model=0.0, pure forecast). New converter `AnyData/anydata/converters/yukon_yam.py` (jpg+pkl format, **resamples 15→10 fps** at conversion; web stride 1 — note no `_str1` suffix in webbed dir name). 2 views (scene→top_left + right wrist) → ~1.7 s/step on the 96GB pair, 3000 iters ≈ 85 min, T5 on. Videos: pp70_fc_video{1..4}.mp4 (finetuned) vs pp70_zeroshot_video{1..4}.mp4 (s52h base) in agents_stuff/agents/yams_any4d/. wandb run 07-09-19-49_pp70_fc_lora1 (group pickplace). Full runbook in yams_any4d outbox 2026-07-10.

## 2026-07-10 — LoRA eval gotcha: MERGE REQUIRED (noise-video bug + fix)

LoRA ckpts save unmerged peft keys (`base_layer` + `lora_A/B`); infer_anydata builds a plain model unless told otherwise, silently drops all wrapped weights → predictions are pure noise (PSNR ~12) while wandb train visuals look fine. Fix: offline merge W += B@A (α/r=1 for r16/α16), rename `base_layer.`→``, drop lora keys, eval merged .pt. Verified pp70: finetuned 25.6–29.5 PSNR vs 21.1 zero-shot. Merge snippet + full postmortem in yams_any4d outbox 2026-07-10. Applies to ALL LoRA runs (russet raiden ones were re-done too).

## 2026-07-10 (overnight) — v4 action head on DiT features ✅

Cameron's action-head plan implemented + trained overnight: V4Head (port of DinoVolumeSceneV4, scene-view voxel volume) reads pre-final-layer DiT features at the sampled diffusion sigma, joint with continued LoRA from the merged pp70 ckpt, pure-forecast task mix. Train CE collapsed to ~0 (vol 12.5→0, grip 3.5→0, rot 4.2→0.002) in 6000 iters — DiT features demonstrably encode EE position/gripper/rotation. NO held-out eval yet (that's next). Full arch, file map, 6 integration gotchas + panels in yams_any4d outbox 2026-07-10. wandb run 07-10-01-17_pp70_v4head1.

## 2026-07-11 — cached-feature pipeline VALIDATED: ~2cm held-out EE from video-model features

Scene-only derisk pipeline (Cameron's design) complete: frozen scene-LoRA video model → teacher-forced feature cache (3263 windows, σ=0.05) → standalone fp32 v4head → **held-out (8 unseen eps) EE mean error 1.96cm**, grip solid, rot ≈ baseline (weak — wrist view likely needed). Full runbook + caveats in yams_any4d outbox 2026-07-11. wandb runs: 07-10-18-50_pp70_scene_lora1, m9xpjxz5, ad2qb98r.