# Outbox — yams_any4d

Write progress, plans, and cross-agent messages here.

---

## 2026-07-08 — GOAL: generate a video using the s52h yam4 ckpt

Cameron's directive: generate a video using YAM weights at `s3://tri-ml-sandbox-16011-us-west-2-datasets/any4d/a4d2/sm/06-30-09-28_s52h_yam4_r480_4n/model/iter_000044000_005632000.pt`. Take my time.

### Blockers (unblock ASAP)

~~1. AWS SSO creds~~ ✅ received (both `tri` + `tri2` profiles, both work; ckpt = 4.19 GB)
~~2. HF_TOKEN~~ ✅ received (HF auth as `omidcameronsmith`)

### Progress

| Step | Status |
|---|---|
| Clone Any4D to VPS + confirm SSO auth | ✅ 38 MB at `/data/cameron/repos/Any4D/` |
| Clone AnyData to VPS | ✅ 13 MB at `/data/cameron/repos/AnyData/` |
| Locate exp config for s52h yam4 ckpt | ✅ `custom/experiment/basile/rwm4_yamxdof.py` (uses `basile/` per README) |
| Rsync Any4D + AnyData → yukon local | ✅ `/home/robot-lab/any4d_work/Any4D/` |
| Install Miniconda on yukon | ✅ conda 26.5.3 at `~/miniconda3` |
| Accept conda TOS for main + r channels | ✅ |
| Create `cosmos-predict2` conda env (py 3.10, CUDA 12.4, gcc 12.4) | ✅ |
| `pip install -r requirements-conda.txt` (torch 2.6.0 + deps) | 🔄 in progress (~3-5 min) |
| `pip install flash-attn==2.6.3 --no-build-isolation` | pending |
| Symlink CUDA includes | pending |
| `pip install transformer-engine[pytorch]==1.13.0` | pending |
| `pip install "git+facebook/pytorch3d@stable"` | pending |
| `pip install -e externals/AnyData` + inner webdataset | pending |
| Install AWS CLI locally (no sudo) | pending |
| Download s52h yam4 ckpt to yukon local | **BLOCKED on AWS creds** |
| Download Cosmos-Predict2-2B base weights | **BLOCKED on HF_TOKEN** |
| Run `infer_anydata.py` on YAMxdof + ckpt | pending |
| Video out at `/home/robot-lab/any4d_work/output/first_yam/` | pending |

### Target inference command (once installed + downloaded)

```bash
cd /home/robot-lab/any4d_work/Any4D
source ~/miniconda3/bin/activate cosmos-predict2
export CUDA_HOME=$CONDA_PREFIX
export AWS_PROFILE=tri  # or paste creds into ~/.aws/credentials
export HF_TOKEN=<paste>

python custom/eval/infer_anydata.py \
    --exp_cfg=custom/experiment/basile/rwm4_yamxdof.py \
    --ckpt_path=s3://tri-ml-sandbox-16011-us-west-2-datasets/any4d/a4d2/sm/06-30-09-28_s52h_yam4_r480_4n/model/iter_000044000_005632000.pt \
    --dset_cfg=custom/config/anydata/web/debug/yamxdof_ila4_41.yaml \
    --output_dir=/home/robot-lab/any4d_work/output/first_yam \
    --num_samples=3 --num_steps=25 --stop_after=6
```

**Warning per README:** ckpts from 2026 may need `legacy_network_behavior=2` + `legacy_logistics_behavior=2` in `any4d_config`. If first run errors on state-dict mismatch, edit `rwm4_yamxdof.py` to add those.

**Autoregressive rollout** (for longer videos): add `--run_autoregressive=1 --cond_frames_raw=13 --num_segments=2 --extrapolation_strategy=backtrack`.

### Path context

- Any4D repo (yukon): `/home/robot-lab/any4d_work/Any4D/`
- Conda env: `/home/robot-lab/miniconda3/envs/cosmos-predict2/`
- Ckpts: `/home/robot-lab/any4d_work/ckpts/`
- HF cache: `/home/robot-lab/any4d_work/hf_cache/`
- Output: `/home/robot-lab/any4d_work/output/`
- GPUs: 2× H100 (96 GB each), both mostly free

— yams_any4d

## 2026-07-09 — ✅ FIRST YAM VIDEO GENERATED (russet)

**Gallery video:** https://omidlab.net/browse/agents_stuff/agents/yams_any4d/first_yam_video.mp4?raw
**Full outputs on russet:** `~/any4d_work/output/first_yam/` (gal/cat/act mp4s + per-view entry videos: x0=GT, y0_pred=prediction, xt=noised, rgb0-3 = 4 cams)
**Ckpt:** s52h_yam4_r480_4n iter 44000 (4-view YAM world model)
**Data:** YAMxdof webshards streamed from S3 (bimanual YAM teleop, 4 cams, 41 frames @ 10 fps)

### Runbook (russet)
```bash
ssh robot-lab@100.123.182.78
cd ~/any4d_work/Any4D && source ~/miniconda3/bin/activate a4d
export AWS_PROFILE=tri HF_TOKEN=<token> HF_HOME=~/any4d_work/hf_cache PYTHONPATH=$PWD
python custom/eval/infer_anydata.py \
  --exp_cfg=custom/experiment/basile/rwm4_yamxdof_russet.py \
  --ckpt_path=$HOME/any4d_work/ckpts/s52h_yam4_iter_44000.pt \
  --dset_cfg=custom/config/anydata/web/debug/yamxdof_ila4_41.yaml \
  --output_dir=$HOME/any4d_work/output/<name> \
  --num_samples=1 --num_steps=25 --stop_after=6
```

### Fixes required (all applied; VPS master + russet in sync)
1. **PEP 701 f-strings** (7 lines, 3 files) — repo needs py3.12 syntax; we're on 3.11 for AnyData. Inner quotes swapped.
2. **TE 1.13.0** — needs `pip install pybind11` + `--no-build-isolation`.
3. **apex** — required by imaginaire fused_adam import; built from git with `--cpp_ext --cuda_ext`, MAX_JOBS=8.
4. **sklearn, lpips, plotly** — missing from requirements.
5. **local_root** — `/any4d` hardcoded default unwritable; russet exp variant sets `~/any4d_work/local_root`.
6. **T5 disabled** (`text_encoder_path='disable'`) — T5-11B doesn't fit alongside DiT on 24 GB; a4d_vae handles None encoder with zero embeddings (matches training text-dropout). VRAM: 11 GB with T5 off.
7. **MAX_JOBS=8 on ALL compiles** — uncapped flash-attn build OOM-crashed yukon (needed physical power-cycle).

### Env
- russet `~/miniconda3/envs/a4d`: py3.11, torch 2.6.0+cu124, flash-attn 2.6.3, TE 1.13.0, pytorch3d 0.7.8, apex, AnyData editable
- conda cuda-toolkit 12.4 + gcc 12.4 in-env (no system deps)
- Yukon note: Blackwell RTX PRO 6000 (sm_120) needs torch 2.11+cu128 — env `a4d-bw` exists there, unfinished; revisit if yukon returns

— yams_any4d

## 2026-07-09 — ✅ ANY4D ON OUR OWN RAIDEN DATA (flip_pink_cup)

**Videos:** https://omidlab.net/browse/agents_stuff/agents/yams_any4d/raiden_yam_video1.mp4?raw (+ video2)
**Pipeline (reusable for any raiden task):** raw S3 raiden session → `AnyData/anydata/converters/raiden_yam.py` (ours) → web_local 32b3 --per_camera → `Any4D custom/config/anydata/web/debug/yamraiden_ego3.yaml` → infer_anydata.
**Cameras:** ego→top_left_camera (primary), wrists→wrists, scene excluded (Cameron 2026-07-09). OOD vs ckpt (different mounts).

### Runbook (russet)
```bash
# 1. Download session(s): boto3 sync from s3://tri-ml-datasets-uw2/raiden_datasets/raw/<task>/ to ~/any4d_work/raw/<task>/
# 2. Convert (a4d env):
cd ~/any4d_work/Any4D/externals/AnyData
ANYDATA_LOCAL_ROOT=/tmp/anydata_root python anydata/converters/raiden_yam.py ~/any4d_work/raw/<task> --videos --num_procs 0 --restart
# 3. Web (edit configs/YAMraiden.yaml path first):
bash ~/any4d_work/run_web.sh   # wraps: web_local.py YAMraiden/raiden 32b3 --videos --from_videos --local_path /tmp/anydata_root --per_camera + AWS env exports
# 4. Infer:
bash ~/any4d_work/run_raiden_infer.sh   # exp_cfg=rwm4_yamxdof_russet.py, dset=yamraiden_ego3.yaml
```

### Webbing gotchas (hard-won)
1. **Hyphen-free ANYDATA_LOCAL_ROOT required** — clean_string mangles `-`→`_` across the whole dst path (`/home/robot-lab`→`/home/robot_lab` = EACCES). Use `/tmp/anydata_root` symlink → real storage.
2. **`--per_camera` required for multi-cam webbing** — default mode parses camera name from split filename and gets garbage (`'spli'`), silently webbing 0 sequences.
3. **AWS creds must be ENV VARS** — `Paginator()` reads `os.environ['AWS_ACCESS_KEY_ID']` directly in every worker; profile file isn't read. Export from ~/.aws/credentials before webbing.
4. **`--videos --from_videos` + `--local_path` are required flags** (README example is stale).
5. **Webbing must run via nohup'd script** — takes ~15 min for 2 eps × 3 cams (3,969 shards); an ssh-foreground run dies with the session.
6. **Don't pkill with a pattern matching your own launcher** (cost us one run).

### Converter notes
- `raiden_yam.py`: npz `follower_{l,r}_joint_{pos,vel,eff}` → `{left,right}_arm_proprio` (18d); gripper triplets → eef_proprio (3d); `joint_cmd[:6]`/`[6:7]` → arm/eef_leader. Language from metadata.json task_instruction. Nearest-timestamp sync of 91.5 Hz robot onto 30 fps ego timeline. No intrinsics (YAMxdof webbed without them too).
- SVO2s decoded inline via pyzed (installed into a4d env from /usr/local/zed/get_python_api.py).
- 2 of 4 flip_pink_cup sessions usable (other 2 have 0-byte right_wrist SVO2s).

— yams_any4d

## 2026-07-09 (later) — FPS fix + LoRA training live

**FPS bug found+fixed:** first webbing used `32b3` = 32-frame/3-snippets-per-tar (NOT stride) → data at 30 fps vs model's 10 fps training. Re-webbed with `--stride 3` → `raiden__ILA_32b3_str3_res-1x512`, inference now reports `FPS from data_batch: 10.0`. Corrected zero-shot videos: raiden_str3_video{1,2,3}.mp4 (omidlab ?raw).
**Upstream patch:** web_local.py merge validator off-by-stride (last tar end vs num_frames//stride*stride) — tolerance patch in our AnyData copy (VPS+russet), marked NOTE(yams_any4d).

**LoRA run 1** (sanity/overfit, 2 episodes):
- Exp: `custom/experiment/basile/rwm4_yamraiden_lora.py` (4-view arch kept; 3 cams masked fine)
- `++model.config.train_architecture=lora` (rank 16, α16, q/k/v/output_proj/mlp targets — cosmos-predict2 built-in, peft)
- Warm-start s52h_yam4 iter44k; lr 1e-4; max_iter 2000; bs1; T5 disabled; wandb off
- **Step 1 peak VRAM 19.1 GB** on the 4090 ✓
- Run dir: russet `~/any4d_work/local_root/a4d2/raiden/07-08-22-25_lora1/`
- Launcher: `~/any4d_work/run_lora.sh`

Next: on completion → inference with LoRA ckpt → before/after galleries. Scaling story: 2 eps is loop-validation only; real finetune wants a recording push (ping yams/yam_calib for a flip_pink_cup-style collection session with N≥20 episodes).

— yams_any4d

## 2026-07-09 — ✅ LORA FINETUNE COMPLETE + EVAL VIDEOS

**Finetuned galleries:** raiden_lora_video{1..4}.mp4 (omidlab ?raw) — compare vs zero-shot raiden_str3_video{1..3}.mp4
**Training:** 2000 iters, LoRA r16 on s52h_yam4, our 2 flip_pink_cup eps @10fps, final loss ~0.24 (from 0.42 start), ~35 min on russet 4090 (peak 19.1 GB). wandb: cameronsmithbusiness/yam_any4d (loss + train galleries).
**Eval ckpt:** `~/any4d_work/local_root/a4d2/raiden/07-09-00-41_lora1/model/iter_000001500_000001500.pt`

### Run history + gotchas (runs 1-5)
1. Run 1-2: validation crashed (`Tp=34` non-1+4k tail window) → disabled run_validation for sanity runs.
2. Run 3: SAME bug hit TRAINING at step 1517 (random tail window `Tp=23`) → patched `custom/dataloader/utils.py` to trim windows to largest valid 1+4k length (NOTE(yams_any4d) marker; VPS + russet).
3. Run 4: full 2000 iters BUT no ckpt — `save_iter=2000` never fires because the loop ends at sample 1999. → save_iter=500.
4. Run 5: complete, ckpts at 500/1000/1500.
5. Eval gotcha: ckpts live under **`model/`** (registry layout), NOT `trainer/` (that's grad-scaler state). a4d_surgery loads the LoRA ckpt fine from model/.

### What's proven end-to-end
raw raiden S3 → converter → webbing (stride 3 = 10 fps) → zero-shot eval → LoRA finetune → finetuned eval. Every stage one command. Scaling = more episodes.

— yams_any4d

---

## 2026-07-10 — pickplace_70 UNCONDITIONAL LoRA ✅ (yukon 2×96GB, T5 on)

**What Cameron asked:** switch training from the raiden flip_pink_cup (AWS) data to the 70-ep cup pick-and-place, and make it NOT action-conditioned.

**Results (finetuned vs zero-shot, same 4 val samples, 2 views scene+right-wrist):**
- Finetuned (iter_2500, pure forecast): pp70_fc_video{1..4}.mp4
- Zero-shot s52h base on same data: pp70_zeroshot_video{1..4}.mp4
- All at https://omidlab.net/browse/agents_stuff/agents/yams_any4d/<file>?raw
- wandb: https://wandb.ai/cameronsmithbusiness/yam_any4d/runs/07-09-19-49_pp70_fc_lora1 (group pickplace)
- Training: 3000 iters, ~1.7 s/step (2 views ≈ 2.5× faster than the 3-view raiden run), ~85 min wall, 38 GB/GPU, loss →~0.05-0.13 band. Ckpts every 500 under run dir model/ (final save = iter_2500; the 2999-loop-end gotcha again).

**Pipeline (reusable for any yukon-format single-arm task):**
1. rsync raw eps yukon:~/any4d_work/raw/<task>/ (jpg + per-frame pkl format)
2. `ANYDATA_LOCAL_ROOT=/tmp/anydata_root python anydata/converters/yukon_yam.py ~/any4d_work/raw/<task> --videos --num_procs 8 --restart` (a4d-bw env)
3. `bash ~/any4d_work/run_pp70_prep.sh` pattern: web `YAMyukon/yukon 32b3 --stride 1 --videos --from_videos --local_path /tmp/anydata_root --per_camera`
4. Train: `~/any4d_work/run_lora_pp70.sh` (experiment=any4d_rwm4_pp70_lora_yukon)
5. Eval pair: `~/any4d_work/run_pp70_eval.sh`

**New gotchas:**
- **pickplace_70 is ~15 fps, not 30** — no integer stride hits 10 fps, so `yukon_yam.py` resamples the reference timeline to exactly 10 fps at conversion (nearest-frame ≤33 ms error); web with --stride 1.
- **stride 1 webbing gets NO `_str1` suffix** in the output dir name (`yukon__ILA_32b3_res-1x512`), unlike `_str3`.
- **Unconditional = task_probs world_model=0.0** (pure forecast). The AdaLN action entries stay in the config for ckpt compatibility; forecast masks them.
- **pkill self-match ×3**: any literal appearing later in the same ssh command (sed args!) can match `pkill -f`. Bracket EVERY pattern (`torchru[n]`) and keep kills in their own ssh invocation.
- Converter mapping: single right arm real, left channels zero-filled; joints(7) serve as both proprio (pos only, vel/eff zeros) and leader action.

**Killed runs (ckpts kept on yukon):** 07-09-17-39_lora_yukon1 (raiden 3-view, killed @598, iter_500 saved); 07-09-19-13_pp70_lora1 (action-conditioned pp70, killed @~1400, iter_500+1000 saved).

— yams_any4d

---

## 2026-07-10 (later) — CORRECTION: LoRA eval requires a merge step; noise bug fixed ✅

**Bug (found by Cameron):** published finetuned eval videos were pure noise while wandb train visuals looked fine. Root cause: LoRA training saves ckpts in **unmerged peft format** (280 modules as `X.base_layer.weight` + `X.lora_A/B.default.weight`). `++model.config.train_architecture=lora` was only a CLI override at train time, so infer_anydata built a PLAIN model; attention/MLP keys silently failed to load → DiT ran with garbage attention → noise. Zero-shot evals (plain-key ckpts) were unaffected — that's why only finetuned outputs were noise.

**This also invalidated the earlier russet raiden "finetuned" videos** (raiden_lora_video{1..4}.mp4) — their prediction panes were noise too; I'd shipped them without visually checking the prediction quadrant. Corrected.

**Fix — mandatory post-training merge step (script pattern, ran on both boxes):**
```python
sd = torch.load(ckpt, map_location="cpu")
for ka in [k for k in sd if k.endswith("lora_A.default.weight")]:
    base = ka[:-len("lora_A.default.weight")]
    A = sd.pop(ka).float(); B = sd.pop(base+"lora_B.default.weight").float()
    kw = base+"base_layer.weight"
    sd[kw] = (sd[kw].float() + B @ A).to(sd[kw].dtype)   # scale = alpha/r = 16/16 = 1
for k in list(sd): 
    if ".base_layer." in k: sd[k.replace(".base_layer.",".")] = sd.pop(k)
```
Merged ckpts: yukon `~/any4d_work/ckpts/pp70_fc_lora2500_merged.pt`, russet `~/any4d_work/ckpts/raiden_lora1500_merged.pt`.

**Verified results (pp70, merged, frames visually checked):** finetuned PSNR 25.6–29.5 / SSIM 0.85–0.89 vs zero-shot 21.1 / 0.51 vs broken 11.7 / 0.13. pp70_fc_video{1..4}.mp4 overwritten with real predictions. raiden_lora_video{1..4}.mp4 re-eval running on russet with its merged ckpt.

**Lesson recorded:** always eyeball the prediction pane of at least one gallery before publishing; PSNR in the strip catches this instantly (11-12 = noise).

— yams_any4d

**2026-07-10 addendum:** raiden_lora_video{1..4}.mp4 refreshed from the merged russet ckpt (raiden_lora1500_merged.pt) — prediction panes visually verified coherent (bimanual scene, not noise). All published LoRA eval videos are now from merged checkpoints.

---

## 2026-07-10 (overnight) — V4 ACTION HEAD ON ANY4D FEATURES: TRAINED ✅

**Cameron's ask:** bolt our v4 action model onto the video model's last-layer features (differentiable, at the sampled diffusion timestep), continue LoRA, v4-style visualizations, balanced losses.

**Result — the DiT features carry manipulation state, decisively:**
| loss | random baseline | final (train) |
|---|---|---|
| v4h_vol (EE position CE over 262k voxels) | 12.47 | **~0.00** |
| v4h_grip (32 bins) | 3.47 | **~0.00** |
| v4h_rot (64 kmeans clusters) | 4.16 | **0.002** |

Total train loss 8.1 → 0.13 over 6000 iters (~3h, 2×96GB, ~1.9s/step, 39GB/GPU).
wandb: https://wandb.ai/cameronsmithbusiness/yam_any4d/runs/07-10-01-17_pp70_v4head1
Run dir: yukon `~/any4d_work/local_root/a4d2/pickplace/07-10-01-17_pp70_v4head1` (ckpts model/iter_500..5500; contain LoRA + v4head weights — REMEMBER merge step for the DiT part at eval).
CAVEAT: train-set numbers (70 eps, no held-out split) — memorization not excluded; held-out eval is the required next step before celebrating generalization.

**Architecture (v1):** pre-final-layer DiT features (B,Tl,Hp,Wp,2048) per view stashed each forward at the SAMPLED sigma → per-frame 1×1 proj (64ch) → mean+max temporal pool → fuse conv → 64×64 scene feature map → scene-cam voxel volume (64 z-bins over (0,0.5)m, sin/cos height PE, per-voxel MLP; grid tiled from scene view so no grid_sample) → dot-product volume logits (B,41,64,64,64) + global grip/rot heads. Wrist features via mean-pool into global query. GT from data_batch['a4d_raw']['action'] (action_format_v2 EE pose via repo's own yam_fk — no re-webbing needed!). Mean rig calibration (v4head_prep.npz: rot kmeans64, grip range, z range, mean K/T_w2c).

**Files:**
- `custom/any4d/v4_head.py` (head + targets + losses), `custom/any4d/v4_head_viz.py` (PCA strip / GT-vs-pred keypoints / bin grids, v4-trainer style)
- Patches: `a4d_model.py` (`_v4head_compute` after compute_loss; panels in Any4DModel.training_step visuals; wandb.Image branch + guard), `a4d_network.py` (head init + feature stash), `a4d_config.py` (v4head_* fields), `video2world_model.py` (post-peft unfreeze), `text2image_dit.py` (v4head as own FSDP unit)
- Exp: `custom/experiment/basile/rwm4_pp70_v4head_yukon.py` (from merged pp70 fc ckpt, pure forecast so head can't shortcut via action conditioning); launcher yukon `~/any4d_work/run_v4head.sh`
- Prep: `~/any4d_work/v4head_prep.npz` (regen script in transcript; kmeans on FK quats of all eps)

**Panels (visuals_train/v4head/* on wandb, samples in agents_stuff):** v4head_panel_kp_step2.png (random scatter at step 2) → v4head_panel_kp.png (converged: pred coincides with GT at grasp point); v4head_panel_pca.png (features cleanly segment table/robot/background); v4head_panel_{grip,rot}.png (one-hot rows matching GT outlines).

**6 gotchas burned into this integration (5 crashes):**
1. Head must follow module dtype (trainer casts bf16), geometry fp32, CE fp32.
2. Registered BUFFERS get bf16-cast too — cast at use (rot_centroids einsum).
3. peft inject freezes ALL non-adapter params — re-unfreeze head after injection, BEFORE the fp32 upcast loop.
4. Head called outside DiT forward ⇒ must be its OWN fully_shard unit or params stay sharded DTensors.
5. Gate png writes to rank 0 — concurrent same-path writes corrupt files (and wandb.Image then kills training; now guarded).
6. a4d_model has TWO training_step visuals blocks (VidarModel legacy + Any4DModel) — patch the Any4DModel one.
Plus: head submodule names must avoid peft target suffixes (q_proj etc.).

**Next steps (my recs, in order):** (1) held-out split eval (webbing has 70 eps — hold out ~8, re-web or subset json); (2) fixed-low-sigma deploy readout path (features from a single denoise call on real frames + rollout); (3) per-episode calibration + wrist-view voxel geometry v2; (4) per-latent-frame voxel features (temporal pooling is v1's biggest info loss); (5) compare against puget v4 baseline numbers on the same episodes.

— yams_any4d

---

## 2026-07-10 — CORRECTION + FIX: v4head1 trained on ZEROED targets (Cameron caught it)

Cameron noticed rot/grip bins constant across batches/timesteps. Root cause found:
`a4d_raw['action'] = input*input_mask + output*output_mask` and under PURE-FORECAST task
mix (world_model=0.0 — my own anti-shortcut choice) unified_anyact sets cond_frames_action
= total_frames_action = 0 → both masks zero → the tensor I used as GT was ALL-ZERO every
batch. Symptoms all consistent: grip bin 0 = clamp of (0−g_lo), one constant rot bin =
quat of zero rot6d, GT keypoints at arm-base origin (xyz=0), and vol CE→0 was the head
memorizing a constant voxel — **v4head1's "features carry action info" result is INVALID.**
Data itself verified healthy at every stage (converter npz, webbed tars, AnyDataset sample:
41×20 with metres-scale xyz, varying rot6d, grip 0.04–1.0).

**Fix:** `prepare_action_anydata` now always exports `a4d_raw['action_gt_full']` (unmasked
output stack); `_v4head_compute` reads that key + degenerate-GT warning every 500 steps
(the old step≤2 z-median check existed but its print never surfaced — now louder + abs-sum
check). Panels also upgraded: 2x render, bigger markers, GT trajectory line.
**Rerun: pp70_v4head2** (same config otherwise). Lesson: an aux head reading trainer
tensors must check for TASK MASKING; and CE→0 in hours on 262k classes should have been
treated as a red flag, not a win.

— yams_any4d

---

## 2026-07-11 — v4head runs 6+7 (multiview per-timestep, Cameron's spec) — postmortem + pivot

**v6 head (Cameron's spec, implemented faithfully):** multiview scene+wrist, per-latent-timestep
volumes (11 volumes, 4-action chunks → 44→41), cross-view sampling both views at every voxel,
wrist grid at frame-0 pose (T_c2w = T_base_ee(0)·hand_eye — rot6d-rows convention verified via
wrist kp panel), production params (pred 64, 128 z-bins/view → V=1.05M, rot kmeans 256, grip 0..1),
adaptive per-view z. Geometry verified visually in BOTH views at step 2.

**Run 6 stall:** all losses pinned at baselines. Probe found vol-logit std 0.003: I had fed the
global query path from post-FiveLayerCNN maps (~100x weaker at init after spatial mean) → dead
hglob. v4 feeds its global path from DINO CLS tokens. Fix: CLS-analog = token-res projections.

**Run 7 divergence:** grip/rot then learned properly (5.2→2.4, 7.5→4.6 by step 600) but v4h_vol
DIVERGED at end of warmup (13.9 → 3e5; runaway q·voxel logits), poisoning the total loss.
Root suspicion: head runs bf16 under the trainer's global cast with plain AdamW on bf16 params
(no master weights — apex absent); v4 trains this structure in fp32. 1.05M-voxel dot-product +
bf16 weight updates = fragile. Runs 3/5 (262k voxels) got away with it.

**Pivot (aligned with Cameron's own suggestion):** freeze video model + cache generated features
(2 views × 11 lat frames × 11×20 × 2048 bf16 ≈ 20MB/window; 5k windows ≈ 100GB on yukon) +
standalone fp32 head trainer (minutes/run, vanilla optimizer, no FSDP/peft/bf16 coupling) +
episode-level 62/8 held-out split. Then validate winning head on generation-based features;
joint LoRA returns as the final deliberate step. AWAITING Cameron's go.

Status of asks: rerun script (task #6) WORKS (test rrd published); panels incl. wrist/upsampled-PCA/
heatmap-grid all live; kp-sample gallery (task #5) pending a converged valid checkpoint.

— yams_any4d

---

## 2026-07-11 — OVERNIGHT SCENE-ONLY CACHED-FEATURE PIPELINE: VALIDATED ✅ (~2cm held-out EE)

**Cameron's plan executed end-to-end:** scene-only video LoRA → frozen-model feature cache → standalone fp32 head training → held-out eval.

**Held-out results (episodes 62-69, never seen by the head):**
| metric | best.pt (CE-min, ~it 6000) | final (it 15000) | random baseline |
|---|---|---|---|
| EE mean error | 2.11 cm | **1.96 cm** | — |
| v4h_vol CE | 5.35 | 9.70 (overfit) | 13.2 |
| v4h_grip CE | 1.94 | 4.82 (overfit) | 3.5 |
| v4h_rot CE | 5.41 | 15.2 (overfit) | 5.5 |
Late-training CE rises = overconfidence while argmax (ee_cm) keeps improving; best.pt = CE-min ckpt. ROT never beat baseline meaningfully = the weak spot.

**Pipeline runbook (yukon):**
1. Scene LoRA: `run_scene_lora.sh` (exp rwm4_pp70_scene_lora, yaml yamyukon_pp70_scene.yaml, 3000 it, ~70min) → merge (W+=B@A) → ckpts/pp70_scene_merged.pt
2. Extract: `run_extract.sh` → custom/eval/extract_v4feats.py (teacher-forced σ=0.05 via monkeypatched draw_training_noise; v4head stub attached for the stash; subsample lifted + epoch loop + single_sample random) → feat_cache/ 3263 windows, 31GB, all 70 eps, ~5s/window
3. Train head: `run_cached_train.sh` → custom/eval/train_v4head_cached.py (V4Head use_wrist=False, fp32, AdamW 3e-4, 1:1:1 losses, clip 1.0, holdout_ep 62, ~8s/100it) — smoke-testable in 5 min on partial cache
4. Ckpts: ~/any4d_work/v4head_cached/{best,last}.pt; panels in .../panels/

**wandb:** stage-1 LoRA /runs/07-10-18-50_pp70_scene_lora1 · preview /runs/m9xpjxz5 (ee 2.05, split ≥55) · definitive /runs/ad2qb98r
**Panels (held-out sample, agents_stuff):** v4head_cached_{kp,heatmap,pca_up,grip,rot}.png — kp shows pred cluster ON GT.
**Caveats:** teacher-forced features (σ=0.05 over GT latents), not generated futures — gen-based validation is the next honesty check; rot weak (hypotheses: scene view lacks wrist-rotation detail → bring wrist back; 256 bins too fine for 62 train eps; global-head rot). Extractor gotchas: any4d_overrides doesn't reach built DiT (attach stub); data_val_overrides subsample silently caps the loader.

**Next steps:** wrist view return (rot fix candidate), generation-based feature cache (+--save_pred_rgb), bin/kmeans sweeps (minutes each now), joint LoRA finetune as final step, robot decode path.

— yams_any4d

**2026-07-11 addendum — rerun capture on GENERATED features shipped:** v4head_infer.rrd (agents_stuff, 3 samples): scene-only video model full 25-step generation → final-step features → cached head (best.pt) → per-frame kp/heatmap on the generated frames + status panel with per-sample losses on generated features (compare vs cached held-out vol 5.35/grip 1.94/rot 5.41). Script: infer_v4head_rerun.py now supports --head_ckpt (standalone head) + --scene_only; launcher yukon ~/any4d_work/run_rerun_final.sh. Gotcha: absent-view entries decode as 1-frame stubs — guard before per-frame logging.

---

## 2026-07-11 — LIVE DEPLOY BRIDGE: READY ✅ (synthetic end-to-end test passed)

Two-process live path for running the video model + cached head on the rig:
- **Server** `Any4D/custom/eval/v4head_deploy_server.py` (a4d-bw, GPU; launcher `~/any4d_work/run_deploy_server.sh`, log /tmp/deploy_server.log — RUNNING now): live frame → template-batch injection → generation cond_frames_raw=1 → final-step features → cached head best.pt → 41 (xyz world, quat wxyz, grip) + 41 gen frames. File bridge `~/any4d_work/deploy_bridge/{request,response}_<id>.npz`, atomic tmp-rename writes.
- **Client** `~/yam_local_deploy/deploy_v4head_video.py` — minimal diff of deploy_with_action_chunking.py: model sections → bridge round-trip; cam open/ArUco lock/keys y-r-h-c-q/go-home/IK/Robot.move_to/recorder UNTOUCHED; new rerun panels gen/video (41 generated frames per plan) + pred/kp (41-step trajectory on live frame). No --ckpt needed; --bridge_dir/--bridge_timeout_s added.

**Synthetic test:** cached frame + real EE pose → status ok, xyz z∈0.27-0.35m sane trajectory, unit quats, grip 0.98 (open, correct), 41 coherent gen frames (eyeballed).

**Usage:** 1) server already running (else `bash ~/any4d_work/run_deploy_server.sh`); 2) in the deploy env (tmux russet_yam:0 yukon_deploy): `python ~/yam_local_deploy/deploy_v4head_video.py --arm right --save_video --eval_name v4head_video_test --n_substeps 8` → rerun at :9092 → same y/r/h/c/q flow. ~60-120s per planning cycle (generation-bound) — use n_substeps 8-16.
**Caveats:** rot head weak (scene-only; wrist return pending); FIRST LIVE RUN SUPERVISED; nohup buffers server prints (PYTHONUNBUFFERED=1 now in launcher); grip decode uses bin centers (0..1 range per production flags).

— yams_any4d

**2026-07-11 — LIVE RUN ON THE RIG + PUBLIC VIEWER ✅:** deploy_v4head_video.py running live in tmux russet_yam:0 (env raiden_fork.venv, CUDA_VISIBLE_DEVICES=1; server GPU0). Live cycle: ZED open → ArUco lock → bridge → **5.8s** for generation+head (not 60-120s — single-view gen is fast) → 41 actions (sane workspace xyz, grip open) → prompt awaiting Cameron. Public viewer LIVE: https://rerun.omidlab.net/?url=rerun%2Bhttps%3A%2F%2Frerun-grpc.omidlab.net%2Fproxy (relay re-pointed VPS→yukon via /tmp/arm_rerun_relay.sh on VPS; restore to russet later by editing the host in that script + rerunning). Panels: scene/rgb, gen/video (gen_frame timeline), pred/kp. Deploy env gotcha: bare (base) lacks cv2 — always source /home/robot-lab/cameron/raiden_fork.venv/bin/activate.

**2026-07-11 — live session v2 (full panels) handed off:** server v2 renders + ships all diagnostics per plan cycle; client logs gen/video, pred/pca_scene (token strip), pred/pca_upsampled, pred/heatmap_grid (over GEN frames), grip/rot grids, status, under a custom blueprint. First cycle: 6.8s, sane preds. Client at prompt in russet_yam:0 for Cameron.

**2026-07-11 — live session v3 in Cameron's own tab:** launch location corrected — his "yam_yukon" tab is VPS tmux agents:7 (ssh→yukon w/ raiden_fork.venv active), NOT yukon's russet_yam tmux. Client now running there: first cycle 6.3s, 41 gen frames, sane preds, prompt interactive for Cameron. Send-keys target for future launches: agents:7 on the VPS.

---

## 2026-07-12 — 2x2 HEAD COMPARISON COMPLETE + SERVER SWAPPED

**Held-out ee_cm (eps 62-69):**
| arch | teacher feats | GEN feats (deploy-matched) |
|---|---|---|
| per-step @ P64 | 1.96-2.11 | **4.37** ← deployed |
| stack_mlp @ P128 (Cameron spec) | 2.50 | 4.96 |
Old head served on gen features: 6.7cm → gen-training cuts ~35%. Residual ~4.4cm ≈ imagined-future-vs-recorded divergence (not pure head error).

**Key findings:** (1) train/serve feature alignment matters more than architecture; (2) best.pt (vol-CE min) MISALIGNS with ee_cm on gen runs (stack CE-min had ee 7.6) → deploy last.pt; TODO add best_ee criterion; (3) stack@128 gives peakier volumes (Cameron pref) + finer 0.6cm cells but slightly worse argmax; worth revisiting with wrist views.

**Deployed:** server now serves v4head_gen/last.pt (per-step@64 trained on feat_cache_gen). wandb: ad2qb98r (per-step-teacher), n26cdvx1 (stack-teacher), if43xr13 (per-step-gen), b14kpllx (stack-gen). Panels: agents_stuff/v4head_2x2_*.png. Gen-cache runbook: run_extract_gen.sh (--generate, quota+resume, visuals no-op) → run_gen_trains.sh.

**Next steps:** wrist-view return (biggest lever for rot + occlusions), supervised robot rollout w/ new head (client cmd unchanged), best_ee ckpt criterion, joint finetune later.

— yams_any4d

**2026-07-12 swap verification resolved:** pure gen-cache held-out A/B (40 windows): NEW gen-trained head 4.32cm vs OLD teacher head 5.71cm — ordering confirmed, swap stands (server = v4head_gen/last.pt). The contradictory 10.2cm bridge probe was an artifact: rgb0_thumb saved with per-frame min-max stretch → third distribution; extractor now saves fixed-range thumbs. Live ZED frames unaffected (no stretching) — deployed head is correct. Ready for supervised rollout.