# YAM — SVO2 → PNG + lowdim

`rd convert` turns each episode's per-camera SVO2 stream into PNG frames
+ a per-frame lowdim PKL with state, intrinsics, and extrinsics.

## Launch

On russet (the rig host, where the SVO2 files are decodable by the ZED
SDK):

```bash
ssh robot-lab
cd ~/cameron && rd convert \
    --dataset_root ~/yam_para/data/place_mug_wrist_cam/ \
    --cams right_scene_camera right_wrist_camera
```

Both cams must be listed. The converter walks each episode, decodes each
SVO2 with the ZED SDK, writes `<ep>/<cam>/png/<frame>.png` and updates
`<ep>/lowdim/<frame>.pkl`.

## PKL contents (per-frame)

| Key | Type | Meaning |
|---|---|---|
| `joints` | (14,) float32 | Bimanual state: `[L7, R7]` |
| `joint_velocities` | (14,) float32 | derivative |
| `action` | (14,) float32 | Commanded joint deltas |
| `T_wrist_in_world` | (4,4) float32 | Per-frame wrist EE in world (= left_arm_base) |
| `right_scene_camera_extrinsic` | (4,4) float32 | Scene cam in world |
| `right_scene_camera_intrinsic` | (3,3) float32 | K at recording resolution (HD1080) |
| `right_wrist_camera_intrinsic` | (3,3) float32 | K at HD720 |
| `_scene_in_lbase` | bool | True if extrinsic already in left_arm_base (post-hot-patch) |
| `T_left_from_right` | (4,4) float32 | Reusable handle for downstream code |

## Extrinsic-convention chain (read carefully)

1. **At record time**: live exo aruco solves `T_scene_in_right_arm_base`
   (right arm carries the aruco board).
2. **In archival JSON**: stored with `"success": True` for the converter
   to pick it up.
3. **In `rd convert`**: copied verbatim into each per-frame PKL as
   `right_scene_camera_extrinsic`. **At this point it is still in
   right_arm_base.**
4. **Training / inference expects left_arm_base** (= world). The
   hot-patch script applies `T_lfr` once:

   ```python
   T_scene_in_world = T_lfr @ T_scene_in_rbase
   ```

   And sets `_scene_in_lbase = True` to mark idempotent.

5. **Inference** (`deploy_yam_2view.py`) does the live recal → world
   transformation in memory; no PKL touched.

If you see workspace mask collapsing from ~92k → ~30k valid voxels at
inference startup, you forgot step 4 on a fresh dataset (or step 4's
JSON read returned an inverted matrix — verify against the misnamed-key
convention in `memory.md`).

## Hot-patch script

For datasets that already have right_arm_base extrinsics (legacy):

```bash
python ~/yam_para/code/hot_patch_scene_to_lbase.py \
    --dataset_root ~/yam_para/data/place_mug_wrist_cam/ \
    --t_lfr_path ~/yam_para/data/place_mug_wrist_cam/0000/lowdim/0000000000.pkl
```

Idempotent: skips any PKL where `_scene_in_lbase = True`. Safe to re-run.

## When the JSON is missing `success: True`

`rd convert` will write PKLs with `right_scene_camera_extrinsic = I_4`.
The fix:

1. Re-write the JSON with `"success": True` (the rest of the JSON is
   unchanged).
2. Re-run `rd convert` for that episode only:

   ```bash
   rd convert --dataset_root ... --episodes 0042 0043 ...
   ```

The recorder now always writes `success: True`; legacy episodes from
before the fix may need this treatment.

## After convert: sanity scan

```bash
# Number of episodes
ls ~/yam_para/data/place_mug_wrist_cam/ | wc -l

# Spot-check first / last PKL extrinsic
python - <<'PY'
import pickle, numpy as np
for ep in ["0000", "0500", "1000"]:
    p = f"~/yam_para/data/place_mug_wrist_cam/{ep}/lowdim/0000000000.pkl"
    with open(p, "rb") as f:
        fd = pickle.load(f)
    E = fd["right_scene_camera_extrinsic"]
    if np.allclose(E, np.eye(4)):
        print(f"  {ep}: IDENTITY ← problem")
    else:
        print(f"  {ep}: t={E[:3,3]}")
PY
```

Healthy: `t ≈ [-0.5..0.5, 0.3..0.8, 0.2..0.5]`. Identity = converter
dropped extrinsics; re-check JSON success flag.

## Rsync russet → puget for training

Training runs on puget (GPU is there). Sync the converted data:

```bash
# from puget
ssh dev
rsync -avh --info=progress2 \
    robot-lab@russet:/home/robot-lab/cameron/yam_para/data/place_mug_wrist_cam/ \
    ~/yam_para/data/place_mug_wrist_cam/
```

PNG frames are big; expect ~25 GB for a few hundred episodes. Doing this
over sshfs would be glacial; use rsync.