# Data recording — real-side reference

> yam_calib's canonical doc for the YAM real-robot recording pipeline: `record.py` variants, ZED handling, CAN handling, storage schema, dataset conventions, calibration-frame writes. Downstream training + eval consumption is yams's slice.

---

## record.py variants + where they live

| Script | Machine | Purpose |
|---|---|---|
| `raiden.record` (upstream) | Any | Vanilla single-arm recorder from public raiden |
| `~/yam_local_record/record.py` | yukon | Yukon-local mirror of the recording script — used because `~/lab/` sshfs to phe108 keeps wedging. Runs from `raiden_fork.venv` |
| `raiden.record_bimanual_*` (variants) | russet | Bimanual recorders. Root of truth is `raiden_internal/raiden/record*.py` on russet |

**Yukon** is the single-arm station (right arm). **Russet** is the bimanual station. Both write into episode dirs on their local disks; sync to puget for training happens later.

## Storage schema per episode

Directory layout (single episode):
```
<eval_videos_root or dataset_root>/<name>_<timestamp>/
├── metadata.json                    # scene name, ArUco IDs, timestamps, robot config
├── calibration_results.json         # cameras + T's as of RECORD start (see calibration.md schema)
├── robot_data.npz                    # joint pos/vel/eff/cmd, gripper, timestamps — the training-signal npz
├── scene_camera.svo2 / .mp4          # scene ZED footage (usually SVO2 native, decoded later)
├── ego_camera.svo2 / .mp4           # optional egocentric ZED (russet)
├── left_wrist_camera.svo2 / .mp4    # bimanual only
├── right_wrist_camera.svo2 / .mp4    # bimanual (or `wrist_camera` on single-arm)
└── <camera>_timestamps.json          # per-camera frame timestamps (post-decode sidecar)
```

**Video codec convention:**
- Native record: SVO2 (ZED's own compressed format, mmap-friendly, has depth if enabled)
- Decoded for browser viewers: mp4 with **H.264 (avc1)** + `yuv420p` + `+faststart` (mp4v codec FAILS in browsers — always re-encode with `ffmpeg -c:v libx264 -pix_fmt yuv420p -movflags +faststart`)

**Robot data npz keys** (from `robot_data.npz`):
- `qpos`, `qvel`, `qeff`, `qcmd` — 7-DOF (single arm) or 14-DOF (bimanual concat)
- `gripper` — [0,1] normalised
- `timestamps` — per-frame Unix time (aligned with camera_timestamps for later joint↔frame lookup)

## Camera resolutions in use

| Camera | Record resolution | Notes |
|---|---|---|
| scene_camera (ZED) | Currently HD720@30 (default) | **Blocks scene auto-cal** — needs bump to HD1080 for board detection at record-start |
| ego_camera (ZED) | HD720@30 | Closer to base boards; can detect at 720p |
| wrist_camera (ZED) | HD720@30 | Close to its board target; fine at 720p |

**Empirical detection at 720p** (from flip_pink_cup 2026-06-30 investigation):
- ego_camera: 16/18 markers detected across 5 timestamps
- scene_camera: 0/18 markers → scene auto-cal not possible at 720p

**Fix plan:** open scene cams at HD1080 for at least the first ~15 frames (calibration window), then decide whether to keep HD1080 for the whole record (bigger files) or downshift to HD720 after cal completes.

## ZED handling (SVO2 pipeline)

- Native ZED reads use `pyzed_sl` (`pip install pyzed-sl-py`) — vendor SDK bindings. Not pure python.
- Decoding SVO2 → mp4: `/data/cameron/code/decode_svo2.py` (I wrote this) — takes SVO2 + writes mp4 with H.264 encoding + a `.timestamps.json` sidecar
- SVO2 files can be huge — sync to puget selectively, decode locally on the training box if needed
- ZED opening at HD1080 vs HD720: same API, different `sl.RESOLUTION` constant. Frame rate options limit at HD1080 to 15 fps.

## CAN handling (yukon)

- Named CAN interfaces via udev rules by USB serial. See `/etc/udev/rules.d/*-can*.rules`:
  - `can_follower_l` = `00460031594E501820313332`
  - `can_follower_r` = `0064002C594E501820313332`
  - `can_leader_l` = `0069003B594E501820313332`
  - `can_leader_r` = `0063002D594E501820313332`
- Enumerate present adapters: `for d in /sys/bus/usb/devices/*/serial; do dev=$(dirname $d); vid=$(cat $dev/idVendor 2>/dev/null); pid=$(cat $dev/idProduct 2>/dev/null); [ "$vid$pid" = "1d50606f" ] && echo "serial=$(cat $d) path=$dev"; done`
- Reset all CAN interfaces to 1 Mbps: `rd reset_can` (runs as regular user; passwordless sudo baked in for the specific `ip link` calls)
- If a CAN interface is missing (not in `ip link`), the adapter isn't plugged in / not enumerated. Check USB, don't try software fixes.

## Filesystem gotchas

### `/data/cameron` mount on omid-fleet
- Is a sshfs mount to phe108's `/data/cameron`
- Inside it, `/data/cameron/yukon_remote` symlinks to phe108's `~/mnt/yukon` which is ANOTHER sshfs (to yukon). Two-hop sshfs across a symlink often fails with "Transport endpoint is not connected" from omid-fleet's POV even when both mounts work individually
- **Fix:** run scripts that touch `yukon_remote` from phe108, not omid-fleet

### Never write big data on the agent-fleet box
- Data goes to `/data/cameron` (phe108) or `~/cameron` on russet/yukon/puget
- Do NOT write to `/tmp`, `/var/tmp`, or `~` on omid-fleet — agent-fleet has no data storage
- Memory reminder: `data-storage-agent-fleet.md`

### Yukon sshfs to phe108 (`~/lab`)
- Wedges regularly. When it wedges: `df -h /home/robot-lab/lab` hangs
- Recovery: `sudo fusermount -uz /home/robot-lab/lab` then remount. If D-state processes are stuck on the wedged FUSE, do `sudo bash -c 'for c in /sys/fs/fuse/connections/*/abort; do echo 1 > $c; done'` — this returns EIO to the syscall and the process becomes killable
- Long-term: yukon-local mirrors of scripts avoid the wedge entirely. Existing mirrors: `~/yam_local_record/`, `~/yam_local_train/`, `~/yam_local_deploy/`

## Calibration frame writes into episodes

**Current state (2026-07-02):** record.py does NOT auto-calibrate at start. Calibration is a separate ceremony run before the record session. `calibration_results.json` is copied into the episode dir either at record start (if freshly available) or downstream via post-processing.

**Planned change (open task, mine):** on record.py startup, before the operator can press `c` to start:
1. Open scene cams at HD1080 for calibration window
2. Pool 15 frames per cam per arm
3. Run `exo_calibrate_bimanual_fidex._solve_one_arm` for each arm
4. Compute + save `T_cam_in_left_arm_base`, `T_right_base_in_left_base`
5. Write `calibration_results.json` into the episode dir
6. Optionally downshift scene cams to HD720 for main recording (bandwidth)

Referenced planning memory: `raiden-record-autocal-plan.md`.

## Env / launcher recipes

**Yukon record with wrist cams**:
```bash
cd ~/yam_local_record
source ~/cameron/raiden_fork.venv/bin/activate
python record.py --name <session_name> --wrist   # add --wrist for wrist cam view
```

**Russet bimanual record**:
```bash
ssh robot-lab@100.123.182.78
cd ~/raiden_internal
source .venv/bin/activate
python -m raiden.record_bimanual --name <session_name>
```

## S3 upload / sync (records → training-side storage)

Records land on the recording box's local disk. Sync to S3 or puget for training use:
- **russet → S3**: uploads to `s3://tri-ml-datasets-uw2/raiden_datasets/raw/<task>/<station>_<timestamp>/`
- **yukon → puget**: `rsync -avz ~/recordings/<session>/ cameronsmith@puget:~/yam_para/data/<session>/`
- Puget has the `~/yam_para/data/*` conventions used by training scripts (see yams's `training.md` if it exists)

## AWS S3 credentials

TRI Okra portal issues temporary creds. Account 124224456861 is the working one for `tri-ml-datasets-uw2`. Creds expire in 1-4 hours; refresh from portal + export as env vars.

## Viewer for S3 records (self-serve)

`/data/cameron/code/build_viewer.py` (mine) — takes a decoded record session dir + writes an HTML viewer with:
- 4 real first-frame jpgs (from `frames/<cam>_first.jpg`)
- Robot proprio timelines (Plotly): joints, pos/vel/eff/cmd, grippers
- Play button + scrub slider synced across camera videos
- Sync tolerance: 100ms drift threshold between master + secondaries
- URL convention: images/videos linked with `?raw&v=<build_ts>` for browser MIME (the `/browse/` endpoint returns an HTML preview without `?raw`)

Live example: `/data/cameron/s3rec/flip_pink_cup_2026-06-30T16-52-57/viewer.html`

## Cross-agent handoff

- **Recording pipeline start-to-finish** → I own
- **Storage schema (calibration_results.json, robot_data.npz key names)** → I own; but yams's training consumers depend on this — if I change the schema I ping yams's inbox
- **Sim-derived data / synthetic-record generation** → yam_sim owns
- **What training scripts do with the recorded data** → yams owns

## Known open work (as of 2026-07-02 fork)

- Bake auto-cal into `record.py` + open scene cams at HD1080 (planning memory: `raiden-record-autocal-plan.md`)
- Extract russet v2-reverse arm-2 patch (duplicated across three `_fidex.py` files) — if I touch those files next
- The `~/lab` sshfs stability problem — long-term fix is to reduce reliance on it entirely from yukon-side record/calib scripts, prefer local mirrors

## Related

- Calibration outputs schema + PnP: `calibration.md`
- Fiducial exo (marker geometry): `fiducial_exo.md`
- Sim-side / synthetic data: `simulation.md` (yam_sim's slice)