# Calibration — real-side reference

> yam_calib's canonical doc for **real-robot calibration** on the YAM stack: scene-cam, wrist-cam, hand-eye, and how the outputs are saved / consumed. Fiducial-exo details live in `fiducial_exo.md` (marker geometry, PnP recipe, russet patches). This file is about the **calibration pipeline as a whole**, not just the fiducial math.

---

## Types of calibration + where each lives

| Type | Purpose | Script (single arm) | Script (bimanual russet) |
|---|---|---|---|
| **Scene camera** (extrinsic wrt arm base) | Recover `T_cam_in_arm_base` for each mounted ZED viewing the scene | `raiden/calibration/exo_calibrate.py` | `raiden/calibration/exo_calibrate_bimanual_fidex.py` |
| **Wrist camera** (hand-eye, extrinsic wrt EE) | Recover `T_cam_in_ee` for the ZED mounted on the wrist | `raiden/calibration/wrist_calibrate.py` | `raiden/calibration/wrist_calibrate_bimanual_fidex.py` |
| **Wrist verify** (sanity-check hand-eye) | Reproject base board into wrist view using computed T | — | `raiden/calibration/wrist_verify_rerun_bimanual_fidex.py` |
| **Arm-zero / kinematic** | Verify FK matches the physical hardware zero pose | Handled inside the recorder + robot init; no dedicated script | Same |

The **_fidex** suffix is the russet-specific fork of these scripts. It embeds the arm-2 v2-reverse patch and the auto-discover-cams flag. All russet calibration must go through `_fidex` files. The upstream (non-russet) rig can use the vanilla names.

## What each script writes

`calibration_results.json` — one file per station, consumed by `LiveVisualizer` and any downstream teleop / record / eval that reads camera poses. Schema (as of 2026-07-01):

```json
{
  "cameras": {
    "scene_camera": {
      "extrinsics": {           // = T_cam_in_left_arm_base  (bimanual world = left)
        "R": [[...]], "t": [...]
      },
      "K": [[...]],
      "distortion": [...],
      "image_size": [W, H]
    },
    "left_wrist_camera": {
      "hand_eye_calibration": {  // = T_cam_in_ee (constant relative to EE frame)
        "R": [[...]], "t": [...]
      },
      "K": [[...]],
      "distortion": [...]
    },
    "scene_camera__arm1": { "extrinsics": <T_cam_in_arm1_base>, ... },
    "scene_camera__arm2": { "extrinsics": <T_cam_in_arm2_base>, ... }
  },
  "bimanual_transform": {
    "right_base_to_left_base": <T_left_base_in_right_base>  // inverse per that field's naming — see visualizer._load_calibration
  }
}
```

Key convention (do NOT drift): scene cams' primary `extrinsics` key is `T_cam_in_left_arm_base` (=world), plus `__arm1` and `__arm2` siblings for the per-arm PnP results. This is Shun's convention; LiveVisualizer patches on top of it to render both arms.

## Pooled multi-frame PnP (the standard everywhere)

Every real calibration uses the pooled-frame recipe, not single-shot. It's dramatically more stable and accounts for:
- Marker detection noise per frame
- Occasional partial-marker frames (only some of the 3×3 grid visible)
- Arm/gripper pose drift during pooling (mitigated by taking the K lowest-residual frames)

Helper class: `_PoseHistory` in `raiden/calibration/exo_calibrate_fidex.py`. Uses `max_size=30` for the ring buffer, `top_k_concatenated(10)` at solve time. Both parameters have been stable — don't tune without a reason.

The multi-frame solver is `_solve_multiframe_pose(obj_pool, img_pool, K, dist, board_length)` — returns `(T_aruco_in_cam, residual_px)`. If residual > ~2 px the pool contained mixed poses; scrap and re-pool.

## Bimanual-specific plumbing

`_capture_loop_bimanual_multicam` (in `exo_calibrate_bimanual_fidex.py`) — the current single-thread iterator over cameras. Was refactored from a multi-thread version because `mujoco`'s EGL renderer race-conditions when two threads call `eglMakeCurrent` at once (`EGL_BAD_ACCESS`). Don't try to re-parallelize the camera loop.

Steps per bimanual scene calibration:
1. `_discover_role_scene_cams(camera_config_file)` — walk `camera.json` for entries with `role == "scene"`
2. For each scene cam: open ZED at HD1080, run pooled PnP for arm 1 (IDs 217–225) and arm 2 (IDs 226–234) separately using `_partition_corners`
3. `_solve_one_arm(...)` per arm, writing `T_cam_in_arm1_base` and `T_cam_in_arm2_base`
4. Compute `T_cam_in_left_arm_base` and `T_right_base_in_left_base` from the two per-arm solves
5. `_save_bimanual_calibration(...)` writes the JSON

## Live overlay (real-time visual verify)

Every scene calibration renders a **MuJoCo overlay** on top of the live ZED feed showing the reconstructed arm + board where the calibration says they are. Yellow (arm 1) and magenta (arm 2). If the overlay lines up with the physical arm → calibration is good. If it's offset by ~cm → re-pool or check board mounting.

Helper: `_render_with_intrinsics(model, data, T_world_in_cam, K, W, H)` in `exo_calibrate_fidex` returns `(rgb, seg_mask)`. Composite via `_compose_overlay(...)`.

**On russet specifically** — MuJoCo initialisation needs `MUJOCO_GL=egl PYOPENGL_PLATFORM=egl` env vars. Without them: `MujocoException: GLFW not supported`.

## Known-good hyperparameters (do not tune without cause)

| Param | Value | Where |
|---|---|---|
| `_PoseHistory.max_size` | 30 | `exo_calibrate_fidex.py` |
| Top-K frames pooled at solve | 10 | `top_k_concatenated(10)` |
| ZED input resolution for calib | **HD1080** @ 15 fps | Camera open call |
| Board_length | `BOARD_LENGTH_COARSE` (~9.5 cm — check `panda_exo.py` for exact) | `panda_exo.BOARD_LENGTH_COARSE` |
| ArUco marker margin | 0 (zero internal margin — see `fiducial_exo.md`) | Board PNG regen |

## Gotchas

- **cv2.aruco.detectMarkers missing (4.12)**: raiden's default `.venv` initially had cv2 4.12 which dropped the module-level function. Fixed by pinning `opencv-contrib-python==4.10.0.84` (numpy-2-compatible, has detectMarkers). Not `4.6.0.66` (too old for numpy 2) or `4.9.0.80` (numpy issues).
- **EGL multi-thread race**: two capture threads' mujoco renderers colliding on `eglMakeCurrent` gives `EGL_BAD_ACCESS`. The single-thread iterator was the fix. Don't undo it.
- **v2-reverse tuple bug**: earlier viz-autocal used `tuple` for `aruco_offset_pos`, causing `TypeError: unsupported operand type(s) for /: 'tuple' and 'int'`. Always use `np.array([...])` when patching arm-2.
- **Scene-cam @ 720p can't detect the board.** Recorded episodes at 720p → no auto-cal possible. Bump to 1080p for record OR do manual pre-record calibration ceremony (see `data_recording.md`).
- **`~/lab` sshfs on yukon wedges periodically**. Real calibration scripts on russet run against `~/raiden_internal` local mirror — sshfs breakage is a yukon problem, not russet.

## Auto-calibration on record (planned; blocking on record.py)

Plan: on `record.py` startup (before operator hits `c`), pool the first ~15 frames from every scene + ego camera, run per-arm PnP via `exo_calibrate_bimanual_fidex` helpers, write to `calibration_results.json` inline. **Passive** — no operator input as long as base boards are visible.

**Currently blocked** by `record.py` opening ZED scene cameras at HD720@30 (not HD1080). Bump the recorder's ZED init to HD1080 for at least the calibration frames, or add a bimodal open (1080p at start, downshift to 720p after calibration completes to save bandwidth).

Referenced memory: `raiden-record-autocal-plan.md` in agent memory.

## Cross-agent handoff

- **PnP math / board geometry / STL variants** → I own; see `fiducial_exo.md`
- **MuJoCo rendering of the overlay** → yam_sim owns rendering; I own the `_render_with_intrinsics` wrapper that CALLS the render
- **Record.py auto-cal integration** → I own the writer path + camera-open logic; see `data_recording.md`
- **Downstream code that READS `calibration_results.json`** (deploy, visualizer, training) → yams owns those consumers

## Related

- Fiducial exo geometry + PnP math: `fiducial_exo.md`
- Record.py, ZED, CAN, storage schema: `data_recording.md`
- Sim-side rendering conventions: `simulation.md` (yam_sim's slice)