# Fiducial exoskeleton — real-side reference

> yam_calib's canonical doc for the **real-side** fiducial exoskeleton system: physical board, printed markers, PnP pose recovery, single-arm + bimanual variants, russet-specific patches. Cross-agent bright line: **real-side calibration + PnP = yam_calib**. **Sim-side rendering / validation = yam_sim** (see `simulation.md`). Overlap files (STL geometry, marker geometry, MJCF-side rendering) may be edited on either side; ping the other's inbox when you touch them.

The comprehensive external doc (originally written 2026-07-01, still the authority) lives at:
- Path: `/data/cameron/para/robot/yam/docs/fiducial_exoskeleton_board.md`
- URL: https://omidlab.net/browse/para/robot/yam/docs/fiducial_exoskeleton_board.md?raw

This vault file is the "what yam_calib specifically needs to remember" summary — geometry constants, script paths, and the operational recipes.

---

## Board geometry (unchanged since 2026-07-01)

| Constant | Value | Source of truth |
|---|---|---|
| ArUco dict | `cv2.aruco.DICT_6X6_250` | `raiden_fork/third_party/exo_redo/exo_utils.py::ARUCO_DICT` |
| Layout | 3×3 grid of "coarse" markers per arm base | `panda_exo.link_boards` |
| Marker length | `BOARD_LENGTH_COARSE` (check `ExoConfigs/panda_exo.py`; ~9.5 cm) | `panda_exo.BOARD_LENGTH_COARSE` |
| Arm-1 IDs (right, `larger_coarse_board`) | **217 – 225** (9 markers) | `panda_exo.link_boards` |
| Arm-2 IDs (left, `larger_coarse_board_arm2`) | **226 – 234** (9 markers) | `panda_exo.link_boards` |
| Offset from arm-base link to ArUco origin | `LinkConfig.aruco_offset_pos` (mm) + `.aruco_offset_rot` (rad) | `ExoConfigs/yam_exo.py:52-53` (arm 1) & `:101-102` (arm 2) |

**Default arm-1 config values** (verify against current repo):
- `aruco_offset_pos = np.array([175.23, -19.6, -4.1])` mm
- `aruco_offset_rot = np.array([0.0, 0.0, 0.0])` rad
- `aruco_board_name = "larger_coarse_board"`
- `board_length = BOARD_LENGTH_COARSE`

**Physical print rules** (my responsibility, don't drift):
- Print at 100% scale — measured marker side length with calipers must match `BOARD_LENGTH_COARSE` to ±0.5 mm
- Regenerate PNG at `marginSize=0` (opencv's `GridBoard.generateImage(outSize=(1200,1200), marginSize=0)`)
- Mount vinyl with the registration lip aligned — mirror-flipped install → wrong marker positions → PnP wrong

## STL variants (2026-07-01 state)

| File under `raiden_fork/third_party/exo_redo/so100_blender_testings/` | Which arm | Notes |
|---|---|---|
| `yam_base_board.stl` | v1 (deprecated) | Original single-arm mount |
| `yam_base_board_v2.stl` | Both arms, standard rig | Default (`YAM_EXO_MESH` in `yam_exo.py`) |
| `yam_base_board_v2_reverse_clearance_fixed.stl` | **arm 2 only on russet** | Mirror-flipped along X — see russet patch below |

## PnP pose recovery — single frame

Standard call (any real camera with K + dist known):
```python
from exo_utils import do_est_aruco_pose, ARUCO_DICT
from ExoConfigs.exoskeleton import link_to_aruco_transform

link_cfg     = YAM_BASE_ONLY_CONFIG.links["larger_coarse_board"]
aruco_board  = YAM_BASE_ONLY_CONFIG.aruco_board_objects["larger_coarse_board"]
board_length = link_cfg.board_length
T_link_to_aruco = link_to_aruco_transform(link_cfg)     # 4×4, meters+radians

r = do_est_aruco_pose(frame_bgr, ARUCO_DICT, aruco_board, board_length,
                       cameraMatrix=K, distCoeffs=dist)
if r == -1: return None
T_aruco_in_cam = r["est_aruco_pose"]                    # 4×4 (OpenCV convention)
T_cam_in_link  = T_link_to_aruco @ np.linalg.inv(T_aruco_in_cam)
```

## PnP pose recovery — multi-frame pooled (production)

This is what all real calibration scripts use. Pool corner detections across N frames, keep the K lowest-residual, solve ONE PnP on the union:
```python
from raiden.calibration.exo_calibrate_fidex import _PoseHistory, _solve_multiframe_pose

hist = _PoseHistory(max_size=30)
for bgr in frames:
    r = do_est_aruco_pose(bgr, ARUCO_DICT, aruco_board, board_length,
                          cameraMatrix=K, distCoeffs=dist)
    if r == -1: continue
    # bookkeeping — see _solve_one_arm in exo_calibrate_bimanual_fidex.py for the exact accounting
    hist.add(obj_pts=obj_board, img_pts=img_pts, residual_px=resid_px,
             single_frame_t=T_aruco_in_cam[:3, 3])

obj_pool, img_pool, n_used, _ = hist.top_k_concatenated(10)
T_aruco_in_cam, resid_px = _solve_multiframe_pose(
    obj_pool, img_pool, K, dist, board_length)
T_cam_in_link = T_link_to_aruco @ np.linalg.inv(T_aruco_in_cam)   # this is what raiden saves
```

## Bimanual — partition IDs before per-arm solve

Bimanual PnP does ONE `cv2.aruco.detectMarkers` call, then splits by ID range:
- Arm 1: IDs 217–225 → arm-1 solver
- Arm 2: IDs 226–234 → arm-2 solver

Helper: `raiden.calibration.exo_calibrate_bimanual_fidex._partition_corners(corners, ids)` returns `(arm1_corners_est, arm2_corners_est)` where each is either `None` or `(sub_corners, sub_ids, [])` consumable by `do_est_aruco_pose(..., corners_est=...)`.

## Frame conventions (memorize)

- `T_a_in_b` = pose of frame `a` expressed in frame `b` — 4×4 that maps `p_a` to `p_b = T_a_in_b @ p_a`
- `link_to_aruco_transform(link_cfg)` returns `T_aruco_in_link` — despite the misleading "to" naming
- OpenCV PnP output convention: `+X` right, `+Y` down, `+Z` forward from camera
- What raiden saves per camera: `T_cam_in_arm_base`
- Bimanual saves: both `T_cam_in_arm1_base` and `T_cam_in_arm2_base` under `__arm1` / `__arm2` suffixed keys
- Bimanual world = `left_arm_base`. Right arm placed via `T_right_base_in_left_base = T_cam_in_left_base @ inv(T_cam_in_right_base)` — written to `calibration_results.json` under `bimanual_transform.right_base_to_left_base` (inverted per that field's naming; see `raiden/visualizer.py`'s reader for the exact convention).

## russet-specific v2-reverse arm-2 patch

**Only applies to russet's bimanual station.** The arm-2 base board is physically mirrored (clearance envelope). Both edits are required at import time, before `link_to_aruco_transform` OR any MJCF/XML build:

```python
_ARM2_V2_STL = str(_EXO_DIR / "so100_blender_testings" /
                   "yam_base_board_v2_reverse_clearance_fixed.stl")
for _link in YAM_BASE_ONLY_CONFIG_ARM2.links.values():
    _link.exo_mesh_path = _ARM2_V2_STL
    _old = np.asarray(_link.aruco_offset_pos, dtype=np.float64).copy()
    _link.aruco_offset_pos = np.array([-_old[0], _old[1], _old[2]])   # flip x sign
```

This patch is embedded in every `_fidex.py` sibling on russet:
- `raiden/calibration/exo_calibrate_bimanual_fidex.py`
- `raiden/calibration/wrist_calibrate_bimanual_fidex.py`
- `raiden/calibration/wrist_verify_rerun_bimanual_fidex.py`

**Don't apply the patch on the upstream single-arm rig.** yukon uses `yam_base_board_v2.stl` on both arms (or its single arm in the current single-arm YUKON case).

## Resolution requirements

- Boards laid out for reliable decode at **1080p** (~30 px per marker in scene-cam FOV)
- At 720p: markers shrink to ~20 px, 6×6 patterns near threshold, scene_cam fails
- Empirical: on `flip_pink_cup_2026-06-30T16-52-57`, ego cam @ 720p detects 16/18, scene cam @ 720p detects 0/18
- **Calibrate at HD1080@15fps. Don't try scene-cam self-calibrate at 720p.**

## Key file map (real-side)

| Purpose | Path |
|---|---|
| Config: dict + IDs + board_length | `raiden_fork/third_party/exo_redo/ExoConfigs/panda_exo.py` |
| Config: YAM link → aruco offset + STL path | `raiden_fork/third_party/exo_redo/ExoConfigs/yam_exo.py` |
| Config: `LinkConfig`, `link_to_aruco_transform` | `raiden_fork/third_party/exo_redo/ExoConfigs/exoskeleton.py` |
| PnP wrapper + mocap positioning | `raiden_fork/third_party/exo_redo/exo_utils.py` — `do_est_aruco_pose`, `position_exoskeleton_meshes`, `get_link_poses_from_robot` |
| Board PNGs | `raiden_fork/third_party/exo_redo/aruco_images/*.png` |
| STL mounts | `raiden_fork/third_party/exo_redo/so100_blender_testings/yam_base_board_v2*.stl` |
| Scene calib (single arm) | `raiden/calibration/exo_calibrate.py` (or `_fidex` on russet) |
| Scene calib (bimanual) | `raiden/calibration/exo_calibrate_bimanual.py` (or `_fidex` on russet) |
| Wrist calib (single arm) | `raiden/calibration/wrist_calibrate.py` |
| Wrist calib (bimanual) | `raiden/calibration/wrist_calibrate_bimanual.py` (or `_fidex` on russet) |
| Wrist verify (bimanual) | `raiden/calibration/wrist_verify_rerun_bimanual.py` (or `_fidex` on russet) |
| Live visualizer (consumes calibs) | `raiden_internal/raiden/visualizer.py::LiveVisualizer._load_calibration` |

## Recipes (real hardware)

**Bimanual scene calibration on russet** (both boards visible):
```bash
ssh robot-lab@100.123.182.78
cd /home/robot-lab/raiden_internal
source .venv/bin/activate
MUJOCO_GL=egl PYOPENGL_PLATFORM=egl \
  python -m raiden.calibration.exo_calibrate_bimanual_fidex \
  --vis_port 9877 --auto_discover_scene_cams --no-teleop
# in rerun viewer: `c` start pooling, `y` save, `q` quit
```

**Bimanual wrist calibration on russet** (needs teleop / spacemouse):
```bash
python -m raiden.calibration.wrist_calibrate_bimanual_fidex --vis_port 9877
```

**Bimanual wrist verify** (reprojects base board into wrist view):
```bash
python -m raiden.calibration.wrist_verify_rerun_bimanual_fidex --vis_port 9877
```

## Cross-agent handoff points

- Anything about **MuJoCo rendering** of the board (mocap-geom orientation, `build_bimanual_xml`, `position_exoskeleton_meshes`) → yam_sim owns rendering; call the vault `simulation.md` first
- Anything about **STL editing** (new mount shapes, printing) — either side may touch; ping the other's inbox with a one-line FYI
- Anything about **PnP conventions / `T_aruco_in_cam` / calibration_results.json schema** → yam_calib owns

## Known open work (as of 2026-07-02 fork)

- **Bake auto-cal into `record.py`** — pool first N frames on record startup and write extrinsics. Blocked on record.py opening scene cams at HD1080 (currently HD720). See `data_recording.md`. Referenced from prior planning memory `raiden-record-autocal-plan.md`.
- **Russet v2-reverse patch** currently duplicated across three `_fidex.py` files — candidate for extraction into a shared helper if we edit those files again.

## Related

- Sim-side rendering conventions + MJCF composition: `simulation.md` (yam_sim's slice)
- Recording pipeline that consumes calibration outputs: `data_recording.md` (my slice)
- Calibration output schema (what LiveVisualizer reads): `calibration.md` (my slice)