# Fiducial Exoskeletons (fidex) — notes for parametric robot building

Source: `github.com/cameronosmith/FiducialExoskeletons` (cloned to lab `/data/cameron/repos/FiducialExoskeletons`). Paper: "Fiducial Exoskeletons: Image-Centric Robot State Estimation" (cameronosmith.github.io/fidex). Cameron's working robot config: `smith300_para_stuff/` (medium_single_redo).

## The one-sentence idea
Rigidly mount a known ArUco **board** on **every link** of the robot; then a single uncalibrated RGB image → detect each board → per-link pose → solve **camera pose + all joint angles** (no encoders, no calibration). The markers are *part of the printed structure* (on the clamps/connectors), not stickers.

## Pipeline (exo_utils.py)
1. **Detect** (`detect_link_poses` → `do_est_aruco_pose`): one `detectMarkers` pass; per link `getBoardObjectAndImagePoints` + `solvePnPGeneric(SOLVEPNP_IPPE)` (planar). Disambiguate the IPPE 2-fold flip by picking the solution whose board **normal faces the camera** (`normal[2]<0` & `tvec[2]>0`). If no intrinsics passed, `calibrateCameraAruco` estimates `cam_K` from the markers themselves (uncalibrated path).
2. **Link pose from board pose**: `T_link_cam = T_aruco_cam @ inv(T_link→aruco)`. `T_link→aruco` is the FIXED offset of the board on the link = `LinkConfig.aruco_offset_pos (mm)` + `aruco_offset_rot (euler rad)`. **This is the key CAD↔vision coupling: the marker's pose on the part must be known exactly.**
3. **Anchor frame**: a reference link (e.g. `larger_base`) defines the camera pose; all other link poses re-expressed as `inv(cam_pose) @ link_pose`. Output: per-link poses (base frame) + `camera_pose_world` + `cam_K`.
4. **Joint angles** (`estimate_robot_state`): `mink` differential IK — one `FrameTask` per detected link (target = detected pose, set on `<name>_link_mesh` mocap body), `solve_ik` ("daqp") for `ik_iterations` (~15–55). `position_cost=1.0`, `orientation_cost=0.03` (position dominates).
5. **Overlay** (`render_from_camera_pose`): render estimated robot from estimated camera + fovy `2·atan(H/(2·fy))`, blend over the image.

## Data structures
- `LinkConfig` (exo_configs/exoskeleton.py): `mujoco_name`, `pybullet_name` (the IK body), `robot_mesh_path`, `exo_mesh_path` (the printed mount that carries the board), `aruco_offset_pos` (mm, link→board), `aruco_offset_rot` (euler rad), `aruco_board_name`, `board_length` (m).
- `ExoskeletonConfig._generate_xml`: builds a MuJoCo scene of **mocap** bodies per link — `<name>_link_mesh`, `<name>_exo_mesh`, `<name>_exo_plane` (textured aruco quad = `plane.obj` scaled + board-image material) — plus an `estimated_camera` (fovy 95). Plane sizes: small (≤0.05m) vs large.
- Per-link board object = `cv2.aruco.GridBoard(grid, markerLength, markerSeparation, dict, ids)`. In `medium_single_redo`/`OurSO100ExoConfig`: `markerLength:markerSeparation = 0.8:0.2`; `marker_len = 0.8·board/unit_span`, `unit_span = gx·0.8+(gx-1)·0.2`.

## smith300 / medium_single_redo board layout (the current robot)
- dict **DICT_4X4_250**. base = **3×3 @ ~50mm**, id 0. each of 6 shoulders = **2×2 @ 15mm**, ids 20,24,28,32,36,40 (4 ids each). Boards placed via `aruco_positions[link]` with `frame` = `local` | `parent_link` | `world`; sizes use `size_is_blender_plane_scale` (×2). Auto-emits a **true-scale A4 print PDF** with a 50mm calibration square so prints match the model.

## Gotchas
- **Dict mismatch**: `exo_utils.ARUCO_DICT = DICT_6X6_250` (default), but the SO100/smith300 configs generate **DICT_4X4_250**. Detection MUST use the same dict the markers were generated with — check per config.
- **Unique IDs per link** (sequential blocks); 250 ids per dict.
- **Units**: `aruco_offset_pos` in **mm** (divided by 1000 in code); board sizes in **m** (or mm if >2). Robot SO100 meshes are in **m**; Blender exo/holder STLs in **mm** (×0.001). (Same meters-vs-mm trap as `sts3215_03a_v1.stl`.)
- Board normal/ID-corner convention sets the marker's local frame → the `aruco_offset_rot` must match how the board was placed in CAD.

## What our parametric CAD must emit to be fidex-ready
When we parametrically generate the robot (build123d parts + URDF), generate from ONE source of truth per link:
1. **The printed part** (clamp/connector/holder) with a **recessed board pocket** at a known pose (so the physical marker sits exactly where the model says).
2. **`board_pose_in_link`** = the `aruco_offset_pos`/`aruco_offset_rot` for that pocket → feeds `LinkConfig`.
3. **The board spec** (dict, grid, ids, size) → the GridBoard + the printed marker image.
4. Keep the **printed marker sheet** (true-scale) and the **CAD pocket** dimensionally identical.
So a parametric "link module" should output: part STEP/STL + its `LinkConfig` row + its board image, all consistent. That's the clean abstraction — the marker pocket is just another parametric feature of the connector, and its pose is a free output we hand to fidex.

See also [[smith300-current-robot]] context in memory.md.