# ar_view — live real-vs-sim AR overlay

Renders the arm + UMI gripper (posed by the **live calibrated servos**) over the real
camera streams, localized from the arm-base ArUco board — the live version of the cad
capture-relocalization montages.

## First: calibrate + register each camera (once)

```
python3.11 calibrate_camera.py scene --index 0     # third-person camera
python3.11 calibrate_camera.py wrist --index 1     # UMI wrist camera
```

Hold the arm-base ArUco board in front of the camera and move it around (vary distance,
angle, position). It auto-captures ~60 frames and runs `cv2.aruco.calibrateCameraAruco`
(a multi-view solve, not a per-frame median), then saves `intrinsics/<name>.json` and
registers the camera (index + resolution) in `cameras.json`. Keys: `c` capture, `q` finish
(≥8 frames), ESC abort.

**Registration by role, not port:** cameras are referenced as `scene` / `wrist`. `arview`
resolves each role via `cameras.json`; if the registered index no longer has the registered
resolution (macOS reshuffled indices), it finds the index whose resolution matches and uses
that. Two cameras with *different* resolutions self-identify regardless of index; if they're
the same resolution, keep them on stable ports or pass `--scene-index/--wrist-index`.

## Run

```
python3.11 arview.py --config arm
```

Each camera uses its saved intrinsics (frames undistorted, pose solved with the calibrated
K); a camera with no saved intrinsics falls back to a live single-view focal estimate.

Fixed 3-column grid, re-rendered every frame from the live joint state:

| row | col 0 | col 1 | col 2 |
|-----|-------|-------|-------|
| **scene cam (cam 0)** | rgb | scene render | umi render *(if the UMI board is observed)* |
| **wrist cam (cam 1)** | rgb | scene render *(if the base board is observed)* | blank |

- **scene render** — the arm+UMI model, localized from the **arm-base** board.
- **umi render** — the UMI gripper model (with its own ArUco plane), localized from the
  **UMI board** — a second, independent render source (scene-cam row only).
- Each render is a real|sim 50% blend; a cell shows a dim "not observed" panel when its
  board isn't in view. The wrist row's umi cell is intentionally blank (the UMI board is
  on the gripper, never in the wrist camera's own view).

`q` quits. Servos are read-only (torque off, so you can back-drive).

Render layers live in `render_core.py` (a `Layer` = one model + one board; measures its
board frame, poses itself from joints, renders from a localized pose). `arview.py` is the
thin driver that reads the servos and drives both layers.

## How it works (reused from the cad pipeline)
- `calib_utils.py` + `aruco_boards.py` — copied from the cad relocalization pipeline
  (single-image intrinsics, PnP, the RX180 OpenCV↔MuJoCo camera convention).
- Board frame in the sim is recovered once at startup by **render loop-closure** (render
  the sim board, detect it, back out its world frame) — no hand-guessed UV conventions.
- The model is `arm_umi_v2.xml` (from the `arm` config), which already carries the base
  board (`aruco_plane80`) and a `wrist_cam`.

Verified by loop-closure self-test (`_selftest.py`, runs against the sim only): both
boards recover cleanly — arm-base 0.9 px / 0.7 mm, UMI board 2.7 px / 0.4 mm.

## Flags / tuning
- `--cam0 N` / `--cam1 N` — camera indices (default 0 and 1).
- `--width W` — working width per pane (default 640; raise if board detection is flaky).
- `--cam0-focal PX` / `--cam1-focal PX` — fix a camera's focal length instead of estimating
  it from the first board view. For best alignment on the real cameras, do a proper
  intrinsics calibration and pass the focal here (the auto-estimate is a single-view guess).
- Alignment quality on real streams depends on the servo calibration (`calib/arm.json`) and
  the camera intrinsics — the sim math is verified; the physical fit is what you tune here.