# YAM — exo+arm mujoco rendering

The "cam_check" overlay that draws the arm (with the exo aruco board) on
top of the live scene cam. Used by both recorder (visual calib sanity)
and deploy (`--no_render` skips it). Two pitfalls dominate: principal
point and K-resolution.

## Source model

`YAM_BASE_ONLY_CONFIG` from `raiden_fork/third_party/exo_redo/`. Includes
the YAM arm chain + the larger_coarse_board aruco plate at the right
arm base.

```python
import sys
sys.path.insert(0, "raiden_fork/third_party/exo_redo")
from ExoConfigs.yam_exo import YAM_BASE_ONLY_CONFIG
```

The config carries `mjcf_xml` + `texturedir` separately; we need both
for textured rendering.

## Texturedir injection

Out of the box, the MuJoCo XML references the aruco PNG by a relative
path. Without `texturedir`, the renderer fails unless `cwd` matches.
Patch the `<compiler>` tag at load time:

```python
_EXO_DIR = os.path.dirname(YAM_BASE_ONLY_CONFIG.mjcf_path)
xml = open(YAM_BASE_ONLY_CONFIG.mjcf_path).read()
xml = xml.replace("<compiler ", f'<compiler texturedir="{_EXO_DIR}/" ')
model = mujoco.MjModel.from_xml_string(xml)
```

## Principal point warp (the silent gotcha)

MuJoCo's `cam_fovy` rasterizer assumes the principal point is at the
image center: `cx = W/2, cy = H/2`. Real ZED 2i intrinsics put it ~14
pixels off-center in y, ~13 px off in x at HD1080. Without compensation
the overlay drifts.

```python
M = np.float32([[1, 0, cx - W / 2], [0, 1, cy - H / 2]])
rendered = cv2.warpAffine(rendered, M, (W, H))
```

Apply this **after** the MuJoCo render, **before** compositing with the
segmentation mask. See `feedback_mujoco_principal_point_warp.md` in
auto-memory.

## K scaling for render resolution

MuJoCo `cam_fovy = 2 * arctan(H_render / (2 * fy))` — uses the render
resolution, not the native intrinsic resolution. If you pass the
HD1080-native K (fx, fy ≈ 1122.83) into a 1280×720 render, you get
fovy ≈ 35.6° instead of the correct ~52°. Scale K before computing fovy:

```python
DW, DH = 1280, 720          # render resolution
W_native = 1920 if cx > 800 else 1280   # rough heuristic: HD1080 vs HD720
H_native = 1080 if W_native == 1920 else 720
K_render = K.copy()
K_render[0, 0] *= DW / W_native
K_render[0, 2] *= DW / W_native
K_render[1, 1] *= DH / H_native
K_render[1, 2] *= DH / H_native
fovy_deg = 2 * np.degrees(np.arctan(DH / (2 * K_render[1, 1])))
```

Then use `K_render`'s cx, cy in the principal-point warp above.

## Per-thread GL context

`mujoco.MjrContext` and `mujoco.GLContext` are **not** thread-safe. If
the renderer is called from a background overlay worker (rerun) and the
main thread, give each thread its own context:

```python
# Per-thread, lazy-init
def get_gl_ctx():
    if not hasattr(_tls, "ctx"):
        _tls.ctx = mujoco.GLContext(W, H)
        _tls.ctx.make_current()
        _tls.mjr_ctx = mujoco.MjrContext(model, ...)
    return _tls.ctx, _tls.mjr_ctx
```

## Caching the model

`mujoco.MjModel.from_xml_string` is slow (~200 ms). Cache the model +
shared baseline `mjData`; only `qpos` changes per frame from live
joints.

## Where this is used

| Site | Purpose |
|---|---|
| `record_skip_wrist.py` — per-cam overlay workers | Visual sanity that the live calib + arm pose match the real image |
| `deploy_yam_2view.py` — `render_arm_with_exo_mask` | "cam_check" rerun panel; defaults off via `--no_render` |
| `wrist_verify_rerun.py` | Visual check that the wrist extrinsic is correct after calibration |

These three should share **one** renderer module post-refactor
(`lib/mujoco_render.py`).

## Composing with the live image

The renderer returns a `(rgb, mask)` pair (mask = where the model is
visible). Composite:

```python
out = live_bgr.copy()
out[mask > 0] = (0.5 * live_bgr[mask > 0] + 0.5 * rendered[mask > 0]).astype(np.uint8)
```

50/50 blend lets you see both layers; the misalignment shows up as
ghosting.

## Visual debug fingerprints

| Symptom | Likely cause |
|---|---|
| Overlay arm is ~half the size of real arm | K not scaled to render res (see K scaling) |
| Overlay arm offset by ~10-20 px | Principal point warp missing |
| Overlay arm is *exactly* the real arm but mirror-flipped | Forgot the `T_lfr` on the cam extrinsic — extrinsic is still in right_arm_base |
| Aruco board renders solid black | texturedir not injected |
| Black or empty frame | GL context not current on this thread |