# YAM — known bugs and recovery procedures

The fragile-ness inventory. Workarounds inline.

## 1. chiral `stop_obs_stream()` orphans a frame

**Symptom**: After calling `env.stop_obs_stream()` then an RPC, the RPC
returns garbage (e.g. `smooth_move_to` → `{}` no `"ok"` field). Then
`env.start_obs_stream()` restarts → next obs decode raises
`KeyError('cameras')`.

**Cause**: `_aobs_stream` holds the single asyncio lock across `send` +
`recv` of the obs request:

```python
async with lock:
    await self._send(encode_obs_request())
    data = await self._recv()
```

When we cancel between `send` and `recv`, the obs response sits on the
wire unconsumed. The next RPC's `recv` (under the lock again) consumes
that orphan frame. Then the real response sits on the wire until the
*next* recv — typically the restarted obs stream — which tries to
decode a `smooth_move_response` as an obs frame.

**Don't do this** (caused real bug 2026-06-02):

```python
env.stop_obs_stream()
for cmd in joint_cmds:
    env.smooth_move_to(cmd, ...)
env.start_obs_stream(hz=30)
```

**Safer mitigations**:
- Lower obs hz instead of stopping: `env.start_obs_stream(hz=5)` at
  session start.
- Stop the stream at session start, run all RPCs without it, restart at
  session end (no chunk-loop mid-flight cancellation).
- Patch chiral to put `decode` inside the `async with lock:` block and
  to drain any pending recv on cancel (proper fix, but means modifying
  chiral).

## 2. `rd serve` dies on client disconnect

**Symptom**: After deploy quits (or disconnects), `rd serve` prints:

```
[RaidenPolicyServer] Client disconnected — triggering emergency stop.
  EMERGENCY STOP ACTIVATED
  All arms reached home position.
  System will now shut down.
```

Then `rd serve` exits. Next deploy attempt: `websockets.exceptions.ConnectionClosedError`.

**Cause**: Intentional safety: the server treats every disconnect as an
estop request from the client side.

**Workaround**: Restart `rd serve` before each new deploy session.
There is no in-server "soft disconnect" yet. (Possible future feature:
client-side `goodbye` message that distinguishes intentional quit from
crash — needs a raiden PR.)

## 3. ZED 2i lockup recovery — physical only

**Symptom**: ZED SDK calls hang or return zero-shape buffers. `rd serve`
prints zed init errors and refuses to start.

**Cause**: Usually after a `Killed` process while streaming HD1080
(OOM, hard SIGKILL). The ZED firmware enters a wedge state that no
software reset clears.

**Recovery** (only known methods):
1. Unplug the camera USB for **30+ seconds**, replug.
2. Reboot the host.

**Failover**: yukon has a spare ZED 2i — point russet at it (USB
re-routing) if recovery takes > 5 min.

See `feedback_polling_cadence.md` in auto-memory: don't `sleep 30s` to
poll for recovery — terminal feedback comes in seconds when it does
recover.

## 4. opencv-python missing `aruco.detectMarkers`

**Symptom**:

```
AttributeError: module 'cv2.aruco' has no attribute 'detectMarkers'
```

**Cause**: pip's stock `opencv-python` ships without contrib modules.
`cv2.aruco.detectMarkers` is in **contrib only**.

**Fix**:

```bash
uv pip install --force-reinstall opencv-contrib-python==4.13.0.92
```

Force-reinstall is required because pip leaves a broken `cv2/` dir
behind on bare uninstall.

## 5. `success: True` missing in archival JSON → identity extrinsics

**Symptom**: After `rd convert`, the per-frame PKLs have
`right_scene_camera_extrinsic = I_4`. Workspace mask collapses to ~30k
valid voxels.

**Cause**: Raiden's converter requires `"success": True` in
`calibration_results.json` for the recording. If absent, it silently
drops the extrinsics block.

**Fix**: ensure `record_skip_wrist.py` (and any custom recorder) writes
`"success": True` at the end of each episode. Re-convert affected
episodes after fixing the JSON.

## 6. `cv2.warpAffine` shift missing → overlay drift ~10 px

**Symptom**: cam_check overlay arm is offset from the real arm by
roughly 10-20 pixels in both axes.

**Cause**: MuJoCo's `cam_fovy` rasterizer assumes principal point at
image center. Real ZED intrinsics are off-center.

**Fix**: see `rendering.md` for the warp recipe.

## 7. K not scaled to render res → overlay arm half-size

**Symptom**: cam_check overlay arm is visibly smaller than the real
arm, ~50%.

**Cause**: HD1080-native K (fx ≈ 1122) passed to a 1280×720 render →
fovy comes out at ~35° instead of ~52°.

**Fix**: scale K to render res before computing fovy. See
`rendering.md`.

## 8. `vis_every_steps=0` zero-div

**Symptom**: Training crashes immediately with `ZeroDivisionError`.

**Cause**: Step counter modulo 0.

**Fix**: Use `--vis_every_steps 100000` to effectively disable viz.

## 9. SSH `localhost` resolution on russet

**Symptom**: SSH `LocalForward localhost:9094` to russet fails;
`127.0.0.1` works.

**Cause**: russet's nsswitch doesn't resolve "localhost" reliably under
some tunnel paths.

**Fix**: always use `127.0.0.1` literal in `LocalForward` directives
when russet is in the chain.

## 10. Mis-named `right_base_to_left_base` key

**Symptom**: After loading `T_lfr` directly from JSON, all wrist
projections are inverted across the bimanual midline.

**Cause**: raiden's calibration JSON labels the matrix
`right_base_to_left_base` but stores its inverse.

**Fix**:

```python
T_lfr = np.linalg.inv(json_data["right_base_to_left_base"])
```

See `memory.md` and `feedback_raiden_bimanual_transform_misnamed.md`.

## 11. Volume-head dot products must NOT be L2-normalized

**Symptom**: Volume CE loss plateaus around ~11 nats. The per-T marginal
heatmap stays flat across timesteps even hours into training. Rotation
+ gripper heads converge normally; only the volume head misbehaves.

**Cause**: Bilinear scoring under cosine-similarity (`F.normalize(q_F)`,
`F.normalize(F_scene)`, etc.) bounds all dot products to [-1, 1]. The
per-T softmax over (Z·P·P) ≈ 524k voxels needs logit *spreads* of ~ln(524k)
≈ 13 to sharpen onto the GT voxel — physically impossible when each
logit is in [-1, 1].

**Fix**: raw dot products, no `F.normalize`, no `log_temperature`:

```python
score_yx = einsum("btc,bchw->bthw", q_F, F_scene)     # raw
score_z  = einsum("btc,zc->btz",   q_z, z_sin)        # raw
score_t  = einsum("btc,tc->bt",    q_t, t_sin)        # raw  (diag — q_t[T] · t_sin[T])
vol = score_yx[:, :, None] + score_z[:, :, :, None, None] + score_t[:, :, None, None, None]
```

Magnitudes are allowed to grow as training aligns the queries.
Confirmed 2026-06-02: switching from normalized to raw dropped step-100
volume loss 11.0 → 5.9.

## 12. IK chain collapses to seed when fall-back-on-fail throws away partial-q

**Symptom**: Per-timestep mujoco renders during inference look identical
across all 16 timesteps even though the 2D predicted-keypoint
trajectory (in wandb / rerun) is clearly diverse. IK debug shows
all-FAIL with pos errors > 10mm at later timesteps.

**Cause**: ``ee_chain_to_joints`` seeded each step's IK from the
previous step's solution. If the IK was treated as "failed" (didn't
hit the strict 1e-4 pos/ori thresholds) and the implementation
replaced ``q`` with the seed, the next step started from the same
seed, cascading "stay at home" across the whole chunk.

**Fix**: keep the partial-q on threshold-miss. The pre-refactor
``ee_actions_to_joint_cmds`` does this implicitly by ignoring the
``ok`` flag — at 1e-4 thresholds even excellent solutions (~5-10mm)
nominally "fail," so the chain has to use them anyway.

```python
ok, q, info = ik_solve(target, seed)
if not ok:
    pass        # keep partial q, do NOT fall back to seed
```

Confirmed 2026-06-03: switching fixed-fallback → keep-partial dropped
chained per-T pos error 8-25 mm → 0.3-8 mm and made the per-T joint
renders visibly sweep the trajectory.

## 13. Workspace-mask + cell-center pixel offsets must match the pre-refactor lifting

**Symptom**: undertrained model picks voxels behind the robot base
(or behind the camera) for the argmax of ``volume_logits``; lifted
world XYZ targets are physically invalid; IK can't reach them;
chunk collapses to seed.

**Fix**: at decode time, apply ``vol.masked_fill(~ws_mask, -1e9)``
**before** argmax. ``ws_mask`` is ``compute_workspace_mask`` (see
`lib/inference.py`) — same logic as the pre-refactor
``compute_workspace_mask`` in deploy_yam_2view.py:919. Also use
**cell-center** pixel coords (``(bin + 0.5) * cell_size``), not
top-left corner, for the unproject.

## 15. `'Robot' object has no attribute 'smooth_move_to'`

**Symptom**:

```
AttributeError: 'Robot' object has no attribute 'smooth_move_to'
```

…and the arm collapses under gravity (deploy crashes mid-session,
torque released without a homing move).

**Cause**: Conflating the chiral `PolicyClient.smooth_move_to(...)` API
(remote, used when deploy ran on puget) with the local raiden-direct
`Robot.move_to(...)` wrapper used by the post-rehaul on-russet deploy.
The vault's `control.md` documents the chiral API; `lib/robot.py` is
the local class.

**Fix in code**: use `robot.move_to(target14, vel_rad_s=...)` and
**wrap robot mains in `try/finally + go_home`** so any future crash
homes the arm before releasing torque:

```python
try:
    while True:
        ...
finally:
    go_home()
    robot.shutdown()
    cam.close()
```

Confirmed 2026-06-03 in `deploy.py`.

## 16. Tailscale SSH session expires (recurring sshfs breakage)

**Symptom**: sshfs mount between two TRI machines goes I/O-error;
processes block in D-state on mount reads (uninterruptible by
`kill -9`). Plain ssh to the same host returns:

```
# Tailscale SSH requires an additional check.
# To authenticate, visit: https://login.tailscale.com/a/<token>
```

**Cause**: Tailscale SSH auth on the outbound machine expired (sessions
are time-limited even after the one-time browser approval). All sshfs
mounts and outbound ssh stop working until the user re-auths in a
browser. **Recurs** — has bit puget→lab and russet→lab on 2026-06-03
within hours of each other.

**Recovery procedure**:

1. **Unwedge the dead sshfs mount**: `fusermount3 -uz ~/lab` (lazy
   unmount). D-state processes will then unblock and die.
2. **Trigger the auth URL** by running plain ssh:
   `ssh <outbound-machine> 'ssh cameronsmith@100.74.71.38 hostname'`.
   The Tailscale URL prints; user opens it in a browser and approves.
3. **Remount lab** with the standard options:

   ```bash
   ssh <outbound-machine> 'sshfs cameronsmith@100.74.71.38:/data/cameron ~/lab \
     -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,follow_symlinks,cache=yes,kernel_cache,compression=no'
   ```

4. **Verify**: `mountpoint ~/lab && ls ~/lab/para/robot/yam/`.

For the **school server → russet** mount (`~/mnt/robot-lab/` →
`/data/cameron/yam_remote/`), use:

```bash
sshfs robot-lab:/home/robot-lab/ ~/mnt/robot-lab/ \
  -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,follow_symlinks,cache=yes,kernel_cache,compression=no
```

(Uses the `robot-lab` SSH alias which already has `ProxyJump dev`.)

Future fix worth considering: switch outbound from Tailscale SSH to a
pubkey-pinned identity so sessions don't expire.

## 17. torch wheels: cu126 works on Ada, fails on Blackwell

**Symptom**:

```
UserWarning: NVIDIA RTX PRO 6000 Blackwell Max-Q ... with CUDA capability sm_120
is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 ... sm_90.
...
RuntimeError: CUDA error: no kernel image is available for execution on the device
```

…even though `torch.cuda.is_available()` returns `True`.

**Cause**: torch wheels carry compiled CUDA kernels for a fixed set of
compute capabilities. The `cu126` wheels (great for russet's Ada
sm_89) include kernels up to sm_90 only — they have no Blackwell
support. Yukon's RTX PRO 6000 is **sm_120** which needs cu130+ wheels.

**Fix**:

```bash
# yukon (sm_120, driver 595, CUDA 13.2)
~/.local/bin/uv pip install --reinstall "torch==2.12.0" torchvision \
    --index-url https://download.pytorch.org/whl/cu130
```

Verify with a real matmul, not just `cuda.is_available()`:

```python
import torch
x = torch.randn(64, 64, device="cuda"); float((x @ x.T).sum())
```

**Avoid the trap**: `torch.cuda.is_available()` returns True even when
no compatible kernels exist — every cuda call then errors at launch.
Always verify with a real op. See FAQ "Per-rig torch matrix".

## 18. DA3 patch_size=14 strict — img_size must divide by 14

**Symptom**:

```
File ".../depth_anything_3/model/dinov2/layers/patch_embed.py", line 73
    H % patch_H == 0
AssertionError: Input image height 405 is not a multiple of patch height 14
```

**Cause**: DA3's DINOv2 patch embed enforces `H % 14 == 0` strictly (no
interpolation fallback like our DINOv3 ViT-S/16+). Our default
`--img_size 405` (chosen for DINOv3's 16+ stride) breaks DA3.

**Fix**: train DA3 with `--img_size 504` (=14×36, DA3's native training
res) or any other multiple of 14. `lib/model_da3.py` has `patch_size=14`
on its `_backbone_features` patch-grid math and assumes a square
multiple of 14 input.

## 19. Local `wandb/` dir shadows pip-installed wandb (yukon)

**Symptom**:

```
AttributeError: module 'wandb' has no attribute 'init'
```

`python -c "import wandb; print(wandb.__file__)"` from
`~/lab/para/robot/yam/` returns `None`.

**Cause**: When `train.py` does `sys.path.insert(0, parent_dir)` to
enable `from lib.* import …`, that parent dir takes precedence on
import. Any sibling `wandb/` subdirectory (left over from past offline
wandb runs) is found first as a **PEP 420 namespace package** and
shadows the pip-installed wandb. Namespace packages have no `__init__.py`
and expose no attributes — `wandb.init` raises.

**Fix**: rename or remove the local `wandb/` dir. Renamed to
`wandb_old_offline/` on lab so future runs can't shadow but the old
offline run data is preserved if needed.

## 14. Inter-substep delays during chunk execution (open)

**Symptom**: After each `smooth_move_to` returns, there's a ~3 s pause
before the next substep fires. Robot moves at full speed when moving,
but is idle between substeps.

**Cause (suspected)**: chiral lock contention + slow round-trips. The
obs stream coroutine grabs the lock at 30 Hz; each grab takes
~100-300 ms when payloads are HD1080+wrist. After each substep,
`smooth_move_to` must wait for the obs stream to release.

**Tried + failed**: pausing the obs stream mid-loop — caused bug #1.

**Not yet tried**:
- Lower obs hz to 5 for the whole deploy session.
- Lower obs hz to 5 just during chunk execution (without
  start/stop — see if chiral supports dynamic rate).
- Drop wrist cam from obs (keep scene only) during deploy.

This is task `tasks.md` → "Investigate inter-substep delay".