# YAM — inference / deploy How the model runs on the robot. Cross-view fusion math, live-recal, fusion modes, ergonomics flags. ## Topology ``` [ russet ] [ puget (GPU 0) ] ZED 2i (HD1080) ws ┌────────────────────┐ ZED Mini × 2 (HD720) ─────────► │ deploy.py │ i2rt YAMs :8765 │ ┌──────────────┐ │ │ │ chiral client│ │ │ └──────┬───────┘ │ │ ▼ │ │ DinoVolume... │ │ │ │ │ ┌──────▼───────┐ │ │ │ fusion decode│ │ │ └──────┬───────┘ │ │ ▼ │ │ smooth_move_to │ │ │ │ ws ◄──────┘ │ │ │ │ rerun web :9094 │ └────────────────────┘ ``` `rd serve` on russet is the gateway. Inference + decoding runs on puget. The model never leaves puget. Joint commands stream back via `smooth_move_to`. ## Launch (current canonical) On russet: ```bash ssh robot-lab # password: robot-lab cd ~/cameron && rd serve --action-type joint --resize-images 1080x1920 # wait for "Raiden policy server ready" ``` On puget: ```bash ssh dev cd ~/yam_para/code CUDA_VISIBLE_DEVICES=0 \ DINO_REPO_DIR=$HOME/yam_para/dinov3 \ DINO_WEIGHTS_PATH=$HOME/yam_para/dinov3/weights/dinov3_vits16plus_pretrain_lvd1689m-4057cbaa.pth \ python deploy_yam_2view.py \ --ckpt ~/yam_para/checkpoints/yam_wrist_xattn_v1/latest.pth \ --t_lfr_path ~/yam_para/data/place_mug_wrist_cam/0000/lowdim/0000000000.pkl \ --server_uri ws://10.110.22.11:8765 \ --arms_mode single --arm right \ --action_type joint --use_smooth_move \ --chunk_size 8 \ --max_joint_vel_rad_s 0.3 \ --first_move_vel_rad_s 0.2 \ --vis_port 9094 \ --use_wrist 1 \ --scene_only_decode 1 \ --fuse_mode sum \ --no_render \ --eval_name xattn_v1_test ``` Drops you at the deploy prompt: ``` [y]=deploy / [r]=re-query / [h]=home (stay) / [c]=recalibrate cam / [q]=quit+home > ``` - **y**: deploy the next chunk and roll forward. - **r**: re-run inference (don't move). - **h**: home the arm, keep deploy session alive. - **c**: re-run live exo aruco recal. - **q**: home + quit (kills `rd serve` — restart for next session). ## Flags | Flag | Default | Purpose | |---|---|---| | `--use_wrist 1` | required if ckpt has wrist | Forward both views; load `F_oof_token` | | `--wrist_cam_name right_wrist_camera` | | Which obs key holds the wrist frame | | `--fuse_mode sum` | sum | `sum` or `max` for cross-view log-prob fusion | | `--scene_only_decode 1` | 0 | Forward both views (keep CLS concat + viz) but **skip** the geometric fusion step — decode comes from scene only | | `--use_train_cam` | off | Load scene extrinsic from `--t_lfr_path` PKL instead of live recal — use when live recal is wrong | | `--no_render` | off | Skip the exo+arm cam_check render (headless / fast spin-up) | | `--save_video` | off | Save MP4 of rerun viewport; heavy, only for showcase runs | | `--eval_name ` | autogen | Names the saved video + the wandb-style eval record | | `--max_joint_vel_rad_s 0.3` | | Per-substep velocity cap | | `--first_move_vel_rad_s 0.2` | | Slower velocity for first substep of each chunk | | `--chunk_size 8` | 8 | Substeps deployed per inference call | | `--min_detections 8` | 8 | Aruco frames pooled at recal | | `--max_wait_s 20.0` | 20 | Hard cap on recal wait | ## Fusion math For each predicted (t, a) (timestep, arm): 1. **Scene decode**: argmax marginal Z → `z*`. The model emits a 56×56 YX log-softmax per (t, a). 2. **For each (y, x) in the scene grid**, lift to 3D at the bin center `z*`: ```python p_world = unproject(K_scene, T_scene_in_world, (x, y), bin_centers_z[z*]) ``` 3. **Project into wrist**: ```python p_wrist_pix = project(K_wrist, T_world_to_wrist, p_world) ``` 4. **Sample wrist log-softmax** at `p_wrist_pix`. If out of frame, mark abstain. 5. **Combine scene + wrist log-probs**: - `sum`: scene + wrist (when wrist not abstain; else use scene alone) - `max`: elementwise max (with abstain = -inf for max, 0 for sum) 6. **argmax** the fused volume → final pixel + 3D point. When `--scene_only_decode 1`, steps 2-5 are skipped (still forward both views and log the wrist panels for inspection). ## Live recal at startup Same logic as recorder — see `calibration.md` for pool-until-N pattern. Deploy reports: ``` training K diag = (1122.9, 1122.9); cx,cy=(946.8,571.3) live K diag = (1123.0, 1123.0); cx,cy=(946.8,571.3) ``` If translation drift > a few cm or rotation > ~5°, deploy warns and recommends `--use_train_cam`. Don't suppress the warning — investigate. ## Eval videos When `--save_video` is on, MP4s are written to `~/yam_para/eval_videos/__.mp4` on puget. Russet's `~/cameron/puget/` sshfs-mounts puget's `~/yam_para/`, exposing the videos at: ``` https://omidlab.net/browse/yam_remote/cameron/puget/eval_videos/ ``` Naming convention: `___.mp4`. E.g. `wrist_eval_first_take__2026-06-01_154212.mp4`. ## Sessions interleave with rd serve Every `q` (or unexpected disconnect) homes the arm + triggers `rd serve` estop + shutdown. Restart `rd serve` before each new deploy. See `known-bugs.md`. ## Performance notes - HD1080 scene + HD720 wrist obs frame is ~1-2 MB encoded. Round-trip over ws can be 100-300 ms when the rig is busy. - Lock contention is the dominant slowdown — inter-substep delays come from the obs stream coroutine grabbing the lock before the next `smooth_move_to`. - The **safe** mitigations: - Lower obs stream hz (`env.start_obs_stream(hz=5)` instead of 30) while a chunk is executing. - Or set hz low for the whole session if you only need fresh frames between inference calls, not every 33 ms. - The **unsafe** mitigation (do not use): `env.stop_obs_stream()` then RPC. Orphans a frame on the wire — see `known-bugs.md`. ## Rerun viewer Web on port 9094 (default 9092 may be taken on puget). SSH tunnel from your Mac: ``` ssh -L 9094:127.0.0.1:9094 -L 9095:127.0.0.1:9095 dev ``` **127.0.0.1, not "localhost"** — see `memory.md`. URL printed at deploy startup: ``` http://localhost:9094?url=rerun%2Bhttp%3A%2F%2F127.0.0.1%3A9095%2Fproxy ``` What's logged: see `rerun-panels.md`.