# panda_streaming/ — Real Panda streaming, dataset, training, deploy

Everything that touches the real robot lives here. Read
[`../CLAUDE.md`](../CLAUDE.md) first for the project-wide context, then
this file for what's in this directory specifically.

## Files

| File | What it does |
|---|---|
| `data_panda_para.py` | `PandaTrajectoryDataset`. Loads pre-cached 448×448 JPGs + per-frame joint NPYs, computes FK + projection at init time. Exports the **calibrated** `T_CAM_WORLD` and `CAM_K` that the rest of the pipeline uses. |
| `train_panda_para.py` | Training entry point. Imports `TrajectoryHeatmapPredictor` from `../para/model.py`, computes height/gripper/rotation stats from the dataset, runs the standard CE loss combo, logs per-timestep visualization strips to wandb. |
| `simple_dataset_record_panda.py` | Live recording: subscribes to `/joint_states` over rosbridge, grabs RealSense frames, writes per-frame PNG + joint NPY pairs to `scratch/`. |
| `parse_video_into_episodes_panda.py` | Interactive matplotlib tool: scrub through a recording, press keys to mark episode boundaries, output goes to `parsed_<run>/`. |
| `vis_dataset_gt.py` | Render an episode's GT keypoints + masks for visual sanity check. |
| `stream_panda_with_vis.py` | MuJoCo viewer with live joint states (no camera). Useful for debugging joint-name mapping and base pose. |
| `stream_panda_with_cam.py` | Live RealSense feed with MuJoCo overlay using ArUco-based per-frame camera pose. Replace the per-frame ArUco pose with calibrated `T_cam_world` once you have it (see `TASKS.md` Stage 1). |
| `test_ik_recovery.py` | Stage 3 sanity check: take recorded EEF poses, run damped least-squares IK, render and compare to GT. |
| `deploy_ik_sequence.py` | Closed-loop deployment driver: model → 3D point → IK → publish on `/gello/joint_states`. Handles publisher startup ordering, position settling, velocity-limited ramps. |
| `exo_utils.py` | ArUco detection + pose estimation, plus `position_exoskeleton_meshes` and `get_link_poses_from_robot` used by every renderer. |
| `requirements.txt` | Just `roslibpy>=1.7.0`. The rest comes from the conda env. |

## Subdirectories

| Path | Contents |
|---|---|
| `ExoConfigs/` | Panda + ArUco board configs. Concrete configs: `panda_exo.py` (6×6 base), `panda_exo_4x4.py`, `panda_exo_5x5.py`, `panda_exo_6x6.py`, `panda_exo_7x5.py`, `panda_exo_handeye_4x2.py` (current hand-eye board). Base classes in `exoskeleton.py`. The `__init__.py` was trimmed to only export the base classes; import the concrete configs directly. |
| `robot_models/franka_emika_panda/` | MJCF + meshes for the Emika Panda. `panda.xml` is the "with hand" version used here. |
| `robot_models/board_imgs/` | PNG textures for the ArUco boards. |
| `scripts/` | `start_robot_server.sh`, `run_teleop.sh`, `kill_panda_server.sh`. Drive the robot box's tmux sessions from your local machine. |
| `hand_eye_calib/` | All calibration scripts. See [`hand_eye_calib/CLAUDE.md`](hand_eye_calib/CLAUDE.md). |

## Data format on disk

Per recorded frame `NNNNNN`:

| File | Contents |
|---|---|
| `NNNNNN.png` | RGB from the RealSense (raw, not resized). |
| `NNNNNN.npy` | joint state vector — `[q1..q7, gripper_pos]`, gripper in `[0, 0.04]`. |
| `NNNNNN_gripper_pose.npy` *(optional)* | 4×4 world pose of the MuJoCo body `virtual_gripper_keypoint`. |
| `NNNNNN_camera_pose.npy` *(optional)* | 4×4 camera-in-world pose, only for sessions where it was logged per-frame. |
| `NNNNNN_cam_K_norm.npy` *(optional)* | normalized camera intrinsics. |

After `parse_video_into_episodes_panda.py` you also get:

| File | Contents |
|---|---|
| `episodes.json` | `{"episodes": [{"start": int, "end": int, "task": str, ...}, ...]}` |
| `cached_448/<frame>.jpg` | pre-resized 448×448 RGB (run a pre-cache script after parsing). |

`PandaTrajectoryDataset` requires the `cached_448/` directory and
`episodes.json`. It will raise `FileNotFoundError` if either is missing.

## Calibration values

`data_panda_para.py` ships hardcoded `T_CAM_WORLD` and `CAM_K` from a
historical capture. They are almost certainly wrong for any new mount.
**Replace them** after Stage 1 — either edit the constants or load from a
JSON. The helper `project_to_pixel(pos_world, T_cw, K)` is exported for
reuse in visualizations.

## Joint name conventions

The streaming scripts try several joint-name patterns (`panda_joint{i}`,
`joint{i}`, `fr3_joint{i}`, `fr3v2_joint{i}`) so the same script works
across different driver flavors. `deploy_ik_sequence.py` publishes on
`fr3_joint{i}` because the production driver uses the FR3 names.

## Things to watch out for

- **Quaternion order.** `/joint_states` and MuJoCo `xquat` both use
  `[w, x, y, z]`. SciPy `Rot.from_quat` wants `[x, y, z, w]`. Reorder.
- **MuJoCo vs OpenCV camera convention.** See `../CLAUDE.md` for the
  conversion. Pretty much every "the overlay is rotated 180°" bug traces
  back to forgetting this.
- **Gripper sign.** In NPY frames `gripper_pos ∈ [0, 0.04]`; in training
  it's normalized to `[-1, 1]`; in policies typically `+1` = close,
  `-1` = open.
- **Image resolution drift.** RealSense raw is 1920×1080. Cached training
  images are 448×448. Intrinsics for projection must be scaled
  consistently — `data_panda_para.CAM_K_448` is the rescaled version.

## Don'ts

- Don't commit `wandb/`, `checkpoints/`, `scratch/`, or `__pycache__/`.
  All of these are gitignored at the repo root.
- Don't run `train_panda_para.py` from a directory other than
  `panda_streaming/` — it uses `os.path.dirname(__file__)` for relative
  imports of `data_panda_para` and `../para/model.py`.