# panda_streaming/ — Real Panda Robot + Streaming

## Purpose
Real-robot deployment and data collection pipeline for a Franka arm (driver may use `arm_id:=fr3` on hardware).
Handles live joint state streaming over ROS2/rosbridge, camera feed, MuJoCo visualization, and dataset recording.

**MuJoCo model:** `robot_models/franka_emika_panda/panda.xml` (Emika Panda MJCF), configured via `ExoConfigs/panda_exo.py`. Joint names on `/joint_states` are matched as `panda_joint1`…`panda_joint7` (typical franka_ros), `joint1`…`joint7`, or legacy `fr3_joint*` / `fr3v2_joint*` for compatibility.

## Folder Structure

```
panda_streaming/
├── CLAUDE.md                        ← you are here
├── scripts/
│   ├── start_robot_server.sh        ← launch SSH tunnel + ROS2 services (tmux)
│   └── run_teleop.sh                ← launch GELLO blue teleop (tmux)
├── stream_panda_with_vis.py         ← MuJoCo viewer with live joint states
├── stream_panda_with_cam.py         ← live camera feed + MuJoCo overlay
├── simple_dataset_record_panda.py   ← dataset recording (RGB + joints)
├── exo_utils.py                     ← ArUco pose estimation + rendering helpers
├── ExoConfigs/                      ← Panda exoskeleton MuJoCo configs
├── robot_models/                    ← MuJoCo robot XML assets
└── requirements.txt                 ← pip dependencies (roslibpy, mujoco, etc.)
```

## Remote Server Access

- **Connection:** ngrok TCP tunnel — host and port change each session, check ngrok dashboard
- **User:** `root`
- **Password:** `elitesmasher99` (hardcoded in scripts)
- **SSH flag for new ngrok address:** `-o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=/dev/null`

## Workflow: Start Joint State Streaming

### Step 1 — Launch remote ROS2 services + local tunnel

```bash
./scripts/start_robot_server.sh <ngrok_host> <ngrok_port>
# e.g.: ./scripts/start_robot_server.sh 4.tcp.us-cal-1.ngrok.io 12345
```

This starts 3 tmux sessions:
| Session | What it runs |
|---|---|
| `panda_tunnel` | Local SSH tunnel: `localhost:9090` → remote `9090` |
| `panda_rosbridge` | `ros2 launch rosbridge_server rosbridge_websocket_launch.xml` |
| `panda_driver` | `ros2 launch franka_bringup franka.launch.py robot_ip:=169.254.2.83 arm_id:=fr3 load_gripper:=true` |

Wait for the Franka driver to print `Successfully connected to robot` before running Mac-side scripts.

### Step 2 — Run on Mac

```bash
# MuJoCo viewer (joint state visualization only)
python stream_panda_with_vis.py

# Camera feed + MuJoCo overlay (ArUco-based camera pose)
python stream_panda_with_cam.py

# Dataset recording
python simple_dataset_record_panda.py <stream_host> <stream_port>
```

### Step 3 — (Optional) GELLO Teleop

```bash
./scripts/run_teleop.sh <ngrok_host> <ngrok_port>
# Starts tmux session: panda_teleop
# Runs: cd /workspace/ros2; python3 scripts/calib.py --gello_name blue_gello; ./scripts/blue_gello_teleop.sh
```

## Attaching to tmux sessions

```bash
tmux attach -t panda_tunnel
tmux attach -t panda_rosbridge
tmux attach -t panda_driver
tmux attach -t panda_teleop
tmux ls   # list all sessions
```

## Remote Machine Layout

| Path | Contents |
|---|---|
| `/opt/ros/humble/` | ROS2 Humble base install |
| `/root/franka_ws/` | Franka FR3 ROS2 driver workspace |
| `/workspace/ros2/` | GELLO teleop workspace |
| Robot IP | `169.254.2.83` (FR3) |

## Notes

- The SSH tunnel (`panda_tunnel`) must stay running for rosbridge to be accessible at `localhost:9090`
- The `panda_driver` session must show `Configured and activated joint_state_broadcaster` before joint states flow
- ArUco markers on the robot are used by `stream_panda_with_cam.py` for camera pose estimation
- Data format: per-frame PNG + `_gripper_pose.npy` (4×4 world pose of MuJoCo body `virtual_gripper_keypoint` on the Panda hand) + `_camera_pose.npy` + `_cam_K_norm.npy` + `.npy` (joint states)
- Camera intrinsics are seeded from a hardcoded estimate and refined per-frame via ArUco detection