# AR-view dataset format

Datasets produced by `record_dataset.py` (UMI gripper + arm rig). Each frame captures the
scene (third-person) and wrist camera images, the poses of the scene camera, wrist camera,
and UMI gripper in a common world frame, and the gripper servo state — all at a fixed FPS.

- **Recorder:** `ar_view/record_dataset.py` (runs on the Mac).
- **Local path:** `ar_view/datasets/<task>/`
- **Synced to puget:** `~/ar_datasets/<task>/` via `ar_view/sync_datasets.sh` (incremental rsync, on demand).

```
python record_dataset.py <task> [--fps 4] [--scene-width 500] [--wrist-width 500] ...
```

## Directory layout

```
datasets/<task>/
  meta.json
  scene/000000.jpg 000001.jpg ...   # third-person camera RGB (BGR on disk via OpenCV)
  wrist/000000.jpg 000001.jpg ...   # UMI wrist camera RGB
  state/000000.npz 000001.npz ...   # poses + gripper + timestamp
```

Frame index `NNNNNN` is **aligned across `scene/`, `wrist/`, `state/`** — `scene/000042.jpg`,
`wrist/000042.jpg`, and `state/000042.npz` are the same instant.

## Images

- **JPEG, quality 95.** Downscaled from the camera's native capture to `save_width`
  (default **500px** wide), **aspect ratio preserved** (height follows). Read with `cv2.imread`
  → BGR `uint8`.
- Native (pre-downscale) resolution is recorded per camera in `meta.json` (`scene.native_wh`,
  `wrist.native_wh`).

## `meta.json`

| field | meaning |
|---|---|
| `task`, `fps`, `started`, `ended`, `num_frames` | run info |
| `camera_convention` | `"opencv_z_forward"` — camera looks +Z, X right, Y down |
| `distortion` | `"zero (pinhole)"` — no distortion coeffs; **no undistort needed** |
| `world_frame` | the calib board's ArUco GridBoard frame (ids 109–132) |
| `boards.calib` / `boards.umi` | `{ids:[first,last], shape:[cols,rows], marker_m, sep_m}` for each board |
| `scene.native_wh` / `wrist.native_wh` | `[W,H]` full capture resolution before downscale |
| `scene.save_width` / `wrist.save_width` | saved image width (default 500) |
| `scene.K_native` / `wrist.K_native` | 3×3 intrinsics at the **native** resolution |
| `scene.K_save` / `wrist.K_save` | 3×3 intrinsics **scaled to the saved images** — use these with the JPEGs |
| `scene.intrinsics_source` | `"live (this run)"` — the iPhone is re-calibrated each run (it drifts) |
| `wrist.intrinsics_source` | `"saved intrinsics/wrist.json"` — fixed B0332 lens |
| `scene.intrinsics_reproj_px` | reprojection error of the live scene solve (quality check) |
| `jaw_servo_id`, `jaw_sign` | the gripper servo id and its calibration sign |

Intrinsics are a pinhole model: `fx=fy` (square pixels), principal point at the image center,
zero distortion. `wrist.K_*` is `null` if the wrist had no saved calibration at record time
(then wrist RGB is still saved but there is no `wrist_cam_pose`).

## `state/NNNNNN.npz`

Load with `np.load(path)`. All poses are **4×4 homogeneous, local→world** (`X_world = T @ X_local`),
in the calib-board world frame, OpenCV camera convention.

| key | dtype | meaning |
|---|---|---|
| `scene_cam_pose` | 4×4 | scene camera pose in world (from PnP of the calib board in the scene image) |
| `wrist_cam_pose` | 4×4 | wrist camera pose in world (from PnP of the calib board in the **wrist** image) |
| `umi_pose` | 4×4 | UMI gripper board pose in world (UMI board seen by the scene camera) |
| `scene_visible` / `wrist_visible` / `umi_visible` | bool | was that board actually detected this frame? |
| `gripper_tick` | int | raw jaw encoder tick (0–4095, ~2048 = calibrated zero); `-1` if the read failed |
| `gripper_rad` | float | calibrated jaw angle = `jaw_sign*(tick-2048)*2π/4096`; `NaN` if read failed |
| `t` | float | wall-clock timestamp (`time.time()`), seconds |

**Visibility / last-known:** when a board isn't detected in a frame, its pose **reuses the last
frame it was visible** and the `*_visible` flag is `False`. The scene camera is normally fixed
(so its pose is stable regardless), but the UMI moves — **filter on `umi_visible` if you need
only measured gripper poses.**

## Frames / conventions

- **World = the calib board's GridBoard frame** (fixed during a recording as long as the physical
  board doesn't move). Origin/axes are whatever `cv2.aruco.GridBoard` defines for ids 109–132.
  It is board-relative: moving the board between recordings moves the world frame with it.
- **Poses are the entity expressed in world** (its columns are the local axes in world; the
  translation is its position in world). To go the other way, invert.

## Loading example

```python
import json, cv2, numpy as np
d = "~/ar_datasets/testing_cube_in_bowl"
meta = json.load(open(f"{d}/meta.json"))
K = np.array(meta["scene"]["K_save"])          # matches the 500px scene JPEGs

f = 42
scene = cv2.imread(f"{d}/scene/{f:06d}.jpg")    # BGR uint8
wrist = cv2.imread(f"{d}/wrist/{f:06d}.jpg")
st = np.load(f"{d}/state/{f:06d}.npz")

scene_cam_pose = st["scene_cam_pose"]           # 4x4 camera->world
umi_pose       = st["umi_pose"]                 # 4x4 umi->world
jaw_rad        = float(st["gripper_rad"])
umi_seen       = bool(st["umi_visible"])

# reproject the UMI origin into the scene image (opencv pinhole, no distortion):
X_world = umi_pose[:, 3]                                  # umi origin, homogeneous
X_cam   = np.linalg.inv(scene_cam_pose) @ X_world
uv      = K @ (X_cam[:3] / X_cam[2])
print("umi projects to", uv[:2], "visible:", umi_seen)
```

To build trajectories, stack across frames (sorted by index) and drop where `*_visible` is False
if you only want measured poses.