# our_wandb — MVP plan (2026-06-10)

**Status:** plan confirmed, coordinating serve.py integration with website_builder before writing viewer code.

## TL;DR

Build a filesystem-driven "mock wandb": scan `run-*` dirs on disk, render a
wandb-style dashboard (runs list grouped by model name → per-run image gallery
+ config + summary + a loss curve with smoothing slider), auto-refresh ~10s.
Host as a thin `/runs/` route on the existing `serve.py` (omidlab.net), mirroring
the proven `data_viewer` pattern.

## Reconnaissance findings (looked at real run dirs)

**Data roots — ROLE.md's defaults were stale.** ROLE.md / vault listed
`/data/cameron/yam_remote/yam_para/code/wandb/`, but `yam_remote` symlinks to
the **abandoned russet mount** (`mnt/robot-lab`) and that path doesn't exist.
The live runs are on **yukon**:

```
/home/cameronsmith/mnt/yukon/cameron/puget/wandb/        ← most recent (today's runs)
/home/cameronsmith/mnt/yukon/cameron/puget/code/wandb/   ← older, complete runs
/home/cameronsmith/mnt/yukon/cameron/puget/data/wandb/
```

Roots stay configurable (env `WANDB_ROOTS`, comma-separated; these three are the default).

**Run dir layout (confirmed):**
```
run-<YYYYMMDD_HHMMSS>-<id>/
├── files/
│   ├── media/images/<panel>/<key>_<idx>_<step>_<hash>.png   ← logged viz panels
│   ├── wandb-metadata.json     ← ALWAYS present: args[], program, host, gpu, startedAt
│   ├── output.log              ← ALWAYS present: per-ckpt loss lines (see below)
│   ├── config.yaml             ← only on completed runs (nested _wandb.value.e.<hash>.args)
│   └── wandb-summary.json      ← only on completed runs: final metrics + image refs
└── run-<id>.wandb              ← binary protobuf history (SKIP in MVP)
```

**Model-grouping key (Cameron's one new feature):** the `--name` value inside
`wandb-metadata.json` → `args`. Example: `--name mv_scene_wrist_clsconcat_wristpp`.
Reliable on every run (metadata is always written). Fallbacks: `program` basename,
then run id. I'll group the run list by this `--name`.

**Loss curve source — output.log, no binary parsing needed.** Newer runs log:
```
  ckpt @ 200  total=14.1160 vol=10.3021 grip=0.9913 rot=2.8226  2.13 it/s
```
→ regex `ckpt @ (\d+) ... total=.. vol=.. grip=.. rot=..` gives step + per-term
series for free. Older runs use per-epoch lines; I'll parse both shapes. This
sidesteps the proprietary `.wandb` protobuf entirely for the MVP. (If Cameron
later wants the full per-step history wandb logs internally, the `.wandb`
datastore parser is the v2 path.)

**Config + summary:** read `config.yaml` / `wandb-summary.json` when present;
otherwise reconstruct config from `wandb-metadata.json` args, and skip summary.
Must not crash when these files are absent (in-progress runs lack them).

## Architecture — extend serve.py the `data_viewer` way

`serve.py` is already 2980 lines; I will NOT bloat it. The `data_viewer` route is
the template to copy: a static UI dir served via `send_from_directory` + small
`/api/...` JSON endpoints. Proposed split:

- **`reports/wandb_runs.py`** (new, ~150 lines, mine) — all scan/parse logic:
  `list_runs(roots)`, `run_detail(run_dir)`, `loss_series(output_log)`. Pure
  functions, fail-loud per GUIDELINES, but tolerant of a single unreadable run
  dir (skip it, don't crash the list — stale FUSE handling).
- **`reports/runs/index.html` + `runs.js`** (new, mine) — the UI: grouped run
  list, run detail with gallery + config/summary + Plotly/Chart loss curve +
  EMA smoothing slider, `setInterval` 10s poll of the JSON API.
- **~15-line route block in `serve.py`** (needs website_builder): 
  - `GET /runs/` and `/runs/<path>` → `send_from_directory(REPORTS_DIR/"runs")`
  - `GET /api/runs/list` → grouped runs JSON
  - `GET /api/runs/<run_id>` → detail JSON
  - `GET /api/runs/<run_id>/media/<path>` → serve a run's PNG
  - register a `WANDB_ROOTS` default.

This keeps serve.py's diff tiny and one-responsibility. Final URL:
**omidlab.net/runs/**.

## Coordination ask → website_builder

You own `serve.py`. Two ways to land the route block; your call:
1. I write `wandb_runs.py` + `runs/` static UI and hand you the ~15-line route
   block to paste + restart serve.py, **or**
2. You're fine with me adding the thin route block directly (I'll keep it minimal
   and ping you to restart in your tmux window).

Either way the heavy code lives in separate files I own. Replying in your inbox too.

## MVP checklist

1. [ ] website_builder coordination on the route block (this msg)
2. [ ] `wandb_runs.py` — scan + parse (name grouping, loss-from-log, graceful FUSE)
3. [ ] `runs/index.html` + `runs.js` — grouped list → detail (gallery/config/summary/loss+smoothing), 10s refresh
4. [ ] route block in serve.py + restart
5. [ ] verify on live yukon mount in browser
6. [ ] update vault infra/overview.md routes table + correct the stale yam_remote root in ROLE/overview

— our_wandb