# Fleet infrastructure

The agent-system infrastructure itself: dashboard, cron jobs, web services.

## Hosts (the dual-box architecture, as of 2026-06-21)

The fleet is migrating from a single-box (lab) setup to a **dual-host architecture**:

| Host | Role | What lives here |
|---|---|---|
| **omidhub** — Hetzner CCX33 VPS, `5.78.127.30`, US-West | Fleet host. Predictable uptime. | All agent state: `agents_stuff/`, `vault/`, `~/.claude/projects/`, `~/.cloudflared/`, `serve.py`. The tmux session lives here. Cloudflared tunnel terminates here. |
| **phe108** — lab box (USC, shared) | Data + GPU server. | All large files: `/data/cameron/para/libero/checkpoints/`, training datasets, eval videos, model code that runs on lab GPUs. Reachable from omidhub via sshfs + ssh-for-commands. |
| **puget** (TRI rig, 100.104.232.94) | Heavy training compute. | yams' YAM/PARA training. trackio still here for now. Reachable from omidhub directly. |
| **yukon** (TRI YAM rig) | Robot deploy + on-site data. | YAM robot state. Reachable from omidhub directly. |
| **mac** (Cameron's laptop) | Robot data collection + finance fetcher (planned). | Local data + planned: Plaid fetcher pushing to omidhub. |

**Key principle (Cameron, 2026-06-21):** agent-related things on omidhub; large files + serious computation on phe108. Don't `pip install torch` on omidhub. Don't write training code to omidhub local disk — write it under the sshfs-mounted lab path (which physically lives on phe108).

**Path equivalence:** omidhub mirrors phe108's path layout. `/data/cameron/agents_stuff/` is local on omidhub but the SAME absolute path. `/data/cameron/para/libero/` is an sshfs mount from phe108. This is deliberate — JSONLs reference absolute paths and agents resume mid-conversation after cutover without rewriting anything.

Migration scripts live at `/data/cameron/agents_stuff/migration/` (see README there). Cutover hasn't happened yet — VPS is being staged in parallel; lab box continues to run the live fleet.

## Canonical paths

| Component | Path | Notes |
|---|---|---|
| Bootstrap script | `/data/cameron/agents_stuff/bootstrap.sh` | Recreates the tmux session + launches every agent |
| Daily scrollback save | `/data/cameron/agents_stuff/shared/save_scrollback.sh` | Cron job, captures panes → commits to agents_stuff repo |
| Dashboard server | `/data/cameron/para/.agents/reports/serve.py` | Flask + cloudflared; serves omidlab.net |
| Cloudflared tunnel | dashboard tmux window | `para-dashboard` tunnel; auto-restarts in loop |
| Comms helpers | `/data/cameron/agents_stuff/shared/comms.py` | Python helpers for inter-agent messaging |
| Machines config | `/data/cameron/agents_stuff/shared/machines.yaml` | Mac, robot, TRI hosts |

## omidlab.net routes

- `/` — main agents dashboard
- `/para_website` — PARA project site (built by `website_builder`)
- `/chat` — mobile chat (`mobile_chat` agent)
- `/paper` — live LaTeX PDF
- `/agents` — per-agent terminal viewer (password protected)
- `/data_viewer` — dataset viewer (`data_visualizer`)
- `/runs/` — filesystem-driven wandb viewer (`our_wandb`); JSON API under `/api/runs/*`. Scans wandb run dirs on the yukon mount, groups by model name. Logic in `reports/wandb_runs.py`; UI in `reports/runs/`.
- `/browse/` — file browser
- `/upload` — file upload (≤100 MB through Cloudflare)
- `/api/transcribe` — webdictation (Whisper)
- `/api/tts` — text-to-speech

## `/browse/` allowlist (`serve.py:_BROWSE_ALLOWED_ROOTS`)

`/browse/<path>` resolves under `BROWSE_ROOT=/data/cameron` and accepts
symlinks ONLY when the resolved target lands inside one of these roots:

| Path | Purpose | Exposed at |
|---|---|---|
| `/data/cameron` | school server (this box) — primary working dir | `/browse/<anything>` |
| `/data/libero` | LIBERO benchmark data | `/browse/libero/` (via `/data/cameron/libero` symlink) |
| `/home/cameronsmith/mnt/robot-lab` | sshfs to russet's `/home/robot-lab/` (legacy, see [[../tri/overview.md]]) | `/browse/yam_remote/` |
| `/home/cameronsmith/mnt/yukon` | sshfs to yukon's `/home/robot-lab/` via tailscale — current rig host | `/browse/yukon_remote/` |

Adding a new root: append to `_BROWSE_ALLOWED_ROOTS` in serve.py (~line 1997)
and restart serve.py in the `agents:website` tmux window.

## Open infra work

### Migrate durable state off the lab box (added 2026-06-13 by Cameron)

The lab server has no uptime guarantee — could be shut down or wiped on short notice. The fleet has hardware ties (lab GPUs, TRI SSH chain, sshfs mounts to puget/yukon) so the *runtime* can't fully relocate, but the *durable state* should mirror to external git remotes + an external host that can rehydrate from those repos. Sub-tasks ordered by exposure:

1. **memory** (`~/.claude/projects/-data-cameron-agents-stuff/memory/`) is NOT a git repo. Zero backup. Highest exposure. Convert to a git repo + push to a private GitHub remote.
2. **vault** is a local git repo but has **no remote**. Add a private remote + push.
3. **Daily push automation** (cron) for memory + vault, mirroring the existing scrollback daily-commit cron. One `*-scrollback`-style cron job per repo, or one job that walks all three.
4. **External host for the public surfaces** — small VPS (Hetzner / DO / Render) running a copy of `serve.py` + cloudflared. So `omidlab.net` keeps resolving even if the lab box is gone. Yams/backbones/etc. would still be lab-tied (no fix without re-homing the training too), but the dashboard, mobile chat, voice interface, and our_wandb viewer can all run anywhere with the right repos checked out.
5. **Rehydration playbook** — `bootstrap.sh` already exists; test it end-to-end on a fresh VM with just the three repos (agents_stuff, vault, memory) checked out. Document any gaps in a `RECOVERY.md` follow-up section.
6. **Side leak to clean up**: `/data/cameron/para/.git/config` has a PAT embedded in the remote URL (visible in `git remote -v`). Same exposure pattern as the leaked PAT from 2026-06-04 — rotate to SSH like `agents_stuff` uses.

We discussed external hosting options on 2026-06-09 — context: the wandb / trackio path (Hetzner, Render, HF Space). Same shortlist applies.

## Known gotchas (move to memory.md when stable)

- Cloudflare free tier caps body size at ~100 MB; uploads above that 413 / `Failed to fetch`.
- serve.py must run from the `uva` conda env, not base miniconda (faster_whisper installed there).
- Daily scrollback cron commits to GitHub — don't paste secrets into agent windows.
- Stale FUSE mounts cause `Transport endpoint not connected` + can wedge agents in D-state.