# Fleet migration to VPS (Hetzner CCX33, `5.78.127.30`, `omidhub`)

Scripts to migrate the agent fleet from the shared lab box (phe108) to a dedicated VPS.

## Architecture (post-migration)

```
omidhub (Hetzner CCX33, US-West)
├── /data/cameron/agents_stuff   ← local: fleet code + per-agent dirs
├── /data/cameron/vault          ← local: vault
├── /data/cameron/para/.agents/  ← local: serve.py + UI
├── /home/cameronsmith/.claude/  ← local: session JSONLs + memory
├── /home/cameronsmith/.cloudflared/ ← local: tunnel cert + config
└── sshfs mounts (to lab box):
    ├── /data/cameron/para/{libero,robot,panda_streaming,...}   ← lab disks
    ├── /data/cameron/repos                                     ← lab disks
    ├── /data/cameron/mac_robot_datasets                        ← lab disks
    └── /data/cameron/{yam_remote,yukon_remote}                 ← lab disks (chain mounts)

phe108 (lab box, USC)
└── stays as: GPU server + data store
    ├── /data/cameron/para/libero/checkpoints/  ← training output stays here
    ├── /data/cameron/para/robot/yam/data/      ← yam datasets stay here
    └── (reachable from VPS via sshfs + ssh-for-commands)
```

## Rationale

- **Agent state on the VPS** = predictable uptime, no risk of being wiped by lab admins, no shared multi-user box for credentials.
- **Heavy files + GPU jobs on the lab box** = no need to migrate TBs of data; agents that need GPUs (yams, backbones) just SSH from VPS to lab/puget/yukon like yams already does to puget.
- **Same absolute paths** (e.g. `/data/cameron/...`) on both boxes → zero JSONL path rewriting. Agents resume mid-conversation after cutover.

## Sequence (rough timing once SSH access is sorted)

| Step | Where | Script | Time |
|---|---|---|---|
| 0 | Cameron's Mac | SSH into VPS, paste lab box's ssh pubkey into root's authorized_keys (see "Access unblock") | 30 s |
| 1 | VPS (as root) | `01_vps_bootstrap.sh` | ~5 min (apt + npm + cloudflared) |
| 2 | Lab box (after step 1's `cameronsmith` user is created) | `02_rsync_from_lab.sh` | ~3 min (≈150 MB over wire) |
| 3 | Lab box (one-time) | Add VPS's pubkey to lab's `~/.ssh/authorized_keys` (script 01 prints it) | 30 s |
| 4 | VPS (as cameronsmith) | `03_vps_post_setup.sh` | ~3 min (sshfs mounts + systemd units) |
| 5 | VPS | Verify: `tmux ls`, `ls /data/cameron/agents_stuff`, `cat ~/.claude/projects/*/memory/MEMORY.md` | manual |

Steps 1–4 leave the VPS **fully provisioned but the fleet NOT running**. The lab box keeps running unchanged. Cutover is a separate later step.

## Cutover (when Cameron is ready to flip the switch)

```
# 1. On lab box: stop the old cloudflared so the tunnel cleanly fails over
tmux send-keys -t agents:cf-omidlab C-c

# 2. On VPS: start the new cloudflared
sudo systemctl start cloudflared-omidlab
# (verify: curl -s https://omidlab.net/ should return 200)

# 3. On VPS: start the fleet (resumes every agent mid-conversation from rsync'd JSONLs)
cd /data/cameron/agents_stuff && ./resume_fleet.sh

# 4. (optional) On lab box: stop the old fleet so messages don't go to two managers
#    Wait until VPS is verified before this step.
```

## Access unblock (current blocker as of 2026-06-21)

The VPS is provisioned with Cameron's Mac SSH key (set in Hetzner console at provision time), but the lab box's key isn't there yet, and root password login appears disabled despite the auth-method banner. Two ways to unblock:

**Option A (fastest, Cameron's Mac):**
```
ssh root@5.78.127.30 'mkdir -p ~/.ssh && chmod 700 ~/.ssh && echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILuBCzIz1TAQfeOQMSZ60glDJVZXd9ZCgcmY4BPutjNn cameronsmith@phe108-yuewang-01" >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys && echo done'
```

**Option B (if Cameron prefers password auth on for the bootstrap):**
SSH from Mac, edit `/etc/ssh/sshd_config` to set `PermitRootLogin yes` + `PasswordAuthentication yes`, then `systemctl restart ssh`. Re-share the correct password (mine guesses didn't match; possible typo when sent through chat).

## Files

- `01_vps_bootstrap.sh` — run on VPS as root. Installs packages, creates user, scaffolds paths.
- `02_rsync_from_lab.sh` — run on lab box as cameronsmith. Pushes fleet state to VPS.
- `03_vps_post_setup.sh` — run on VPS as cameronsmith. Mounts lab disks via sshfs, registers cloudflared.
- `README.md` — this file.

## Note on data files

Per Cameron's instruction: **large files and serious computation stay on the lab box.** This split is enforced by:
- The sshfs mounts (read/write to lab disks, but the data lives there)
- The agents' default to SSH out to puget/yukon for GPU work
- The bootstrap NOT installing CUDA drivers or training frameworks on the VPS

Don't add `pip install torch` or similar to the VPS unless we change this principle.