# YAM code migration: phe → VPS + GitHub backup

**Written 2026-07-09. Nothing executed yet.** Refines the earlier `POST_PHE_RECOVERY_CHECKLIST.md` around Cameron's specific priority (yam para v4 + baselines + viz) and adds an automatic backup strategy.

## Goal

**Detach all code from phe.** Phe stays useful only as a GPU host for inference/training when we need it. Every file we'd want to keep long-term or edit interactively lives on VPS + backed to a private GitHub repo.

Two lines of change:

1. **Migrate code** (one-shot rsync from phe to VPS local disk)
2. **Automate backup** (nightly cron: git commit + push to private GitHub) so we never re-live 2026-07-05's "server room cooling failure = fleet degraded" scare.

## Scope

Priority-ordered per Cameron:

1. **yam para v4** — model code, config, training script, wrappers. This is the pillar.
2. **Baselines** — ACT, DiT, DA3, VLM variants, whatever else got compared.
3. **Visualizations** — viz/*, overlay, keypoints, rerun panels, wandb code.
4. **Data-loading + augmentation** — dataset PKL schema readers, augmentation, calibration-frame reader.
5. **Eval + rollout** — deploy scripts, eval_videos writer, annotation page, matrix_state.json.
6. **Dependencies** — anything hardcoded to phe paths.

Explicitly OUT of scope (stays on the compute host per 2026-07-06 policy):
- Checkpoints (puget/yukon local)
- Pretrained backbone weights (puget)
- Training PKLs, PNG frames (puget/yukon)
- Eval video MP4s (puget/yukon)
- Recorded episodes (yukon)

Yams's manifest (pinged 2026-07-09) will refine this list — the manifest is the source of truth. This doc is the framework.

## Phase A — Info-gathering (yams manifest received 2026-07-09)

- [x] Pinged yams for the manifest
- [x] **Yams's manifest is in — see `agents/yams/outbox.md` §1–7**. Comprehensive, treats itself as canonical documentation going forward.
- [ ] Ping yam_calib for its scope-slice (record.py, raiden.calibration.*, ArUco boards, calibration schema) — yams flagged these under "sibling ownership" but yam_calib should confirm its own list
- [ ] Ping yam_sim for its scope-slice (exo_redo/Demos, robot_models/i2rt_yam_fidex, ExoConfigs, MuJoCo sim assets) — same reason

### Highlights from yams's manifest

- **v4 variants**: base + `_da3, _vlm, _dynaflip, _paligemma, _pi0, _radio, _clip` — all share the volumetric head, swap backbone. All under `lib/model_volumetric_v4*.py`.
- **Baselines**: CLS→XYZ regression, motion_tracks (`model_baseline_motion_tracks.py`), diffusion DDPM (`model_baseline_diffusion.py`).
- **Viz — three layers**: offline (`lib/viz/*`, `lib/run_viz.py`), training-time (wandb via `train.py:_viz_batch`), deploy-time (rerun via `deploy_with_action_chunking.py`).
- **Deploy canonical**: `deploy_with_action_chunking.py` + `lib/inference.py` (model dispatch) + `lib/robot.py` (raiden wrapper) + `lib/render.py` + `lib/deploy_panels.py`.
- **Matrix + annotation**: `code/exp_grid_v4_vs_baselines.py`, `code/exp_grid_annotations.py`.

### Hardcoded phe paths to fix DURING migration (yams §6)

Naive rsync will silently break these — fix each before Saturday, or in-line during rsync:

1. `lib/model_volumetric_v4_da3.py` + `lib/model_da3.py` — eager `os.environ.get(k, default)` where default is `~/lab/da3_repo/src`. Fix: proper lazy fallback.
2. `code/exp_grid_v4_vs_baselines.py` — `EVAL_VIDEOS_DIR = /data/cameron/yukon_remote/cameron/eval_videos` (phe sshfs symlink). Fix: env-var override + fresh mount on VPS.
3. `code/exp_grid_annotations.py` — `OUT_DIR` hardcoded phe path. Fix: env-var override matching `HTML_DIR` pattern.
4. `~/yam_para/data/pickplace_*` symlink trees on puget — encode phe/puget layout. Fix: `rsync -L` to resolve, or rebuild from the 70-ep source.
5. Yukon-local sed patches (yams applied during phe outages) not yet merged back to canonical. Fix: merge before Saturday rsync.
6. Deploy tmux sessions use `~/lab/para/robot/yam/` via sshfs — Cameron already set up `/home/robot-lab/yam_local_train/` etc. as escape hatches on yukon.

Non-issues: `train.py` writes to `~/yam_para/checkpoints/` (env-relative), wandb config in `~/.netrc` (per-machine).

### Yams's Q1 answer: two repos

Yams recommends a two-repo split (§Suggested repo structure):
- **`yam-v4`** — yams's own scope: `train.py`, `deploy_with_action_chunking.py`, `lib/model_volumetric_v4*.py`, `lib/model_baseline_*.py`, `lib/inference.py`, `lib/viz/`, `lib/run_viz.py`, `code/exp_grid_*.py`, `code/launch_train_*.sh`.
- **`yam-shared`** — schemas + docs (`docs/fiducial_exoskeleton_board.md`, calibration/PKL schemas, ArUco ID table). Shared across yams + yam_calib + yam_sim.
- `raiden_fork/` — leave as a git submodule pointer (TRI/Shun's upstream).

Adopting this recommendation unless Cameron overrides. It's cleaner than one giant `para-code` repo, matches the sibling-scope split, and lets nightly backups run in parallel.

### Sibling-ownership flags (route during migration)

Yams identified files that belong to siblings — DON'T bundle these in yams's rsync:

**→ yam_calib**: `record.py`, `raiden.calibration.*` (`exo_calibrate*.py`, `wrist_calibrate*.py`, `wrist_verify_rerun_bimanual*.py`), ArUco board PNGs + STL mounts, calibration_results.json schema + reader, calibration/PnP sections of `docs/fiducial_exoskeleton_board.md`.

**→ yam_sim**: `exo_redo/Demos/*` (sim_yam, sim_yam_bimanual, exo_img_bimanual_yam, render_bimanual_views), `robot_models/i2rt_yam_fidex/*` (MJCF, meshes, XML variants, finray), `ExoConfigs/*` (yam_exo.py, panda_exo.py, ExoskeletonConfig), `exo_utils.py`, MuJoCo composition sections of the fiducial-exo doc, `sim_assets/` directory.

**→ yams_any4d**: nothing existing yet — this is a newer agent, its Any4D-specific code lives at `/data/cameron/repos/Any4D/`.

### Secrets to filter (never push to any repo — private included)

From yams §"Secrets-adjacent":
- Yukon sudo password (memory-only, off-vault ✓)
- AWS SSO temporary credentials (env-only, ~1-4h TTL)
- TRI internal auth: `ANZU_RESOURCES_APIKEY` from `anzu-resources-apikey.awsinternal.tri.global`
- wandb API key in `~/.netrc`
- Tailscale auth keys
- Cloudflare tokens for omidlab.net
- TRI internal repo source (anzu bazel-fetched blobs — URLs are fine, actual source is not)

Bake into `.gitignore` and add a grep-pre-push guard in `nightly_backup.sh` (regex: `sk-|BEGIN.*PRIVATE|api[_-]?key|password.*=.*['\"][A-Za-z0-9]{8,}`).

## Phase B — Migration (one-shot rsync)

**Approach: rsync from phe to VPS, preserving directory structure.** Standard git repos come along with their `.git`, so `git status` still works post-move.

```bash
# Illustrative — actual paths from yams's manifest
rsync -avz --exclude='__pycache__' --exclude='.venv' --exclude='*.pyc' \
      --exclude='eval_videos/' --exclude='checkpoints/' \
      cameronsmith@phe:/data/cameron/para/robot/yam/ \
      /data/cameron/para/robot/yam/

rsync -avz cameronsmith@phe:/data/cameron/repos/<repo>/ \
      /data/cameron/repos/<repo>/
```

Excludes:
- `__pycache__`, `.venv`, `*.pyc` — regenerable
- `eval_videos/`, `checkpoints/`, `data/*.pkl`, `*.mp4`, `*.png` — large, host-local per policy

Post-rsync:
- [ ] Verify checkouts are clean (`git status` per repo)
- [ ] Fix any hardcoded phe paths surfaced by the manifest (Phase A step 3)
- [ ] Delete redundant sshfs mount entries pointing at those paths (or repoint if it's still useful for cross-check reads)

## Phase C — Compute-host access (sshfs FROM VPS)

Per the 2026-07-06 policy inversion: compute hosts (phe, puget, yukon) sshfs-mount code FROM VPS at the same `/data/cameron/…` paths. Cameron confirmed this direction 2026-07-06.

```bash
# on phe / puget / yukon (illustrative)
sshfs cameronsmith@vps:/data/cameron/para/robot/yam \
      /data/cameron/para/robot/yam \
      -o reconnect,ServerAliveInterval=15,follow_symlinks
```

- [ ] Draft the compute-host mount script (`08_compute_hosts_mount_vps.sh` — one script that runs on each compute host)
- [ ] Verify: after mount, running `python train.py` on phe reads code from VPS transparently, writes checkpoints to phe-local disk (per policy)

## Phase D — Backup: private GitHub, nightly sync

**Design:**

- One private GitHub repo per top-level tracked dir on VPS. Suggested:
  - `cameronosmith/para-code` — the migrated yam para + baselines + viz
  - `cameronosmith/vault` — the vault (per drafted `07_vault_memory_github_remote.sh`)
  - `cameronosmith/manager-memory` — the manager memory files
  - `cameronosmith/agents-stuff` — already exists
- Cron on VPS, nightly at a quiet hour (say 04:15 local):
  ```bash
  15 4 * * *  /data/cameron/agents_stuff/scripts/nightly_backup.sh
  ```
- `nightly_backup.sh` walks a whitelist of repo paths, per repo:
  - `git add -A`
  - Skip if `git diff --cached --quiet` (nothing changed)
  - `git commit -m "nightly $(date -Iseconds)"`
  - `git push origin main`
  - Log success/failure to `logs/nightly_backup.log`
  - On push failure, notify manager pane (same pattern as `watch_phe.sh`)

**Frequency options** Cameron mentioned:
- **Every night** — safest. ~5 min job typically. My default recommendation.
- **Every couple nights** — cheaper GitHub traffic, but if VPS dies mid-day you might lose ≤3 days.

I'd lean nightly — the marginal cost is zero and the marginal safety is real. Cron override is trivial if Cameron prefers less.

**Secret filtering:**

- Add a `.gitignore` in each repo that excludes `.env`, `credentials.json`, `~/.aws`, `~/.config/rclone/rclone.conf`, `*.pem`, `*token*`, etc.
- Yams's manifest should flag any secrets-adjacent file for pre-migration audit.
- The nightly script also `grep -RIn -E 'sk-|BEGIN.*PRIVATE|api[_-]?key' <repo>` before pushing; abort + notify if it finds a suspicious blob.

**What lives outside git** (per 2026-07-06 policy):

- Big model outputs — on compute host, never in repo
- Session JSONLs (`~/.claude/projects/`) — they're archived by save_scrollback.sh already, not here

## Phase E — Retire phe as a code host

- [ ] Once compute-host sshfs is verified: on phe, rename the old code dirs (`mv .../old-yam yam-DEPRECATED`) so no one accidentally edits stale copies
- [ ] Add a `README.DEPRECATED` file pointing at the VPS-canonical path
- [ ] Optionally, symlink the old paths to the VPS sshfs mount for a couple weeks so any hardcoded ssh-into-phe scripts silently keep working during the tail
- [ ] Update GUIDELINES.md's "Where does code live?" section (currently still says the old policy in places) and retire `feedback_code_lives_on_lab.md` in favor of a new `feedback_code_lives_on_vps.md` (drafted post-migration)

## Phase F — Validation + close-out

- [ ] Simulated failure test: `sudo systemctl stop tailscaled` on phe from inside a maintenance window; verify VPS + compute hosts + agents still function
- [ ] Verify a nightly backup ran successfully after Phase D lands
- [ ] Update `project_migrate_offlab_pending` memory to closed
- [ ] Update `feedback_code_lives_on_lab` to superseded

## Open questions for Cameron

**Q1 auto-answered** by yams: two repos (`yam-v4` + `yam-shared`) instead of one big `para-code`. Adopting unless Cameron overrides.

Remaining:

2. **Vault + memory to GitHub** — my earlier `07_vault_memory_github_remote.sh` proposed `cameronosmith/vault` and `cameronosmith/manager-memory`. Confirm naming.
3. **Compute-host sshfs credentials** — after migration, phe/puget/yukon need to ssh-mount FROM VPS. Adding VPS's key to each host's `authorized_keys` is a one-time setup. Confirm you're OK with that path.
4. **Backup cadence** — nightly (my rec) or every-other-night?
5. **Any repos on GitHub already** we'd merge into vs. new repos? (Are `yam-v4` / `yam-shared` fresh, or is there an existing `cameronosmith/yam-para` or similar to fold into?)

## Deliverables when we execute

- Migration script: `08_migrate_yam_code_from_phe.sh` — driven by yams's manifest
- Compute-host mount script: `09_compute_hosts_mount_vps.sh` — run on each of phe/puget/yukon
- Nightly backup script: `scripts/nightly_backup.sh` + cron entry
- Updated GUIDELINES.md + retired memories