# Outbox — sufi_lectures

## 2026-07-06 — bootstrap pipeline sketch (v0.1)

Drafted after reading ROLE.md + inbox.md. **Not yet running anything.** Waiting on Cameron's answers to the open questions (posted in his pane) before committing.

### Inherited state I can build on (from life_manager fork context)

On Puget (`puget-232243-01` via Tailscale):
- `~/khanega/` is a going concern from 2026-07-05:
  - `~/bin/rclone`, `~/bin/ffmpeg`, `~/bin/ffprobe` — user-local static binaries, no sudo
  - `~/.config/rclone/rclone.conf` — Google Drive OAuth configured for the Khanega folder (`160vGgXJbWSp68TfTjD7craJVPNeGWimS` — ~24 `Jalaseh` mp4s, March→June 2026)
  - `~/khanega/.venv` — Python 3.12 with `faster-whisper large-v3`, `nvidia-cublas-cu12`, `nvidia-cudnn-cu12`, `flask`, `anthropic`, `requests`
  - `~/khanega/data/jalaseh_2026-06-21/segments.json` — one lecture fully transcribed (751 Persian segments, 23-min audio, ran ~12x realtime on GPU)
- **Reusable for me**: rclone + ffmpeg + faster-whisper stack, plus the Persian-language Whisper prompt tuning
- **Not reusable for me** (out of my scope): the translate/synthesize/viewer layers built for the sibling `khanega` agent (see open question about khanega overlap below)

### Pipeline sketch (source → archive → persian handoff)

**Stage 1 — Discovery + download**
- Google Drive folder (already known) — pull via `rclone copy` on Puget
- YouTube channels (if Cameron adds) — `yt-dlp` with audio-only, `--extract-audio --audio-format mp3`
- Naming convention: `<source>_<date>_<slug>.<ext>` (e.g. `khanega_2026-06-21_jalaseh.mp4`)
- Store originals on Puget: `~/sufi_lectures/audio/`

**Stage 2 — Transcription**
- Host: **Puget GPU** (until `phe` returns). faster-whisper `large-v3`, `language="fa"`, `vad_filter=True`, `beam_size=5`
- If a lecture is bilingual (Persian teacher + English asides in Q&A), run with `language=None` and let Whisper auto-detect per-segment
- Output: segments with `[start, end, text]` (Persian script only at this stage)
- **Do NOT do** translation or transliteration or English synthesis — those were khanega-agent scope; my job stops at Persian source transcript

**Stage 3 — Post-processing on VPS**
- **Merge very short segments** (< 3s) into paragraph units for readability. Preserve original fine-grained segments in a separate `raw_segments.json` for anyone who needs word-level alignment
- **Cleanup** obvious ASR errors via a Claude pass (typos, hallucinated repetitions Whisper is known for). Log every edit so revert is possible
- **Generate metadata via Claude** — one short pass per lecture:
  - `title` (if not obvious from filename)
  - `speaker` (default Nirvana until Cameron says otherwise per lecture)
  - `topics` (5-10 tags — e.g. `dhikr`, `nafs`, `sohbat`, `fana`, `Rumi-quote`, `personal-story`)
  - `summary_short` (2-3 sentences)
  - `summary_long` (~200 words)
  - `notable_phrases` — 10-30 phrases persian's SRS should consider

**Stage 4 — Archive layout (VPS, per Cameron's storage policy)**

```
/data/cameron/life/projects/khanega/transcripts/
├── index.json                          # master list — all lectures + status + metadata
├── <lecture_id>/                        # e.g. khanega_2026-06-21
│   ├── meta.json                        # title, source_url, speaker, date, duration, topics[], summaries
│   ├── transcript.json                  # paragraph-level, cleaned
│   ├── raw_segments.json                # Whisper's original fine segments (audit trail)
│   ├── transcript.md                    # human-readable Farsi paragraphs
│   └── phrases_for_srs.json             # notable_phrases[] persian can ingest
```

- Audio/video stays on Puget at `~/sufi_lectures/audio/<lecture_id>.mp4`. VPS only holds text + metadata + a `source_host` pointer so the archive is portable if Puget dies.

**Stage 5 — Search + retrieval**
- v1: grep-based (all transcripts are markdown, ripgrep against `transcripts/**/transcript.md`)
- v2 (if Cameron wants it): tiny Flask endpoint or a Claude Code slash-command; still lives here
- Explicitly **not** doing embedding search v1 — grep gets us far with 24 lectures

**Stage 6 — Handoff to persian**
- After each new lecture lands, append a one-liner to `agents/persian/inbox.md`:
  > `2026-07-06 — new transcript batch: khanega_2026-06-21 · path /data/cameron/life/projects/khanega/transcripts/khanega_2026-06-21/ · ~15 phrases_for_srs.json ready to ingest`
- Persian pulls the phrases file; I don't touch anything under `persian/`

**Stage 7 — life_manager surfaces**
- If a lecture's topics include something Cameron's actively working on (e.g. a passage about `nafs` when he's mid-Brandon "self-scanning" work), drop a one-liner in `agents/life_manager/inbox.md`. Otherwise silent.

### Open questions posted to Cameron

(Also in his pane; keeping a copy here.)

1. **khanega agent overlap** — a sibling `khanega/` agent exists on disk at `agents/khanega/` from 2026-07-05 with a broader scope (translate + synthesize + bilingual viewer at `puget:8091`). Am I superseding / rolling that in, or does khanega still exist for translation/viewer while I handle just source→transcript→persian-handoff?
2. **Sources** — Google Drive folder (~24 lectures) is the known backlog. Any YouTube channels or other feeds to include from day 1?
3. **Priority** — backfill the 24 Drive lectures now (Puget is warm), or wait for new lectures forward?
4. **Retention** — keep source `.mp4`s on Puget forever, or delete after transcript + metadata land on VPS?
5. **Speaker metadata** — is every `Jalaseh` recording Nirvana (Cameron's grandfather), or are there other speakers I should distinguish? Any manifest with speaker per lecture?
6. **Cleanup autonomy** — for the Claude-pass ASR-cleanup step, do you want me to make edits inline or produce a diff you review? First pass I'd suggest inline with a full audit trail in `raw_segments.json` so nothing is lost.
7. **Persistent shell** — I don't have one; will spawn one for the actual transcription runs when green-lit. Not needed for bootstrap.

### Next action
Waiting for Cameron on the above. Status set to `idle` after this write.