---
name: fuse-wedge-recovery-2026-06-28
description: "Recurring failure: agents wedge in D-state when find/ls/recursive ops walk into a broken sshfs chain-mount. Diagnose via wchan=request_wait_answer, fix by killing the sshfs daemon (NOT touching the mount path)."
metadata: 
  node_type: memory
  type: feedback
  originSessionId: cd07777c-8add-4afc-9cb9-9113caa0331f
---

**Symptom:** An agent's pane shows a stuck spinner with no timer progression. The agent claude process or one of its tool subprocesses is in `D` state (uninterruptible sleep) with `wchan=request_wait_answer` — that's FUSE waiting for an sshfs response that's never coming. Cameron says "agent X got stuck investigating a mount, like it does."

**Root cause (every time so far):** A chain-sshfs mount on the VPS has lost its remote endpoint. Examples:
- `/data/cameron/russet_remote` = omid-fleet → lab → robot-lab. Breaks when robot-lab is unreachable (motors error, ssh dies, host powers off).
- `/data/cameron/yam_remote` = chain through yukon. Breaks when yukon is down or its inner mount is broken.

The agent then runs *any* operation that walks the broken mount — even a wide `find /data/cameron -name <pattern>` or a `ls -R` — and the kernel FUSE call hangs forever waiting for the dead sshfs daemon. Sometimes only a subprocess wedges; sometimes the agent's MAIN thread blocks on FUSE too, which freezes the whole pane.

**Why:** Cameron's sshfs chain-mounts are fragile single points of failure for the agent that touches them. Once wedged, neither `kill -9` nor `fusermount -uz` (even with sudo) cleanly recovers — both return Permission denied / no-op on the wedged path.

**How to apply (diagnostic + fix recipe):**

1. **NEVER `ls`/`cd`/`stat`/`df` the broken mount yourself** — your shell will wedge too.
2. **Find the wedge** safely via:
   - `tmux capture-pane -t agents:<name> -p` — view the agent's state (safe, no FS access)
   - `ps -eo pid,stat,wchan:25,etime,cmd` + filter for `D` state, `wchan=request_wait_answer` (FUSE), or `wchan=poll_schedule_timeout` (network hang)
   - Walk the agent's process tree from its pane PID; D-state procs there are the wedged ones
3. **Find which mount is broken:**
   - Iterate sshfs mounts with `timeout 2 stat <path>`. The one that times out is broken.
   - WARNING: even `mountpoint -q` on a broken sshfs can hang. Use the timeout wrapper.
4. **Fix:** kill the sshfs daemon process directly: `kill -9 $(pgrep -f 'sshfs.*<mount_name>')`. This forces the kernel to return EIO to all pending FUSE ops on that mount, unblocking every D-state caller.
5. **Cleanup:** `sudo umount -l <mount>` to clear the stale `/proc/mounts` entry. After daemon death this succeeds cleanly.
6. **Remount later** when the remote endpoint is back up (the broken mount's original sshfs command is recoverable from the dead daemon's argv — keep a copy before killing).

**Prevention guidance for agents (added to GUIDELINES.md eventually):**
- Don't blanket-recurse across `/data/cameron` with `find`, `bfs`, `ls -R`, `grep -r`, etc. The chain-mounts wedge your whole agent.
- Scope file searches to a specific subdir you know is healthy, OR run the search via ssh on the host where the data actually lives (`ssh phe108 'find ~/cameron/raiden_fork -name ...'`).
- When in doubt about a mount path's health, probe first: `timeout 2 stat <mount_root>` returns within 2s if healthy, hangs+exits with code 124 if broken.

**Side-channel fact:** when the russet_remote chain breaks because robot-lab is down, the **russet_yam tmux pane's `ssh -t cameronsmith@lab ssh -t robot-lab` also hangs**. Same root cause, different process. Kill that ssh process too (find via `ps -ef | grep 'ssh.*robot-lab'`).