agent-orchestrator/packages/web/server/__tests__
Harshit Singh Bhandari 5c1d56aea4
fix(web): bound PTY re-attach loop with grace-period counter reset (#1640)
* fix(web): bound PTY re-attach loop with grace-period counter reset (#1639)

When `ao stop` (or any external action) kills a tmux session out from
under a still-subscribed dashboard, the mux server's PTY exit handler
attempts to re-attach. The MAX_REATTACH_ATTEMPTS=3 cap was supposed to
prevent unbounded respawning, but it was never engaging because the
counter was reset to 0 immediately after each "successful" `open()` —
where success only meant the new PTY was *spawned*, not that it
*survived*. When the underlying tmux session is gone, attach-session
exits ~40 ms after spawn, the exit handler fires again with counter=0,
and the loop runs at ~80 spawns/sec.

Diagnostic data captured on the issue: a single 1.5-second burst
produced 119 spawn↔exit cycles, raising the process's PTY fd count
from ~15 to ~153. Sustained for a few seconds, this exhausts the
macOS system PTY pool (kern.tty.ptmx_max=511), after which nothing
on the system can spawn a new PTY (tmux, VS Code terminal, ao spawn,
etc.) until the leaking process is killed.

Fix:
- Remove the `terminal.reattachAttempts = 0` reset inside the exit
  handler.
- Schedule a delayed reset via setTimeout in `open()`, gated on the
  closure-captured `pty` reference still being terminal.pty after
  REATTACH_RESET_GRACE_MS (5 s).

Effect: tight crash loops cannot reset the counter (PTY exits before
grace expires) and hit MAX_REATTACH_ATTEMPTS within ~150 ms, after
which the server emits "exited" and stops respawning. A long-lived
PTY that crashes hours later still gets a fresh retry budget.

Adds an integration test that reproduces the runaway scenario by
killing the tmux session externally and asserting "exited" arrives
within 2 s. Without this fix the test hangs and times out — the
exact symptom of the bug.

Note: this addresses the dominant runaway behaviour. A separate
~1 fd/cycle leak in node-pty 1.1.0 itself (each spawn opens 3
PTY-class fds in the parent, each exit releases only 2) remains and
will be tracked separately — likely a node-pty upgrade.

Fixes: #1639

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fixup(web): track grace timer + add recovery test (#1639 PR review)

Address two PR review notes on #1640:

1. Greptile P2 — store the grace-period timer handle on ManagedTerminal
   and clearTimeout it in the unsubscribe cleanup path. The closure
   guard already prevented any incorrect counter reset, so this is
   tidiness rather than a bug fix: it eliminates the up-to-5 s window
   where the timer's closure kept the killed PTY and evicted terminal
   object reachable. Also clears any prior timer when scheduling a new
   one in open() so back-to-back re-attaches don't pile up dead closures.

2. Copilot — add an integration test for the recovery path. The
   existing runaway test exercises the case where the counter must NOT
   reset (PTY crashes inside grace); the new test exercises the case
   where the counter MUST reset (PTY survives grace, then crashes
   later). Without the grace timer firing correctly, a single transient
   blip during startup would permanently consume the retry budget.

Test takes ~5.5 s because it uses the production grace period; per-test
timeout raised to 15 s. Vitest runs tests in parallel so this doesn't
serialize the suite.

Refs: #1639, #1640

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 18:37:44 +05:30
..
direct-terminal-ws.integration.test.ts fix(web): bound PTY re-attach loop with grace-period counter reset (#1640) 2026-05-07 18:37:44 +05:30
mux-websocket.test.ts refactor(web): replace SSE with WebSocket polling for session updates (#1259) 2026-04-29 13:05:04 -07:00
server-compatibility.test.ts fix: address bugbot comments — missed renames in preflight, tests, and shell scripts 2026-04-09 16:00:31 +00:00
tmux-utils.test.ts fix(web): scope terminal tmux resolution by project (#1551) 2026-05-01 16:15:43 +05:30