10 KiB

Raw Blame History

LCM + Session Manager — architecture

This is the deterministic core of the backend daemon. It supervises agent
sessions and keeps exactly one true status per session.

1. Mental model: OBSERVE → DECIDE → ACT

The backend owns no agent state. git/GitHub own PR/CI/review truth; the agent's
own files own its activity. The job, per session, is one loop:

OBSERVE            →   DECIDE              →   ACT
(impure, external)     (pure, total)           (impure)
raw facts              one canonical status    persist + react

In the rewrite the OBSERVE step lives outside the LCM (separate owners),
and the LCM is a synchronous reducer invoked with facts:

SCM poller     ─ ApplySCMObservation ──┐
reaper         ─ ApplyRuntimeObservation┤
activity hooks ─ ApplyActivitySignal ───┼─▶ LCM:  load canonical
Session Mgr    ─ OnSpawnCompleted ──────┘         → pure DECIDE
               ─ OnKillRequested                  → diff → persist (merge-patch)
reaper tick    ─ TickEscalations                  → if transition: react (ACT)

The LCM never polls. The reaper (a timer, owned elsewhere) drives liveness
sampling and duration-based escalation by calling in.

2. Canonical state model — the crown jewel

The only thing persisted per session is CanonicalSessionLifecycle
(backend/internal/domain/lifecycle.go). The single-word display status is
derived on read and never stored — this is the most important invariant; it
prevents canonical truth and display from drifting.

CanonicalSessionLifecycle
  Version    schema version of the record shape
  Revision   monotonic write counter (optimistic-concurrency token)
  Session    (state, reason)   working/idle/needs_input/stuck/detecting/done/terminated
  PR         (state, reason)   none/open/merged/closed
  Runtime    (state, reason)   unknown/alive/exited/missing/probe_failed
  Activity   last-known agent activity (+ timestamp, source)   ← decider input
  Detecting  anti-flap quarantine memory (nil unless quarantined) ← decider input

DeriveLegacyStatus (domain/status.go) is the sole producer of the
display SessionStatus. Precedence: terminal/hard session states map directly
(they outrank PR facts) → a merged PR wins → an open PR maps by reason → else the
soft session state. So an idle worker with a CI-failing open PR displays
ci_failed, but a needs_input session shows needs_input regardless of the PR.

Session (domain/session.go) is the read-model: a SessionRecord
(persistence shape, identity + lifecycle + metadata) plus the derived Status.
The Session Manager is the single producer of Status — it attaches it on
read; the store and API never recompute or persist it.

3. Package layout (`backend/internal/`)

domain/                 the vocabulary (imports only the std lib → no cycles)
  lifecycle.go          CanonicalSessionLifecycle + all sub-states/enums
  status.go             SessionStatus + DeriveLegacyStatus (sole display producer)
  session.go            SessionRecord (persisted) + Session (read-model) + id types
  decide/               the PURE core — total, deterministic, zero I/O
    types.go            LifecycleDecision + Probe/OpenPR/Detecting inputs + tuning consts
    decide.go           the deciders + the anti-flap quarantine + HashEvidence
ports/                  the boundaries (interfaces + DTOs)
  inbound.go            LifecycleManager, SessionManager (we implement)
  outbound.go           LifecycleStore, Notifier, AgentMessenger, Runtime/Agent/Workspace
  facts.go              SCMFacts, RuntimeFacts, ActivitySignal, SpawnOutcome, KillReason
lifecycle/              the LCM implementation (DECIDE + ACT)
  manager.go            the Apply* pipeline, per-session lock, patch diffing
  decide_bridge.go      fact→decide-input translation + the composition rules
  reactions.go          the reaction table + escalation engine + TickEscalations
session/                the SM implementation (explicit mutations)
  manager.go            Spawn/Kill/Restore/Cleanup/List/Get/Send + rollback

domain + ports are the committed, stabilized integration boundary.
Everything else implements behind it.

4. The pure DECIDE core (`domain/decide`)

Total, deterministic, side-effect-free functions — the highest-value test
surface (table-tested to 100%). Key ones:

ResolveProbeDecision — runtime/process liveness. An explicit kill
short-circuits to terminal; a failed probe is never read as death (routes
to detecting), as does any probe disagreement; only runtime-dead +
process-dead + no-recent-activity reaches killed.
ResolveOpenPRDecision — the PR ladder: ci_failing → changes_requested →
mergeable → approved → review_pending → idle-beyond → else pr_open.
ResolveTerminalPRStateDecision — merged → merged (park idle awaiting a
human decision); closed → idle.
CreateDetectingDecision — the anti-flap quarantine. Counts attempts and
hashes the timestamp-stripped evidence; escalates to stuck only after 3
consecutive unchanged-evidence ticks or 5 minutes since first entering
detecting (StartedAt is preserved across the whole episode). Changing
evidence resets the counter.

5. The LCM (`lifecycle`)

Implements ports.LifecycleManager. Every Apply*/On* entrypoint runs the
same pipeline (manager.go):

withLock(session):                       ← per-session serialization
  load canonical → decideFn (build sparse patch) → if changed: persist → load after
return transition (before, after)

then, after the lock releases, react() fires the mapped reaction.

Per-session serialization — keyedMutex hands out one lock per session id
(parallel across sessions, serial within one). Entries are reference-counted
and evicted when the last holder releases, so the map stays bounded.
Composition rules (decide_bridge.go) — two observers must not fight over
the session axis. Liveness (runtime probes) owns the runtime + death/detecting
axis; activity owns working/idle/waiting. isLivenessOwned decides when a
healthy probe may recover a state (e.g. detecting → working) vs. when it
must not clobber an activity-owned needs_input/blocked. A high-confidence
activity signal may resolve a detecting session; an open PR writes only the
PR axis and lets DeriveLegacyStatus surface it.
Detecting-memory lifecycle — a decision with Detecting == nil clears the
persisted quarantine memory (LifecyclePatch.ClearDetecting) so a stale prior
can't leak into a later episode.
ACT — reactions + escalation (reactions.go) — on a genuine status
transition, react() maps it to a reaction (send-to-agent / notify;
auto-merge exists but is off by default) and dispatches it. A
per-(session,reaction) escalation tracker counts attempts; it escalates
(notifies a human and silences further auto-dispatch) when a numeric cap or a
duration is exceeded. The ci-failed budget is persistent across CI
oscillation within an open PR and re-arms on genuine recovery. TickEscalations
(called by the reaper) fires the duration-based escalations the synchronous
LCM can't wake itself for; it notifies outside the lock.

6. The Session Manager (`session`)

Implements ports.SessionManager — the explicit-mutation plumbing. It never
derives/observes lifecycle state; it routes outcomes to the LCM.

Spawn — Workspace.Create → build prompt → Runtime.Create (env
AO_SESSION_ID/AO_PROJECT_ID/AO_ISSUE_ID) → seed the initial record
(not_started/spawn_requested) via the store → LCM.OnSpawnCompleted.
Eager rollback unwinds prior steps on failure; an OnSpawnCompleted failure
routes the seeded orphan to terminal-errored (the store has no delete; a later
Cleanup reclaims it).
Kill — LCM.OnKillRequested → Runtime.Destroy → Workspace.Destroy,
honoring the worktree-remove safety: after git worktree prune, a still-
registered path is never rm -rf'd (it may hold the agent's uncommitted work)
— the refusal is surfaced, not forced.
Restore — reopen via PatchLifecycle (not re-seed): session →
not_started, PR → cleared_on_restore; relaunch with the agent's resume
command; runtime is rolled back on a post-create failure.
List/Get — read records and attach the derived Status. Send — via
AgentMessenger. Cleanup — tear down terminal/stale sessions, skipping
paths with uncommitted work.

7. Load-bearing invariants

Persist canonical; derive display. Never store the display status.
One authority for death. Only the DECIDE pipeline (via detecting) writes
inferred terminal states; the SM's explicit-kill path goes through
OnKillRequested. Everything else that notices a dead runtime persists
detecting, never terminated.
Failed probe ≠ dead. Timed-out/errored probes route to detecting.
Evidence-hash debounce prevents flapping signals from terminating live
work; the 5-minute cap is a whole-episode wall-clock safety net.
PR facts dominate the soft session states once a PR exists.
Merge-patch persistence — writes touch only changed keys; the store is the
single disk writer (atomic write + lock + CDC).
Sticky activity states (waiting_input/blocked) do not decay by clock.
Worktree-remove safety on teardown.

8. Concurrency & testing

Within a session, the per-session lock serializes the load→decide→persist
read-modify-write. react() runs outside the lock (so a busy-waiting
send-to-agent never holds the session mutex) — see status.md for the
integration-time follow-up this implies.
Tests use in-memory fakes for every outbound port, so the LCM and SM are
fully testable with no real adapters. The SM tests drive the real
lifecycle.Manager for spawn/kill round-trips, so the SM↔LCM contract is
genuinely exercised. The decide package is table-tested in isolation.

10 KiB Raw Blame History