## Approach
The orchestrator polling loop previously made individual API calls for each PR's
state, CI status, and review decision - 3 separate calls per PR per poll.
With multiple PRs being monitored, this quickly exhausted GitHub's 5,000-point
hourly rate limit.
This PR implements GraphQL batching using aliases, which allows fetching data
for up to 25 PRs in a single GraphQL query. Additionally, a 2-Guard ETag
strategy is used to skip queries entirely when nothing has changed.
## Implementation
### GraphQL Batching
- `generateBatchQuery()` creates a single GraphQL query with unique aliases (pr0, pr1, pr2...)
- Each PR gets the same set of fields: state, CI status, review decision, mergeability
- Uses inline fragments for union types (CheckRun/StatusContext)
- Variable types: String! for owner/repo, Int! for PR numbers
### 2-Guard ETag Strategy
Before running expensive GraphQL queries, two lightweight REST ETag checks detect if
anything changed:
**Guard 1 (PR List ETag):**
- Checks `/repos/{owner}/{repo}/pulls` with If-None-Match header
- Returns 304 if no changes → skips GraphQL (0 points)
- Detects: New commits, title/body edits, labels, reviews, state changes
**Guard 2 (Commit Status ETag):**
- Checks `/repos/{owner}/{repo}/commits/{sha}/status` per cached PR
- Returns 304 if no changes → skips GraphQL (0 points)
- Detects: CI status transitions (failing → passing, passing → failing, etc.)
### Caching
- LRU caches for PR metadata (max 200 entries), ETags (100/500 entries)
- Cache misses trigger individual API fallback via lifecycle-manager
- No placeholder caching on errors - allows proper fallback behavior
## Impact
- **API reduction:** ~88% fewer REST calls (216 vs 1,800 calls/hour for 5 PRs)
- **GraphQL efficiency:** Batch query fetches 25 PRs for ~40 points vs ~400 for individual calls
- **Polling interval:** Still 30s, but most polls return cached data (0 cost)
- **Fallback:** Individual SCM calls still work for edge cases (permissions, cache misses)
## Testing
- Unit tests for query generation and parsing helpers
- Integration tests for real GraphQL API calls (skipped by default)
- Covers batch failures, partial success, empty arrays, edge cases
Create test-utils.ts with reusable factories (createTestEnvironment,
createMockPlugins, createMockRegistry, createMockSCM, createMockNotifier,
createMockSessionManager, makeSession, makePR) and refactor
lifecycle-manager.test.ts to use them, reducing setup boilerplate by >70%.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace CPU-burning spin loops in running-state.ts with async setTimeout
and jittered backoff; make all exports async
- Fix TOCTOU race in acquireLock by extracting tryAcquire helper with
retry loop instead of force-remove-and-retry-once
- Hoist dynamic imports in addProjectToConfig/choice-2 to static imports
- Add try/finally around readline in detectAgentRuntime
- Validate session prefix uniqueness in "Start new orchestrator" menu
- Add ConfigNotFoundError class to @composio/ao-core, replace fragile
string matching in ao start with instanceof check
- Fix setup.sh to recommend `ao start` instead of deprecated `ao init`
- Fix start-all.ts: resolve next binary with fallback, wait for children
on SIGTERM instead of immediate process.exit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- spawn now takes only `[issue]` — project is always auto-detected
- batch-spawn now takes only `<issues...>` — no project prefix
- Commander rejects extra positional args automatically
- Update orchestrator prompt to remove projectId from spawn example
- Rewrite spawn tests to use auto-detected project (single project in config)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Change != null to !== undefined && !== null in caller-context.ts and
session-manager.ts (3 locations) to satisfy eqeqeq lint rule
- Add displayName field to all 4 agent plugin test manifest assertions
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace require() with execFileSync in all 4 agent plugin detect()
functions — fixes ReferenceError in ESM-only environments
- Remove non-existent -p flag from orchestrator prompt spawn/batch-spawn
docs — prevents orchestrator agent from generating broken commands
- Return actual port from runStartup() and pass to register() — fixes
running.json storing wrong port when auto-escalation picks a free port
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reduces onboarding to `npm install -g @composio/ao && ao start`.
- Absorb init and add-project logic into `ao start` with auto-config
creation, environment detection, and project type detection
- Add single-instance tracking via running.json with already-running
interactive menu (human) and info+exit (agent)
- Add caller context detection (human/orchestrator/agent) via TTY and
AO_CALLER_TYPE env var
- Add plugin-based agent runtime detection — no hardcoded binary paths,
each plugin exports detect() and displayName
- Simplify `ao spawn` to just `ao spawn <issue>` with auto-detected
project (backward compat: `ao spawn <project> <issue>` still works)
- Set AO_CALLER_TYPE, AO_PROJECT_ID, AO_CONFIG_PATH, AO_PORT env vars
on all spawned sessions (worker, orchestrator, restore)
- Add `ao config-help` subcommand for config schema reference
- Deprecate `ao init` to thin wrapper, fully remove `ao add-project`
- Register/unregister in running.json on start/stop
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Publish @composio/ao-web dashboard as npm package (removed private flag,
added files field, production entry point, node-pty made optional)
- CLI auto-detects dev vs production mode for dashboard startup
- Use local next binary instead of npx in production start-all.ts
- findWebDir() throws with install-specific guidance instead of returning
broken path
- Fix CI-silent failure: setup.sh and ao-update.sh exit 1 on non-interactive
npm link failure
- Deduplicate detectDefaultBranch into shared cli/lib/git-utils.ts
- Add EACCES permission guidance to README.md and SETUP.md
- Move resolveProjectIdForSessionId to @composio/ao-core
- Update design doc with problems #8-13, changes #9-12, known limitations
section, and expanded test plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(lifecycle): implement stuck detection using agent-stuck threshold
The agent-stuck reaction config supported a threshold field (e.g.
"10m"), but determineStatus() never returned "stuck" — there was no
code path that consumed the threshold or transitioned sessions based
on idle time. Sessions would stay parked at pr_open/working forever
even when the agent had been idle for hours.
Added idle-time check in determineStatus(): when getActivityState()
reports "idle" or "blocked" with a timestamp, compare the idle
duration against the agent-stuck.threshold config. If exceeded,
return "stuck" so the reaction system can fire notifications.
Also removed the priority !== "info" guard on transition
notifications, so all priority levels (including info) are routed
through notificationRouting. This lets the config control which
notifiers receive each priority level, rather than silently dropping
info-level transition events.
* fix(lifecycle): add post-PR stuck detection as safety net
The original stuck check in step 2 (before PR checks) can be bypassed
when getActivityState() returns null (session file not found, cache miss,
I/O failure). When this happens, the code falls through to the PR path
which returns 'pr_open' without ever checking idle duration.
Fix: extract isIdleBeyondThreshold() helper and call it in three places:
1. Step 2: before PR checks (fast path, catches most cases)
2. Step 4b: after PR checks return 'pr_open' (safety net)
3. Step 5: after all checks, for agents that finish without a PR
This ensures stuck detection fires even when the JSONL activity detection
fails to return idle state. Sessions can no longer get permanently stuck
at 'pr_open' when the agent has been idle beyond the threshold.
Also removes the debug console.error calls from the previous commit.
* fix(lifecycle): treat reviewDecision 'none' as approved for merge readiness
PRs with no required reviewers never reached 'mergeable' status because
getReviewDecision returned 'none', which was not handled. The lifecycle
poll fell through to 'review_pending' or the default, so merge.ready
never fired and the approved-and-green reaction never triggered.
Also: skip stuck short-circuit when session has an open PR so merge
readiness checks in step 4 can still run. Without this, idle agents
with open PRs get stuck status and never transition to mergeable.
Closes composio#0 (internal fix)
* fix(opencode): classify activity state from session timestamps
* test(lifecycle): cover opencode idle-threshold stuck transition
* fix(lifecycle): preserve global stuck threshold with project overrides
* fix(lifecycle): run PR auto-detection before stuck transitions
---------
Co-authored-by: Harsh <harshb012@gmail.com>
Treat persisted tmuxName as metadata-derived runtime identity when runtimeHandle is missing so active sessions are not misclassified as exited/unknown across agents.
* fix: gate fresh session sends on interactive readiness
Wait for spawning tmux-backed sessions to settle before injecting input, and use OpenCode session timestamps to confirm delivery without relying on pane churn alone.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* fix: satisfy lint in opencode send regression helper
Remove unnecessary escaping in generated shell snippets used by the new OpenCode confirmation tests so CI lint passes cleanly.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* fix: avoid false opencode delivery confirmation
Only treat OpenCode timestamp confirmation as delivered when updatedAt advances beyond a known baseline, and add regression coverage for transient baseline visibility.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
---------
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* feat: add end-to-end observability across core, web, and terminal
Instrument lifecycle/API/websocket flows with correlation-aware metrics and operator health surfaces so the system can self-diagnose and escalate failures.
* fix: realign session restore set and unblock claim PR typecheck
* fix(web): restore project-filtered sessions route after main merge
* fix(core): remove unused orchestrator import to unblock lint
* fix(core): always record lifecycle poll failures and remove dead review branches
* fix(web): harden websocket metrics and reuse SSE observers
* fix(web): preserve primary session API errors when services bootstrap fails
* fix(web): use full orchestrator config type in SSE observer helper
* fix(web): restore project-scoped SSE and guard observability error paths
* fix(web): address remaining Bugbot review gaps
* fix(web): align SSE project attribution and share session project resolver
* fix(web): require request arg for SSE route and align tests
* fix(web): record websocket error disconnects as failures
* fix(web): use path alias for session project resolver import
* fix(web): include active connection count in disconnect metrics
* fix: skip PR auto-detection for orchestrator sessions
Orchestrator sessions sit on the base branch (e.g. master) and should
never own a PR. When the lifecycle worker ran detectPR for these
sessions, any PR whose head branch matched master (e.g. a master->prod
deploy PR) would get incorrectly attached and keep re-attaching after
manual cleanup.
Add a role=orchestrator check to the PR auto-detection guard so
orchestrator sessions are skipped entirely.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add session ID suffix fallback for orchestrator PR auto-detection skip
Pre-existing orchestrator sessions spawned before the role metadata field
was added lack role=orchestrator. Mirror the session-manager pattern by
also checking session.id.endsWith("-orchestrator") as a fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Drop the lifecycle manager's local orchestrator predicate so the final remaining core polling path follows the same shared detection logic as session cleanup, web filters, and CLI status.
Carry legacy project ownership through recovery, drop the redundant dashboard worker filter, and remove the now-unused orchestrator helper so the PR branch stays aligned with the shared per-project orchestration contract.
Preserve the legacy project fallback in the shared session loader and keep the sessions route filtering by project after the clean branch cherry-pick, so the final PR branch matches the verified multi-project behavior.
Recover the owning project from the sessions directory and archived metadata so older session files still resolve to the correct project and reuse the right orchestrator mapping.
* feat(core): add scm webhook contract
Defines a provider-agnostic SCM webhook contract in core types and
config so SCM plugins can verify and normalize inbound webhook events
without reshaping project config later.
* feat(scm): trigger lifecycle checks from github webhooks
Adds GitHub webhook verification and event parsing, exposes a web
webhook endpoint, and routes matching PR/branch events through the
existing lifecycle manager so CI and review reactions update immediately.
* fix(scm): verify github signatures with raw webhook bytes
Preserves the original webhook bytes alongside the decoded payload so
GitHub HMAC verification uses the exact request body while the route
continues to drive lifecycle checks through the existing manager.
* fix(web): wire scm webhook route into main branch services
Restores the main-branch service and route-test wiring while keeping the
new webhook route coverage and scoped lifecycle helper in place.
* fix(webhooks): use singleton lifecycle manager and fail closed on scm API errors
Reuses the existing services lifecycle manager for webhook-triggered checks
so reactions and state transitions don't replay from a fresh instance, and
restores fail-closed behavior for GitHub review comment fetch failures.
* fix(webhooks): tighten project matching and restore scm compatibility methods
Prevents repository-less webhook events from matching all projects, restores
GitHub SCM PR utility methods and CI status rollup fallback, and adds tests
covering the compatibility paths and safer project matching behavior.
* fix(webhooks): pre-check content length and continue on parse errors
Adds an early content-length guard against configured maxBodyBytes before
reading the body and changes candidate parse failures to fail-forward so
one malformed payload path does not abort other valid candidate handling.
* fix(scm-github): parse review-comment timestamps from comment payload
Use comment.updated_at/created_at for pull_request_review_comment webhook
timestamps so normalized events retain temporal data for comment events.
* fix(webhooks): apply early size guard only when all candidates are bounded
Uses the broadest candidate limit for pre-read content-length checks and
skips early rejection when any matching project has no configured limit,
while retaining per-candidate verification limits.
* fix(webhooks): normalize repo matching and skip terminal sessions
Match webhook repository names case-insensitively against configured project
repos and avoid lifecycle checks for terminal sessions when resolving
webhook-affected sessions.
* fix(webhooks): fail forward when lifecycle checks throw
* fix(scm-github): parse push webhook branch and sha
* refactor(scm-github): dedupe cli exec helper wrappers
* fix(scm-github): tighten exec helper type and comment timestamps
* fix(scm-github): prefer head_commit timestamp for push events
* fix(webhooks): tighten repository parsing and helper visibility
* fix(scm-github): ignore non-head refs for push branch
* chore: trigger bugbot rerun
* feat(scm-gitlab): add webhook verification and event parsing
* fix(webhooks): share parser utils and handle check_run branch
* fix(webhooks): ignore gitlab tag refs in ci branch mapping
* fix(scm-gitlab): harden token and tag ref handling
* chore(scm-gitlab): update webhook helpers around bugbot threads
* fix: preserve PR number in recovery and dedupe validation utilities
Bugbot #2908822306: Use parsePrFromUrl utility to correctly extract
PR number from URL instead of defaulting to 0. Applied to both
recovery/actions.ts and session-manager.ts for consistency.
Bugbot #2908822310: Extract safeJsonParse and validateStatus to
shared utils/validation.ts to eliminate duplication between
recovery/validator.ts and session-manager.ts.
Clean rebuild from origin/main with only recovery-specific changes.
* fix: add PR number fallback and remove unused import
Bugbot #2908822306: Add fallback regex to extract PR number from URL
ending when full GitHub URL pattern doesn't match (e.g., non-GitHub URLs).
Bugbot #2908822310: Remove unused SessionStatus import from session-manager.ts
(now imported from utils/validation.ts).
Tests: 406 passed, typecheck clean.
* fix(core): harden recovery action selection and escalation
* fix(core): preserve recovered agent summary metadata
* fix(core): reuse canonical PR type in URL parser
Avoid shadowing the core PRInfo type in the recovery URL parser while keeping the non-GitHub trailing-number fallback covered by tests.
* fix(core): align recovery metadata and session reconstruction
Persist restored timestamps under the canonical metadata key, share
session reconstruction logic between recovery and session loading, and
log the real escalation reason when recovery aborts on retry limits.
* fix(core): keep dry-run escalation reasons accurate
Use the assessment's actual escalation reason in dry-run results so preview output matches real execution behavior for partial-session escalations.
* fix(core): dedupe recovery scanning and honor custom log path
Reuse metadata listing rules in scanner to avoid duplicated session ID filters,
and preserve user-supplied recovery logPath in recovery manager APIs.
* fix(core): align dry-run recovery behavior with real actions
Compute recovery escalation decisions before dry-run returns and route
single-session dry runs through executeAction so action-specific fields
(reason/manual-intervention) are preserved.
* fix(core): derive dry-run recovery report from action execution
Run dry-run recovery classification through executeAction so report actions
match real execution logic, including max-attempt escalation decisions.
* feat: wire lifecycle manager, backlog auto-claim, and dashboard overhaul
- Start LifecycleManager in dashboard server (30s polling) so reactions
actually fire: CI failures, review comments, merge conflicts are now
auto-forwarded to agents
- Add backlog auto-claim poller (60s interval) that watches for issues
labeled `agent:backlog` and auto-spawns agent sessions up to max
concurrent limit (5)
- Add tabbed dashboard UI: Board (kanban), Backlog (issue queue), PRs
- Add issue creation form in dashboard — creates GitHub issues with
`agent:backlog` label for immediate agent pickup
- Add API routes: /api/backlog, /api/issues, /api/setup-labels
- Pass notifier config through plugin registry (slack webhook fix)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add task decomposition layer (classify → decompose → recurse)
Adds LLM-driven recursive task decomposition upstream of session spawning.
Complex issues are broken into atomic subtasks before agents start working.
Each agent receives lineage context (where it fits in the hierarchy) and
sibling awareness (what parallel agents are doing).
Core changes:
- New decomposer module (core/src/decomposer.ts) — classify, decompose,
plan tree, lineage formatting, using Claude API
- Extended SessionSpawnConfig with lineage/siblings fields
- Prompt builder Layer 4: decomposition context (hierarchy + siblings)
- ProjectConfig.decomposer config section with Zod validation
- Tracker plugin: added removeLabels support for label management
CLI:
- `ao spawn <project> <issue> --decompose` flag
- `--max-depth <n>` option for decomposition depth
- Spawns multiple sessions with lineage context for composite tasks
Backlog poller:
- Respects project.decomposer.enabled for auto-decomposition
- Posts plan as issue comment when requireApproval=true
- Auto-spawns subtasks with lineage when requireApproval=false
Config example:
projects:
my-app:
decomposer:
enabled: true
maxDepth: 3
requireApproval: true
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add verification gate — issues stay open until human confirms fix
PR merge no longer auto-closes GitHub issues. Instead:
1. On PR merge: issue labeled `merged-unverified`, stays open
2. Human checks staging, then runs `ao verify <issue>` to close
3. Or `ao verify <issue> --fail` to flag verification failure
Changes:
- services.ts: labelIssuesForVerification() replaces closeIssuesForMergedSessions()
- New CLI command: `ao verify` (verify/fail/list modes)
- New API route: GET/POST /api/verify
- Dashboard: new Verify tab with one-click verify/fail buttons
- ao status: shows count of issues awaiting verification
- Idle session detection + auto-nudge reaction
- Use TERMINAL_STATUSES in batch-spawn dedup check
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: rename decomposerConfig to avoid variable shadowing
Addresses Bugbot medium severity issue where inner variable
shadowed outer from getServices().
* fix: update pnpm-lock.yaml for new @anthropic-ai/sdk dependency
* fix: resolve remaining merge conflicts and syntax errors
- Remove leftover conflict markers in types.ts
- Remove orphaned code in services.ts
- Fix semicolon to comma in config.ts
- Remove unused import in verify.ts
* fix: address final Bugbot issues
- requireApproval path now exits early with continue to prevent
fall-through to in-progress label and session spawned comment
- remove packages/core/package-lock.json (pnpm workspace should only
use root pnpm-lock.yaml)
* fix: idle sessions now transition back to working
When agent resumes activity after being idle, the status correctly
transitions to 'working' instead of remaining stuck in 'idle' state.
* fix(backlog): remove agent:backlog label when claiming issues
When claiming issues from the backlog, the poller now removes the
agent:backlog label in addition to adding agent:in-progress. This
prevents duplicate work if all spawned sessions reach terminal status
and the poller rediscovers the issue.
* fix(test): use Set for TERMINAL_STATUSES mock
The mock for TERMINAL_STATUSES was an array, but the real export is a
ReadonlySet. Changed to use a Set so tests with non-empty sessions won't
crash when calling .has().
* fix(web): resolve backlog/dashboard regressions after branch sync
* fix(web): align dashboard events hook and SSE test mocks
* fix(notifier-openclaw): apply exponential delay from retry index
* fix(integration-tests): align openclaw retry delay expectation
* fix(web): keep dashboard header stats in sync
* fix(openclaw): keep first retry at base delay
---------
Co-authored-by: Agent <agent@example.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Harsh <harsh@Ubuntu-24-Forrest.lan>
Co-authored-by: Harsh <harsh@example.com>
* fix(core): enforce single-owner PR claim consolidation
* refactor: remove dead `takeover` option from claimPR
Since PR consolidation is now automatic (single-owner enforcement),
the `--takeover` flag became dead code. Users passing it got no
error but it had zero effect.
This commit:
- Removes takeover from ClaimPROptions interface
- Removes --takeover flag from spawn and session CLI commands
- Updates orchestrator-prompt.ts examples
- Updates tests to reflect automatic consolidation
Tests: ao-core 403 passing, ao-cli 189 passing (spawn+session)
* fix: remove stale --takeover from Quick Start example
Remove remaining --takeover reference in orchestrator prompt Quick Start section.
* feat(core): document and test asymmetric PR ownership model
- Add JSDoc to claimPR documenting RULE A (exclusive PR->Agent) and
RULE B (Agent->Many PRs) ownership contract
- Add tests for:
- Same session claiming multiple PRs sequentially (RULE B)
- Idempotent re-claim by same owner
- Stale/dead prior owner handoff
- Exclusive PR ownership enforcement (RULE A)
139 tests passing
---------
Co-authored-by: Harsh <harsh@Ubuntu-24-Forrest.lan>
Dhruv Sharma
Gautam Tayal
Deepak Veluvolu
harshitsinghbhandari
ruskaruma
AO Bot
github-actions[bot]
suraj-markup
Joakim Sigvardt
Harsh Batheja
Jayesh Sharma
Harsh
prateek