5-real-agent run on todo-app exhausted GraphQL bucket in 31 min (~9572 pts/hr,
~37x quiet-steady at the same session count). AO polling consumed ~10 calls;
the rest came from agents themselves via the metadata-only ~/.ao/bin/gh
wrapper, which has no tracing. Captures findings, adds Track D (agent-side
gh consumption) plus B5 (migrate remaining bare gh callsites to
execGhObserved), and includes the runbook + benchmark scripts Adil will
build on for the cross-machine reproduction.
Complete scaling curve measured: 50 sessions uses only ~28% of GraphQL
budget with 100% ETag guard hit rate at every scale. Poll cycle lag
identified as first bottleneck (66s at 50 sessions vs 30s target).
CI churn benchmark shows ETag invalidation is a latency problem, not
a rate-limit problem (+9% GraphQL, +4.4x p50 latency).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add read-through caching to the ~/.ao/bin/gh wrapper, targeting the two
largest agent-side waste buckets identified in D4 analysis:
1. PR discovery (gh pr list --head): infinite TTL for positive results.
598 calls → ~10 per 10-session run (98% reduction).
2. Issue context (gh issue view): 300s TTL.
75 calls → ~20 per 10-session run (73% reduction).
The wrapper now caches successful read-only responses in
$AO_DATA_DIR/.ghcache/$AO_SESSION/ and serves them on subsequent
identical calls. Negative results (empty []) are never cached.
gh pr create populates the PR discovery cache immediately.
Also lifts PATH wrapper installation from individual agent plugins into
session-manager, making it universal for all agents including Claude Code:
- session-manager injects PATH + GH_PATH into every runtime.create()
- session-manager calls setupPathWrapperWorkspace() for all agents
- Removes duplicate buildAgentPath/setupPathWrapperWorkspace boilerplate
from codex, aider, opencode, and cursor plugins
Includes D4 implementation plans in experiments/.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`gh api -i` exits code 1 on HTTP 304 responses, causing the catch blocks
in checkPRListETag and checkCommitStatusETag to assume the resource changed
and trigger unnecessary GraphQL batch queries every poll cycle.
Fix: inspect stdout/stderr in the catch block for the 304 status line before
falling back to "assume changed". Also unifies the 304 detection regex to
handle HTTP/1.1, HTTP/2, and HTTP/2.0 status lines, and adds rateLimit
introspection to the batch GraphQL query.
Benchmark result (quiet-steady, 5 sessions, 15 min):
- GraphQL points/hr: 260/5,000 (5%) — down from 820–1,416 pre-fix
- ETag guard 304 rate: 100%
- GraphQL batch calls during measurement: 0
Also fixes the benchmark harness to create placeholder tmux sessions with a
claude symlink so the lifecycle actually polls sessions instead of
short-circuiting to "killed".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three modes: setup (spawn sessions, wait for PRs), measure (trace API
calls over a fixed window, produce scorecard), report (recompute from
existing trace). Node.js stdlib only, shells out to ao CLI and gh CLI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Baseline measurements, discussion notes, benchmark harness spec,
and updated master plan from two independent trace runs at 5-6 sessions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Addresses code review findings:
1. Guard Buffer.byteLength and parseIncludedHttpResponse against
undefined stderr/stdout — fixes 48 SCM test regressions where
mocked execFile paths don't populate stderr
2. extractOperation() now takes only the first path segment of REST
URLs (e.g. "repos" from "repos/acme/repo/pulls/123/...") to keep
operation bucket cardinality bounded and stable across runs
3. Fix A2 runbook /rate_limit snapshots to produce valid JSON using
jq's now|todate instead of appending raw timestamp
4. Add blocker 5 dependency to runbook prereqs and per-session cells
All 140 SCM tests pass (0 failures).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Practical execution plan for the Phase A2 scenario x scale x topology
matrix: 7 priority cells, per-cell procedure, /rate_limit bracketing
for Gap 1 subcommand burn, output format for baseline.md, and the
scorecard that gates Track B.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add -i flag to executeBatchQuery in graphql-batch.ts and split HTTP
headers from JSON body before parsing, making all gh.api.graphql-batch
rows visible to status and rate-limit analysis (was 186 invisible rows)
- Fix extractOperation() in gh-trace.ts to walk past -* flags before
picking the operation segment, eliminating the gh.api.--method bucket
- Add per-reset-window burn segmentation to both analyzers so runs
straddling a reset boundary produce per-window deltas instead of a
single invalid cross-reset delta
- Add experiment scripts: analyze-trace.mjs (deep trace analysis) and
drill-tracer.mjs (standalone tracer exerciser)
- Document Gap 1 decision in PLAN.md: accept CLI subcommands as opaque
for A1, bracket A2 runs with /rate_limit snapshots for coarse burn
- Add progress timeline to PLAN.md showing A→B→C track dependencies
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Record the five A1b pre-freeze blockers surfaced by Adil's 1,487-row
baseline and an independent drill run: graphql-batch missing -i,
extractOperation flag mis-bucketing, analyzer not segmenting burn by
reset window, CLI-subcommand opacity (GH_DEBUG=api stderr vs coarse
/rate_limit bracket — not equivalent), and sessionId/projectId not
threaded through plugin callsites. Note bare gh() helper cleanup as
known-open follow-up.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces execGhObserved() in @aoagents/ao-core: a thin wrapper around
execFile("gh", ...) that writes a JSONL trace row to $AO_GH_TRACE_FILE
on both success and failure. Captures status line, HTTP status, ETag,
rate-limit headers, duration, stdout/stderr byte counts, exit code, and
signal. No-op when the env var is unset, so default behavior is
unchanged.
Migrates three call sites to the observer:
- scm-github/graphql-batch.ts — PR-list guard, commit-status guard,
GraphQL batch query
- scm-github/index.ts — gh() and ghInDir() helpers
- tracker-github/index.ts — internal gh() helper
This is Phase A1a of experiments/PLAN.md: tracer infrastructure +
migration. The full GhRunner contract (Promise<GhResult>,
GhRunnerError.ghResult on reject, body capture, redaction, 64 KB cap)
lands in A1b along with the scorecard baseline.
Also adds experiments/ reference docs: the v2.3 plan, the gh-CLI call
catalog, two ETag verification writeups, and a trace harness + summary
script.
Dhruv Sharma
adil