diff --git a/experiments/DISCUSSION-NOTES.md b/experiments/DISCUSSION-NOTES.md index cdb4e7527..299431a2b 100644 --- a/experiments/DISCUSSION-NOTES.md +++ b/experiments/DISCUSSION-NOTES.md @@ -62,7 +62,7 @@ The ETag guard is broken: Also: Bug #2 — HTTP status check misses `HTTP/2.0 304` (only matches `HTTP/1.1` and `HTTP/2`). -**Assigned to:** Adil (@whoisasx) as B1 task. PR comment drafted but not yet posted. +**Status:** ✅ Fixed (commit `cd0b16ca`). Both `checkPRListETag` and `checkCommitStatusETag` catch blocks now inspect stdout/stderr for 304 before falling back to "assume changed". Also added `rateLimit { cost remaining resetAt }` to the GraphQL batch query for free cost attribution. PR comment posted to Adil for independent verification. --- @@ -116,7 +116,77 @@ Scorecard metrics: GraphQL points/hr, REST core requests/hr, graphql-batch count 4. After Bug #1 fix: re-run same three sizes 5. Compare before/after scorecards -**Status:** Spec complete and reviewed. Ready for implementation plan. +**Status:** ✅ Built and working (`experiments/benchmark.mjs`). Three modes: `setup`, `measure`, `report`. Validated end-to-end with B1 fix — see benchmark results below. + +### Benchmark Results (2026-04-17, B1 fix applied) + +15-minute quiet-steady benchmark, 5 sessions, single repo (`illegalcall/todo-app`): + +| Metric | Value | +|--------|-------| +| GraphQL points/hr | 260 / 5,000 (5%) — **~70% reduction from pre-fix baseline** | +| REST core requests/hr | 0 / 5,000 (0%) | +| Total GH calls | 250 (16.7/min) | +| graphql-batch count | **0** (all skipped by ETag guards) | +| guard-pr-list 304s | 30 (100.0%) | +| guard-pr-list errors | 0 | +| ETag guard 304 rate | **100%** | +| p50 / p95 / p99 latency | 746 / 1,165 / 1,261 ms | + +**Scorecard:** `experiments/out/scorecard-quiet-steady.single-repo.5-1776384105.json` +**Trace:** `experiments/out/gh-trace-bench-1776383083.jsonl` (281 rows) + +### 10-Session Benchmark (2026-04-17, B1 fix applied) + +| Metric | Value | +|--------|-------| +| GraphQL points/hr | 640 / 5,000 (13%) | +| REST core requests/hr | 0 / 5,000 (0%) | +| Total GH calls | 470 (31.3/min) | +| graphql-batch count | **0** | +| guard-pr-list 304s | 30 (100.0%) | +| p50 / p95 / p99 latency | 803 / 1,968 / 2,509 ms | + +**Scorecard:** `experiments/out/scorecard-quiet-steady.single-repo.10-1776419128.json` +**Trace:** `experiments/out/gh-trace-bench-1776418105.jsonl` (526 rows) + +### Scaling Analysis (5 → 10 sessions) + +| Metric | 5 sessions | 10 sessions | Factor | +|--------|-----------|-------------|--------| +| GraphQL points/hr | 260 | 640 | 2.46x | +| Total calls/min | 16.7 | 31.3 | 1.88x | +| Opaque calls | 70 | 140 | 2.0x | +| Guard 304 count | 30 | 30 | 1.0x (repo-scoped) | +| p99 latency | 1,261ms | 2,509ms | 1.99x | + +Scaling is slightly super-linear for GraphQL (2.46x for 2x sessions). Guard checks are repo-scoped and don't scale with session count. Opaque calls (per-session subcommands) scale linearly. + +### 20-Session Benchmark (2026-04-17, B1 fix applied) + +| Metric | Value | +|--------|-------| +| GraphQL points/hr | 680 / 5,000 (14%) | +| REST core requests/hr | 0 / 5,000 (0%) | +| Total GH calls | 910 (60.7/min) | +| graphql-batch count | **0** | +| guard-pr-list 304s | 30 (100.0%) | +| p50 / p95 / p99 latency | 761 / 2,798 / 3,052 ms | + +**Scorecard:** `experiments/out/scorecard-quiet-steady.single-repo.20-1776424159.json` +**Trace:** `experiments/out/gh-trace-bench-1776423135.jsonl` + +### Key Finding: Sub-Linear Scaling + +GraphQL cost barely increased from 10→20 sessions (640→680, +6%). The guard-pr-list check is repo-scoped (constant 30 checks regardless of session count), and graphql-batch stays at 0 during steady state. Most of the per-session cost comes from opaque `gh pr view/checks` subcommands which are individually cheap. + +**Revised 50-session projection:** ~800–1,000 GraphQL pts/hr (16–20% of budget). Far better than the earlier 64% estimate. **B2 structural reductions are NOT required for quiet-steady state.** The 50-session target is safely achievable with B1 alone. + +**Key harness implementation notes:** +- Creates placeholder tmux sessions with a `claude` symlink → `/bin/sleep 86400` so lifecycle polls sessions instead of short-circuiting to "killed" +- macOS `/bin/sleep` doesn't accept `infinity` — use `86400` (24h) +- Must set `AO_CONFIG_PATH` to the todo-app config when running from the AO repo directory +- The todo-app config auto-infers `scm: { plugin: "github" }` from the `repo` field --- @@ -130,15 +200,19 @@ Scorecard metrics: GraphQL points/hr, REST core requests/hr, graphql-batch count | `experiments/analyze-trace.mjs` | Detailed trace analyzer (per-window burn) | | `experiments/summarize-gh-trace.mjs` | Summary trace analyzer | | `experiments/drill-tracer.mjs` | Standalone tracer exercise script | -| `packages/core/src/gh-trace.ts` | The tracer (execGhObserved) | -| `packages/plugins/scm-github/src/graphql-batch.ts` | Where Bug #1 lives (+ `-i` flag fix) | +| `experiments/benchmark.mjs` | **Repeatable benchmark harness** (setup/measure/report) | | `experiments/benchmark-spec.md` | Benchmark harness spec | +| `experiments/out/scorecard-*.json` | Benchmark scorecards (JSON) | +| `experiments/out/gh-trace-bench-*.jsonl` | Benchmark trace files | +| `packages/core/src/gh-trace.ts` | The tracer (execGhObserved) | +| `packages/plugins/scm-github/src/graphql-batch.ts` | B1 fix: ETag 304 handling + rateLimit instrumentation | --- ## Open decisions -1. **B1 PR comment to Adil** — drafted, not yet posted. Waiting for user approval. +1. ~~**B1 PR comment to Adil** — drafted, not yet posted.~~ ✅ Posted. Awaiting Adil's independent verification run. 2. **Benchmark with bugbot/CI** — Dhruv enabled bugbot on todo-app. Want to verify empirically that CI/reviews don't change polling cost. 3. **Blocker #5 (sessionId/projectId threading)** — deferred. Needed for per-session attribution in the remaining A2 matrix cells. -4. **50-session validation** — target tier, not first measurement tier. Get real data at 5/10/20 first. +4. **Scale-up validation (10, 20 sessions)** — next step. Can run locally now without waiting for Adil. Replaces extrapolation with measured data. +5. **50-session validation** — target tier, not first measurement tier. Get real data at 5/10/20 first. diff --git a/experiments/PLAN.md b/experiments/PLAN.md index 66dd55ee3..e480ed8a9 100644 --- a/experiments/PLAN.md +++ b/experiments/PLAN.md @@ -78,19 +78,29 @@ Bursts of `N = 20, 60, 120, 200` via `gh api /user` in parallel and `N = 150` vi The most important structural change from v1: **the baseline recorder (Track A) and any behavior-changing work (Track B) are strictly separated.** Track A ships first, unchanged behavior, just instrumentation. Only after we have baseline numbers does Track B start landing fixes. -### Progress & Status (updated 2026-04-16) +### Progress & Status (updated 2026-04-17) **Tracks are strictly sequential:** A → B → C. Each track depends on the previous track's output. ``` Track A ── Measure ────────────────────────────────────────────────────── A1a Ship execGhObserved() + JSONL recorder ✅ Done (PR #1238) - A1b Fix tracer blind spots (5 blockers) 🔄 In progress - A2 Baseline scenario × scale × topology matrix ⏳ Blocked on A1b + A1b Fix tracer blind spots (5 blockers) ✅ Done (blockers 1-4 fixed, #5 deferred) + A2 Benchmark harness + baseline data ✅ Done — harness built, 5-session baseline captured + Scorecard: experiments/out/scorecard-quiet-steady.single-repo.5-*.json Track B ── Fix bugs ───────────────────────────────────────────────────── - B1 Safe behavioral fixes (304, status parsing) ⏳ Blocked on A2 baseline - B2 Structural reductions (detectPR dedup, batch) ⏳ Blocked on B1 + B1 Safe behavioral fixes (304, status parsing) ✅ Done (cd0b16ca, pushed, PR #1238) + — ETag 304-as-error fix in graphql-batch.ts + — is304() + extractErrorOutput() helpers + — rateLimit { cost remaining resetAt } added to batch query + — Verified: 100% guard 304 rate, 0 graphql-batch calls in quiet-steady + — Awaiting Adil's independent verification (PR comment posted) + B2 Structural reductions (detectPR dedup, batch) ⏳ Blocked on B1 verification + B3 Scale-up validation (10, 20 sessions) ✅ Done — sub-linear scaling confirmed + 5→260, 10→640, 20→680 GraphQL pts/hr + 50-session projection: ~800-1000 pts/hr (16-20% budget) + B2 structural reductions NOT required for quiet-steady Track C ── Octokit migration (optional) ───────────────────────────────── C1 OctokitRunner behind flag + compare ⏳ Blocked on B scorecard