agent-orchestrator/experiments/baseline.md

6.4 KiB
Raw Blame History

Baseline Measurements

Branch: feat/gh-rate-limiting
Purpose: Scenario × scale × topology matrix of GH API consumption under current (unfixed) behavior. Gates Track B fixes.


Completed Cells

Cell: S2-T1-5 — Quiet Steady State, Single Repo, 5 Sessions

The first real measurement. Two independent runs by different operators on separate machines to verify reproducibility.

Run 1 — Adil (@whoisasx), 2026-04-15

Field Value
Operator @whoisasx
Trace file gh-trace-1776260288.jsonl (gist linked in PR #1238 comment)
Repo illegalcall/todo-app
Sessions 5
Duration 32 min 41 sec
Total calls 974
Calls/min 29.8

By operation:

Operation Count %
gh.pr.list 184 18.9%
gh.api.graphql 159 16.3%
gh.pr.view 139 14.3%
gh.api.--method 128 13.1%
gh.api.graphql-batch 106 10.9%
gh.api.guard-pr-list 106 10.9%
gh.pr.checks 99 10.2%
gh.issue.view 53 5.4%

HTTP status (of rows that have it):

Status Count Notes
none 868 CLI subcommands — no HTTP visibility (Gap 1)
304 91 ETag guard hit but treated as error (Bug #1)
200 15 Fresh fetches

Rate-limit burn (from headers):

Resource Window Remaining start→end Delta Elapsed Burn rate
(naive) mixed 4995 → 4981 14 32.7 min 25.7/hr

Note: Adil's run had gh.api.--method as an operation name (128 calls, 13.1%). This is the unbounded-cardinality bug in extractOperation() — fixed in blocker #2 (first path segment extraction). The 128 calls are gh api --method GET repos/.../pulls/:n/comments — REST per-PR comment fetches.

Note: The naive burn rate of 25.7/hr is misleadingly low. Only 106/974 rows (10.9%) had rate-limit headers. The 304s (91 calls) don't decrement the bucket but aren't visible per-call either. True burn is underreported because CLI subcommands (868 rows) don't expose headers.


Run 2 — Dhruv (@illegalcall), 2026-04-16

Field Value
Operator @illegalcall
Trace file gh-trace-verify-1776345900.jsonl
Repo illegalcall/todo-app
Sessions 6 (5 issues + 1 project)
Duration 22 min 21 sec
Total calls 234
Calls/min 10.5
Success rate 81.6% (191 ok, 43 failed)

By operation:

Operation Count %
gh.api.graphql 46 19.7%
gh.api.repos 46 19.7%
gh.api.guard-pr-list 35 15.0%
gh.api.graphql-batch 35 15.0%
gh.pr.view 25 10.7%
gh.issue.view 16 6.8%
gh.pr.list 16 6.8%
gh.pr.checks 15 6.4%

HTTP status (of rows that have it):

Status Count Notes
none 164 CLI subcommands — no HTTP visibility
200 43 Fresh fetches
304 27 ETag guard hits (treated as failures — Bug #1)

Failure breakdown:

Exit code Operation Count Cause
1 gh.api.guard-pr-list 27 304 treated as error (Bug #1)
1 gh.pr.checks 15 No PR exists yet for new sessions
1 gh.issue.view 1 Unknown

Rate-limit burn (per reset window):

Resource Reset window Remaining start→end Delta Elapsed Burn rate
graphql 13:41:05Z 4714 → 4537 177 7.5 min 1,416/hr
core 13:49:03Z 4994 → 4987 7 15.0 min 28/hr
graphql 14:41:10Z 4995 → 4872 123 9.0 min 820/hr

Rate-limit before/after snapshots:

Bucket Before After Notes
core remaining=4997, used=3 remaining=4940, used=60 Same reset window → 57 tokens burned
graphql remaining=4909, used=91 remaining=4875, used=125 Different reset windows — crossed boundary

Cross-Run Comparison

Metric Run 1 (Adil, 33min) Run 2 (Dhruv, 22min) Notes
Calls/min 29.8 10.5 Run 1 was longer, more PRs existed → more per-PR calls
GraphQL burn/hr ~25.7 (naive, unreliable) 8201,416/hr (per-window) Per-window is the reliable number
Core burn/hr (not split) 28/hr Negligible
304 rate 91/974 (9.3%) 27/234 (11.5%) Consistent — ~10% of calls hit the dead ETag guard
Header coverage 106/974 (10.9%) 70/234 (29.9%) Run 2 has better coverage due to blocker #1 fix (-i flag)
graphql-batch calls 106 35 ~3/min in both runs
guard-pr-list calls 106 35 1:1 with graphql-batch (expected — guard runs before batch)

Key finding: graphql-batch is the dominant measured budget consumer. Per-window burn of 8201,416 tokens/hr at 56 sessions. REST core at 28/hr is negligible by comparison.

Bug #1 observation: 304-as-error causes most guard-pr-list calls to be treated as changes, which drives unnecessary graphql-batch calls. Bug #1 is the first high-confidence cause to remove.


Extrapolation Table (Linear, Unfixed Behavior)

Sessions GraphQL burn/hr (low est) GraphQL burn/hr (high est) Budget (5,000/hr) Headroom
5 820 1,416 5,000 ~3.56x safe
10 1,640 2,832 5,000 ~1.83x safe
25 4,100 7,080 5,000 OVER at high end
50 8,200 14,160 5,000 1.62.8x OVER

Linear extrapolation is a floor estimate. Real scaling may be superlinear due to batch query complexity growing with session count.


Pending Cells

The full A2 matrix (6 scenarios × 2 topologies × 5 scales) is defined in a2-baseline-runbook.md. Priority cells:

Cell Scenario Topology Scale Status
S2-T1-5 Quiet steady Single repo 5 Done (above)
S1-T1-5 Cold start Single repo 5 Pending
S3-T1-5 Spawn storm Single repo 5 Pending
S2-T1-10 Quiet steady Single repo 10 Pending
S2-T1-25 Quiet steady Single repo 25 Pending
S4-T1-5 Review backlog Single repo 5 Pending
S2-T2-5 Quiet steady Multi repo 5 Pending

These cells are blocked on blocker #5 (sessionId/projectId threading) for per-session attribution, per the runbook prereqs.