--- name: bug-triage description: Triage bugs reported in chat/issues, search for duplicates, file or update GitHub issues with full context, and push fix PRs. trigger: User reports a bug, or asks to triage/file an issue for a reported problem. --- # Bug Triage Skill Triage bugs into well-structured GitHub issues on the upstream **`AgentWrapper/agent-orchestrator`** repo (issues are enabled there; the `origin` fork is not the issue tracker). > **Agent Orchestrator (AO) is Go + Electron.** The backend is a Go daemon > (`backend/`) exposing a loopback HTTP API on `127.0.0.1:3001`; the frontend is an > Electron + React supervisor (`frontend/`). There is **no** pm2/tmux-per-session > Node runtime here: the daemon owns lifecycle and terminals run under the **tmux** > runtime adapter (ConPTY on Windows). Triage against _this_ Go rewrite, not the old > TypeScript agent-orchestrator implementation. ## ⚠️ Which `ao` are you running? **A bare `ao` on your PATH may resolve to a different AO install** (for example an old npm build at `~/.nvm/.../bin/ao` that talks to port **:3000**). Triaging with the wrong binary produces bugs that don't exist in this rewrite (and misses ones that do). Before any diagnostics: ```bash which -a ao # see every ao on PATH; expect surprises ao status 2>/dev/null # if this shows port 3000, it is NOT this rewrite ``` Use a rewrite binary explicitly: ```bash # Option A: build from this repo (preferred during triage) cd backend && go build -o /tmp/ao ./cmd/ao /tmp/ao status # must report port: 3001 # Option B: the packaged app's bundled daemon "/Applications/Agent Orchestrator.app/Contents/Resources/daemon/ao" status ``` **Confirm `ao status` reports `port: 3001` before trusting any output.** Throughout this skill, `ao` means _your verified rewrite binary_ (`/tmp/ao` or the bundled one), never a bare PATH lookup. > Note: spawned sessions get a PATH pin so the _session's_ `ao` resolves to the > daemon's own executable (see `hookPATH` in > `backend/internal/session_manager/manager.go`). That pin only applies inside > sessions; your interactive shell is still on its own PATH, so pin it yourself. ## 1. Pre-flight - **Pull latest code:** `git fetch upstream && git log --oneline upstream/main -5`. Stale code means bad triage. (`upstream` = `AgentWrapper/agent-orchestrator`.) - **Target repo:** Always file on **`AgentWrapper/agent-orchestrator`** (the upstream product repo, where issues live). Never file on the `origin` fork or on `aoagents/*`. - **Verify your binary:** confirm `ao status` shows port **3001** (see warning above). - **Record source:** chat URL, reporter name, attachments. ## 2. Gather Context ### 2a. Extract the report | Source | How to gather | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | **Discord/Slack thread** | Read full thread. Extract: reporter name, original description (the thread starter, not whoever tagged you), screenshots, follow-ups | | **GitHub issue** | `gh issue view --repo AgentWrapper/agent-orchestrator --json body,comments` | | **Live observation** | Pull live state via the daemon: `ao status`, `ao session ls`, `ao session get ` | ### 2b. Minimum viable report gate Before tracing code, verify the report has enough substance: **Required (ALL):** what happened, where (page/command/feature), when (after upgrade? first time?) **Required (2 of 4):** OS/shell, AO version (`ao version`), reproducibility (consistent vs intermittent), reproduction steps If insufficient, ask: > "I'd like to triage this but need more info: (1) **What happened?** (error/behavior), (2) **Where?** (page/command), (3) **When did it start?**, (4) **How to reproduce?**" ### 2c. Local diagnostics (if bug is on same machine) Gather everything yourself before asking the reporter. Use your **verified** rewrite binary (`/tmp/ao` here) for every `ao` call: ```bash # Environment /tmp/ao version && go version && echo $SHELL && uname -a which -a ao # confirm no rogue ao shadows the build cat ~/.ao/running.json # PID + port handshake (expect port 3001) # Daemon health /tmp/ao status # daemon up? port? health/ready probes /tmp/ao doctor # local health checks lsof -i :3001 # who's bound to the daemon port tail -n 100 ~/.ao/daemon.log # daemon log # Sessions & runtime /tmp/ao session ls # all sessions and their state /tmp/ao session get # one session: spawn config, runtime, lifecycle tmux ls # tmux runtime sessions backing terminals (macOS/Linux) # Durable state (SQLite at ~/.ao/data) sqlite3 ~/.ao/data/ao.db '.tables' # inspect schema/rows if state looks wrong ``` The daemon owns lifecycle, sessions, storage, and the terminal mux; structured state lives in `~/.ao/data/ao.db` (WAL: `ao.db-wal`, `ao.db-shm`). The PID+port handshake is `~/.ao/running.json`. **Try the reproduction steps.** Running the actual command against the daemon on :3001 is worth 100 lines of code tracing. ## 3. Investigate ### 3a. Trace the code path **Always trace the actual code**; do not surface-level diagnose. A symptom that looks like a simple `ao stop` issue is often a lifecycle/session-manager problem one layer down. The layers: - CLI (Cobra, thin client over daemon HTTP): `backend/internal/cli/`, entrypoint `backend/cmd/ao/main.go` - Daemon (loopback HTTP on :3001): `backend/internal/daemon/daemon.go`, controllers under `backend/internal/httpd/controllers/` - Sessions & lifecycle: `backend/internal/session_manager/manager.go` - Runtime adapter (tmux / ConPTY): `backend/internal/adapters/runtime/` (`tmux/`, `conpty/`, `ptyexec/`, selected by `runtimeselect/`) - Agent harness adapters: `backend/internal/adapters/agent//` - Terminal mux: `backend/internal/terminal/` - Agent hooks: `backend/internal/cli/hooks.go` ```bash git fetch upstream main && git log --oneline upstream/main -5 # current HEAD # Record the commit hash you're analyzing against ``` **Git archaeology**: find which commits introduced/removed specific code: ```bash git log --oneline -S 'exact-string' -- git show -- | grep -B 5 -A 10 'pattern' ``` **Research dependencies** (tmux, the agent harness binary, Electron, React, the SQLite driver): check installed vs latest version, search their issue trackers, check changelogs. Root cause is sometimes in a dependency, not AO itself. ### 3b. Cross-platform check AO targets **macOS, Linux, and Windows**. If env info indicates Windows (or is unknown), check for these patterns: - **Path separators**: hardcoded `/` or `\`; use `filepath.Join`, not string concat - **Shell syntax**: PowerShell lacks `&&`, `$VAR`, `$(cat ...)`, `/dev/null`, here-docs - **`runtime.GOOS == "windows"` scattered inline**: centralize platform checks - **Runtime backend**: tmux on macOS/Linux vs ConPTY on Windows (`runtimeselect` picks per platform); Windows has no tmux - **Process-tree kills**: POSIX process groups vs Windows job objects - **`localhost`**: Windows resolves to `::1` first; the daemon binds the explicit loopback host (see `config.LoopbackHost`) to avoid IPv4/IPv6 stalls - **Case-insensitive filesystems**: do not compare paths with raw `==` - **PATH / binary resolution**: `.exe`/`PATHEXT` lookup, the session PATH pin (`hookPATH` in `backend/internal/session_manager/manager.go`) Key files: `backend/internal/config/config.go`, `backend/internal/session_manager/manager.go`, `backend/internal/adapters/runtime/`, `backend/internal/terminal/`. ### 3c. Stop-and-ask triggers Stop and ask for more info if: - **3 failed hypotheses**: traced 3 code paths, none explain it - **Root cause is a dependency**: file with the dependency reference, don't guess a local fix - **UI-only bug** and you can't screenshot, ask reporter to describe - **Can't reproduce**: ask for different config/sequence ## 4. Search for Duplicates Search with multiple strategies, always using `--state all` (closed bugs regress): ```bash gh issue list --repo AgentWrapper/agent-orchestrator --state all --search "" gh issue list --repo AgentWrapper/agent-orchestrator --state all --search "" gh issue list --repo AgentWrapper/agent-orchestrator --state all --search "" gh pr list --repo AgentWrapper/agent-orchestrator --state all --search "" ``` ### Duplicate found → comment on existing issue ```bash gh issue comment --repo AgentWrapper/agent-orchestrator --body "$(cat <<'EOF' ## New Report **Reported by:** @ in [chat]() **Date:** | **Checkout:** `` EOF )" ``` ### No duplicate → file new issue (next section) ## 5. File New Issue ### 5a. Pre-submission checklist - [ ] Reporter attribution correct (original reporter, not who tagged you) - [ ] Commit hash recorded - [ ] AO version recorded (`ao version`) - [ ] Reproduced against the rewrite (:3001 / Go code path), not another AO install - [ ] Root cause confidence scored (see 5c) - [ ] Related issues cross-linked - [ ] Reproduction steps are concrete - [ ] Screenshots uploaded with real URLs (see 5b) ### 5b. Upload screenshots **⛔ NEVER use placeholder URLs.** Upload BEFORE creating the issue. ```bash SLUG="descriptive-slug" # Create asset branch gh api -X POST repos/AgentWrapper/agent-orchestrator/git/refs \ -f ref="refs/heads/issue-assets-${SLUG}" \ -f sha=$(git rev-parse upstream/main) # Upload (portable base64) IMG_B64=$(base64 < /path/to/screenshot.png | tr -d '\n') gh api -X PUT "repos/AgentWrapper/agent-orchestrator/contents/.issue-assets/${SLUG}/name.png" \ -f message="chore: upload screenshot" \ -f content="$IMG_B64" \ -f branch="issue-assets-${SLUG}" # Use: ![screenshot](https://raw.githubusercontent.com/AgentWrapper/agent-orchestrator/issue-assets-/.issue-assets/) ``` ### 5c. Create the issue ```bash gh issue create --repo AgentWrapper/agent-orchestrator --title "" --body "$(cat <<'EOF' ## Bug <summary> **Source:** <url> | **Reported by:** @<reporter> | **Analyzed against:** `<hash>` **Confidence:** High/Medium/Low ## Reproduction 1. <step> ## Root Cause <file paths, line numbers, explanation> ## Fix <suggested approach> ## Impact - <effects> EOF )" ``` ### 5d. Label and prioritize **Check which labels actually exist first**, then apply only those: ```bash gh label list --repo AgentWrapper/agent-orchestrator # source of truth; apply only these gh issue edit <number> --repo AgentWrapper/agent-orchestrator --add-label "bug" ``` The repo currently carries `bug`, `enhancement`, `documentation`, `question`, `priority: critical/high/medium/low` (plus the hyphen-free `priority:high` / `priority:medium` variants), status labels (`todo`, `in-progress`, `review`, `blocked`, `verify-if-fixed`, `to-reproduce`, `to-explore`), `sub-issue`, `ready-for-agent`, `good-first-issue`, `help wanted`, and `upstream complication` (for bugs rooted in Claude Code / Codex / tmux etc.). **Do not invent labels**: if a priority or confidence label you want doesn't exist, **state it in the issue body instead** (e.g. "**Priority:** high: core feature broken, no workaround" / "**Confidence:** medium"). | Priority | Criteria | | -------------------- | ----------------------------------- | | `priority: critical` | Data loss, security, system down | | `priority: high` | Core feature broken, no workaround | | `priority: medium` | Feature degraded, workaround exists | | `priority: low` | Cosmetic, edge case | **Confidence scoring** (always state in the issue body): | Level | Meaning | | ---------- | ----------------------------------------------------------- | | **High** | Traced exact code path, specific lines, mechanism explained | | **Medium** | Strong hypothesis but unconfirmed | | **Low** | Can't trace, multiple conflicting theories | ### 5e. Cross-link related issues Search by subsystem and add a `## Related` section to the issue body: ``` ## Related - [#20](url): stale session blocking ao start (same subsystem) - [#35](url): same race condition ``` ### 5f. Push a fix PR (always attempt) This is a Go repo: fixes go through a local branch, build, and `gh pr create` against upstream. There is no remote-patch script. Branch off `upstream/main`. - **Unclear fix:** Don't push a guess. Document and flag in the issue. - **Trivial, verifiable fix** (you can build and test it yourself): ```bash git fetch upstream main && git checkout -b fix/<slug> upstream/main # make the edit cd backend && go build ./... && go test ./... # must pass before pushing git commit -am "fix(<scope>): <summary> Fixes #<n>" git push -u origin fix/<slug> gh pr create --repo AgentWrapper/agent-orchestrator --fill \ --title "fix(<scope>): <summary>" \ --body "Fixes #<n> ## Summary <what changed> ## Test cd backend && go build ./... && go test ./..." ``` - **Non-trivial fix** (broad change, needs iteration, or you can't fully verify): spawn a worker session to do the work in its own worktree instead of pushing a guess: ```bash ao spawn --project agent-orchestrator --prompt "Fix #<n>: <one-line problem statement>. \ Root cause: <file:line + mechanism>. Suggested approach: <approach>. Branch off upstream/main. \ Build with 'cd backend && go build ./... && go test ./...' before opening a PR against AgentWrapper/agent-orchestrator." ``` Note the issue with which path you took (PR or spawned worker). ### 5g. Report back Issue URL, PR URL (if created) or spawned worker session ID, labels applied (and any priority/confidence stated in the body), root cause summary. --- ## Appendix ### A. Subsystem Quick Reference | Subsystem | Collect | Key files | | ------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------- | | **CLI** (`ao start/stop/spawn`) | Version, install method, OS, which binary | `backend/internal/cli/`, `backend/cmd/ao/main.go` | | **Daemon / HTTP API** | `ao status`, port, daemon.log | `backend/internal/daemon/daemon.go`, `backend/internal/httpd/controllers/` | | **Sessions / Lifecycle** | Session ID, spawn config, runtime, state | `backend/internal/session_manager/manager.go` | | **Runtime (tmux / ConPTY)** | tmux version, `tmux ls` (macOS/Linux) | `backend/internal/adapters/runtime/` | | **Terminal mux** | Runtime type, shell, attach behavior | `backend/internal/terminal/` | | **Agent harness** | Harness name + version | `backend/internal/adapters/agent/<harness>/` | | **Storage** | DB state, migrations | `backend/internal/storage/sqlite/`, `~/.ao/data/ao.db` | | **Hooks** | Hook event, agent, payload | `backend/internal/cli/hooks.go` | | **Frontend (Electron/React)** | Screenshot, viewport, daemon connectivity | `frontend/src/` | **Misrouting patterns:** - Terminal bugs → tmux/ConPTY runtime adapter vs the terminal mux vs the Electron xterm surface. Trace where bytes flow (daemon → mux → frontend). - "Session stuck" → lifecycle/session-manager state vs agent harness process vs tmux runtime connection. - "Config not saving" → config loading (`backend/internal/config/config.go`) vs project registration vs SQLite write (`~/.ao/data/ao.db`). - "Command does nothing / wrong port" → you're on the wrong `ao` binary (:3000 vs :3001). Re-check `which -a ao` and `ao status`. ### B. Remote Code Inspection (no local clone) ```bash gh api repos/AgentWrapper/agent-orchestrator/git/trees/main?recursive=1 --jq '.tree[].path' # list files gh api repos/AgentWrapper/agent-orchestrator/contents/{path} --jq '.content' | python3 -c "import base64,sys; sys.stdout.buffer.write(base64.b64decode(sys.stdin.read()))" # read file gh search code "term" --repo AgentWrapper/agent-orchestrator --json path --jq '.[].path' # search code gh api "repos/AgentWrapper/agent-orchestrator/commits?path={path}&per_page=10" --jq '.[] | "\(.sha[0:8]) \(.commit.message | split("\n")[0])"' # file history ``` ### C. Build / Version Diagnostics AO is built from source in this rewrite, not published to npm. Pin the binary under test and reproduce against a known build: ```bash cd backend && go build -o /tmp/ao ./cmd/ao # build the binary under test /tmp/ao version # record version/commit go version # toolchain (build issues are often here) git log --oneline upstream/main -1 # the commit you're analyzing against ``` To bisect a regression, build `ao` at two commits and compare behavior: ```bash git checkout <good-sha>; (cd backend && go build -o /tmp/ao-good ./cmd/ao) git checkout <bad-sha>; (cd backend && go build -o /tmp/ao-bad ./cmd/ao) git checkout - # run the repro against /tmp/ao-good vs /tmp/ao-bad ``` ## Formatting Rules - **Linkify all issue/PR refs:** `[#123](https://github.com/AgentWrapper/agent-orchestrator/issues/123)`, `[PR #456](url)`. Never bare `#123`. ## Pitfalls - **Wrong `ao` binary.** A bare `ao` may be a different AO install (old npm build on :3000). Always pin a rewrite binary and confirm `ao status` shows port **3001**. - **Verify the bug reproduces against the rewrite (:3001 / Go code path) before filing** (symptoms first seen in another AO install may not reproduce here). - **File on upstream, not the fork.** Issues go to `AgentWrapper/agent-orchestrator`; `origin` is a personal fork with no issue tracker. - **Reporter ≠ person who tagged you.** Always attribute to the original reporter. - **Record the commit hash** you analyzed; code changes fast. - **GitHub issue is mandatory**: every triaged bug gets one, even if fix is trivial. - **Only apply labels that exist** (`gh label list --repo AgentWrapper/agent-orchestrator`). State priority/confidence in the body when no matching label exists. - **Build before you push.** `cd backend && go build ./... && go test ./...` must pass; never open a PR with an unverified Go change. - **`gh api --jq .content` truncates large files** (>~100KB). Use local git instead. - **Don't invent Go file paths.** Grep the repo to confirm a path before citing it in an issue.