diff --git a/.github/TESTING_GUIDE.md b/.github/TESTING_GUIDE.md new file mode 100644 index 000000000..dca521006 --- /dev/null +++ b/.github/TESTING_GUIDE.md @@ -0,0 +1,163 @@ +# Testing Guide for Agent Orchestrator + +## Quick Start (For Future Agents/Contributors) + +**Before making changes to orchestrator setup logic, run this test:** + +```bash +./scripts/test-orchestrator-setup.sh +``` + +**Expected:** All 5 tests pass in ~5 seconds. + +## What This Tests + +The test verifies the critical path for new users: + +1. `ao init --auto` → Creates config **without** `sessionPrefix` +2. `ao start` → Creates orchestrator session with correct naming +3. Session naming follows pattern: `{projectId}-orchestrator` (NOT `undefined-orchestrator`) +4. Metadata file created in correct location +5. tmux session exists and is functional + +## Why This Test Exists + +**Bug that was fixed (2026-02-16):** + +When users ran `ao init`, the config didn't include `sessionPrefix`. Then `ao start` would construct the orchestrator session ID as: + +```typescript +const sessionId = `${project.sessionPrefix}-orchestrator`; +// Result: "undefined-orchestrator" ❌ +``` + +The dashboard looks for sessions ending with `-orchestrator`, but couldn't find `undefined-orchestrator`. + +**The fix:** + +```typescript +const sessionId = `${project.sessionPrefix || projectId}-orchestrator`; +// Result: "my-project-orchestrator" ✅ +``` + +## Manual Testing Procedure + +If you need to test manually (see [QUICKTEST.md](../QUICKTEST.md)): + +```bash +# 1. Create test environment +mkdir /tmp/ao-test && cd /tmp/ao-test +git init && git remote add origin git@github.com:test/test.git + +# 2. Initialize +ao init --auto + +# 3. Start orchestrator +ao start --no-dashboard + +# 4. Verify session name +tmux list-sessions | grep orchestrator +# Should see: ao-test-orchestrator (NOT undefined-orchestrator) + +# 5. Cleanup +tmux kill-session -t ao-test-orchestrator +rm -rf /tmp/ao-test ~/.agent-orchestrator/ao-test-orchestrator +``` + +## Testing Dashboard Integration + +To verify the dashboard shows the "orchestrator terminal" button: + +```bash +# 1. Setup test environment (as above) +ao init --auto && ao start + +# 2. Check dashboard +curl -s http://localhost:3000 | grep "orchestrator terminal" +# Should find the button text + +# 3. Or manually visit +open http://localhost:3000 +# Look for "orchestrator terminal" button in top-right header +``` + +## Common Test Failures + +### Test 1 fails: "Session name contains 'undefined'" + +**Cause:** The `sessionPrefix` fallback bug is present in `start.ts` + +**Fix:** Check these locations in `packages/cli/src/commands/start.ts`: +- Line ~177: `const sessionId = \`\${project.sessionPrefix || projectId}-orchestrator\`` +- Line ~369: Same fix in the `stop` command + +### Test 2 fails: "Session not found in tmux" + +**Cause:** The tmux session creation failed but was not reported + +**Debug:** +```bash +# Check tmux sessions +tmux list-sessions + +# Check for error in metadata +ls -la ~/.agent-orchestrator/ + +# Try manual session creation +cd /tmp/ao-test +tmux new-session -d -s test-session +``` + +### Test 3 fails: "Metadata file not found" + +**Cause:** Metadata write failed or wrong path + +**Debug:** +```bash +# Check data directory exists +ls -la ~/.agent-orchestrator/ + +# Check permissions +ls -ld ~/.agent-orchestrator/ + +# Check config dataDir setting +grep dataDir agent-orchestrator.yaml +``` + +### Test 5 fails: "Session name doesn't end with '-orchestrator'" + +**Cause:** Orchestrator session naming logic changed + +**Fix:** Check session ID construction in `start.ts` command + +## Integration Tests + +For testing as part of CI/CD: + +```bash +# Run all tests including orchestrator setup +pnpm test +./scripts/test-orchestrator-setup.sh + +# Or add to package.json +{ + "scripts": { + "test:e2e": "./scripts/test-orchestrator-setup.sh" + } +} +``` + +## Files Changed in the Fix + +- `packages/cli/src/commands/start.ts` - Added `|| projectId` fallback +- `packages/web/src/components/Dashboard.tsx` - Added helpful tooltip for missing orchestrator +- `scripts/test-orchestrator-setup.sh` - Created automated test +- `QUICKTEST.md` - Created manual test guide +- `README.md` - Added testing section + +## Related Documentation + +- [QUICKTEST.md](../QUICKTEST.md) - Detailed manual testing procedures +- [SETUP.md](../SETUP.md) - User setup guide +- [TROUBLESHOOTING.md](../TROUBLESHOOTING.md) - Common issues +- [CLAUDE.md](../CLAUDE.md) - Code conventions (includes test command) diff --git a/CLAUDE.md b/CLAUDE.md index 9319ee225..58914a6a2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -145,6 +145,9 @@ pnpm test # run tests # Before committing pnpm lint && pnpm typecheck + +# Quick verification test (orchestrator setup) +./scripts/test-orchestrator-setup.sh # 5-second end-to-end test ``` ## Development Workflow diff --git a/QUICKTEST.md b/QUICKTEST.md new file mode 100644 index 000000000..a62352047 --- /dev/null +++ b/QUICKTEST.md @@ -0,0 +1,240 @@ +# Quick Verification Test + +**Purpose:** Verify the orchestrator setup works correctly from scratch (5 minutes). + +## Prerequisites + +```bash +# Ensure you've built the project +cd /path/to/agent-orchestrator +pnpm install && pnpm build +``` + +## Test Setup (Fresh Environment) + +```bash +# 1. Create a test directory +mkdir -p /tmp/ao-test +cd /tmp/ao-test + +# 2. Initialize a fake git repo (ao init needs this) +git init +git remote add origin git@github.com:test/test-repo.git + +# 3. Run ao init with auto mode +ao init --auto + +# This creates agent-orchestrator.yaml with: +# - Project ID: ao-test (from directory name) +# - NO sessionPrefix defined (this is the key test case) +# - Default settings +``` + +## Expected Config + +The generated `agent-orchestrator.yaml` should look like: + +```yaml +dataDir: ~/.agent-orchestrator +worktreeDir: ~/.worktrees +port: 3000 + +defaults: + runtime: tmux + agent: claude-code + workspace: worktree + notifiers: [desktop] + +projects: + ao-test: + repo: test/test-repo + path: /tmp/ao-test + defaultBranch: main + # NOTE: No sessionPrefix - this is correct! +``` + +## Run the Test + +```bash +# Start the orchestrator (creates session + dashboard) +ao start --no-dashboard + +# Expected output: +# ✔ Orchestrator prompt ready +# ✔ CLAUDE.local.md configured +# ✔ Agent hooks configured +# ✔ Orchestrator session created +# +# Orchestrator: tmux attach -t ao-test-orchestrator +``` + +## Verify Success + +### 1. Check tmux session exists +```bash +tmux list-sessions | grep orchestrator + +# Expected: ao-test-orchestrator: 1 windows (created ...) +``` + +### 2. Check metadata file exists +```bash +cat ~/.agent-orchestrator/ao-test-orchestrator + +# Expected: +# worktree=/tmp/ao-test +# branch=main +# status=working +# project=ao-test +# runtimeHandle={"id":"ao-test-orchestrator",...} +``` + +### 3. Check session name pattern +```bash +# The session ID should be: {projectId}-orchestrator +# NOT: undefined-orchestrator (this was the bug) + +tmux list-sessions | grep -E '^ao-test-orchestrator:' +echo $? # Should print: 0 (success) +``` + +### 4. Test dashboard detection +```bash +# Start dashboard +ao start & +sleep 5 + +# Check page source for orchestrator button +curl -s http://localhost:3000 | grep -q "orchestrator terminal" +echo $? # Should print: 0 (button found) +``` + +## Expected Results + +✅ **PASS**: Session created as `ao-test-orchestrator` +✅ **PASS**: Metadata file exists +✅ **PASS**: Dashboard shows "orchestrator terminal" button + +❌ **FAIL**: Session created as `undefined-orchestrator` +❌ **FAIL**: No metadata file +❌ **FAIL**: Button missing or shows "No orchestrator session" tooltip + +## Cleanup + +```bash +# Kill test session +tmux kill-session -t ao-test-orchestrator + +# Remove test data +rm -rf /tmp/ao-test +rm -rf ~/.agent-orchestrator/ao-test-orchestrator + +# Stop dashboard +pkill -f "next dev.*3000" || pkill -f "node.*next-server" +``` + +## Common Issues + +### Session named `undefined-orchestrator` +**Problem:** The `sessionPrefix` fallback bug exists in `start.ts` +**Fix:** Ensure lines 177 and 369 use `${project.sessionPrefix || projectId}` + +### Button still missing +**Problem:** Dashboard filtering logic not finding orchestrator session +**Fix:** Check `packages/web/src/app/page.tsx` line 23 uses `.endsWith("-orchestrator")` + +### Metadata file missing +**Problem:** Session created but metadata write failed +**Check:** Permissions on `~/.agent-orchestrator/` directory + +## Testing on Real Project + +For testing on an existing project (like integrator): + +```bash +cd ~/secondary_checkouts/integrator-1 + +# Create test config (use unique port to avoid conflicts) +cat > agent-orchestrator.yaml << 'EOF' +dataDir: ~/.agent-orchestrator-test +port: 4567 +projects: + integrator: + repo: ComposioHQ/integrator + path: ~/secondary_checkouts/integrator-1 + defaultBranch: next + # NOTE: No sessionPrefix - tests the fallback +EOF + +# Run test +ao start --no-dashboard + +# Verify +tmux list-sessions | grep integrator-orchestrator +cat ~/.agent-orchestrator-test/integrator-orchestrator + +# Cleanup +tmux kill-session -t integrator-orchestrator +rm -rf ~/.agent-orchestrator-test +rm agent-orchestrator.yaml +``` + +## Automated Test Script + +```bash +#!/bin/bash +# test-orchestrator-setup.sh + +set -e + +echo "🧪 Testing orchestrator setup..." + +# Setup +TEST_DIR="/tmp/ao-quicktest-$$" +mkdir -p "$TEST_DIR" +cd "$TEST_DIR" +git init +git remote add origin git@github.com:test/test.git + +# Init +ao init --auto --output agent-orchestrator.yaml + +# Start (capture session name from output) +OUTPUT=$(ao start --no-dashboard 2>&1) +echo "$OUTPUT" + +# Extract session name +SESSION=$(echo "$OUTPUT" | grep -o 'tmux attach -t [^ ]*' | awk '{print $4}') +echo "Session created: $SESSION" + +# Verify session name is NOT undefined +if [[ "$SESSION" == *"undefined"* ]]; then + echo "❌ FAIL: Session name contains 'undefined'" + exit 1 +fi + +# Verify session exists +if ! tmux has-session -t "$SESSION" 2>/dev/null; then + echo "❌ FAIL: Session not found in tmux" + exit 1 +fi + +# Verify metadata +if [[ ! -f ~/.agent-orchestrator/"$SESSION" ]]; then + echo "❌ FAIL: Metadata file not found" + exit 1 +fi + +echo "✅ PASS: Orchestrator setup working correctly!" + +# Cleanup +tmux kill-session -t "$SESSION" +cd / +rm -rf "$TEST_DIR" +``` + +Save as `scripts/test-orchestrator-setup.sh` and run: +```bash +chmod +x scripts/test-orchestrator-setup.sh +./scripts/test-orchestrator-setup.sh +``` diff --git a/README.md b/README.md index 2b249fa64..c5f6d1ab1 100644 --- a/README.md +++ b/README.md @@ -205,6 +205,21 @@ packages/ terminal-*/ - Terminal plugins ``` +## Testing + +### Quick Verification Test + +Verify your setup is working correctly: + +```bash +# Automated test (5 seconds) +./scripts/test-orchestrator-setup.sh + +# Manual test - see QUICKTEST.md for details +``` + +See [QUICKTEST.md](QUICKTEST.md) for detailed test procedures and troubleshooting. + ## Troubleshooting See [TROUBLESHOOTING.md](TROUBLESHOOTING.md) for common issues and solutions. diff --git a/packages/cli/src/commands/start.ts b/packages/cli/src/commands/start.ts index 09eb1105c..735d0393c 100644 --- a/packages/cli/src/commands/start.ts +++ b/packages/cli/src/commands/start.ts @@ -174,7 +174,7 @@ export function registerStart(program: Command): void { try { const config = loadConfig(); const { projectId, project } = resolveProject(config, projectArg); - const sessionId = `${project.sessionPrefix}-orchestrator`; + const sessionId = `${project.sessionPrefix || projectId}-orchestrator`; const port = config.port; console.log(chalk.bold(`\nStarting orchestrator for ${chalk.cyan(project.name)}\n`)); @@ -365,8 +365,8 @@ export function registerStop(program: Command): void { .action(async (projectArg?: string) => { try { const config = loadConfig(); - const { projectId: _projectId, project } = resolveProject(config, projectArg); - const sessionId = `${project.sessionPrefix}-orchestrator`; + const { projectId, project } = resolveProject(config, projectArg); + const sessionId = `${project.sessionPrefix || projectId}-orchestrator`; const port = config.port; console.log(chalk.bold(`\nStopping orchestrator for ${chalk.cyan(project.name)}\n`)); diff --git a/packages/web/src/components/Dashboard.tsx b/packages/web/src/components/Dashboard.tsx index fbee50abc..a6b10fcdb 100644 --- a/packages/web/src/components/Dashboard.tsx +++ b/packages/web/src/components/Dashboard.tsx @@ -88,13 +88,29 @@ export function Dashboard({ sessions, stats, orchestratorId }: DashboardProps) { Agent Orchestrator
- {orchestratorId && ( + {orchestratorId ? ( orchestrator terminal + ) : ( +
+ orchestrator terminal + + + No orchestrator session + + Run: + + ao start + + +
)}
diff --git a/scripts/test-orchestrator-setup.sh b/scripts/test-orchestrator-setup.sh new file mode 100755 index 000000000..f9e5b45b8 --- /dev/null +++ b/scripts/test-orchestrator-setup.sh @@ -0,0 +1,110 @@ +#!/bin/bash +# Test orchestrator setup from scratch +# Usage: ./scripts/test-orchestrator-setup.sh + +set -e + +echo "🧪 Testing orchestrator setup..." +echo "" + +# Setup +TEST_DIR="/tmp/ao-quicktest-$$" +echo "📁 Creating test directory: $TEST_DIR" +mkdir -p "$TEST_DIR" +cd "$TEST_DIR" + +# Initialize fake git repo (required by ao init) +git init > /dev/null 2>&1 +git remote add origin git@github.com:test/test-repo.git + +# Run ao init +echo "⚙️ Running: ao init --auto" +ao init --auto --output agent-orchestrator.yaml > /dev/null 2>&1 + +# Verify config was created +if [[ ! -f agent-orchestrator.yaml ]]; then + echo "❌ FAIL: agent-orchestrator.yaml not created" + exit 1 +fi + +# Check that sessionPrefix is NOT set (this is the test case) +if grep -q "sessionPrefix:" agent-orchestrator.yaml; then + echo "⚠️ Note: sessionPrefix is set in config (expected to be unset for this test)" +fi + +# Start orchestrator +echo "🚀 Running: ao start --no-dashboard" +OUTPUT=$(ao start --no-dashboard 2>&1) + +# Extract session name from output +SESSION=$(echo "$OUTPUT" | grep -o 'tmux attach -t [^ ]*' | awk '{print $4}') + +if [[ -z "$SESSION" ]]; then + echo "❌ FAIL: Could not extract session name from ao start output" + echo "$OUTPUT" + exit 1 +fi + +echo "📋 Session created: $SESSION" +echo "" + +# Test 1: Verify session name is NOT undefined +echo "Test 1: Session name should not contain 'undefined'" +if [[ "$SESSION" == *"undefined"* ]]; then + echo "❌ FAIL: Session name contains 'undefined': $SESSION" + echo "This indicates the sessionPrefix fallback bug is present" + exit 1 +fi +echo "✅ PASS: Session name is valid: $SESSION" + +# Test 2: Verify session exists in tmux +echo "Test 2: Session should exist in tmux" +if ! tmux has-session -t "$SESSION" 2>/dev/null; then + echo "❌ FAIL: Session not found in tmux" + tmux list-sessions + exit 1 +fi +echo "✅ PASS: Session exists in tmux" + +# Test 3: Verify metadata file exists +echo "Test 3: Metadata file should exist" +METADATA_FILE=~/.agent-orchestrator/"$SESSION" +if [[ ! -f "$METADATA_FILE" ]]; then + echo "❌ FAIL: Metadata file not found: $METADATA_FILE" + ls -la ~/.agent-orchestrator/ || echo "Directory not found" + exit 1 +fi +echo "✅ PASS: Metadata file exists" + +# Test 4: Verify metadata content +echo "Test 4: Metadata should contain correct project" +if ! grep -q "project=" "$METADATA_FILE"; then + echo "❌ FAIL: Metadata missing 'project' field" + cat "$METADATA_FILE" + exit 1 +fi +echo "✅ PASS: Metadata is valid" + +# Test 5: Verify session name matches pattern {projectId}-orchestrator +echo "Test 5: Session name should match pattern {projectId}-orchestrator" +if [[ ! "$SESSION" =~ -orchestrator$ ]]; then + echo "❌ FAIL: Session name doesn't end with '-orchestrator': $SESSION" + exit 1 +fi +echo "✅ PASS: Session name matches expected pattern" + +echo "" +echo "🎉 All tests passed!" +echo "" +echo "Session: $SESSION" +echo "Metadata: $METADATA_FILE" +echo "" + +# Cleanup +echo "🧹 Cleaning up..." +tmux kill-session -t "$SESSION" 2>/dev/null || true +rm -f "$METADATA_FILE" +cd / +rm -rf "$TEST_DIR" + +echo "✨ Test complete!"