bkinnightskytw
/
agent-orchestrator
mirror of https://github.com/ComposioHQ/agent-orchestrator.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
agent-orchestrator/tests/integration
History
Prateek 105f26253e fix: strengthen integration test to catch WebSocket bug 
**Problem:**
- Test only warned about missing WebSocket servers (didn't fail)
- Wouldn't have caught the bug your friend encountered

**Solution:**
1. Make WebSocket server check mandatory (fail if not responding)
2. Add retry logic (wait up to 10s for servers to start)
3. Add end-to-end test: verify orchestrator terminal page loads

**Now catches:**
✅ Config discovery bug (API endpoint fails if config not loaded)
✅ WebSocket server bug (test fails if servers not started)
✅ Terminal page bug (test fails if session page doesn't load)

All bugs from your friend's experience are now covered!

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 		2026-02-18 03:27:12 +05:30
..
Dockerfile feat: add Docker-based onboarding integration test 2026-02-18 03:27:07 +05:30
README.md feat: add Docker-based onboarding integration test 2026-02-18 03:27:07 +05:30
docker-compose.yml feat: add Docker-based onboarding integration test 2026-02-18 03:27:07 +05:30
onboarding-test.sh fix: strengthen integration test to catch WebSocket bug 2026-02-18 03:27:12 +05:30
run-test.sh feat: add Docker-based onboarding integration test 2026-02-18 03:27:07 +05:30

README.md

Integration Tests

Docker-based integration tests that simulate real-world onboarding and usage scenarios.

Onboarding Test

Tests the complete first-time developer experience:

  1. Fresh environment (no prior setup)
  2. Repository access
  3. ./scripts/setup.sh (install + build + link)
  4. ao init (configuration)
  5. ao start (dashboard + services)
  6. Verify dashboard responds
  7. Verify API endpoints work
  8. Measure total onboarding time

Run Locally

cd tests/integration
docker-compose up --build

Run Manually (for debugging)

# Build image
docker-compose build

# Run container interactively
docker-compose run --rm onboarding-test /bin/bash

# Inside container, run test manually
/workspace/agent-orchestrator/tests/integration/onboarding-test.sh

Expected Results

  • Total time: < 180s (3 minutes) for fresh install
  • All steps pass: ✓ green checkmarks
  • Dashboard accessible: http://localhost:4000

Metrics Tracked

  • Setup time (pnpm install + build)
  • Dashboard startup time
  • API response time
  • Total onboarding time

CI Integration

The test runs automatically on:

  • Pull requests (when packages/ or scripts/ change)
  • Pushes to main branch

View results: .github/workflows/onboarding-test.yml

Future Tests

  • Multi-project test: Verify multiple projects in one config
  • Port conflict test: Verify auto-port-selection
  • Config discovery test: Verify config found from subdirectories
  • Session spawn test: Verify ao spawn creates working sessions
  • Terminal test: Verify WebSocket terminal connection
  • Upgrade test: Verify upgrade from previous version

Adding New Tests

  1. Create test script in tests/integration/<test-name>.sh
  2. Add to docker-compose.yml as new service
  3. Add to .github/workflows/integration-tests.yml
  4. Document expected behavior and metrics

Troubleshooting

Test fails at setup step:

  • Check scripts/setup.sh for errors
  • Verify pnpm is installed in Dockerfile

Dashboard doesn't start:

  • Check port 4000 is not already in use
  • Verify pnpm dev runs all required services
  • Check dashboard logs: docker logs ao-onboarding-test

Test times out:

  • Increase timeout in docker-compose.yml
  • Check if any step hangs (tmux, npm link, etc.)