# Observability Signals

This document describes runtime observability emitted by Agent Orchestrator.

## Goals

- Structured, low-noise telemetry for session lifecycle and operator workflows.
- Correlated traces across core services, API routes, SSE, and websocket terminal health.
- Clear failure reasons and current health surfaces for fast diagnosis.

## Emission Model

- **Structured logs**: JSON lines on stderr, controlled by `AO_LOG_LEVEL`.
  - Supported levels: `debug`, `info`, `warn`, `error`.
  - Default level: `warn` (production-safe, avoids high-volume info logs).
- **Durable snapshots**: process-local JSON snapshots under:
  - `~/.agent-orchestrator/{config-hash}-observability/processes/*.json`
- **Aggregated view**: merged by project via:
  - `GET /api/observability`

## Correlation

- API routes accept `x-correlation-id`; when absent, AO generates one.
- Responses include `x-correlation-id` for traceability from UI or CLI.
- SSE snapshots include `correlationId` and `emittedAt`.

## Metrics

Counters are emitted per project and operation:

- `spawn` (`session.spawn`)
- `restore` (`session.restore`)
- `kill` (`session.kill`)
- `claim_pr` (`session.claim_pr`)
- `cleanup` (`session.cleanup`)
- `send` (`session.send`)
- `lifecycle_poll` (`lifecycle.poll`, `lifecycle.transition`)
- `api_request` (web API routes)
- `sse_connect`, `sse_snapshot`, `sse_disconnect`
- `websocket_connect`, `websocket_disconnect`, `websocket_error` (websocket servers)

Each metric counter tracks:

- `total`, `success`, `failure`
- `lastAt`, `lastSuccessAt`, `lastFailureAt`
- `lastFailureReason`

## Trace Fields

Recent traces keep operation-level diagnostics:

- `id`
- `timestamp`
- `component`
- `operation`
- `outcome`
- `correlationId`
- `projectId`
- `sessionId`
- `path`
- `reason`
- `durationMs`
- `data`

## Health Surfaces

Health records provide current status and failure context per surface:

- `surface` (for example: `lifecycle.worker`, `sse.events`)
- `status` (`ok`, `warn`, `error`)
- `updatedAt`
- `component`
- `projectId`
- `correlationId`
- `reason`
- `details`

## Operator-Facing Diagnostics

- **Dashboard**: observability banner shows overall status, SSE stream state, last correlation id, and latest failure reason.
- **API**: `/api/observability` returns merged per-project diagnostics (`overallStatus`, metrics, health, recent traces, session state).
- **Terminal websocket health**: `/health` endpoints include active sessions and websocket/terminal health counters with last error/disconnect reasons.

## Rollout Notes

1. Deploy with default `AO_LOG_LEVEL=warn` to avoid noisy logs.
2. Validate `/api/observability` and dashboard banner in a canary environment.
3. If deeper triage is needed, temporarily raise `AO_LOG_LEVEL=info` (or `debug`), then revert to `warn`.
4. Monitor `lastFailureReason` and surface-level `reason` fields before enabling broader rollout.