# Observability Signals

This document describes runtime observability emitted by Agent Orchestrator.

## Goals

- Structured, low-noise telemetry for session lifecycle and operator workflows.
- Correlated traces across core services, API routes, SSE, and websocket terminal health.
- Clear failure reasons and current health surfaces for fast diagnosis.

## Emission Model

- **Structured logs**: JSON lines on stderr when enabled by config or `AO_OBSERVABILITY_STDERR`.
  - Supported levels: `debug`, `info`, `warn`, `error`.
  - Default level: `warn` (production-safe, avoids high-volume info logs).
  - Default stderr mirroring: disabled.
  - Runtime env vars override YAML:
    - `AO_LOG_LEVEL=info`
    - `AO_OBSERVABILITY_STDERR=1`
- **Durable snapshots**: process-local JSON snapshots under:
  - `~/.agent-orchestrator/{config-hash}-observability/processes/*.json`
- **Aggregated view**: merged by project via:
  - `GET /api/observability`

## Correlation

- API routes accept `x-correlation-id`; when absent, AO generates one.
- Responses include `x-correlation-id` for traceability from UI or CLI.
- SSE snapshots include `correlationId` and `emittedAt`.

## Metrics

Counters are emitted per project and operation:

- `spawn` (`session.spawn`)
- `restore` (`session.restore`)
- `kill` (`session.kill`)
- `claim_pr` (`session.claim_pr`)
- `cleanup` (`session.cleanup`)
- `send` (`session.send`)
- `lifecycle_poll` (`lifecycle.poll`, `lifecycle.transition`)
- `lifecycle_poll` (`lifecycle.merge_cleanup.completed`) — auto-cleanup ran after a PR was detected as merged; session runtime + worktree + metadata were torn down
- `lifecycle_poll` (`lifecycle.merge_cleanup.deferred`) — auto-cleanup is waiting for the agent to idle (or for the `mergeCleanupIdleGraceMs` window to elapse) before tearing down
- `lifecycle_poll` (`lifecycle.merge_cleanup.failed`) — auto-cleanup threw during `sessionManager.kill()`; the session stays in `merged` so the next poll retries
- `notification_delivery` (`notification.deliver`) — per-target notifier dispatch success/failure, with event ID/type, priority, project/session IDs, target reference, plugin name, and delivery method
- `api_request` (web API routes)
- `sse_connect`, `sse_snapshot`, `sse_disconnect`
- `websocket_connect`, `websocket_disconnect`, `websocket_error` (websocket servers)

Each metric counter tracks:

- `total`, `success`, `failure`
- `lastAt`, `lastSuccessAt`, `lastFailureAt`
- `lastFailureReason`

## Trace Fields

Recent traces keep operation-level diagnostics:

- `id`
- `timestamp`
- `component`
- `operation`
- `outcome`
- `correlationId`
- `projectId`
- `sessionId`
- `path`
- `reason`
- `durationMs`
- `data`

## Health Surfaces

Health records provide current status and failure context per surface:

- `surface` (for example: `lifecycle.worker`, `sse.events`)
- `status` (`ok`, `warn`, `error`)
- `updatedAt`
- `component`
- `projectId`
- `correlationId`
- `reason`
- `details`

## Operator-Facing Diagnostics

- **Dashboard**: use **Copy debug info** in the hero toolbar (desktop) to copy `/api/observability` plus page URL, project scope, and correlation id to the clipboard for issue reports. The observability banner shows overall status, SSE stream state, last correlation id, and latest failure reason.
- **API**: `/api/observability` returns merged per-project diagnostics (`overallStatus`, metrics, health, recent traces, session state).
- **Terminal websocket health**: `/health` endpoints include active sessions and websocket/terminal health counters with last error/disconnect reasons.
- **Notifications**: successful deliveries update `notification_delivery` metrics and per-target health; failed/missing targets also appear in `ao events`.

## Rollout Notes

1. Deploy with default `observability.logLevel: warn` and `observability.stderr: false` to avoid noisy logs.
2. Validate `/api/observability` and dashboard banner in a canary environment.
3. If deeper triage is needed, temporarily raise `AO_LOG_LEVEL=info` (or `debug`) and set `AO_OBSERVABILITY_STDERR=1`, then revert to defaults.
4. Monitor `lastFailureReason` and surface-level `reason` fields before enabling broader rollout.