Canvases

A right-hand rail in the session detail page where agents render structured output as interactive artifacts. Pluggable enough that anyone can add a canvas in 3 minutes by dropping a JSON file. Constrained enough that no third-party JavaScript runs in the supervisor dashboard.

1. The problem

An AO session is a tmux pane with a coding agent inside it. Today, when an agent emits structured output — a diff, a test summary, a cost breakdown, an error trace, a plan — it scrolls past in the terminal and the human supervisor has to know it happened, scroll back, and parse the unstructured shell output. The richer the agent's output, the worse this gets:

• A test runner produces 200 passing rows the user has to scroll past to find the 3 failures.
• A code reviewer produces a 1500-token markdown report that wraps in the terminal and is impossible to navigate.
• A cost tracker prints token counts on every call; by the end of the session you can't reconstruct the total.
• The agent's diff is the highest-value artifact and it's buried under build chatter.

Cursor solved this with canvases — interactive panels that render alongside the chat. Canvases brings the same idea to AO sessions, shaped for an open-source plugin ecosystem.

2. Architecture

Why files instead of an SDK

• Zero coupling. Agents are CLIs running in their own processes. They don't link AO libraries. A file write from a shell hook works for every agent — Claude Code, Codex, Aider, OpenCode, future ones — without an integration step.
• Matches the existing pattern. AO already uses {workspacePath}/.ao/activity.jsonl for agent-to-dashboard activity signaling. Canvases reuses the same convention.
• Survives restarts. Agent crashes, dashboard restarts, even ao stop — the JSON file is still there for the next read.

Why pull instead of push

The dashboard already has a WebSocket mux (MuxProvider.tsx) and an SSE channel for sessions. v0.1 deliberately doesn't use either — a single 5-second REST poll is the simplest thing that works, has no failure modes (no reconnect logic, no event ordering, no message replay), and can be replaced with a push channel in v0.3 without changing the on-disk format. Visibility-aware: pauses when the tab is hidden.

3. Schema and what renders

Four built-in renderer types. The schema is a Zod discriminated union on type; all variants share the same envelope.

Common envelope

{
  "version": 1,
  "id": "a-z0-9-, 1-64 chars",        // "core-" prefix is reserved
  "type": "markdown" | "diff" | "table" | "stats",
  "title": "Display title",
  "createdAt": "ISO 8601",
  "updatedAt": "ISO 8601",           // sort key, descending
  "source": "optional label",
  "payload": { /* per-type, see below */ }
}

Sample stats payload

{
  "version": 1,
  "id": "test-run",
  "type": "stats",
  "title": "vitest",
  "createdAt": "2026-05-06T10:00:00Z",
  "updatedAt": "2026-05-06T10:02:14Z",
  "source": "agent",
  "payload": {
    "metrics": [
      { "label": "Pass",    "value": 847, "tone": "good" },
      { "label": "Fail",    "value": 3,   "tone": "bad"  },
      { "label": "Skipped", "value": 12,  "tone": "warn" },
      { "label": "Duration","value": "2m 14s" }
    ]
  }
}

4. The synthesized core-git-diff canvas

Every session gets a free diff canvas without any agent integration. Core runs:

1. git merge-base origin/<default> HEAD (falls back to local <default>) — this is the diff anchor.
2. git diff <merge-base> — captures committed and uncommitted changes against the working tree.
3. git ls-files --others --exclude-standard — finds untracked files.
4. For each untracked file, git diff --no-index NUL/&dev/null <file> — synthesizes "fully added" hunks.
5. Concatenate, parse the unified diff, render via CanvasDiff.

The canvas is reserved under id core-git-diff. Agent file canvases that try to use the core- prefix are rejected by the reader, so an agent can't shadow this trusted artifact.

5. How to extend it

Three tiers of effort, smallest first.

interface CanvasProducer {
  listCanvases(session: Session, project: ProjectConfig): Promise<CanvasArtifact[]>;
}

@aoagents/ao-plugin-canvas-flamegraph/
├── package.json          # exports: { ".": "./dist/index.js", "./renderer": "./dist/renderer.js" }
├── dist/
│   ├── index.js          # manifest + Zod payload schema (Node-safe)
│   └── renderer.js       # React component, default export (browser-safe)

plugin discovery
  → Node schema registry        // core uses to validate at runtime
  → generated web renderer map  // next bundles statically
  → optional generated TS union // web/internal ergonomics only

// Core types stay strict for built-ins, with a single fallback for plugin types.
type BuiltInCanvasArtifact =
  | { type: "markdown"; payload: { markdown: string }; ... }
  | { type: "diff"; payload: { files: CanvasDiffFile[] }; ... }
  | { type: "table"; payload: { ... }; ... }
  | { type: "stats"; payload: { ... }; ... };

type PluginCanvasArtifact = {
  id: string;
  type: string;       // any canvasType registered by a plugin
  title: string;
  createdAt: string;
  updatedAt: string;
  source?: string;
  payload: unknown;   // plugin's Zod schema validates in core before serving
};

type CanvasArtifact = BuiltInCanvasArtifact | PluginCanvasArtifact;

5b. Validation rules

Canvases that fail validation are dropped silently and logged. Rules core enforces today:

• version must be 1.
• type must be a known CanvasType (one of the four built-ins, or — in v0.4 — a registered plugin canvas type).
• id must match [a-z0-9][a-z0-9-]{0,63}.
• The core- id prefix is reserved for canvases synthesized by AO core. File canvases using this prefix are dropped at the reader.
• Total serialized size capped at 256 KB per canvas.
• Per-session count capped at 32 canvases (oldest by updatedAt evicted).
• Payload must structurally match the type's Zod schema.
• For diff canvases specifically: max 200 files / 50 hunks per file / 1000 lines per hunk (added in pass 13 to prevent DOM blowup).

5c. Canvas-id namespacing

Two separate concerns, often conflated:

• id — the filesystem identity of a canvas file ({workspacePath}/.ao/canvases/{id}.json) and its stable UI identity for animations / collapse state. Reserved prefix: core- for canvases synthesized by AO core (core-git-diff, etc). File canvases using core- are dropped by the reader.
• canvasType — the renderer dispatch key. Maps to a built-in renderer or a plugin renderer. Does not reserve ids. A Claude agent can legitimately produce a type: "flamegraph" canvas with id auth-perf-2026-05-06 without being the flamegraph plugin author. Producers and renderers are separate concerns.

Recommended (not enforced): plugins suggest a producer-scoped id prefix to avoid wild collision — e.g. flamegraph plugin's example agent emits flamegraph-* ids. Enforced: only core-* reservation and per-session id collision (last-writer-wins by updatedAt).

6. Built with paranoia

The feature went through 13 codex review passes. 18 distinct corner-case fixes landed pre-merge.

Security

Correctness

Robustness

7. File map

8. Try it locally

# 1. Start the dashboard
pnpm dev

# 2. Open any session detail page in your browser

# 3. From a terminal, drop a JSON file in the session's worktree
WS=$(jq -r .worktree ~/.agent-orchestrator/projects/<your-project>/sessions/<session-id>.json)
mkdir -p "$WS/.ao/canvases"

cat > "$WS/.ao/canvases/hello.json" <<'EOF'
{
  "version": 1,
  "id": "hello",
  "type": "stats",
  "title": "Demo",
  "createdAt": "2026-05-06T00:00:00Z",
  "updatedAt": "2026-05-06T00:00:00Z",
  "payload": {
    "metrics": [
      { "label": "Tests", "value": 42, "tone": "good" },
      { "label": "Failures", "value": 0, "tone": "neutral" }
    ]
  }
}
EOF

Within 5 seconds the canvas appears. Edit the file, save, watch it update.

9. Roadmap

10. Why no html canvas type?

The most-asked design question — answered honestly here so reviewers don't have to ask.

Two reasons, both load-bearing.

Trust hierarchy inversion

Agents are the lowest-trust surface in the system. They emit content on every poll cycle, and large parts of that content come from LLMs that can be prompt-injected. If an agent ingests a malicious URL, summarizes a poisoned document, or just hits a jailbreak, its next "render this nice summary" call could include attacker-controlled HTML. Putting that HTML into the same-origin dashboard means the attacker now has:

• Read access to every same-origin auth token / cookie / localStorage value.
• Permission to call every API the user can call (kill sessions, restore them, modify config).
• The ability to inject scripts that survive across navigation.
• Keylogging on every page the dashboard renders.

That's the same blast radius v0.4 renderer plugins have — but renderer plugins clear npm install review once. An agent emits a fresh canvas every poll. The trust gap is enormous.

HTML sanitization is a leaky abstraction

DOMPurify, sanitize-html, etc. are honest attempts but every year ships new bypasses — <svg> foreign objects, <math> payloads, mutation-XSS via clipboard, CSS exfiltration via attr(). A sanitizer-based HTML canvas is a 90% solution that fails open on the 10%, and the 10% is "execute arbitrary JS in your supervisor". Even GitHub, GitLab, and Slack treat HTML rendering as a multi-quarter security investment, not a feature you slot in next to markdown.

What the existing types cover

The legitimate "I need raw HTML" path — sandboxed iframe

An iframe with sandbox="allow-scripts" (no allow-same-origin) runs in a null origin — it can't reach your auth tokens or call APIs. Add a postMessage protocol for size negotiation and limited event-out, document the trust trade-off explicitly, and you've got a real HTML surface that's safe by construction.

But — per codex's pass-13 advice — half-baked iframe support is worse than no sandbox. Sandbox protocols have to be designed end-to-end (CSP headers, postMessage validation, focus management, accessibility, viewport sizing) before they ship. v0.4 renderer plugins is the bigger unlock; the iframe canvas comes after, only if a real use case shows up.

11. Known trade-offs in v0.1

Honest acknowledgment of where the v0.1 surface has real limits:

• 5-second poll latency. Visible-tab polls every 5s, hidden tabs pause entirely. For fast-updating canvases (live test runs, streaming logs) this feels sluggish. The fix is the v0.3 mux topic — core publishes on file write, the rail subscribes per session — which gives sub-second updates without polling. Shortening the interval to 1s in v0.1 just shifts cost (10× the API calls for the same end state); the right call is to wait for push.
• Plugin renderers (v0.4) require an AO rebuild. Next.js can't bundle dynamic plugin paths — only static imports. npm install @aoagents/ao-plugin-canvas-flamegraph alone won't make the renderer appear; the AO build has to regenerate the static-import barrel. The ao plugin install CLI ships in v0.4 (not deferred) and auto-rebuilds, so users see one command, not three. The rebuild itself is unavoidable; the CLI hides it.
• Markdown renderer is intentionally limited. Headings, bold, italic, code, fenced blocks, lists, and (as of post-merge) safe HTTP/HTTPS links work. Images, HTML pass-through, tables, nested lists, and footnotes don't. Tables have a dedicated table canvas type; rich layouts get a renderer plugin in v0.4. Agents that produce rich reports will hit this ceiling on the markdown canvas — that's deliberate, but real.
• Read-only surface in v0.1. No write-back, no action buttons, no canvas-driven mutation of session state. This is where the highest-leverage UX would land — approve a diff, retry a failed test, kill a stuck session — but it's a different feature with a different trust model (every interactive control needs explicit consent gating, CSRF protection, auth scope review). Explorable post-v0.4 once the renderer plugin path is live and we have a real consent UI.

12. Deliberate non-goals

• Remote-loaded JS at runtime. No URL-fetched code, no agent-emitted React, no on-the-fly module fetching. v0.4 plugins are discovered at AO startup from node_modules (Node loads each plugin's schema entrypoint) and renderers are bundled at web build time via a generated barrel of static imports. That's the OSS extensibility line: trust at install, not at runtime.
• Image rendering in markdown canvases. Image URLs would need an allowlist policy (which origins are trusted? what about data: with embedded SVG that runs JS via onload?). The risk-to-value ratio is bad; v0.5+ might add a curated { src, alt } form once an actual use case shows up. Until then: agents that want to render an image should attach the bytes to a session and reference it from a `markdown` canvas with workspace-relative path support (also v0.5+).
• HTML pass-through in markdown. Even sanitized HTML opens the supervisor to mutation-XSS, CSS exfil, and same-origin token theft on every poll. See section 10.
• A 9th plugin slot for canvas producers. Considered and rejected — producing a canvas is session output, not infrastructure. Existing plugins opt-in via CanvasProducer. (v0.4's canvas-renderer slot is different: that's UI infrastructure, where a slot makes sense.)