Lesson 8.4 — Observability before autonomy

You may not run agents you cannot trace. An unattended run you can't replay isn't a process — it's a hope.

TL;DR: Repo telemetry (lint firings, flaky tests — 6.4) tells you a rule fired; it never tells you why the agent tripped it. That needs run telemetry: a trace of the session itself — prompts, tool calls, sub-agent spans, tokens, cost, duration, terminal state. This isn't hypothetical: Claude Code exports exactly that over OpenTelemetry ¹², and OpenAI's Agents SDK traces model calls, tool calls, and handoffs by default ³. Enforce it the same way as any other non-negotiable — a PreToolUse gate, not a prose request ⁴ — and mine the traces afterward to find where agents retry, which skills misfire, and what a run actually cost ¹.

ELI5: the black box everyone forgets to plug in

A flight recorder that isn't wired up doesn't help you after the crash — it only helps if it was running before it.

You wouldn't run a fleet of aircraft without a flight recorder, and you wouldn't wait until after something goes wrong to ask "was it even recording?" Autonomous coding agents are the same: the moment you stop watching every turn (Phase 6's fan-out, 6.4's harness), the only way to know what actually happened is a trace that was already running. Bolting on observability after an incident is too late — it has to be a precondition, not a incident-response afterthought.

Repo telemetry vs. run telemetry

Lesson 6.4's pillar 4 covers repo signals; this is the other half — the session itself.

	Repo telemetry (6.4)	Run telemetry (this lesson)
Answers	which rule fired, which test flaked	why the agent tripped it, turn by turn
Granularity	aggregated across the repo	one session, one prompt, one tool call
Source	lint/CI/test exit codes	the agent's own prompts, tool calls, spans

Both matter; neither substitutes for the other. A lint-rule dashboard tells you that agents keep importing upward — it can't tell you what prompt led to that, or whether it was one agent retrying five times or five different agents making the same mistake once each.

What a trace must capture

Prompts, tool calls, sub-agent spans, tokens, cost, duration, and terminal state — and this is already shipping, not a wishlist.

flowchart LR
    S["agent session"] --> M["METRICS (time series)<br/>session count · cost · tokens<br/>active time · lines of code"]
    S --> E["EVENTS (per-action)<br/>user_prompt · assistant_response<br/>api_request · tool_result · tool_decision"]
    S -. "beta: enhanced telemetry" .-> SP["SPANS<br/>one prompt → its API calls →<br/>its tool executions, linked"]
    M --> OTEL[("OpenTelemetry<br/>(CNCF standard)")]
    E --> OTEL
    SP --> OTEL
    OTEL --> BACKEND["your backend<br/>(dashboards · Jaeger/Datadog · alerts)"]

Claude Code exports this natively over OpenTelemetry: metrics like claude_code.session.count, claude_code.cost.usage, and claude_code.token.usage; per-action events (user_prompt, tool_result, api_request) that reconstruct exactly what happened; and, in beta, distributed spans that link one prompt to every API call and tool execution it triggered, filterable by a single prompt.id ¹. OpenAI's Agents SDK ships the equivalent by default: LLM generations, tool calls, and handoffs are each wrapped in their own span, viewable in a traces dashboard with no extra setup ³. Either way, the underlying wire format is the same open standard — OpenTelemetry, the CNCF's observability specification for traces, metrics, and logs ².

🧠 Test Yourself: Your CI dashboard shows a lint rule firing five times this week (6.4). What can't you answer from that signal alone?

Answer

Why the agent kept tripping it — that lives in the session, not the repo. Run-level tracing (prompts, tool calls, spans) answers it; repo telemetry only tells you a rule fired ¹².

Attribution: per developer, per session, per run

A trace with no owner is just noise at scale — attribution is what makes cost and quality controllable, not merely visible.

Every metric and event Claude Code exports carries standard attributes — session ID, user ID, organization ID, version, entrypoint — plus optional custom attributes you set via OTEL_RESOURCE_ATTRIBUTES ¹. That's the difference between "our agents cost $X this month" (a number nobody can act on) and "session abc123, user jdoe, cost $4.20, 3 tool retries" (a number you can go debug). Tag a custom resource attribute with whatever unit of work you track upstream — a ticket ID, a PR number — and cost/failure attribution rolls up against it for free, no separate bookkeeping system needed.

The enforcement move: a precondition, not a request

Same move as 6.4: don't ask a team to "please enable tracing" — gate the first tool call on it.

Prose asking people to turn on tracing doesn't survive a deadline. The durable version, per 6.4's own harness-engineering thesis applied to the harness's own instrumentation: a PreToolUse hook that checks tracing is configured and blocks the first tool call until it is, with an explicit CI bypass. Not SessionStart — Claude Code's hooks reference is explicit that SessionStart cannot block; only PreToolUse (among others) can refuse to proceed on a non-zero exit ⁴. This repo's own git-safety.sh/secret-scan.sh already demonstrate the pattern — the same mechanism, aimed at a different precondition.

🧠 Test Yourself: Why does the tracing precondition need to be a PreToolUse hook and not a SessionStart hook?

Answer

SessionStart can only inject context or show a message on exit — it can't stop the session from proceeding. PreToolUse can block via a non-zero exit, so it's the only one of the two that can actually enforce a precondition ⁴.

What traces are for, afterward

Not just an audit trail — the raw material for improving the harness and for building the eval set.

A trace archive isn't just for post-incident forensics. Filter events by prompt.id to reconstruct every API call and tool execution triggered by one prompt ¹ — that's how you find where agents loop-and-retry, which skill's description isn't triggering when it should, and which prompts are burning tokens for no payoff. And because production traces are exactly what feeds the improvement loop in an eval-driven system, this is where observability and evals meet: mined traces become tomorrow's eval set (9.4's "traces become the next eval set" flywheel). Skip the tracing precondition and you don't just lose debuggability today — you lose the raw material the eval flywheel runs on.

Worked example

Same symptom, two harnesses — one can explain it, one can't.

An agent's PR gets merged, but review comments keep saying "this took three tries to get right."

• ❌ Untraced: nobody can say why — was it an ambiguous prompt, a flaky tool, a skill that didn't fire when it should have? The only evidence is whatever's still visible in the chat transcript, if anyone kept it open.
• ✅ Traced: filter events by that session's prompt.id. The trace shows two failed tool_result events for the same lint command before it passed, and the user_prompt event shows the original ask never mentioned the lint config the agent kept violating. Now you know exactly what to fix — the prompt was under-specified, not the agent incompetent.

Your turn (exercise)

If you use Claude Code, turn on telemetry for one real session:

export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317   # or your backend
claude

Run a real task, then find the claude_code.user_prompt event and every tool_result/api_request event sharing its prompt.id. Could you have answered "why did this take three turns?" from that trace alone? If yes, you've just proven the precondition is worth gating on. If no, note what's missing — that gap is the next bumper to add.

---

← Lesson 8.3 · Phase 8 home · → Check your understanding

---

1. Monitoring — Anthropic (Claude Code docs; OpenTelemetry metrics, per-action events, standard + custom resource attributes, and beta distributed tracing keyed by prompt.id) ↩↩↩↩↩↩

2. What is OpenTelemetry? — CNCF ↩↩↩

3. Tracing — OpenAI (Agents SDK; default tracing of generations, tool calls, and handoffs) ↩↩

4. Hooks reference — Anthropic (Claude Code docs; SessionStart cannot block, PreToolUse can via exit 2) ↩↩↩