Lesson 9.4 — The AI application lifecycle

The recipe is not the dish — the taste test is. For an LLM app, evals are the taste test.

TL;DR: Building on top of a foundation model is the normal software lifecycle plus two new properties it was never designed for — non-determinism (same input, different output) and a data feedback loop (production traces become tomorrow's eval set). Because you can't assertEqual a probabilistic system, evals are the oracle — the heartbeat that tells you whether a change is an improvement or a regression ¹⁸. Teams that fail almost always fail for the same reason: no robust evaluation system ¹.

This lesson maps the lifecycle you'll later automate with agents. We explain the loop here; we don't build automation. This is also not about training models — that's Lesson 9.3. This is about apps built on models you don't train.

ELI5: the improvisational kitchen

A brilliant chef who never cooks a dish the same way twice forces you to taste-test continuously instead of trusting the recipe.

You hire a genius chef who improvises — every plate is a little different (non-determinism). You can't hand them a rigid recipe and walk away. So you taste-test constantly: a few standard dishes you re-taste after every change (regression / golden-set evals), trained tasters who agree with your head chef (human ↔ LLM-judge alignment) ², and comment cards from real diners (online A/B + user feedback). Health-and-safety rules stop a prankster slipping a fake order to the chef (prompt-injection defense) ⁵. Each night you read the comment cards and adjust tomorrow's prep (the improvement loop) ¹. The taste test — not the recipe — keeps the food good.

The lifecycle as a loop

It's the software loop with an eval heartbeat in the middle and a trace-mining feedback edge that turns production back into the next iteration.

flowchart LR
    A["0 · Scope<br/>use-case &amp; success metrics"] --> B["1 · Design<br/>prompt → RAG → tools → agent"]
    B --> E(["2 · EVALUATE<br/>the heartbeat /<br/>the oracle"])
    E --> S["3 · Safety &amp; guardrails<br/>red-team, injection defense"]
    S --> D["4 · Deploy<br/>gated / canary release"]
    D --> M["5 · Monitor<br/>hallucination · drift · cost · latency · abuse · feedback"]
    M --> I["6 · Improve<br/>mine traces → fix / fine-tune"]
    I -. "traces become the next eval set" .-> A
    E -. "regression gate on every change" .-> B
    M -. "production traces feed evals" .-> E

The classic SDLC (Lesson 9.2) runs scope → build → test → deploy → maintain. This lifecycle keeps all of that and adds the dashed edges: evals gate every change (not a one-time benchmark) and production traces flow back into both the eval set and the next design pass ¹⁷. The loop never "finishes" — it's a flywheel.

🧠 Test Yourself: Name the two properties an LLM app has that the classic software lifecycle never had to model.

Answer

Non-determinism (same input can yield different output, so you can't unit-test with assertEqual) and a data feedback loop (production traces become the next eval set / fine-tune data). Together they're why evals must act as CI and trace-mining is a first-class stage, not just bug-fixing ¹⁷.

Each stage: WHAT / WHY / HOW

Seven stages; the early ones decide complexity, evaluation is the oracle that makes the rest measurable, and the back half (monitor → improve) is where most of the real cost lives.

#	Stage	WHAT	WHY	HOW
0	Scope	Decide if it needs an LLM at all; define measurable "good"	Without a measurable target, every later eval is ungrounded; over-scoping to agents is the most common waste 3	Start with the simplest thing that works; reserve agents for open-ended tasks you can't hardcode 34
1	Design	Pick the least-complex architecture: prompt → augmented LLM (retrieval/tools/memory) → workflow → agent	Complexity compounds errors, cost, and latency; agents amplify failures 3	Build on the "augmented LLM"; add complexity only when it demonstrably helps; invest in the tool interface as much as the prompt 3; mirror RAG/agent failure modes 7
2	Evaluate	A measurement harness built before you scale	The only oracle for a non-deterministic system; powers debugging, regression detection, and the data flywheel 1	Three levels (below) + golden/offline sets + regression evals on every change 18
3	Safety	Defend against prompt injection, leakage, excessive agency	RAG and tool-use open injection paths; neither RAG nor fine-tuning fully fixes injection 5	Defense-in-depth: least-privilege tools, output handling, human approval for high-risk actions, adversarial red-teaming 56
4	Deploy	Ship behind controls	Non-determinism makes staged exposure essential	Sandboxed testing + guardrails before autonomy 3; canary/gated rollout tied to eval gates
5	Monitor	Observe quality, hallucination, drift, cost, latency, abuse, feedback in prod	Offline evals never cover the real input distribution; quality silently drifts as models/data change 7	Trace-level observability, online metrics, capture implicit + explicit user feedback 7; track confabulation/hallucination and integrity risks 6
6	Improve	Mine traces → fix prompts/RAG/tools or fine-tune → re-run regression evals	Closes the loop; production data is the highest-value eval and training data 1	The "eval flywheel" — eval infra is reused for data curation and fine-tuning 1

🧠 Test Yourself: Anthropic and OpenAI both open their agent guides with the same advice. What is it, and why?

Answer

"Start simple" — use the least-complex pattern (a plain prompt or an augmented LLM) and only reach for agents on open-ended tasks you can't hardcode. Agents compound errors, cost, and latency, so the flashiest pattern is usually the wrong first move ³⁴.

The heartbeat: eval-driven development

Evals are the oracle for a system with no deterministic right answer; they come in three levels and run continuously, not once.

Evals convert "vibes-based" prompt tweaking into engineering — without a harness you literally cannot tell whether a change helped ¹. Husain's three levels ¹:

flowchart TB
    L1["L1 · Unit-test assertions<br/>cheap code-based checks, run constantly"] --> L2["L2 · Human + LLM-as-judge<br/>over logged traces"]
    L2 --> L3["L3 · A/B · online eval<br/>real users = ground truth"]
    GOLD["Golden / offline set<br/>+ regression evals on every change"] -.-> L1
    L2 -. "continuously check judge vs human labels" .-> ALIGN["judge ↔ human alignment"]

• L1 — assertions. Cheap, code-based unit tests (does it return valid JSON? call the right tool? avoid a banned phrase?), run on every change ¹.
• L2 — human + LLM-as-judge over traces. A judge model scores logged traces. GPT-4-class judges reach >80% agreement with humans ² — credible, but they carry position, verbosity, and self-enhancement bias ², so the judge needs its own eval against human labels, checked continuously ¹². "Remove all friction from looking at data" — error analysis on real traces is the actual work ¹.
• L3 — A/B / online. Real-user comparison; the only ground truth for user-facing quality ¹.

Around all three: a pinned golden set and regression evals so an edit can't silently break untouched cases — the LLM-app equivalent of shipping without tests ¹⁸.

🧠 Test Yourself: Your LLM-as-judge agrees with your gold set 92% of the time. Safe to trust it blindly?

Answer

No. Judges carry position/verbosity/self-enhancement bias ²; an unvalidated judge launders vibes into fake rigor. You must keep validating it against fresh human labels — the judge needs its own ongoing eval, and offline agreement ≠ online ground truth ¹².

What curricula get wrong

The common failures cluster at the eval and the back half of the loop — the parts tutorials skip.

Pitfall	The reality
Eval as a one-time benchmark	It's a living system; the hard work is ongoing error analysis on real traces + judge↔human alignment 1
Teaching only the build phase	The cost center is monitoring, drift, and the improvement loop — the back half 7
Worshipping LLM-as-judge	It's a biased instrument that needs its own validation against human labels 2
No regression / eval CI gate	Prompt edits silently regress untouched cases without pinned golden sets 1
Security as "add a guardrail"	RAG and fine-tuning don't fully fix injection; the fix is architectural — least privilege, output handling, human-in-the-loop 5
Jumping straight to agents	Both labs say start simple; agents compound errors, cost, latency 34
Ignoring cost/latency/quality	Quality is rarely free, and offline distributions never match production traffic 7

Your turn (exercise)

Take any LLM feature you've used or built (a support bot, a summarizer, a code helper). On one page, map it through all seven stages. For the Evaluate stage specifically: write 5 L1 assertions (concrete, code-checkable — e.g. "output is valid JSON", "never returns an empty answer", "cites at least one source"), then pick one that an LLM-as-judge would handle better than code (e.g. "answer is faithful to the retrieved context") and describe how you'd validate that judge against human labels ¹². Finally, draw the feedback edge: which production signal from Monitor would you mine to grow your eval set ⁷?

Where agents fit (teaser)

Every stage in the loop is a job a future module hands to agents — but mapping comes before automating.

Each box in the lifecycle diagram is something a later phase will automate: an agent that drafts eval cases from production traces, a judge-agent that scores them, a red-team agent that probes for injection, a monitor that opens an issue on quality drift. The point of Phase 9 is to map these loops first — you can only safely put an agent on a stage once you know what "good" looks like and have the eval gate to catch it when the agent gets it wrong.

---

← Lesson 9.3 · Phase 9 home · → Check your understanding

---

1. Your AI Product Needs Evals — Hamel Husain ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩

2. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., NeurIPS 2023 ↩↩↩↩↩↩↩↩

3. Building Effective Agents — Anthropic ↩↩↩↩↩↩↩

4. A Practical Guide to Building Agents — OpenAI ↩↩↩

5. LLM01:2025 Prompt Injection — OWASP GenAI Security Project, Top 10 for LLM Applications 2025 (direct/indirect prompt injection; mitigations: least-privilege, output handling, human approval, adversarial testing) ↩↩↩↩

6. Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1) — NIST ↩↩

7. AI Engineering: Building Applications with Foundation Models — Chip Huyen, O'Reilly ↩↩↩↩↩↩↩↩

8. Evaluating model performance — OpenAI Platform docs (write evals to measure an app against expectations, run them as test inputs, and catch prompt regressions before shipping) ↩↩↩