Lesson 2.6 — From memory loop to knowledge base (the LLM Wiki)

Stop re-deriving what you already learned. Compile it once; let it compound.

TL;DR: RAG re-derives understanding on every query. An LLM Wiki has the agent compile knowledge once into a curated, interlinked markdown wiki that compounds as you add sources — and below ~50–100k tokens that wiki beats retrieval. Your Lesson 2.5 memory loop is a baby version of it.¹²

The shift: from retrieving to compiling

RAG looks things up at query time; an LLM Wiki builds a durable, growing knowledge artifact.

The default move with a pile of documents is RAG: embed everything, retrieve the top-k chunks at query time, answer. But that re-derives the same understanding on every query — nothing accumulates. Karpathy's LLM Wiki flips it: the agent maintains a persistent, interlinked wiki of markdown pages between you and your raw sources. Add a source once; the agent folds it into the wiki (a summary page, cross-links, a log entry). Knowledge is compiled once and compounds — one research topic of his grew to ~100 pages / ~400k words, none typed by hand.¹²

flowchart TD
    SRC[("① RAW SOURCES (immutable)<br/>papers · notes · URLs you curate")]
    SCHEMA["③ SCHEMA — AGENTS.md / a skill<br/>tells the agent HOW to behave"]
    WIKI[("② THE WIKI (agent-written)<br/>concept pages · <code>[[wiki-links]]</code><br/>index.md · append-only log.md")]
    SRC -->|"INGEST: read → summarize →<br/>update ~10–15 linked pages → log"| WIKI
    WIKI -->|"QUERY: answer from the wiki + cite;<br/>file good answers back as pages"| WIKI
    WIKI -->|"LINT: fix contradictions, stale<br/>claims, orphans, missing links"| WIKI
    SCHEMA -. governs .-> WIKI

RAG vs context — when each wins

Small, dense, high-signal corpora belong in context; RAG is for scale past the window.

Karpathy's argument: below ~50–100k tokens, context wins decisively — a personal knowledge base often fits in 5–20k tokens, giving 100% retrieval reliability, no embedding pipeline, and global reasoning over the whole corpus (the model sees everything at once, not k disconnected chunks). RAG still wins past that scale.¹ This is the same lesson as the rest of Phase 2 — the window is the scarce resource — applied to knowledge: pre-compile a small, dense wiki that fits, instead of retrieving fragments forever.³

🧠 Test Yourself: You're researching one focused topic and your notes total ~15k tokens. RAG or an LLM Wiki — and why?

Answer

The wiki. At 15k tokens the whole thing fits in context, so you get 100% recall + global reasoning with zero embedding/retrieval machinery. RAG's retrieve-top-k only earns its complexity once the corpus can't fit in the window (past ~50–100k tokens).¹

Past the window: navigate, don't embed

When the wiki outgrows the window, the agent reads an index and pulls only the pages it needs — direct corpus interaction, not embedding-RAG.

So what happens once the wiki does grow past the window? You still don't reach for embeddings — the agent navigates. Load a small index.md, then read the specific pages it needs on demand and follow their [[links]]. Researchers call this Direct Corpus Interaction (DCI): the agent searches the raw corpus with plain file tools (read, grep), with no embedding model or vector index, and it can beat semantic retrieval on multi-step lookups.⁵⁶ It's the same move Anthropic calls just-in-time / agentic context — keep lightweight identifiers (file paths) and "navigate its environment and retrieve files just-in-time," rather than pre-loading everything.³ Rule of thumb: inline while the wiki is small, navigate once it's big — exactly what this repo's load-memory hook does (inline under a byte budget, index-only above it).

You already run a baby version

Our .agent/memory/ loop is a primitive LLM Wiki — same bones, missing the curation.

LLM Wiki	Our memory loop (Lesson 2.5)
interlinked .md pages, [[wiki-links]]	.agent/memory/**/*.md (semantic paths) — links missing
append-only log.md	session-log.md ✅
index.md catalog	missing
ingest / query / lint (LLM curates)	deterministic merge only — no curation
the schema (AGENTS.md)	AGENTS.md + memory conventions ✅

So the "upgrade" is concrete: add an index.md, real [[links]], source attribution, and an LLM ingest/query/lint skill (the scaffolder emits one — Phase 2's second lockstep artifact).

The honest part: keep a floor, cap the cost

Don't put an unbounded LLM in the automatic path — keep a deterministic inbox + a hard cost ceiling.

LLM curation is powerful but costs tokens and can hallucinate or run away (a lesson worth taking seriously). So the safe shape: keep the deterministic capture as an un-loseable inbox (a failed or capped curation never loses a note), run curation on demand and hard-capped (a total-agent + token ceiling, like any bounded loop), and keep the read path (re-injecting the wiki) deterministic and cheap. The wiki is the primary knowledge model; determinism is the floor under it.

The deeper idea: ship the idea, not the code

In the agent era, a good spec is a distributable program.

Karpathy didn't ship an app — he shipped an "idea file": a prompt spec you paste into your agent, which then builds the wiki for you.⁴ That's Software 3.0 applied to distribution — and it's exactly how this repo's skills work (SKILL.md is the spec; the agent executes it). The artifact you share is increasingly the idea, legibly written, not the implementation.

Your turn (exercise)

Take 3–4 notes you've saved this week (or 3 .agent/memory/ files). By hand, do one ingest pass: write a one-line summary per page, add a [[link]] between any two that relate, and draft a 3-line index.md listing them. That's the whole LLM-Wiki loop, minus the automation — and it shows you what "compiled, interlinked knowledge" buys over a flat folder of notes.

---

← Lesson 2.5 · Phase 2 home · → Check your understanding

---

1. LLM Wiki (idea file) — Andrej Karpathy (primary source: the canonical artifact + its three-layer / ingest-query-lint architecture) ↩↩↩↩

2. LLM Knowledge Bases — Andrej Karpathy (the origin post: "manipulating knowledge" rather than code) ↩↩

3. Effective context engineering for AI agents — Anthropic (the window as the scarce resource; the "just-in-time" / agentic retrieval pattern — agents navigate with glob/grep and load files at runtime instead of pre-loading) ↩↩

4. On sharing the idea, not the code — Andrej Karpathy (the "idea file" framing) ↩

5. Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction — arXiv, May 2026 (defines DCI: agents query the raw corpus directly with terminal tools — grep, file reads — with no embedding model, vector index, or retrieval API) ↩

6. GrepSeek: Training Search Agents for Direct Corpus Interaction — arXiv 2026 (a trained, shell-based DCI search agent over large text corpora) ↩