In a demo, that's invisible. One call costs fractions of a cent and nobody's watching the clock. In production serving real traffic, ignoring this data is the difference between a system with predictable unit economics and one that quietly bleeds money on patterns nobody's measuring.

Where the Data Actually Comes From

// message_start — input tokens
{ type: "message_start", message: { usage: { input_tokens: 342 } } }

// message_delta — output tokens + stop reason
{ type: "message_delta", delta: { stop_reason: "end_turn" }, usage: { output_tokens: 156 } }

Token counts combined with model pricing give cost. Timestamp deltas between stream start and stream close give latency. That's the entire data source — no separate API call, no additional request, just reading fields that were already streaming past.

The Wrapper Pattern

The design goal was tracking usage without touching any calling code in the earlier modules. The solution is a generator that wraps the raw stream, intercepts the fields it needs, and yields every chunk through unchanged:

async function* wrappedStream() {
  for await (const chunk of rawStream) {
    if (chunk.type === "message_start") {
      model = chunk.message.model;
      inputTokens = chunk.message.usage.input_tokens;
    }
    if (chunk.type === "message_delta") {
      outputTokens = chunk.usage.output_tokens;
      stopReason = chunk.delta.stop_reason ?? "";
    }
    yield chunk; // transparent — caller sees the exact same stream
  }
  // stream done — compute and persist
}

The consumer of wrappedStream() has no idea tracking is happening. It receives the identical sequence of chunks it would have gotten from the raw Anthropic stream. This pattern generalizes well beyond usage tracking — any time you want to observe a stream without changing what downstream code does with it, wrap and re-yield is the shape to reach for.

Cost Calculation — and the Asymmetry That Changes How You Think About It

export function calculateCost(model: string, inputTokens: number, outputTokens: number): number {
  const pricing = MODEL_PRICING[model];
  const inputCost = (inputTokens / 1_000_000) * pricing.input;
  const outputCost = (outputTokens / 1_000_000) * pricing.output;
  return inputCost + outputCost;
}

The detail that matters most here: input and output tokens are not priced the same. On Sonnet, output tokens cost roughly 5x more than input tokens (\(15 vs \)3 per million). This inverts an intuition that's easy to carry over from naive token counting — a long, detailed prompt is cheap. A long, verbose response is expensive. Optimizing prompt brevity to save cost is usually optimizing the wrong side of the ledger.

Three Calls, Three Different Lessons

Running three representative query types and logging the results surfaced patterns that weren't visible from token counts alone:

Label                        In    Out    Total   Cost       Latency   Stop
usage-demo/structured        722   110    832     $0.003816  1993ms    tool_use
usage-demo/long                40  1024   1064    $0.0155    22408ms   max_tokens
usage-demo/short                16    64     80    $0.001008 2079ms    end_turn

The Schema Tax

722 input tokens for a prompt that was a single sentence. That gap is almost entirely the tool schema — property names, descriptions, enums, the required array. Every field definition in a tool's input_schema gets tokenized and sent on every single call, not just the first one. For any system making heavy use of forced tool use (the structured output pattern from 04-structured-output), the schema itself is a meaningful and recurring cost line item that's invisible if you're only watching output tokens.

The Most Expensive Call Didn't Even Finish

stop_reason: max_tokens on the long query — this is the truncation failure mode from the error handling module showing up in cost data. The response was cut off mid-generation. $0.0155 and 22 seconds spent on an answer the user never actually got to read in full.

This is the sharpest lesson in the whole dataset: an incomplete response can be the most expensive call in the system. It's not just a UX failure — it's wasted spend with literally nothing delivered. Any production cost monitoring needs to cross-reference cost against stop_reason, not just against token count, or this failure mode is invisible in the aggregate numbers.

The Efficient Baseline Isn't as Cheap as It Looks

16 input tokens, 64 output tokens, \(0.001, 2 seconds. In raw token terms this used 10x fewer tokens than the structured call. In cost terms it was less than 4x cheaper (\)0.001 vs $0.0038) — because the output tokens in the short response carry 5x the cost weight of input tokens. Token count and dollar cost diverge in exactly the direction the input/output pricing asymmetry predicts.

What the Numbers Actually Tell You

• Token count alone doesn't tell you cost. You have to look at the input/output split. structured moved 10x more total tokens than short but cost less than 4x more, because the bulk of its tokens were cheap input tokens (the schema), not expensive output tokens.

• An incomplete response can be the most expensive call you make. max_tokens truncation is a cost problem, not just a correctness problem.

• The structured output tax is the schema, not the generation. If the goal is reducing cost on tool-heavy calls, the lever is shortening tool descriptions and property definitions — not reducing how much the model generates.

• Always check stop_reason alongside cost. A cheap call that didn't finish is strictly worse than a slightly more expensive call that did — and that comparison is invisible if you're only tracking dollars.

Why This Belongs in the Same System as Evals

This module pairs naturally with 07-evals. Evals answer "is the agent's output good enough?" Cost and latency tracking answers "is the agent's output good enough to justify what it costs?" The earlier evals work already surfaced that the agent runs ~3.7 seconds slower than basic RAG per query (12.3s vs 8.6s) because of the extra tool-execution round trip. Layering cost data on top of that completes the picture: is the accuracy gain from live tool data worth the additional dollars and latency, for this specific use case?

That's a question every FDE and applied AI role eventually has to answer for a real customer, and it's a question that's impossible to answer without instrumentation like this in place from the start — not bolted on after a surprise bill.

Without explicit error handling, any of these silently breaks the user experience or crashes the server. The goal isn't to prevent errors — they're unavoidable — it's to build a system that degrades gracefully, retries intelligently, and gives the user something useful when things go wrong.

The Three Categories

Most engineers handle one category of LLM errors and miss the other two. All three are distinct and require different responses.

Transient errors are temporary infrastructure failures: API timeouts, rate limits, momentary network drops. The characteristic of a transient error is that the same request, retried after a short wait, has a reasonable chance of succeeding. Examples: HTTP 429, 503, 502, connection reset.

Permanent errors are request or auth failures that won't resolve on retry. Retrying a 401 wastes API budget and adds latency to a failure that's already final. Examples: invalid API key (401), forbidden (403), malformed request payload (400).

Output errors are the category most tutorials omit entirely. The API call succeeded — HTTP 200, stream closed cleanly — but the output is wrong. Truncated responses from hitting max_tokens. Malformed JSON when using prompt-based structured output. Tool execution failures. These require different handling than transport errors because there's nothing to retry at the transport layer; the problem is in the content.

The Retry Utility

The core principle: the retry decision is determined by error type, not retry count. Count limits how many times you try. Type determines whether you should try at all.

export async function withRetry<T>(
  fn: () => Promise<T>,
  options: RetryOptions = {}
): Promise<T> {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      const isLastAttempt = attempt === maxAttempts;
      if (isLastAttempt || !shouldRetry(error, attempt)) {
        throw error; // permanent error or exhausted attempts — stop
      }
      await delay(exponentialBackoff(attempt)); // transient — wait and retry
    }
  }
}

shouldRetry checks isTransientError first. A 401 doesn't match any transient pattern, falls through, returns false, and throws immediately. A 429 matches, returns true, and gets retried after a delay. The separation of "should I retry at all" from "how many times have I retried" keeps both concerns independently testable.

Exponential Backoff With Jitter

Each retry waits longer than the last:

const baseDelay = Math.min(initialDelayMs * Math.pow(2, attempt - 1), maxDelayMs);
const jitter = Math.random() * 0.3 * baseDelay;
const delayMs = Math.round(baseDelay + jitter);

The jitter is the part that's easy to skip and worth not skipping. If multiple clients all hit a rate limit simultaneously and retry at identical intervals, they'll hammer the server again in lockstep. The synchronized spike is called a thundering herd — it can trigger the same rate limit that caused the original failure, creating a feedback loop. Adding a small random offset (here, up to 30% of the base delay) spreads retries across a window and breaks the synchronization.

For a single user this doesn't matter much. For any system running parallel requests or serving multiple users — which applies to every production LLM product — jitter is the difference between exponential backoff that actually relieves pressure and exponential backoff that just delays the next spike.

Detecting Permanent vs. Transient Errors

function isTransientError(error: unknown): boolean {
  const message = (error as Error).message.toLowerCase();
  return (
    message.includes("429") ||
    message.includes("rate limit") ||
    message.includes("timeout") ||
    message.includes("503") ||
    message.includes("502")
  );
}

This is intentionally conservative — it only matches patterns that are clearly transient. Anything that doesn't match (including 401s, 403s, and any unknown error) falls through to non-retriable. The safer default in error classification is to not retry, because a retry on a permanent error is strictly worse than throwing immediately: it adds latency and burns API budget on a request that was already failed.

Handling Output Errors

Truncated responses: check stop_reason

stop_reason === "max_tokens" means the model ran out of tokens mid-response. The HTTP call succeeded, the stream closed, but the content is incomplete. Without checking this field the truncated response gets returned silently.

if (response.stop_reason === "max_tokens") {
  // retry with a higher max_tokens budget
}

This is one of the more common silent failures in LLM systems. Long prompts with large context windows are particularly susceptible — the model can hit the token limit well into the response with no visible indication other than the output ending abruptly mid-sentence.

Malformed JSON: extract before giving up

When using prompt-based structured output (not forced tool use), the model occasionally wraps JSON in a greeting, explanation, or markdown block. Before failing, attempt extraction:

try {
  return JSON.parse(text);
} catch {
  const jsonMatch = text.match(/\{[\s\S]*\}/);
  if (jsonMatch) {
    return JSON.parse(jsonMatch[0]);
  }
  throw new Error("Could not extract valid JSON from response");
}

This is also the argument for the forced tool use pattern from 04-structured-output. The model's tool-calling pathway is specifically trained for schema compliance — it doesn't produce mixed content. Prompt-based JSON requires this kind of defensive extraction. If structured output is feeding downstream application logic, the reliability difference is significant enough to prefer forced tool use.

Tool failures: pass the error through tool_result

When a tool execution fails, the instinct is to catch the error and handle it in application code. The better approach is to send the error back to the model through the normal tool_result channel:

try {
  toolResultContent = await executeTool(name, input);
} catch (err) {
  toolResultContent = JSON.stringify({
    error: (err as Error).message,
    retry: true
  });
}

messages.push({
  role: "user",
  content: [{ type: "tool_result", tool_use_id: id, content: toolResultContent }]
});

The model now has the error in its conversation context. It can explain to the user what failed, suggest alternatives, or ask to retry with different parameters — rather than silently returning a degraded response while the application pretends the tool succeeded.

In a test with a deliberately flaky tool that returned intermittent failures, the model received two consecutive errors and accurately summarized both in its final response: "I tried to look that up twice and hit an error each time — you may want to try again in a moment." It had full context for the failure history because both error responses were in the messages array. The model handled the degraded path better than most application code would.

The Decision Tree

When something goes wrong in an LLM system, the question to answer in order:

Error occurred
├── Is it transient? (timeout, rate limit, 5xx)
│   └── Yes → retry with exponential backoff + jitter
└── No — is it permanent? (401, 403, malformed request)
    ├── Yes → throw immediately, don't retry
    └── No — is it an output error?
        ├── Truncated → retry with higher max_tokens
        ├── Malformed JSON → extract or retry with better prompt / switch to forced tool use
        └── Tool failure → pass error to model via tool_result, let model reason over it

The structure matters because collapsing these into a single retry loop — the most common mistake — means permanent errors get retried (wasting budget), output errors get treated as transport failures (wrong fix), and tool failures get silently swallowed (broken UX with no explanation).

This is part of a series on building applied AI systems. Start from the beginning with streaming →

That's a recommendation problem, and recommendation problems have a familiar failure mode: every signal you could rank by feels arbitrary, and it's tempting to bolt on a manually-maintained "priority" field to break ties. This phase is mostly about resisting that temptation and finding signals that are actually derivable from data already in the system.

The Schema Gap: You Can't Query for What Doesn't Exist

The first design problem wasn't about ranking — it was about a query that's structurally impossible with the existing schema.

One of the most useful recommendations is "topics you've never worked on." But problems.topic was a free-text enum value. If no problem with topic = "dynamic_programming" exists, there's no row to query — the topic is invisible to the database. You can't SELECT your way to a gap.

The fix is a dedicated topics table — id, name, slug, seeded upfront with the canonical topic list. Now "never worked on" becomes a standard pattern:

SELECT t.* FROM topics t
LEFT JOIN problems p ON p.topic_id = t.id
WHERE p.id IS NULL

This is a small schema change with an outsized effect: it makes an entire category of question answerable that simply wasn't representable before.

One-to-Many vs. Many-to-Many: Choosing the Simpler Model on Purpose

Some real interview problems genuinely span multiple topics — a problem might be both "graphs" and "dynamic programming." A many-to-many problem_topics join table would model that correctly.

It was considered and rejected for this phase. Most DSA problems map cleanly to one topic, and the join table would be solving a problem that doesn't exist yet in the actual data. problems.topic_id stays a single foreign key. If multi-topic problems become common enough to matter, the join table is a clean migration — but building it now would be future-proofing against a hypothetical.

This is the same instinct as the earlier SM-2 schema decision: model what the data actually needs, not what it could theoretically need.

Failing Loudly on Drift

generate_problem() looks up topic_id from the Topic enum value via SELECT id FROM topics WHERE slug = ?. If no match exists, it raises — it does not auto-create a row.

Auto-creation seems convenient, but it papers over a real bug: the Topic enum (in code) and the seeded topics table (in the database) are two independent sources of truth that can drift. If someone adds a new enum value and forgets to seed the corresponding row, auto-creation would silently produce a topic with no name and no metadata — a broken row that looks like it worked.

Raising immediately turns a silent data integrity problem into a loud, obvious one at the point where it's cheapest to fix: setup, not three weeks later when someone wonders why a topic has no display name.

Designing suggest_topics()

The function signature is deliberately simple:

def suggest_topics(limit: int = 3) -> list[TopicSuggestion]:

class TopicSuggestion(TypedDict):
    id: int
    name: str
    slug: str
    problems: list[ProblemRow]
    explanation: str

A topic name alone isn't actionable. Each suggestion bundles up to 2 concrete problems to work on and an explanation of why this topic was surfaced now. The explanation matters as much as the ranking — "dynamic programming: you haven't started this topic yet" is something a user can act on; a bare topic name in a list is not.

Three Signals, and the Honest Admission That the Order Is a Judgment Call

The function checks three signals in priority order:

1. Never worked on — zero rows in problems for this topic (the LEFT JOIN pattern from earlier)

2. Low average score — topics where average session score is lowest

3. Overdue for review — topics with problems past next_review_date, ranked by how overdue the worst one is

The ordering reflects a judgment call: coverage gaps matter most (you can't improve at something you haven't tried), then struggling topics, then scheduled maintenance. There's no objectively correct order here, and the notes say so directly — this is a reasonable default that can be revisited, not a derived truth.

That kind of explicit acknowledgment is worth more than it looks. A system that ranks by an unstated, unexamined priority order is harder to debug and harder to adjust later, because nobody documented why the order is what it is.

Rejecting the Manual Priority Field — Twice

For signal 1, an early idea was a manual "priority" or "interview frequency" column on topics — weighting "never worked on" suggestions by how often a topic actually shows up in interviews.

This was rejected for the same reason the SM-2 phase rejected manual scheduling fields: it requires ongoing manual upkeep and goes stale. For topics with genuinely no differentiating signal, an arbitrary tiebreaker — insertion order — is fine, because there's no real signal being discarded by using it. Adding a field that someone has to remember to update is worse than admitting the tiebreaker is arbitrary.

This is a recurring theme across the project: anywhere a manually-maintained field is proposed as a fix, the question is whether it's actually encoding a real signal or just deferring the discomfort of an arbitrary choice. Usually it's the latter.

When a Pure Function Should Stay Pure

Signal 1 topics — by definition — have zero rows in problems. So what goes in the problems: [] list for those suggestions?

The tempting fix is to call generate_problem() inline, backfilling a problem on the spot so the suggestion is immediately actionable. This was considered and rejected: it would turn suggest_topics() from a pure read function into something that calls the LLM and writes to the database as a side effect. That's a different function with a different contract — and a much bigger scope than "suggest topics."

The decision: return [] for signal 1 topics, and let the caller decide whether to call generate_problem() separately. Keeping read and write concerns in separate functions keeps both independently testable and keeps suggest_topics() fast and side-effect-free.

Extracting the Shared Pattern

Signals 2 and 3 both need the same thing: "up to 2 problems for this topic where next_review_date <= today, most overdue first." The first implementation was two nearly-identical inline loops — one per signal.

That duplication got extracted into _update_topics_with_problems(connection, topics, limit=2), a helper that mutates the TopicSuggestion list in place rather than returning a new one — since the dicts are already mutable references, an in-place mutation is simpler than threading a return value through.

The N+1 Query: An Accepted Tradeoff, Not an Oversight

Fetching due problems per-topic in a loop is N+1 — one query per topic, rather than a single windowed query using ROW_NUMBER() OVER (PARTITION BY topic_id ...).

For a personal tool where limit is capped at a handful of topics, the N+1 cost is negligible — a few extra queries on a SQLite database measured in milliseconds. The windowed-query version would be more "correct" in a scale sense, but it adds real SQL complexity for a benefit that doesn't exist at this scale.

This is worth stating explicitly because "N+1 queries" is often treated as an automatic red flag in code review. It's a red flag when N is large or growing. Here, N is bounded by limit and small by construction — the tradeoff is genuinely a non-issue, and saying so explicitly is more useful than either ignoring it or over-engineering around it.

Deduplication, Early Exit, and a Hashability Bug

A topic could theoretically satisfy multiple signals. The combination logic iterates signal 1 → 2 → 3, tracks seen ids in a set, skips duplicates, and stops once result reaches limit. The explanation shown is whichever signal first matched — the highest-priority one.

The first attempt at deduplication used dict.fromkeys([*signal1, *signal2, *signal3]), expecting it to dedupe while preserving order. This fails because TypedDicts are plain dicts at runtime, and dicts aren't hashable — they can't be dict keys. The fix was an explicit loop with a set of seen IDs.

There's also an early-exit optimization: if signal 1 alone already meets limit, signals 2 and 3 are skipped entirely. Signal 3 is skipped if len(signal1) + len(signal2) >= limit. This sum-based check is safe specifically because signals 1 and 2 are mutually exclusive by construction — signal 1 requires zero problems rows, signal 2 requires sessions rows (which require problems rows), so no topic can appear in both and no double-counting is possible.

Python Syntax Notes Worth Remembering

Dict spread for building suggestions:

{**dict(row), "problems": [], "explanation": "..."}

Same concept as {...obj} in JS — unpacks key/value pairs into a new dict, with later keys overriding on collision.

What's Still Open

avg_score and max_overdue are rounded for display but otherwise unvalidated — no handling for unusual values like negative deltas if a next_review_date ends up in the future but gets queried anyway. Signal 1's tiebreaker remains arbitrary insertion order, which is acceptable given there's no real signal to use instead. And main.py wiring plus the seed data reset — carried over from Phase 4 — are still outstanding before any of this is user-facing.

None of these are blocking. They're documented so the next phase starts with full context instead of rediscovering the same edge cases.

The question this module explores: can you render structured fields progressively as the model generates them, the same way streaming text renders word by word?

The short answer is yes, with a caveat worth understanding before you reach for it.

The Core Problem: Partial JSON Is Invalid JSON

When the model streams a structured output via input_json_delta, you receive the JSON object in chunks:

{"title": "Forw
{"title": "Forward Deployed Eng
{"title": "Forward Deployed Engineer", "company": "Anthr
{"title": "Forward Deployed Engineer", "company": "Anthropic", "requ

Every intermediate state throws on JSON.parse. The string is only valid JSON for exactly one moment: when the final closing } arrives at content_block_stop.

This is the fundamental tension: streaming gives you the data early, but JSON requires completeness to parse.

Two Approaches

Option 1: Buffer and parse on content_block_stop. Accumulate the full input_json_delta string across all deltas, wait for the block to close, parse once. This is what 02-tool-use already does for tool inputs — it's safe, simple, and correct. Nothing renders until the complete object is ready.

Option 2: Incremental regex extraction. After each delta, scan the accumulated string for completed key-value pairs and render them as they finish. Fields pop in one by one. The full parse at content_block_stop then replaces the partial state with the guaranteed-correct final version.

This module implements the second approach for a job description parser — a use case where title and company are genuinely useful to show before requirements finish streaming.

How the Incremental Extraction Works

After each input_json_delta, the accumulated JSON string is scanned with two patterns:

// Completed string fields: "key": "value", or "key": "value"}
const stringPattern = /"(\w+)"\s*:\s*"([^"\\]*(?:\\.[^"\\]*)*)"\s*[,}]/g;

// Completed array fields: "key": ["item1", "item2"]
const arrayPattern = /"(\w+)"\s*:\s*\[([^\]]*)\]/g;

The key signal is the closing delimiter — a quote followed by a comma or closing brace for strings, a closing ] for arrays. If those aren't present in the accumulated string yet, the field isn't complete and gets skipped until the next delta arrives.

This is a heuristic, not a real parser. It works reliably for flat objects with string and array fields. It breaks on nested objects, arrays of objects, or fields with unusual escaped characters. For anything more complex, a proper streaming JSON parser library is the right tool.

The Two-Phase Rendering Pattern

The stream sends two distinct event types to the frontend:

// During stream — partial state for progressive rendering
send({ type: "delta", partial_json: chunk.delta.partial_json, accumulated: accumulatedJson });

// On content_block_stop — full parse guaranteed to succeed
send({ type: "complete", data: parsed });

The frontend maintains both states and switches when the complete event arrives:

const display = completeData ?? partialData;

During streaming, partialData drives the render — fields appear as they complete. When complete fires, completeData takes over and streaming indicators disappear. Fields that the regex didn't extract incrementally just appear all at once in the final swap. The user never sees a blank field — at worst, a field is slightly delayed.

When This Is Actually Worth It

Not always. The incremental approach adds complexity — two event types, two state objects, a regex that needs to be maintained, a fallback for fields the regex misses. For backend pipelines, evals, or anything where a machine consumes the output, buffering until content_block_stop is simpler and equally correct.

The incremental approach earns its complexity when three conditions are true:

The object has fields that are useful before the whole thing is done. A job description with title, company, location, and requirements — title and company render almost immediately. By the time requirements finish streaming, the user has already read the header. Compare this to a single summary field: there's no useful partial state to show.

Generation is slow enough that the user would notice the wait. Short objects complete quickly. Long objects with multi-sentence fields are where the latency gap between streaming and buffering becomes perceptible.

The fields are independent enough that partial state makes sense to display. Rendering a partial requirements list while title is already visible is fine. Rendering half a structured error message is not.

For the JD parser in this module, all three conditions hold. That's the right use case. For most structured output use cases — grading rubrics, eval scores, tool inputs — content_block_stop is the right place to parse, and reaching for incremental extraction would be premature.

The Broader Pattern

This module sits at the intersection of two earlier ones: the streaming event model from 01-streaming and the forced tool use schema from 04-structured-output. The combination surfaces a real product engineering question — not "does this technically work" but "when is the complexity justified" — which is a different kind of problem than getting the API call right.

In production, the answer to that question depends on the specific object shape, the typical generation time, and what partial state actually means for your UI. Getting that judgment right matters more than being able to implement the pattern.

It was. The system wasn't broken. The eval was.

That distinction — system problem vs. eval problem — is the most important thing to understand about building reliable LLM systems. Without it, you end up "fixing" things that aren't broken while the actual issues go unmeasured.

What Evals Actually Are

An eval is a script that runs your AI system against a fixed set of known inputs and scores the outputs. The inputs are test cases. The scores tell you whether the system is getting better or worse as you make changes — prompts, retrieval strategy, model version, chunk size, similarity threshold.

Without evals, every change is a guess. With evals, you can say: "I changed the retrieval threshold from 0.3 to 0.4 and citation quality went from 6.2 to 7.8 on average." That's the difference between iterating on intuition and iterating on evidence.

The eval setup here runs against two systems: 03-rag-basic (fixed retrieval pipeline) and 06-tool-use-rag (agent that decides which tools to call). Ten cases each, scored across three dimensions: accuracy, citation quality, and confidence.

The Test Case Structure

Each test case captures what the system should do, not just what it should say:

type EvalCase = {
  id: string;
  question: string;
  expected_topics: string[];  // what the answer should cover
  should_use_tool?: string;   // which tool should fire, if any
  case_type: "retrieval" | "live_data";
};

The case_type field turned out to be the most important one — and it wasn't in the original design. More on that shortly.

The Scorer: LLM-as-Judge at the Eval Layer

Scoring each response manually would take hours across a full test suite. The solution is the same pattern from 04-structured-output: force the model to return a structured score using tool_choice.

type EvalScore = {
  accuracy: { score: number; reasoning: string };
  citation_quality: { score: number; reasoning: string };
  confidence: { score: number; reasoning: string };
  verdict: "pass" | "fail";
  overall_score: number;
  flags: string[];
};

LLM-as-judge isn't perfect, but it's scalable. Twenty cases run in a few minutes. The reasoning field in each dimension is as important as the score — it's what tells you whether the model is applying the criteria correctly or drifting.

The False Negative Problem

First run results before any fixes:

agent-03: How much PTO does alice.bob have remaining?       3/10
agent-06: What is alice.bob's role and department?          1.5/10
agent-09: How much PTO does alice.bob get given his tenure? 4/10

All three failures were employee lookups. The agent called the right tools, got the correct data, and returned accurate answers. The scorer flagged them for missing_citation and insufficient_sources — because it was looking for document-style citations, and these responses came from a mock HR API, not retrieved documents.

The system was behaving correctly. The eval was scoring it wrong.

This is a false negative: a working system failing on a bad criterion. It's more dangerous than a false positive because it actively misdirects you. If I'd trusted those numbers and started investigating why the agent's retrieval was weak, I would have been debugging a problem that didn't exist while the actual citation gap in retrieval responses went unaddressed.

The Fix: Case-Aware Scoring

The solution was passing case_type into the scorer so it could apply different citation criteria depending on the data source:

const citationCriteria = caseType === "live_data"
  ? "citation_quality: Does the answer accurately reflect the tool data? Explicit document citations are NOT expected for live data responses."
  : "citation_quality: Does the answer explicitly cite sources inline (e.g. [1], [2])?";

After the fix, both systems hit 100% pass rate. The agent's overall delta dropped from -1.5 to -0.8 — a real gap, but a much smaller and correctly measured one.

What the Final Numbers Actually Tell You

REPORT: Basic RAG (03)
─────────────────────
Overall score:    9.1/10   Pass rate: 100%   Avg latency: 8.6s
Accuracy:         9.3/10
Citation quality: 9.3/10
Confidence:       8.6/10

REPORT: Handbook Agent (06)
────────────────────────────
Overall score:    8.3/10   Pass rate: 100%   Avg latency: 12.3s
Accuracy:         9.8/10
Citation quality: 6.4/10
Confidence:       8.7/10

The remaining gap is entirely in citation quality (9.3 vs 6.4). Everything else is equal or the agent is better — accuracy is actually higher (9.3 vs 9.8) because it can pull live data that retrieval alone can't access. The citation gap is real and fixable with a targeted system prompt addition, the same fix applied to the RAG system already.

The insufficient_sources flags on two RAG cases are genuine — those questions aren't well covered by the document corpus. That's a knowledge gap in the data, not an eval artifact, and it's worth the distinction.

The Latency Tradeoff

Agent averages 12.3s vs RAG's 8.6s. The agent makes more API calls: query, tool executions, and final generation vs. query and generation. In production, the mitigations are caching tool results, parallelizing independent calls where possible, and setting latency budgets per query type. But the cost is real and has to be explicitly accounted for in system design — not discovered after launch.

The Broader Principle

The eval design is as important as the system being evaluated. Criteria that don't match the system's actual behavior produce scores that are meaningless at best and actively misleading at worst.

When something fails in an eval, the first question shouldn't be "what's wrong with my system?" It should be "is this a system problem or an eval problem?" Answering that correctly is what separates teams that iterate productively from teams that spin on phantom issues.

The pattern that keeps eval problems from compounding:

• Separate test cases by response type — retrieval responses and live-data responses have different quality criteria

• Read the reasoning, not just the score — the scorer's reasoning field shows when it's applying criteria incorrectly

• Flag patterns, not individual failures — three failures on employee lookups is a signal about the eval design, not three independent system bugs

• Distinguish real gaps from eval artifacts — the citation gap in the agent is real; the original 1.5/10 on employee lookups was not

Phase 4 adds spaced repetition via the SM-2 algorithm. The implementation forced three interesting design decisions: a schema change that clarified a data modeling confusion, a scoring conversion that rejected the obvious naive approach, and a function boundary that made the scheduling logic actually testable.

What SM-2 Does

SM-2 is the algorithm behind Anki. Instead of reviewing problems on a fixed schedule, each problem is scheduled based on how well you did last time. Struggle → see it again soon. Do well → longer gap before it resurfaces.

Each problem tracks three persistent values:

• interval — days until the next review

• ease_factor — a multiplier that controls how fast the interval grows (starts at 2.5, floors at 1.3)

• repetitions — how many consecutive successful reviews have happened

After each session, a 0–5 quality score drives the update. Scores below 3 reset the interval and repetition count. Scores of 3 and above grow the interval by the ease factor.

The result is a scheduling system that adapts to actual performance — problems you consistently get right disappear from the queue for weeks; problems you keep missing come back the next day.

The Schema Problem: Two Tables, Two Concerns

Before Phase 4, the system had one table: sessions. Each row was a logged study session with topic, difficulty, and grading results. Adding SM-2 to that table would mean mixing two fundamentally different concerns.

Sessions are historical records — immutable, append-only, one row per session. SM-2 state is current state — mutable, one row per problem, updated in place after every review. Mixing them would produce a table where some columns are historical facts and others are running totals, with no clean way to query either correctly.

The solution is a problems table that holds one row per generated problem. SM-2 fields (interval, ease_factor, repetitions, next_review_date) start as NULL on first insert and get updated in place after each session. sessions gets a problem_id foreign key and drops the fields now derivable via join.

An earlier design considered storing a new SM-2 row after each session to preserve a history of scheduling states. The problem: there's no practical query that needs past SM-2 states. The only thing the scheduler ever needs is the current state — what's the next review date, what's the current ease factor. Keeping historical rows would add storage cost and query complexity for zero benefit.

The Scoring Conversion Problem

SM-2 expects a 0–5 quality rating with specific semantics: 0–2 means the review failed, 3–5 means it passed. That threshold is what drives whether the interval resets or grows — so the mapping from session score to SM-2 score has to respect it.

The naive approach is a linear mapping: round(total_score / max_score * 5). The problem is that rubrics with multiple criteria rarely produce scores near 0 even for completely wrong answers. A solution with correct structure but wrong output might still score 40–50% on the rubric because it passes naming, modularity, and style criteria. A linear mapping compresses everything into the 2–3 range and makes the threshold meaningless.

The correct approach uses correct_output as the gate:

def calculate_initial_sm2_score(
    is_correct: bool, is_uncertain: bool, total_score: int, max_score: int
) -> int:
    sm2_score = math.floor(total_score / max_score * 2)
    if is_correct:
        sm2_score += 3
    if is_uncertain and sm2_score > 0:
        sm2_score -= 1
    return sm2_score

If correct_output failed, the score lands in 0–2 regardless of how the other criteria performed. If it passed, it starts at 3 and scales up based on the remaining criteria. The is_uncertain flag (set when the two grading passes diverge by more than 5%) decrements the score by 1 — a signal that the grade itself isn't reliable, which should slow the scheduling down rather than reward it.

Criteria with evaluation_dependency: "correct_output" — things like edge case handling — are excluded from the secondary score entirely. Including them when correctness failed would double-penalize and distort the 0–2 range.

SM-2 as a Pure Function

The actual SM-2 calculation is implemented as a pure function: takes current state + the 0–5 score, returns updated state, touches no database.

def calculate_sm2(
    repetitions: int,
    interval: int,
    ease_factor: float,
    quality: int
) -> SM2Result:
    if quality < 3:
        return SM2Result(repetitions=0, interval=1, ease_factor=ease_factor)

    new_ease = max(1.3, ease_factor + 0.1 - (5 - quality) * 0.08)

    if repetitions == 0:
        new_interval = 1
    elif repetitions == 1:
        new_interval = 6
    else:
        new_interval = round(interval * ease_factor)

    return SM2Result(
        repetitions=repetitions + 1,
        interval=new_interval,
        ease_factor=new_ease
    )

The caller (log_session()) handles DB reads and writes. First-review defaults are resolved before the function is called:

ease_factor = problem["ease_factor"] if problem["ease_factor"] is not None else 2.5
repetitions = problem["repetitions"] if problem["repetitions"] is not None else 0
interval = problem["interval"] if problem["interval"] is not None else 0

Keeping defaults out of the pure function means the function is fully testable without a database — pass in any combination of state and score, verify the output.

One Date Decision That Matters

An earlier version calculated next_review_date as the old due date plus the new interval. If a problem was due on Monday and reviewed on Friday, the next review would be scheduled from Monday — not Friday.

That's wrong. Spaced repetition should adapt to when the review actually happened, not when it was supposed to happen. Late reviews shouldn't cascade into further compressed scheduling. next_review_date is always calculated from today.

What's Still Not Wired Up

Phase 4 adds the schema, the scoring conversion, and the SM-2 calculation. Two things are explicitly deferred:

main.py hasn't been updated yet — it still uses the old log_session() signature. End-to-end wiring is blocked on resetting seed data for the new schema, which is a Phase 5 task.

get_due_problems() exists as a function but isn't exposed to the user yet. It returns all problems where next_review_date <= today, and accepts an anchor_date argument so it's testable with arbitrary dates. Surfacing it in the session flow — and deciding whether a session should start with due reviews or new problems — is also Phase 5.

The deferred scope is intentional. Phase 4's job was getting the scheduling logic right. Wiring it into the user flow is the next phase's job.

That works fine for a single-purpose search tool. It breaks down the moment you have multiple data sources, multiple tools, and questions that don't all need the same retrieval path.

The fix is conceptually simple: make RAG a tool. Instead of retrieval being a mandatory pre-processing step, it becomes one capability among many that the model can choose to invoke — or skip — based on what the question actually needs.

What Changes

In 03-rag-basic, the flow was linear and predetermined:

query → embed → retrieve → prompt → generate

In 06-tool-use-rag, the flow is dynamic:

query → model decides → [search_handbook? get_employee_info? get_pto_balance? none?] → generate

Given the question "How much PTO does alice.bob have left?", the model might call get_employee_info and get_pto_balance in parallel. Given "What's our parental leave policy?", it calls only search_handbook. Given "What does RAG stand for?", it answers directly without calling anything.

That routing decision — which is made by the model, not hardcoded — is what makes the agent pattern fundamentally more capable than a fixed retrieval pipeline.

Parallel vs. Sequential Tool Calls

This is one of the more important distinctions to understand for production agent work.

Parallel: the model calls multiple tools in a single turn because it already knows what it needs and the results don't depend on each other. Both calls go out simultaneously, both results come back, the model synthesizes them into a single response. This is the common case for questions that span data sources.

Sequential: the model calls one tool, receives the result, then decides it needs another tool based on what came back. This requires multiple loop iterations. A question like "find the most severe open bug and draft a response" can't call get_issue_details until it knows which issue ID to look up from the search results.

This is exactly why the routing layer needs to be a while loop, not an if/else. A single branch only handles one round of tool calls. The loop keeps running until stop_reason === "end_turn" — meaning the model is done and ready to generate a final response.

while (true) {
  const response = await streamAndCollect(messages, tools);
  messages.push({ role: "assistant", content: response.content });

  if (response.stop_reason === "end_turn") break;

  // execute tool calls, append results, loop
  const toolResults = await executeTools(response.content);
  messages.push({ role: "user", content: toolResults });
}

Why System Prompt Ordering Is Not Optional

Tool descriptions tell the model what each tool does. The system prompt tells it when and in what order to use them. Both are required for reliable behavior — and confusing the two is one of the most common failure modes in production agent systems.

Without explicit instructions, the model tends to answer policy questions from its training data instead of actually searching the handbook. The answers sound confident and can be completely wrong.

system: `Guidelines:
- Always search the handbook before answering policy questions — don't rely on your own knowledge
- If a question involves a specific employee, look them up first
- If a question needs both employee info and a policy, use both tools
- If you can't find relevant information, say so — don't make things up`

The tool description handles the "what." The system prompt handles the "when" and "in what order." Omitting the system prompt guidance means you're hoping the model infers the right sequencing from tool descriptions alone — which it sometimes will and often won't.

The Cross-Referencing Behavior

The most interesting result from testing this module: when asked about PTO for an employee with three years of tenure, the model:

1. Called get_employee_info to confirm tenure (3 years)

2. Called search_handbook to retrieve the PTO policy

3. Matched the tenure against the correct policy bracket (2–5 years = 20 days)

4. Added the remaining balance by combining pto_balance and pto_used from the employee data — without being asked

No single tool returned that answer. The model synthesized two results into a response richer than what either source contained alone. That synthesis across data sources — rather than lookup from one — is the core value of the agent pattern over a fixed retrieval pipeline.

What This Looks Like in Production

The setup in this module is a simplified version of what real enterprise FDE deployments actually look like:

The model doesn't care that these are mocked. The architecture is identical. Swapping a mock for a real API is one line of change in the tool execution layer.

The hard problem in this kind of work isn't the model call. It's the integrations — real auth, real rate limits, real data permissions, real failure modes. That's where most of the engineering work actually lives in production agent systems, and it's the part that's invisible in most demos.

This post is the reference I wish I'd had at the start — framed around the JS/TS concepts you already know.

Setting Up a Project: The Node.js Mapping

The setup workflow maps almost 1:1. The key difference is that Python packages install globally by default, which is why virtual environments exist — they're Python's equivalent of node_modules.

Creating and activating a virtual environment:

python -m venv .venv
source .venv/bin/activate   # macOS/Linux

Important: Unlike Node, you have to activate the virtual environment every time you open a new terminal session. Forgetting this is the most common setup error.

Imports: The __init__.py Requirement

JavaScript imports work based on file paths. Python imports work based on module paths — and every directory that contains code you want to import needs an __init__.py file. This tells Python the directory is a module.

project/
  main.py
  src/
    __init__.py          ← required
    generator/
      __init__.py        ← required
      problem.py

# Python equivalent of: import { generateProblem } from './src/generator/problem'
from src.generator.problem import generate_problem

Omitting __init__.py in any directory in the import chain causes a ModuleNotFoundError that's easy to mistake for a path problem.

Type System: dict vs TypedDict vs dataclass

This is the decision that matters most when you start building anything non-trivial. The JavaScript equivalent is choosing between a plain object, an interface, and a class — but Python's options have different tradeoffs.

Use TypedDict for API responses and data shapes that flow between functions — it gives you editor autocomplete and static type checking with no runtime overhead:

from typing import TypedDict, NotRequired, Literal

class Criterion(TypedDict):
    label: str
    points: int
    description: str
    evaluation_type: Literal["independent", "cascading"]
    evaluation_dependency: NotRequired[str]  # equivalent to key?: string in TS

Use dataclass for core application objects that need methods or validation.

Use dict for small, temporary data structures or when keys are generated at runtime.

The Mutable Default Argument Bug

This one will bite you silently and is probably the most important Python gotcha for JS engineers.

# WRONG — the list is shared across ALL calls to this function
def generate_problem(topic: str, past_problems: list = []):
    past_problems.append(topic)  # modifies the shared list

# CORRECT — use None, create a fresh list inside the function
def generate_problem(topic: str, past_problems: list = None):
    past_problems = past_problems or []

In JavaScript, default arguments are re-evaluated on every call. In Python, mutable defaults (lists, dicts) are created once when the function is defined and shared across all calls. The bug manifests as state accumulating unexpectedly across calls — hard to catch in testing, easy to miss in code review.

Spreading, Ternary, and f-strings

Three syntax patterns that have direct JS equivalents:

Spread operator:

# JavaScript: [...BASE_RUBRIC, ...EXERCISE_RUBRICS[type]]
# Python:
return [*BASE_RUBRIC, *EXERCISE_RUBRICS[exercise_type]]

Ternary:

# JavaScript: condition ? value_when_true : value_when_false
# Python:
value_when_true if condition else value_when_false

String interpolation (f-strings):

prompt = f"Generate a {difficulty} problem on the topic of {topic}."

# If your string contains literal curly braces, double them:
prompt = f"Return JSON in this format: {{\"key\": \"value\"}}"

Environment Variables

Unlike frontend frameworks that handle .env files automatically, Node.js and Python both require explicit loading. The Python equivalent of dotenv:

pip install python-dotenv

from dotenv import load_dotenv
import os

load_dotenv()
api_key = os.getenv("ANTHROPIC_API_KEY")

This trips up JS engineers who've been working primarily in Next.js or Vite, where env loading is handled by the framework. In a pure Python or pure Node script, you manage it yourself.

This post documents what I learned building the problem generation component of my study system, where a prompt had to produce interview-caliber problems with consistent structure on every run.

What the Component Needs to Produce

The output isn't a free form answer: it's a structured problem that can be immediately worked on:

{
  "topic": "React state management",
  "difficulty": "medium",
  "prompt": "Build a shopping cart component that...",
  "constraints": ["Must handle concurrent updates", "No external state library"],
  "examples": [
    { "input": "addItem({ id: 1, name: 'Widget', price: 9.99 })", "output": "Cart: 1 item, $9.99 total" }
  ],
  "setup_code": "const initialCart = { items: [], total: 0 };"
}

If any field is missing or vague, the downstream session breaks. That constraint forced me to treat output reliability as a first-class requirement from the start.

Iteration 1: The Prompt Was Too Rigid

The first version of the generation prompt was fully hardcoded: topic, difficulty, and output structure baked in as static text. It worked but couldn't adapt. As the system matured to accept history, difficulty calibration, and topic filtering, a rigid prompt became a bottleneck.

The lesson: prompts for production components should be built as functions, not strings. Dynamic inputs (topic, difficulty, past problems, constraint filters) compose into the prompt at call time. The static portions are the instructions and schema. The dynamic portions are the context.

The Model Sometimes Ignores Formatting Instructions

I explicitly told the model not to wrap the response in markdown code blocks. It ignored this instruction often enough to be a problem — not on every call, but on enough that I couldn't rely on clean JSON coming back.

The fix is defensive stripping in addition to the instruction, not instead of it:

raw = response.content[0].text
cleaned = raw.strip()
if cleaned.startswith("```"):
    cleaned = "\n".join(cleaned.split("\n")[1:])
if cleaned.endswith("```"):
    cleaned = "\n".join(cleaned.split("\n")[:-1])
result = json.loads(cleaned)

The instruction reduces frequency; the stripping handles the remainder. Both are necessary. This is a general pattern: for any structured output that feeds application code, you need both a clear prompt instruction and a parsing layer that handles model non-compliance gracefully.

Scope Decisions That Saved Time

Two scope decisions I explicitly deferred kept Phase 1 on track:

setup_code as a string, not an array. The ideal design for multi-file problems (HTML + CSS, for example) would be an array of file objects. But for a CLI-based tool in the current scope, a single string is sufficient and eliminates the complexity of file-aware rendering. I documented this as a known limitation to revisit.

Problem formatting as plain text, not HTML. The model naturally wanted to format problems with rich structure. Useful eventually, but not useful when the output surface is a terminal. Deferring this prevented UI work from blocking the core prompt engineering work.

Both decisions share a pattern: identify the simplification that keeps the current phase moving without closing doors, and write down what you're deferring and why.

When Past Problems Scope Outgrew the Component

One feature I initially planned to include in the generator (passing in past problems to avoid repetition) revealed a scope issue mid-build. Avoiding repetition is a scheduling concern, not a generation concern. The generator should generate; a higher-level orchestrator should decide whether to generate a new problem or surface a review problem from history.

Keeping the generator's interface narrow (topic + difficulty → structured problem) made it more composable and easier to test in isolation. The past problem logic belongs in the component that calls the generator, not inside it.

This is a prompt engineering lesson as much as a software design one: if your prompt is growing to handle multiple concerns, it's often a sign the abstraction is wrong, not that the prompt needs more instructions.

What This Looks Like in Practice

After several iterations, the generator produces problems specific enough to start immediately and structured enough to parse reliably. More importantly, the prompt is a function — inputs compose cleanly, the output schema is stable, and failures are handled defensively at the parsing layer.

The study system is built on top of this component's reliability. If the generator is flaky, every downstream session is degraded. Treating prompt reliability as a first-class engineering concern — not just "get the words right" but "design a prompt that fails gracefully and consistently" — is the lesson that transferred most directly to how I think about building on LLMs in general.

There are two approaches. Choosing the right one changes how reliable your system is.

Approach 1: Prompt-Based

Tell the model in the system prompt to return JSON matching a specific shape.

const systemPrompt = `
You are a grading assistant. Return your evaluation as a JSON object with this structure:
{
  "score": number (0-10),
  "passed": boolean,
  "feedback": string
}
Return only the JSON object. No markdown, no explanation, no code blocks.
`;

This works most of the time. The model is good at following formatting instructions for common shapes. The problem is "most of the time". Occasionally, it wraps the response in markdown code blocks, adds a preamble sentence, or returns a subtly malformed object. You end up parsing defensively:

const text = response.content[0].text;
const cleaned = text.replace(/```json|```/g, '').trim();
const parsed = JSON.parse(cleaned);

For low-stakes or high-volume use cases where occasional failures are acceptable, prompt-based is fine. It's also simpler to iterate on — just edit the prompt.

Approach 2: Forced Tool Use

Define a tool whose input schema is exactly the shape you want, then force the model to call it:

tools: [{
  name: "submit_evaluation",
  description: "Submit the structured evaluation result",
  input_schema: {
    type: "object",
    properties: {
      score: {
        type: "number",
        description: "Score from 0-10 where 0 is completely wrong and 10 is perfectly accurate"
      },
      passed: { type: "boolean" },
      feedback: { type: "string", description: "One to two sentences of specific, actionable feedback for the learner" }
    },
    required: ["score", "passed", "feedback"]
  }
}],
tool_choice: { type: "tool", name: "submit_evaluation" }

With tool_choice forcing the model to call this tool, the response is always valid JSON conforming to the schema. The model's tool-calling pathway is specifically trained for schema compliance. You extract the input directly:

const toolUse = response.content.find(b => b.type === "tool_use");
return toolUse.input as EvaluationResult;

The tool never "runs" anything. Its only purpose is to give the model a schema to conform to.

When Property Descriptions Carry All the Weight

With tool_choice forcing the call, the top-level tool description matters less — the model has no choice but to use it. What matters are the property descriptions, because those guide what value the model generates for each field.

// Vague — model guesses what's expected
score: { type: "number" }

// Specific — model knows exactly what scale to use and what the extremes mean
score: {
  type: "number",
  description: "Score from 0-10 where 0 is completely wrong and 10 is perfectly accurate. Use the full range — a partial correct answer should score 4-6, not cluster near the top."
}

This is a direct parallel to how description quality drives tool selection reliability in multi-tool setups. Whether the model is choosing which tool to call or filling in a field value, the description is the only lever you have to influence the output toward what you actually want.

How to Debug When the Model Picks Wrong

When you're not using tool_choice and the model selects the wrong tool or generates an unexpected value, there's no stack trace. The decision is opaque. The places to look:

1. Tool and property descriptions: is there enough differentiation between tools? Are the property descriptions specific enough about valid values?

2. System prompt: can you add explicit ordering instructions ("always call X before Y")?

3. tool_choice override: if a specific call must happen, force it rather than nudging

You're always nudging when you're not using tool_choice. Evals matter here because you can't inspect the decision — you can only measure outcomes across many runs and catch regressions.

The Decision Framework

The tradeoff is always reliability vs. flexibility. Forced tool use is more reliable but locks you into a schema. Prompt-based is easier to iterate but requires defensive parsing. In production systems where structured output feeds other components, the reliability is usually worth it.

This post documents what I learned building the grading component of my personal study system, where I had to solve this problem for real.

The Four Failure Modes

Position bias. If you ask a judge to compare two answers (A vs. B), it tends to favor whichever comes first. Swap the order and you can get the opposite verdict. For pairwise evaluation, always test with swapped order and check consistency.

Verbosity bias. Longer, more confident-sounding answers score higher even when they're less accurate. The judge rewards the appearance of thoroughness. This is particularly dangerous in code review — a long but wrong implementation can outscore a short but correct one.

Self-preference bias. A model tends to rate outputs from itself (or similar models) more favorably than outputs from other models. If the same model generates the problem and grades the solution, this bias is active.

Sycophancy. If the prompt gives any hint of what answer you want, the model leans that way. "Confirm this is correct" versus "evaluate this objectively" produces meaningfully different results even when grading identical content.

The Design Decisions I Made

Rubric Decomposition Over Holistic Judgment

Instead of asking "is this a good solution?", every criterion is an atomic yes/no check with a specific description:

{
  "label": "Edge cases handled",
  "points": 2,
  "description": "Handles empty input, null/None, zero, or boundary values without crashing or returning wrong output.",
  "evaluation_type": "cascading",
  "evaluation_dependency": "correct_output"
}

Two things to note here. First, the description gives the model a concrete test to apply, not a judgment call to make. Second, the evaluation_type and evaluation_dependency fields encode a relationship I discovered was being missed: edge case handling is meaningless to evaluate if the primary output is wrong. Adding cascading dependencies prevents the judge from awarding points for edge cases in a solution that doesn't produce correct output for the happy path.

This dependency modeling took several iterations to get right. Initially I omitted it, and the grader was being too lenient on incorrect solutions because it evaluated each criterion independently.

Two-Pass Evaluation With Framing

To combat sycophancy and improve consistency, I run the grading prompt twice with different framings:

• Strengths framing: "Focus on what the learner is doing well"

• Gaps framing: "Focus on what the learner is missing or could fail on"

The gaps framing is dominant — it's the one that drives the final score. If the difference between the two passes exceeds 5% of the total point value, the result is flagged as uncertain. The threshold is arbitrary but functions as an early signal that the prompt needs tuning, since high discrepancy usually means the criteria descriptions are ambiguous.

The tradeoff I accepted: this approach isn't maximally accurate, but it's calibrated for the right direction. For a personal learning tool, being harder on gaps than strengths is a reasonable bias — I want the system to catch things I missed, not validate things I did right.

Criterion Descriptions Take Most of the Work

The most time-consuming part of building this wasn't the code — it was iterating on the criterion descriptions until the grader produced sensible results. One specific example: the time complexity criterion was initially docking points for an O(2n) solution, claiming a more efficient approach existed, when no such approach was possible. Two fixes resolved this:

1. Added explicit language to the description: "If a simpler approach has the same Big-O complexity class, prefer that. Only flag if a significantly better class is straightforward to achieve."

2. Added explicit language about O(n) and O(2n) being the same complexity class.

The underlying issue was the model hallucinating an O(n) approach that didn't exist and penalizing the solution accordingly. Clear criterion descriptions prevent this by constraining the model's judgment to what's actually being measured.

Chain-of-Thought Before the Verdict

The prompt explicitly asks for reasoning before the score:

## Instructions
1. For each rubric criterion, reason through whether the learner's answer satisfies it.
2. Assign points: full points if satisfied, 0 if not.
3. Return your response as a JSON object in exactly this format, with no other text.

Asking the model to reason first measurably reduces snap judgments. The reasoning also surfaces when the model is confused — if the reasoning field contains contradictory logic but is_satisfied is true, that's a signal to revisit the criterion description.

The Broader Point for Production Systems

Every production LLM evaluation system faces this problem in some form. Whether you're evaluating RAG retrieval quality, agent decision quality, or generated content — you need automated scoring you can trust.

The patterns I used here (rubric decomposition, multi-pass with framing, explicit reasoning before verdict, dependency modeling) all generalize. They're not study-buddy-specific. They're the standard toolkit for making LLM-as-judge reliable enough to act on.

This series documents a different starting point: I had a real, recurring problem in my daily workflow, and I decided to build an LLM-powered system to solve it. The problem forced me into every major skill category that matters for applied AI engineering: prompt engineering, eval frameworks, agent architecture, structured output, and spaced repetition scheduling. Not as checkboxes, but as genuine design decisions with real tradeoffs.

The Problem

My daily technical practice sessions were almost entirely manual. Each morning I had to: decide what to work on, search through past notes to find problems I'd struggled with, prompt an AI step by step through a structured exercise, grade my own output, and log the session somewhere. This took meaningful time and was inconsistent. On high-friction mornings, the prep overhead itself became an excuse to skip.

The other problem was invisible: I had no spaced repetition. Problems I struggled with didn't resurface at the right time. I'd nail something in a session and not see it again for weeks, or hit the same failure mode repeatedly because I had no system tracking it.

What "actually works" looks like: wake up, run one command, get a study plan and a problem to work on. At the end of the session, submit the solution and get a graded report that tells me what I missed. Have that logged automatically so the system knows what to surface tomorrow.

Why Not Just Use an Existing Tool

LeetCode has no model of how I study. Anki handles repetition but not problem generation or grading. ChatGPT can do pieces of this but has no memory or state across sessions. The combination I needed (personalized problem generation, structured grading against my own rubric, and spaced repetition driven by actual performance data) didn't exist off the shelf. Building it was also the point: every component maps directly to applied AI skills.

The System's Six Jobs

When I broke down what the system actually needs to do, it decomposed into six functional components:

1. Problem generation: generate a relevant problem based on current skill gaps and history

2. Grading + feedback: evaluate a submitted solution against both objective and qualitative criteria

3. Progress tracking: automatically log sessions, scores, and patterns over time

4. Spaced repetition: resurface problems I struggled with at the right interval

5. Topic suggestion: recommend what to focus on next based on patterns in what I'm getting wrong

6. Schedule generation: produce a daily study plan that incorporates all of the above

The build order matters. Problem generation has the fewest unknowns and can be built with current knowledge. Grading is blocked on a design decision about evaluation reliability. Everything downstream depends on progress tracking. I sequenced the build to unblock design decisions as quickly as possible rather than building in order of appearance.

The Hardest Design Problem: Who Grades the Grader?

Before writing a line of code, I identified the blocking design decision: if the same model generates a problem AND grades the solution against it, it's evaluating its own output. Lenient generation leads to easy grades. The system becomes self-congratulatory.

This isn't just a personal project concern. It's one of the central problems in production LLM evaluation systems: how do you prevent model-generated rubrics from being too accommodating of model-generated solutions?

I researched three patterns before making a design decision:

• Multi-model evaluation: use a different model to grade than the one that generated the problem. Breaks the self-preference loop.

• Rubric decomposition: instead of asking "was this good?", break the evaluation into atomic yes/no checks. Harder to be lenient when each criterion has a specific description.

• Adversarial test generation: prompt a model specifically to find edge cases the solution might miss, rather than just verifying the happy path.

The decision I landed on, and why, is the subject of the next post in this series.

What This Looks Like as an Architecture Decision

The FDE and applied AI roles I'm targeting care about this kind of thinking. Not "I used the Anthropic API" but "I identified a reliability problem at the system design level, researched the known solutions, and made an explicit decision with documented tradeoffs." That's the work — not the code.

The component build plan, the sequencing logic, the blocking research question — these are the artifacts of that thinking. The code comes after.

Query → embed query → search vector store → retrieve top N chunks → prompt + chunks → generate

At its core, it's a retrieval problem with a generation step at the end. The model doesn't have access to your data — it reasons over whatever you include in the prompt. RAG is the mechanism for deciding what to include.

What Embeddings Actually Are

An embedding is a numerical representation of text — a list of floats (a vector) that captures semantic meaning. Text with similar meaning produces vectors that are close together in high-dimensional space.

This is what makes semantic search work. When you embed a query and search for the stored vectors nearest to it, "nearest" means semantically similar — not lexically similar. The query "how do I cancel my subscription" will find documents about "account cancellation" and "ending a membership" even if neither phrase appears in the query.

Traditional keyword search matches words. Embedding-based search matches meaning. That distinction matters a lot when users phrase things differently than your documentation does.

The Stack

For a basic RAG implementation in TypeScript/Node.js:

• Anthropic API for the generation step

• OpenAI embeddings for creating and querying vectors (Anthropic's SDK doesn't expose an embeddings API, so OpenAI fills that gap)

• pgvector on PostgreSQL for the vector store

• NDJSON streaming to push results to the client incrementally

The pgvector setup is straightforward. It's a Postgres extension that adds a vector column type and similarity search operators. You store your document chunks with their embeddings, then query for the closest matches at retrieval time.

Balancing the Similarity Threshold

Every RAG implementation needs a similarity threshold — a cutoff below which retrieved chunks are considered too dissimilar to be relevant.

Setting this wrong in either direction causes real problems:

Too high: You filter out chunks that are relevant but not close to an exact phrasing match. The model gets less context than it should and either makes things up or says it doesn't know.

Too low: You retrieve chunks that aren't genuinely relevant, adding noise that degrades the quality of the generated response. And it costs tokens.

There's no universal right answer here. The threshold needs to be tuned against real queries from your specific use case. Start conservative (higher threshold, fewer results) and loosen it as you observe misses.

The pgvector Bug That Trips You Up

Here's the production debugging story that makes this post worth reading.

When I was building the RAG module, I hit a case where the similarity search was returning empty results even for queries that clearly matched stored documents. The data was there. The embeddings were correct. The query looked right.

The culprit: a known pgvector behavior where referencing the same parameterized vector expression more than once in a single query causes it to return nothing.

This query fails silently:

SELECT id, content
FROM documents
WHERE 1 - (embedding <=> \(1::vector) > \)2
ORDER BY embedding <=> $1::vector

The vector $1::vector is referenced twice — once in the WHERE clause and once in the ORDER BY. pgvector evaluates it twice, and the second evaluation returns empty.

The fix is a subquery that evaluates the expression once and references the result:

SELECT id, content, similarity
FROM (
  SELECT id, content, 1 - (embedding <=> $1::vector) AS similarity
  FROM documents
) AS ranked
WHERE similarity > $2
ORDER BY similarity DESC

This pattern evaluates the vector expression a single time in the inner query, then filters and sorts against the pre-computed similarity score in the outer query. The results come back correctly.

The practical rule: never reference the same parameterized vector more than once in a single pgvector query.

NDJSON for Streaming Mixed Content

Basic streaming pushes raw text strings to the client. RAG adds a retrieval step before generation — and the client needs to know about both. What got retrieved? When does generation start?

The answer is NDJSON (newline-delimited JSON): each chunk pushed through the stream is a JSON object with a type field:

// Retrieval result
controller.enqueue(encoder.encode(JSON.stringify({ type: "sources", data: retrievedChunks }) + "\n"));

// Generated text
controller.enqueue(encoder.encode(JSON.stringify({ type: "text", delta: chunk.delta.text }) + "\n"));

The client splits incoming data on newlines and parses each line independently. A partial reader.read() result gets buffered until the next \n arrives. This is also why TextEncoder becomes necessary here — ReadableStream expects Uint8Array, and NDJSON requires explicit encoding rather than relying on runtime tolerance for plain strings.

The Broader Pattern

The pgvector bug is a good example of a class of problems that's common in applied AI work: the integration layer between the model and your data infrastructure has its own failure modes that have nothing to do with the model. Debugging them requires treating each layer (the embedding generation, the vector store query, the retrieval pipeline) as independently testable components.

In production RAG systems, most failures happen in retrieval, not generation. The model does a reasonable job if given good context. The hard part is reliably getting it that context.

Anthropic never executes your code. The model reads the description and schema you provide, decides this is the right tool for the job, generates a valid set of arguments, and stops. You run the function. You send the result back. The model then continues generating based on that result.

Understanding this distinction changes how you think about building reliable tool-based systems.

The Conversation Flow

Tool use turns a single API call into a multi-turn exchange:

1. You send a message + your tool definitions

2. The model responds with a tool_use block containing a name and arguments

3. You run the actual function with those arguments

4. You send the result back as a tool_result message

5. The model generates its final response using that result

This is fundamentally different from a normal completion. The model is mid-sentence when it decides to call a tool — it needs you to act on that, then resumes where it left off.

What This Looks Like in Code

In a streaming context, adding tool use changes the stream in two ways. First, instead of only seeing text_delta events, you'll now see input_json_delta events — the model streaming the JSON arguments for a tool call in chunks. Second, content_block_start becomes load-bearing, because it tells you what kind of block is opening:

// content_block_start for a tool call looks like:
{
  type: 'content_block_start',
  content_block: {
    type: 'tool_use',
    id: 'toolu_01A09q90qw90lq917835lq9',
    name: 'get_weather',
    input: {}
  }
}

When you see a tool_use block starting, you capture the name and id, then accumulate the incoming input_json_delta chunks into a string. You don't parse that string until content_block_stop fires — that's your signal that the block is complete and safe to process.

Why wait for content_block_stop? Because a single message can contain multiple content blocks arriving in sequence. Parsing at message end would concatenate them incorrectly. Parsing at block stop means each tool call is handled as its own logical unit.

The Conversation State Problem

With basic streaming, the messages array was static — one request, one response. With tool use, it grows:

// After the model calls a tool:
messages = [
  { role: "user", content: "What's the weather in Chicago?" },
  { role: "assistant", content: [{ type: "tool_use", id: "...", name: "get_weather", input: { city: "Chicago" } }] },
  { role: "user", content: [{ type: "tool_result", tool_use_id: "...", content: "72°F, partly cloudy" }] }
]

If you don't append the tool calls and results to the conversation history before the next request, the model has no memory of what it called or what came back. The full conversation state must be sent with every turn.

This is also why the routing layer becomes a loop, not a single pass. Each iteration sends the current conversation state, streams the response, and either breaks (model is done) or continues (model called a tool and needs the result before it can finish).

Writing Tool Descriptions That Actually Work

Since the model selects tools based on their descriptions — not their implementations — description quality is where reliability lives. A useful mental model:

The key difference from a function: the output is not deterministic. Given the same inputs, there's no guarantee the model selects the same tool or generates the same arguments every time. This is why description quality matters more than implementation quality for reliability.

When you have multiple tools and the model needs to pick between them, descriptions need to clearly differentiate. A good test: if a new engineer read only the description, would they know when to use this tool vs. the others?

// Too vague — model will guess when to use this
{
  name: "search",
  description: "Searches for information"
}

// Specific enough to be reliable
{
  name: "search_products",
  description: "Searches the product catalog by name, category, or SKU. Use this when the user is looking for a specific product or browsing a category. Do not use for order status, shipping, or account questions."
}

Why This Matters Beyond Demos

The tool use pattern shows up in almost every production LLM system worth building: agents that can query databases, orchestrators that delegate to specialist models, copilots that can take actions in your application. The underlying mechanism is always the same — the model generates intent, your code executes, the result flows back.

The engineering challenge isn't the API call. It's building the conversation state management, the result routing, and the error handling that makes this loop reliable at scale. Those are the parts that look easy in tutorials and break in production.

Streaming is the difference between a chatbot that feels alive and one that feels like a form submission.

What Streaming Actually Is

Streaming generates and delivers text incrementally as the model produces it, rather than waiting for the full response to be ready. The transport mechanism is Server-Sent Events (SSE) — a one-way channel where the server pushes data to the client as it's available.

From a product perspective, this matters for two reasons:

Perceived latency drops dramatically. Users see the first token in under a second instead of waiting for the full response. Even if the total generation time is the same, the experience feels faster because something is happening immediately.

You can cancel early. If the model starts going in the wrong direction, the user (or your system) can cut the stream and save tokens. In agentic workflows where model calls chain together, this adds up.

When should you not use streaming? Backend pipelines where a machine is consuming the output, data extraction tasks where you need the complete structured response before you can do anything, and automated agent workflows where partial state is worse than no state. Streaming is a mechanism for incrementally delivering content — it's not always the right one.

The Bridge Problem

Here's where it gets interesting at the implementation level. The Anthropic SDK exposes streaming as an async iterator — it gives you chunks as they arrive through a for await loop. That's great for server-side code. But a Response object in a Next.js API route expects a ReadableStream, not an async iterator. They're not the same thing.

The solution is a ReadableStream wrapper that bridges the two:

const readable = new ReadableStream({
  async start(controller) {
    for await (const chunk of stream) {
      if (
        chunk.type === 'content_block_delta' &&
        chunk.delta.type === 'text_delta'
      ) {
        controller.enqueue(chunk.delta.text);
      }
    }
    controller.close();
  },
});

What this does: ReadableStream maintains an internal queue. As chunks arrive from the Anthropic stream, we filter for the ones we care about (content_block_delta events with text_delta type) and push them into that queue via controller.enqueue(). The client reads from the queue via reader.read(), which suspends when the queue is empty and resumes when a new chunk arrives. When controller.close() is called, read() returns done: true on the next call.

It's a producer/consumer pattern — the Anthropic stream is the fast producer, the client is the slower consumer, and ReadableStream is the buffer between them.

Understanding the Stream Structure

Every Anthropic streaming response follows the same envelope pattern:

message_start
  content_block_start
    content_block_delta
    content_block_delta
    ...
  content_block_stop
  content_block_start
    content_block_delta
    content_block_delta
  content_block_stop
message_stop

message_start carries the outer metadata — model, token usage, stop reason. The actual content lives in content_block_delta events. For basic text responses, you only need to track content_block_delta with type: 'text_delta'. Once you add tool use to the mix (covered in the next post), content_block_start becomes critical because it tells you what type of block is opening — text or a tool call.

Why This Matters for Production Systems

The ReadableStream wrapping pattern isn't just a TypeScript quirk — it's a concrete example of a problem that comes up constantly in applied AI work: the model's output format doesn't match what your application layer expects, and you need an adapter layer between them.

That adapter layer — whether it's a stream bridge, a JSON parser, or a response transformer — is often where the real engineering work happens in LLM-powered products. The model call itself is straightforward. Making its output integrate cleanly with the rest of your stack is not.