# Topic Suggestion — Designing a Function That Knows What to Recommend Without Magic Numbers

Phase 4 gave the study system memory of individual problems — when each one is due for review based on SM-2. Phase 5 asks a different question: zoomed out across all topics, what should I actually be working on today?

That's a recommendation problem, and recommendation problems have a familiar failure mode: every signal you could rank by feels arbitrary, and it's tempting to bolt on a manually-maintained "priority" field to break ties. This phase is mostly about resisting that temptation and finding signals that are actually derivable from data already in the system.

## The Schema Gap: You Can't Query for What Doesn't Exist

The first design problem wasn't about ranking — it was about a query that's structurally impossible with the existing schema.

One of the most useful recommendations is "topics you've never worked on." But `problems.topic` was a free-text enum value. If no problem with `topic = "dynamic_programming"` exists, there's no row to query — the topic is invisible to the database. You can't `SELECT` your way to a gap.

The fix is a dedicated `topics` table — `id`, `name`, `slug`, seeded upfront with the canonical topic list. Now "never worked on" becomes a standard pattern:

```sql
SELECT t.* FROM topics t
LEFT JOIN problems p ON p.topic_id = t.id
WHERE p.id IS NULL
```

This is a small schema change with an outsized effect: it makes an entire category of question answerable that simply wasn't representable before.

## One-to-Many vs. Many-to-Many: Choosing the Simpler Model on Purpose

Some real interview problems genuinely span multiple topics — a problem might be both "graphs" and "dynamic programming." A many-to-many `problem_topics` join table would model that correctly.

It was considered and rejected for this phase. Most DSA problems map cleanly to one topic, and the join table would be solving a problem that doesn't exist yet in the actual data. `problems.topic_id` stays a single foreign key. If multi-topic problems become common enough to matter, the join table is a clean migration — but building it now would be future-proofing against a hypothetical.

This is the same instinct as the earlier SM-2 schema decision: model what the data actually needs, not what it could theoretically need.

## Failing Loudly on Drift

`generate_problem()` looks up `topic_id` from the `Topic` enum value via `SELECT id FROM topics WHERE slug = ?`. If no match exists, it raises — it does not auto-create a row.

Auto-creation seems convenient, but it papers over a real bug: the `Topic` enum (in code) and the seeded `topics` table (in the database) are two independent sources of truth that can drift. If someone adds a new enum value and forgets to seed the corresponding row, auto-creation would silently produce a topic with no name and no metadata — a broken row that looks like it worked.

Raising immediately turns a silent data integrity problem into a loud, obvious one at the point where it's cheapest to fix: setup, not three weeks later when someone wonders why a topic has no display name.

## Designing `suggest_topics()`

The function signature is deliberately simple:

```python
def suggest_topics(limit: int = 3) -> list[TopicSuggestion]:
```

```python
class TopicSuggestion(TypedDict):
    id: int
    name: str
    slug: str
    problems: list[ProblemRow]
    explanation: str
```

A topic name alone isn't actionable. Each suggestion bundles up to 2 concrete problems to work on and an `explanation` of why this topic was surfaced now. The explanation matters as much as the ranking — "dynamic programming: you haven't started this topic yet" is something a user can act on; a bare topic name in a list is not.

## Three Signals, and the Honest Admission That the Order Is a Judgment Call

The function checks three signals in priority order:

1.  **Never worked on** — zero rows in `problems` for this topic (the `LEFT JOIN` pattern from earlier)
    
2.  **Low average score** — topics where average session score is lowest
    
3.  **Overdue for review** — topics with problems past `next_review_date`, ranked by how overdue the worst one is
    

The ordering reflects a judgment call: coverage gaps matter most (you can't improve at something you haven't tried), then struggling topics, then scheduled maintenance. There's no objectively correct order here, and the notes say so directly — this is a reasonable default that can be revisited, not a derived truth.

That kind of explicit acknowledgment is worth more than it looks. A system that ranks by an unstated, unexamined priority order is harder to debug and harder to adjust later, because nobody documented why the order is what it is.

## Rejecting the Manual Priority Field — Twice

For signal 1, an early idea was a manual "priority" or "interview frequency" column on `topics` — weighting "never worked on" suggestions by how often a topic actually shows up in interviews.

This was rejected for the same reason the SM-2 phase rejected manual scheduling fields: it requires ongoing manual upkeep and goes stale. For topics with genuinely no differentiating signal, an arbitrary tiebreaker — insertion order — is fine, because there's no real signal being discarded by using it. Adding a field that someone has to remember to update is worse than admitting the tiebreaker is arbitrary.

This is a recurring theme across the project: anywhere a manually-maintained field is proposed as a fix, the question is whether it's actually encoding a real signal or just deferring the discomfort of an arbitrary choice. Usually it's the latter.

## When a Pure Function Should Stay Pure

Signal 1 topics — by definition — have zero rows in `problems`. So what goes in the `problems: []` list for those suggestions?

The tempting fix is to call `generate_problem()` inline, backfilling a problem on the spot so the suggestion is immediately actionable. This was considered and rejected: it would turn `suggest_topics()` from a pure read function into something that calls the LLM and writes to the database as a side effect. That's a different function with a different contract — and a much bigger scope than "suggest topics."

The decision: return `[]` for signal 1 topics, and let the caller decide whether to call `generate_problem()` separately. Keeping read and write concerns in separate functions keeps both independently testable and keeps `suggest_topics()` fast and side-effect-free.

## Extracting the Shared Pattern

Signals 2 and 3 both need the same thing: "up to 2 problems for this topic where `next_review_date <= today`, most overdue first." The first implementation was two nearly-identical inline loops — one per signal.

That duplication got extracted into `_update_topics_with_problems(connection, topics, limit=2)`, a helper that mutates the `TopicSuggestion` list in place rather than returning a new one — since the dicts are already mutable references, an in-place mutation is simpler than threading a return value through.

## The N+1 Query: An Accepted Tradeoff, Not an Oversight

Fetching due problems per-topic in a loop is N+1 — one query per topic, rather than a single windowed query using `ROW_NUMBER() OVER (PARTITION BY topic_id ...)`.

For a personal tool where `limit` is capped at a handful of topics, the N+1 cost is negligible — a few extra queries on a SQLite database measured in milliseconds. The windowed-query version would be more "correct" in a scale sense, but it adds real SQL complexity for a benefit that doesn't exist at this scale.

This is worth stating explicitly because "N+1 queries" is often treated as an automatic red flag in code review. It's a red flag when N is large or growing. Here, N is bounded by `limit` and small by construction — the tradeoff is genuinely a non-issue, and saying so explicitly is more useful than either ignoring it or over-engineering around it.

## Deduplication, Early Exit, and a Hashability Bug

A topic could theoretically satisfy multiple signals. The combination logic iterates signal 1 → 2 → 3, tracks seen `id`s in a set, skips duplicates, and stops once `result` reaches `limit`. The `explanation` shown is whichever signal first matched — the highest-priority one.

The first attempt at deduplication used `dict.fromkeys([*signal1, *signal2, *signal3])`, expecting it to dedupe while preserving order. This fails because `TypedDict`s are plain dicts at runtime, and dicts aren't hashable — they can't be dict keys. The fix was an explicit loop with a `set` of seen IDs.

There's also an early-exit optimization: if signal 1 alone already meets `limit`, signals 2 and 3 are skipped entirely. Signal 3 is skipped if `len(signal1) + len(signal2) >= limit`. This sum-based check is safe specifically because signals 1 and 2 are mutually exclusive by construction — signal 1 requires zero `problems` rows, signal 2 requires `sessions` rows (which require `problems` rows), so no topic can appear in both and no double-counting is possible.

## Python Syntax Notes Worth Remembering

**Dict spread for building suggestions:**

```python
{**dict(row), "problems": [], "explanation": "..."}
```

Same concept as `{...obj}` in JS — unpacks key/value pairs into a new dict, with later keys overriding on collision.

## What's Still Open

`avg_score` and `max_overdue` are rounded for display but otherwise unvalidated — no handling for unusual values like negative deltas if a `next_review_date` ends up in the future but gets queried anyway. Signal 1's tiebreaker remains arbitrary insertion order, which is acceptable given there's no real signal to use instead. And `main.py` wiring plus the seed data reset — carried over from Phase 4 — are still outstanding before any of this is user-facing.

None of these are blocking. They're documented so the next phase starts with full context instead of rediscovering the same edge cases.