You Don't Need a Bigger Model. You Need Better Retrieval.

The console includes a small assistant, Ask EA, that answers architecture questions from the knowledge base and cites the pages it used. It runs on the same server as the console: CPU only, 8 cores, 16 GB of memory, no GPU. The model is a 4-billion-parameter open model served locally, so nothing leaves the box.

The first version was bad. More than 200 seconds to the first token, and citations pointing at unrelated research notes. The obvious fix was a bigger model. The server could not hold one next to the console — and, it turned out, the model was not the problem.

Retrieval was the real problem

The first retrieval matched on shared words. That is good at names and codes and weak at paraphrase: a question about an "owner" missed pages that said "custodian". The model was answering in good faith from the wrong pages.

Switching to semantic retrieval fixed the citations more than any model change could have. Every page is embedded with a small embedding model, the vectors sit in the console's existing local SQLite store, and the index updates incrementally when pages change. When the embedding model is unavailable, the assistant falls back to keyword retrieval and says so in the chat.

Latency was residency, not size

Most of those 200 seconds went to loading the model, not to thinking. Two changes fixed it: keep the model resident in memory, and warm it with a one-token request as soon as the chat panel opens. A fast mode that skips extended reasoning is the default.

Measure retrieval separately from answers

Generation is slow and non-deterministic. Retrieval is neither. So there are two evaluations, and they fail for different reasons:

• Retrieval eval. For each golden question, does the canonical page appear in the top one, three and five results? Reported with mean reciprocal rank. It is cheap enough to run on every change, and when it drops you know the index or the chunking broke.
• Answer eval. Does the answer contain the required facts, cite the right page, and decline when the knowledge base cannot answer? Some questions also carry must-not strings that fail on presence, which is how privacy leaks get caught.

The golden set is 35 questions across rules, concepts, frameworks, figures and decisions, including three where the only correct answer is "I don't know". Every expectation was verified against the knowledge base, not recalled. When the knowledge base changes, you fix the expectation: a failing question is a finding, not noise.

The gate is a floor, not a target. Raise it when a change earns it.

The floors started at 70% for the right page in the top five and 30% for the top one. After the retrieval work measured 90% and 66%, I ratcheted them to 80% and 50%. A regression now breaks the build instead of quietly degrading answers.

Fall back, but only locally

When the chosen model fails, the assistant retries with the next model in an ordered list — on the same local endpoint only, before the first byte reaches the browser — and it always says which model answered. A service that offered to route prompts across dozens of hosted providers was declined. It would have put knowledge-base content in front of processors nobody had reviewed, and that is a procurement decision, not a setting.

Copy this: two golden questions

The shape of the golden set, one answerable question and one that must be refused:

{
  "questions": [
    {
      "id": "decision-record-sections",
      "q": "What sections does our decision record template require?",
      "page": "templates/decision-record",
      "must": ["Considered Options", "Consequences"],
      "tag": "rule"
    },
    {
      "id": "refuse-vendor-quote",
      "q": "What price did the vendor quote us last week?",
      "refuse": true,
      "must_not": ["$"],
      "tag": "refuse"
    }
  ]
}

Before you reach for a bigger model, check whether the right page is even in front of the one you have.