RAG / Evaluation
RAG Against the Machine
Building a RAG got cheaper. Evaluating its accuracy and efficiency remains challenging.
AI-assisted coding makes it increasingly easy to build something that looks like a working product. Two questions remain harder: is it useful, and how do we know how well it works?
I’m leaving the business case aside. Here, I’ll focus on answer quality, efficiency and the engineering trade-offs behind them.
1. Wake Up
During the hype around Retrieval-Augmented Generation (RAG), I was a Machine Learning Engineer working on more classical problems: supervised learning on tabular data, pipelines, A/B testing. Then 2026 arrived, and RAG expertise seemed to have become part of the job description. My side projects had mostly explored music, so I had some catching up to do.
I’m still learning. What I can share is a three-day side project answering questions about documents from the Boletín Oficial del Estado, Spain’s official state gazette. The app runs on Azure. You can try it here.
Building from scratch can be a good way to learn and an inefficient way to deliver. This time, I started from boe-rag-assistant, an open-source project by Gonzalo López Crespo, then used Claude Code to help me investigate, deploy and iterate on it.
2. Sleep Now in the Fire
As a rule of thumb, I expect more computation and better frontier models to solve many of today’s technical puzzles (Sutton’s Bitter Lesson), whether in six months, two years or longer. So part of engineering is knowing where craft beats force (as the Spanish saying goes, más vale maña que fuerza) and where the latest established model or solution is the no-brainer.
My hypothesis is that parts of today’s RAG stack (chunking recipes, embeddings, vector databases) may have a shorter shelf life than we assume.
The test I now try to apply: what would have to change before I stopped building this component? Context must be large enough, and reading all of it on every question must be cheap enough. My blog passes both today: about 12,000 words, around 17,000 tokens, or about $0.002 per question with GPT-6 Luna and $0.04 with GPT-6 Sol at list prices. My BOE sample fails both: it would need a context window about 15 times larger, and at long-context rates each question would cost about $3 with Luna and $63 with Sol.
Even so, in an earlier local experiment I built retrieval and an MCP server for that same blog, probably without needing either. A prompt with the whole blog could ask for cited answers. Retrieval earns its place when the data doesn’t fit in the prompt. MCP earns its place when your tools serve different AI applications outside your control.
3. Bulls on Parade
RAG can’t send the whole BOE to the LLM, so it splits documents into chunks and retrieves the few most relevant to the question. There are two main ways to rank them. BM25, the classical one, derives from Term Frequency-Inverse Document Frequency (TF-IDF) and matches words. It represents each text as a sparse vector, with one coordinate per vocabulary word, weighted by how often the word appears and how rare it is. Embeddings, the modern one, come from a neural model that represents each text as a dense vector, placing similar meanings close together even when they share no words. Vector databases exist mainly to search these dense vectors quickly. A hybrid combines both rankings, so you might expect it, or embeddings alone, to beat a technique from the 1990s.
When I compared retrieval approaches, adding components did not automatically help. On 20 manually curated questions (the gold set), a hybrid of BM25 and embeddings reached 95% Recall@5 (how often a labeled passage appears in the top 5) against 85% for either alone, though that gap is just 2 questions. On 14 questions asking what a given law covers, BM25 alone found the labeled passage every time, while the hybrid found it for 9. On an earlier corpus, a cross-encoder (a model that rereads each question and passage together to rerank them) improved ranking on the gold set but worsened it on those 14 questions and on a larger set generated by an LLM (the silver set), while adding latency. Here, BM25 matched or beat embeddings (multilingual-e5-small, a small off-the-shelf model) on Recall@5. Another corpus, other questions or a larger model could reverse that.
So the useful work was simplification: finding which components the system could do without. There is always value in distilling and simplifying, though the effort of doing it is not always worth it (when the more complex alternative already meets the requirements). Removing the cross-encoder left a simpler, faster system. The experiments behind that decision may stop being worth running once sending many more passages is cheap and doesn’t hurt the answers.
4. Take the Power Back
What I expect to last longest is measuring whether answers are right, at what cost, and with what user experience. I’m less sure about knowing your data or understanding retrieval and ranking techniques, which agents may absorb in part.
It’s a bitter thought, because my own expertise is on that list. Not the augmented generation that came with LLMs, but the older foundations of retrieving information from text: the subtle differences between TF-IDF and BM25, embeddings or tokenization. They were clever, carefully crafted solutions in their time, and the kind of puzzle engineers love to solve. They can still make a difference today (keeping 39/2015 as one token and indexing document titles changed what my system could find). But I expect them to become commoditized, operated from higher levels of abstraction and no longer required as hands-on skills.
I could be wrong. If context stops growing, prices stop falling, or large prompts keep losing to good retrieval, these skills will stay useful for much longer.
Either way, what will probably remain valuable is the ability to manage trade-offs, even if low-level technical puzzles no longer need as much human effort. Better models can change what is possible, but we still have to decide which balance serves the user, considering how accuracy, latency, cost and coverage are tied together. When I doubled my BOE corpus, memory rose and ranking quality fell slightly. To decide well, we need to measure what matters. And evaluating can be the hardest part.
5. Testify
Evaluation produced the most interesting questions.
What makes an answer right? Faithful to the retrieved fragments? Matching a reference answer? Complete enough to address the question? Supported by the particular source being cited? Some combination of these, and to what extent?
I found it useful to separate three checks:
| Check | Compares | Labels | LLM judge here |
|---|---|---|---|
| Retrieval | Retrieved passages with labeled passages | Yes | No |
| Groundedness | Answer with retrieved passages | No | Yes |
| Correctness | Answer with expected facts | Yes | Yes |
These checks overlap, but none replaces the others. An answer can be grounded in the retrieved evidence while addressing the wrong question, repeat an error in a source, or provide the right answer without supporting retrieved evidence.
6. Know Your Enemy
Retrieval scores also have blind spots. They only hold for the collection they were measured on, and they can’t see what is missing from it, because the questions only cover documents already in the corpus. In an audit of 50 sampled days, my corpus contained 26 of 207 Section I documents. The app holds a sample for cost reasons.
Refusal belongs in this picture too. If the available evidence cannot answer the question, acknowledging that limitation can be the correct response. Refusing an answerable question is a different failure.
A correctness evaluation can cover both cases, provided the references and rubric specify the expected behavior. I would still report them separately to distinguish excessive refusal from unsupported answering.
The project’s existing evaluation also taught me to read percentages carefully. Its 90.5% “corrección” was an average of judge-assigned ratings, including partial credit. It did not mean that exactly 90.5% of answers passed a binary test. Likewise, 98% fidelity was not a count of individually verified claims. It also measured a different system: it dated from July and used dense-only retrieval, not the system I serve today.
An LLM judge made scoring easier to automate. It did not make the scoring objective. References still needed validation, partial-credit ratings needed a clearer rubric, and judge behavior needed checking.
7. Mic Check
So I replaced those scores with two of MLflow’s built-in judges: RetrievalGroundedness, to catch claims the passages don’t support, and Correctness, to check answers against a list of expected facts. Each gives a yes or no per answer rather than partial credit. For questions the corpus cannot answer, I set the expected response to the refusal.
On 50 questions (10 of them unanswerable), with GPT-6 Luna as judge and each verdict decided by the majority of three votes, Correctness was 94% and groundedness 92.5% of answered questions. There were no wrong refusals and no wrong answers to unanswerable questions. Treat these as provisional: it was one run, and the labels still need a full audit. The report, questions, verdicts and code are public.
Interesting results emerged from disagreements between the checks. One answer retrieved the right article but gave only 1 of its 3 measures, and it claimed the passages contained no concrete details, although another retrieved passage did. It omitted required information and misrepresented the evidence, so both judges rejected it, while Recall counted it as a success.
Hans Rosling wrote that the world cannot be understood without numbers, nor with numbers alone. The same goes for evaluation: scores told me where to look, and reading the answers told me why.
These judges have limits too. Correctness checks whether the expected facts are present, but it does not guarantee that every additional claim is supported. Judges can also disagree with themselves: 3 of the 50 questions got split votes.
On my test set, the app reached 94% correctness with no wrong refusals. Then I typed my own open questions, and it refused six of seven. Sending 20 passages instead of 5 rescued none of them. The test questions were written from known passages, so they never measured this.
Concision is the dimension I’m least happy with. Answers that passed both judges had a median of 22 extra words over the reference answer (the references are terse, so this is a rough measure).
Lo bueno, si breve, dos veces bueno.
Short answers reduce reading effort and leave fewer opportunities to introduce unsupported claims. But omitting an exception can make a legal answer misleading.
The goal is the shortest answer that remains correct and complete, preserving the relevant conditions, exceptions and supporting citations.
Concision is hard. I probably failed to produce a Minimum Viable Post too (in my defense, structuring it around Rage Against the Machine song titles didn’t make it easier ;)