RAG: is it still a differentiating skill for an AI professional?

A short reminder, because it is not the point

A RAG (retrieval-augmented generation) system fetches, at question time, document passages judged relevant, then hands them to a language model that writes an answer grounded in those passages. The usual diagram fits on one line: documents → embeddings → vector database → retrieval → LLM. The diagram is correct, it is useful for orientation, and today almost every candidate applying for a GenAI role can draw it.

Which is exactly why it no longer demonstrates anything. Knowing the chain shows you read the documentation or followed a tutorial. The professional question sits elsewhere: what do you do once the chain is in place and the answers are still wrong? This piece describes what actually distinguishes someone able to work on a RAG system, and how to place your own level honestly.

Why the architecture diagram proves nothing

Assembling a working RAG on a clean corpus has become a short exercise. Libraries hide most of the difficulty, defaults behave reasonably, and a convincing demo can be produced in a few hours. What the demo does not tell you is how the system behaves on real documents: mixed formats, several versions of the same document, tables, scans, domain vocabulary, and different access rights per user.

So the skill being looked for is not knowledge of the components, it is the ability to diagnose. When an answer is wrong, can you say whether the problem came from ingestion, chunking, retrieval, the context passed to the model, or the model itself? Asked plainly in an interview, that question separates profiles very quickly.

None of the practices described here is mandatory in the abstract. A few hundred pages of homogeneous content do not call for the same machinery as a multilingual enterprise document base with fine-grained permissions. Part of the skill is knowing what you can leave out.

What happens before the model: data, chunking, metadata

Data quality sets the ceiling

A retrieval system cannot return information that was lost at ingestion. A PDF whose tables were flattened into running text, a document whose contextual header disappeared, two contradictory versions of the same procedure indexed without distinction: none of that is recovered downstream by a better model or a better prompt. A significant share of real RAG work consists of looking at documents one by one, which demos rarely show.

Chunking is not a default setting

Chunking determines what can be found later. Fixed-size splitting cuts through arguments and separates a definition from its example; splitting guided by document structure — headings, sections, clauses, questions — keeps units that mean something to a reader. Overlap between segments, repeating the section title inside each segment, special handling for tables and lists: these are decisions, not values to copy.

Metadata is what makes a corpus usable

Source, date, version, issuing entity, language, applicable scope, confidentiality level. Without them you cannot filter, cite properly, manage freshness, or apply access rights. Metadata is often what is most missing in projects picked up after a prototype phase, and adding it later means reindexing.

Retrieval strategy: where the skill actually shows

Vector search alone brings together texts that talk about the same thing. That helps when the question is phrased differently from the document, and it breaks down as soon as the query contains an identifier, a product reference, an error code, a rare proper noun or an internal acronym. Lexical search finds exactly those and fails on rephrasings.

• Lexical search: robust on exact terms, references, acronyms; blind to synonyms.
• Semantic search: robust to rephrasing; can surface documents that are topically close but factually wrong for the question.
• Hybrid search: covers part of both blind spots, at the cost of a score fusion that has to be tuned and evaluated.
• Metadata filters: narrow the search space to what is applicable — scope, date, entity — before any similarity computation.
• Reranking: reorders a wider candidate set with a costlier, sharper model, and often improves perceived quality more than swapping the generative model.

Listing these options is not enough either. What counts is being able to say why a given combination was chosen for a given corpus, what it improved, and what it cost in latency.

What the model receives: context, citations, permissions, freshness

The handover from retrieval to the model is where a lot of quality is lost. How many passages to pass, in what order, truncated how, should the section title and document date travel with them, and what does the system do when no passage clears a relevance threshold? A system willing to answer "I did not find sufficient grounding" is often more usable than one that always answers.

• Citations: tie each claim to an identifiable passage with its source and version, otherwise the user cannot verify the answer.
• Permissions: filter by the actual rights of the person asking, at search time rather than after generation. A system that filters the answer but leaks a confidential document title already has a problem.
• Freshness: define reindexing frequency, handling of obsolete documents, and expected behaviour when two versions coexist.
• Traceability: for a given answer, keep the query, the selected passages and the configuration version used.

Evaluating retrieval separately from generation

This is probably the most discriminating marker. As long as evaluation targets only the final answer, you are measuring a blend of two stages and cannot tell which one to fix. Separating them answers a simple question: was the passage needed to answer present in what the model received?

1. 1

   Build a set of real questions

   step 1

   Questions users actually ask, not questions reverse-engineered from the documents. Together with domain experts, each question is mapped to the passage or passages that contain the answer. A few dozen well-chosen cases beat a large automatically generated set.

2. 2

   Measure retrieval first

   step 2

   For each question, does the expected passage appear in the returned results, and at what rank? That alone separates a search failure from a writing failure, and the measurement does not depend on the generative model.

3. 3

   Then measure generation

   step 3

   Given identical passages, is the answer faithful to them, complete, correctly cited, and does it abstain when the information is missing? Judgement can be human on a sample, or automated — provided you checked that the automation agrees with the experts on known cases.

4. 4

   Replay on every change

   step 4

   New chunking, new embedding model, new threshold, new prompt, new provider: the same set is replayed. Without that, every improvement stays an impression and regressions go unnoticed until a user reports them.

Case in point: swapping models does not fix bad retrieval

A common situation. An internal assistant answers incompletely on internal procedures. The team concludes the model is too weak and moves to a stronger, more expensive one. Answers become better written, more confident — and remain wrong on the same questions.

Latency, cost, observability, failures

A system queried daily by a whole team does not have the constraints of a demo. Every added stage — hybrid search, reranking, multiple calls — potentially improves quality while adding delay and cost. Making that trade-off means knowing what quality loss is acceptable to stay within a sustainable envelope, and being able to explain it.

• Observability: keep queries, selected passages, scores, per-stage latency and cost per query. Without those traces, every diagnosis becomes a reconstruction.
• Failure modes: provider outage, context overflow, silent degradation after a partial reindex, documents deleted at the source but still indexed.
• User feedback: a simple channel to flag a wrong answer, linked to traces, feeds the evaluation set with real cases.
• Cost: the dominant line item is not always the obvious one; reranking and repeated calls sometimes weigh more than generation itself.

Three levels, and what they let you claim

These are not job titles, they are degrees of autonomy. Placing yourself honestly is stronger than claiming the level above: in an interview the gap shows on the first diagnostic question.

1. 1

   Knowing RAG

   level 1

   Understanding the principle, the vocabulary, the role of each component, and why you do not simply send the question to the model. Enough for a product, project or business role that has to talk with an engineering team. Not enough to claim implementation skill.

2. 2

   Being able to build a prototype

   level 2

   Assembling the components for a use case, making chunking choices, getting correct answers on a corpus you control, and knowing the limits of what you built. That is a real level, and describing it as such is perfectly acceptable. Presenting it as production experience is not.

3. 3

   Being able to run RAG in production

   level 3

   Evaluating retrieval and generation separately, diagnosing a wrong answer, handling rights and freshness, trading quality against cost and latency, instrumenting the system, absorbing a regression and a provider outage, maintaining it as the corpus evolves. This is the level most job ads mean when they write "RAG experience".

How to demonstrate it without overclaiming

What convinces is not the tool list but the chain of decisions. A strong answer looks like: here was the problem, here is what I measured, here is what I changed, here is what it cost, here is what I did not solve. That last part matters as much as the rest: naming a remaining limitation signals level, it does not admit weakness.

If your experience stops at the prototype, say so and describe precisely what you would do before going to production. That is a credible, checkable answer, and it spares you the follow-up question you could not handle.

• RAG
• Evaluation
• Production