What the exercise actually measures
An AI system design prompt is almost always one sentence too short: "design an assistant that answers internal questions", "propose a system that triages incoming requests". The vagueness is deliberate. What is being observed is not your ability to produce a complete diagram in forty minutes, but how you move forward while the problem is still undefined.
Three things are watched in parallel: how you frame, how you choose while accepting a trade-off, and how you talk about failure. A candidate who draws a full architecture immediately loses the first. One who stacks components without hierarchy loses the second. One who describes a system without ever mentioning its failure modes loses the third.
This guide covers the design exercise only. Preparing for the rest of an AI Engineer loop, with its other formats, is a separate subject.
Frame before you design
The first minutes turn a request into a specification. What is expected is that you ask questions out loud, and say why each answer would change the design. A question whose answer changes nothing is not worth asking in an interview.
User and decision
Who uses the system, at what point in their work, and what do they decide next? An assistant used by a support agent during a live call has nothing in common with an analysis tool opened once a week. Usage frequency, the user's expertise and the cost of an error for them drive most of the trade-offs that follow.
Functional and non-functional constraints
- Functional: which questions are in scope, which must be refused, what answer format is expected.
- Volume: number of users, peak traffic, corpus size and update frequency.
- Latency: perceived as instant, a few seconds tolerated, or asynchronous processing acceptable.
- Accuracy: what a wrong answer costs, and whether a source must be cited.
- Deployment: hosting, region, and which data may or may not leave the internal systems.
Available data and access rights
Data comes before the model. Where does it live, in what formats, who is allowed to read it, and does that right vary by user? A corporate document corpus is almost never uniform in permissions. Asking early avoids designing a system that has to be rebuilt at the security review.
Choose an architecture, and justify it
The expected progression runs from simple to complex, each layer introduced by a need that came out of the framing. Opening with the most elaborate architecture is a method error, even when it turns out to be needed.
- No retrieval: a model call with instructions and application-provided context is enough when scope is narrow and knowledge stable.
- Retrieval (RAG): justified when the answer depends on a large, evolving corpus, or when the source must be cited.
- Tools: as soon as the answer requires transactional data, a computation or an action in another system.
- Multi-step or agentic orchestration: reserved for genuinely decomposable tasks with verifiable steps, and only by accepting the latency and variability cost out loud.
Phrase each addition as a consequence, not a preference: "because the corpus changes weekly and citing the source is mandatory, I add a retrieval layer". That sentence is what the panel remembers, more than the diagram.
Retrieval and context management
In a retrieval-based system, perceived quality rarely comes from the model. It comes from what the model is given to read. So the indexing and selection decisions are the ones to detail.
- Chunking: by unit of meaning rather than fixed size, keeping title, section and original date.
- Metadata: source, scope, version, rights — used for filtering as much as for citation.
- Selection: lexical and vector search combined when domain vocabulary is specific, reranking when precision matters most.
- Context budget: how many passages you keep, how you handle contradictory documents, what happens when nothing relevant is found.
- Freshness: reindex on update, and make deletions effective.
The "no relevant passage" case deserves an explicit answer. A system that replies anyway is a system that invents.
The model's role, without a benchmark
You are not asked to crown a model. You are asked what role it plays and under which constraints it is picked: usable context length, cost per request at the target volume, acceptable latency, ability to return structured output, hosting compatible with the data involved. Naming those criteria and saying the decision is settled by a run on the evaluation set is a stronger answer than naming a model.
The real differentiator is isolation: the system must survive a model swap. A single call interface, versioned instructions and schema-validated output make that swap possible without a rewrite.
Evaluation and failure handling
This is the most frequently skipped part, and the one that separates levels most clearly. Designing without saying how you measure means shipping an intuition.
- An evaluation set built from real cases, including out-of-scope and ambiguous questions.
- On retrieval: are the right passages returned, and at which rank.
- On the answer: accuracy, grounding in the retrieved sources, compliance with the expected format.
- Periodic human review on a sample, for what automation cannot capture.
- A tracked rate of refusals and of answers produced without a source.
On failure, describe expected behaviour: provider outage, timeout, output that fails schema validation, no source found. An explicit fallback — partial answer, handover to a human, unavailability message — beats an answer produced at any cost.
Latency, cost and observability as design constraints
These three are not end-of-project adjustments; they decide the architecture. A two-second requirement rules out five sequential calls. High volume forces a cache, a smaller transmitted context, or a lighter model on intermediate steps.
- Latency: break it down per stage (retrieval, generation, post-processing) and say which dominates.
- Cost: reason per request and at target volume, naming the main lever instead of promising general optimisation.
- Observability: trace every request, the retrieved passages, the instruction and model versions, per-stage latency and validation results.
- Drift: watch behaviour change after a corpus update or a provider-side model update.
Security and permissions
A question-answering system over an internal corpus is an information access system. Permissions must be enforced at retrieval time, filtered on the user's identity, not after generation: a confidential passage that enters the context has already leaked, even if the final answer looks harmless.
- Filter documents by the user's rights at the index level.
- Treat instructions embedded in retrieved documents as data, never as directives.
- Scope tools tightly: minimal permissions, human confirmation on sensitive actions.
- Log access, handle personal data, and set a retention period for traces.
Worked case: an internal question-answering assistant
Deliberately vague prompt: "a support team spends too much time searching internal documentation, design an assistant". Here is how to unfold it.
- 1
Frame
5 minTier-1 support, during a live call, answer expected within seconds, source mandatory so the agent can verify. Corpus: product documentation and procedures, updated weekly, with sections restricted to certain teams.
- 2
State the simple version
5 minSearch the corpus filtered by rights, generate a short cited answer, refuse explicitly when no relevant passage is found. No agent, no tools: the task is grounded question answering.
- 3
Detail the retrieval chain
10 minSection-level chunking with titles kept, scope and version metadata, hybrid search to absorb product vocabulary, reranking so only a few passages are passed on.
- 4
Define evaluation
8 minA hundred real questions taken from tickets, ten of them out of scope. Measure passage recall, answer accuracy and the share of answers without a source. Monthly human review on a sample.
- 5
Handle the constraints
7 minLatency dominated by generation: stream the answer, cache frequent questions. Security: rights filtering at the index. Observability: full per-request trace, alert on refusal rate.
- 6
Own the limits
5 minNo answers on individual contractual cases, handover to tier 2. Possible extension to customer record lookups, but that means tools and a separate security review.
Weak answer
"I would build a RAG with a vector database and an agent that calls tools when needed, with a good model and a cache for performance."
Credible answer
"The hard constraint is use during a live call: a few seconds and a verifiable source. So I start with rights-filtered retrieval and grounded generation, no orchestration. I would add tools only for customer record lookups, which pushes latency past the target — that becomes a second, asynchronous path."
The second answer has no more components. It has the reasons.
Own the trade-offs and the limits
Close by stating what the system does not do, what you sacrificed and why, and what you would measure first after go-live. That last minute often decides the evaluation: it shows you design for a real context rather than for a diagram.
The same demand for justification applies when you talk through a project you have already delivered: the reasoning matters more than the stack. Different exercise, same lens.
Before the design exercise
Seven habits to have ready. The one you cannot tick is the one to work on first.
- I ask at least three framing questions before drawing anything.
- I can name the actual user and the decision they make next.
- I start from the simplest solution and justify every layer I add.
- I discuss data and access rights before discussing the model.
- I describe how I measure the system before describing how I improve it.
- I treat latency, cost and security as design constraints, not operational details.
- I close by stating the limits I accept and what I would measure first in production.
Key takeaways
- The exercise scores your framing as much as your solution.
- An architecture only counts in an interview if you can name the option you rejected.
- Retrieval and permissions are designed together: an index that ignores rights is an incident, not an optimisation.
- Without an evaluation protocol, an AI system is not designed, only hoped for.
- Latency and cost shape the architecture; they are not fixed afterwards.
- Stating a limit is stronger than promising full coverage.
- AI Engineering
- Evaluation
- Cloud & production