Preparing an AI Engineer interview: the three things actually assessed

What an AI Engineer process actually assesses

Job titles vary from one company to the next, but the interview almost always tries to answer three questions: can you design a system around a model, can you tell whether it works, and can you keep it alive once real users depend on it.

That explains a common frustration. Technically strong candidates get turned down because they treat the interview as a knowledge exam. Knowing model families, vector stores and orchestration frameworks is assumed. What is being assessed is how you decide under constraint.

• Design: turning a vague need into a system whose components each have a reason to exist.
• Evaluation: defining what “it works” means, then measuring it.
• Operations: keeping the system alive over time, with a cost, a latency budget and regressions.

Most processes also include a product or domain conversation. It is not a formality: it is where your ability to discuss a use case without collapsing it into an implementation is judged.

The screening call: place your scope in one sentence

The first conversation exists to work out where you fit. Walking through your CV chronologically wastes it. State your current scope first — what kind of systems you build, for what use, with what level of ownership — then connect it to the need described in the ad.

Designing a system around an LLM

The most common exercise is deliberately open: “how would you build an assistant that answers customer questions from our documentation?” What is observed is the order in which you work. Naming an architecture before framing the problem is often penalised.

1. 1

   Frame the use

   5 min

   Who asks the questions, how often, and what happens when an answer is wrong? The cost of an error drives everything else: guardrails, human in the loop, traceability requirements.

2. 2

   Frame the data

   5 min

   Who owns the documents, how do they change, are they clean, versioned, subject to different access rights? Many systems fail here rather than at the model. The most sensitive case: a system returning a document the user was never allowed to see. Say how permissions follow the data all the way to the answer — filtering at retrieval, separated indexes, traceable cited sources.

3. 3

   Propose a simple first version

   10 min

   A minimal architecture that meets the need, with each component justified. Say explicitly what you are leaving out of version one, and why.

4. 4

   Name the breaking points

   5 min

   Volume, freshness, ambiguous questions, contradictory sources. Describe what would break first and how you would notice.

A strong candidate also says what they rule out: no multi-tool agent until plain retrieval is reliable, no fine-tuning while the problem is one of context rather than model behaviour. An explicit trade-off carries further than an exhaustive answer. When the exercise turns into a full system design round, our dedicated guide to the AI system design interview covers that format; there is no need to replay it here.

Evaluation: the fastest differentiator

“How did you know it was good?” That question splits an interview in two minutes. A vague answer — “we tested it, feedback was positive” — signals a prototype. A precise one signals a system.

• A representative example set, built with the people who understand the domain rather than sampled at random.
• Explicit criteria: correct answer, grounded in sources, no fabrication, usable format.
• A deliberate mix of automated scoring and human review, with the share of each stated.
• Version-to-version comparison: what a prompt, model or retrieval change actually shifts.
• Deliberately hard cases: out-of-scope questions, contradictory documents, ambiguous phrasing.

If you have never built a formal evaluation, say so, then describe how you would build one for the case being discussed. Structured reasoning about a gap you own lands better than a recited generic answer.

Operations: cost, latency, regressions

Many organisations are past the exploration phase. They want people who can hold an existing system, not only start one. Prepare a real incident: what happened, how you saw it, what you changed.

• What drove the cost: context size, number of calls, model choice, caching.
• What you watched continuously, and why: end-to-end latency because it governs whether the system gets used, the rate of failed or off-format calls because it exposes provider drift, the share of questions left unanswered or answered without a cited source because it signals quality loss before users report it.
• How you detected a regression after a prompt change or a model version bump.
• What triggered a rollback, and who made that call.

A model version change that silently degrades quality is one of the most discussed topics in these interviews. Being able to say how you would catch it is often enough.

The coding round: what is really being watched

The coding exercise in an AI Engineer process is rarely a hard algorithmic puzzle. It usually sits around a model call: an output contract to honour, a service that answers badly, a result that changes between two runs.

• An off-format response: validate against the expected schema and define what happens when validation fails, rather than parsing optimistically.
• A model or service call that fails or times out: say what you retry, what you degrade, and what the user gets when nothing answers.
• Non-deterministic outputs: show how you still make testing possible — frozen cases, an explicit tolerance, checking a property rather than an exact string.
• A deliberate fallback when quality is not there: returning “I don't know” with the documents found is preferable to a fabricated answer.

Prepare three examples, not ten projects

A whole process can be covered with three examples worked in depth: a system you designed, an evaluation you set up, a production problem you solved. They can all come from the same project.

Use the same frame for each: the problem, the dominant constraint, two decisions and their reasons, how quality was measured, one limitation you own. That frame lets you answer very different questions without reciting. Our guide on presenting an AI project in an interview sets out the full method for building those examples.

The questions to ask at the end

Your questions signal your level as much as your answers. Three are enough, provided they are about reality.

• Who owns the data this depends on, and what state is it in today?
• Is anything already in production, and who operates it day to day?
• Who owns the roadmap, and how is the system's value judged internally?

The answers will tell you more than the job title. An organisation with no available data and nothing in production usually needs someone to break ground, not someone to operate — and the reverse holds too.

• Interviews
• AI Engineering
• Evaluation
• Production