Product Owner in the AI era: which skills become essential?

What actually changes in the role

A Product Owner working on a product with an AI component does not switch jobs: they still own user value, a backlog, trade-offs and stakeholder relationships. What changes is the nature of the thing being shipped. A conventional feature is described by rules: given this input, the system returns exactly that. A feature built on a language model or a recommendation system produces behaviour, not a rule. Two similar phrasings can return different answers, and the same phrasing can be fine today and mediocre tomorrow because the retrieved context changed.

The practical consequence is that accepting a feature after testing it once no longer works. The PO has to describe what is expected on average, what is unacceptable in every case, and what should happen when the system is wrong. That is a specification and evaluation skill, not an engineering one.

A PO does not need to train models or read inference code. They need to frame a problem, define what a good answer means, and decide how much error the business can absorb.

Frame the use case before talking about models

Requests rarely arrive as problems. They arrive as solutions: "we want an assistant", "we want tickets summarised", "we want an agent that handles requests". The PO's first job is to step back one level: which human decision or task are we trying to speed up, who does it today, how often, and what happens when it is done badly?

• Which precise task is targeted, and where it sits in the journey.
• Who the real user is: an end customer, an agent, an internal operator.
• What the expected output is: text, a classification, a suggestion, an action.
• What an error costs: wasted time, a wrong decision, a contractual risk.
• What alternative already exists, and why it is not enough.

This framing drives everything else. An internal drafting assistant whose output is always reviewed tolerates a high imperfection rate. A system that sends a reply straight to a customer does not. That is a question of accountability, not of model choice.

Define user value, not technical novelty

Many AI features ship because they are possible, not because they are useful. The PO remains the guardian of the value question: what can the user do after release that they could not do before, and how will we see it? The answer has to be observable — a shorter delay, a step removed, information found without switching tools, a decision documented faster.

A value indicator is not the same as a model quality indicator. The latter measures system behaviour; the former measures the effect on real work. A system can score well internally and still go unused, because it arrives too late in the journey or because users do not trust it.

From acceptance criteria to evaluable criteria

A classic acceptance criterion is binary: the feature meets it or it does not. With probabilistic behaviour the question becomes: across a representative set of cases, what share of outputs is acceptable, and by whose definition of acceptable? The PO does not need statistical metrics, but must supply the raw material used to build them: real examples, expected outputs, and examples of unacceptable outputs.

Build a test set and work the edge cases

The test set is the most durable asset a PO can produce on an AI product. It survives changes of model, prompt and vendor. Keep it small but representative: ordinary cases, frequent painful cases, and edge cases chosen for their consequence rather than their strangeness.

• Nominal cases: what the system will meet most of the time.
• Ambiguous cases: partial information, vague wording, contradictory requests.
• Out-of-scope cases: what the system must decline to answer.
• Sensitive cases: personal data, contractual commitments, amounts, health, legal matters.
• Regression cases: errors already observed, kept forever.

Each case carries an expectation written in business language. That collection, reviewed regularly, is what lets the engineering team compare two versions objectively.

Hallucinations, wrong answers and acceptable quality

A generative system can produce a confident, wrong answer. That is expected behaviour for this class of system, not an isolated bug to be fixed once. So the PO decides, use case by use case, what is absorbable and what is not, then shapes the product around that decision.

• Visible error the user can correct: often acceptable when correction is cheap.
• Invisible error that propagates into another system: rarely acceptable without a control.
• Error touching a commitment, a price or a legal obligation: designed out, not warned against.

Three product levers exist before any model work: narrow the scope of the feature, require the answer to cite its source, and design the "I don't know" path explicitly. A system able to abstain is almost always more usable than one that always answers.

Decide where the human sits in the loop

Human-in-the-loop is not a checkbox, it is a design choice with a cost. Reviewing every output protects quality but often cancels the time saved. Reviewing nothing maximises the gain but pushes the risk onto the end user. In between there are positions the PO must choose deliberately: review only outputs flagged uncertain, sample-based review, mandatory approval only before an irreversible action.

A useful framing question: "what is the last action after which the error becomes expensive?" Human control belongs just before that action, not everywhere.

Trading quality, latency and cost

On an AI product three variables move together. A more reliable answer usually needs more context, more steps or a heavier model: it costs more and arrives later. These trade-offs are product decisions, not technical ones. The PO should be able to say when the user prefers to wait, and when an imperfect immediate answer beats a perfect late one.

Cost deserves to be treated as a design constraint from the framing stage, exactly like performance. A feature whose unit cost exceeds the value it creates will not scale, however good the demo was.

Data, dependencies and working with the team

An AI feature rarely depends on the model alone. It depends above all on what it is fed: up-to-date documents, access scope, freshness, structure. A PO who ignores the real state of the sources risks discovering late that the promised product cannot be supplied.

• Which sources are required, and who owns them in the organisation.
• Who may see what: an assistant must never widen a user's permissions.
• How often the data changes, and what an answer based on a stale version becomes.
• Which traces are kept so an answer can be explained afterwards.

With AI engineering and data teams, a PO benefits from separating what they require (expected behaviour, forbidden cases, thresholds) from what they do not impose (architecture, model choice, retrieval strategy). That boundary avoids the worst of both worlds: dictating a technical solution while remaining accountable for the business outcome.

A concrete case: a user story that does not hold

A support team wants to offer agents a pre-drafted reply. The initial story: "As an agent, I want a reply suggested automatically so I save time. Acceptance criterion: the suggested reply is correct." This story is untestable: "correct" is undefined, no scope is set, no behaviour is planned for doubt, and nothing says what happens when the reply relies on an outdated procedure.

The same need, framed differently, becomes workable:

• Scope: order-tracking requests only, in English, on orders from the last 90 days.
• Allowed sources: the published procedure base and order status; nothing else.
• Expected behaviour: the suggestion cites the procedure used and the order number.
• Abstention: if the status is missing or the procedure ambiguous, the system offers an empty template and flags the uncertainty instead of inventing.
• Hard limits: no commitment on delivery dates, refunds or goodwill gestures in the suggestion.
• Control: the reply is never sent automatically; the agent approves or edits it.
• Evaluation: across 80 anonymised real conversations, at least 60 suggestions are sent with at most a minor edit, and zero suggestion contains a forbidden commitment.
• Follow-up: actual adoption rate, heavy-rewrite rate, agent reports.

Nothing here assumes technical skill. Everything is product work: scope, sources, behaviour under doubt, hard limits, controls and measurement. That is exactly what an AI team expects from a PO.

From POC to a feature people can actually use

A successful demo proves a behaviour is possible on a handful of chosen examples. A usable feature assumes something else: stable behaviour on cases nobody picked, error handling, a trace of decisions, controlled cost and a team able to keep it alive. That gap is where many AI projects stop, usually because it was never anticipated during framing.

After release

An AI product degrades without anyone touching the code: usage drifts, documents change, users phrase things differently. Post-release monitoring is therefore part of the PO's scope, just like the backlog.

• Real usage: who uses the feature, how often, and who dropped it.
• Reports: a simple way to flag a problematic answer, reviewed regularly.
• Re-evaluation: replay the test set on every change of model, prompt or source.
• Cost: cost per use and how it moves with volume.
• Decisions: what changed, when, and the observed effect.

Questions a PO should know how to ask

• What data does this answer rely on, and how is it selected?
• What does the system do when it cannot find the information?
• How will we know a new version is better than the current one?
• Which cases did we decide not to handle, and is that visible to the user?
• What does one call cost, and how does that move if usage doubles?
• How long does an answer take in the worst case, not on average?
• What can we explain to a user who disputes an answer?
• Which part of the behaviour depends on an external vendor, and what happens if it changes?

These questions are not there to police the engineering team. They exist to make explicit the choices that would otherwise be made implicitly — and usually discovered in production.

• AI product
• Use-case framing
• Evaluation