AI agent product management
Designing an AI Agent Product: From User Problem to Evaluation Framework
An AI agent product management framework for designing autonomy, human-in-the-loop workflows, and reliable evaluation.
The first question in an AI agent project should not be “Which agent framework should we use?”
It should be: What workflow are we improving, and why does it require agency?
Turning an agent demo into a reliable enterprise product is a different challenge.
I learned this while developing a multi-agent operations prototype. The crucial decisions concerned autonomy, evidence, failure, and measurable value.
Do you need an agent?
Many products are better served by automation, search, classification, or an AI assistant.
An agent becomes useful when a task requires interpreting a goal, selecting tools, adapting its sequence, or escalating incomplete cases.
Before choosing an agentic product strategy, I ask:
- Is the workflow variable?
- Must the system choose among bounded actions?
- Is the required context current and authorised?
- Can experts evaluate acceptable behaviour?
If the last two answers are no, the team is not ready to grant meaningful autonomy.
Define the outcome
“Automate operations with multiple agents” is not a product definition.
A stronger framing is:
Reduce the time specialists spend preparing operational cases while maintaining escalation accuracy and expert control over high-risk decisions.
In the evaluated prototype, the workflow reduced manual triage time by approximately 90% and improved escalation accuracy by approximately 60%, connecting system behaviour to operational value.
Map the workflow first
Document the trigger, required information, decisions, errors, and accountability.
Then classify each step:
- Deterministic automation: rules execute an action.
- AI recommendation: the system suggests an action.
- AI-drafted action: a human approves before execution.
- Bounded execution: the agent acts within permissions.
- Human-only decision: autonomy is prohibited.
The architecture should emerge from clear responsibilities, not the desire to maximise agents.
Make autonomy progressive
I treat autonomy as a ladder:
- Retrieval: present relevant information.
- Recommendation: propose the next step.
- Drafted action: prepare an action for approval.
- Bounded execution: act within defined permissions.
- Adaptive workflow: plan several steps within policy boundaries.
Start at the lowest level that creates meaningful value. Increase autonomy only when evaluation, observability, and user trust support it.
For each agent, define its objective, permissions, boundaries, escalation, evaluation, and owner.
Build evaluation into the product
Evaluation is part of the specification, not a final quality gate.
An AI agent evaluation framework may include:
- Task success.
- Decision and escalation quality.
- Relevance of retrieved evidence.
- Correct tool use.
- Safety and policy compliance.
- Human-review effort.
- Latency, reliability, and cost.
- Business outcomes.
No single score captures this. An agent may complete tasks while making serious errors.
Tests should include normal, ambiguous, contradictory, and high-risk scenarios, including cases where the correct behaviour is to stop.
Design human review
“A human will review it” is not a control unless the process works.
Reviewers need the request, evidence, uncertainty signals, and a correction mechanism. Reviewed corrections can feed evaluation datasets, not automatic retraining.
Monitor workload too. If reviewers repeat the whole analysis, the agent may have moved work rather than removed it.
Launch progressively: map the workflow, test a prototype, build evaluations, run a bounded pilot, and automate only reversible actions supported by evidence.
The best agent is not the most autonomous one in a demo. It is the one that improves a real workflow reliably, within clear boundaries, at an acceptable cost.
Agency should be earned.