How to build an AI agent that takes actions for users
Scope, evals and guardrails for an agent people let touch real money
1 min read
TL;DR
- Start with one task that is boring, frequent and easy to check.
- Write the evals before the prompts.
- Ask for confirmation on anything that costs money.
The problem
Users want the outcome, not the steps. An agent that books, buys or files for them saves real time. It also makes real mistakes, so trust is the product.
The VP calls
The call
Which task first?
High value · High frequency · Easy to verify
Easy to verify. You can't improve what you can't check.
The call
How much autonomy?
Suggest · Confirm · Act
Confirm for v1. Move to act task by task, as evals allow.
The build
Define done
Owner: PM1 week
Write the task, the inputs and what a correct result looks like.
Build the eval set
Owner: PM + Engineering2 weeks
Collect 100 real requests and grade the expected result for each one.
Mistakes to avoid
- Launching with many tools before one works well.
- Letting the agent spend money with no confirm step.
Go to market
Price per completed task, not per seat. It matches what the user values.
Checklist
0/3Next episode
How to build sponsored answers in an AI assistant
An ad unit for a world with no results page and no click