In this article
One change to the agent, with and without evals
Today · tested by eye
- 1ChangeSomeone tweaks the agent
- 23 manual tries"Looks fine"
- 3Goes liveNo threshold
- 4ComplaintThe customer finds the error
With evals · example 5 minutes per run
- 1ChangeSomeone tweaks the agent
- 250 cases, automaticAnswer vs. the correct one
- 3ThresholdUnder 95 %: it does not ship
- 4New failureAdded as a case
What gets tested: 3 levels
| Level | What it checks | Bank example | Red flag |
|---|---|---|---|
| 1 · Tools | Each lookup into your systems on its own | Balance lookup with a mistyped account number | Returns another account's data |
| 2 · Steps | The order in which the agent uses its tools | Asks for confirmation before blocking a card | Acts without confirming |
| 3 · Answer | What the customer reads, against a written rubric | SME loan requirements under the current policy | Quotes an expired policy |
1 · Tools
- What it checks
- Each lookup into your systems on its own
- Bank example
- Balance lookup with a mistyped account number
- Red flag
- Returns another account's data
2 · Steps
- What it checks
- The order in which the agent uses its tools
- Bank example
- Asks for confirmation before blocking a card
- Red flag
- Acts without confirming
3 · Answer
- What it checks
- What the customer reads, against a written rubric
- Bank example
- SME loan requirements under the current policy
- Red flag
- Quotes an expired policy
The 4 numbers that decide if it ships
| Metric | What it measures | Threshold to ship (example) |
|---|---|---|
| Success rate | % of cases solved per the rubric | 95 % or more |
| Handoff to a human | % of cases the agent passes to a person | Under 20 %, and always for sensitive cases |
| Cost per solved case | Model + infrastructure ÷ solved cases | Lower than the person's cost |
| Response time | Seconds to the final answer | Under 10 s in chat |
Success rate
- What it measures
- % of cases solved per the rubric
- Threshold to ship (example)
- 95 % or more
Handoff to a human
- What it measures
- % of cases the agent passes to a person
- Threshold to ship (example)
- Under 20 %, and always for sensitive cases
Cost per solved case
- What it measures
- Model + infrastructure ÷ solved cases
- Threshold to ship (example)
- Lower than the person's cost
Response time
- What it measures
- Seconds to the final answer
- Threshold to ship (example)
- Under 10 s in chat
What it costs and when it pays back
What evals cost and when they pay back
1 · What errors cost today
2 · What evals cost
- Model (tests): gpt-6-lunaUSD 0.38 · 0.9 %
- TeamUSD 40.09 · 99 %
- ToolUSD 0.00 · 0.0 %
Pays back in 9.6 months. Start with 50 real cases.
estimatedModel prices: OpenAI API pricing, checked 11 Oct 2026 (USD per million tokens, input / output): gpt-6-luna 0.10 / 0.50; gpt-6.1-sol 2.00 / 10.00. Each case = one agent answer (4,000 in + 500 out) + one grading call against the written rubric (2,000 + 200).
Hourly cost: Paraguay, minimum Gs. 12,683 per hour (MTESS, Res. 670/2026) at Gs. 5,694.47 per USD (BCP, 9 Oct 2026) = USD 2.23. Ecuador: basic wage USD 482 ÷ 173.33 h (Ministry of Labor) = USD 2.78. The multiplier is an example for a skilled profile.
Net savings = hours not lost to errors × hourly cost − total cost. Months = setup ÷ net savings. Excludes lost customers and fines, so the real value is usually higher. Error %, % stopped, minutes, tool and setup are examples.
Customer data in tests
Use real cases without personal data: replace names, ID numbers and accounts before saving them. In Paraguay, Law 7593/2025 requires it (in force from 27/11/2027, art. 57); in Ecuador, the LOPDP. In banks, keep the report of every run: it is your evidence for the BCP's SIB or Ecuador's Superintendencia de Bancos. Country cases: Paraguay · Ecuador.
Your first 50 test cases
Example evals rollout
- Week 1CasesReal history, past errors and cases to refuse.KPI: 50 cases with approved answers
- Week 2Rubric and thresholdWhat counts as a correct answer and when it does not ship.KPI: threshold signed off by the business
- Week 3Automatic runThe agent does not ship below the threshold.KPI: report on every change
- Week 4Loop with productionEvery new complaint joins the exam.KPI: real failures turned into cases
Is your agent already talking to customers without an exam? In the free diagnostic we review your cases and tell you where to start.