AI agent observability: what to measure in production

In this article

A production error, with and without observability

Today · only the server is watched

1. 1ComplaintA customer got wrong data
2. 2Review"The server is fine"
3. 3No trailNobody knows what it answered
4. 4RepeatsThe error goes on for days

With observability · bank example

1. 1AlertSuccess drops from 95 % to 88 %
2. 2ConversationStep 3: empty balance lookup
3. 3FixThe lookup is corrected
4. 4New caseAdded to the evals

What to monitor: your score

Tick what you already measure. The heaviest items are the ones that take longest to discover without data.

What to monitor in your agent

0 % · 0 of 8 controls that applyFirst missing: full conversation and daily success rateCopy what is missing

Flying blind: if the agent starts answering badly, you find out from a complaint.

Higher-risk controls weigh more in the score.

Logging

• Every conversation is saved in full: message, steps, tools and answer.
• Every conversation records which agent version answered.

Quality

• A daily sample is graded against the rubric and gives a success rate.
• You measure the % of conversations passed to a person.

Cost

• You know what each conversation costs, not just the monthly bill.

Speed

• You measure total response time and the time of each step.

Tools

• There is an alert if a lookup into your systems fails or comes back empty.

Alerts

• Every alert has a named owner and a response time.

Alerts worth having

Signal	What to watch	Alert (example)
Quality	Success rate in the daily sample	Drops 5 points vs. the previous week
Cost	Cost per conversation	Rises more than 50 % day over day
Speed	Time of the slowest step	Doubles
Tools	Lookups that fail or come back empty	Over 2 % in one hour
Handoff	% passed to a person	Rises 10 points

Quality

What to watch

Success rate in the daily sample

Alert (example)

Drops 5 points vs. the previous week

Cost

What to watch

Cost per conversation

Alert (example)

Rises more than 50 % day over day

Speed

What to watch

Time of the slowest step

Alert (example)

Doubles

Tools

What to watch

Lookups that fail or come back empty

Alert (example)

Over 2 % in one hour

Handoff

What to watch

% passed to a person

Alert (example)

Rises 10 points

Why cost is tracked per conversation: switching models multiplies spend without anyone noticing on the server.

Saved conversations are personal data

Saving full conversations means saving customer data: define access, purpose and retention under Law 7593/2025 (Paraguay) or the LOPDP (Ecuador). In banks, the log is also evidence for the BCP's SIB or Ecuador's Superintendencia de Bancos. Cases: Paraguay · Ecuador.

Example: observability in the first 4 weeks

1. Week 1Full loggingMessage, steps, tools, cost and version.KPI: 100 % of conversations saved
2. Week 2BaselineTwo weeks of data before setting alerts.KPI: average success, cost and time
3. Week 3AlertsQuality, cost, speed, tools and handoff.KPI: 5 alerts with an owner
4. Week 4Loop with evalsEvery resolved alert adds a case to the evals.KPI: real failures turned into cases

Is your agent already live and you don't know what it answers? In the free diagnostic we review what to measure first.