Evaluation, Error Analysis & Tuning

Before you can evaluate an agent you have to decide what 'good' looks like. This topic teaches you to translate a business goal into measurable success criteria and the concrete signals — task completion, tool-call accuracy, groundedness, safety — that prove the agent met them.

Healthy agent programmes combine cheap quantitative signals that run on every build with slower qualitative signals that catch what numbers can't see. This topic shows how to pair them so neither side ships blind.

Automated scanning is how you catch the bad change before users do. This topic covers running evaluators in CI/CD, gating releases on thresholds, scheduling production evaluation, and using continuous evaluation plus red teaming to detect regressions and drift.

When an agent run goes wrong, the trace is your crime scene. This topic shows how to read a Foundry / OpenTelemetry-style trace, identify the failing span, and use the surrounding artifacts (instructions, tool args, retrieved context, model output) to assign root cause.

A useful failure taxonomy turns a vague 'the agent broke' into an actionable fix. This topic teaches the three core agent failure modes — reasoning, tool, and context — and how to match observable signals to each so you fix the right layer.

Tuning an agent is rarely about changing the model — it is about tightening the instructions, the workflow steps, and the constraints around tool use. This topic walks through how to iterate on those levers safely using evaluators as a guard.

Once instructions and workflow are tight, the next levers are the agent's memory layer and its tool catalog. This topic shows how to refine what the agent remembers and which tools it can pick from — using evaluator signals to decide what to add, prune, or scope.