Agent Evaluation

Catch regressions in response quality, tool use, latency, and memory.

Changes to models, instructions, tools, and knowledge can introduce regressions. Agno evals turn response criteria and expected tool use into executable cases. Run them during development, gate CI with their exit code, and evaluate selected production outputs through hooks.

import sys

from agno.agent import Agent
from agno.eval import Case, cli
from agno.tools.calculator import CalculatorTools

calculator = Agent(
    id="calculator",
    model="openai:gpt-5.5",
    tools=[CalculatorTools()],
    instructions="Use the calculator tools for every calculation.",
)

CASES = (
    Case(
        name="factorial_uses_calculator",
        agent=calculator,
        input="What is 10 factorial?",
        criteria="States that 10 factorial equals 3,628,800.",
        expected_tool_calls=("factorial",),
    ),
)

if __name__ == "__main__":
    sys.exit(cli(CASES))

Create a virtual environment, install the OpenAI integration, and set OPENAI_API_KEY before running the suite:

uv venv --python 3.12
uv pip install -U "agno[openai]"
uv run python evals.py --json-output tmp/evals.json

Each case runs the component once and applies the configured judge, reliability, and scorer checks to the same output. The CLI returns a nonzero exit code when a case fails, so the suite can gate CI.

Evaluation types

TypeMeasuresGuide
AccuracyCorrectness against an expected answerAccuracy evals
Agent as judgeCustom quality criteria scored by an evaluator modelAgent-as-judge evals
ReliabilityExpected tool calls and argumentsReliability evals
ScorerAn in-process Score from custom code, optionally combined with other case checksScorer cases
PerformanceRuntime latency and memory usePerformance evals

Where evals run

StagePattern
Local developmentRun one case while changing an agent.
CIRun tagged eval suites and keep the JSON report.
ProductionEvaluate selected outputs with a synchronous or background post-hook.
AgentOSStore eval results in a configured database and manage them through the AgentOS API.

Next steps

TaskGuide
Build an eval suiteEval suites
Add evals to an agent platformAgent platform evals
Inspect the API surfaceAgent API