Eval Suites

An eval suite runs multiple Cases against your Agents and Teams in one pass. Each case sends one input to one agent or team, then applies any combination of judge (criteria), reliability (expected_tool_calls), and scorer (scorer) checks. The built-in CLI adds case selection, a summary table, a machine-readable JSON report, and exit codes for CI.

Before running these examples, install the dependencies in your Python environment and set your OpenAI key:

uv pip install -U agno openai
export OPENAI_API_KEY="your-api-key"

On PowerShell, use $Env:OPENAI_API_KEY="your-api-key".

evals.py

import sys

from agno.agent import Agent
from agno.eval import Case, cli
from agno.models.openai import OpenAIResponses
from agno.tools.calculator import CalculatorTools

agent = Agent(
    id="math-tutor",
    model=OpenAIResponses(id="gpt-5.2"),
    tools=[CalculatorTools()],
    instructions="Use the calculator tools for any arithmetic.",
)

CASES = (
    Case(
        name="factorial_uses_calculator",
        agent=agent,
        input="What is 10! (ten factorial)?",
        tags=("smoke",),
        criteria="States that 10! equals 3628800.",
        expected_tool_calls=("factorial",),
    ),
    Case(
        name="explains_compound_interest",
        agent=agent,
        input="Explain compound interest in one short paragraph.",
        criteria="Explains that interest is earned on both the principal and previously earned interest.",
    ),
)

if __name__ == "__main__":
    sys.exit(cli(CASES))

python evals.py                 # run all cases
python evals.py --list          # list cases without running
python evals.py --tag smoke     # run a tagged subset
python evals.py --json-output tmp/evals.json

Setting criteria runs an Agent as Judge eval on the response. Setting expected_tool_calls runs a Reliability eval on the tool calls. A case passes when every configured check passes and no error occurred.

Score a response in code

A scorer can check a completed output without a judge model call. It implements ascore(run, expected) and returns Score; the agent or team under test still runs normally. For example, append this case to CASES above:

from agno.scorer import AnyRunOutput, Score


class ExactAnswer:
    async def ascore(self, run: AnyRunOutput, expected=None) -> Score:
        passed = str(run.content).strip() == str(expected)
        return Score(value=float(passed), passed=passed, reason="Exact text comparison")


CASES += (
    Case(
        name="exact_sum",
        agent=agent,
        input="What is 2 + 2? Return only the number.",
        scorer=ExactAnswer(),
        expected="4",
    ),
)

Put this definition before the if __name__ == "__main__" block. The scorer receives the completed RunOutput or TeamRunOutput and runs inside the case timeout. Score.value must be in [0, 1]; the case combines Score.passed with every other configured check. Scorers are skipped for runs that cannot be graded, such as cancelled or paused runs.

Case

Case is a frozen dataclass. Construction raises ValueError unless exactly one of agent or team is set and at least one check (criteria, expected_tool_calls, or scorer) is configured.

Field	Type	Default	Description
`name`	`str`	Required	Case name. Used by the `--name` selector and in results.
`input`	`str`	Required	Input sent to the agent or team.
`agent`	`Optional[Agent]`	`None`	Agent under test. Set exactly one of `agent` or `team`.
`team`	`Optional[Team]`	`None`	Team under test.
`tags`	`Tuple[str, ...]`	`()`	Labels for `--tag` / `tag=` selection.
`timeout_seconds`	`Optional[int]`	`None`	Per-case timeout. Falls back to the runner's `default_timeout` (120s).
`criteria`	`Optional[str]`	`None`	Enables the judge check (`AgentAsJudgeEval`).
`judge_model`	`Optional[Model]`	`None`	Per-case judge model. Falls back to the runner's `judge_model`, then the eval default (`gpt-5-mini`).
`judge_mode`	`JudgeMode`	`JudgeMode.BINARY`	How the judge grades the answer. See the table below.
`judge_threshold`	`int`	`7`	Pass bar (1-10). Read only when `judge_mode` is `NUMERIC`.
`expected_tool_calls`	`Optional[Tuple[str, ...]]`	`None`	Enables the reliability check (`ReliabilityEval`).
`allow_additional_tool_calls`	`bool`	`True`	Whether tool calls beyond the expected ones are allowed.
`scorer`	`Optional[Scorer]`	`None`	Object with `async ascore(run, expected)` returning a `Score`. Enables the scorer check.
`expected`	`Optional[Any]`	`None`	Expected value passed to the scorer.
`setup`	`Optional[Callable]`	`None`	Runs before the case, outside the timeout. Its return value is passed to `teardown`.
`teardown`	`Optional[Callable]`	`None`	Runs once `setup` has completed, on pass, fail, error, or timeout. Receives `(context, result)`.

setup and teardown may be sync or async callables. A teardown failure is recorded on the case result instead of being swallowed.

Judge modes

Mode	Verdict	When to use
`JudgeMode.BINARY` (default)	Pass/fail	Clear-cut criteria: "states that 10! equals 3628800"
`JudgeMode.NUMERIC`	1-10 score, passes when the score meets `judge_threshold`	Graded qualities like tone or clarity. The score is reported as `judge_score` so you can track quality over time.

Running Programmatically

cli() is built on the same runner as run_cases(). To run a suite from code, call run_cases() (or await arun_cases() inside an event loop) and read the SuiteResult.

from agno.eval import run_cases

suite = run_cases(CASES, tag="smoke")
print(suite.status)  # "PASS" or "FAIL"
payload = suite.to_dict()

Parameter	Type	Default	Description
`cases`	`Sequence[Case]`	Required	The cases to select from.
`tag`	`Optional[str]`	`None`	Keep only cases with this tag.
`name`	`Optional[str]`	`None`	Keep only the case with this name.
`default_timeout`	`int`	`120`	Per-case timeout in seconds, when `Case.timeout_seconds` is `None`.
`judge_model`	`Optional[Model]`	`None`	Suite-wide judge model. `Case.judge_model` overrides it.
`db`	`Optional[Union[BaseDb, AsyncBaseDb]]`	`None`	Logs judge and reliability results to storage.
`on_case_start`	`Optional[Callable[[Case], None]]`	`None`	Presentation hook, called before each case runs.
`on_case_end`	`Optional[Callable[[Case, CaseResult], None]]`	`None`	Presentation hook, called with each case and its result.
`on_run_event`	`Optional[Callable[[Case, RunOutputEvent], None]]`	`None`	Presentation hook, called with every streamed run event.

Cases run sequentially on a single event loop. The runner performs no console I/O; all presentation flows through the hooks, which must be sync callables. A hook that raises is recorded on the case as a hook: error without aborting the suite.

An empty selection (for example a mistyped tag) yields status == "FAIL", so a CI gate never passes a run that executed zero cases. A cancelled run aborts the suite; the unrun cases are recorded as failed with skipped=True so the payload accounts for every selected case.

With db= set, results log to storage through the same path as standalone evals. A failed write logs a warning without failing the case, and a hung write cannot stall a case past its timeout. Each case runs in a dedicated eval session (the payload's session_id), so eval traffic stays out of the agent or team's history.

CLI

cli(CASES) parses sys.argv, runs the selected cases with a progress display, prints a summary table, and returns an exit code. Wire it into __main__ with sys.exit(cli(CASES)). Inside an already-running event loop (a server or notebook), use await acli(CASES) instead.

Both accept db=, judge_model=, and default_timeout= keyword arguments, which set the runner defaults behind the flags. Pass mcp_tools= for toolkits used by your case components:

# mcp_tools is the list of MCP toolkits already attached to your case components.
exit_code = cli(CASES, mcp_tools=mcp_tools)

The CLI connects these toolkits in the same event loop as the suite and closes connections it opened. Already initialized toolkits remain caller-owned. Listing or selecting no cases makes no connections. Failed connections produce warnings while cases still run; the checks determine whether each case passes.

Flag	Description
`--name NAME`	Run only the case with this name.
`--tag TAG`	Run only cases with this tag.
`--timeout SECONDS`	Default per-case timeout. Defaults to the `default_timeout` passed to `cli()` (120s).
`--json-output PATH`	Write the machine-readable JSON results to this path.
`--list`	List the selected cases without running them.
`-v`, `--verbose`	Render the full run panels (Message, Tool Calls, Response) after each case.

Exit codes

Code	Meaning
`0`	All selected cases passed.
`1`	Any case failed, or the `--json-output` write failed.
`2`	No cases matched the selector.

JSON report

--json-output writes SuiteResult.to_dict(). The shape is a stable contract for CI consumers.

evals.json

{
  "summary": { "total": 1, "passed": 1, "failed": 0, "status": "PASS" },
  "cases": [
    {
      "name": "factorial_uses_calculator",
      "agent_id": "math-tutor",
      "team_id": null,
      "tags": ["smoke"],
      "session_id": "eval-factorial_uses_calculator-1a2b3c4d",
      "duration_seconds": 6.412,
      "judge_passed": true,
      "judge_reason": "The response states that 10! equals 3628800.",
      "judge_score": null,
      "reliability_passed": true,
      "output": "10! = 3,628,800...",
      "tools_called": ["factorial"],
      "timed_out": false,
      "skipped": false,
      "passed": true,
      "error": null,
      "score_value": null,
      "score_passed": null,
      "score_reason": null
    }
  ]
}

judge_passed and reliability_passed are null when the check is not configured. score_value, score_passed, and score_reason are null when no scorer result exists. Programmatic callers can read the full Score from CaseResult.score. judge_score carries the 1-10 score in numeric mode. error joins every error recorded for the case: run, judge, reliability, setup/teardown, and hooks.

CI usage

The exit code gates the job, and the JSON report is the artifact you keep.

- name: Run evals
  run: python evals.py --tag smoke --json-output tmp/evals.json

Teams

Pass team= instead of agent=. Reliability sees the members' tool calls, so expected_tool_calls can name the tool a member fires rather than only the leader's delegation.

team_evals.py

import sys

from agno.agent import Agent
from agno.eval import Case, JudgeMode, cli
from agno.models.openai import OpenAIResponses
from agno.team import Team
from agno.tools.calculator import CalculatorTools

calculator = Agent(
    id="calculator",
    model=OpenAIResponses(id="gpt-5.2"),
    tools=[CalculatorTools()],
    instructions="Use the calculator tools for every arithmetic operation.",
)
writer = Agent(
    id="writer",
    model=OpenAIResponses(id="gpt-5.2"),
    instructions="Answer in one clear paragraph.",
)
assistant_team = Team(
    id="assistant-team",
    model=OpenAIResponses(id="gpt-5.2"),
    members=[calculator, writer],
    instructions="Delegate arithmetic to the calculator member and writing to the writer member.",
)

CASES = (
    Case(
        name="team_uses_calculator",
        team=assistant_team,
        input="What is 4891 multiplied by 7238?",
        tags=("smoke",),
        criteria="States that the product is 35,401,058.",
        judge_mode=JudgeMode.NUMERIC,
        judge_threshold=7,
        expected_tool_calls=("multiply",),
    ),
    Case(
        name="team_explains_clearly",
        team=assistant_team,
        input="Explain compound interest in one paragraph.",
        criteria="Explains that interest is earned on both the principal and previously earned interest.",
        judge_mode=JudgeMode.NUMERIC,
        judge_threshold=7,
    ),
)

if __name__ == "__main__":
    sys.exit(cli(CASES))