Evals | Agno

Evals are regression tests for your agents. Rerun the same prompts against the same agents and behavior drift becomes visible.

/improve-agent generates probes from an agent's instructions to find new weaknesses. Evals preserve known behavior as repeatable cases.

Cases

Cases live in evals/cases.py. Each case sends one input to an agent (agent=) or team (team=) and optionally checks two things:

judge: AgentAsJudgeEval scores the response against criteria (binary pass/fail) using an LLM.
reliability: ReliabilityEval checks which tools fired against expected_tool_calls.

The template includes this live-web case. Keep its imports and cleanup hooks when adding it to evals/cases.py:

evals/cases.py

from os import getenv

from agno.eval import Case
from evals.hooks import LEARNING_HOOKS
from teams.lead import agno_team

# The team uses Parallel with a key, or the keyless web tools otherwise.
_WEB_TOOL = "parallel_search" if getenv("PARALLEL_API_KEY") else "web_search"

CASES: tuple[Case, ...] = (
    Case(
        name="agno_answers_from_live_web",
        team=agno_team,
        input="What has the James Webb Space Telescope found recently? Just tell me — no need to file it.",
        tags=("live",),
        timeout_seconds=120,
        **LEARNING_HOOKS,
        criteria=(
            "Answers the question by citing at least one real URL from the fetched "
            "results (nasa.gov, webbtelescope.org, or another real source domain). "
            "The response is grounded in fetched content rather than refusing to answer."
        ),
        expected_tool_calls=(_WEB_TOOL,),
    ),
    # Keep the template's other cases here.
)

A case can use either check or both. If both are set, the target runs once and feeds the same response into both.

Cases can write to the platform's shared stores. Keep LEARNING_HOOKS for learning-store cases and BUILDER_HOOKS for cases that can reach component creation or editing, including through delegation. These hooks remove newly created records; they cannot undo changes to existing records. Use fictional, unique fixtures and a dedicated test database for development.

Add tags to group your cases into suites. The template uses three tags: smoke, release, and live. This case uses the live tag because its answer depends on the open web.

Run the suite

The suite runs on the host, calls the model, and logs results to your local Postgres through eval_db. Start the platform first (docker compose up -d) and make sure .env has your OPENAI_API_KEY.

Create a virtual environment

The eval suite runs on the host and needs a local virtual environment:

./scripts/venv_setup.sh

Activate it:

source .venv/bin/activate

Run the eval suite

python -m evals --tag smoke     # fast suite
python -m evals                # full suite

Other options:

python -m evals -v             # stream the agent run with full panels
python -m evals --name <case>  # single case while iterating

Each case prints its response and the verdicts for the checks it defines. The run ends with an Eval Summary table.

Results write to Postgres via eval_db. The eval history shows up on os.agno.com alongside your sessions and traces, so you can see when a case started failing and what changed.

Diagnose failures with your coding agent

Run /create-evals to add coverage for an agent. The skill maps the agent's behavior, proposes cases, writes them to evals/cases.py, and verifies the new cases.

Open your coding agent and run:

/eval-and-improve

The coding agent runs the suite, triages every failure (bad criteria, real regression, flaky LLM judge), and proposes in-scope fixes. It edits the agent or the case and re-runs until the suite is green.

When to run evals

Trigger	Frequency
Before deploying a change to an agent	Every time
As part of CI	Every PR
Against a dedicated test environment	On a daily schedule
After bumping a model version	Every time

The template registers a daily run-evals schedule in the disabled state because it uses model calls. Enable it from the AgentOS UI when you want the smoke-tagged cases to run daily. See Scheduler for the cron API.

What good cases look like

Specific. "Returns a JSON object with ticker and price" beats "Returns the right answer".
Stable. Avoid prompts whose correct answer changes daily. Use phrasing like "describes a real, recent..." instead of locking in a specific result.
Scoped to one behavior. One case per behavior makes failures easy to read.
Anchored to tools. expected_tool_calls catches the failure mode where the agent confidently makes things up instead of calling a tool.

Next steps