Evals

The reliability case verifies the assistant calls the calculator's ``multiply`` tool with the expected arguments.

evals.py
"""Run accuracy and reliability evaluations through AgentOS.

The reliability case verifies the assistant calls the calculator's
``multiply`` tool with the expected arguments.

Prerequisites: start ``_server.py`` and set OPENAI_API_KEY.
Run: .venvs/demo/bin/python cookbook/05_agent_os/03_python_client/05_evals.py
Try: compare the stored accuracy and reliability result payloads.
"""

import asyncio

from agno.client import AgentOSClient
from agno.db.schemas.evals import EvalType

BASE_URL = "http://localhost:7778"
AGENT_ID = "assistant"

# ---------------------------------------------------------------------------
# Create the Client
# ---------------------------------------------------------------------------

async def run_evaluations() -> None:
    """Run two evaluations, then list and fetch their stored results."""
    client = AgentOSClient(base_url=BASE_URL)

    accuracy = await client.run_eval(
        eval_type=EvalType.ACCURACY,
        input_text="What is 2 + 2?",
        agent_id=AGENT_ID,
        expected_output="4",
    )
    if accuracy is None:
        raise RuntimeError("Accuracy evaluation returned no result")
    print(f"Accuracy evaluation: {accuracy.id}")
    print(accuracy.eval_data)

    reliability = await client.run_eval(
        eval_type=EvalType.RELIABILITY,
        input_text="Use the calculator to multiply 10 by 5.",
        agent_id=AGENT_ID,
        expected_tool_calls=["multiply"],
        expected_tool_call_arguments={"multiply": {"a": 10, "b": 5}},
    )
    if reliability is None:
        raise RuntimeError("Reliability evaluation returned no result")
    print(f"Reliability evaluation: {reliability.id}")
    print(reliability.eval_data)

    eval_runs = await client.list_eval_runs(
        agent_id=AGENT_ID,
        eval_types=[EvalType.ACCURACY, EvalType.RELIABILITY],
    )
    print(f"Stored evaluations: {len(eval_runs.data)}")

    for run_id in (accuracy.id, reliability.id):
        detail = await client.get_eval_run(run_id)
        print(f"Fetched {detail.id}: {detail.eval_type.value}")

# ---------------------------------------------------------------------------
# Run the Example
# ---------------------------------------------------------------------------

if __name__ == "__main__":
    asyncio.run(run_evaluations())

This program calls the shared server on port 7778. The steps below start it without authentication for local use. The OpenAI key belongs in the server's environment; the client sends HTTP requests and does not need its own model key.

Run the Example

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U "agno[os]" chromadb openai

Export your API keys

export OPENAI_API_KEY="your_openai_api_key_here"

Clone Agno

Clone the pinned Agno source and run the remaining commands from its root:

git clone https://github.com/agno-agi/agno.git
cd agno
git checkout v3.0.4

Start the AgentOS server

Open another terminal in the parent directory where .venv and the cloned agno directory are siblings, then start the shared server on port 7778:

source .venv/bin/activate
export OPENAI_API_KEY="your_openai_api_key_here"
unset OS_SECURITY_KEY
cd agno
python cookbook/05_agent_os/03_python_client/_server.py

Run the example

Run the example from the repository root:

python cookbook/05_agent_os/03_python_client/05_evals.py

Full source: cookbook/05_agent_os/03_python_client/05_evals.py

Calling a protected server

Set OS_SECURITY_KEY to the same value in the server and client environments, then restart the server. Add this configuration to the client program:

import os

auth_headers = {"Authorization": f"Bearer {os.environ['OS_SECURITY_KEY']}"}

Pass headers=auth_headers on every call to run_eval, list_eval_runs, and get_eval_run, including calls inside helpers or loops. AgentOSClient does not read OS_SECURITY_KEY automatically and does not accept constructor-wide headers.