Reliability Evaluation with Database Logging

Persist ReliabilityEval tool-call results to a Postgres eval_runs table via PostgresDb.

Demonstrates storing reliability evaluation results in PostgreSQL.

db_logging.py
"""
Reliability Evaluation with Database Logging
============================================

Demonstrates storing reliability evaluation results in PostgreSQL.
"""

from typing import Optional

from agno.agent import Agent
from agno.db.postgres.postgres import PostgresDb
from agno.eval.reliability import ReliabilityEval, ReliabilityResult
from agno.models.openai import OpenAIChat
from agno.run.agent import RunOutput
from agno.tools.calculator import CalculatorTools

# ---------------------------------------------------------------------------
# Create Database
# ---------------------------------------------------------------------------
db_url = "postgresql+psycopg://ai:ai@localhost:5432/ai"
db = PostgresDb(db_url=db_url, eval_table="eval_runs")

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------
agent = Agent(
    model=OpenAIChat(id="gpt-5.2"),
    tools=[CalculatorTools()],
)

# ---------------------------------------------------------------------------
# Create Evaluation
# ---------------------------------------------------------------------------

# ---------------------------------------------------------------------------
# Run Evaluation
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    response: RunOutput = agent.run("What is 10!?")
    evaluation = ReliabilityEval(
        db=db,
        name="Tool Call Reliability",
        agent_response=response,
        expected_tool_calls=["factorial"],
    )
    result: Optional[ReliabilityResult] = evaluation.run(print_results=True)
    if result:
        result.assert_passed()

Run the Example

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U agno "psycopg[binary]" openai sqlalchemy

Export your OpenAI API key

export OPENAI_API_KEY="your_openai_api_key_here"

Start Postgres

Start Postgres on the port used by this example:

docker run -d --name postgres -e POSTGRES_USER=ai -e POSTGRES_PASSWORD=ai -e POSTGRES_DB=ai -p 5432:5432 postgres:17

Run the example

Save the code above as db_logging.py, then run:

python db_logging.py

Full source: cookbook/09_evals/reliability/db_logging.py