Background Output Evaluation

Use Agent as Judge evaluation to assess the main agent's output as a background task. Unlike blocking validation, background evaluation:

Does NOT block the response to the user
Logs evaluation results for monitoring and analytics
Can trigger alerts or store metrics after the main response

Use cases:

Quality monitoring in production
Compliance auditing
Detecting hallucinations or inappropriate content

These examples use background callbacks attached to a normal AgentOS HTTP response. They execute after the response completes in the serving process; they are not durable queue jobs. A process shutdown or callback failure can prevent pending callbacks from finishing. Use the durable job queue for run execution that must survive a restart.

Create a Python file

background_output_evaluation.py

from agno.agent import Agent
from agno.db.sqlite import AsyncSqliteDb
from agno.eval.agent_as_judge import AgentAsJudgeEval
from agno.hooks import hook
from agno.run.agent import RunOutput
from agno.models.openai import OpenAIResponses
from agno.os import AgentOS

# Setup database for agent and evaluation storage
db = AsyncSqliteDb(db_file="tmp/evaluation.db")

@hook(run_in_background=True)
async def evaluate_response(run_output: RunOutput):
    # Each invocation owns its evaluator because post-check temporarily changes db.
    evaluator = AgentAsJudgeEval(
        db=db,
        name="Response Quality Check",
        model=OpenAIResponses(id="gpt-5.2"),
        criteria="Response should be helpful, accurate, and well-structured",
        additional_guidelines=[
            "Evaluate if the response addresses the user's question directly",
            "Check if the information provided is correct and reliable",
            "Assess if the response is well-organized and easy to understand",
        ],
        scoring_strategy="numeric",
        threshold=7,
    )
    await evaluator.async_post_check(run_output)


# Create the main agent with Agent as Judge evaluation
main_agent = Agent(
    id="support-agent",
    name="CustomerSupportAgent",
    model=OpenAIResponses(id="gpt-5.2"),
    instructions=[
        "You are a helpful customer support agent.",
        "Provide clear, accurate, and friendly responses.",
        "If you don't know something, say so honestly.",
    ],
    db=db,
    post_hooks=[evaluate_response],  # Queues evaluation after the response
    markdown=True,
)

# Create AgentOS
agent_os = AgentOS(agents=[main_agent])
app = agent_os.get_app()


if __name__ == "__main__":
    agent_os.serve(app="background_output_evaluation:app", port=7777, reload=True)

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U "agno[os]" openai aiosqlite

Export your OpenAI API key

export OPENAI_API_KEY="your_openai_api_key_here"

Run the server

python background_output_evaluation.py

Test the endpoint

curl -X POST http://localhost:7777/agents/support-agent/runs \
  -F "message=How do I reset my password?" \
  -F "stream=false"

The response is returned after the model and foreground work complete. The evaluation runs in the background and results are stored in the database.

The hook constructs a fresh evaluator per response. In current Agno, async_post_check() temporarily changes the evaluator's database field while awaiting the judge; sharing one instance across concurrent requests can lose database logging. A separate instance prevents that cross-request mutation. Judge scores assess the supplied criteria and do not establish factual correctness on their own.

What Happens

User sends a request to the agent
The agent processes and generates a response
The response is sent after the main agent finishes
Background evaluation runs:
- AgentAsJudgeEval automatically evaluates the response against the criteria
- Scores the response on a scale of 1-10
- Stores results in the database

Production Extensions

In production, you could extend this pattern to:

Extension	Description
Database Storage	Store evaluations for analytics dashboards
Alerting	Use `on_fail` callback to send alerts when evaluations fail
Observability	Log to platforms like Datadog or OpenTelemetry
A/B Testing	Compare response quality across model versions
Training Data	Build datasets for fine-tuning

Background evaluation is ideal for quality monitoring without impacting user experience. For scenarios where you need to block bad responses, use synchronous hooks instead.

Global Background Hooks

Run all hooks as background tasks

Per-Hook Background

Mix synchronous and background hooks