LLM as judge
Score model outputs against a rubric. The same machinery as labeling, applied to evaluation.
A judge is a classifier whose input is a (prompt, response) pair and whose output is a score. Use int with ge and le so validated scores stay on scale.
output_schema requests structured output and Agno attempts to parse it. A failed or unparseable run can leave content as text. Check RunStatus.completed and the expected Pydantic type before reading fields, indexing, or writing downstream. These examples stop on failure; an application can instead send the original input to a review queue. Schema validation checks the shape and constraints, so verify extracted facts against the source separately.
from agno.run.base import RunStatus
from agno.agent import Agent
from agno.models.google import Gemini
from pydantic import BaseModel, Field
class Score(BaseModel):
overall: int = Field(
..., ge=1, le=5, description="Overall quality, 5 is excellent"
)
agent = Agent(
model=Gemini(id="gemini-3.5-flash"),
instructions=(
"Score the response on overall quality from 1 (unusable) to 5 "
"(excellent). Use the full scale. Reserve 5 for genuinely "
"excellent responses."
),
output_schema=Score,
)
def build_input(prompt: str, response: str) -> str:
return f"Prompt:\n{prompt}\n\nResponse:\n{response}"
prompt = "Explain why the sky is blue, in one sentence."
result_run = agent.run(build_input(prompt, "It just is."))
if result_run.status != RunStatus.completed or not isinstance(result_run.content, Score):
raise RuntimeError("No validated Score; send the input to review before continuing")
result = result_run.content
# Score(overall=1)Add a rationale
A free-text rationale makes the score auditable and surfaces rubric drift.
from pydantic import BaseModel, Field
class Score(BaseModel):
overall: int = Field(..., ge=1, le=5, description="Overall quality")
rationale: str = Field(..., description="Why this score, citing the response")Multi-dimension rubric
Use one bounded field per rubric dimension and a separate field for the overall assessment.
from pydantic import BaseModel, Field
class RubricScore(BaseModel):
correctness: int = Field(..., ge=1, le=5, description="Factually correct")
completeness: int = Field(..., ge=1, le=5, description="Covers what was asked")
clarity: int = Field(..., ge=1, le=5, description="Easy to follow")
concision: int = Field(..., ge=1, le=5, description="No padding")
overall: int = Field(..., ge=1, le=5, description="Holistic quality")Picking the shape
| You need | Schema |
|---|---|
| One quality number | int with ge=1, le=5 |
| Number plus justification | Add a rationale field |
| Per-criterion breakdown | One bounded int field per dimension |
| Pairwise comparison | Preference data |
Relationship to evals
This applies a classification schema to model outputs. When the judge is the deliverable, it lives here. When it scores a system under test, see Evals.
Next steps
| Task | Guide |
|---|---|
| Rank two responses | Preference data |
| Reduce single-model bias | Quality pipeline |