Preference data for RLHF
Rank two responses to the same prompt. The data shape for reward models and DPO.
Given a prompt and two candidate responses, pick the better one. Constrain the verdict to A, B, or tie.
output_schema requests structured output and Agno attempts to parse it. A failed or unparseable run can leave content as text. Check RunStatus.completed and the expected Pydantic type before reading fields, indexing, or writing downstream. These examples stop on failure; an application can instead send the original input to a review queue. Schema validation checks the shape and constraints, so verify extracted facts against the source separately.
from agno.run.base import RunStatus
from typing import Literal
from agno.agent import Agent
from agno.models.google import Gemini
from pydantic import BaseModel, Field
class Preference(BaseModel):
winner: Literal["A", "B", "tie"] = Field(
..., description="Which response is better, or 'tie' if equal"
)
agent = Agent(
model=Gemini(id="gemini-3.5-flash"),
instructions=(
"Decide which response better answers the prompt. Return 'A', 'B', "
"or 'tie'. Use 'tie' only when the two are genuinely "
"indistinguishable in quality."
),
output_schema=Preference,
)
def build_input(prompt: str, a: str, b: str) -> str:
return f"Prompt:\n{prompt}\n\nResponse A:\n{a}\n\nResponse B:\n{b}"
prompt = "Explain why the sky is blue, in one sentence."
a = "Shorter blue wavelengths scatter more off air molecules, so the sky looks blue."
b = "Because of physics."
result_run = agent.run(build_input(prompt, a, b))
if result_run.status != RunStatus.completed or not isinstance(result_run.content, Preference):
raise RuntimeError("No validated Preference; send the input to review before continuing")
result = result_run.content
# Preference(winner='A')Each (prompt, A, B, winner) row is a raw pairwise comparison. Convert the winner into the chosen and rejected fields or other schema required by your reward-model or DPO trainer. Ties usually need separate handling.
Add a rationale
A rationale per comparison gives annotators something to audit and helps debug a noisy reward model.
from typing import Literal
from pydantic import BaseModel, Field
class Preference(BaseModel):
winner: Literal["A", "B", "tie"] = Field(..., description="Better response")
rationale: str = Field(..., description="Why this comparison received its verdict")Compare against a rubric
When preference should follow explicit criteria, put the rubric in the instructions and keep the output constrained to A, B, or tie.
instructions = """\
Compare the two responses on these criteria, in priority order:
1. Correctness - is the information accurate
2. Completeness - does it fully answer the prompt
3. Clarity - is it easy to follow
Return the response that wins on the highest-priority criterion where
they differ. Use 'tie' only if they are equal on all three.
"""Picking the shape
| You need | Schema |
|---|---|
| Bare preference label | Literal["A", "B", "tie"] |
| Preference plus justification | Add a rationale field |
| Criteria-driven preference | Rubric in instructions, closed-set output |
Reducing position bias
A single judge can favor whichever response is shown first. Run the comparison twice with A and B swapped, or send both orderings to two providers and adjudicate. See the Quality pipeline for the two-model agreement pattern.
Next steps
| Task | Guide |
|---|---|
| Score a single response | LLM as judge |
| Adjudicate disagreements | Quality pipeline |