Multimodal inputs

Feed images, audio, video, and PDFs into any labeling or extraction agent.

Pass media through the matching Agent.run() argument and choose a model that supports the input modality. The output_schema pattern stays the same.

output_schema requests structured output and Agno attempts to parse it. A failed or unparseable run can leave content as text. Check RunStatus.completed and the expected Pydantic type before reading fields, indexing, or writing downstream. These examples stop on failure; an application can instead send the original input to a review queue. Schema validation checks the shape and constraints, so verify extracted facts against the source separately.

from agno.run.base import RunStatus

from typing import Literal

from agno.agent import Agent
from agno.media import Image
from agno.models.google import Gemini
from pydantic import BaseModel, Field


class Classification(BaseModel):
    label: Literal["wildlife", "landscape", "sports", "architecture", "other"] = Field(
        ..., description="The primary scene type of the image"
    )


agent = Agent(
    model=Gemini(id="gemini-3.5-flash"),
    instructions="You classify images by scene type.",
    output_schema=Classification,
)

url = "https://storage.googleapis.com/generativeai-downloads/images/generated_elephants_giraffes_zebras_sunset.jpg"
result_run = agent.run("Classify this image.", images=[Image(url=url)])
if result_run.status != RunStatus.completed or not isinstance(result_run.content, Classification):
    raise RuntimeError("No validated Classification; send the input to review before continuing")
result = result_run.content
# Classification(label='wildlife')

Input argument per modality

ModalityImportArgument
Imagefrom agno.media import Imageimages=[Image(url=...)]
Audiofrom agno.media import Audioaudio=[Audio(content=...)]
Videofrom agno.media import Videovideos=[Video(content=..., format="mp4")]
PDFfrom agno.media import Filefiles=[File(url=...)]

All four classes accept url, filepath, or raw bytes via content. The cookbook examples fetch audio and video bytes first. Agno installs httpx.

from agno.run.base import RunStatus

import httpx

from agno.agent import Agent
from agno.media import Audio
from agno.models.google import Gemini
from pydantic import BaseModel, Field


class Transcript(BaseModel):
    text: str = Field(..., description="Verbatim transcript of all spoken audio")


transcription_agent = Agent(
    model=Gemini(id="gemini-3.5-flash"),
    instructions="Transcribe all spoken audio. Return only the transcript.",
    output_schema=Transcript,
)

url = "https://agno-public.s3.us-east-1.amazonaws.com/demo_data/QA-01.mp3"
response = httpx.get(url, timeout=30.0)
response.raise_for_status()

result_run = transcription_agent.run(
    "Transcribe this audio.",
    audio=[Audio(content=response.content)],
)
if result_run.status != RunStatus.completed or not isinstance(result_run.content, Transcript):
    raise RuntimeError("No validated Transcript; send the input to review before continuing")
result = result_run.content
# Transcript(text='...')

Bounding boxes

For region detection, return normalized coordinates so the result is resolution-independent.

from pydantic import BaseModel, Field


class BoundingBox(BaseModel):
    label: str = Field(..., description="What the box contains")
    x: float = Field(..., ge=0.0, le=1.0, description="Top-left x in [0, 1]")
    y: float = Field(..., ge=0.0, le=1.0, description="Top-left y in [0, 1]")
    width: float = Field(..., ge=0.0, le=1.0, description="Width in [0, 1]")
    height: float = Field(..., ge=0.0, le=1.0, description="Height in [0, 1]")

State the [0, 1] coordinate system in both the field descriptions and the agent instructions. This keeps the four values consistent across images.

The bounding boxes cookbook has runnable single-object, multi-object, and per-box confidence variants.

Transcription and diarization

Audio extraction covers transcription, speaker diarization, and timestamped segments. Each is a schema change over the same API.

OutputSchema shape
Flat transcript{ text: str }
Speaker turns{ turns: List[{ speaker, text }] }
Timestamped segments{ segments: List[{ start_seconds, end_seconds, text }] }

Model choice

The cookbook uses gemini-3.5-flash across its text, image, audio, video, and PDF recipes. A replacement model must support the input modality and structured output.

Next steps

TaskGuide
Define the output schemaData extraction
Assign labels to mediaClassification
Review media labelsQuality pipeline

Developer Resources