Send Media to an Agent over AG-UI

Accept AG-UI image, audio, video, and document content parts and pass them to a Gemini multimodal agent through one AgentOS interface.

agent_with_media.py
"""
Send Media to an Agent over AG-UI
=================================

Accept AG-UI image, audio, video, and document content parts and pass them to
a Gemini multimodal agent through one AgentOS interface.

Prerequisites: GOOGLE_API_KEY
Run: .venvs/demo/bin/python cookbook/05_agent_os/16_agui/agent_with_media.py
Try: Attach an image and ask what it shows through http://localhost:7777/media/agui
"""

from agno.agent import Agent
from agno.db.sqlite import SqliteDb
from agno.models.google import Gemini
from agno.os import AgentOS
from agno.os.interfaces.agui import AGUI

# ---------------------------------------------------------------------------
# Create Multimodal Agent
# ---------------------------------------------------------------------------

db = SqliteDb(
    id="agui-media-db",
    db_file="tmp/agui_media.db",
)

media_agent = Agent(
    id="agui-media-agent",
    name="AG-UI Media Agent",
    model=Gemini(id="gemini-3.5-flash"),
    db=db,
    instructions=(
        "Inspect the user's attached image, audio, video, or document. "
        "Answer only from the supplied content and say when a detail is unclear."
    ),
)

agent_os = AgentOS(
    id="agui-media-os",
    description="A Gemini multimodal agent served through AG-UI.",
    agents=[media_agent],
    interfaces=[AGUI(agent=media_agent, prefix="/media")],
)
app = agent_os.get_app()

# ---------------------------------------------------------------------------
# Run Media Server
# ---------------------------------------------------------------------------

if __name__ == "__main__":
    agent_os.serve(app=app)

Run the Example

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U "agno[agui,os]" google-genai

Export your Google API key

export GOOGLE_API_KEY="your_google_api_key_here"

Run the example

Save the code above as agent_with_media.py, then run:

python agent_with_media.py

Attach an image

In the request below, replace the user message's content string with this array. Replace the URL with an image reachable by the AgentOS server and use its actual MIME type:

[
  {"type": "text", "text": "What does this image show?"},
  {
    "type": "image",
    "source": {
      "type": "url",
      "value": "https://example.com/your-image.png",
      "mimeType": "image/png"
    }
  }
]

For this Gemini example, Agno extracts the media part from the last user message, downloads the image in the AgentOS process, and sends its bytes to Gemini. The AgentOS server needs network access to the URL.

Send a request

With the server running in another terminal:

Save this complete AG-UI request as agui-input.json. Replace content with the prompt for this example. Use a fresh runId for each run; keep threadId only when continuing the same session.

agui-input.json
{
  "threadId": "example-thread-1",
  "runId": "example-run-1",
  "state": {},
  "messages": [
    {"id": "example-message-1", "role": "user", "content": "Hello"}
  ],
  "tools": [],
  "context": [],
  "forwardedProps": {}
}

The endpoint accepts this object, rather than a JSON string or the REST run endpoint's message form field. The response is an SSE stream whose data values are AG-UI event objects. Inspect RUN_ERROR and RUN_FINISHED events; an HTTP 200 only establishes that the stream opened.

curl --no-buffer -H "Content-Type: application/json" --data-binary @agui-input.json http://localhost:7777/media/agui

Full source: cookbook/05_agent_os/16_agui/agent_with_media.py