vLLM Basic

Run a Qwen2.5-7B agent on a local vLLM server in sync, async, and streaming modes.

basic.py
"""
Vllm Basic
==========

Cookbook example for `vllm/basic.py`.
"""

import asyncio

from agno.agent import Agent
from agno.models.vllm import VLLM

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------

agent = Agent(
    model=VLLM(id="Qwen/Qwen2.5-7B-Instruct", top_k=20, enable_thinking=False),
    markdown=True,
)

# ---------------------------------------------------------------------------
# Run Agent
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    # --- Sync ---
    agent.print_response("Share a 2 sentence horror story")

    # --- Sync + Streaming ---
    agent.print_response("Share a 2 sentence horror story", stream=True)

    # --- Async ---
    asyncio.run(agent.aprint_response("Share a 2 sentence horror story"))

    # --- Async + Streaming ---
    asyncio.run(agent.aprint_response("Share a 2 sentence horror story", stream=True))

Run the Example

The server needs a supported serving environment with hardware and memory suitable for the selected model. For the standard GPU setup, use supported Linux hardware; Windows users can use a supported WSL environment or a separate serving host. The Python client can run separately.

Start the server in the first terminal

In the serving environment, install vLLM and start the model. Leave this foreground process running:

uv venv .venv-vllm
source .venv-vllm/bin/activate
uv pip install -U vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --host 127.0.0.1 --port 8000

Open a second terminal in your example directory for the client steps below.

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install client dependencies in the second terminal

uv pip install -U agno openai

Configure the local vLLM client

In this second terminal, select the server started above. These examples use an unauthenticated loopback server, so the SDK key is a nonempty placeholder. If you enable server authentication, use its configured key instead.

export VLLM_BASE_URL="http://127.0.0.1:8000/v1"
export VLLM_API_KEY="vllm-local"

The requested model ID must match the model served above. For a server on another supported host, configure its reachable URL and authentication instead of the loopback URL.

Run the example

Save the code above as basic.py, apply the listed edits, then run in the second terminal:

python basic.py

Full source: cookbook/90_models/vllm/basic.py