vLLM | Agno

vLLM is a fast and easy-to-use library for LLM inference and serving, designed for high throughput and memory efficiency.

Prerequisites

Install vLLM in a server environment using the installation guide for your hardware. The GPU commands below require a supported Linux accelerator environment. The Agno client can run in a separate Python environment with agno, openai and the example's other dependencies.

Run vllm serve in a dedicated server terminal with its environment active and keep it running. In a client terminal, activate the environment for your saved Python file, export VLLM_API_KEY, and run the agent. The examples use port 8000; for a server on another host, set the matching base_url on VLLM. A server started without --api-key accepts a placeholder key, but Agno still requires a nonempty VLLM_API_KEY.

start vLLM server

vllm serve Qwen/Qwen2.5-7B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --dtype float16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9

This spins up the vLLM server with an OpenAI-compatible API.

Set the VLLM_API_KEY environment variable. VLLM raises an error without it. If you started the server without --api-key, any value works:

set API key

export VLLM_API_KEY=xxx

VLLM connects to http://localhost:8000/v1/ by default. Override with the base_url parameter or the VLLM_BASE_URL environment variable.

Example

In the client terminal, install the client dependencies:

uv pip install -U agno openai

from agno.agent import Agent
from agno.models.vllm import VLLM

agent = Agent(
    model=VLLM(
        id="Qwen/Qwen2.5-7B-Instruct",
        base_url="http://localhost:8000/v1/",
    ),
    markdown=True
)

agent.print_response("Share a 2 sentence horror story.")

Advanced Usage

With Tools

vLLM models work with Agno tools:

with_tools.py

from agno.agent import Agent
from agno.models.vllm import VLLM
from agno.tools.hackernews import HackerNewsTools

agent = Agent(
    model=VLLM(id="Qwen/Qwen2.5-7B-Instruct"),
    tools=[HackerNewsTools()],
    markdown=True
)

agent.print_response("What's the latest news about AI?")

View more examples here.

For the full list of supported models, see the vLLM documentation.

Params

Parameter	Type	Default	Description
`id`	`str`	`"not-set"`	The id of the model to use with vLLM
`name`	`str`	`"VLLM"`	The name of the model
`provider`	`str`	`"VLLM"`	The provider of the model
`api_key`	`Optional[str]`	`None`	The API key. Falls back to the `VLLM_API_KEY` environment variable and raises an error if neither is set
`base_url`	`Optional[str]`	`None`	The base URL for the vLLM server. Falls back to the `VLLM_BASE_URL` environment variable, then `"http://localhost:8000/v1/"`
`temperature`	`float`	`0.7`	Sampling temperature
`top_p`	`float`	`0.8`	Nucleus sampling probability
`presence_penalty`	`float`	`1.5`	Penalty for repeating tokens already present
`top_k`	`Optional[int]`	`None`	Top-k sampling, sent via `extra_body`
`enable_thinking`	`Optional[bool]`	`None`	Sets `chat_template_kwargs.enable_thinking` via `extra_body`

VLLM is a subclass of the OpenAILike class and has access to the same params.