vLLM Tool Use

Add web search tools to a vLLM agent and stream responses sync and async.

tool_use.py
"""Build a Web Search Agent using xAI."""

import asyncio

from agno.agent import Agent
from agno.models.vllm import VLLM
from agno.tools.websearch import WebSearchTools

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------

agent = Agent(
    model=VLLM(
        id="NousResearch/Nous-Hermes-2-Mistral-7B-DPO", top_k=20, enable_thinking=False
    ),
    tools=[WebSearchTools()],
    markdown=True,
)

# ---------------------------------------------------------------------------
# Run Agent
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    # --- Sync ---
    agent.print_response("Whats happening in France?")

    # --- Sync + Streaming ---
    agent.print_response("Whats happening in France?", stream=True)

    # --- Async + Streaming ---
    asyncio.run(agent.aprint_response("Whats happening in France?", stream=True))

Run the Example

The server needs a supported serving environment with hardware and memory suitable for the selected model. For the standard GPU setup, use supported Linux hardware; Windows users can use a supported WSL environment or a separate serving host. The Python client can run separately.

Start the server in the first terminal

In the serving environment, install vLLM and start the model. Leave this foreground process running:

uv venv .venv-vllm
source .venv-vllm/bin/activate
uv pip install -U vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --host 127.0.0.1 --port 8000 --enable-auto-tool-choice --tool-call-parser hermes

Open a second terminal in your example directory for the client steps below.

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install client dependencies in the second terminal

uv pip install -U agno ddgs openai

Configure the local vLLM client

In this second terminal, select the server started above. These examples use an unauthenticated loopback server, so the SDK key is a nonempty placeholder. If you enable server authentication, use its configured key instead.

export VLLM_BASE_URL="http://127.0.0.1:8000/v1"
export VLLM_API_KEY="vllm-local"

The requested model ID must match the model served above. For a server on another supported host, configure its reachable URL and authentication instead of the loopback URL.

Match the served model

Replace NousResearch/Nous-Hermes-2-Mistral-7B-DPO with Qwen/Qwen2.5-7B-Instruct in the saved model constructor. This model has a tool-aware template supported by vLLM's Hermes parser. Follow the current serving command below instead of the historical source comments.

Run the example

Save the code above as tool_use.py, apply the listed edits, then run in the second terminal:

python tool_use.py

Full source: cookbook/90_models/vllm/tool_use.py