vLLM Embedder

Generate embeddings with vLLM, either in-process or against a vLLM server.

VLLMEmbedder runs in two modes. Local mode loads the model in-process with no server or API key. Remote mode connects to a running vLLM server through its OpenAI-compatible API, selected by setting base_url.

from vllm import LLM
from agno.knowledge.embedder.vllm import VLLMEmbedder

# Inject a client using the current vLLM pooling API
model_id = "sentence-transformers/all-MiniLM-L6-v2"
client = LLM(
    model=model_id,
    runner="pooling",
    enforce_eager=True,
    max_model_len=256,
)
embedder = VLLMEmbedder(
    id=model_id,
    dimensions=384,
    vllm_client=client,
)
embedding = embedder.get_embedding("The quick brown fox jumps over the lazy dog.")

print(embedding[:5])
print(len(embedding))

# Remote mode: connect to a running vLLM server
remote_embedder = VLLMEmbedder(
    id="sentence-transformers/all-MiniLM-L6-v2",
    dimensions=384,
    base_url="http://localhost:8000/v1",
    api_key="your-api-key",  # optional, also read from VLLM_API_KEY
)

Local mode downloads the model from Hugging Face on first use and requires a platform and hardware supported by vLLM. Check the model's memory requirements. The injected client uses the current pooling API; Agno's automatic client construction still passes the older task="embed" keyword.

Run the Example

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U agno vllm openai

Run the example

python vllm_embedder.py

Developer Resources