vLLM
Parameter reference for VLLMEmbedder, which embeds locally with a vLLM engine or remotely via an OpenAI-compatible server.
The vLLM Embedder provides high-performance embedding inference with support for both local and remote deployment modes. It can load models directly for local inference or connect to a remote vLLM server via an OpenAI-compatible API.
Usage
Use a platform and hardware supported by vLLM; the first run downloads the model. Install dependencies with uv pip install -U agno vllm openai. This local example injects the current pooling engine because Agno's automatic engine construction still passes the obsolete task="embed" argument.
from vllm import LLM
from agno.knowledge.embedder.vllm import VLLMEmbedder
model_id = "sentence-transformers/all-MiniLM-L6-v2"
client = LLM(model=model_id, runner="pooling", enforce_eager=True, max_model_len=256)
embedder = VLLMEmbedder(id=model_id, dimensions=384, vllm_client=client)
print(len(embedder.get_embedding("A document to embed.")))Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
id | str | "sentence-transformers/all-MiniLM-L6-v2" | Model identifier (HuggingFace model name) |
dimensions | int | 4096 | Expected vector width; does not resize returned vectors. The default MiniLM model produces 384 values, so set dimensions=384 for it. |
base_url | Optional[str] | None | Remote vLLM server URL (enables remote mode) |
api_key | Optional[str] | getenv("VLLM_API_KEY") | API key for remote server authentication |
enable_batch | bool | False | Enable batch processing for multiple texts |
batch_size | int | 100 | Number of texts to process per batch |
enforce_eager | bool | True | Use eager execution mode (local mode) |
vllm_kwargs | Optional[Dict[str, Any]] | None | Additional vLLM engine parameters (local mode) |
vllm_client | Optional[LLM] | None | Pre-configured vLLM engine instance (local mode) |
request_params | Optional[Dict[str, Any]] | None | Additional request parameters (remote mode) |
client_params | Optional[Dict[str, Any]] | None | OpenAI client configuration (remote mode) |
remote_client | Optional[OpenAI] | None | Pre-configured OpenAI-compatible client for the vLLM server (remote mode) |
async_remote_client | Optional[AsyncOpenAI] | None | Pre-configured async OpenAI-compatible client for the vLLM server (remote mode) |