Memory

Personalized memory and session summaries with vLLM.

memory.py
"""
Personalized memory and session summaries with vLLM.
Prerequisites:
1. Start a Postgres + pgvector container (helper script is provided):
       ./cookbook/scripts/run_pgvector.sh
2. Install dependencies:
       uv pip install sqlalchemy 'psycopg[binary]' pgvector
3. Run a vLLM server (any open model).  Example with Phi-3:
       vllm serve microsoft/Phi-3-mini-128k-instruct \
         --dtype float32 \
         --enable-auto-tool-choice \
         --tool-call-parser pythonic
Then execute this script – it will remember facts you tell it and generate a
summary.
"""

from agno.agent import Agent
from agno.db.postgres import PostgresDb
from agno.models.vllm import VLLM
from rich.pretty import pprint

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------

# Change this if your Postgres container is running elsewhere
DB_URL = "postgresql+psycopg://ai:ai@localhost:5532/ai"

agent = Agent(
    model=VLLM(id="microsoft/Phi-3-mini-128k-instruct"),
    db=PostgresDb(db_url=DB_URL),
    update_memory_on_run=True,
    enable_session_summaries=True,
)

# -*- Share personal information
agent.print_response("My name is john billings?", stream=True)
# -*- Print memories and summary
if agent.db:
    pprint(agent.get_user_memories(user_id="test_user"))
    pprint(
        agent.get_session(session_id="test_session").summary  # type: ignore
    )

# -*- Share personal information
agent.print_response("I live in nyc?", stream=True)
# -*- Print memories and summary
if agent.db:
    pprint(agent.get_user_memories(user_id="test_user"))
    pprint(
        agent.get_session(session_id="test_session").summary  # type: ignore
    )

# -*- Share personal information
agent.print_response("I'm going to a concert tomorrow?", stream=True)
# -*- Print memories and summary
if agent.db:
    pprint(agent.get_user_memories(user_id="test_user"))
    pprint(
        agent.get_session(session_id="test_session").summary  # type: ignore
    )

# Ask about the conversation
agent.print_response(
    "What have we been talking about, do you know my name?", stream=True
)

# ---------------------------------------------------------------------------
# Run Agent
# ---------------------------------------------------------------------------

if __name__ == "__main__":
    pass

Run the Example

The server needs a supported serving environment with hardware and memory suitable for the selected model. For the standard GPU setup, use supported Linux hardware; Windows users can use a supported WSL environment or a separate serving host. The Python client can run separately.

Start the server in the first terminal

In the serving environment, install vLLM and start the model. Leave this foreground process running:

uv venv .venv-vllm
source .venv-vllm/bin/activate
uv pip install -U vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --host 127.0.0.1 --port 8000 --enable-auto-tool-choice --tool-call-parser hermes

Open a second terminal in your example directory for the client steps below.

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install client dependencies in the second terminal

uv pip install -U agno "psycopg[binary]" openai sqlalchemy

Configure the local vLLM client

In this second terminal, select the server started above. These examples use an unauthenticated loopback server, so the SDK key is a nonempty placeholder. If you enable server authentication, use its configured key instead.

export VLLM_BASE_URL="http://127.0.0.1:8000/v1"
export VLLM_API_KEY="vllm-local"

The requested model ID must match the model served above. For a server on another supported host, configure its reachable URL and authentication instead of the loopback URL.

Run PgVector

docker run -d \
  -e POSTGRES_DB=ai \
  -e POSTGRES_USER=ai \
  -e POSTGRES_PASSWORD=ai \
  -e PGDATA=/var/lib/postgresql \
  -v pgvolume:/var/lib/postgresql \
  -p 5532:5432 \
  --name pgvector \
  agnohq/pgvector:18

Match the served model

Replace microsoft/Phi-3-mini-128k-instruct with Qwen/Qwen2.5-7B-Instruct in the saved model constructor. This model has a tool-aware template supported by vLLM's Hermes parser. Follow the current serving command below instead of the historical source comments.

Use consistent memory and session identities

Add user_id="test_user", session_id="test_session" to Agent(...). The stored person and conversation must match all three inspection calls. The memory and summary managers inherit the vLLM model, so their tool calls need the same supported server configuration.

Replace each summary print block with this guarded version:

session = agent.get_session(session_id="test_session")
pprint(session.summary if session is not None else None)

These fixed IDs are for a single-user demo. Applications should use authenticated user IDs and distinct session IDs for each conversation.

Run the example

Save the code above as memory.py, apply the listed edits, then run in the second terminal:

python memory.py

Full source: cookbook/90_models/vllm/memory.py