LiteLLM Knowledge

Answer questions from a PDF knowledge base stored in PgVector using a LiteLLM agent.

The same OPENAI_API_KEY is used for the answer model and the default OpenAI embedder.

knowledge.py
"""
Litellm Knowledge
=================

Cookbook example for `litellm/knowledge.py`.
"""

from agno.agent import Agent
from agno.knowledge.knowledge import Knowledge
from agno.models.litellm import LiteLLM
from agno.vectordb.pgvector import PgVector

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

knowledge = Knowledge(
    vector_db=PgVector(table_name="recipes", db_url=db_url),
)
# Add content to the knowledge
knowledge.insert(url="https://agno-public.s3.amazonaws.com/recipes/ThaiRecipes.pdf")

agent = Agent(model=LiteLLM(id="gpt-5.6-luna"), knowledge=knowledge)
agent.print_response("How to make Thai curry?", markdown=True)

# ---------------------------------------------------------------------------
# Run Agent
# ---------------------------------------------------------------------------

if __name__ == "__main__":
    pass

Run the Example

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U agno "psycopg[binary]" beautifulsoup4 litellm openai pgvector pypdf sqlalchemy

Set your OpenAI credentials

Use an OpenAI API key with access to the requested model. The LiteLLM SDK calls the provider directly. An existing LITELLM_API_KEY overrides provider-specific credentials, so clear it for this example.

unset LITELLM_API_KEY
export OPENAI_API_KEY="your_provider_api_key_here"

Run PgVector

docker run -d \
  -e POSTGRES_DB=ai \
  -e POSTGRES_USER=ai \
  -e POSTGRES_PASSWORD=ai \
  -e PGDATA=/var/lib/postgresql \
  -v pgvolume:/var/lib/postgresql \
  -p 5532:5432 \
  --name pgvector \
  agnohq/pgvector:18

Set compatible sampling options

Add temperature=None, top_p=None to every LiteLLM(...) using id="gpt-5.6-luna" or id="openai/gpt-5.6-luna" in your saved file. The adapter defaults to temperature=0.7 and top_p=1.0; the LiteLLM SDK rejects those sampling settings for this model's default reasoning mode before sending a request.

Run the example

Save the code above as knowledge.py, then run:

python knowledge.py

Full source: cookbook/90_models/litellm/knowledge.py