Website Knowledge

Ingest web pages and PDFs into a PgVector knowledge base with WebsiteTools.

WebsiteTools(knowledge=kb) lets the agent add pages to the same PgVector-backed knowledge base it searches.

website_tools_knowledge.py
"""
Website Tools Knowledge
=============================

Demonstrates website tools knowledge.
"""

from agno.agent import Agent
from agno.knowledge.knowledge import Knowledge
from agno.tools.website import WebsiteTools
from agno.vectordb.pgvector import PgVector

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------


db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

# Create PDF URL knowledge base
kb = Knowledge(
    vector_db=PgVector(
        table_name="documents",
        db_url=db_url,
    ),
)

# ---------------------------------------------------------------------------
# Run Agent
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    kb.insert_many(
        urls=[
            "https://agno-public.s3.amazonaws.com/recipes/ThaiRecipes.pdf",
            "https://docs.agno.com/introduction",
        ]
    )

    # Initialize the Agent with the combined knowledge base
    agent = Agent(
        knowledge=kb,
        search_knowledge=True,
        tools=[
            WebsiteTools(knowledge=kb)  # Set combined or website knowledge base
        ],
    )

    # Use the agent
    agent.print_response(
        "How do I get started on Mistral: https://docs.mistral.ai/getting-started/models/models_overview",
        markdown=True,
        stream=True,
    )

With knowledge=kb, the registered tool is add_website_to_knowledge; it inserts the URL but does not return the extracted page. The agent’s separate knowledge search retrieves it afterward. The initial PDF and Agno page are inserted before the run. Use an accessible current Mistral documentation URL in the final request.

Run the Example

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U agno "psycopg[binary]" beautifulsoup4 openai pgvector pypdf sqlalchemy

Export your OpenAI API key

export OPENAI_API_KEY="your_openai_api_key_here"

Run PgVector

docker run -d \
  -e POSTGRES_DB=ai \
  -e POSTGRES_USER=ai \
  -e POSTGRES_PASSWORD=ai \
  -e PGDATA=/var/lib/postgresql \
  -v pgvolume:/var/lib/postgresql \
  -p 5532:5432 \
  --name pgvector \
  agnohq/pgvector:18

Run the example

Save the code above as website_tools_knowledge.py, then run:

python website_tools_knowledge.py

Full source: cookbook/91_tools/website_tools_knowledge.py