LLMs.txt

LLMsTxtTools reads llms.txt files. The format is a standardized way for websites to publish an LLM-friendly documentation index. The toolkit operates in two modes depending on whether you pass a Knowledge instance.

Prerequisites

The following examples require the openai library.

uv pip install -U agno openai

The Knowledge mode example stores pages in PgVector, which also needs:

uv pip install -U sqlalchemy "psycopg[binary]" pgvector

Agentic Mode

Without knowledge, the agent reads the index and decides which pages to fetch.

cookbook/91_tools/llms_txt_tools.py

from agno.agent import Agent
from agno.models.openai import OpenAIResponses
from agno.tools.llms_txt import LLMsTxtTools

agent = Agent(
    model=OpenAIResponses(id="gpt-5.4"),
    tools=[LLMsTxtTools()],
    instructions=[
        "First use get_llms_txt_index to see what pages are available.",
        "Then use read_llms_txt_url to fetch only the pages relevant to the question.",
    ],
    markdown=True,
)

agent.print_response(
    "Using the llms.txt at https://docs.agno.com/llms.txt, "
    "find and read the documentation about how to create an agent with tools",
    stream=True,
)

Knowledge Mode

Pass a Knowledge instance and the toolkit exposes read_llms_txt_and_load_knowledge, which ingests the indexed pages into your knowledge base for retrieval.

cookbook/91_tools/llms_txt_tools_knowledge.py

from agno.knowledge.knowledge import Knowledge
from agno.tools.llms_txt import LLMsTxtTools
from agno.vectordb.pgvector import PgVector

knowledge = Knowledge(
    vector_db=PgVector(
        table_name="llms_txt_docs",
        db_url="postgresql+psycopg://ai:ai@localhost:5532/ai",
    ),
)

tools = LLMsTxtTools(knowledge=knowledge)

Toolkit Params

Parameter	Type	Default	Description
`knowledge`	`Optional[Knowledge]`	`None`	When set, switches to Knowledge mode (load pages instead of returning).
`max_urls`	`int`	`20`	Maximum linked pages ingested in Knowledge mode; does not limit index entries, direct reads, or the agent's total calls.
`timeout`	`int`	`60`	HTTP timeout in seconds.
`skip_optional`	`bool`	`False`	Skip pages listed under the index's optional section.
`allowed_hosts`	`Optional[List[str]]`	`None`	Exact, case-insensitive host allowlist, checked across redirects. `None` allows any host; this is not private-IP or DNS validation.

Toolkit Functions

Function	Mode	Description
`get_llms_txt_index`	Agentic	Fetch and parse an llms.txt index.
`read_llms_txt_url`	Agentic	Read a single page referenced by the index.
`read_llms_txt_and_load_knowledge`	Knowledge	Read indexed pages and load them into Knowledge.

All functions have sync and async variants.

Knowledge mode database

Before constructing the Knowledge example, start a PgVector-enabled PostgreSQL instance at its configured URL (ai:ai on localhost port 5532, database ai). For a new local demo instance:

docker run -d --name llms-txt-pgvector -e POSTGRES_USER=ai -e POSTGRES_PASSWORD=ai -e POSTGRES_DB=ai -p 5532:5432 pgvector/pgvector:pg17

Wait until PostgreSQL is ready before running the Python code. If port 5532 already serves your database, use that instance or choose a different port and update db_url to match. Knowledge ingestion also requires the OpenAI key for the default embedder.