Spider Tools

Search, scrape, and crawl websites into LLM-ready Markdown with SpiderTools.

spider_tools.py
"""
Spider Tools
=============================

Demonstrates spider tools.
"""

from agno.agent import Agent
from agno.tools.spider import SpiderTools

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------


# Example 1: All functions available (default behavior)
agent_all = Agent(
    name="Spider Agent - All Functions",
    tools=[SpiderTools(optional_params={"proxy_enabled": True})],
    instructions=["You have access to all Spider web scraping capabilities."],
    markdown=True,
)

# Example 2: Include specific functions only
agent_specific = Agent(
    name="Spider Agent - Search Only",
    tools=[SpiderTools(enable_crawl=False, optional_params={"proxy_enabled": True})],
    instructions=["You can only search the web, no scraping or crawling."],
    markdown=True,
)


# Use the default agent for examples
agent = agent_all

# ---------------------------------------------------------------------------
# Run Agent
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    agent.print_response(
        'Can you scrape the first search result from a search on "news in USA"?'
    )

The script runs agent_all. The configuration named “Search Only” still enables scrape; add enable_scrape=False alongside enable_crawl=False to make that configuration search-only, then call agent_specific with a search prompt.

Search returns result metadata by default (fetch_page_content=False); retrieving the first result's page is a separate scrape call. Scrape and crawl return JSON envelopes containing Markdown, rather than bare Markdown strings.

Run the Example

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U agno openai spider-client

Export your API keys

export OPENAI_API_KEY="your_openai_api_key_here"
export SPIDER_API_KEY="your_spider_api_key_here"

Run the example

Save the code above as spider_tools.py, then run:

python spider_tools.py

Full source: cookbook/91_tools/spider_tools.py