Website Reader
Parameters and methods of WebsiteReader, which crawls websites and reads pages into documents.
WebsiteReader reads data from websites.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
chunking_strategy | Optional[ChunkingStrategy] | FixedSizeChunking() | Strategy used to chunk documents |
max_depth | int | 3 | Maximum depth level for crawling links |
max_links | int | 10 | Maximum number of links to crawl |
timeout | int | 10 | Request timeout in seconds |
proxy | Optional[str] | None | Proxy URL used for requests |
allowed_hosts | Optional[List[str]] | None | Hostnames the reader is allowed to fetch from. See Restricting URL Fetches. |
WebsiteReader also accepts the base Reader constructor parameters.
Methods
read()
Crawls a website starting from the given URL and returns a list of documents.
| Parameter | Type | Default | Description |
|---|---|---|---|
url | str | Required | URL of the website to crawl and read |
name | Optional[str] | None | Name override for the resulting documents |
async_read()
Async variant of read(). Accepts the same parameters.