Run it yourself

The CLI.

The command line came first; the HTTP API wraps the same engine. Running snakereach with no arguments in a terminal opens a guided interface that shows the command it is about to run, so it teaches the CLI while you use it.

Install

shell

pip install snakereach            # core
pip install "snakereach[all]"     # + rendering, stealth, PDFs, parquet

Commands

Command	What it does
scrape URL...	One URL to one normalized document. Accepts several at once.
crawl URL	Whole-site acquisition into a local store. Resumable.
discover URL	Every link on a page, canonicalized and classified.
sitemap URL	Find and expand sitemaps or feeds.
download URL...	Raw files as-is. PDFs, archives, images. No extraction.
ingest PATH	Local html/md/txt/pdf files through the same pipeline.
search QUERY	Full-text search across everything stored locally.
export	Stored documents to jsonl, csv, markdown or parquet.
benchmark	Time every pipeline stage against bundled fixtures.
doctor	Environment and which optional capabilities are active.
plugins	Extension points and installed plugins.
config	Show and edit configuration, with the origin of every value.
cache	Inspect or clear the HTTP cache.
setup	Install the command system-wide and enable optional extras.

Common runs

shell

snakereach scrape example.com                    # to the terminal
snakereach scrape example.com -f json -o out.json
snakereach crawl docs.example.com -d 2 --scope prefix
snakereach export -o corpus.jsonl --dedup --chunk 2000
snakereach search "rate limit"                   # over what you stored

Output formats: markdown with front-matter, JSON, text, HTML, CSV, JSONL, parquet, or a filesystem tree. Add --json to any command for machine-readable output; piping or redirecting skips the interface automatically.

Configuration

Layered, lowest to highest: built-in defaults, then ~/.config/snakereach/config.yaml, then ./snakereach.yaml, then SNAKEREACH_* environment variables, then flags.

shell

snakereach config show          # every value, with where it came from
snakereach config init          # write a commented starter file

Politeness is on by default

robots.txt is respected, per-domain delays are honoured, and Crawl-delay is obeyed. --no-robots exists for content you own, and is deliberately not exposed over the HTTP API.