snakereach

Endpoint

Crawl a site.

A crawl outlives a request, so it runs as a job. Start it, get an id back immediately, then poll until it finishes.

POST/v1/crawlAPI key

Start a crawl. Returns 202 with a job id, because crawls outlive a request.

Budgets are clamped server-side. Asking for 10,000 pages gets you the deployment's ceiling, not 10,000 pages.

Body

FieldTypeDetail
url requiredstringSeed URL.
max_pagesintegerPage budget. Clamped by the server.
max_depthintegerLink depth from the seed. Clamped by the server.
scope"domain" | "subdomains" | "prefix" | "any"Which discovered URLs belong to the crawl. Default domain.
dedupbooleanDrop pages whose extracted text repeats. Default true.
request
curl https://api.snakereach.com/v1/crawl \
  -H "X-API-Key: $SNAKEREACH_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url": "https://example.com/docs", "max_depth": 2, "max_pages": 20}'
202 response
{
  "id": "7cdb3c13697f4adf88abed66a21b969d",
  "url": "https://example.com/docs",
  "status": "queued",
  "created_at": 1785411826.888277,
  "finished_at": null,
  "stats": { "crawled": 0, "stored": 0, "failed": 0,
             "skipped": 0, "duplicates": 0, "pending": 0 },
  "errors": [],
  "document_count": 0
}

Errors

  • 429 too many crawls already in flight, or you hit the rate limit
GET/v1/jobs/{id}API key

Poll a crawl. Add ?documents=true to include the pages themselves.

status moves through queued → running → one of done, failed, cancelled. Poll every second or two.

request
curl "https://api.snakereach.com/v1/jobs/7cdb3c13697f4adf88abed66a21b969d?documents=true" \
  -H "X-API-Key: $SNAKEREACH_API_KEY"
200 response
{
  "id": "7cdb3c13697f4adf88abed66a21b969d",
  "url": "https://example.com/docs",
  "status": "done",
  "created_at": 1785411826.888277,
  "finished_at": 1785411826.981174,
  "stats": { "crawled": 7, "stored": 0, "failed": 0,
             "skipped": 0, "duplicates": 1, "pending": 22 },
  "errors": [],
  "document_count": 6
}

Errors

  • 404 no job with that id. It may have expired
DELETE/v1/jobs/{id}API key

Cancel a running crawl. Finished jobs come back unchanged.

request
curl -X DELETE https://api.snakereach.com/v1/jobs/7cdb3c13697f4adf88abed66a21b969d \
  -H "X-API-Key: $SNAKEREACH_API_KEY"

Errors

  • 404 no job with that id

Jobs are not storage

Jobs live in the server's memory. A restart loses them, and they are not shared between processes. Collect results soon after a crawl finishes rather than treating the job as somewhere to keep them.

Finished jobs are evicted after 15 minutes by default. A 404 on a job id you know was valid usually means it expired or the service restarted.

Scope

domain stays on the exact host. subdomains allows anything under the same registrable domain. prefix restricts to the seed's path prefix, the right choice for crawling /docs without pulling in a marketing site. any follows every link and is rarely what you want.

Politeness applies throughout: robots.txt is respected, per-domain delays are honoured, and Crawl-delay is obeyed where a site declares it.