snakereachReference
Rate limits.
Defaults for the hosted API. If your workload does not fit inside them, say so and we will look at it.
| Limit | Default | Variable |
|---|---|---|
| Requests | 10 per minute, per client IP | SNAKEREACH_API_RATE_LIMIT |
| Window | 60 seconds | SNAKEREACH_API_RATE_WINDOW |
| Concurrent requests | 8 across the deployment | SNAKEREACH_API_MAX_CONCURRENCY |
| Concurrent crawls | 4 jobs in flight | SNAKEREACH_API_MAX_JOBS |
| Crawl pages | 25 per job | SNAKEREACH_API_CRAWL_MAX_PAGES |
| Crawl depth | 2 from the seed | SNAKEREACH_API_CRAWL_MAX_DEPTH |
| Response size | 10 MB, on decoded bytes | SNAKEREACH_API_MAX_RESPONSE_BYTES |
| Request timeout | 30 seconds | SNAKEREACH_API_REQUEST_TIMEOUT |
| Job retention | 15 minutes after finishing | SNAKEREACH_API_JOB_TTL |
Crawl budgets are clamped, not rejected
Asking for 10,000 pages does not fail. It gives you the deployment's ceiling. The client asks; the server decides. Check stats.crawled on the finished job for what you actually got.
Counted per process
Limits are held in memory, so a horizontally scaled deployment gives a client the allowance once per instance. A shared store is the fix when that starts to matter.
Staying under
Leave use_cache on, repeated URLs are served from cache and cost you almost nothing. Batch work with a delay between requests rather than firing everything at once. And read the detail on a 429: it says exactly how long to wait.
The playground on this site is capped harder than the API, five requests a minute, because it runs on our key. That limit is not the API's.