Writing · Fetching

Why your scraper returns an empty page

30 July 2026 · 6 min read

The request succeeded. The status was 200. The response had a perfectly reasonable content length. And the extracted text is three words long, or empty.

Nothing failed. You fetched exactly what the server sent, and what the server sent was not the page.

What actually arrived

A client-rendered application typically responds with something close to this:

the whole body

<!doctype html>
<html>
  <head>
    <title>Dashboard</title>
    <link rel="stylesheet" href="/assets/index-4f2a.css">
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/assets/index-9b7c.js"></script>
  </body>
</html>

That is the complete document. Everything a visitor reads is assembled afterwards, in the browser, by the script. A fetch gets the shell because the shell is genuinely all there is at that moment.

The word for this is client-side rendering, and it is the normal way single-page applications work. It is not an anti-scraping measure and there is nothing to defeat.

Telling a shell from a thin page

“Not much text” is not enough of a signal, because some pages really are short. The pattern is more specific than that.

A single empty mount element, often #root, #app or #__next.
Very little text against substantial markup, with most of the bytes in script tags.
Almost no links in the body, while the head is fully populated with meta tags.
Sometimes a <noscript> block saying the obvious thing.

Any one of these alone produces false positives. Together they are reliable, which is why the detection is worth doing properly rather than thresholding on text length.

Rendering is the fix, and it is expensive

The only real answer is to run the page in a browser, wait for the network to settle, and take the DOM afterwards. That works. It also changes the economics of your pipeline completely.

A static fetch is one request and a few tens of milliseconds. A rendered fetch launches a browser context, downloads and executes the JavaScript, waits for whatever that triggers, and then serialises the result. It is routinely two orders of magnitude slower, and it needs a couple of gigabytes of memory sitting around to do it.

Which is why rendering everything is the wrong default. It is also why rendering nothing is the wrong default. Detect, then render only the pages that need it.

Measure it rather than assume

The useful thing is knowing which of your targets are actually costing you. If your acquisition reports per-stage timings, that question answers itself: a render stage that dominates the total is the page telling you it is an application, not a document.

People are frequently wrong about which of their own targets those are. Marketing sites that look static are often framework-built; heavy-looking documentation sites are often prerendered and cheap. Per-stage timings turn that from a guess into a number.

Before you reach for a browser

Two things are worth checking first, because both are far cheaper.

The metadata may be enough. Shells usually have complete OpenGraph and JSON-LD tags in the head, because that is what social previews and search engines read. If you need a title, description, author and date, they are already there in the response you have.

There may be a static version. Many applications have an underlying API, an RSS feed, or a sitemap pointing at server-rendered pages. Fetching JSON directly is faster and steadier than driving a browser to watch it render.

Fail loudly

The worst outcome is not a slow render. It is a pipeline that silently ingests thousands of empty documents and reports success, which you discover weeks later when retrieval cannot answer anything about that source.

A page detected as a shell and not rendered should say so in the result. An empty document that knows it is empty is a bug you can find. One that does not is a bug you inherit.

snakereach turns any URL into the shape described here: markdown first, structure preserved, with per-stage timings. Try it on a page of your own, or read what comes back.

More writing