Why do agents skip pages that return 404?

We record the status of every page we crawl, and a page that did not answer 200 receives no further Checks.

What it is

A Deep Scan fetches up to 50 addresses taken from your sitemap and your own links. Each one is recorded with the status it returned.

This Check appears in a Report only when a page answered with something other than 200. A page that returned 200 raises no Issue here and moves on to the Machine Readability Checks.

Where the request failed outright, the Evidence names the cause in words: no answer within the time limit, or a refused connection.

Why an agent needs it

A page you list in a sitemap or link from your navigation is a page you are telling an agent to fetch. A 404 or a 500 there spends the agent's request budget on nothing.

Non-200 pages are also the reason a Report can look thin. We do not measure headings or prices on a page we never received.

The Fix

Fetch every address in your sitemap and confirm it returns 200. Remove the ones that do not, or restore the pages.

Where a page has moved, return a 301 to its new address. A soft 404 that answers 200 with an error page is worse than either.

Fix internal links that point at the dead address. An agent following your own navigation should never arrive at an error.

This one is a change in your own markup or edge configuration, so the Fix Pack cannot write it for you. A Report names every page it affects, with the Evidence.

More in Machine Readability

All 24 Checks·Check library 2026-08-27