Does Google crawl 404 pages?

Does Google crawl and index 404 pages (not found)? I had heard conflicting theories from multiple people about how the Googlebot responded when it discovered a 404 pages. Does it immediately consider the 404 to be a hard stop, or will it crawl this like any other and possibly discover any linked pages.

I set out to find out by conducting the following test.

  1. I created a brand new page on an authoritative domain. There were no internal nor external links to this page
  2. The URL and title of the page was a single keyword that did not exist in Google
  3. Added a link to the new page on a 404 page of another authoritative domain
  4. The anchor text of the link was a word a word that did not exist in Google
  5. Edited a footer link on the domain to contain a typo, so Googlebot would crawl the 404 as fast as possible

Results:

  • Googlebot discovered the 404 in 6 hours.
  • Google immediately crawled the “hidden” page
  • The hidden page became the only result ranked on the non-existent keyword
  • There are still no results for the non-existent keyword used in the anchor text on the 404 page

Conclusions:

  • Google does crawl links discovered on a 404 of an authoritative domain
  • Googlebot does not trust the anchor text

I will continue testing to learn if the results change on a non-authoritative domain, and if a page can get ranked on a competitive keyword if the only link is on a 404 page.

 

Leave a Comment

Your email address will not be published. Required fields are marked *