Does Google crawl and index 404 pages (not found)? I had heard conflicting theories from multiple people about how the Googlebot responded when it discovered a 404 pages. Does it immediately consider the 404 to be a hard stop, or will it crawl this like any other and possibly discover any linked pages.
I set out to find out by conducting the following test.
- I created a brand new page on an authoritative domain. There were no internal nor external links to this page
- The URL and title of the page was a single keyword that did not exist in Google
- Added a link to the new page on a 404 page of another authoritative domain
- The anchor text of the link was a word a word that did not exist in Google
- Edited a footer link on the domain to contain a typo, so Googlebot would crawl the 404 as fast as possible
- Googlebot discovered the 404 in 6 hours.
- Google immediately crawled the “hidden” page
- The hidden page became the only result ranked on the non-existent keyword
- There are still no results for the non-existent keyword used in the anchor text on the 404 page
- Google does crawl links discovered on a 404 of an authoritative domain
- Googlebot does not trust the anchor text
I will continue testing to learn if the results change on a non-authoritative domain, and if a page can get ranked on a competitive keyword if the only link is on a 404 page.