What are crawl errors? And how to fix crawl errors for SEO

Crawling and indexing are crucial steps in the SEO process because achieving a top ranking on search engine result pages begins with ensuring that Google crawls and indexes your web pages. Even if you have exceptional content and high-quality backlinks, any issues related to crawlability or indexability can adversely impact your site's rankings and visibility.

What are Crawlability and Indexability?

Crawlability refers to the ability of search engines to access your website's content. Search engines utilize search bots to navigate through your web pages by following the links between them.

Indexability, on the other hand, pertains to the search engine's capability to analyze the web pages it crawls and add them to its index. This process ensures that your website pages can be found by users when they perform relevant searches.

Now, let's discuss crawl errors. A crawl error occurs when search engine crawlers attempt to reach and explore every page of your website but encounter obstacles that prevent them from doing so. These errors have a negative impact on your website's SEO performance. To maintain your website's organic position, it is crucial to address crawl errors promptly.

Google categorizes crawl errors into two types:

1. Site errors: These errors indicate that your entire website cannot be crawled. They may arise due to issues with DNS (Domain Name System), server connectivity, or problems with the robots.txt file.
2. URL errors: This type of crawl error occurs when Googlebot is unable to access and read a specific page on your website. It includes soft 404 errors, which are relatively easier to identify and fix.

To delve deeper into these errors, it is important to understand the reasons behind their occurrence and learn how to resolve them effectively.

How to fix crawl errors for SEO?

DNS errors

DNS stands for Domain Name System. When a search engine encounters a DNS error, it means it cannot establish a connection to the server hosting your website. This can happen if the server is currently unavailable or if there is an issue with the DNS route to your website's domain. As a result, your website cannot be visited by users.

It's important to note that DNS errors are typically temporary in nature, and there is usually no need to worry. Google will revisit your website at a later time and attempt to crawl it again. In most cases, once the server or DNS issue is resolved, Google will be able to successfully access and crawl your website.

There are two types of DNS errors that Google encounters:

1. DNS Lookup: This error occurs when the website cannot be found. Google is unable to access your website because the DNS server couldn't locate the domain name.
2. DNS Timeout: This error indicates that Google was able to look up your URL on the DNS server, but the server fails to respond within a specific time limit.

While DNS errors do not directly impact your website's promotion, it is important to address them promptly to ensure a positive user experience.

How to fix?

To resolve a DNS error, Google suggests using Fetch as a Google tool to check how Googlebot crawls an important page on your website, such as the homepage. If the tool does not report any problems, it indicates that Google is able to access your website correctly.

However, if Google cannot fetch and render your page, it is recommended to contact your DNS provider to investigate if there are any issues on their side. They can help ensure that the DNS settings are configured correctly for your website.

Additionally, it is important to make sure that your server displays a proper error code, such as a 404 (page not found) or 500 (server error), instead of a DNS error. These error codes provide more meaningful information to both search engines and visitors, indicating that the requested page is unavailable or that there is a server issue.

Server errors

A server error occurs when the server hosting your website takes too long to respond, resulting in the Googlebot being unable to access your site. The Googlebot has a specific time limit for crawling a site, and if the server response exceeds this limit, the crawling process is halted, and an error message is returned. These errors are often indicated by 5xx status codes, such as 500 and 503.

The main reasons for server connectivity errors include:

1. Timeout: The server takes too long to respond, exceeding the crawling time limit.
2. Truncated or oversized headers: The headers sent by the server are either too large or improperly formatted, causing issues with the crawling process.
3. Truncated responses: The server sends incomplete or truncated responses, which can prevent the Googlebot from fully accessing and indexing your web pages.
4. Refused connections: The server refuses the connection request from Googlebot, denying access to your site.
5. Failed connections: The connection attempt between the Googlebot and the server fails due to various reasons, such as network issues or server overload.
6. Connection resets: The server abruptly terminates the connection with the Googlebot, leading to a failed crawling attempt.
7. Connection timeouts: The connection between the Googlebot and the server times out, indicating a failure to establish a successful connection.
8. No responses: The server does not provide any response to the crawling request, causing the crawling process to fail.

If you encounter server connectivity errors, it is recommended to investigate and resolve the underlying issues with your server configuration or hosting environment to ensure that the Googlebot can successfully access and crawl your website.

How to Fix?

Server errors may occur if your site receives a large number of visitors that your server cannot handle. If you experience connection response or timeout issues, ensure that your hosting provider can handle sudden bursts of traffic.

As mentioned earlier, you can use the "Fetch as Google" tool to verify if the Googlebot can successfully crawl your site.

Difference between DNS error and server error

DNS errors occur when the Googlebot is unable to access your site at all due to issues with the domain name system. This means that the server cannot be reached either because it is down or there is a problem with the DNS route to your website's domain. In contrast, server errors occur when the Googlebot can access your site, but it encounters difficulties loading the page due to server-related issues.

DNS errors prevent the Googlebot from visiting your website, while server errors allow the Googlebot to access your site but hinder its ability to load the page properly. Both types of errors can negatively impact your website's visibility and user experience.

Robots.txt failure

To understand this error, let's first understand what the "robots.txt" file is.

The "robots.txt" file is a code that instructs web crawlers on which pages of your website they should or should not crawl.

A robots.txt failure occurs when Googlebot is unable to access your robots.txt file. Initially, the Googlebot attempts to crawl your robots.txt file to determine if there are any specific areas of your website that you prefer not to be indexed. If the bot is unable to access the robots.txt file, Google will delay the crawl until it can access the file.

How to Fix?

Make sure that your robots.txt file is properly configured. An improperly configured robots.txt file can prevent important pages of your website from being indexed. Therefore, it is crucial to double-check which pages of your website you don't want the Googlebot to crawl. The robots.txt file is only necessary when you have specific URLs that you don't want the Googlebot to index. If you want the Googlebot to crawl every page on your website, you don't need to add even an empty robots.txt file to your domain.

URL errors

URL errors only affect specific pages of your website, not the entire website, and as a result, those pages will not be displayed in search results. These errors occur when a search engine crawler is unable to crawl a particular page due to incorrect redirects or an outdated sitemap.

URL errors can be categorized into the following types:

• Soft 404 errors
• Access denied errors
• Not followed errors
• Not found errors

Soft 404 errors

A soft 404 error occurs when the server sends a 200 OK status code for a requested webpage, but Google mistakenly interprets that the page should return a 404 error code. This typically happens when a page on your website has minimal or no content.

Soft 404 errors can have a negative impact on your website's crawl budget because Googlebot ends up spending a significant amount of time crawling non-existent pages instead of visiting your unique URLs. It is important to address these soft 404 errors to optimize the crawling process and ensure that Googlebot focuses on crawling valuable content on your website.

Reasons for Soft 404s

• The page has thin or no content
• Broken links
• A non-existent page redirects users to non-related pages like a home page.
• Soft 404 errors are also caused by misspelling URLs and outdated links.

How to fix?

If a page is truly unavailable or invalid, it is important to configure the server to display the appropriate error code, such as 404 (Page Not Found) or 410 (Gone). Additionally, ensure that the page contains sufficient and relevant content. If the page lacks content, you have several options: update the page with relevant information, redirect it to another relevant page, or delete the page entirely so that the URL leads to a 404 error page.

Avoid redirecting numerous dead pages to your home page. Instead, use 301 redirects to direct users and search engines to relevant or related pages on your website. It is also essential to regularly check and update your sitemap and internal links to ensure they are accurate and up to date. This helps search engines properly crawl and index your website's pages.

Access denied

"Access denied" means that the Google bot does not have access to a particular page. Common causes for this type of error include your hosting provider blocking the Google bot, pages being disallowed by the robots.txt file, and password protection preventing access to the page.

How to Fix?

• Remove the login requirement from the pages that you want Google to crawl.
• Ensure that your robots.txt files are accessible to search robots.
• Contact your hosting provider to whitelist the Google bot.

Not followed

The "Not followed" error occurs when Google is unable to follow a specific URL. It should not be confused with the "NO FOLLOW" link directives. Flash content, HTML frames, and JavaScript are common reasons for this error to occur.

How to fix?

• Check your redirect chains and ensure that the number of redirects is minimal. If there are too many redirects, Google may stop following the redirect chain.
• Verify that there are no redirect loops. Mutual redirects, where two pages redirect to each other, can cause a Google bot to get stuck in a loop.
• Review the redirect destinations and ensure they are redirected to relevant pages.
• Exclude the redirected URLs from your sitemap.
• Pay attention to your site architecture and make sure that every page on your website can be reached through static links.

Not found (404 error)

"Not found" URLs refer to actual 404 errors on your site, indicating that the URL you submitted for crawling corresponds to a page that does not exist. These errors can occur in both internal and external contexts and can manifest in various ways.

Internal "not found" errors typically occur when a page on your own website is no longer available or has been removed or renamed without proper redirection. This could happen due to changes in your site's structure, deletion of pages, or incorrect internal linking.

External "not found" errors occur when external links pointing to your site lead to non-existent pages. This could be due to outdated or incorrect links from other websites, or the pages being removed or relocated on your site without proper redirects.

How to fix?

1. Check your website's internal links and ensure they point to existing and relevant pages. Update or remove any broken or outdated links.
2. Implement proper 301 redirects for any removed or renamed pages to redirect visitors and search engines to the appropriate new location.
3. Monitor your site for external "not found" errors by regularly reviewing your website's backlink profile and identifying any broken or outdated links pointing to your site. Reach out to the referring websites and request them to update the links or remove them.
4. Customize your 404 error page to provide a helpful message and navigation options for users who encounter broken links.

Conclusion

I hope now you can understand the crawl errors to look out for. You can't ignore the crawl errors. If you notice any crawl errors, fix them as soon as possible and keep them on your site's maintenance checklist to improve your crawl budget and increase visibility in search engine results.