What Does the Error Mean?
A “Blocked due to unauthorized request (401)” error indicates that Googlebot, the search engine crawler for Google, is being blocked from accessing certain pages on your website.
This can happen for a few reasons, such as:
Password protection: If your website requires a password to access certain pages, Googlebot won’t be able to crawl them.
IP restrictions: If you have restrictions in place that block certain IP addresses, you may be inadvertently blocking Googlebot.
Authentication errors: There may be an issue with your website’s authentication process that is preventing Googlebot from logging in.
How to Fix
The solution to this error depends on whether you want Google to index the affected pages. Here are two options:
1. Allow Googlebot to Index the Pages
If you want Google to index the pages, you’ll need to change your website’s settings to allow Googlebot access. Here’s how:
Check your password protection: If your pages are password-protected, you’ll need to remove the password protection. This approach is ideal if the restricted content offers value to searchers. However, displaying different content to Googlebot than to human users is considered “cloaking” and can violate Google’s Webmaster Guidelines.
Use schema markup for paywalled content: If removing access restrictions isn’t an option, you can signal to Google the nature of your restricted content. Use schema markup for paywalled content to provide Google with details about the content behind the paywall. This helps Google understand the content’s value without actually accessing it.
Review your IP restrictions: If you have IP restrictions in place, make sure that Googlebot’s IP addresses are not being blocked. You can find a list of Googlebot’s IP addresses on Google’s Search Console Help website.
Fix authentication errors: If you suspect there is an issue with your website’s authentication process, you’ll need to troubleshoot the problem and fix it.
2. Block Googlebot from Indexing the Pages
If you don’t want Google to index the affected pages, you can use the Disallow directive in your robots.txt file to block Googlebot from accessing them. Here’s how:
Create a robots.txt file: If you don’t already have a robots.txt file, you’ll need to create one. You can add a Disallow directive to this file to block Googlebot from specific pages or directories.
Add the Disallow directive: The Disallow directive should look like this:
Disallow: /path/to/blocked/page/
You can add multiple Disallow directives to your robots.txt file to block multiple pages or directories.
By following these steps, you should be able to fix the “Blocked due to unauthorized request (401)” error and ensure that your website is being properly indexed by Google.
Additional Tips
Once you’ve made changes to your website’s settings or robots.txt file, it may take some time for Google to recrawl your website and update its index.
You can use the Test Live URL in Google Search Console to test how Googlebot sees your website. This can help you identify any issues that may be preventing Googlebot from accessing your pages.
FAQs
A 401 Unauthorized Error occurs when a request lacks valid authentication credentials, preventing access to the resource.
It prevents Googlebot from accessing and indexing affected pages, potentially leading to lower search rankings and visibility.
Yes, misconfigured security plugins can block access and trigger 401 errors.
Changes can take a few days to a couple of weeks to reflect in Google Search Console.
