What Does the Error Mean?
A “Blocked due to unauthorized request (401)” error indicates that Googlebot, the search engine crawler for Google, is being blocked from accessing certain pages on your website.
This can happen for a few reasons, such as:
Password protection: If your website requires a password to access certain pages, Googlebot won’t be able to crawl them.
IP restrictions: If you have restrictions in place that block certain IP addresses, you may be inadvertently blocking Googlebot.
Authentication errors: There may be an issue with your website’s authentication process that is preventing Googlebot from logging in.
How to Fix
The solution to this error depends on whether you want Google to index the affected pages. Here are some options:
Allow Googlebot to Index the Pages
If you want Google to index the pages, you’ll need to change your website’s settings to allow Googlebot access. Here’s how:
Step 1: Verify the Error in GSC
- Go to Google Search Console → Pages
- Click on Blocked due to unauthorized request (401)
- Note the affected URLs
- Click Inspect URL on a sample URL to see when Google last attempted to crawl it
Step 2: Test the URL Yourself
In Your Browser:
- Open an incognito/private window
- Visit the affected URL
- Do you see a login prompt or 401 error page?
Using cURL (for technical users):
curl -I https://yoursite.com/affected-page
Look for:
HTTP/1.1 401 Unauthorized WWW-Authenticate: Basic realm="Restricted Area"
Using Online Tools:
– HTTP Status Code Checker Enter your URL and check if it returns 401
Step 3: Check for Password Protection
If your pages are password-protected, you’ll need to remove the password protection. This approach is ideal if the restricted content offers value to searchers. However, displaying different content to Googlebot than to human users is considered cloaking and can violate Google’s Webmaster Guidelines.
WordPress:
- Go to Settings → Reading
- Check if Search Engine Visibility is enabled (should be UNchecked)
- Check if you’re using a coming soon/maintenance plugin
- Verify individual pages: Edit page → Page Attributes → Password Protection
Shopify:
- Go to Online Store → Preferences
- Check Password protection section
- Ensure it’s disabled for public-facing pages
cPanel/Server Level:
- Check for .htpasswd files in affected directories
- Review .htaccess for authentication rules:
AuthType Basic AuthName "Restricted Area" AuthUserFile /path/to/.htpasswd Require valid-user
Step 4: Inspect Security Plugin Settings
Many security plugins block Googlebot accidentally. Check these:
Wordfence (WordPress):
- Go to Wordfence → Firewall → All Firewall Options
- Scroll to Rate Limiting Rules
- Ensure Googlebot isn’t being blocked
- Check Wordfence → Tools → Whois → Enter Googlebot IP
- Verify it’s not blacklisted
iThemes Security (WordPress):
- Go to Security → Settings → 404 Detection
- Ensure Ignore 404 errors from logged-in users is enabled
- Check Security → Settings → Network Brute Force
- Whitelist Google’s IP ranges if needed
Cloudflare:
- Log into Cloudflare dashboard
- Go to Security → WAF
- Check for rules blocking user agents
- Review Firewall Events for Googlebot blocks
Step 5: Check Server Logs
Apache:
tail -f /var/log/apache2/error.log | grep 401
Nginx:
tail -f /var/log/nginx/error.log | grep 401
Look for entries showing Googlebot requests being denied.
Step 6: Test as Googlebot
Use the URL Inspection tool:
- Go to Google Search Console
- Enter the affected URL
- Click Test Live URL
- Click View Tested Page → Screenshot
- Compare what Googlebot sees vs. what you see
If Googlebot sees a login page but you don’t, you’ve found your issue.
Use schema markup for paywalled content: If removing access restrictions isn’t an option, you can signal to Google the nature of your restricted content. Use schema markup for paywalled content to provide Google with details about the content behind the paywall. This helps Google understand the content’s value without actually accessing it.
Review your IP restrictions: If you have IP restrictions in place, make sure that Googlebot’s IP addresses are not being blocked. You can find a list of Googlebot’s IP addresses on Google’s Search Console Help website.
Fix authentication errors: If you suspect there is an issue with your website’s authentication process, you’ll need to troubleshoot the problem and fix it.
Block Googlebot from Indexing the Pages
If you don’t want Google to index the affected pages, you can use the Disallow directive in your robots.txt file to block Googlebot from accessing them. Here’s how:
Create a robots.txt file: If you don’t already have a robots.txt file, you’ll need to create one. You can add a Disallow directive to this file to block Googlebot from specific pages or directories.
Add the Disallow directive: The Disallow directive should look like this:
Disallow: /path/to/blocked/page/
You can add multiple Disallow directives to your robots.txt file to block multiple pages or directories.
By following these steps, you should be able to fix the “Blocked due to unauthorized request (401)” error and ensure that your website is being properly indexed by Google.
Once you’ve made changes to your website’s settings or robots.txt file, it may take some time for Google to recrawl your website and update its index.
You can use the Test Live URL in Google Search Console to test how Googlebot sees your website. This can help you identify any issues that may be preventing Googlebot from accessing your pages.
Common Causes by Platform
WordPress 401 Causes
1. Security Plugins Blocking Googlebot
Plugins like Wordfence, Sucuri, or iThemes may mistakenly identify Googlebot as a threat.
Fix:
// Add to functions.php or custom plugin
add_filter('wordfence_is_googlebot', '__return_true');
2. Membership/LMS Plugins
Plugins like MemberPress, Restrict Content Pro, or LearnDash may protect too many pages.
Fix:
– Review which content is members-only
– Ensure public preview pages aren’t restricted
– Add exception rules for search engines
3. HTTP Authentication in .htaccess
Check your `.htaccess` file for:
<Files wp-login.php> AuthType Basic
AuthName "Protected"
AuthUserFile /path/to/.htpasswd
Require valid-user
</Files>
Fix:
Ensure this ONLY protects wp-login.php, not public pages.
4. Server-Level IP Blocking
Your host may block Google’s IP ranges.
Whitelist Googlebot IPs:
# In .htaccess
SetEnvIf User-Agent "Googlebot" AllowBot
Order Deny, Allow
Deny from all
Allow from env=AllowBot
Shopify 401 Causes
1. Password-Protected Store
Most common cause – store is in password mode while in development.
Fix:
- Shopify Admin → Online Store → Preferences
- Scroll to “Password protection”
- Uncheck “Enable password” (unless intentional)
2. Hidden Collections
Collections set to “Hidden” in search engines.
Fix:
- Go to Products → Collections
- Click on the collection
- Scroll to “Search engine listing preview”
- Ensure it’s not hidden
3. Shopify Scripts Blocking Access
Custom Shopify Scripts may inadvertently block crawlers.
Fix:
Review theme.liquid and Scripts for authentication checks.
FAQs
A 401 Unauthorized Error occurs when a web server requires authentication credentials to access a resource, but the request either lacks credentials or provides invalid ones. For websites, this means Googlebot cannot access your pages because they’re password-protected or behind authentication barriers.
A 401 error prevents Googlebot from accessing and indexing affected pages, which can lead to:
1) Pages being removed from search results,
2) Loss of search rankings and visibility,
3) Decreased organic traffic, and
4) Wasted crawl budget on inaccessible pages. If important pages return 401 errors, your site’s overall search performance will decline significantly.
Yes, security plugins like Wordfence, iThemes Security, Sucuri, and membership plugins like MemberPress can cause 401 errors by either blocking Googlebot’s IP addresses, requiring authentication for public pages, or implementing overly aggressive rate limiting that treats legitimate crawlers as threats. Check your security plugin settings and whitelist Googlebot user agents.
After fixing 401 errors, changes typically take 24-48 hours to reflect in Google Search Console when using the ‘Request Indexing’ feature. Natural recrawling without manual requests can take anywhere from a few days to 2-3 weeks depending on your site’s crawl frequency and the page’s importance. Monitor the URL Inspection tool to track when Google successfully recrawls the fixed pages.
To allow Googlebot to access paywalled or protected content, use structured data markup for paywalled content (schema.org/NewsArticle with isAccessibleForFree property). Show preview content to all users including search engines, clearly indicate the paywall, and never show completely different content to bots versus users (that’s cloaking). You can also whitelist Googlebot’s user agent in your authentication middleware while keeping the page protected for regular users.
