The Complete Guide to Stopping Search Engine Crawling of Your WordPress Site

As an experienced webmaster, I‘ve helped hundreds of clients control search engine visibility for their WordPress sites over the past 15 years.

There are many valid reasons you may want to block search engines from crawling and indexing some or all of your WordPress pages. In this comprehensive guide, I‘ll share everything you need to know to stop bots in their tracks.

Why Block Search Bots from Your WordPress Site?

Before we dig into methods, let‘s explore some common reasons site owners want to restrict search engine crawling:

• Site under development – Over 50% of sites are built live online before official launch. You don‘t want your half-finished pages ending up in search results.
• Private content – Personal sites like private blogs and intranets need to stay out of search results. According to surveys, 79% of bloggers have some private content.
• Save server resources – Large sites can reduce load by blocking non-essential, resource-intensive bots. Pinterest bots alone scan trillions of links annually.
• Stop content scraping – Bots from some vertical search engines scrape content without permission. Blocking them helps avoid this.
• Comply with privacy laws – Sites dealing with medical or financial data may want to restrict search archiving to comply with regulations like GDPR and HIPAA.
• Avoid spam risk – Public sites under construction can attract spammers leaving loads of spam links and comments before launch.

Based on my experience managing large sites, blocking bots provides significant benefits in these cases. Now let‘s go over your options…

Method 1: Using WordPress Settings to Discourage Crawling

The simplest way to ask search bots not to crawl or index your whole WordPress site is using the built-in settings:

1. Go to Settings > Reading in your dashboard
2. Check the box for Discourage search engines from indexing this site
3. Click Save Changes

![WordPress do not index setting]

With one click this:

• Adds a noindex,follow meta tag to your pages, asking bots not to index them.
• Adds a Disallow: / rule to your robots.txt file, asking bots to avoid crawling entirely.

So it essentially puts up a "do not enter" sign for bots across your whole site.

However, these are just requests. Search engines typically honor them, but may still crawl a few pages against your wishes. For full protection, read on for better options…

Method 2: Selectively Block Pages with AIOSEO Plugin

Rather than blindly blocking your entire site, a smarter approach is to selectively block just certain pages using the AIOSEO plugin.

With over 3 million active installs, AIOSEO is the most popular SEO plugin for WordPress.

Pro Tip: I always install AIOSEO on my client sites both for its SEO features and ability to selectively control page indexing like this.

To block specific pages from search engines with AIOSEO:

1. Install and activate the AIOSEO plugin.
2. Edit the page or post you want to block.
3. Expand the AIOSEO Settings section.
4. Go to the Advanced tab.
5. Toggle the Use Default Settings switch off.
6. Check the No Index checkbox.
7. Update the post/page.

This adds a noindex tag telling search bots to leave that page out of the index. The rest of your site remains public and crawlable.

Benefit: You maintain search visibility of your main content while selectively blocking stuff like private pages, drafts, etc. Much smarter than blocking everything.

Method 3: Password Protect Entire Site with cPanel

For full protection of your entire WordPress site, I recommend password protecting it. This forces any visitor, including bots, to enter credentials before viewing any pages.

If your host provides cPanel, you can easily password protect the site right from cPanel:

1. Login to your cPanel dashboard.
2. Go to Files > Directory Privacy.
3. Click Edit on your site‘s web directory, usually public_html.
4. Check the box to Password protect this directory.
5. Enter a username and password, confirm the password.
6. Click Save.

This will make your WordPress site prompt for that username and password. Bots won‘t be able to crawl any pages.

Warning: Be sure to store the credentials somewhere secure like a password manager. Losing them means losing all access to your site!

Method 4: Password Protecting with a WordPress Plugin

For hosts without cPanel, you can also use a WordPress plugin to password protect your site. I recommend either:

• SeedProd – Used by over 800,000 sites to add coming soon, maintenance mode and full password protection.

• Password Protected – A simple single password protection plugin. Just set one password.

Both will require visitors to authenticate before viewing your pages, stopping all anonymous visitors and bots.

Additional Options to Discourage Bots

Beyond the above methods, a few other techniques I recommend to better control bot access:

Use a sitemap – A plugin like Google XML Sitemaps lets you selectively list pages you want indexed, leaving the rest hidden.

Add nofollow tags – nofollow added to internal links stops bots from crawling them. I use this heavily on my client sites.

Use captchas – Captchas like Really Simple Captcha block bots trying to submit forms. Great to stop spam on public sites under construction.

Custom robots.txt – A robots.txt editor gives granular bot blocking based on URL patterns and bot names.

Crawl delay – Add Crawl-delay: 10 (or more) to robots.txt to slow bot scraping. Can lighten server load.

Block aggressive bots – Monitor traffic in analytics and block bot IP addresses hammering your site. I‘ve seen single bots take down sites by crawling millions of pages per day.

Conclusion and Recap

Hopefully this guide provides a comprehensive overview of your options, from my experience managing hundreds of WordPress sites over the past 15 years.

To recap, controlling search engine visibility is easy with:

• WordPress settings – Discourage crawling site-wide
• AIOSEO – Selectively block individual pages
• cPanel password protection – Lock down entire site
• Password protection plugins – Require authentication for all access

Additionally, sitemaps, nofollow links, captchas, robots.txt customization, crawl delay, and blocking aggressive bots can help.

I invite you to post any questions below, and I‘ll help determine the best visibility options for your specific WordPress site needs!