Are you looking for a way to prevent search engines from crawling in WordPress? Ranking higher on search engines is the primary goal of almost every single website. This is because ranking higher will increase the amount of traffic your website will receive, but what if you do not want to receive any web traffic at all?
While this may be a strange thought for web developers that have poured countless hours into strengthening their website’s SEO and marketing, it is not uncommon. There are actually quite a few reasons a web developer would not want a website to appear on search engines. Today, I will demonstrate how to prevent search engines from crawling in WordPress.
Why Prevent Search Engine Crawling in WordPress
You’re probably still questioning why you would want to prevent search engines from crawling your website and indexing your pages and there are many possibilities. For example, while most web developers will use a local server they can create using Xampp or Wamp to test their latest design changes, some do not.
Instead, they choose to do it on a live website, but they don’t want to receive any traffic on this website since it is for testing purposes. Thus, it becomes a necessity to block crawlers in WordPress, but that is not something many web developers know how to do since they are desperately doing the opposite.
Of course, this is not the only reason, imagine you run a small business and use a website to share information with staff members. Odds are, you probably don’t want to let customers see this. Thus, making it private is important, which is essentially what blocking search bots will do.
Another great example is for educational purposes. Many colleges require some form of a website for your final projects, but you may not want your work to be shared. Other students could find it and try to copy, this may be a patentable idea (depending on the major), or maybe it’s a requirement of the professor.
In any case, learning how to stop search engines in WordPress can be a valuable skill.
How to Prevent Search Engines From Crawling in WordPress
Today, I will demonstrate how to prevent search engines from crawling in WordPress. There are a few ways to go about it, but they have varying results. For example, the first method I will show you is to use the built-in WordPress feature to ask search bots not to index your pages, but search engines are free to ignore that request.
The more surefire way, but more difficult, is to password protect your website to ensure search bots do not have access to the website. Failure to enter a correct username and password will result in a 401 error.
Asking Search Engines Not To Index Your Pages
The key word here is asking because search engines can choose to ignore this completely, but most will respect your request. Why demonstrate this method then you ask? Because it is beginner friendly and much simpler and convenient than password protecting your website.
On the left-hand admin panel, click on Settings and select the Reading option.
The last option of this page is Search Engine Visibility. Check the box to discourage search engines from indexing your website. There is also a disclaimer underneath the option indicating that it is up to the search engine to honor this request.
Click on the “Save Changes” button to save your changes. Now, as long as the search engines honor your request, your website will not be indexed.
Password Protecting Your Entire Website
To even begin this method, you will need access to your website’s cPanel. The login information is provided to you by your web hosting provider.
Log in to your website’s cPanel and click on the Directory Privacy option.
This area of the cPanel will allow you to password protect any individual directory. Almost every web host will store your website’s root files within the public_html directory, but there is also a chance it could use the www directory, although it’s unlikely. You can tell which one has your website’s information if you find “wp-” prefixes inside one of them.
Click on the name of the directory that stores all of your website’s information. If you click on the folder icon, you will enter the folder and not actually select it for use.
Now it’s time to create a username and password for the directory. You can add a directory name, but I would just recommend using whatever the actual directory name is. Create a username and password for your directory. Click on the “Save” button when you are done.
Congratulations, you have successfully password protected your entire website. This will ensure that search engine bots cannot get access to your pages, which will mean they will not be indexed. This also improves the security of your website, but make sure you don’t forget your username and password.
Make Your Website Private
While the idea of a private website is not new, it is becoming rarer. Most web developers are looking to share their website with the world, but keeping your website isolated has its moments. This is a great way to make a testing server on the actual Internet. Since you are testing ideas that could be stolen, you don’t want those accessible for everyone to see.
Why do you want to prevent search engines from indexing your website? How easy do you find the password protection method to be?