Google constantly scours the Internet looking for content. But what if there are things on your website you don’t want search engines to access or index for search results? Not everyone wants every post or page immediately available to bots. This is when you hide WordPress pages and posts.
When you hide content from bots, the idea is the post or page becomes hidden from search engines. This means only the direct users of the site should have access to them. This is often done for privacy as well as protecting information from being scraped by others and copied.
In this article, I’m going to show you how to hide WordPress posts and pages from Google. There are several ways you can do this, and all of them are relatively simple.
Using Noindex/Nofollow Command
The “Noindex” and “Nofollow” commands are intended to guide bots away from certain articles. For instance, noindex tells crawlers such as Google that a particular post or page is not to be indexed into the search engine. If a bot comes across a page with the meta data for noindex, the entire content is skipped.
One of the best tools to use for this is Yoast SEO. This plugin has a lot of features for enhancing search engine optimization including an easy-to-use system to turn on or off noindex and nofollow.
To install Yoast, go to the Plugins area and click “Add New.”
Install and active the Yoast SEO plugin.
Yoast will install a new section to the WordPress editor. This gives you access to a variety of tools that are beneficial for creating and marketing content. It also has a function to control some of the meta data within a post or page.
Access a post or create a new one to open the WordPress editor.
Scroll down until you see the Yoast SEO window. It should be directly under the content section in the post.
You’ll see a series of icons along the left of this window. Click on the image that looks like a gear.
Under “Meta robots index,” you will see a drop down option window. Click this to change the post type to “noindex.”
Under “Meta robots follow,” click the option for “nofollow.” This makes it so search engines will not use links to influence the searching index. It helps stop the post you want to hide from scoring in the results page.
Once you update or publish the post, search engines will then ignore its contents. This option is also available in pages and some custom post types. Yoast is capable of modifying the meta data for a variety of content.
Using Password Protection in WordPress
Some bots are designed to ignore the meta data information of a post or page. Sometimes, it’s more secure to simply add a password to your content. This means no one else will be able to access the post or page even if its a bot. Luckily, WordPress comes built with its own privacy settings.
When looking at a post or page, you’ll see an option in the Publish window for “Visibility.” Click the “Edit” link to open privacy settings of WordPress.
This will open a drop down section. If you click the “Password protected” or “Private” options, search engines like Google will be unable to crawl the content. For the sake of this tutorial, I’ll select the “Password protected” option to secure the post.
The “Password” text box will appear immediately. Set a password for the post or page and click, “ok.”
This will keep visitors and search engines out of the post or page. Unfortunately, this also means you need to give the password to the people who need to read the contents.
If you use the “Private” option, any of your users who have an Editor role or higher can read the content. This requires managing roles in WordPress, which is a good thing to know if you plan on having a lot of people helping build the website.
This actually has quite a few possibilities ranging from paid content to business memos and information.
Modifying the robots.txt File
In reality, I’m not a fan of modifying the robots.txt file. Although some search engines will abide by the requests of this file, not all of them will. In fact, a lot of malicious bots and content scrapers will ignore the file altogether.
Another reason is that the robots.txt file is publicly accessible. Otherwise, the file would be pointless. This also means anyone can read the robots.txt file and see the URLs you’re trying to hide.
But in the event of preventing Google from indexing a post, let’s make a few adjustments anyway. It’s a simple single line of text you add and only takes a minute or two.
For this tutorial, we’re using Yoast SEO. This plugin is like the Swiss Army Knife of SEO tools for WordPress. It has a wide range of functions that help you deal with search engine optimization.
Go to the “SEO” area on the left and click “Tools.” You may have to turn on the advanced options from the Yoast dashboard to see this ability.
Yoast comes with a couple different editors. For editing robots.txt, click the “File editor” link from the list.
If you have a robots.txt file available, it will be displayed here. But as you can see, I do not have one on this site. If that is the case, click the “Create robots.txt file” button and Yoast will build it.
Coincidentally, you can also edit your .htaccess form this page as well. This is a good thing to know for future use.
NOTE: If the page changes and shows a bunch of HTML coding, just hit the back button on your browser. This happens sometimes when creating the robots.txt file in Yoast. It will be created, but sometimes the “success” page doesn’t load correctly. The file will be available for editing.
With the Robots.txt editor ready, enter the following code:
[ht_message mstyle=”info” title=”” show_icon=”” id=”” class=”” style=”” ]Disallow: /2017/7/your-article[/ht_message]
Change the “/2017/7/your-article” to the URL of the page or post you want to hide. This will be everything after your address domain suffix. For instance, if your address was “https://bobsmeathut.com/tutorials/cooking-with-pork/”, you would put:
[ht_message mstyle=”info” title=”” show_icon=”” id=”” class=”” style=”” ]Disallow: /tutorials/cooking-with-pork/[/ht_message]
Once you enter the URL you wish to hide, click the “Save changes to Robots.txt” button on the bottom.
You can use single posts, categories or even block access to specific directories of the site using the robots.txt file. Unfortunately, this doesn’t stop all bots from crawling your content. However, Googlebot does respond to this file and respects your privacy.
Not Everything Needs to Be Search-Engine Friendly
Although you want as much content as possible in search engines to help drive traffic, not everything needs to be publicized. The above methods help reduce showing those elements in sites like Google. If you want to make the entire site inaccessible, you may want to consider setting your website to be completely private. This way, neither search engines or scrapers will be able to access the content.
What additions have you made for improving the SEO value of your site? What have you done to hide your content from search engines?
thanks that was very helpful