{"id":116716,"date":"2019-04-30T16:26:33","date_gmt":"2019-04-30T08:26:33","guid":{"rendered":"https:\/\/seo-hacker.com\/?p=16716"},"modified":"2023-02-08T11:26:56","modified_gmt":"2023-02-08T03:26:56","slug":"complete-guide-robots-txt-noindex-meta-tag","status":"publish","type":"post","link":"https:\/\/seo-hacker.com\/complete-guide-robots-txt-noindex-meta-tag\/","title":{"rendered":"The Complete Guide to Robots.txt and Noindex Meta Tag"},"content":{"rendered":"

\"The<\/a><\/p>\n

How do I use Robots.txt and the noindex meta tag?<\/h2>\n

Quick Answer:<\/strong> To create a Robots.txt file, you can use any text editor (such as Notepad). Make sure to save the file with UTF-8 encoding during the save file dialog. This file must also be named \u201crobots.txt,\u201d and your site can only have one such file. This file must also be located at the root of the website host you\u2019re applying it to. To use a noindex tag for pages that you do not want to be included in search results, add \u201c<meta name=\u201drobots\u201d content=\u201dnoindex\u201d>\u201d to the <head> section. Or, you can add the noindex tag using a X-Robots-Tag in the HTTP header: \u201cX-Robots-Tag: noindex\u201d.<\/p>\n

<\/p>\n

Overview<\/h2>\n

The Robots.txt file and Noindex meta tag are important for doing on-page SEO. This gives you the power to tell Google which pages they should crawl and which pages they should index – display in the search results. <\/span><\/p>\n

Knowing how to use these two and <\/span>when <\/b>to use them is important for all SEOs since this involves a direct relationship between the websites we\u2019re handling and the search engine crawlers. Being able to direct the search engine crawlers on where they should go and which pages they should include in the database is a massive advantage for us, and we can use that to make sure that only our website\u2019s important pages are the ones that Google and other search engines crawl and index. But before we delve into the details of <\/span>how <\/b>and <\/span>when <\/b>to use these two, we must first know what they are and their specific functions. <\/span><\/p>\n

What is a Robots.txt file?<\/span><\/h2>\n

The Robots Exclusion Protocol, or more commonly known as Robots.txt<\/a> is a file that directs web crawlers and robots such as Googlebot and Bingbot to which pages of your websites should not be crawled.<\/span><\/p>\n

What is the use of a Robots.txt file?<\/span><\/h2>\n

The robots.txt file is only a crawling directive and it cannot control how fast a bot should crawl your website and other bot behaviors. This is just a set of instructions for bots on what parts of your website should not be accessed.<\/span><\/p>\n

You should also take note that while some bots respect robots.txt file, some can ignore it. Some robots can exploit files on your website or even harvest information so to completely block malware robots, you should increase your site security or protect private pages by putting a password. If you have other questions about the robots.txt, check out some <\/span>frequently asked questions on robots<\/span><\/a> here.<\/span><\/p>\n

How to Create a Robots.txt File?<\/span><\/h2>\n

By default, a robots.txt file would look like this:<\/span><\/p>\n

\"Robots<\/a><\/p>\n

You could create your own robots.txt file in any program that is in .txt file type. You could block different URLs such as your website\u2019s blog\/categories or \/author pages. Blocking pages like this would help bots prioritize important pages on your website more. The robots.txt file is a great way of managing your crawl budget.<\/span><\/p>\n

Robots crawling directives<\/span><\/h3>\n\n\n\n\n\n\n
User-agent<\/span><\/td>\nSpecifies the crawl bot you want to block from crawling a URL eg. Googlebot, Bingbot, Ask, Yahoo. Here\u2019s a link to a <\/span>directory of known web crawlers<\/span><\/a><\/td>\n<\/tr>\n
Disallow<\/span><\/td>\nSpecifies that a URL and all other URLs under it should be blocked<\/span><\/td>\n<\/tr>\n
Allow<\/span><\/td>\nThis is only followed by Googlebot. It tells it that a page can be crawled even if the parent page is disallowed<\/span><\/td>\n<\/tr>\n
Sitemap<\/span><\/td>\nSpecifies the location of your website\u2019s sitemap<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Proper usage of wildcards<\/span><\/h3>\n

In the robots.txt, a wildcard, represented as the (*) symbol, can be used as a symbol for any sequence of characters.<\/span><\/p>\n

A directive for all types of crawl bots:<\/span><\/i><\/p>\n

User-agent:*<\/span><\/p>\n

 <\/p>\n

The wildcard could also be used to disallow all URLs under the parent page except for the parent page.<\/span><\/p>\n

User-agent:*<\/span><\/p>\n

Disallow: \/authors\/*<\/span><\/p>\n

Disallow: \/categories\/*<\/span><\/p>\n

 <\/p>\n

This means all page URLs under the main author page and categories page are blocked except for them.<\/span><\/i><\/p>\n

 <\/p>\n

A good example of a robots.txt file would look like this<\/span><\/p>\n

User-agent:*<\/span><\/em><\/p>\n

Disallow: \/testing-page\/<\/span><\/em><\/p>\n

Disallow: \/account\/<\/span><\/em><\/p>\n

Disallow: \/checkout\/<\/span><\/em><\/p>\n

Disallow: \/cart\/<\/span><\/em><\/p>\n

Disallow: \/products\/page\/*<\/span><\/em><\/p>\n

Disallow: \/wp\/wp-admin\/<\/span><\/em><\/p>\n

Allow: \/wp\/wp-admin\/admin-ajax.php<\/span><\/em><\/p>\n

 <\/p>\n

Sitemap: yourdomainhere.com\/sitemap.xml<\/span><\/em><\/p>\n

 <\/p>\n

After editing your robots.txt file, you should upload in the <\/span>top-level directory of your website\u2019s code<\/b> so when a bot enters your website for crawling, it would see the robots.txt file first.<\/span><\/p>\n

What is Noindex?<\/span><\/h2>\n

Noindex is a <\/span>meta robots tag<\/span><\/a> that tells search engines not to include a page in the search results. <\/span><\/p>\n

How to Implement Noindex Meta Tag?<\/span><\/h2>\n

There are three ways to put a noindex tag on pages you don\u2019t want search engines to index:<\/span><\/p>\n

Meta Robots Tag<\/span><\/h3>\n

In the <head> section of the page, place the following code:<\/span><\/p>\n

<meta name=\u201drobots\u201d content=\u201dnoindex\u201d><\/span><\/p>\n

 <\/p>\n

The code may vary depending on your decision. The code mentioned tells all types of crawl bots from indexing a page. Alternatively, if you only want to noindex a page from a specific crawl bot, you could place the name of that bot in the meta name.<\/span><\/p>\n

 <\/p>\n

To prevent Googlebot from indexing a page:<\/span><\/p>\n

<meta name=\u201dgooglebot\u201d content=\u201dnoindex\u201d><\/span><\/p>\n

 <\/p>\n

To prevent Bingbot from indexing a page:<\/span><\/p>\n

<meta name=\u201dbingbot\u201d content=\u201dnoindex\u201d><\/span><\/p>\n

 <\/p>\n

You can also instruct bots to follow or don\u2019t follow links that are found on the page you noindexed.<\/span><\/p>\n

 <\/p>\n

To follow links in the page:<\/span><\/p>\n

<meta name=\u201drobots\u201d content=\u201dnoindex,follow\u201d><\/span><\/p>\n

 <\/p>\n

To tell bots to not crawl the links in the page:<\/span><\/p>\n

<meta name=\u201drobots\u201d content=\u201dnoindex,nofollow\u201d><\/span><\/p>\n

X-Robots-Tag<\/span><\/h3>\n

The x-robots-tag allows you to control the indexing of a page in the HTTP response header of the page. The x-robots-tag is similar to the meta robots tag but it also allows you to tell search engines not to show specific file types in the search results such as images and other media files.<\/span><\/p>\n

To do this, you need to have access to your website\u2019s .php, .htaccess, or server access file. Directives in the meta robots tag are also applicable to the x-robots-tag. Here\u2019s a great article about the <\/span>X-Robots-Tag in HTTP headers<\/span><\/a>.<\/span><\/p>\n

Through YoastSEO<\/span><\/h3>\n

If you\u2019re using YoastSEO in WordPress, there is no need for you to manually place these codes. Just go to the page or post you want to noindex, scroll down to the YoastSEO interface, go to the settings of the post by clicking the gear icon and then select \u201cNo\u201d under \u201cAllow Search Engines to Show this Post in Search Results?\u201d<\/span><\/p>\n

\"Yoast<\/a><\/p>\n

You could also put a noindex tag sitewide for pages such as categories, tags, and author pages so you don\u2019t have to go to every individual page on your website. To put a noindex tag, go to the Yoast plugin itself and then go to Search Appearance. Selecting \u2018no\u2019 under \u2018Show Categories in Search Results\u2019 would place a noindex tag on all category pages.<\/span><\/p>\n

\"Yoast<\/a><\/p>\n

Best Practices<\/span><\/h2>\n

Many people are still confused by these two. It is critical as an SEO to know what the difference is in between. This is crucial in making sure that the pages that you want the users to see in the search results are the only pages that appear and the pages you want bots to crawl are the only pages that get crawled.<\/span><\/p>\n

 <\/p>\n