{"id":106349,"date":"2013-03-21T17:36:57","date_gmt":"2013-03-21T09:36:57","guid":{"rendered":"https:\/\/seo-hacker.com\/?p=6349"},"modified":"2022-05-02T11:00:05","modified_gmt":"2022-05-02T03:00:05","slug":"robotstxt-meta-tags-affects-search-engine-crawling","status":"publish","type":"post","link":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/","title":{"rendered":"How Robots.txt and Meta Tags Affect Search Engine Crawling"},"content":{"rendered":"

\"Googlebot\"<\/a><\/span><\/p>\n

If you are concerned about the privacy of your website and you do not want the search engine crawlers or bots to crawl certain pages of your website, then \u201cRobots.txt\u201d is the one-stop solution that will keep the crawlers away from the \u2018No Entry\u2019 zone.<\/span><\/p>\n

<\/span><\/p>\n

Webmaster’s Note:<\/strong> This is a guest post by Sarah Bruce<\/em><\/span><\/p>\n

Confused? Probably, you are wondering about the need of keeping the search engine bots away from the pages, when everyone wants their website to be indexed in the search engines. Sure.<\/span><\/p>\n

Reason for stopping the bots from entering certain pages of a website<\/span><\/h2>\n

\"noindex\"<\/a><\/span><\/p>\n

If yours is an e-commerce website and you store your database on it, would you like to disclose the database of your clients\u2019 information to the entire world? Definitely not! But, if you do not take any precautionary measures to indicate the crawlers not to crawl those pages with vital information, then search engine spiders will crawl them eventually and index those pages in the search engine results. From there, anybody can view the detail of your clients and use it unethically, to put you and your clients in a position of legal nightmare.<\/span><\/p>\n

To avoid such disaster, you should use robots.txt.\u2018Robots.txt\u2019 plays the similar role as a bouncer in a club. Like how bouncers do not allow certain guests to enter private sections of the club, so does robots.txt. Consider it as a file which includes the directories that shouldn\u2019t be entered by specific or all crawlers.<\/span><\/p>\n

Now, this question arises: Are your pages safe with robots.txt?<\/span><\/h2>\n

Search Engine crawlers are built from artificial intelligence and before visiting any page of the website, these bots look out for the existence of robots.txt file, where they can see the pages that they are prevented from accessing.<\/span><\/p>\n

Don’t worry about search engine bots violating the robots.txt file of your website. If they do so, they have to face severe legal consequences, which is why they have no option but to respect your robots.txt file.<\/span><\/p>\n

The Bad news is that there are malicious spammers who also make use of robots to crawl the website’s ‘private’ pages, which you pretty much can’t do anything about. So, it is highly recommended to use firewalls, encryption methods, password protection and other security services besides robots.txt.<\/span><\/p>\n

In and out of \u2018robots.txt\u2019!<\/span><\/h2>\n

Not everyone needs robots.txt. Unless you have some serious content in your website, which you do not want anybody to look into, there is no mandatory need to upload a robots.txt file and not even an empty one.<\/span><\/p>\n

Robots.txt file contains a set of instructions for the search engine crawlers, as in the files and directories that are not supposed to be crawled. A noteworthy point here is that this file should be installed in the highest level directory of your website because crawlers search for robot.txt file in the root domain of your website and not in any sub-domain.<\/span><\/p>\n

For example, http:\/\/www.abc.com\/robots.txt<\/i> is a valid location, but http:\/\/www.abc.com\/mysite\/robots.txt<\/i> is invalid.<\/span><\/p>\n

How to create a robots.txt file?<\/b><\/span><\/h2>\n

There are two important parts of a robots.txt file:<\/span><\/p>\n

\"Google-Spider\"<\/a>User-agent:<\/i><\/b> It symbolizes a search engine bot. You can indicate either all the search engine bots or a specific bot.<\/span><\/p>\n

Disallow:<\/i><\/b> This is the field, which allows or disallows the search engines to crawl specific files or directories.<\/span><\/p>\n

If you want all search engines not to crawl a directory, then use a * on the User-Agent section then follow the directory name with a forward slash:<\/span><\/p>\n

    \n
  1. User-agent: *<\/span><\/li>\n<\/ol>\n

    Disallow: \/directoryA\/<\/span><\/p>\n

    If you want particularly, Bingbot not to crawl a directory, then follow the directory name with a forward slash:<\/span><\/p>\n

      \n
    1. User-agent: Bingbot<\/span><\/li>\n<\/ol>\n

      Disallow: \/ directoryA \/<\/span><\/p>\n

      If you want all search engines not to crawl the complete website, then:<\/span><\/p>\n

        \n
      1. User-agent: *<\/span><\/li>\n<\/ol>\n

        Disallow: \/<\/span><\/p>\n

        If you want to restrict the search engine bots from crawling a page, then:<\/span><\/p>\n

          \n
        1. User-agent: *<\/span><\/li>\n<\/ol>\n

          Disallow: \/abc_file.html<\/span><\/p>\n

          Google uses many bots, such as Googlebot-Image\u00a0and Googlebot-Mobile, however the conditions applied to Googlebot will be applied to all, but the case is not vice-versa. You can set specific rules for the specific bots, as well. <\/i><\/span><\/p>\n

          To block an image from Google Images, use the following:<\/span><\/p>\n

            \n
          1. User-agent: Googlebot-Image<\/span><\/li>\n<\/ol>\n

            Disallow: \/images\/ watch.jpg<\/span><\/p>\n

            To remove all images from Googlebot Images, use:<\/span><\/p>\n

              \n
            1. User-agent: Googlebot-Image<\/span><\/li>\n<\/ol>\n

              Disallow: \/<\/span><\/p>\n

              If you want to block a specific file type, for example\u2014.png, then:<\/span><\/p>\n

                \n
              1. User-agent: Googlebot<\/span><\/li>\n<\/ol>\n

                Disallow: \/.png<\/span><\/p>\n

                You can be certain of the pages not crawled by search engine bots, if you have indicated them in your robots.txt. However, if the URLs of those pages are found in other pages of your website, then there’s a certain narrow chance that those pages will also be indexed.<\/span><\/p>\n

                To avoid such kind of trouble, it is recommended that you use \u2018robots meta tag\u2019, to restrict any kind of access to the specific page. Let us dig out little information about robots Meta tag, to understand it better.<\/span><\/p>\n

                Robots Meta Tag: In Depth<\/b><\/span><\/h2>\n

                \u2018Index\u2019<\/b> and \u2018noindex\u2019<\/b> are the two major instructions of a Meta tag, as it allows you to have a control on the indexing page-by-page. If you do not want the search engine bot to index a specific page, then put the following Meta tag at the head section of your page:<\/span><\/p>\n

                <meta name=”robots” content=”noindex”><\/i><\/span><\/p>\n

                If you do not want a specific bot to index a page, for example\u2014Googlebot, then:<\/span><\/p>\n

                <meta name=”Googlebot” content=”noindex”> <\/i><\/span><\/p>\n

                Search engine crawlers will only crawl the pages that they are allowed to. But, if they find the links on other pages, they may not overlook those URLs and end up in indexing those pages. It is not necessary that the bots will index the pages, where you have used the Meta tag to \u2018index\u2019. However, the certain thing is that search engine bots will abruptly drop the pages, which are asked to \u201cnoindex\u201d, even if they have been linked to other pages.<\/span><\/p>\n

                Remember that if you have included a \u2018noindex\u2019 meta tag in a page, but that page is not included in the robots.txt, search engine bots will crawl that page and the moment it comes across \u2018noindex\u2019 tag, it will drop it.<\/span><\/p>\n

                There could be a possibility that despite of adding a \u2018noindex\u2019 Meta tag, the page still appears in the search result. Don\u2019t panic – the reason could be: the crawlers didn\u2019t appear back to crawl your page since you have added the Meta tag. It will be definitely removed the next time the crawler crawls your page.<\/span><\/p>\n

                To speed up the index removal process, you can also make use of Google’s URL removal tool.<\/a><\/span><\/p>\n

                Final Touch: Test your robots.txt file through Google Webmaster Tools
                \n<\/b><\/span><\/h2>\n

                This test is advised to be performed on a \u2018Test robots.txt\u2019<\/b> tool, before you upload the robots.txt file in your website\u2019s root domain. This test will give you the actual result, as it reads the website as Googlebot does.<\/span><\/p>\n

                Performing this test is a plus, as you will know if the robots.txt file is blocking or permitting a page, accidentally. Accordingly, you can fix the problems, if any found. Let us see, how to use the tool:<\/span><\/p>\n

                \u00a8\u00a0\u00a0\u00a0\u00a0 Click on the website that you want to check, in the Webmaster Tools home page.<\/span><\/p>\n

                \u00a8\u00a0\u00a0\u00a0\u00a0 Under \u2018Health\u2019 section, click \u2018Blocked URLs.<\/span><\/p>\n

                \u00a8\u00a0\u00a0\u00a0\u00a0 \u2018Test robots.txt\u2019 tab must be selected, by default. If it is not, then click on the tab.<\/span><\/p>\n

                \u00a8\u00a0\u00a0\u00a0\u00a0 You need to copy the content of your robots.txt file and paste it in the first box.<\/span><\/p>\n

                \u00a8\u00a0\u00a0\u00a0\u00a0 Copy and paste the sites that need to be tested in the \u2018URLs\u2019 box<\/span><\/p>\n

                \u00a8\u00a0\u00a0\u00a0\u00a0 List the user-agents in the \u2018User-agents\u2019 box.<\/span><\/p>\n

                Do remember that you cannot make any change from within the tool, but you need to edit the content of the robots.txt file.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

                If you are concerned about the privacy of your website and you do not want the search engine crawlers or bots to crawl certain pages of your website, then \u201cRobots.txt\u201d is the one-stop solution that will keep the crawlers away from the \u2018No Entry\u2019 zone.<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"wl_entities_gutenberg":"","footnotes":""},"categories":[100013],"tags":[101807,101775,101774,100442,101773,101772,101771],"wl_entity_type":[102583],"class_list":["post-106349","post","type-post","status-publish","format-standard","hentry","category-seo-tips-and-tricks","tag-abc-comrobots-txt","tag-google-crawler","tag-google-spider","tag-meta-robots","tag-no-index","tag-nofollow","tag-robots-txt","wl_entity_type-article"],"acf":[],"yoast_head":"\nHow Robots.txt and Meta Tags Affect SEO and Crawling<\/title>\n<meta name=\"description\" content=\"If you are concerned about the privacy of your website and you do not want the search engine crawlers or bots to crawl certain pages of your website, then \u201cRobots.txt\u201d is the one-stop solution that will keep the crawlers away from the \u2018No Entry\u2019 zone.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How Robots.txt and Meta Tags Affect SEO and Crawling\" \/>\n<meta property=\"og:description\" content=\"If you are concerned about the privacy of your website and you do not want the search engine crawlers or bots to crawl certain pages of your website, then \u201cRobots.txt\u201d is the one-stop solution that will keep the crawlers away from the \u2018No Entry\u2019 zone.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Services Agency in Manila, Philippines\" \/>\n<meta property=\"article:published_time\" content=\"2013-03-21T09:36:57+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-05-02T03:00:05+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/seo-hacker.com\/wp-content\/uploads\/2013\/03\/Googlebot.jpg\" \/>\n<meta name=\"author\" content=\"Sean Si\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/\"},\"author\":{\"name\":\"Sean Si\",\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/#\\\/schema\\\/person\\\/5565b7823135bb49dd1618bbcaec2dbf\"},\"headline\":\"How Robots.txt and Meta Tags Affect Search Engine Crawling\",\"datePublished\":\"2013-03-21T09:36:57+00:00\",\"dateModified\":\"2022-05-02T03:00:05+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/\"},\"wordCount\":1330,\"commentCount\":12,\"image\":{\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/seo-hacker.com\\\/wp-content\\\/uploads\\\/2013\\\/03\\\/Googlebot.jpg\",\"keywords\":[\"abc.com\\\/robots.txt\",\"google crawler\",\"google spider\",\"meta robots\",\"no index\",\"nofollow\",\"robots.txt\"],\"articleSection\":[\"SEO tips and tricks\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/\",\"url\":\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/\",\"name\":\"How Robots.txt and Meta Tags Affect SEO and Crawling\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/seo-hacker.com\\\/wp-content\\\/uploads\\\/2013\\\/03\\\/Googlebot.jpg\",\"datePublished\":\"2013-03-21T09:36:57+00:00\",\"dateModified\":\"2022-05-02T03:00:05+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/#\\\/schema\\\/person\\\/5565b7823135bb49dd1618bbcaec2dbf\"},\"description\":\"If you are concerned about the privacy of your website and you do not want the search engine crawlers or bots to crawl certain pages of your website, then \u201cRobots.txt\u201d is the one-stop solution that will keep the crawlers away from the \u2018No Entry\u2019 zone.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/#primaryimage\",\"url\":\"https:\\\/\\\/seo-hacker.com\\\/wp-content\\\/uploads\\\/2013\\\/03\\\/Googlebot.jpg\",\"contentUrl\":\"https:\\\/\\\/seo-hacker.com\\\/wp-content\\\/uploads\\\/2013\\\/03\\\/Googlebot.jpg\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/robotstxt-meta-tags-affects-search-engine-crawling\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/seo-hacker.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How Robots.txt and Meta Tags Affect Search Engine Crawling\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/#website\",\"url\":\"https:\\\/\\\/seo-hacker.com\\\/\",\"name\":\"SEO Services Agency in Manila, Philippines\",\"description\":\"SEO Hacker is an SEO Agency and SEO Blog in the Philippines. Let us take your website to the top of the search results with our holistic white-hat strategies. Inquire today!\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/seo-hacker.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/seo-hacker.com\\\/#\\\/schema\\\/person\\\/5565b7823135bb49dd1618bbcaec2dbf\",\"name\":\"Sean Si\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/3225fbc3fa29eafa997934ff429b9b1949121b469f7a110079f055ad4eeffd25?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/3225fbc3fa29eafa997934ff429b9b1949121b469f7a110079f055ad4eeffd25?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/3225fbc3fa29eafa997934ff429b9b1949121b469f7a110079f055ad4eeffd25?s=96&d=mm&r=g\",\"caption\":\"Sean Si\"},\"description\":\"Sean Si is a Filipino motivational speaker and a Leadership Speaker in the Philippines. He is the head honcho and editor-in-chief of SEO Hacker. He does SEO Services for companies in the Philippines and Abroad. Connect with him at Facebook, LinkedIn or Twitter. Check out his new project...\",\"sameAs\":[\"https:\\\/\\\/sean.si\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How Robots.txt and Meta Tags Affect SEO and Crawling","description":"If you are concerned about the privacy of your website and you do not want the search engine crawlers or bots to crawl certain pages of your website, then \u201cRobots.txt\u201d is the one-stop solution that will keep the crawlers away from the \u2018No Entry\u2019 zone.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/","og_locale":"en_US","og_type":"article","og_title":"How Robots.txt and Meta Tags Affect SEO and Crawling","og_description":"If you are concerned about the privacy of your website and you do not want the search engine crawlers or bots to crawl certain pages of your website, then \u201cRobots.txt\u201d is the one-stop solution that will keep the crawlers away from the \u2018No Entry\u2019 zone.","og_url":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/","og_site_name":"SEO Services Agency in Manila, Philippines","article_published_time":"2013-03-21T09:36:57+00:00","article_modified_time":"2022-05-02T03:00:05+00:00","og_image":[{"url":"https:\/\/seo-hacker.com\/wp-content\/uploads\/2013\/03\/Googlebot.jpg","type":"","width":"","height":""}],"author":"Sean Si","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/#article","isPartOf":{"@id":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/"},"author":{"name":"Sean Si","@id":"https:\/\/seo-hacker.com\/#\/schema\/person\/5565b7823135bb49dd1618bbcaec2dbf"},"headline":"How Robots.txt and Meta Tags Affect Search Engine Crawling","datePublished":"2013-03-21T09:36:57+00:00","dateModified":"2022-05-02T03:00:05+00:00","mainEntityOfPage":{"@id":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/"},"wordCount":1330,"commentCount":12,"image":{"@id":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/#primaryimage"},"thumbnailUrl":"https:\/\/seo-hacker.com\/wp-content\/uploads\/2013\/03\/Googlebot.jpg","keywords":["abc.com\/robots.txt","google crawler","google spider","meta robots","no index","nofollow","robots.txt"],"articleSection":["SEO tips and tricks"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/","url":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/","name":"How Robots.txt and Meta Tags Affect SEO and Crawling","isPartOf":{"@id":"https:\/\/seo-hacker.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/#primaryimage"},"image":{"@id":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/#primaryimage"},"thumbnailUrl":"https:\/\/seo-hacker.com\/wp-content\/uploads\/2013\/03\/Googlebot.jpg","datePublished":"2013-03-21T09:36:57+00:00","dateModified":"2022-05-02T03:00:05+00:00","author":{"@id":"https:\/\/seo-hacker.com\/#\/schema\/person\/5565b7823135bb49dd1618bbcaec2dbf"},"description":"If you are concerned about the privacy of your website and you do not want the search engine crawlers or bots to crawl certain pages of your website, then \u201cRobots.txt\u201d is the one-stop solution that will keep the crawlers away from the \u2018No Entry\u2019 zone.","breadcrumb":{"@id":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/#primaryimage","url":"https:\/\/seo-hacker.com\/wp-content\/uploads\/2013\/03\/Googlebot.jpg","contentUrl":"https:\/\/seo-hacker.com\/wp-content\/uploads\/2013\/03\/Googlebot.jpg"},{"@type":"BreadcrumbList","@id":"https:\/\/seo-hacker.com\/robotstxt-meta-tags-affects-search-engine-crawling\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/seo-hacker.com\/"},{"@type":"ListItem","position":2,"name":"How Robots.txt and Meta Tags Affect Search Engine Crawling"}]},{"@type":"WebSite","@id":"https:\/\/seo-hacker.com\/#website","url":"https:\/\/seo-hacker.com\/","name":"SEO Services Agency in Manila, Philippines","description":"SEO Hacker is an SEO Agency and SEO Blog in the Philippines. Let us take your website to the top of the search results with our holistic white-hat strategies. Inquire today!","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/seo-hacker.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/seo-hacker.com\/#\/schema\/person\/5565b7823135bb49dd1618bbcaec2dbf","name":"Sean Si","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/3225fbc3fa29eafa997934ff429b9b1949121b469f7a110079f055ad4eeffd25?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/3225fbc3fa29eafa997934ff429b9b1949121b469f7a110079f055ad4eeffd25?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/3225fbc3fa29eafa997934ff429b9b1949121b469f7a110079f055ad4eeffd25?s=96&d=mm&r=g","caption":"Sean Si"},"description":"Sean Si is a Filipino motivational speaker and a Leadership Speaker in the Philippines. He is the head honcho and editor-in-chief of SEO Hacker. He does SEO Services for companies in the Philippines and Abroad. Connect with him at Facebook, LinkedIn or Twitter. Check out his new project...","sameAs":["https:\/\/sean.si"]}]}},"_wl_alt_label":[],"jetpack_featured_media_url":"","wl:entity_url":"http:\/\/data.wordlift.io\/wl0320\/post\/how_robots-txt_and_meta_tags_affect_search_engine_crawling","_links":{"self":[{"href":"https:\/\/seo-hacker.com\/wp-json\/wp\/v2\/posts\/106349","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/seo-hacker.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/seo-hacker.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/seo-hacker.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/seo-hacker.com\/wp-json\/wp\/v2\/comments?post=106349"}],"version-history":[{"count":0,"href":"https:\/\/seo-hacker.com\/wp-json\/wp\/v2\/posts\/106349\/revisions"}],"wp:attachment":[{"href":"https:\/\/seo-hacker.com\/wp-json\/wp\/v2\/media?parent=106349"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/seo-hacker.com\/wp-json\/wp\/v2\/categories?post=106349"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/seo-hacker.com\/wp-json\/wp\/v2\/tags?post=106349"},{"taxonomy":"wl_entity_type","embeddable":true,"href":"https:\/\/seo-hacker.com\/wp-json\/wp\/v2\/wl_entity_type?post=106349"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}