Secure Robots.txt settings and Special Robot Header Tags on Blogger

How to Set Robots.txt and Custom Robot Header Tags that are Safe on Blogger - Configuring settings on Robot.txt and Special Robot Header Tags or Special Robot Header Tags are intended so that posts can be easily crawled/indexed by search engines. By implementing this, the blog becomes SEO friendly.

How To Set Robots.txt Specially Safe For Bloggers

1. Use Robots.txt Default Only

The default robots.txt setting defaults to the blogger is the safest, but that doesn't mean it's not effective. You don't need to activate the robots.txt setting on your blog, because for the default robots.txt it has been active since you first created the blog. Check this https://your-blog.blogspot.com/robots.txt. Here's the default robots.txt from blogger.

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Allow: /

Sitemap: https://your-blog.blogspot.com/sitemap.xml

Information:
User-agent: Mediapartners-Google {Markers that blogs use AdSense}
Disallow: {Which is not allowed to exist}
User-agent: * {All search engine robots}
Disallow: /search {Does not allow after /search url like ... /search/label and ... search/search?Updated ...}
Allow: / {which is allowed to be crawled. (/) Is the name of the blog.}
Sitemap: https://your-blog.blogspot.com/sitemap.xml {Sitemap/sitemap blog.}

We focus on the code Disallow: /search, meaning we limit or not allow crawling on the page like https://your-blog.blogspot.com/search/label/SEO, because the link does not go to one page of a full article.

2. Custom robots.txt

If you still want to customize it on robots.txt, then you can apply the robots.txt modification below.

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /p/about.html
Disallow: /p/contact.html
Allow: /
Sitemap: https://your-blog.blogspot.com/sitemap.xml

Customization of the robots.txt above does not allow search engines to crawl the About page and Contact pages on static pages. if you want certain pages not to be crawled by search engines, then you just add as above.

How to Apply Robots.txt on Blogger/Blogspot

Here are the steps to implement custom robots.txt in blogger.

After logging in, select the blog that you want to customize the robots.txt, then click Settings > Search Preferences. After that, activate the special robots.txt and place your custom robots.txt code like this.

How to Set a Special Robot Header Tag

To activate a special robot header tag please click Settings > Search Preferences > Special robot header tags. Then do the settings as shown below.

The above settings aim to allow search engines to crawl home pages, default for posts and pages (all blog contents), but do not allow search engines to index archive and search pages (search models that occur in blogs).

Thus the article on how to set robots.txt and custom robot header tags specifically on the blog. Hopefully this article is useful for the continuity of your blog.

source:blog.krakentechno.com