Enable Custom Robots.txt File in Blogger

Previously we had done the custom robots header tags . Today I share some knowledge about the second part of crawlers and indexing secti...

Previously we had done the custom robots header tags. Today I share some knowledge about the second part of crawlers and indexing section means Custom Robots.txt. These both things do a same work, but in custom robots header tags we tick the boxes and in custom robots.txt, we write some codes to tell the search engine crawlers that whose pages are able to index and whose pages are not able for the index. For the newbie who don't know the definition we also share this below.

What is Custom Robots.txt?

Robots.txt is used to tell the search engine crawlers that whose pages can be indexed. These can be done by writing some codes in this section. For example, we write "Disallow:/p/about-us" and search crawlers will always disallow your about us page.

Add Custom Robots.txt File In Blogger

With these robots.txt you get high traffic for your website or blog. These are easy to add codes, just follow the steps as mention below.

• First, check your robots.txt

http://www.yourdomain.com/robots.txt

• Then go to blogger.com
• Go to Setting>> Search Preferences
• Click edit, on custom robots.txt.

For users who use Adsense in their web add the below codes. And the users who don't have Adsense, follow the above first step, copy that and add in your blog.

User-agent: Mediapartners-Google
User-agent: *
Disallow: /search?q=*
Disallow: /*?updated-max=*
Allow: /
Sitemap: http://www.yourdmomain.com/feeds/posts/default?orderby=updated

Code's Definitions

 For your interest we are sharing the full definitions of codes that used in robots.txt.

• User-agent: Mediapartners-Google: These codes define the web crawler that this blog has adsense advertisement. And your all pages indexed with google advertisements. Who don't have Adsense, simply remove this code.
• User-agent:*: In this code User-agent is define the robots while * define the other search engine like Google, Yahoo and Bing.
• Disallow: /search? q=*: With these codes you define the web crawlers that the search pages are not for the crawl. And it's better for your web.
• Disallow: /*?updated-max=*: This code also disallows the label and navigation pages in search engine.
• Allow: /: These codes allow your homepage to index in search engine.

Add Your Own Codes

You can also add your own codes like, you never want to index some pages. Then simply add User-agent:* first, and then apply this code Disallow:/p/Your-page-Here. Remember always add = after the codes.

Last Words

If you do these great you will see the improvement in your traffic. We have done the crawlers and indexing section in blogger. For any point, thoughts or other question that you want to ask from us related to this topic, so write it down in the comment box. In the next we share the knowledge about Creating of 404 page. Good luck and wait for the next post.