Enable Custom Robots.txt File in Blogger

Previously we had done the custom robots header tags . Today I share some knowledge about the second part of crawlers and indexing secti...


Custom robots.txt
Previously we had done the custom robots header tags. Today I share some knowledge about the second part of crawlers and indexing section means Custom Robots.txt. These both things do a same work, but in custom robots header tags we tick the boxes and in custom robots.txt, we write some codes to tell the search engine crawlers that whose pages are able to index and whose pages are not able for the index. For the newbie who don't know the definition we also share this below.

What is Custom Robots.txt?

Robots.txt is used to tell the search engine crawlers that whose pages can be indexed. These can be done by writing some codes in this section. For example, we write "Disallow:/p/about-us" and search crawlers will always disallow your about us page.

Add Custom Robots.txt File In Blogger

With these robots.txt you get high traffic for your website or blog. These are easy to add codes, just follow the steps as mention below.
  • First, check your robots.txt
http://www.yourdomain.com/robots.txt
  • Then go to blogger.com
  • Go to Setting>> Search Preferences
  • Click edit, on custom robots.txt.

Custom robots.txt in blogger

For users who use Adsense in their web add the below codes. And the users who don't have Adsense, follow the above first step, copy that and add in your blog.
User-agent: Mediapartners-Google
User-agent: *
Disallow: /search?q=*
Disallow: /*?updated-max=*
Allow: /
Sitemap: http://www.yourdmomain.com/feeds/posts/default?orderby=updated

Code's Definitions

 For your interest we are sharing the full definitions of codes that used in robots.txt.
  • User-agent: Mediapartners-Google: These codes define the web crawler that this blog has adsense advertisement. And your all pages indexed with google advertisements. Who don't have Adsense, simply remove this code.
  • User-agent:*: In this code User-agent is define the robots while * define the other search engine like Google, Yahoo and Bing.
  • Disallow: /search? q=*: With these codes you define the web crawlers that the search pages are not for the crawl. And it's better for your web.
  • Disallow: /*?updated-max=*: This code also disallows the label and navigation pages in search engine.
  • Allow: /: These codes allow your homepage to index in search engine.

Add Your Own Codes

You can also add your own codes like, you never want to index some pages. Then simply add User-agent:* first, and then apply this code Disallow:/p/Your-page-Here. Remember always add = after the codes.

Last Words

If you do these great you will see the improvement in your traffic. We have done the crawlers and indexing section in blogger. For any point, thoughts or other question that you want to ask from us related to this topic, so write it down in the comment box. In the next we share the knowledge about Creating of 404 page. Good luck and wait for the next post.

COMMENTS

Name

Android,2,backlinks,7,Blogger,46,blogger theme,1,Errors,1,How To,31,Make Money Online,2,SEO,13,SMM,10,Templates,9,Tools,7,Tricks,15,Web designing,18,Widgets,24,WordPress,4,
ltr
item
PRO TECHNIFY: Enable Custom Robots.txt File in Blogger
Enable Custom Robots.txt File in Blogger
http://2.bp.blogspot.com/-z6WgHWdycdE/VCBNHmC_e1I/AAAAAAAAH6M/z5aRg39goEk/s1600/Custom%2Brobots.txt.png
http://2.bp.blogspot.com/-z6WgHWdycdE/VCBNHmC_e1I/AAAAAAAAH6M/z5aRg39goEk/s72-c/Custom%2Brobots.txt.png
PRO TECHNIFY
https://protechnify.blogspot.com/2014/09/enable-custom-robotstxt-file-in-blogger.html
https://protechnify.blogspot.com/
https://protechnify.blogspot.com/
https://protechnify.blogspot.com/2014/09/enable-custom-robotstxt-file-in-blogger.html
true
8223102945718438291
UTF-8
Loaded All Posts Not found any posts VIEW ALL Readmore Reply Cancel reply Delete By Home PAGES POSTS View All RECOMMENDED FOR YOU LABEL ARCHIVE SEARCH ALL POSTS Not found any post match with your request Back Home Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sun Mon Tue Wed Thu Fri Sat January February March April May June July August September October November December Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec just now 1 minute ago $$1$$ minutes ago 1 hour ago $$1$$ hours ago Yesterday $$1$$ days ago $$1$$ weeks ago more than 5 weeks ago Followers Follow THIS CONTENT IS PREMIUM Please share to unlock Copy All Code Select All Code All codes were copied to your clipboard Can not copy the codes / texts, please press [CTRL]+[C] (or CMD+C with Mac) to copy