» » How To Add Custom Robots.txt File in Blogger?

How To Add Custom Robots.txt File in Blogger?

In one of my previous posts, I had discussed Custom Robots Header Tags Settings for blogger. 

If you have read that post then I hope you guys are aware of its importance in search rankings.

Today, I am back with a very useful and must aware blogging term that is Robots.txt.

In blogger it is known as Custom Robots.txt that means now you can customize this file according to your choice.

In today's tutorial, we will discuss this term in deep and come to know about its use and benefits. I will also tell you how to add custom robots.txt file in blogger.

So let's start the guide. 

More...

What is Robots.txt?

​Robots.txt is a text file which contains few lines of simple code.

It is saved on the website or blog's server which instruct the web crawlers on how to index and crawl your blog in the search results.

That means you can restrict any web page on your blog from web crawlers so that it can't get indexed in search engines like your blog labels page, your demo page or any other pages that are not as important to get indexed.

Always remember that search crawlers scan the robots.txt file before crawling any web page.

Each blog hosted on blogger has its default robots.txt file which is something look like this: 

User-agent: Mediapartners-Google

Disallow:

User-agent: *

Disallow: /search

Allow: /

Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED

Explanation 

This code is divided into three sections. Let's first study each of them after that we will learn how to add custom robots.txt file in blogspot blogs.

User-agent: Mediapartners-Google 

This code is for Google Adsense robots which help them to serve better ads on your blog. Either you are using Google Adsense on your blog or not simply leave it as it is.

User-agent: * 

This is for all robots marked with asterisk (*). In default settings our blog's labels links are restricted to indexed by search crawlers that means the web crawlers will not index our labels page links because of below code.

Disallow: /search 

That means the links having keyword search just after the domain name will be ignored. See below example which is a link of label page named SEO.

https://www.abc24.ga/search/label/SEO 

And if we remove Disallow: /search from the above code then crawlers will access our entire blog to index and crawl all of its content and web pages.

Here Allow: / refers to the Homepage that means web crawlers can crawl and index our blog's homepage.

Disallow Particular Post

Now suppose if we want to exclude a particular post from indexing then we can add below lines in the code.

Disallow: /yyyy/mm/post-url.html

Here yyyy and mm refers to the publishing year and month of the post respectively. For example if we have published a post in year 2013 in month of March then we have to use below format.

Disallow: /2013/03/post-url.html

To make this task easy, you can simply copy the post URL and remove the blog name from the beginning.

Disallow Particular Page

If we need to disallow a particular page then we can use the same method as above. Simply copy the page URL and remove blog address from it which will something look like this:

Disallow: /p/page-url.html

Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED 

This code refers to the sitemap of our blog. By adding sitemap link here we are simply optimizing our blog's crawling rate.

Means whenever the web crawlers scan our robots.txt file they will find a path to our sitemap where all the links of our published posts present.

Web crawlers will find it easy to crawl all of our posts.

Hence, there are better chances that web crawlers crawl all of our blog posts without ignoring a single one.

Note: This sitemap will only tell the web crawlers about the recent 25 posts. If you want to increase the number of link in your sitemap then replace default sitemap with below one. It will work for first 500 recent posts.

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

If you have more than 500 published posts in your blog then you can use two sitemaps like below:

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000

Adding Custom Robots.Txt to Blogger 

Now the main part of this tutorial is how to add custom robots.txt in blogger. So below are steps to add it.

  1. Go to your blogger blog.
  2. Navigate to Settings >> Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes
  3. Now paste your robots.txt file code in the box.
  4. Click on Save Changes button.
  5. You are done! 

How to Check Your Robots.txt File? 

You can check this file on your blog by adding /robots.txt at the end of your blog URL in the web browser. For example: 

http://www.yourblogurl.blogspot.com/robots.txt

Once you visit the robots.txt file URL you will see the entire code which you are using in your custom robots.txt file.

See below image.

Final Words! 

This was the today's complete tutorial on how to add custom robots.txt file in blogger.

I tried my best to make this tutorial as simple and informative as possible. But still if you have any doubt or query then feel free to ask me in the comment section below.

Make sure not to put any code in your custom robots.txt settings without knowing about it. Simply ask me to resolve your queries. I'll tell you everything in detail.

Thanks guys for reading this tutorial. If you liked it then please support me to spread my words by sharing this post on your social media profiles. Happy Blogging!

31 Responses to "How To Add Custom Robots.txt File in Blogger?"

  1. RaheemOctober 17, 2020 at 12:30 AM

    Great tips and very easy to understand. This will definitely be very useful for me when I get a chance to start my blog.

    notepad mac download

    ReplyDelete
  2. fgDecember 3, 2020 at 12:47 AM

    reference 91
    reference 92
    reference 93dv
    reference 94dsv
    reference 95dfb

    ReplyDelete
  3. Sam BakerJanuary 15, 2021 at 10:25 PM

    I'm glad I found this web site, I couldn't find any knowledge on this matter prior to.Also operate a site and if you are ever interested in
    doing some visitor writing for me if possible feel free to let me know, im always
    look for people to check out my web site.content://com.android.browser.home/index

    ReplyDelete
  4. Sam BakerJanuary 23, 2021 at 9:43 PM

    I can see that you are an expert at your field! I am launching a website soon, and your information will be very useful for me.. Thanks for all your help
    and wishing you all the success in your business.https://1asiaqq.net

    ReplyDelete
  5. Daniel AnthonyJanuary 24, 2021 at 3:39 AM

    AsiaQQ adalah Situs Judi Online, BandarQQ, DominoQQ, PokerQQ, QQ Online, Situs QQ, Poker Online, Agen BandarQ Online 24 jam...https://1asiaqq.net/

    ReplyDelete
  6. L E G E N D SMarch 4, 2021 at 6:24 AM

    In this article the author is going to carry out custom writing services market overview providing comprehensive survey of the market divided into the following categories: market identification, market size, market segmentation, market growth speed, market customers, principles of operation, financial characteristics, quality of products, pricing policies, regulations, competitors and leaders, current trends within the industry, market success factors, and other categories. The research is structured in such a way as to group related issues in a logical order.
    https://neva24.ru/socium/3584-problemy-vybora-pervoy-raboty.html

    ReplyDelete
  7. spark riderApril 22, 2021 at 3:43 AM

    Fabulous post, you have denoted out some fantastic points, I likewise think this s a very wonderful website. I will visit again for more quality contents and also, recommend this site to all. Thanks. convert pdf to png

    ReplyDelete
  8. sid seoMay 22, 2021 at 11:05 PM

    Just saying thanks will not just be sufficient, for the fantasti c lucidity in your writing. I will instantly grab your rss feed to stay informed of any updates. Mysql online editor

    ReplyDelete
  9. JamesyJune 22, 2021 at 6:50 AM

    I am very happy to discover your post as it will become on top in my collection of favorite blogs to visit. Website Hosting

    ReplyDelete
  10. hidayat aliJune 28, 2021 at 12:17 AM

    Very informative post! There is a lot of information here that can help any business get started with a successful social networking campaign. custom leather patches

    ReplyDelete
  11. king levisterJune 29, 2021 at 2:21 PM

    Nice to be visiting your blog once more, it has been months for me. Well this article that ive been waited for therefore long. i want this article to finish my assignment within the faculty, and it has same topic together with your article. Thanks, nice share. 먹튀사이트

    ReplyDelete
  12. king levisterJuly 11, 2021 at 12:11 AM

    I really like your writing style, great information, thankyou for posting. 먹튀검증

    ReplyDelete
  13. Daniel AnthonyJuly 30, 2021 at 6:37 AM

    I have read your article, it is very informative and helpful for me.I admire the valuable information you offer in your articles. Thanks for posting it..Bandarqq

    ReplyDelete
  14. bombayAugust 5, 2021 at 5:46 AM

    Good website! I truly love how it is easy on my eyes it is. I am wondering how I might be notified whenever a new post has been made. I have subscribed to your RSS which may do the trick? Have a great day! 먹튀사이트

    ReplyDelete
  15. jamesAugust 13, 2021 at 8:31 AM

    The content is utmost interesting! I have completely enjoyed reading your points and have come to the conclusion that you are right about many of them. You are great, and your efforts are outstanding! 토토커뮤니티

    ReplyDelete
  16. haaliAugust 25, 2021 at 1:37 AM

    Overpowering information, epic and bewildering development, as offer astonishing stuff with sharp assessments and bits of information, stores of cerebrum blowing information and inspiration, the two of which I need, by ethics of offer an especially strong information here. https://vograce.com

    ReplyDelete
  17. ・TeamAzPlayerSeptember 14, 2021 at 10:01 AM

    Nice knowledge gaining article. This post is really the best on this valuable topic. check this out

    ReplyDelete
  18. sameeSeptember 16, 2021 at 4:23 AM

    Thank you for taking the time to publish this information very useful! more info

    ReplyDelete
  19. Johnny September 18, 2021 at 3:55 AM

    Thanks for the blog loaded with so many information. Stopping by your blog helped me to get what I was looking for. Canada Visa Types

    ReplyDelete
  20. jackjohnnySeptember 21, 2021 at 4:29 AM

    You’ve got some interesting points in this article. I would have never considered any of these if I didn’t come across this. Thanks!. индийская виза онлайн

    ReplyDelete
  21. canadavisaSeptember 23, 2021 at 10:59 PM

    Hey – decent blog, just looking around plenty of blogs, seems a really nice platform you are using. I’m currently using WordPress for some of my sites but looking to change one amongst these over to a platform similar to yours as a trial run. Anything in particular you’ll recommend about it?
    visa de turista canadá

    ReplyDelete
  22. spark riderSeptember 29, 2021 at 4:06 AM

    An interesting dialogue is price comment. I feel that it is best to write more on this matter, it may not be a taboo topic however usually individuals are not enough to talk on such topics. To the next. Cheers. 대전스웨디시

    ReplyDelete
  23. UnknownOctober 24, 2021 at 12:12 AM

    Hey – decent blog, just looking around plenty of blogs, seems a really nice platform you are using. I’m currently using WordPress for some of my sites but looking to change one amongst these over to a platform similar to yours as a trial run. Anything in particular you’ll recommend about it?
    Chesterfield Footstool

    ReplyDelete
  24. jemmeyNovember 9, 2021 at 5:48 AM

    Thank you for helping people get the information they need. Great stuff as usual. Keep up the great work!!! joker

    ReplyDelete
  25. jemmeyNovember 12, 2021 at 12:45 AM

    Great post, and great website. Thanks for the information! Domanda di visto per il Canada

    ReplyDelete
  26. jemmeyNovember 14, 2021 at 3:21 AM

    I think that thanks for the valuabe information and insights you have so provided here. prediksi sydney

    ReplyDelete
  27. ConfirmNovember 18, 2021 at 11:03 AM

    I wish more authors of this type of content would take the time you did to research and write so well. I am very impressed with your vision and insight. check this link

    ReplyDelete
  28. alamNovember 19, 2021 at 1:45 AM

    This comment has been removed by the author.

    ReplyDelete
  29. haaliMarch 3, 2022 at 3:40 AM

    I truly regard this tangling post that you have obliged us. I guarantee this would be epic for by a wide edge by a wide edge past the thing may be thinking about all that typical a goliath piece of individuals. Miami LED robots

    ReplyDelete
  30. mohsinMarch 24, 2022 at 9:34 PM

    Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include. custom rocker patches

    ReplyDelete
  31. kulsoomApril 9, 2022 at 10:16 PM

    An interesting dialogue is price comment. I feel that it is best to write more on this matter, it may not be a taboo topic however usually individuals are not enough to talk on such topics. To the next. Cheers. visit this site

    ReplyDelete

Subscribe to: Post Comments (Atom)