Setting up Crawlers and Indexing in Blogger | Robots.txt, Meta tag & More

Setting up Crawlers and Indexing in Blogger | Robot.txt, Meta tag & More

Get a high ranking on Google with your Blogspot blog. How? Just Configure the Robots.txt, Meta Tags, Custom redirect, and Custom Robot header tag. It’s easy and interesting to understand!

This Guide on “Setting up Crawlers and Indexing in Blogger” is dedicated to newbie bloggers who just started their blogs on blogger and want to understand & configure their Indexing of their Blog.

These configurations of  Custom Robot header tags, Robots.txt, Custom redirects & description meta tag of your blog in the right way can help your blog to increase your search engine visibility and also, can lead to de-indexing your blog from Search Engine if you misconfigured it. So, I will recommend you to please be serious and more careful while reading this Guide. I am not scaring you but warning you that what the wrong configuration of Meta tags & Robots.txt can lead.

Let’s Start.

For Setting up Crawlers and Indexing in Blogger | Robots.txt, Meta tag & More follow these steps:

*Login to your Blogspot dashboard

*Go to Setting

*Now, go to search preferences section

*You’re there!

Setting up description meta tag in blogger?

This is where you can write an informative short description about your blog in 150 words. Writing better and enough Good description can lead you better search engine ranking and clicks. But don’t forget to add your main keywords. For example Wikiijunction – A Blog where you can get Informative Guides Based on WordPress, Blogger, SEO, and Google Adsense tips & tricks.

In my Meta description, I have told about what my site is about and included my main keywords on which I want to rank. Don’t exceed your meta description more than 150 words as the crawler bots will ignore characters after 150 words.

After adding meta description the crawlers will crawl your blog and shows up the description on the search engine result page.


Setting up Robots.txt in Blogger

Robots.txt is a file which contains some lines of code which give proper instruction to the crawler to how to index, which part of your blog not to index or block crawler bots from some of the pages of crawl your blog. For example, I do not want to crawl some of the pages or posts such as archive, tags, etc so I will add disallow: /URL  in the robots.txt file.

Don’t add or edit the robotx.txt file if you don’t know. Otherwise, it can lead to de-indexing of your blog. Crawlers read the robots.txt file before crawling any blog or website.

This video made by Ex-Google Engineer, Matt Cutts will give you enough basic information about Blogger robots.txt file:

Default robots.txt file

User-agent: Mediapartners-Google


User-agent: *

Disallow: /search

Allow: /



Don’t know what is sitemap and how to create it? Read this post “What is SEO and How it works? | Google SEO Guide For Beginners” to create your blog sitemap.

Let me Explain you these terms

 *[User-agent: *]: –This code is for all bots as it is marked with (*).

*[User-agent: Mediapartners-Google]: This code is for the Google partners such as Google Adsense. If you don’t have the Google Adsense on your blog then leave it.

 *[Disallow: /search]: –  This code means not to index the links which contain “/search” term. For example If we remove this the crawlers will index all the links.

*[Allow: / ]: – This code is for allowing all bots to crawler the homepage of your site.

*[sitemap: ]: – This code is for referring bots the sitemaps of your Blog. Sitemaps are where your all posts, pages, categories, & Images are listed in the form of XML language.

How to make your sitemap of your blog? Go to XML-Sitemap generator and download the sitemap, it will in the form of XML. Add the sitemap on your site and provide the link in the robots.txt file.

Now, your robots.txt is ready! Now, its time to add in the blogger.

How to Add the Custom Robots.txt in Blogger

In this, I am gonna tell you how to add the custom Robots.txt in Blogger. So, follow the below steps to add the robots.txt:

  • Login in Blogger
  • Setting>> search preferences>> Crawlers and Indexing>> Custom Robots.txt
  • Edit>>click on yes
  • Now, paste the code and save the changes.
  • That’s it!

Robots.txt | meta tags


Check your robots.txt file by putting robots.txt after your blog URL.

Setting up Custom Redirects in Blogger

Custom Redirects gives you options for the redirection of your broken links to the right URL. Optimizing your broken links will you give a boost to your blog traffic and helps your readers to come to the right location. You can make any redirection as temporary (302) or permanent (301) by just tick the permanent checkbox or left it if you want to make it 302 temporary redirections.

Custom redirect
Image credit:


Setting up Custom Robots Header Tags in Blogger

These flags are used to set the robots header tags served to search engines. These tags are used to control an individual page indexing for better serving to the users in SERP. These robots meta tags and description tags in the <head> section of your blog posts make it search engine friendly.

Let me explain all the directives and these are not case sensitive.

  • Custom Robots header tags in blogger | meta tags
  • All:- It will allow the bots to crawl everything and index.
  • noindex:- It will tell the bots to crawl but not to index it.
  • nofollow:- It is utilized for telling the bots not follow the links in posts or pages.
  • none:- It is used for providing the information to the bots that nofollow as well as noindex it.
  • noarchive:- This is used for telling the bots not to show cache links in the search results.
  • nosnippet:- This is used for telling the bots that not to show snippet in the search results.
  • notranslate:- This is used for telling the bots not to translate the text into other languages.
  • noodp:- this option can not allow search engines to use metadata from the DMOZ directory.
  • noimageindex:- This is used for telling the spiders not to index the images.
  • unavailable_after: This is very useful as it can be used for deindexing the zombie pages that are too old by just providing the date/month/year after the term.

How to Add Robots Header Tags in Blogger


  • Login to your blogger dashboard

meta tags


  • Setting >> Search Preferences

Robots header tags in blogger

  • Now Navigate Custom Robots header tags and edit it.

Setting up Custom robots header tags in blogger


  • Make the changes and save it.

How to add custom robots header tags in blogger

  • That’s it!


Bottom Line

Custom Robots Headers tags, Meta Tags, and Robots.txt gives the powerful boost in SEO to Blogspot blog. Blogger makes it easy to configure the Custom Robots headers tags and give the proper way to tell the Search Engine bots to What to index or what they have to follow or not.


These Setting gives a huge increase in your traffic and also increase your Search engine ranking. Without the knowledge of coding the, anyone can tell the spiders that they do or do not want to index or not to index the pages, posts, and categories.


Google also gives the proper information about all these tags and how to utilize them for your blog to make it search engine friendly. You can find the link below the read the full guide give my the Google. It will help you a lot to customize your Meta tags and custom robots header tags.




For any query, you can post the comment below. If you like the post then feel free to share!


Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.