ROBOT.TXT file is a very important part of the website because the SEO Performance of your site is very much dependent on ROBOTS.TXT file. I think you are reading this post to increase your website’s search performance, it’s good.
Also read – Custom Robots Header Tags For Blogger
|Custom Robots txt File In Blogger|
Robots.txt is a text file containing some text lines. It is stored on the blogger / website server. Robots.txt determines which page of your blog indexes search engines.
Robots.txt forbids not indexing anything ..
Search engines index your blog through their web crawler. This crawler is also known as Spider / Spiderbot, but it is most often called a crawler
You can restrict any of your page by Robots.txt. You also can restrict Specific Spider By Robots.txt . You can add your Sitemap which will be Indexed.
I will discuss Robots.txt another day, let’s go back to todays topic “Custom Robots txt File In Blogger”
Look Up Default Robots.txt
Blogger has a default robots.txt file on every website .
You can lookup default robots.txt visiting https://your-url.blogspot.com/robots.txt , For custom domain https://www.example.com/robots.txt .
It will look like
Disallow: / search
The following code described below, See the description to get understand .
That means , the adsense spider/crawler can visit all of your pages . This User-Agent is a robots of Adsense.The robot help adsense to serve page related advertise on your site.
Disallow: / search
This command line for all robots (*) .And Disallow: /search means labels or search page will not index lebel page .
Here Allow: / means Your homepage or root page will be indexed , All robots can index homepage.
That’s added a xml sitemap for search bots . Crawler will index links from this sitemap.
How To Add Custom Robots.txt On Blogger
Login to Blogger Dashboard and choose your site .
Go to Settings》Search performance》Crawler and indexing》Custom robots.txt 》Edit 》Yes.
Now Copy the code below
Now a text box will appear , Here write/paste your custom robots.txt code .
Click Save button, After editing Click Save button.
All done , I think you already done this .
Some Important Robots.txt Rules
Here I added some more commands, if needed you can use them. If you do not understand a single line, then avoid it . Because ROBOTS.TXT is associated with your search .
Add this line Disallow specific post
Disallow : /post-link.html
Add this line to Disallow specific page
Disallow : /page-link/
Add this line to Disallow specific user-agent
If you want to prevent a user-agent from indexing , only then use this code.
Before use the code change User-agent name .
Disallow : /
Add multiple sitemap
You can add one more sitemap.xml to robots.txt . follow this code
Sitemap : www.example.com/sitemap-1.xml
sitemap : www.example.com/sitemap-2.xml
sitemap : www.example.com/sitemap-3.xml
How to check / test your live robots.txt File
You can check your live robots.txt by visiting https://www.example.com/robots.txt (change to your url). You also Google Robots.txt Tester Tool .
I tried to present the post beautifully, I do not know how much it has happened.
If the post seems beneficial,please make a comment . I will get inspired to write something better . Comment below if anything else to ask me.
Thanks for reading this article.
(Visited 10 times, 1 visits today)