Custom Robots txt File In Blogger

ROBOT.TXT file is a very important part of the website because the SEO Performance of your site is very much dependent on ROBOTS.TXT file. I think you are reading this post to increase your website's search performance, it's good.
Also read - Custom Robots Header Tags For Blogger

    Custom Robots txt File In Blogger
    Custom Robots txt File In Blogger

    About ROBOTS.TXT


    Robots.txt is a text file containing some text lines. It is stored on the blogger / website server. Robots.txt determines which page of your blog indexes search engines.

    Robots.txt forbids not indexing anything ..
    Search engines index your blog through their web crawler. This crawler is also known as Spider / Spiderbot, but it is most often called a crawler

    You can restrict any of your page by Robots.txt. You also can restrict Specific Spider By Robots.txt . You can add  your Sitemap which will be Indexed.

    I will discuss Robots.txt another day, let's go back to todays topic "Custom Robots txt File In Blogger"

    Look Up Default Robots.txt


    Blogger has a default robots.txt file on every website .
    You can lookup default robots.txt visiting https://your-url.blogspot.com/robots.txt  , For custom domain https://www.example.com/robots.txt .
    It will look like
    User-agent: Mediapartners-Google
    Disallow:

    User-agent: *
    Disallow: / search

    Allow: /

    Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED

    Sitemap: http://example.blogspot.com/sitemap.xml

    Example Explanation



    The following code described below, See the description to get understand .

    User-agent: Mediapartners-Google
    Disallow:
    That means , the adsense spider/crawler can visit all of your pages . This User-Agent is a robots of Adsense.The robot help adsense to serve page related advertise on your site.
    User-agent: *
    Disallow: / search
    This command line for all robots (*) .And Disallow: /search means labels or search page will not index lebel page .

    Allow: / 
    Here Allow: / means Your homepage or root page will be indexed , All robots can index homepage.

    Sitemap: http://example.blogspot.com/sitemap.xml
    That's added a xml sitemap for search bots . Crawler will index links from this sitemap.

    How To Add Custom Robots.txt On Blogger


    Step 1:

    Login to Blogger Dashboard and choose your site .

    Step 2:

    Go to Settings》Search performance》Crawler and indexing》Custom robots.txt 》Edit 》Yes.
    Now Copy the code below

    User-agent: Mediapartners-Google
    Disallow: 
    
    User-agent: *
    Disallow: /search
    Allow: /
    
    Sitemap: https://www.example.com/sitemap.xml
    

    Step 3:

    Now a text box will appear , Here  write/paste your custom robots.txt  code .

    Step 4:

    Click Save button, After editing Click Save button.
    All done , I think  you already done this .

    Some Important Robots.txt Rules


    Here I added some more commands, if needed you can use them. If you do not understand a single line, then avoid it . Because ROBOTS.TXT is associated with your search .
    Add this line Disallow specific post
    Disallow : /post-link.html

    Add this line to Disallow specific page
    Disallow : /page-link/

    Add this line to Disallow specific user-agent
    If you want to prevent a user-agent from indexing , only then use this code.
    Before use the code change User-agent name .
    User-agent: Name
    Disallow : /

    Add multiple sitemap

    You can add one more sitemap.xml to robots.txt . follow this code
    Sitemap : www.example.com/sitemap-1.xml
    sitemap : www.example.com/sitemap-2.xml
    sitemap : www.example.com/sitemap-3.xml

    How to check / test your live robots.txt File


    You can check your live robots.txt by visiting https://www.example.com/robots.txt (change to your url). You also Google Robots.txt Tester Tool .

    Final words

    I tried to present the post beautifully, I do not know how much it has happened.
    If the post seems beneficial,please make a comment . I will get inspired to write something better . Comment below if anything else to ask me.
    Thanks for reading this article.