Dalelorenzo's GDI Blog

Robots.txt: The Deceptively Important File All Websites Need

The robots.txt file facilitates major search engines understand where they're allowed to go on your website.

But, while the major search engines do support the robots.txt file, they may not all adhere to the rules the same way.

Below, let's break down what a robots.txt record is, and how you can use it.

→ Download Now: SEO Starter Pack [Free Kit]

What is a robots.txt document?

Every day, there are visits to your website from bots -- also known as robots or spiders. Search engines like Google, Yahoo, and Bing send these bots to your place so your material can be slithered and indexed and appear in search results.

Bots are a good thing, but there are some cases where you don't want the bot to run your website crawling and indexing everything. That's where the robots.txt record comes in.

By adding certain directives to a robots.txt folder, you're directing the bots to slither only the pages you crave crawled.

However, it's important to understand that not every bot will adhere to the rules you write in your robots.txt file. Google, for instance, won't listen to any directives that you arrange in the datum about slithering frequency.

Time you need a robots.txt enter?

No, a robots.txt document is not required for a website.

If a bot comes to your website and it doesn't have one, it will exactly crawl your website and index pages as it normally would.

A robot.txt file is only needed if you wishes to more restrict over what is being crawled.

Some benefits to having one include 😛 TAGEND

Help manage server overloads

Prevent crawl waste by bots that are visiting sheets you is not are to be able to

Keep specific folders or subdomains private

Can a robots.txt document prevent indexing of the information contained?

No, you cannot stop content from being indexed and shown in search results with a robots.txt file.

Not all robots will follow the instructions the same way, so some may index the contents you set to not be slithered or indexed.

In addition, If the content you are trying to prevent from showing in the search results has external links to it, that they are able to also cause the search engines to index it.

The only way to ensure your content is not indexed is to add a noindex meta tag to the page. This text of system looks like this and will go in the html of your page.

It's important to note that if you crave the search engines to not index a page, you will need to allow the sheet to be slithered in robots.txt.

Where is the robots.txt register located?

The robots.txt file will ever sit at the root arena of an internet site. As an example, our own datum can be found at https :// www.hubspot.com/ robots.txt.

In most websites you should be able to access the actual file so you can edit it in an FTP or by accessing the File Manager in your legions CPanel.

In some CMS scaffolds you can find the folder right in your administrative expanse. HubSpot, for instance, starts it easy to customize your robots.txt enter from your account.

If you are on WordPress, the robots.txt record can be accessed in the public_html folder of your website.

the robots.txt file in the public_html folder on your WordPress website

WordPress does include a robots.txt record by default with a brand-new installation that will include the following 😛 TAGEND


Disallow:/ wp-admin/

Disallow:/ wp-includes/

The above is telling all bots to slither all parts of the website except anything for the purposes of the/ wp-admin/ or/ wp-includes/ directories.

But you may want to create a more robust file. Let's show you how, below.

Employ for a Robots.txt File

There could be countless intellects you just wanted to customize your robots.txt folder -- from ensure creeping budget, to obstruct slice of an internet site from being slithered and indexed. Let's explore a few reasons for using a robots.txt datum now.

1. Block All Crawlers

Blocking all crawlers from retrieving your website is not something you would want to do on an active website, but is a great option for a development website. When you block the crawlers it will help prevent your pages from being shown on search engines, which is good if your sheets aren't ready for contemplating yet.

2. Disallow Certain Pages From Being Crawled

One of the most common and useful ways to use your robots.txt folder is to limit search engine bot access to parts of your website. This can help maximize your creep budget and prevent unwanted sheets from gale up in the search results.

It is important to note that only because you have told a bot to not slither a page, that doesn't mean it will not get indexed. If you don't crave a sheet to show up in the search results, you need to add a noindex meta label to the page.

Sample Robots.txt File Directives

The robots.txt file is made up of blocks of wires of mandates. Each guiding will begin with a user-agent, and then the rules for that user-agent will be placed below it.

When a specific search engine grounds on your website, it will look for the user-agent that applies to them and read the block that refers to them.

There are several ordinances you can use in your datum. Let's undermine those down , now.

1. User-Agent

The user-agent command allows you to target specific bots or spiders to aim. For speciman, if you simply want to target Bing or Google, this is the directive you'd use.

While there are hundreds of user-agents, below are examples of some of the most common user-agent options.

User-agent: Googlebot

User-agent: Googlebot-Image

User-agent: Googlebot-Mobile

User-agent: Googlebot-News

User-agent: Bingbot

User-agent: Baiduspider

User-agent: msnbot

User-agent: slurp( Yahoo)

User-agent: yandex

It's important to note -- user-agents are case-sensitive, so be sure to enter them properly.

Wildcard User-agent

The wildcard user-agent is noted with an (*) asterisk and causes you readily apply a directive to all user-agents that exist. So if you crave a specific rule to apply to every bot, you can use this user-agent.


User-agents will exclusively follow the rules that most closely apply to them.

2. Disallow

The disallow directive tells search engines to not slither or access particular pages or directories on a website.

Below are several examples of how you are able to use the outlaw directive.

Block Access to a Specific Folder

In this instance we are telling all bots to not crawl anything in the/ portfolio directory on our website.


Disallow:/ portfolio

If we only want Bing to not slither that directory, we are to be able included it like this, instead 😛 TAGEND

User-agent: Bingbot

Disallow:/ portfolio

Block PDF or Other File Characters

If you don't demand your PDF or other document natures slithered, then the below directive should be used. We are telling all bots that we do not crave any PDF enters crawled. The$ at the end is telling the search engine that it is the end of the URL.

So if I have a pdf file at mywebsite.com/ place/ myimportantinfo.pdf, the search engines won't access it.


Disallow: *. pdf$

For PowerPoint documents, you could use 😛 TAGEND


Disallow: *. ppt$

A better alternative might be to create a folder for your PDF or other documents and then disallow the crawlers to crawl it and noindex the whole directory with a meta tag.

Block Access to the Whole Website

Particularly beneficial if you have a development website or experiment folders, this directive is telling all bots to not slither your area at all. It's important to remember to remove this when you established your locate live, or you will have indexation issues.


The*( asterisk) you see above is what we call a "wildcard" expression. When we use an asterisk, we are implying that the rules below should apply to all user-agents.

3. Allow

The allow guiding can help you specify certain sheets or indices that you do want bots to access and creeping. This can be an override rule to the disallow option, pictured above.

In the illustration below we are telling Googlebot that we do not want the portfolio index crawled, but we do want one specific portfolio item to be accessed and slithered 😛 TAGEND

User-agent: Googlebot

Disallow:/ portfolio

Allow:/ portfolio/ crawlableportfolio

4. Sitemap

Including the location of your sitemap in your folder can make it easier for search engine crawlers to crawl your sitemap.

If you submitted your sitemaps directly to each search engine's webmaster implements, then it is not necessary to add it to your robots.txt file.

sitemap: https :// yourwebsite.com/ sitemap.xml

5. Crawl Delay

Crawl delay can tell a bot to slow down when slithering your website so your server does not become overwhelmed. The directive pattern below is asking Yandex to wait 10 seconds after each creeping action it takes on the website.

User-agent: yandex

Crawl-delay: 10

This is a directive you should be careful with. On a very large website it can greatly minimize the number of URLs crawled every day, which would be counterproductive. This can be useful on smaller websites, nonetheless, where the bots are inspecting a bit too much.

Note: Crawl-delay is not supported by Google or Baidu. If you want to ask their crawlers to slow their crawling of your website, you will need to do it through their tools.

What are regular formulations and wildcards?

Pattern matching is a more advanced way of controlling the way a bot crawls your website with the use of characters.

There are two express that are common and are used by both Bing and Google. These mandates can be especially beneficial on ecommerce websites.

Asterisk:* is treated as a wildcard and are presenting any string of characters

Dollar sign:$ is used to designate the end of a URL

A good example of using the* wildcard is in the situation whatever it is you want to prevent the search engines from crawling pages that might have a question mark in them. The below system is divulge all bots to disregard crawling any URLs that have a question mark in them.


Disallow: /*?

How to Create or Edit a Robots.txt File

If you do not have an existing robots.txt file on your server, you can easily add one with the steps below.

Open your wished text editor to start a new record. Common editors that may exist on your computer are Notepad, TextEdit or Microsoft Word.

Add the directives you would like to include to the document.

Save the record with the epithet of “robots.txt”

Test your enter as indicated in the next region

Upload your. txt file to your server with a FTP or in your CPanel. How you upload it will depend on the type of website you have.

In WordPress you can use plugins like Yoast, All In One SEO, Rank Math to generate and edit your file.

You can also use a robots.txt generator implement to help you prepare one which might help minimise errors.

How to Test a Robots.txt File

Before you go live with the robots.txt file system you created, you will want to run it through a tester to ensure it's valid. This will help prevent issues with incorrect directives that may have been added.

The robots.txt testing tool is only available on the age-old copy of Google Search Console. If your website is not been incorporated into Google Search Console, you will need to do that first.

Visit the Google Support page then click the "open robots.txt tester" button. Select the property you would like to test for and then you will be taken to a screen, like the one below.

To test your new robots.txt code, simply delete what is currently in the box and supersede with your brand-new code and sound "Test". If the replies to your exam is "allowed", then your code is valid and you can revise your actual register with your brand-new code.

the robots.txt tester on Google Support

Hopefully this berth has moved "youre feeling" less scared of digging into your robots.txt register -- because doing so is one way to improve your ranks and boost your SEO efforts.

SEO Starter Pack

Read more: blog.hubspot.com

Comments (0) Trackbacks (0)

No comments yet.

Leave a comment

No trackbacks yet.