Webmasters and developers often encounter questions about how to correctly create a robots.txt file, what to write in it, and how to block a page from indexing using robots.txt. Let's go through it step by step.
What is a robots.txt file
Sometimes you need to hide certain pages of a site from search engine bots, thus preventing them from being indexed. The robots.txt file must be placed in the root directory of the site on the server. If it is missing or placed elsewhere, the bots will not find it, or will ignore it.
How to correctly create a robots.txt file and what to put in it
Creating robots.txt is very easy — open Notepad or any other text editor (Word, Notepad++, Sublime Text, VS Code, etc.) and create a new text file named robots with the extension .txt. The file content should consist of a list of directives aimed at a specific search engine bot or universal directives for all bots. The first line is the user-agent, the second line lists pages, sections, or subdirectories you want to block from visibility. If you want the bots to see the whole site, a single universal directive is enough, and the robots.txt file will look like:
User-agent: *
Disallow:
Host: www.site.com
In this example User-agent followed by an asterisk means the rule applies to all search engine bots. The Disallow line with nothing after the colon means the entire site is fully open for indexing. The Host line indicates which version of the site the bot should consider the primary one – with www or without it.
How to block a page or site from indexing with robots.txt
To block the entire site from indexing so that search engines do not see it, add the following commands to robots.txt:
User-agent: *
Disallow: /
Host: www.site.com
To block a specific page, add the path after Disallow: without the site domain – for example, to block http://www.site.com/admin.php, specify /admin.php. You can also block a directory by adding a trailing slash, e.g., /api/. All pages within that directory will also be blocked.
User-agent: *
Disallow: /admin.php
Disallow: /api/
Host: www.site.com
Also you can use regular expressions in robots.txt. To check the current configuration and which pages are blocked, you can use Google Search Console -> Crawl -> Blocked URLs.