LinkTrixx
Home / Blog / How to Use a Free Robots.txt Generator to Control Search Crawling

How to Use a Free Robots.txt Generator to Control Search Crawling

How to Use a Free Robots.txt Generator to Control Search Crawling

A robots.txt file is one of the smallest files on a website, and one of the easiest to get wrong. It is a plain text file, usually only a few lines long, that tells search engine crawlers which parts of your site they may visit and which parts they should leave alone. Get it right and crawlers spend their time on the pages that earn you traffic. Get it wrong and a single line can hide your best pages from Google entirely. This guide explains how to use a free robots.txt generator to build that file correctly, what each rule means, and how to check the result after you upload it.

The tool at the centre of this guide is the LinkTrixx free robots.txt generator. It builds robots.txt rules from a simple form. You type a user agent, choose Allow or Disallow, enter a path, add your sitemap URL, and the preview updates live as you type. Then you copy the result or download it as robots.txt and upload it to the root of your site. It is free, with no signup and no limits, and it runs entirely in your browser, so nothing you type is uploaded or stored. You can find it with the other free SEO tools from LinkTrixx, including the Word Counter, the Meta Tag Generator and the Keyword Density Checker.

What people actually search for when they look for a free robots.txt generator

Search results for robots.txt generator, how to create robots.txt, and related queries show a clear pattern. People want a tool that works at once, with no account and no install. They want a live preview that updates as they type, not a file that appears only after they press a button. They want to copy the result or download it as a ready to upload file. The most useful pages in the results offer exactly that, a form for user agents and paths, a sitemap field, and a clean output they can take away in seconds.

The same search results show a second pattern. Many searchers are not looking for a tool alone. They are looking for understanding. Queries such as how to create a robots.txt file, robots.txt examples, and robots.txt syntax appear alongside the tool queries, and the guides that rank for them explain the same few ideas in plain language. What a user agent is, what Allow and Disallow do, where the file must live, and what happens when a rule is wrong. A generator that only produces text without explaining it leaves a beginner guessing, and guessing is how a site gets blocked by accident.

A third pattern appears in the newest results. Searchers now ask about AI crawlers as well as search crawlers, naming bots such as GPTBot and ClaudeBot and asking how to keep them out of a site or let them in. The Google Search Central guide to creating a robots.txt file remains the authority on the syntax itself, and it is the first source to check when two guides disagree. It sets out the four steps that every other good guide follows. Create a file named robots.txt, add rules to it, upload it to the root of your site, and test it. Everything else in this article builds on those four steps.

That pattern shapes how this guide is written. It starts with what the file is and why it matters, then walks through the LinkTrixx generator step by step, then explains each directive, gives copy ready examples for common platforms, lists the mistakes that cause the most damage, and finishes with how to test the file once it is live. If you only need the file, the generator gives it to you in a minute. If you want to understand it, the rest of the guide is here for you.

What a robots.txt file is

A robots.txt file is a plain text file that lives at the root of your website, so it is available at yourdomain.com/robots.txt. Search engine crawlers read it before they fetch your pages, and they use it to decide which URLs they may request and which they should skip. It follows the Robots Exclusion Protocol, a standard that has been in use since 1994 and was later standardised as RFC 9309. The file is public, anyone can open it in a browser, and it works by request rather than by force. Compliant crawlers follow it. Crawlers that ignore it are not stopped by it.

The file is built from groups of rules. Each group starts with a User agent line that names the crawler the group applies to, followed by one or more Allow or Disallow lines that name the paths that crawler may or may not fetch. A Sitemap line can sit at the end of the file and point crawlers at your sitemap. Here is the shape of a simple file, as shown in the Google Search Central guide.

User-agent: Googlebot
Disallow: /nogooglebot/
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml

In that example, the crawler named Googlebot may not fetch anything under the nogooglebot folder. Every other crawler may fetch the whole site. The sitemap line tells crawlers where the sitemap lives. Unless you specify otherwise, all files are implicitly allowed for crawling, so a robots.txt file only needs to list the exceptions, not every page on the site.

Three facts about the file catch beginners out, so it helps to learn them before you write your first rule. First, a robots.txt file applies only to the exact protocol, host and port where it is posted. Rules in the file at https://example.com/robots.txt apply only to https://example.com, not to a subdomain such as https://blog.example.com and not to the http version of the same domain. Each host needs its own file. Second, paths in robots.txt are case sensitive. A rule for /Private/ does not block /private/, and a rule for /Blog/ does not block /blog/. Third, the file must be a plain text file with standard UTF encoding, saved with a plain text editor. Word processors can add unexpected characters that make rules invalid, which is one more reason to build the file with a generator instead of typing it by hand.

The Ahrefs guide to robots.txt and SEO makes the same point in a different way. It calls robots.txt one of the simplest files on a website and one of the easiest to mess up, and notes that misconfigurations are extremely common even among experienced SEO professionals. That is not a reason to avoid the file. It is a reason to build it carefully, preview it before you upload it, and test it after. The generator on this page exists to make the careful version the easy version.

Why robots.txt matters for SEO

Robots.txt matters for SEO because crawling is a limited resource. Search engines do not fetch every page on the internet with equal attention. Each website gets a crawl budget, the number of pages a search engine will fetch from that site in a given period. If crawlers spend that budget on login pages, internal search results, shopping carts and duplicate parameter URLs, the pages that earn you traffic may be crawled less often, and new or updated pages may wait longer to be found. A clean robots.txt file points crawlers at the content worth indexing and away from the rest.

The Similarweb guide to robots.txt for SEO puts the consequence plainly. If your site has more pages than the crawl budget covers, some content will never make it into the index, and pages that are not indexed cannot rank. Blocking low value sections is a practical fix. Stop crawlers from fetching duplicate pages, archives, staging areas and dynamically generated URLs, and the budget that is left goes to the pages that matter. For a small site with a few dozen pages, crawl budget is rarely the limiting factor. For a large site, an ecommerce store with thousands of filtered URLs, or a site that publishes daily, it can be the difference between new pages appearing in search in days or in weeks.

Duplicate content is the second reason the file matters. Many sites generate the same content under many URLs without meaning to. A product page with sorting and filtering options can appear under dozens of addresses that differ only by a question mark and a parameter. An internal search box can create a fresh URL for every query a visitor types. A print friendly version of a page can duplicate the original. Crawlers that fetch all of those versions waste budget and can split ranking signals between copies of the same content. A Disallow rule for the parameter pattern or the search path keeps crawlers on the canonical versions.

The third reason is privacy and focus. Most sites have sections that should never appear in search results. Admin panels, account pages, checkout flows, thank you pages, staging environments and test areas all belong behind a Disallow rule. They are not useful to a searcher, and they can leak information or create confusing results if they are crawled. Blocking them does not make them secure, and the next section explains why, but it does keep them out of the crawl queue and out of the way of your real content.

The fourth reason is discovery. The Sitemap line in robots.txt tells every crawler, not just Google, where your sitemap lives. A sitemap lists the pages you want crawled and indexed, so it works as the positive half of the file. Robots.txt says what to skip, the sitemap says what to find. Used together, they give crawlers a clear map of the site instead of leaving them to guess.

None of this makes a page rank on its own. Crawling is only half the job. A page also needs authority to rank, and that comes from other sites linking to it. That is why many sites pair technical fixes like robots.txt with link building services that earn editorial links from real publications. If you sell software, our SaaS link building service does the same for product and comparison pages, and guest posting on relevant sites is a practical way to earn those first links. For agencies that deliver links for clients, white label link building keeps the work under your own brand. Robots.txt makes sure crawlers can reach your best pages. Links help those pages rank once they are found.

What the LinkTrixx free robots.txt generator does

The LinkTrixx free robots.txt generator turns the syntax above into a form you can fill in without memorising it. You do not need to remember whether Allow or Disallow comes first, how to spell a user agent, or where the sitemap line goes. You fill in the fields, the preview builds the file for you, and you copy or download the result.

When you first open the page, the preview already shows a sensible starting point for a WordPress site.

User-agent: *
Disallow: /wp-admin/

That default says every crawler should stay out of the WordPress admin area. It is a safe starting point, and you can keep it, change it, or build on it. The page also offers a sample that fills in three rules plus a sitemap line, so you can see at once how a finished file looks before you type anything of your own.

The generator has four parts. The first is the rule builder. You type a user agent, use an asterisk for all crawlers or name a specific crawler such as Googlebot, choose Allow or Disallow, and enter the path to allow or block. The second is the Add rule button, which adds another rule row each time you click it, so you can build a file with as many groups and paths as your site needs. The third is the sitemap field, where you paste your sitemap URL so the generator can add the Sitemap line for you. The fourth is the live preview, which updates as you type, so you always see the exact file you are about to download.

When the file looks right, two buttons finish the job. Copy copies the file to your clipboard, so you can paste it into your hosting file manager or a text editor. Download saves it as robots.txt on your computer, ready to upload. Both are free, with no signup and no limits, and the whole tool runs in your browser, so your rules are never uploaded or stored.

It is worth knowing what the tool does not do, so you do not mistake a finished file for a finished job. It does not upload the file to your site for you, because uploading depends on your host and your access. It does not test the file against Google, because testing needs the file to be live first. And it does not edit an existing robots.txt file on your server. It builds a new file from your rules, and the steps later in this guide cover uploading and testing it. That split is deliberate. Building the file is the part where syntax errors happen, and that is the part the generator takes care of.

How to use the free robots.txt generator step by step

Open the free robots.txt generator in your browser. There is nothing to install and no account to create. The page loads with the default preview already filled in, and the form is ready to use. Work through the steps below in order. The whole process takes a few minutes once you know what you want to block.

Step 1. Decide what crawlers should skip

Before you type a single rule, list the parts of your site that crawlers do not need. Admin areas, login pages, account pages, shopping carts, checkout flows, internal search results, thank you pages, staging areas and duplicate parameter URLs are the usual candidates. Write the list on paper or in a note first, because a rule typed in a hurry is the most common source of a blocking mistake. If you are unsure whether a section should be blocked, leave it open. Blocking too little is easy to fix later. Blocking too much can hide pages that earn you traffic.

Check your existing file first if you have one. Open yourdomain.com/robots.txt in your browser and read what is already there. If your site runs on WordPress, Shopify or another platform, the platform may already manage a default file for you. Note the rules that are already working before you replace them, so you do not lose a rule you needed. The generator builds a new file from scratch, so anything in the old file that you want to keep must be typed into the new one.

Step 2. Set the user agent for your first rule

Type an asterisk in the user agent field to apply the rule to all crawlers. That is the right choice for most rules, because a rule that keeps crawlers out of your admin area should apply to every crawler, not just to Google. Use a named crawler only when a rule should apply to that crawler alone. Type Googlebot when a rule is only for Google, Bingbot when it is only for Bing, or GPTBot when it is only for the AI training crawler from OpenAI. Each named user agent starts a new group, and the rules in one group do not apply to another.

Remember that the asterisk matches all crawlers except the various AdsBot crawlers, which must be named explicitly if you want to target them. For most sites that detail does not matter, because AdsBot rules are only needed when you want to control how Google Ads crawlers fetch your pages. If you do not run ads, you can ignore it and use the asterisk for your general rules.

Step 3. Choose Allow or Disallow and enter the path

Choose Disallow for a path crawlers should not fetch, and Allow for a path they may fetch. Enter the path starting with a forward slash, as it appears after your domain name. To block the WordPress admin area, enter /wp-admin/ with a trailing slash, because it is a directory. To block a single page, enter the full page name, such as /thanks.html. To allow everything, use Allow with a single forward slash, which means the whole site.

Paths are case sensitive, so type them exactly as they appear on your site. A rule for /Admin/ will not block /admin/, and a rule for /Blog/ will not block /blog/. If your site mixes cases in its URLs, check the real address in your browser before you type the rule. A single wrong capital letter can leave a whole section open that you meant to close, or close a section you meant to leave open.

Step 4. Add more rules with the Add rule button

Click Add rule for each extra user agent or path. A typical small site needs only a handful of rules. A WordPress blog might block /wp-admin/, /wp-login.php and the internal search pattern /?s=, then allow /wp-admin/admin-ajax.php so the admin ajax endpoint that themes and plugins need stays reachable. An online store might block /cart/, /checkout/ and /account/ as well. A larger site might add a separate group for one named crawler with its own rules.

Keep the file short. Google reads only the first 500 KB of a robots.txt file, and a file that long would be unmanageable in any case. Most sites need fewer than twenty rules. If your file is growing past that, you are probably blocking individual pages that would be better handled with a single pattern or a noindex tag. Review the list and look for a pattern you can replace several rules with, such as a folder rule that covers a whole section at once.

Step 5. Add your sitemap URL

Paste your sitemap URL into the sitemap field so the generator adds the Sitemap line for you. Use the full absolute URL, including https, such as https://example.com/sitemap.xml. The sitemap line tells crawlers where your sitemap lives, so they can find every page worth indexing without waiting for a Search Console submission. It is especially useful for crawlers other than Google, which may not have your sitemap submitted anywhere else.

If you have more than one sitemap, you can list more than one Sitemap line, and the generator can hold the first while you add the rest by hand after download. If you do not have a sitemap yet, leave the field empty and create one before you publish the file. A sitemap is not required for robots.txt to work, but a file without one gives crawlers no positive guidance about what to find, only negative guidance about what to skip.

Step 6. Read the live preview line by line

Read the preview before you copy or download anything. Check each group in turn. Does the user agent line name the crawler you intended. Does each Disallow line name the exact path you meant to block, with the right case and the trailing slash where it belongs. Is there a Disallow line for the root, Disallow: /, anywhere in the file. That single line blocks your entire site from every crawler in its group, and it is the most damaging line in robots.txt. It belongs in a staging file and nowhere else. If you see it in your preview and you did not mean to block the whole site, remove that rule before you go any further.

Check the Allow lines too. An Allow line only matters when it carves out an exception inside a blocked folder. If you blocked /internal/ but want one page inside it to stay crawlable, the Allow line for that page must name the full path, such as /internal/presskit.html. If no Allow line is needed, the file is simpler without it. Simplicity is a feature in robots.txt. Every line you add is a line a crawler must interpret, and a short file that you understand beats a long file that you do not.

Step 7. Copy or download, then upload to the root of your site

Click Copy to copy the file, or Download to save it as robots.txt on your computer. Then upload it to the root directory of your site, the same folder that holds your homepage, so it is available at yourdomain.com/robots.txt. How you upload it depends on your host. Most hosts offer a file manager in the hosting control panel, or you can use FTP. WordPress users can also use an SEO plugin that manages the file, but uploading the plain file to the root works on every platform and leaves no doubt about where it lives.

Do not place the file in a subdirectory. A file at yourdomain.com/pages/robots.txt will not be found, because crawlers only look at the root. Do not save it with a different name or a different extension. It must be named robots.txt, all lower case, and saved as a plain text file. Once it is uploaded, open yourdomain.com/robots.txt in a private browser window. If you see the contents of your file, it is live. The section near the end of this guide covers how to test it properly in Google Search Console.

What each part of your robots.txt file means

A robots.txt file has only four parts, and learning what each one does is enough to read any file you will ever meet. The sections below explain each directive in plain language, with the detail that decides whether a rule works as intended.

User agent

The user agent names the crawler a group of rules applies to. It is always the first line of a group. An asterisk means the group applies to all crawlers. A name such as Googlebot, Bingbot, GPTBot or ClaudeBot means the group applies only to that crawler. Crawlers follow the rules in the group that most accurately applies to them, so a group named for Googlebot takes priority over the asterisk group for Googlebot, while every other crawler follows the asterisk group.

That priority rule is the reason a file can say two different things to two different crawlers without contradiction. You can block all crawlers from a folder in the asterisk group, then allow Googlebot into one page inside it with an Allow line in a Googlebot group. You can also block one named crawler from the whole site while letting everyone else in. The Ahrefs guide shows this pattern clearly. A group that disallows everything for the asterisk, followed by a group that allows everything for Googlebot, leaves Google as the only crawler that can fetch the site. Use that pattern with care, because it is easy to write the groups in the wrong order and block the crawler you meant to welcome.

Disallow

Disallow names a path a crawler should not fetch. The path is relative to the root of the domain and must start with a forward slash. A Disallow line for a directory must end with a forward slash, such as Disallow: /admin/. A Disallow line for a single page must name the full page, such as Disallow: /offers.html. A Disallow line with nothing after it, Disallow: on its own, means nothing is disallowed, so it allows everything. A Disallow line for a single forward slash, Disallow: /, blocks the entire site for that group.

Disallow supports two wildcards. An asterisk matches any sequence of characters, so Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /products/*? blocks every product URL that contains a question mark. A dollar sign marks the end of a URL, so the rule only matches URLs that end exactly there. Used well, wildcards replace long lists of individual rules with one clear pattern. Used carelessly, they can block far more than you intended, so test any wildcard rule against a few real URLs before you publish it.

Allow

Allow names a path a crawler may fetch, and it exists to carve out exceptions inside a blocked area. If you block a whole folder with Disallow: /internal/ but want one page inside it to stay crawlable, an Allow line for that page does the job. If you block /wp-admin/ but want the admin ajax endpoint to stay reachable, Allow: /wp-admin/admin-ajax.php does the job. Google and Bing both support Allow, and it is the standard way to keep a useful page open inside a closed folder.

When Allow and Disallow both match the same URL, Google and Bing apply the rule with the most characters, which is usually the more specific rule. If the two rules are equal in length, the least restrictive rule wins, so Allow wins the tie. Other search engines may instead follow the first matching rule, so the order of your lines can matter for them. The safest habit is to make your rules unambiguous in the first place. Block the folder, allow the single exception, and do not write two rules that could both match the same URL with different outcomes.

Sitemap

Sitemap names the full absolute URL of a sitemap, such as Sitemap: https://www.example.com/sitemap.xml. It is optional, it can appear more than once if you have more than one sitemap, and unlike the other directives it is not tied to a user agent group, so it is best placed at the start or the end of the file. Google supports it, as do Bing, Yahoo and Ask, and the Google Search Central guide recommends it as a good way to indicate which content Google should crawl, as opposed to which content it can or cannot crawl.

Including your sitemap in robots.txt is somewhat redundant for Google if you have already submitted it through Search Console, but it is still good practice, because it tells every other search engine where to find it without a separate submission. It also helps after a migration or a sitemap move, when a crawler that has not seen your Search Console submission can still discover the new location from the file itself. Use the full URL with the correct protocol and subdomain, because Google does not assume or check http, https, www and non www alternates for you.

Common robots.txt rules with examples

The examples below cover the situations most sites meet. Each one is a complete file you can adapt, and each one follows the syntax in the Google Search Central guide. Replace example.com with your own domain, and replace the paths with the ones on your site.

Allow everything

A site with nothing to hide needs only a short file, or no file at all. If you want an explicit allow, this is the whole file.

User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml

Because everything is allowed by default, the Allow line here is optional. Its value is clarity. Anyone who opens the file can see at a glance that the whole site is open to crawlers, and the sitemap line still gives crawlers a place to start.

Block everything, for staging only

A staging site, a development copy, or a site under construction should be closed to all crawlers. This is the file for that job, and it must never reach a live site.

User-agent: *
Disallow: /

That single Disallow line is the most dangerous line in robots.txt. It is also the most common cause of a sudden traffic loss, because it often survives a migration from staging to production when nobody remembers to change it. If your live site ever disappears from search results without explanation, this line is the first thing to check.

Block admin and login, allow everything else

This is the most common file on a small business or blog site. It keeps crawlers out of the admin area and the login page, and leaves the rest of the site open.

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap.xml

The Allow line matters here. The admin ajax endpoint is used by themes and plugins to load content, and blocking it can break how pages render for crawlers. Allowing that one path while blocking the rest of the admin folder is the standard WordPress pattern, and it is the pattern the LinkTrixx generator default is built around.

Block a single file

To block one page and nothing else, name the full page in a Disallow line.

User-agent: *
Disallow: /thanks.html
Sitemap: https://www.example.com/sitemap.xml

Use this for pages that have no search value, such as a thank you page after a form submission, a confirmation page, or a single private document. Do not use it for a page you want removed from search results. Blocking a page from crawling does not remove it from the index, which is covered in the section on noindex later in this guide.

Block a file type

To block every PDF on the site, use a wildcard and a dollar sign to match URLs that end in .pdf.

User-agent: *
Disallow: /*.pdf$
Sitemap: https://www.example.com/sitemap.xml

This pattern blocks /file.pdf but allows /file.pdf?id=68937586, because that address does not end in .pdf. The same pattern works for any file type. Replace pdf with the extension you want to block, and keep the dollar sign so the rule only matches URLs that end exactly there.

Allow one page inside a blocked folder

To close a folder but keep one page inside it open, pair a Disallow line for the folder with an Allow line for the page.

User-agent: *
Disallow: /internal/
Allow: /internal/presskit.html
Sitemap: https://www.example.com/sitemap.xml

The longer, more specific rule wins for Google and Bing, so the press kit page stays crawlable while the rest of the folder stays closed. This is the same technique used for the admin ajax endpoint above, and it is the main reason the Allow directive exists at all.

Robots.txt examples for popular platforms

Different platforms create different folders, so a good starting file depends on what your site runs on. The examples below are starting points, not finished files. Check your platform defaults before you replace an existing file, and check the real paths on your site before you block them.

WordPress

WordPress creates an admin area, a login page, an XML RPC endpoint, and an internal search that generates a new URL for every query. A common starting file blocks those and keeps the ajax endpoint open.

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /?s=
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap.xml

Do not block the whole wp-content folder. It holds your images, themes and scripts, and Google needs to fetch those to render your pages correctly. Blocking it is one of the classic WordPress mistakes, and it is covered in the mistakes section below.

Online stores

Stores create the same low value pages on every platform. Carts, checkout flows, account pages and filtered category URLs rarely need to be crawled, and they can multiply into thousands of addresses on a large catalogue.

User-agent: *
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?sort=
Sitemap: https://www.example.com/sitemap.xml

Adjust the folder names to match your platform. Shopify, WooCommerce and Magento use different paths for the same ideas, so open your own store, visit the cart and the account page, and copy the real paths from the address bar before you type the rules.

Blogs and small business sites

A blog or a small business site usually needs the shortest file of all. Block the admin area and the internal search, allow the ajax endpoint if the platform uses one, and add the sitemap. Anything more is probably overkill, and a short file is easier to review, easier to test, and easier to hand to someone else without a long explanation.

User-agent: *
Disallow: /wp-admin/
Disallow: /?s=
Sitemap: https://www.example.com/sitemap.xml

If your site has no admin area, no search box, and no private sections, the allow everything file from the previous section is enough. A robots.txt file is not a ranking factor, and a site does not need a long one to rank well. It needs a correct one, and correct usually means short.

Blocking an AI crawler

Search results for robots.txt now include many questions about AI crawlers, and the answer is a named user agent group. To keep the OpenAI crawler out of your site while letting search crawlers in, add a group for GPTBot.

User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml

Replace GPTBot with ClaudeBot, PerplexityBot, CCBot or any other crawler name to target a different bot. The LinkTrixx generator builds named groups the same way it builds the asterisk group. Type the crawler name in the user agent field, choose Disallow, enter a single forward slash, and the preview shows the finished group. Keep in mind that a crawler which ignores robots.txt will ignore this rule too, so it is a request to compliant bots, not a lock against the rest.

Common mistakes to avoid

Most robots.txt damage comes from a small set of mistakes, and all of them are easy to avoid once you know them. The Similarweb guide lists the same core errors, and the Search Engine Journal guide to robots.txt covers them from the same angle. Work through the list below before you upload any file, and check it again after every change.

Robots.txt vs noindex: what robots.txt cannot do

The most persistent misunderstanding in technical SEO is the idea that robots.txt removes pages from search results. It does not, and the distinction matters every time you decide how to handle a page you do not want in the index. Robots.txt controls crawling, which is whether a crawler may fetch a page. Indexing, which is whether a page appears in search results, is a separate decision, and it is controlled by a different tool.

When you disallow a page, you tell crawlers not to fetch it. That means Google cannot read the content of the page, and it cannot see any noindex tag on the page either, because the tag lives in the page itself. If other sites link to that blocked URL, Google may still list it in search results, showing the address with a note that it is blocked by robots.txt, and without a title or description it could not read. The page is not properly indexed, but it is not gone either, and it can still appear for a search that matches its address.

To keep a page out of search results properly, allow crawling and add a noindex tag to the page, using a robots meta tag in the head of the page or the equivalent HTTP header. That way Google can fetch the page, read the noindex instruction, and remove the page from the index. The Google Search Central guide is explicit on this point. If you want to exclude a page from search engines, use a meta robots tag instead of a robots.txt rule. The Ahrefs guide reaches the same conclusion. Google made clear in September 2019 that a noindex directive inside robots.txt is not supported, so the only reliable route is the meta tag or the header.

That gives you a simple decision rule. Use robots.txt when you want to save crawl budget, keep a section out of the crawl queue, or stop crawlers from fetching low value URLs. Use noindex when you want a page to stay out of search results while remaining fetchable. Use a password or proper authentication when you want a page to stay private. Each tool does one job, and using the wrong one for the job is how pages end up visible when they should be hidden, or hidden when they should be visible.

There is one more case worth knowing. If a page is already indexed and you want it gone fast, blocking it in robots.txt is the slowest way to get there, because it stops Google from reading the very tag that would remove it. Remove the Disallow rule, add the noindex tag, let Google fetch the page once, and then, if you still want it out of the crawl queue, you can add the Disallow rule back after the page has left the index. That sequence, allow, tag, wait, then block, is the correct order for a clean removal.

How to check your robots.txt file after uploading it

Building the file is half the job. Checking it after upload is the other half, and it is the half that catches the mistakes that cost traffic. The Google Search Central guide sets out the checks, and they take only a few minutes.

Open the live file in a private browser window

Open a private browsing window and go to yourdomain.com/robots.txt. If you see the contents of your file, it is publicly accessible and in the right place. If you see a 404 error, the file is not where crawlers expect it, and you need to move it to the root directory. If you see an old version of the file, your host may be caching it, and you should clear the cache or wait and check again. This simple check catches the most common upload error, a file in the wrong folder, before it costs you anything.

Use the robots.txt report in Search Console

Google Search Console has a robots.txt report that shows the file Google has fetched for your site, when it was last fetched, and whether it has any errors. Open it after every upload and read what it says. It shows exactly how Google interprets your rules, which is more reliable than any third party checker, because it uses the same parser Google Search uses. If the report shows an error, fix the line it names and upload the file again. If the report shows a warning about a blocked resource, check whether the blocked path holds CSS, JavaScript or images your pages need.

Test individual URLs with URL Inspection

Use the URL Inspection tool in Search Console to test whether Googlebot can fetch a specific page. Enter the address of a page that should be crawlable, such as a service page or a recent blog post, and check that Google reports it as allowed. Then enter the address of a page that should be blocked, such as an admin page or a thank you page, and check that Google reports it as blocked. Testing one allowed URL and one blocked URL after every change is a fast way to confirm the file does what you intended.

You can also test the file itself against the same check. If a page you expected to be open shows as blocked, read the file again and look for a wildcard or a folder rule that matches it by accident. If a page you expected to be blocked shows as open, check the case of the path, check that the rule sits in the right user agent group, and check that the file you are reading is the file that is actually live, not an older cached copy.

Recheck after migrations, theme updates and platform changes

A robots.txt file is easy to lose. A site migration can copy the staging file to production. A theme update can overwrite a file that was edited by hand. A platform change can replace a custom file with a platform default. That is why the file should be rechecked after every migration, every theme update, and every change to the content management system. Add it to the same checklist as checking internal links, checking the featured image, and checking the title and description. A file that was correct last year is not proof that it is correct today.

If Google has cached an older copy of your file, you can ask it to refresh the cached copy through Search Console. Google will find and start using a new file on its own in time, but a refresh request speeds the process when you have fixed a blocking mistake and need crawlers back on your pages quickly. That request is the right response to a fixed error, not a substitute for fixing the error first.

Pair the generator with meta tags, content length and links

Controlling crawling is one part of technical SEO. A page also needs clear meta tags so search engines and readers understand it, content of the right length so it covers the topic in full, and links from real publications so search engines trust it. The robots.txt generator works best as part of that small workflow, not as a standalone check.

Start with the crawl controls. Build the file with the free robots.txt generator, upload it, and test it as described above. Then check the pages themselves. Use the Meta Tag Generator to build clean title, description, keywords, author and robots tags, with live length counters that warn you when a title goes over 60 characters or a description goes over 160 characters, and read our guide on how to use a free meta tag generator for the full workflow. Then check length with the Word Counter, and read our guide on how to use a free word counter to learn what each count means for your draft. Then check keyword use with the Keyword Density Checker, so a page uses its target phrase often enough to be clear and not so often that it reads as stuffed. All four tools are listed on the free SEO tools hub, with more tools to follow.

Then check where the page stands against the sites ranking above it. A competitor backlink gap analysis shows which links your rivals have that you do not, so you can close the gap page by page. Our guide to link building strategies for 2026 explains 13 tactics that still work, from guest posting to digital PR to resource page outreach, and our guide on how to measure link building ROI shows how to tell whether that work is paying. If you want help with that work, our link building services start every engagement with a gap analysis, then match sites to your budget and get your approval before anything goes live.

Agencies can use the same workflow for clients. Our white label link building service handles prospecting, outreach, content and placement under your brand, with unbranded reports you can forward as your own work. SaaS companies can use our SaaS link building service for product and comparison pages, where buyers research deeply and links need to live where that research happens. Guest posting remains a practical way to earn first links on relevant sites, as explained on our guest posting service page. For a closer look at how long form content earns its place, read our guide to how to write meta titles and descriptions, which covers the snippet that decides whether a searcher clicks.

That sequence, control crawling, check meta tags, check length, then build links, is simple enough to repeat for every post. It keeps robots.txt in its proper place, as a tool that directs crawlers to the right pages, while clear meta tags, full content and real links do the work of earning rankings. A page that crawlers can reach, readers can understand, and other sites link to has the best chance to rank and to keep ranking.

Frequently Asked Questions

Is the free robots.txt generator really free?

Yes. The free robots.txt generator is free to use with no signup, no account and no limits. It runs in your browser, so your rules are never uploaded or stored. You can build as many files as you like, as often as you like.

Where do I put my robots.txt file?

Upload the file to the root directory of your website, so it is available at yourdomain.com/robots.txt. Search engines look for it there before they crawl your site. A file in a subdirectory will not be found.

What does User-agent: * mean?

The asterisk means the rule applies to all crawlers. Use it when a rule should apply to every search engine, or name a specific crawler such as Googlebot to target just that one.

Does robots.txt stop a page from appearing in Google?

Not on its own. Disallow tells crawlers not to fetch a page, but Google may still list its URL if other sites link to it. To keep a page out of search results, use a noindex tag or remove the page.

Should I include a sitemap in robots.txt?

Yes, when you have one. The Sitemap line tells crawlers where your sitemap is, so they can find every page worth indexing. Paste your sitemap URL in the Sitemap URL field and the generator adds the line for you.

How do I check my robots.txt file after uploading it?

Open yourdomain.com/robots.txt in a private browser window to confirm the file is live, then use the robots.txt report in Google Search Console to check for errors and confirm how Google reads your rules. Test one allowed URL and one blocked URL with URL Inspection to confirm the file does what you intended.

We scale your organic traffic with editorial content & natural link building.

Tell us your site and your competitors — we come back with a free link plan, a site list and a custom quote.