Using a Free Robots.txt Generator on a Real Website: A Worked Example

A robots.txt generator worked example is the fastest way to learn how crawl rules really behave, because it turns abstract syntax into a file you can read line by line and test against a live site. This guide does exactly that. It starts with the real robots.txt file that serves on linktrixx.com today, then builds a second file for a realistic small business website with the free Robots.txt Generator, and explains every line of both files so you can judge what each rule allows, what each rule blocks, and what each rule leaves open. Nothing here is invented. The live file was fetched from the site on 9 November 2026, the tool page was read the same day, and the worked example follows the tool as it actually behaves, with a user agent field, an allow or disallow choice, a path, a sitemap field, a live preview, and copy and download buttons.
If you have ever typed a rule into a generator and wondered whether the output would help or harm your site, this robots.txt generator worked example answers that question in practical terms. You will see how a short file can protect an admin area, keep a search results page out of the crawl queue, point crawlers at a sitemap, and still leave every money page open. You will also see the mistakes that turn a helpful file into a harmful one, including the single line that can block a whole site, and the misunderstanding that makes people treat robots.txt as a privacy tool when it is nothing of the kind. By the end you will know how to write, test, and deploy a file with confidence, and you will know how to read the results after it goes live.
What robots.txt is and why it matters for SEO
A robots.txt file is a small plain text file that sits at the root of a website and tells compliant search engine crawlers which parts of the site they may fetch. Google describes it as a plain text file that follows the Robots Exclusion Standard, and notes that a file consists of one or more rules, where each rule blocks or allows access for all or a specific crawler to a specified file path on the domain where the file is hosted. The file lives at a fixed address, so for a site at example.com it answers at example.com/robots.txt, and crawlers ask for that address before they fetch a page. If no file exists, crawlers simply assume they may fetch everything, which is the default behaviour Google documents: unless you specify otherwise, all files are implicitly allowed for crawling.
The reason this matters for SEO is crawl budget. Search engines will fetch only a limited number of pages from a site in a given period, and every request spent on a login page, an internal search results page, a filter URL, or a duplicate copy of a page is a request not spent on a page that could earn traffic. A clean robots.txt file points crawlers at the content worth indexing and away from the rest, so the pages that matter get crawled more often and more promptly. That is the core promise of the Robots.txt Generator: it turns a list of decisions about which paths to open and which to close into a correctly formatted file in minutes, without asking you to memorise syntax or to type a rule from memory and hope the spelling is right.
Crawling is only half of ranking, and this is where many technical guides stop short. A page that crawlers can reach still needs authority to rank, and authority comes from other sites linking to it. That is why many sites pair technical fixes with link building services that earn editorial links from real publications, and why software companies use SaaS link building for product and comparison pages while agencies that deliver links for clients work through white label link building under their own brand. Robots.txt opens the door. Links give Google a reason to walk through it. The free Robots.txt Generator handles the first half, and the worked example below shows how the two halves fit together on a real site.
Search demand around this topic is practical rather than theoretical. People search for a robots.txt generator when they need a file now, and they search for worked examples when they want to see what a good file looks like before they trust a tool with their own site. The SERP for these queries is dominated by generator pages and by long guides from Ahrefs, SE Ranking, and Google Search Central, which tells you that readers want two things at once: a tool that builds the file, and a guide that explains the rules. This article gives you both, with the tool linked throughout and the explanation built around real files rather than invented ones.
How search engines read a robots.txt file
Before you write a single rule, it helps to know how a crawler reads the file, because several common mistakes come from assuming the file works like a firewall or a set of commands that every bot must obey. Google explains that a robots.txt file consists of one or more groups, and each group begins with a user agent line that names the crawler the group applies to, followed by the rules for that crawler. A crawler follows exactly one group: the group whose user agent matches it most specifically. A group that names Googlebot applies to Googlebot, while a group that names every crawler with an asterisk applies to all crawlers that have no more specific group. Specific groups and global groups are not combined, so a rule in the global group does not apply to a crawler that has its own named group. This is one of the most misunderstood points in technical SEO, and it is documented in Google interpretation of the robots.txt specification.
Inside a group, each line is one rule. The crawler compares the start of each URL path against the rules, and the longest matching rule wins. If an allow rule and a disallow rule match a URL with the same length, the allow rule wins. That longest match rule is why an exception such as Allow for one page inside a blocked folder works: the longer, more specific path beats the shorter, broader block. Google also notes that rules are case sensitive, so a disallow for a lowercase path does not block the same path written with capital letters, and that the asterisk can act as a wildcard in a path while the dollar sign can mark the end of a URL. These details are covered in the Google Search Central guide to creating and submitting a robots.txt file, which is the single most authoritative source on the subject and the first page to read before you change a live file.
Two limits deserve emphasis because they shape every decision in the worked example. First, robots.txt controls crawling, not indexing. Google states plainly that it cannot index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet, usually because other sites link to that URL. If you want a page out of search results, you need a noindex tag or an X Robots Tag header on a page that Google can crawl, or you need to remove the page, not a disallow line. Second, robots.txt is a request, not a lock. Reputable crawlers obey it, but nothing in the file forces a badly behaved bot to comply, and the file itself is public, so anyone can read it and see exactly which paths you consider private. Never put a genuinely sensitive path in robots.txt and assume it is hidden, because the file advertises it. The Ahrefs guide to robots.txt and SEO makes the same point with clear examples, and it is worth reading alongside the Google documentation.
The four fields Google supports, and the one it does not
Google supports exactly four fields in a robots.txt file: user agent, allow, disallow, and sitemap. Everything else is parsed and ignored. That short list is good news, because it means the whole format can be learned in a few minutes, and it is also a warning, because several fields that other tools offer are not honoured by Google at all. The most common is crawl delay, which some generators still include as an option. Google names crawl delay explicitly as an unsupported field, so adding it to a file aimed at Googlebot changes nothing. The Robots.txt Generator stays inside the supported set, which is one reason its output is safe to use: it builds groups from a user agent, allow and disallow rules, and a sitemap line, and nothing else.
The user agent field names the crawler a group applies to. An asterisk means all crawlers, and a name such as Googlebot targets just that crawler. The disallow field names a path the crawler should not fetch, relative to the root of the domain, starting with a slash, and ending with a slash when it names a folder. The allow field names a path the crawler may fetch, and its main job is to carve an exception out of a disallow, for example allowing one public page inside a folder that is otherwise blocked. The sitemap field is different from the other three: it is not tied to any specific user agent, it may appear anywhere in the file, and it gives the absolute URL of an XML sitemap so crawlers can find every page worth indexing. Google notes that the sitemap URL must be a fully qualified URL, and that sitemaps are a good way to indicate which content Google should crawl, as opposed to which content it can or cannot crawl.
A practical consequence follows from that list. If a generator offers you a field that Google does not support, the output may look impressive and still do nothing. If a generator limits itself to the four supported fields, the output may look plain and still do exactly what you asked. The LinkTrixx tool takes the second approach. You can read the same four field rule in the SE Ranking robots.txt guide, which walks through setup, syntax, and best practice with examples for each field, and in the SEO Sherpa robots.txt guide, which covers user agents, directives, and examples in the same terms.
What the free LinkTrixx Robots.txt Generator actually does
The tool page at linktrixx.com/tools/robots-txt-generator/ was read on 9 November 2026, and its behaviour is simple enough to describe in five steps, which is exactly how the page itself presents it. First, you add a rule by typing a user agent, using an asterisk for all crawlers, choosing Allow or Disallow, and entering the path to allow or block. Second, you add more rules if you need them by clicking the Add rule button for each extra user agent or path. Third, you add your sitemap by pasting the sitemap URL so crawlers can find every page worth indexing. Fourth, you copy or download the file, because the preview updates live as you type, and the Copy button copies the file while the Download button saves it as robots.txt. Fifth, you upload the file to the root directory of your site and check it in Google Search Console.
Three properties of the tool matter for the worked example. It is completely free, with no signup, no account, and no limits. It runs entirely in your browser, so your rules are turned into a robots.txt file locally and nothing is uploaded or stored. And it shows a live preview, so you can see the file take shape as you add each rule and catch a mistake before you download anything. Those three facts come from the tool page itself, including its frequently asked questions, which confirm that the asterisk means the rule applies to all crawlers, that disallow on its own does not stop a page from appearing in Google, and that the sitemap line tells crawlers where the sitemap is. If you want the conceptual background before you use it, the companion Robots.txt Generator Guide explains what robots.txt is and why it matters, and this article is the second blog in the set: the hands on worked example that shows the tool on a real site.
The tool sits in a wider family of free tools on the site, and the same worked example approach has been used for other tools in the set. The Keyword Density Checker Use Case runs the density checker on a real page and shows what the numbers mean, while the Keyword Density Checker business comparison weighs the free tool against paid alternatives. The Word Counter Worked Example does the same for counting words on a real page, and the Meta Tag Generator business comparison sets the free meta tag tool against paid SEO suites. You can browse the whole family from the free SEO tools hub, which lists every live tool in one place.
The real example: the live linktrixx.com robots.txt, line by line
The live robots.txt file at linktrixx.com/robots.txt was fetched on 9 November 2026 and returned a 200 response with this exact content, seven lines including blank lines:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://linktrixx.com/sitemap_index.xml
Four lines carry rules, and each one earns its place. Read them slowly, because this small file is a good model for most WordPress sites, and because the worked example later in this article builds on the same pattern with extra rules for a realistic shop.
Line one, User-agent with an asterisk, opens a single group that applies to all crawlers. There is no separate group for Googlebot, Bingbot, or any other named crawler, which means every compliant crawler reads the same rules. That is a deliberate choice. A site with one group has one behaviour to test and one behaviour to explain, and there is no risk that a named group accidentally overrides the global group and leaves a crawler with a different set of rules than you intended. Remember the group rule from the Google documentation: specific groups and global groups are not combined, so adding a Googlebot group later would mean Googlebot follows only that group and ignores the global one. If you ever add a named group, you must repeat every rule that should still apply to that crawler inside the named group, or it will stop applying to them.
Line two, Disallow for the path /wp-admin/, tells all crawlers not to fetch anything under the WordPress admin folder. This is the single most common disallow on a WordPress site, and it is there for two reasons. The admin area holds login pages, settings pages, and dashboard pages that no search visitor should ever land on, and crawling them wastes crawl budget on pages that can never rank or earn a click. It also keeps the admin login screen out of the crawl queue, which is good hygiene even though robots.txt is not a security tool. The trailing slash matters: /wp-admin/ names a folder, so it blocks every URL that starts with that path, while /wp-admin without the slash would also match a file of that name and behave slightly differently in edge cases. Google requires a directory rule to end with a slash, and the live file follows that rule exactly.
Line three, Allow for the path /wp-admin/admin-ajax.php, is the exception that proves the value of the longest match rule. WordPress uses admin ajax.php to serve dynamic content on the front end of a site, including some features that visitors and search engines need to see for a page to render correctly. If the whole admin folder is blocked and this one file is not allowed back in, crawlers may fetch pages that depend on it without the data it provides, and the rendered page can look broken or incomplete. The allow line is longer and more specific than the disallow line, so for this one URL the allow wins and crawlers may fetch it, while every other URL under /wp-admin/ stays blocked. This is the classic WordPress pattern, and it appears in the same form in the WsCube Tech robots.txt guide, which shows a WordPress sample with the same admin block and the same admin ajax exception, plus a disallow for internal search URLs and a sitemap line.
Line four, Sitemap with the absolute URL https://linktrixx.com/sitemap_index.xml, points crawlers at the XML sitemap index for the site. The sitemap index lists the sitemaps for posts, pages, and other content types, so a crawler that reads robots.txt can find every page worth indexing without guessing which URLs exist. Note the address: it is a sitemap index, not a single sitemap file, which is normal for a WordPress site that uses a sitemap plugin or a built in sitemap feature. The URL is absolute, with the protocol and the domain, exactly as Google requires, and it sits in the same file as the crawl rules so every crawler sees it on the same request. There is no trailing slash problem here, no relative path, and no guesswork about which host the sitemap belongs to.
What the file does not contain is just as instructive. It does not block the blog, the service pages, the tools hub, or any public content, because every one of those pages is meant to be crawled and indexed. It does not block CSS or JavaScript, which Google needs to render pages, and blocking those resources is one of the most damaging mistakes a site can make. It does not include a crawl delay line, because Google does not support one. It does not include a noindex field, because Google retired support for that field in robots.txt in 2019 and now ignores it. And it does not try to hide anything sensitive, because robots.txt is public and a disallow line would only advertise the path. The file is short, honest, and complete for the site it serves, and that is the standard the worked example below is judged against.
The worked example: building a robots.txt for a realistic small business site
Now the hands on part. The scenario is a realistic small business website, the kind of site that hires a link building agency or a freelancer and needs a clean crawl setup before it spends anything on links. Picture a small online shop that also publishes guides and runs a WordPress site, with a shop, a blog, a contact page, a checkout, a customer account area, an internal search page, and a folder of PDF invoices for staff use. The site has about forty public pages worth indexing, and a larger number of URLs that should never be crawled: admin pages, checkout pages, account pages, search results pages, filter URLs, and the staff PDF folder. The owner wants crawlers to find the shop, the blog, and the service pages quickly, to stay out of the private areas, and to read the sitemap on every visit. That is the whole brief, and it is the brief the Robots.txt Generator is built to answer.
Start by listing the paths before you open the tool, because a generator can only turn your decisions into syntax; it cannot decide for you which parts of your site should be public. Walk the site as a visitor would, then as a crawler would. The public paths are the home page, the shop category pages, the product pages, the blog and its articles, the about page, and the contact page. The private paths are the admin folder, the checkout, the cart, the customer account area, the internal search results, and the staff folder. The special case is the admin ajax file, which must stay open even though its folder is closed, exactly as in the live LinkTrixx file. The sitemap is the one the site already publishes, at https://example.com/sitemap_index.xml for this scenario, and in real use you would paste your own sitemap address instead. Writing this list first takes five minutes and prevents the most common generator mistake, which is adding rules for paths you have not checked and blocking something you meant to keep open.
Open the free Robots.txt Generator and enter the first rule. Type an asterisk as the user agent so the rule applies to all crawlers, choose Disallow, and enter /wp-admin/ as the path. The preview updates live and shows the first two lines of the file. Click Add rule and enter the exception: same user agent, choose Allow, and enter /wp-admin/admin-ajax.php. Add a second disallow for /checkout/ and a third for /cart/, because checkout and cart pages are unique to each visitor, carry no content worth indexing, and waste crawl budget when crawlers follow them from linked buttons. Add a fourth disallow for /my-account/, because account pages are personal to each customer and should never be crawled. Add a fifth disallow for /?s=, the WordPress internal search pattern, because search results pages duplicate content that already exists on the site and can multiply into thousands of thin URLs. Add a sixth disallow for /staff/, the folder that holds internal PDFs. Each rule takes seconds, and the preview grows line by line so you can read the file as it forms.
Next, paste the sitemap URL into the sitemap field. In this scenario the address is https://example.com/sitemap_index.xml, and in real use you would paste your own sitemap address, copied from your sitemap plugin or from your browser address bar after opening the sitemap. The generator adds the sitemap line to the preview, and the file is ready to review. Read it from top to bottom before you copy or download anything, and check each path against the list you wrote at the start. Is the admin folder blocked? Is the admin ajax file allowed? Are the checkout, cart, account, search, and staff paths all listed? Is the sitemap URL absolute, with the protocol and the domain, and does it open in a browser when you paste it in? If every answer is yes, click Copy to copy the file, or Download to save it as robots.txt. The tool does not upload the file for you, and that is correct: deployment is a separate step, and it belongs to you, not to a generator.
The generated file, explained line by line
Here is the complete file the tool produces for the scenario above, exactly as the live preview shows it when the rules are entered in the order described:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /checkout/ Disallow: /cart/ Disallow: /my-account/ Disallow: /?s= Disallow: /staff/ Sitemap: https://example.com/sitemap_index.xml
Every line follows the same logic as the live LinkTrixx file, with extra rules for the extra private areas a shop has. The user agent line opens a single group for all crawlers, so there is one behaviour to test and no risk that a named group overrides it. The admin disallow and the admin ajax allow are identical to the live file, for the same reasons: the admin area stays out of the crawl queue, and the one file the front end needs stays open. The checkout, cart, and account disallows keep crawlers out of pages that are unique to each visitor and carry no content worth indexing. The search disallow keeps crawlers out of internal search results, which are the classic source of crawl budget waste on WordPress sites, because every search a visitor runs creates a new URL that Google may try to crawl and index. The staff disallow keeps internal documents out of the crawl queue, with the reminder that this is a crawl instruction and not a security measure: if those PDFs must stay private, they need authentication, not a robots.txt line.
The sitemap line closes the file by pointing crawlers at the sitemap index, so a crawler that reads robots.txt can find every public page in one step. Compare the shape of this file with the live LinkTrixx file: same user agent line, same admin rules, same sitemap pattern, and extra disallows that match the extra private areas of a shop. That similarity is the point. A good robots.txt file for a small site is short, it blocks only what needs blocking, and it leaves everything else open. A file with thirty rules is usually a file whose owner has not decided what matters, and a file with three rules that blocks the wrong three paths is worse than no file at all. The generator makes it easy to add rules, so the discipline is in deciding which rules to add, and the list you wrote before opening the tool is what keeps the file short.
Comparing the generated file with the real live LinkTrixx file
Put the two files side by side and the comparison teaches more than either file alone. The live LinkTrixx file has one group, one disallow, one allow, and one sitemap line. The generated shop file has one group, six disallows, one allow, and one sitemap line. The overlap is the WordPress core: the admin block, the admin ajax exception, and the sitemap index. The difference is the shop layer: checkout, cart, account, search, and staff paths that LinkTrixx does not have, because LinkTrixx is a service site rather than a shop. Neither file blocks the blog, the service pages, the tools, or any public content, and neither file blocks CSS or JavaScript. Both files use absolute sitemap URLs on the same domain as the site, and both files keep the file short enough to read in seconds.
The comparison also shows what a generated file cannot do on its own. It cannot tell you that your sitemap index is the right address, because only you can check that it opens and lists your pages. It cannot tell you that a path you blocked is linked from your navigation, because only a crawl of your own site shows that. And it cannot tell you that a page you allowed is actually worth indexing, because that is a content decision, not a crawl decision. That is why the worked example pairs the generator with a real file: the generator handles syntax, and the comparison handles judgement. If you run the Keyword Density Checker on a page after you fix its crawl rules, you are doing the same kind of pairing, using one free tool to check the content and another to check the crawl path, and the Keyword Density Checker Use Case shows that pairing on a real page.
There is one more lesson in the comparison, and it concerns the sitemap addresses. The live LinkTrixx sitemap is https://linktrixx.com/sitemap_index.xml, and the scenario sitemap is https://example.com/sitemap_index.xml. Both are sitemap indexes, and both are absolute URLs. If your site uses a single sitemap file instead of an index, the line looks the same but the address ends in sitemap.xml rather than sitemap_index.xml, and if your site has more than one sitemap you can list more than one sitemap line. The generator accepts the address you paste, so the accuracy of the line depends entirely on the accuracy of the address you give it. Open the address in a browser before you paste it, and confirm that it returns a 200 response and lists your pages. A sitemap line that points at a missing file is worse than no sitemap line, because it tells crawlers to fetch a page that does not exist.
How to test the file before you deploy it
Testing is the step that separates a careful site owner from one who uploads a file and hopes. Test in two places: before deployment, against the text you generated, and after deployment, against the live file. Before deployment, read the file once more and check each rule against the list you wrote at the start. Then test the logic with a few URLs. Take a URL you meant to block, such as the admin login page, and confirm that it starts with a disallowed path. Take a URL you meant to allow, such as the admin ajax file, and confirm that its longer allow rule beats the shorter disallow. Take a public URL, such as a blog article, and confirm that no rule matches it, which means it is allowed by default. This manual check takes two minutes and catches the most common errors, including a missing trailing slash on a folder, a capital letter in a path that should be lowercase, and a rule that blocks a folder you meant to keep open.
Google Search Console gives you a stronger test after deployment, and it is free. Open the robots.txt report in Search Console and paste the address of your live file, or open the URL inspection tool and test a specific URL against the live file to see whether Googlebot can fetch it. The report shows the file Google fetched, the date it fetched it, and any syntax warnings, and the URL inspection tool tells you plainly whether a URL is allowed or blocked. If you change the file, ask Google to recrawl it, and check the report again the next day to confirm the new version is the one Google is using. The tool page itself tells you to check the file in Google Search Console after upload, and that advice is worth following every time, because a file that looks right in a preview can still behave differently on a live server if the server returns the wrong response code or serves a cached copy.
There are also free robots.txt tester pages from other tool makers, and the Google introduction to robots.txt links to the concepts behind them, but the strongest test is always the live file on your own domain. Open https://yourdomain.com/robots.txt in a browser and confirm that it returns a 200 response and shows the file you uploaded. Then open the sitemap address from the file and confirm that it also returns a 200 response. Then test one blocked URL and one allowed URL in Search Console. If all four checks pass, the file is live and working. If any check fails, fix the file, upload it again, and repeat the checks. Never assume a file is live because you uploaded it; confirm it by fetching it as a visitor would.
How to deploy the file on your site
Deployment is straightforward, and the tool page describes it in one sentence: upload the file to the root directory of your website, so it is available at yourdomain.com/robots.txt. In practice, that means saving the downloaded file with the exact name robots.txt, in lowercase, with no extra characters, and placing it in the folder that holds your home page. On a WordPress site, that is usually the same folder as the wp config file, and many hosts offer a file manager in the hosting control panel that lets you upload it without FTP software. If you use FTP, connect to the server, open the root folder, and upload the file there. If your site already has a robots.txt file, back up the old one first, then replace it, because the new file overwrites the old one completely and there is no merge step. Keep the backup for a week, so you can restore it if the new file causes a problem you did not expect.
After upload, check the live address in a browser and confirm the content matches the file you downloaded. Then open Google Search Console, open the robots.txt report, and confirm that Google fetched the new version. If your site uses a cache plugin or a content delivery network, clear the cache for the robots.txt address, because a cached copy of the old file can keep serving after you upload the new one. If your site is on WordPress and a plugin manages robots.txt, check whether the plugin generates the file dynamically, because an uploaded file and a plugin generated file can conflict, and the plugin version may win. In that case, paste the new rules into the plugin instead of uploading a file, or turn off the plugin feature and use the uploaded file. Either route works, but only one should be active at a time.
Deployment is also the moment to think about the pages the file opens up. A robots.txt file that allows crawlers into your best pages is only useful if those pages are worth crawling, and that is where content and links come back into the picture. If you run a service business, the pages that matter are your service pages and your guides, and the link building services page explains how editorial links raise the authority of exactly those pages. If you sell software, the SaaS link building page covers product and comparison pages, and guest posting is a practical way to earn the first links for a new page. If you deliver links for clients, white label link building keeps the work under your own brand. The file opens the door, and the links give crawlers a reason to walk through it, so deploy the file and then keep building the pages it protects.
Common mistakes that turn a helpful file into a harmful one
The first mistake is the most dangerous: a disallow for the whole site. The two lines User-agent with an asterisk and Disallow with a single slash tell every crawler to stay out of every page, and a site with that file will slowly disappear from search results as Google stops crawling it. This rule has a legitimate use, for a staging site or a site under construction, but it should never appear on a live site, and it should never be added by accident while editing a file. Always read the file after you generate it and confirm that no rule blocks the root path unless you meant it to. If you find that rule on a live site, remove it immediately and ask Google to recrawl the site in Search Console.
The second mistake is blocking CSS or JavaScript. Google needs those resources to render a page, and a file that blocks the folders where they live can make pages look broken to Google even though they look fine to a visitor. The fix is simple: never disallow the folders that hold your styles and scripts, and if you inherited a file that blocks them, remove those lines and test a page in Search Console to confirm it renders correctly. The third mistake is using robots.txt to hide a page from search results. As the tool page itself warns, disallow does not stop a page from appearing in Google, because Google may still list the URL if other sites link to it. To keep a page out of results, allow crawling and add a noindex tag to the page, or remove the page, and use robots.txt only to control crawling.
The fourth mistake is treating robots.txt as security. The file is public, so a disallow line advertises the path it blocks, and a badly behaved crawler can ignore the file entirely. Private pages need a login, not a robots.txt line. The fifth mistake is forgetting the sitemap, or pointing the sitemap line at an address that does not open. A missing sitemap line does not break a site, but a wrong one wastes a chance to point crawlers at every important page in one step. The sixth mistake is case sensitivity: a disallow for a lowercase path does not block the same path with capital letters, so check the exact spelling of every folder on your server. The seventh mistake is adding a named group for one crawler and forgetting that the named group replaces the global group for that crawler, so rules you meant to apply to everyone stop applying to the one crawler you named. The eighth mistake is copying a file from another site without reading it, because paths that make sense on one site can block the wrong pages on another. Every one of these mistakes is avoidable, and every one of them is caught by the testing routine in the previous section.
Robots.txt and link building: why crawl control pairs with authority
Technical SEO and link building are often treated as separate jobs, but on a real site they work together, and the worked example shows why. A robots.txt file decides which pages crawlers can reach. Links decide which of those pages Google trusts enough to rank. A site with clean crawl rules and no links will be crawled well and rank poorly. A site with strong links and a broken robots.txt file will have authority that Google cannot use, because the pages the links point to are blocked or buried under crawl waste. The sites that rank are the ones that do both, and the LinkTrixx service pages are built around that pairing: link building services for editorial links, SaaS link building for software sites, guest posting for relevant placements, and white label link building for agencies that deliver links under their own brand.
The pairing is visible in the articles on the site as well. The guide to link building for ecommerce stores covers shops like the one in this worked example, and the same crawl discipline applies to a shop, a SaaS site, or a local business. The best link building agencies for SaaS companies listicle shows how buyers compare providers, and the white label link building guide explains how agencies deliver that work for clients. Local businesses face the same pairing: the guides to link building for plumbers, link building for roofing companies, link building for landscaping companies, link building for coffee shops, link building for moving companies, and link building for solar companies all start from the same premise, that a site must be crawlable before links can help it, and that links are what turn a crawlable site into a visible one. Fix the crawl path first with the Robots.txt Generator, then earn the links that make the open pages rank.
What the results mean after the file goes live
A robots.txt file does not produce a ranking report, and that is worth saying plainly, because anyone who promises a ranking lift from a robots.txt change alone is overselling it. What the file produces is crawl behaviour, and crawl behaviour can be read in Search Console. After deployment, watch the crawl stats report for a change in the number of requests to the paths you blocked: requests to the admin folder, the checkout, the cart, the account pages, and the internal search pages should fall, while requests to the blog, the shop, and the service pages should hold steady or rise. Watch the pages report for the number of valid pages, and check that the pages you meant to keep open are still listed as valid. Watch the sitemap report for the number of discovered URLs, and confirm that the sitemap index is being read. These are the results the file is meant to produce, and they are the results you should judge it by.
Give the change time. Google does not recrawl a whole site in a day, and a file that is correct on Monday may not show its full effect in crawl stats for a week or more. Check the robots.txt report first to confirm Google fetched the new version, then check crawl stats weekly for a month, and compare the pattern before and after the change. If a blocked path is still being crawled, check the file for a spelling error, a missing slash, or a named group that overrides the global group. If an allowed page is not being crawled, check that no other rule blocks it, that the page is linked from somewhere crawlers can reach, and that the page returns a 200 response. If the sitemap shows errors, open the sitemap address and confirm that it lists the pages you expect. The generator gives you the file, the testing routine gives you the checks, and Search Console gives you the evidence, and the three together are what turn a robots.txt generator worked example into a routine you can repeat on every site you manage.
That routine is the real takeaway. List the paths, enter the rules in the free Robots.txt Generator, read the preview, download the file, test it, upload it to the root, confirm it in a browser and in Search Console, and watch the crawl stats. The same routine works for a shop, a blog, a SaaS site, or an agency site, and it works because the format is small and the rules are few. If you want to check the content on the pages you have just opened up, the Keyword Density Checker and the Meta Tag Generator are free tools in the same hub, and the Word Counter and the XML Sitemap Generator cover the two jobs that sit closest to robots.txt: checking page length and building the sitemap the file points to. The free SEO tools hub lists every live tool, so you can build the same worked example habit across the whole technical checklist, one free tool at a time.
Frequently asked questions
What does a robots.txt generator worked example show me that a blank generator does not?
A worked example shows the decisions behind the file, not just the syntax. This article starts with the real live linktrixx.com robots.txt, explains each of its four rule lines, then builds a second file for a realistic shop with the free Robots.txt Generator, and explains every line of that file too. You see which paths were blocked, which were allowed back in as an exception, and why the sitemap line points where it does, so you can judge the same decisions on your own site instead of copying a file you do not understand.
Can I use the free Robots.txt Generator for a WordPress site?
Yes. The generator works for any site, and WordPress is the most common case. Enter an asterisk as the user agent, disallow the path /wp-admin/, allow the path /wp-admin/admin-ajax.php so the front end still renders correctly, add disallows for checkout, cart, account, and internal search pages if your site has them, paste your sitemap address, and download the file. Upload it to the root directory of your site so it answers at yourdomain.com/robots.txt, then check it in Google Search Console.
Does a robots.txt file stop a page from appearing in Google?
No, not on its own. A disallow line tells crawlers not to fetch a page, but Google may still list the URL in search results without a snippet if other sites link to it. To keep a page out of search results, allow crawling and add a noindex tag to the page, or remove the page entirely. Use robots.txt to control crawling and crawl budget, and use noindex to control indexing.
What is the live linktrixx.com robots.txt file, and what does each line do?
The live file, fetched on 9 November 2026, has four rule lines. The user agent line with an asterisk applies the rules to all crawlers. The disallow line for /wp-admin/ blocks the WordPress admin folder. The allow line for /wp-admin/admin-ajax.php lets crawlers fetch the one admin file the front end needs. The sitemap line points crawlers at https://linktrixx.com/sitemap_index.xml. The file blocks nothing public, blocks no CSS or JavaScript, and uses only fields Google supports.
How do I test a robots.txt file before and after I deploy it?
Before deployment, read the generated file line by line, test a blocked URL and an allowed URL against the rules by hand, and confirm the sitemap address opens in a browser. After deployment, open yourdomain.com/robots.txt in a browser and confirm it returns a 200 response with the new content, then open Google Search Console, check the robots.txt report to confirm Google fetched the new version, and use URL inspection to test one blocked URL and one allowed URL. Clear any cache for the robots.txt address if your host or content delivery network serves an old copy.
Should I include a sitemap in my robots.txt file?
Yes, when you have one. The sitemap line gives crawlers the absolute URL of your XML sitemap, so they can find every page worth indexing without guessing which URLs exist. Paste the full sitemap address into the sitemap field of the free Robots.txt Generator and the tool adds the line for you. Open the address in a browser first and confirm it returns a 200 response and lists your pages, because a sitemap line that points at a missing file is worse than no sitemap line at all.