How to Create a Robots.txt File for Search Engines
A properly configured robots.txt file helps search engines understand which parts of your website they should crawl and which areas they should ignore. While it’s a small file, it plays an important role in technical SEO by guiding search engine bots, improving crawl efficiency, and preventing unnecessary pages from being indexed.
As search evolves with AI-powered search experiences and traditional search engines alike, having the right robots.txt configuration is more important than ever. In this guide, you’ll learn how to create, optimize, and test a robots.txt file to improve your website’s visibility and support better search performance.
What Is a Robots.txt File?
A robots.txt file, also known as the Robots Exclusion Protocol (REP), is a plain text file that tells web crawlers, such as search engine bots and AI crawlers which parts of your website they can and cannot access.
In simple terms, it provides instructions about which pages should be crawled and which should be ignored.
When a search engine visits your website, one of the first things it does is check your robots.txt file for crawling instructions before accessing individual pages.
There are many ways to configure a robots.txt file. Here’s a simple example:
User-agent: *
Disallow: /
This is the basic structure of a robots.txt file.
The asterisk (*) after User-agent means the rule applies to all web crawlers. The forward slash (/) after Disallow tells those crawlers not to access any pages on the website.
However, it’s important to understand that blocking a page from being crawled does not necessarily prevent it from being indexed. If other websites link to that page, search engines may still include it in search results.
Why Robots.txt Matters for SEO
At first glance, it might seem counterintuitive to block search engines from crawling parts of your website. After all, one of the main goals of both traditional SEO and AI search optimization is to make your content easy for crawlers to discover.
The key is understanding that not every page on your website deserves equal attention.
Most websites contain pages that provide little or no SEO value, such as admin areas, filtered URLs, duplicate pages, search results, or staging content. If search engines spend time crawling those pages, they have less time available to crawl the pages that actually matter.
This is where crawl budget becomes important.
Googlebot doesn’t have unlimited resources for every website. Instead, Google assigns each site a crawl budget, which consists of two components:
- Crawl capacity limit – the maximum number of requests Googlebot can make without overwhelming your server.
- Crawl demand – Google’s interest in crawling your pages based on factors like popularity, freshness, and how often your content changes.
In other words, crawl budget is simply the number of URLs Googlebot can and wants to crawl.
Your goal is to help Google spend that budget on your most valuable content instead of wasting it on low-value pages.
Google recommends minimizing common factors that waste crawling resources, including:
- Faceted navigation: URL parameters for filtering and sorting can generate thousands of nearly identical pages.
- Duplicate content: Multiple URLs containing the same information force crawlers to spend time indexing redundant pages.
- Redirect chains and soft 404 errors: These create unnecessary work for crawlers without adding value.
- Slow server performance: If your website responds slowly, Google may crawl fewer pages during each visit.
This is where a well-configured robots.txt file becomes valuable.
By preventing crawlers from accessing unnecessary pages, you guide them toward the content that matters most. Instead of wandering through technical clutter, search engines can focus on your most important pages, making every crawl more efficient.
Think of your robots.txt file as a traffic controller for search engines. It directs bots away from distractions and toward the content that helps your website rank.
Now that you understand why it matters, let’s look at how to create a robots.txt file.
How to Create a Robots.txt File
Creating a robots.txt file is straightforward. Follow these steps to build one correctly.
Step 1: Open a Plain Text Editor
Start by opening a plain text editor, such as Notepad on Windows or TextEdit on macOS.
Be sure you’re working in plain text mode rather than a rich text editor like Microsoft Word.
If your website already has a robots.txt file, clear its contents (without deleting the file itself) before adding your new rules.
Step 2: Name and Place the File Correctly
Your file must be named:
robots.txt
The filename is case-sensitive and must be spelled exactly as shown.
Every website can have only one robots.txt file, and it must be placed in the site’s root directory.
For example:
https://www.example.com/robots.txt
Keep these important guidelines in mind:
- Location matters: Crawlers only look for robots.txt in the root directory. If you place it inside a subfolder, it will be ignored.
- Scope is limited: A robots.txt file only controls the protocol, subdomain, and port where it’s hosted. If you have separate subdomains, such as blog.example.com or m.example.com, each requires its own robots.txt file.
- Use UTF-8 encoding: Save the file as plain text using UTF-8 encoding. Invalid characters can cause search engines to ignore your directives.
Step 3: Add Your Robots.txt Rules
Every robots.txt file begins with a User-agent directive, which specifies which crawler the following rules apply to.
Here’s a simple example:
User-agent: Googlebot
Allow: /
Disallow: /nogooglebot/
This configuration allows Googlebot to crawl the entire website except pages inside the /nogooglebot/ directory.
You can also include your XML sitemap:
Sitemap: https://www.example.com/sitemap.xml
Although adding a sitemap isn’t required, it’s highly recommended especially for larger websites. It provides search engines and AI crawlers with a direct path to your most valuable pages.
Once you’re comfortable with the basics, you can create more advanced rules for specific crawlers and directories.
Step 4: Upload the File to Your Root Directory
Once you’ve finished editing your robots.txt file, upload it to your website’s root directory.
The upload process varies depending on your hosting provider, CMS, or server configuration.
For WordPress websites, WordPress may generate a virtual robots.txt file automatically. If you need custom rules, you may have to upload your own file manually or use an SEO plugin that lets you edit it.
If you’re unsure how to upload files to your website, consult your hosting provider’s documentation or support resources.
How to Block AI Crawlers with Robots.txt
Many website owners now use robots.txt to control how AI companies access their content.
Some block AI crawlers to prevent their content from being used for model training. Others want to reduce unnecessary crawler activity, protect proprietary content, or prevent competitors from repurposing their work through AI systems.
The trade-off is visibility.
Blocking AI crawlers may protect your content, but it can also reduce your chances of appearing in AI-generated search results and chatbot responses.
Some of the most important AI crawlers include:
- GPTBot (OpenAI)
- OAI-SearchBot (OpenAI Search)
- ClaudeBot (Anthropic)
- Claude-SearchBot (Anthropic Search)
- Google-Extended (Google AI training)
- CCBot (Common Crawl)
Each crawler supports robots.txt directives and publishes its own user-agent string.
To block them individually, use rules like these:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Notice that training crawlers and search crawlers are separate.
For example, blocking GPTBot does not block OAI-SearchBot, and blocking ClaudeBot does not block Claude-SearchBot. If you want to stop both AI training and AI search crawling, you must create rules for each crawler individually.
Another option is to block all crawlers by default and explicitly allow only the search engines you want.
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
It’s also worth noting that blocking Google-Extended only prevents your content from being used for Google’s AI training. It does not affect your rankings in Google Search.
Before blocking AI crawlers, consider your long-term strategy carefully. Preventing access may protect your content, but it can also reduce opportunities for AI systems to cite or recommend your website.
If AI visibility is part of your marketing strategy, consider implementing an llms.txt file alongside robots.txt to help AI systems discover your highest-value content instead of blocking them entirely.
How to Test Your Robots.txt File
After publishing your robots.txt file, verify that Google can read it correctly.
Google retired its standalone Robots.txt Tester in 2023. Today, you can review your robots.txt file through the robots.txt report in Google Search Console.
The report shows:
- the robots.txt files Google has retrieved,
- when each file was last crawled,
- any syntax errors,
- and warnings that may prevent Google from interpreting your rules correctly.
If you’ve recently updated your file, you can request Google to recrawl it.
To test individual pages, use the URL Inspection tool in Google Search Console. It lets you confirm whether Googlebot can access a specific URL or whether it’s blocked by your robots.txt rules.
This is an excellent way to catch accidental blocking before it affects your search visibility.
Another helpful tip is to view competitors’ robots.txt files directly in your browser.
Simply enter:
https://competitorwebsite.com/robots.txt
Studying competitors’ configurations can reveal which directories they exclude, how they structure their rules, and whether they’re blocking AI crawlers.
Combined with a comprehensive technical SEO audit, this can uncover opportunities to improve your own website.
Common Robots.txt Mistakes to Avoid
Robots.txt errors can be surprisingly easy to make, and some can have serious SEO consequences.
Here are the most common mistakes to avoid.
Blocking Your Entire Website
The following rule prevents every crawler from accessing every page on your website:
User-agent: *
Disallow: /
This often happens when a staging site’s robots.txt file is accidentally deployed to production.
Always review your robots.txt file after launching a new website or completing a migration.
Blocking CSS, JavaScript, or Images
Google renders pages much like a modern browser.
If you block CSS, JavaScript, or important image files, Google may not fully understand how your pages work, potentially hurting rankings and indexing.
Confusing Disallow with Noindex
A Disallow directive only prevents crawling.
It does not guarantee that a page won’t appear in search results. If other websites link to the page, Google may still index it.
If your goal is to keep a page out of search results entirely, use a noindex directive or protect the page with authentication.
Leaving Robots.txt Empty or Missing
A missing robots.txt file won’t prevent search engines from crawling your website. Google simply assumes that everything is allowed.
However, you’ll lose the ability to manage crawl budget, reference your XML sitemap, or control AI crawlers.
For that reason, creating and maintaining a robots.txt file should be part of every technical SEO checklist.
Frequently Asked Questions
Below are some of the most frequently asked questions about robots.txt files, search engine crawling, website indexing, and technical SEO, answered using industry best practices and real-world experience.
A robots.txt file is a simple text file placed in the root directory of your website that provides instructions to search engine crawlers and other web bots. It tells them which pages or directories they are allowed to crawl and which ones they should avoid. When a crawler visits your website, it checks the robots.txt file before crawling other pages. While reputable search engines such as Google and Bing generally follow these directives, robots.txt is not a security feature and cannot prevent users from accessing blocked pages directly if they know the URL.
Most websites benefit from having a robots.txt file, although it is not mandatory. It helps search engines crawl your site more efficiently by preventing them from wasting crawl budget on pages that don’t need to appear in search results, such as admin areas, login pages, staging environments, or duplicate content. For small websites, the impact may be minimal, but for larger websites with thousands of pages, a properly configured robots.txt file can improve crawl efficiency and support better SEO.
A basic robots.txt file is short and easy to understand. It typically specifies the user agent (the crawler the rule applies to) followed by crawl directives such as Disallow or Allow. Many websites also include the location of their XML sitemap to help search engines discover important pages more efficiently.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xmlThe exact rules depend on your website’s structure and SEO goals, so avoid copying another site’s robots.txt file without understanding what each directive does.
There are several ways to edit your robots.txt file in WordPress. The easiest method is to use an SEO plugin such as Yoast SEO, Rank Math, or All in One SEO, all of which include a built-in robots.txt editor. Alternatively, you can manually upload or modify the file using your hosting control panel or an FTP client. Before making changes, always create a backup and verify that important pages are not accidentally blocked from search engine crawlers.
This warning in Google Search Console means Google has indexed a page even though your robots.txt file prevents it from crawling the page. This can happen if other websites link to the URL or if Google discovered it before the block was added. To fix the issue, first determine whether the page should remain indexed. If it should appear in search results, remove the robots.txt block and allow Google to crawl it. If it should not be indexed, remove the robots.txt restriction and instead use a noindex meta tag or HTTP header so Google can crawl the page and process the noindex directive. Blocking pages in robots.txt alone does not guarantee they will be removed from search results.
No. A robots.txt file only controls crawling, not indexing. Search engines may still index a blocked URL if they discover it through backlinks, XML sitemaps, or other sources. If you want a page removed from search results, use a noindex directive instead of relying solely on robots.txt.
Yes. Many AI companies publish the names of their crawlers, allowing website owners to block them using robots.txt directives. However, compliance depends on whether the crawler respects the robots exclusion standard. Most reputable AI providers follow these directives, but robots.txt cannot enforce access restrictions on bots that choose to ignore them.
The robots.txt file must be placed in the root directory of your website so it is accessible at https://yourdomain.com/robots.txt. Search engines automatically look for the file in this location when they begin crawling your website.
Yes. Adding the location of your XML sitemap is considered a best practice because it helps search engines discover your important pages more efficiently. While search engines can often find your sitemap through other methods, including it in robots.txt provides an additional signal and improves crawl efficiency.
Conclusion
Your robots.txt file may be small, but it plays a significant role in how search engines and AI systems interact with your website. When configured correctly, it helps search engines focus on your most valuable content, prevents unnecessary pages from being crawled, and gives you greater control over how your content is accessed by AI platforms.
If your website already has a robots.txt file, take the time to audit it for the common mistakes covered in this guide. Even a minor error can affect how your site is crawled and indexed.
If you’re creating one for the first time, follow the best practices outlined above and test your file in Google Search Console before publishing it. Verifying your configuration helps ensure search engines can crawl the pages that matter while respecting the rules you’ve defined.
As search continues to evolve, so does the role of robots.txt. What was once primarily a tool for managing search engine crawlers has become an important part of controlling how AI systems interact with your content, alongside emerging standards such as llms.txt.
No matter how search technology changes, a properly configured robots.txt file remains a fundamental part of technical SEO and an essential tool for maintaining control over your website’s visibility and content.
Jun Homecillo
Website Developer
Related Articles
No posts found.
Latest Articles
Explore everything you need to know in our latest articles.
How To Make AEO and GEO Profitable
You might be appearing in ChatGPT responses, earning citations in Google’s AI Overviews, and seeing your brand mentioned more frequently across AI-powered platforms. Yet revenue...
How to Create a Robots.txt File for Search Engines
A properly configured robots.txt file helps search engines understand which parts of your website they should crawl and which areas they should ignore. While it’s...
Are Backlinks Still Important for SEO and AI Search?
AI is transforming how people search for information, but one core SEO principle has remained remarkably resilient: backlinks. A backlink is a hyperlink from one...
What is an AI Citation Audit? Why Your Content Needs One in 2026
If you’ve been monitoring your brand’s visibility across AI search platforms but aren’t getting meaningful insights, the problem often begins long before you analyze the...
