// guides / robots and sitemaps
WordPress robots.txt the default file, safe edits and the rules that hurt
The WordPress robots.txt is a plain text file at your site's root that tells crawlers which URLs they may request. WordPress generates one on the fly, plugins and files can replace it, and a single copied rule can block far more than you meant. Here's what the file does, what it should say, and how to change it without hurting your rankings.
What is the WordPress robots.txt file?
A robots.txt is a plain text file that sits at the root of a domain, always at the same address: yourdomain.com/robots.txt. Crawlers request it before anything else, and its rules tell them which paths they're allowed to fetch. Google's documentation is blunt about its purpose: it manages crawler traffic to your site, and it isn't a mechanism for keeping a page out of Google.
On WordPress the file has a twist that catches almost everyone. If no physical robots.txt exists in the root folder, WordPress generates one on request. There's no file to find over FTP, which is why so many owners search for a robots.txt they can't see. That generated version is called the virtual robots.txt, and it answers with sensible defaults until something replaces it.
The rules are voluntary. Reputable crawlers like Googlebot and Bingbot follow them; scrapers and badly built bots don't have to. So robots.txt is a signpost for well-behaved crawlers, not a lock. If you need pages actually kept out of search results, that's a different tool, covered below.
What does the default WordPress robots.txt say?
With no physical file and no SEO plugin rewriting it, WordPress answers /robots.txt with four lines:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlWhat does each line of the WordPress robots.txt do?
Each group of rules opens with a User-agent line naming a crawler, and a crawler obeys only its most specific group. That last part matters more than it sounds: add a User-agent: Googlebot group and Google ignores every rule under User-agent: *, which is how a small edit ends up blocking far more than intended.
| Line | What it does |
|---|---|
| User-agent: * | Opens the group every crawler follows unless it has a group of its own |
| Disallow: /wp-admin/ | Keeps crawlers out of the dashboard, which has nothing worth indexing |
| Allow: /wp-admin/admin-ajax.php | Leaves open the endpoint that front-end features call, so blocking the dashboard doesn't break them |
| Sitemap: https://example.com/wp-sitemap.xml | Points crawlers at core's sitemap; Yoast and Rank Math serve /sitemap_index.xml instead, and this line should name whichever is live |
Where does your robots.txt actually come from?
Several layers can each write or rewrite the file, and they don't warn each other. The only robots.txt that counts is the one fetched from your domain's root, because that's the one Google reads.
A physical file
Upload a text file named robots.txt to the root folder and it replaces the virtual one at once. WordPress stops answering; the server sends your file.
Yoast SEO
Its file editor creates and edits a physical robots.txt, so after using it once, the virtual file is gone for good.
Rank Math
Its editor changes what the virtual file prints, so there's still no file on disk, and a physical file would override its settings.
Cloudflare or the host
Rules added at the edge or by a security layer never show on any WordPress screen, which is why the live URL, not the plugin screen, is the source of truth.
How do you edit robots.txt in WordPress?
Whichever route you take, fetch yourdomain.com/robots.txt in a private browser window afterwards and read what's actually served. Caches and edge layers mean the saved version and the served version aren't always the same thing.
- i.
With Yoast SEO
Yoast SEO, Tools, File editor. If no robots.txt exists yet, Yoast offers to create one, then lets you edit the physical file it wrote.
- ii.
With Rank Math
Rank Math, General Settings, Edit robots.txt. It edits the virtual output, and the box is locked if a physical file already exists, because that file would win anyway.
- iii.
With All in One SEO
All in One SEO, Tools, then switch on the Enable Custom Robots.txt toggle. The rules you add go in the box below it, and the same live-file check applies afterwards.
- iv.
Without a plugin
Write a plain text file named robots.txt and upload it to the site root over SFTP or the host's file manager. Include the admin-ajax.php allow line and your Sitemap line, since your file replaces the default completely.
- v.
Check what's live
Load /robots.txt in the browser, and confirm in Search Console's robots.txt report which version Google last fetched and whether it parsed.
Which robots.txt rules hurt a WordPress site?
These turn up in files copied from old tutorials, and each one costs something today. The pattern behind all of them: robots.txt blocks look free, and they aren't.
| Rule | Why it hurts | Write instead |
|---|---|---|
| Disallow: /wp-content/ | Google can't load the CSS, JavaScript and images it needs to render your pages | Nothing, leave it crawlable |
| Disallow: /category/ | Blocked archives get stuck as Indexed, though blocked by robots.txt, and Google can't see any noindex on them | Noindex thin archives in your SEO plugin, keep them crawlable |
| Noindex: /some-page/ | Google announced in 2019 that it stopped supporting noindex rules in robots.txt | A robots meta tag or an X-Robots-Tag header |
| Disallow: / | A staging leftover that stops all crawling of the whole site | Delete it the day the site launches |
Does robots.txt stop a page appearing in Google?
Not reliably, and Google's own documentation says so. robots.txt controls crawling. A blocked URL that other pages link to can still be indexed from those links alone, and it then shows in results with no description, because Google was never allowed to read it. To keep a page out of search, do the opposite of blocking it: let crawlers reach it and set noindex, through your SEO plugin or an X-Robots-Tag header. If the page is blocked in robots.txt, Google can't see the noindex at all.
This is the single most common robots.txt mistake I see on WordPress sites, and it usually surfaces in Search Console as a pile of URLs under Indexed, though blocked by robots.txt. What that status means, and when it matters, is part of why Google leaves pages out of its index.
Should robots.txt link your sitemap, and which one?
Yes. The Sitemap line is the one place every crawler looks for your sitemap's address without being told, and it's also where mismatches hide. WordPress core serves /wp-sitemap.xml; Yoast SEO and Rank Math switch it off and serve /sitemap_index.xml instead. After a plugin change, it's common to find robots.txt naming one address while Search Console has the other submitted, and Google quietly reading both.
The file matters for sitemaps in a second way: Google respects robots.txt when fetching sitemaps, so a Disallow rule that covers the sitemap's path stops Google reading it at all. If Search Console reports a sitemap it can't fetch, the live robots.txt is the first thing I read, and my WordPress robots.txt service exists for exactly that untangling. If you're setting up Search Console for the first time, my Search Console setup walkthrough covers submitting the right sitemap from day one.
Should a WordPress robots.txt block AI crawlers?
Only if you've decided you don't want AI systems using your content, and it's worth knowing the bots aren't interchangeable. OpenAI, Anthropic and Perplexity each document separate crawlers for separate jobs: training, search results, and a person asking a question. That means a site can allow one and block another, on purpose. One sensible split is to allow the search and user agents and decide about training bots separately, since blocking a training bot doesn't touch your search results.
GPTBot
Collects content for training OpenAI's models. Blocking it keeps your pages out of future training data, and costs you nothing in search.
OAI-SearchBot
Serves ChatGPT's search results. Blocking it means ChatGPT search won't surface or cite your pages, which for most businesses is traffic given away.
Google-Extended
Google's own control for AI training use, separate from Googlebot, so blocking it doesn't touch your normal Google rankings.
ClaudeBot
Anthropic says ClaudeBot collects web content that could contribute to training its models. It also documents Claude-SearchBot, which navigates the web to improve search result quality, and Claude-User, which visits a site when someone asks Claude a question.
PerplexityBot and Perplexity-User
Perplexity says PerplexityBot is designed to surface and link websites in its search results, while Perplexity-User may visit a page to answer a question a person asked. Blocking PerplexityBot means you aren't surfaced there.
Cloudflare's managed robots.txt
If the site sits behind Cloudflare, its managed robots.txt setting can generate and maintain a file that tells known AI crawlers to stay away, and its Content Signals label content as search, ai-input or ai-train. None of that shows on a WordPress screen, so read the live file.
Write the decision down
Whichever way you choose, put a comment line in the file saying why. The next person to open it shouldn't have to guess.
How do you test a WordPress robots.txt?
Three reads, cheapest first. Together they catch nearly every robots.txt problem I get hired for.
- i.
Fetch the live file
Open yourdomain.com/robots.txt in a private window. If what you see doesn't match what your plugin shows, something downstream, a physical file, a cache or Cloudflare, is rewriting it.
- ii.
Read Search Console's robots.txt report
It replaced the old robots.txt Tester and shows the exact copy Google last fetched, when, and any lines it couldn't parse.
- iii.
Run a live URL Inspection
Inspect a page you care about: Crawl allowed should read Yes. Anything else names the rule, and the layer, that's blocking it.
What happens if your robots.txt is missing or returns an error?
A missing file isn't an error to Google. Its documentation says its crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn't exist, so a 404 simply means everything is crawlable.
A server error is a different story. For the first 12 hours of 5xx responses Google stops crawling the site while it keeps trying to fetch the file, and if it still can't get a new version it uses the last good copy for the next 30 days. That's why a host outage that breaks /robots.txt can pause crawling before anyone notices.
Three smaller details save real confusion. Google caches the file for up to 24 hours, so a fix isn't instant. It enforces a limit of 500 KiB on the file. And the path in a rule is case-sensitive while the field names aren't, so Disallow: /Photos/ doesn't block /photos/.
Rather have it fixed for you? See the Robots and sitemaps service.
Robots and sitemapsWhat else do people ask about WordPress robots.txt?
Where is the robots.txt file located in WordPress?
At the site root, served at yourdomain.com/robots.txt. If you can't find a file over FTP, that's normal: WordPress generates a virtual one on request, and a real file only exists once you or a plugin like Yoast creates it.
Do I need a robots.txt file at all?
Practically, yes. Without one, crawlers assume everything is allowed, which is safe but wastes the file's one real job: naming your sitemap and keeping crawlers out of the dashboard. The default four lines plus an accurate Sitemap line covers most WordPress sites.
Why isn't my robots.txt updating?
Because something else is answering. A physical file overrides plugin settings, a page cache can serve a stale copy, and Cloudflare can rewrite the response at the edge. Fetch the live URL, then check Search Console's robots.txt report to see which version Google actually has.
Can robots.txt block bad bots and scrapers?
Not really. The rules are voluntary, so the bots you most want gone are the ones that ignore them. robots.txt is for managing well-behaved crawlers; blocking abusive ones is a job for your firewall or security layer.
Does Google support crawl-delay in robots.txt?
No. Google lists the fields it supports and notes that other fields, crawl-delay among them, aren't supported. A crawl-delay line won't slow Googlebot, so don't rely on it to protect a slow server. Fix the server instead.
Should I use robots.txt to save crawl budget?
Almost never on a small business site. Google's crawl budget guide is written for sites with 1 million or more unique pages that change weekly, or 10,000 or more that change daily. It also says not to use robots.txt to temporarily reallocate crawl budget: use it to block pages or resources you don't want Google to crawl at all.
How long does Google take to see a robots.txt change?
Google generally caches the file for up to 24 hours, so allow a day. Then open Search Console's robots.txt report to confirm which version it last fetched and that it parsed without errors.
Is robots.txt case-sensitive?
Partly. Field names such as User-agent and Disallow aren't case-sensitive, but the path in a rule is. Disallow: /Photos/ blocks that exact capitalisation and leaves /photos/ crawlable, so match the capitalisation your URLs actually use.
Where is the WordPress sitemap located?
Core serves /wp-sitemap.xml. Yoast SEO and Rank Math replace it with /sitemap_index.xml. The Sitemap line in your live robots.txt should name the one that's actually being generated, and that's the address to submit in Search Console.
Where do these rules come from?
Google's own documentation, plus what I see on the WordPress sites I work on. Written by Muhammad Younus, a WordPress technical SEO consultant.
- Google Search Central: Introduction to robots.txt
- Google Search Central: Create and submit a robots.txt file
- Google Search Central: A note on unsupported rules in robots.txt (2019)
- Google Search Central: Block search indexing with noindex
- Google Search Console help: Sitemaps report
- Google Search Console help: robots.txt report
- Yoast help: How to edit robots.txt through Yoast SEO
- Rank Math KB: How to edit robots.txt
- OpenAI: Overview of OpenAI crawlers
- Google Search Central: How Google interprets the robots.txt specification
- Google Search Central: Managing crawl budget for large sites
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler
- Perplexity: Perplexity crawlers
- Cloudflare: Managed robots.txt and Content Signals
- All in One SEO: What is robots.txt in WordPress
- Google Search Central: Google's crawlers and user agents
Want this checked on your own site?
Book a call on Upwork, share the URL and read-only Search Console access, and you'll get back what's actually wrong, ranked by the clicks it's costing.
Book a call on Upwork