Robots.txt in WordPress: Find It, Edit It, Fix It Safely

One small file, outsized consequences. A good robots.txt steers crawl attention toward the pages that earn you money. A careless one can quietly remove your whole site from Google. This guide covers where the file lives, three safe ways to edit it, what belongs inside, and the mistakes I keep finding in real audits.

What is robots.txt in WordPress?

It's the first thing a crawler requests when it visits your domain: a set of instructions saying which URLs each bot may fetch. It works on the honor system, and the major crawlers, Googlebot, Bingbot, and the AI crawlers, respect it. It is not a security tool and not an index-removal tool; it only controls crawling.

That distinction matters more than any syntax detail: robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if other sites link to it. We'll come back to this, because it's the root of the most damaging mistake on this page.

Where is the robots.txt file in WordPress?

At yoursite.com/robots.txt, always, but here's the part most guides skip: by default there is no actual file. WordPress intercepts the request and generates the response in PHP, on the fly. That's why FTP and file managers show nothing at the root while the browser happily displays content.

The virtual default looks roughly like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/wp-sitemap.xml

Sensible, minimal, and doing nothing for you beyond the basics. The moment you create a physical robots.txt file at the root, or a plugin does, WordPress stops generating the virtual one and your file takes over completely.

How do you edit robots.txt in WordPress?

Three methods, from easiest to most control. Whichever you use, copy your current file somewhere safe first, and test after saving.

1. Through your SEO plugin. In RankMath: General Settings, then Edit robots.txt. Paste, save, done. The plugin serves your version instead of the WordPress default. Yoast has the equivalent under Tools, File editor.
2. As a physical file. Create a plain-text file named robots.txt, put your rules in it, and upload it to your site root (the folder holding wp-config.php) via your host's file manager or SFTP. A physical file overrides everything else, including plugins.
3. With a code filter. For developers who keep configuration in the theme: WordPress applies the robots_txt filter to its virtual output, so a few lines in functions.php or a small plugin can append rules without any file. Useful when the file must stay version-controlled.

What should a WordPress robots.txt contain?

The directives are few and the syntax is unforgiving of typos:

A strong starting point for most WordPress sites:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /*?orderby=
Disallow: /*?filter_

Sitemap: https://yoursite.com/sitemap_index.xml

The search and parameter lines are where the real value hides: internal search results and WooCommerce filter URLs multiply into thousands of near-duplicate pages that burn crawl budget. Adjust the parameter patterns to what your own site actually generates, which you can see in Search Console's crawl stats. And never blanket-block /wp-content/: your CSS, JavaScript, and images live there, and Google must render them to evaluate your pages.

Should you block AI crawlers like GPTBot?

Not if you want customers to find you through AI. It's a genuine choice with a real trade-off: blocking GPTBot, ClaudeBot, and PerplexityBot keeps your content out of AI training and answers, and it also keeps your business out of the recommendations those assistants make. For a publisher selling content, blocking can be rational. For a business that wants to be found, it's self-sabotage.

To welcome them explicitly:

User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /

This site runs exactly that policy, paired with an llms.txt file summarizing the business for AI crawlers. Being citable by AI engines is a growing traffic channel, and robots.txt is its front door; the AI search optimization service covers the full setup.

Which robots.txt mistakes kill WordPress rankings?

• Disallow: / left over from development. One character path, entire site invisible. This is the first thing I check on any "our traffic vanished" call, and I've found it live on real production sites.
• Blocking CSS and JavaScript. Old advice said to block /wp-includes/ and asset folders. Google renders pages like a browser now; block the assets and every page looks broken to it.
• Blocking a page to deindex it. Blocked pages can't show Google their noindex tag, so they linger in the index as ghost listings. Crawl-block or index-block, never both on the same URL.
• Blocking the sitemap or forgetting the Sitemap line. Small, silly, common.
• Case and path typos. Rules are case-sensitive and prefix-matched: Disallow: /Blog/ doesn't block /blog/, and Disallow: /shop blocks /shopping-guide too.

Robots.txt vs noindex: which one when?

Use robots.txt to manage crawling: keep bots out of infinite parameter spaces, internal search, and admin areas, so crawl budget flows to real pages. Use a noindex meta tag to manage indexing: let Google crawl thin archives or utility pages, read the tag, and drop them from results.

The test is one question: do I need Google to read this page's instructions? Deindexing requires reading, so noindex needs the crawl open. Pure waste, like ten thousand filter combinations, never needs reading, so robots.txt handles it. Getting this pairing right across tags, archives, and WooCommerce URLs is a big part of my crawlability and indexing service, because the two tools misapplied together cause exactly the ghost-listing problem described above.

How do you test robots.txt?

1. Load it directly. yoursite.com/robots.txt in a browser. It should return your rules instantly, not a 404 or an error page.
2. Use Search Console's robots.txt report. Under Settings, Google shows the file it currently holds, when it last fetched it, and any parse errors, and lets you request a refetch after changes.
3. Spot-check specific URLs. Use URL Inspection on a page you intend to keep crawlable. If it reports "blocked by robots.txt," your rules are catching more than you meant, usually a prefix-match surprise.
4. Watch crawl stats for a week. After a cleanup, Googlebot's fetch distribution should visibly shift away from junk URLs toward content. That shift is the whole point.