How I crawl a WordPress site with Screaming Frog before I change anything

Can I use Screaming Frog for free, and what do I set up first?

Before I change a plugin, a redirect or a title on a WordPress site, I crawl it. It's the first thing I run when I'm finding what's actually broken on a site before anything gets touched, and it's the baseline I measure every later change against.

You can start free. Screaming Frog's product page says the free version crawls 500 URLs and a licence removes the limit; on 28 September 2026 it listed a licence at 279 US dollars a year, or 199 pounds on the UK pricing page, per user. That isn't the only limit. The same page's comparison marks crawl configuration, saving crawls, JavaScript rendering and the Search Console integration as paid. The free version is fine for a quick broken link check on a small site, but the method below needs the licence.

With a licence, I set up storage before the first crawl. The vendor's getting started guide recommends database storage, the default, on a computer with an SSD, and memory storage without one. Database storage lets it crawl more URLs and keeps every crawl saved automatically.

Which Screaming Frog settings do I change before the first crawl?

A crawl that runs on the defaults misses half of what I want, so I change a handful of settings first and save them as a profile. Screaming Frog's own user guide is the reference for each one. Most Screaming Frog tutorials walk through the defaults tab by tab, which suits a quick look at any site, but not the comparison I care about, which needs the sitemaps and Search Console inside the same crawl.

Since version 5.5, WordPress core has served a sitemap index at /wp-sitemap.xml and named it in the robots.txt it generates. An SEO plugin often replaces that with its own index, so I read robots.txt before the crawl and check which sitemap the site really serves.

Exclude takes partial regex, and I only use it for URL patterns that multiply without adding pages, like add-to-cart links on a WooCommerce store. Excluding real sections to make the crawl shorter defeats the point of running it.

Why compare the crawl with the sitemap and Search Console?

Because each list tells you something different, and none of them is the whole site. The crawl is what the site links to. The sitemap is what the site says it wants indexed. Search Console is what Google has actually found. When the three agree, the site's coherent. When they don't, the gap is almost always a fault somebody didn't know about.

Google's sitemap guidance is plain: include the URLs you want to see in search results. So a sitemap listing a redirect, a 404 or a noindexed page sends Google a mixed message, and those 404s are where I start when I fix 404 errors in WordPress. A page that's in the sitemap but that nothing links to is a linking problem long before it's an indexing one, which is the gap my internal linking service closes. That crawl file feeds my internal link audit.

Search Console's Page indexing report helps from the other side. Its filter switches between all known pages, all submitted pages and unsubmitted pages only, so I can see whether the pages Google isn't indexing are ones the sitemap asked for. Its example list stops at 1,000 URLs, though, so on a big site it's a sample, not the full picture. When sitemap pages sit in Discovered or Crawled, currently not indexed for weeks, those two statuses have their own diagnosis.

What can a crawl find that no other tool will?

Mistakes that nobody links to and nobody visits. Search Console reports what Google's done with a page, and an SEO plugin checks the page you're editing. Neither one reads every URL the site serves and lines them up side by side.

Scrap a Vehicle is the example I use. It's a UK scrap car site with 487 location pages generated with Lpagery, and a full crawl turned up two things you'd never find clicking around. The first was /region_name/, live and indexable, with the literal title {seo_title} and the heading {h1_heading}. It's the template page, published by accident, and it only stands out when its title sits in a list next to hundreds of real ones.

The second was /scrap-my-car-edinburgh-2/, a duplicate Edinburgh page. A slug ending in -2 usually means the page was generated twice, so two pages end up competing for one town. Neither fault shows up for someone clicking through the menus.

That's also why I don't fix as I go. On a site with hundreds of generated pages, one change to a template touches every page built from it, so I want the whole list first.