Screaming Frog

Screaming Frog SEO Spider is desktop software and suitable for someone who knows SEO. It runs on your own machine, so a crawl's size is limited by your RAM and storage rather than a subscription tier. You open the app, point it at a domain, and it crawls like a search engine bot would, following links and pulling back status codes, page titles, meta data, headings, and canonical tags for every URL it touches.

That local-first design is also its main trade-off. A five-person agency running large crawls needs five separate licences and five machines capable of handling the memory load, where a cloud crawler would just add a seat.

Screaming Frog doesn't try to be a collaboration platform. What it does instead is stay fast, cheap for what it does, and comprehensive at the level of a single crawl — duplicate content detection, hreflang auditing, structured data validation, and a custom extraction engine that can pull arbitrary data out of the page source using CSS, XPath, or regex.

The free version is capped at 500 URLs per crawl, which is enough to get a feel for the tool on a small site. Nearly every real use case needs the $279/year license, which removes that cap and runs up to what your hardware can handle and turns on JavaScript rendering, crawl comparison between runs, and integrations with Google Analytics, Search Console, and PageSpeed Insights so crawl data and performance data live in the same export.