Alejandro Rioja.
SEO

How To Use Screaming Frog: A Complete Guide For 2026

Alejandro Rioja
Alejandro Rioja
11 min read
TL;DR

Screaming Frog SEO Spider is a desktop crawler that mirrors how Google bots walk your site. Free up to 500 URLs; paid license removes the cap and unlocks JS rendering, integrations, and scheduled crawls. Use it to surface broken links, redirect chains, missing meta tags, duplicate content, and crawlability issues before they cost you rankings.

Free newsletter

Every Wednesday. 28,400+ operators. Zero fluff.

Table of contents

Open Table of contents

What is Screaming Frog?

Screaming Frog SEO Spider is a desktop application (Windows, Mac, Linux) that crawls websites the same way a search engine bot does — following links, reading HTML, collecting on-page data, and flagging issues.

The tool retrieves on-site data and organizes it into filterable tabs and exportable reports. That makes it fast to identify and prioritize technical SEO issues across a site of any size.

Core use cases:

  1. Find broken links (4xx errors) to fix
  2. Identify temporary and permanent redirects — and redirect chains
  3. Audit metadata: titles, descriptions, H1s, canonicals
  4. Find duplicate content and thin pages
  5. Review robots.txt directives and noindex tags
  6. Generate XML sitemaps
  7. Evaluate site architecture and internal linking depth
  8. Render JavaScript-heavy pages to check what the crawler actually sees

Free vs. paid: The free version crawls up to 500 URLs — useful for small sites or spot-checking. The paid annual license removes the URL cap and adds JS rendering, Google Analytics / Search Console integration, custom extraction, scheduled crawls, and more. (Check the Screaming Frog pricing page for current rates — verify before purchasing.)

Factors that affect website crawlability

Before running a crawl, it helps to know what you’re actually measuring. Here are the main signals that determine how efficiently a search engine bot navigates your site.

Domain authority and traffic signals

Domain name structure diagram

Sites with stronger authority tend to get crawled more frequently. Google allocates crawl budget based on how valuable and trustworthy a site is perceived to be. Screaming Frog helps you audit the signals — like crawl depth, internal link structure, and response times — that influence this.

See also: my GoDaddy review if you’re evaluating domain registrars.

What are backlinks diagram

Search engines treat backlinks as votes. Pages with strong external link equity tend to get crawled more often and indexed faster. A Screaming Frog crawl won’t show your backlink profile directly, but it surfaces pages that are orphaned (no internal links) — meaning they depend entirely on external links to get found.

Internal linking

Internal linking example

Internal linking is the practice of linking pages within the same site. It distributes link equity, signals content hierarchy to crawlers, and makes the user experience coherent.

Screaming Frog surfaces pages with zero inlinks — orphaned pages that bots might never reach. Fixing these is a quick win. For more context on anchor text strategy, see my local SEO guide.

XML sitemap

XML sitemap example

A well-structured XML sitemap is a roadmap that tells crawlers which pages to prioritize. Some pages have no internal links pointing to them, making them invisible without a sitemap. Screaming Frog can both audit an existing sitemap and generate a new one.

For a deeper technical audit workflow, see my technical SEO audit guide.

URL canonicalization

Canonical tag in MozBar

If one page is reachable via multiple URLs, or if multiple pages have near-identical content, Google may treat them as duplicates — diluting link equity and indexation. Setting a canonical URL consolidates signals onto the preferred version.

Screaming Frog’s Canonicals tab surfaces missing, conflicting, or self-referencing canonicals at a glance.

Meta tags

Meta description in Google search results

Meta descriptions don’t directly affect rankings, but they influence click-through rate from the SERP — and in AI Overviews, a concise, well-written description can be pulled directly into the cited snippet. Screaming Frog flags missing, duplicate, or over-length titles and descriptions across your entire site in one crawl.

Where to begin

Installation

Screaming Frog download page

Download the installer from screamingfrog.co.uk/seo-spider. It’s available for Windows, macOS, and Linux. The free version works immediately — no account needed. It crawls up to 500 URLs per session.

License activation

To remove the URL cap and access the full feature set, purchase an annual license from the Screaming Frog site (verify current pricing — it changes). You’ll receive a license key to enter under License > Enter License Key. A dialog will confirm the key is valid and show the expiry date.

Familiarize yourself with the UI

The main menus you’ll use:

  1. File — Save crawls, reopen recent sessions (up to six), and export raw data.
  2. Configuration — The most important menu. Configuration > Spider controls what gets crawled: which content types to include, whether to follow nofollow links, JS rendering mode, and more.
  3. Bulk Export — Export URLs filtered by response code, inlinks, directives, images, hreflang, and other dimensions.
  4. Reports — Download structured reports: redirect chains, canonical errors, hreflang errors, pagespeed — useful for handing off issues to a dev team.
  5. Sitemaps — Generate an XML sitemap directly from a completed crawl.

Setting up your device for larger crawls

Memory and storage

For small sites (under a few thousand URLs), default settings are fine. For larger crawls:

After changing the storage mode or RAM allocation, restart Screaming Frog for the changes to take effect.

JavaScript rendering

If you’re crawling a site built on React, Vue, Next.js, or another JS framework, switch to JavaScript rendering: Configuration > Spider > Rendering > JavaScript. This runs a headless browser (Chromium-based) to execute JS before crawling the rendered HTML — the same way Googlebot processes modern sites.

Note: JS rendering is significantly slower than standard crawls and uses more resources. Use it for JS-heavy sites; skip it for traditional HTML sites.

Using Screaming Frog

Start a crawl

Spider mode (default): Crawls an entire site starting from a root URL. Enter the homepage URL in the address bar and click Start.

List mode: Crawls only the specific URLs you provide. Useful when you want to audit a specific set of pages (e.g., all product pages, or URLs pulled from Google Search Console with performance drops). Switch via Mode > List, then upload a CSV or paste URLs directly.

Screaming Frog start crawl interface

During a crawl, the status bar at the bottom shows URLs crawled, speed, and response times in real time. Key notes:

  1. You can pause and resume a crawl (licensed version only).
  2. Exiting the application without saving loses in-progress data.
  3. In database storage mode, crawls auto-save.
  4. You can open up to six recent crawls under the File menu.

Viewing crawl data

The top panel shows tabs by data type: Internal, External, Response Codes, URL, Page Titles, Meta Description, H1, H2, Images, Canonical, Pagination, Directives, Hreflang, Links, Structured Data, and more.

Each tab has filter dropdowns — for example, under Response Codes, you can filter to show only 4xx (client errors) or 3xx (redirects). Click any URL in the top panel to populate the bottom panel with details: inlinks, outlinks, image data, and response headers for that specific URL.

Viewing potential issues in the Overview

The Overview pane on the right summarizes issues across all tabs — broken links, missing titles, duplicate H1s, redirect chains — without requiring you to click through every tab individually. Click any row in the Overview to jump directly to the relevant tab and filter. This is the fastest way to triage a new crawl.

Exporting the data

For a full technical SEO audit workflow using this export data, see my technical SEO audit guide.

Saving and opening crawls

In database storage mode, crawls save automatically. In RAM mode, save manually via File > Save Crawl when the crawl is paused or finished.

The crawl management screen lets you organize, duplicate, export, or delete saved crawls — useful if you want to track the same site over time and compare sessions.

What to do after the crawl

Screaming Frog surfaces the data — it doesn’t tell you what to prioritize. Here’s how I work through a fresh crawl:

  1. Start with the Overview pane. Look for high-count errors: 4xx responses, pages with missing titles, redirect chains longer than one hop.
  2. Fix 4xx errors first. Broken internal links waste crawl budget and create dead ends for users. Export them, find the correct destination, and update or redirect.
  3. Collapse redirect chains. A chain like A → B → C → D should be collapsed to A → D. Every extra hop adds latency and can lose link equity.
  4. Audit metadata. Pages with missing, duplicate, or truncated titles and descriptions are quick fixes with outsized impact — especially now that AI Overviews pull meta content directly.
  5. Check canonical tags. Missing canonicals on paginated or parameterized URLs cause duplicate content issues that compound over time.
  6. Find orphaned pages. Filter the Links tab for pages with zero inlinks. These are invisible to crawlers unless they’re in the sitemap.

Screaming Frog in 2026 — What’s Changed

The tool itself has continued to evolve. As of early 2026 (verify current feature set on their site):

Screaming Frog FAQ — 2026

Is Screaming Frog still worth it when AI tools like ChatGPT can audit sites?

Yes. AI chat tools describe technical SEO concepts; Screaming Frog actually crawls your site and returns a ground-truth dataset. There’s no substitute for having the complete list of your 4xx errors, redirect chains, and missing canonical tags as exportable CSVs. Use AI to interpret and prioritize the data; use Screaming Frog to generate it.

How do I crawl a JavaScript-heavy site correctly?

Go to Configuration > Spider > Rendering and select JavaScript. This uses a headless Chromium instance to execute JS before crawling. It’s the closest approximation to how Googlebot processes modern React/Vue/Next.js sites. Be prepared for crawls to run 3–5x slower than standard mode.

What’s the difference between Spider mode and List mode?

Spider mode crawls an entire site by following links from a starting URL — you’ll discover pages you didn’t know existed. List mode crawls only the specific URLs you provide — useful when you already have a URL set from another data source (Search Console, a database export, a competitor sitemap) and don’t need full-site discovery.

Does Screaming Frog help with AI Overview optimization?

Indirectly but meaningfully. AI Overviews favor well-structured, crawlable pages with clean metadata, valid structured data, and no technical errors. Screaming Frog’s structured data validation tab, canonical audit, and meta tag analysis all surface issues that affect how AI systems read and cite your pages. It won’t tell you if you’re cited, but it helps ensure nothing technical is blocking the path.

Related reading:

This guide is part of alejandrorioja.com — written by Alejandro Rioja, who now builds AI agent systems for founders. Including the agent that keeps this site current. How it works →

Updated for May 2026

SEO in 2026 is unrecognizable from the 2020-era playbook. Three shifts that matter for anything written before mid-2024:

  1. AI Overviews are the new SERP zero position. Google’s AI Overviews default to roughly 60% of US informational queries, eating most “what is” / “how to” CTR. Optimizing for citation inside the AI Overview is now as important as ranking #1.
  2. GEO (Generative Engine Optimization) is the working term for cross-engine optimization — getting cited inside ChatGPT, Perplexity, Claude, and Gemini answers. ~12% of high-intent commercial queries in late 2025 sample studies showed a direct-citation flow from these engines (vs. zero pre-2023).
  3. E-E-A-T (now E^3-A-T, Experience + Expertise + Establishment + Authoritativeness + Trustworthiness) continues to be the framing Google uses internally — “Establishment” was the 2024 addition emphasizing brand-level signals.

Tool landscape (May 2026): Ahrefs and Semrush both shipped Generative Engine tracking. Surfer SEO + the Topical Authority crowd added GEO scoring. Screaming Frog still the standard crawler. AlsoAsked, Keyword Insights, and Frase shifted heavily into AI-Overview snippet engineering.

If this post predates May 2024, treat its core advice as the Google-search baseline and layer the GEO playbook on top.

Keep reading

Get the AI playbook in your inbox

Every Wednesday. 28,400+ operators. Zero fluff.

⌕
↵ ↵ to see all results esc esc to close