# Screaming Frog SEO Spider The industry leading website crawler for Windows, macOS and Linux, trusted by thousands of SEOs and agencies worldwide for technical SEO site audits. ## SEO Spider Tool The **Screaming Frog** SEO Spider is a website crawler that audits your site for over 300 SEO issues to help improve both SEO, and UX. Download & crawl 500 URLs for free, or buy a licence for £199 Per Year to remove the limit & access advanced features. [Read our getting started guide](/content/seo-spider/tutorials/quick-start-guide/index.html) ## What can you do with the SEO Spider Tool? Identify over 300 SEO issues, warnings and opportunities to improve SEO, website health and user experience. ### Find Broken Links Crawl a website instantly and find broken links (404s) and server errors. Bulk export the errors and source URLs to fix, or send to a developer. ### Audit Redirects Find temporary and permanent redirects, identify redirect chains and loops, or upload a list of URLs to audit in a site migration. ### Analyse Page Titles & Meta Data Analyse page titles and meta descriptions during a crawl and identify those that are too long, short, missing, or duplicated across your site. ### Discover Duplicate Content Discover exact duplicate URLs with an md5 algorithmic check, partially duplicated elements such as page titles, descriptions or headings and find low content pages. ### Extract Data with XPath Collect any data from the HTML of a web page using CSS Path, XPath or regex. This might include social meta tags, additional headings, prices, SKUs or more! ### Review Robots & Directives View URLs blocked by robots.txt, meta robots or X-Robots-Tag directives such as ‘noindex’ or ‘nofollow’, as well as canonicals and rel=“next” and rel=“prev”. ### Generate XML Sitemaps Quickly create XML Sitemaps and Image XML Sitemaps, with advanced configuration over URLs to include, last modified, priority and change frequency. ### Integrate with GA, GSC & PSI Connect to the Google Analytics, Search Console and PageSpeed Insights APIs and fetch user and performance data for all URLs in a crawl for greater insight. ### Crawl JavaScript Websites Render web pages using the integrated Chromium WRS to crawl dynamic, JavaScript rich websites and frameworks, such as Angular, React and Vue.js. ### Visualise Site Architecture Evaluate internal linking and URL structure using interactive crawl and directory force-directed diagrams and tree graph site visualisations. ### Schedule Audits Schedule crawls to run at chosen intervals and auto export crawl data to any location, including Google Sheets. Or automate entirely via command line. ### Compare Crawls & Staging Track progress of SEO issues and opportunities and see what's changed between crawls. Compare staging against production environments using advanced URL Mapping. ## Features - Find Broken Links, Errors & Redirects - Analyse Page Titles & Meta Data - Review Meta Robots & Directives - Audit hreflang Attributes - Discover Exact Duplicate Pages - Generate XML Sitemaps - Site Visualisations - Crawl Limit - Scheduling - Crawl Configuration - Save & Open Crawls - JavaScript Rendering - Crawl Comparison - Near Duplicate Content - Custom robots.txt - Mobile Usability - AMP Crawling & Validation - Structured Data & Validation - Spelling & Grammar Checks - Custom Source Code Search - Custom Extraction - Custom JavaScript - Crawl with OpenAI & Gemini - Google Analytics Integration - Search Console Integration - PageSpeed Insights Integration - Accessibility Auditing - Link Metrics Integration - Forms Based Authentication - Segmentation - Looker Studio Crawl Report - Free Technical Support #### Price per licence Licences last 1 year. After that you will be required to renew your licence. ### Free Version - Find Broken Links, Errors & Redirects - Analyse Page Titles & Meta Data - Review Meta Robots & Directives - Audit hreflang Attributes - Discover Exact Duplicate Pages - Generate XML Sitemaps - Site Visualisations - Crawl Limit - 500 URLs - Scheduling - Crawl Configuration - Save & Open Crawls - JavaScript Rendering - Crawl Comparison - Near Duplicate Content - Custom robots.txt - Mobile Usability - AMP Crawling & Validation - Structured Data & Validation - Spelling & Grammar Checks - Custom Source Code Search - Custom Extraction - Custom JavaScript - Crawl with OpenAI & Gemini - Google Analytics Integration - Search Console Integration - PageSpeed Insights Integration - Accessibility Auditing - Link Metrics Integration - Forms Based Authentication - Segmentation - Looker Studio Crawl Report - Free Technical Support ### Paid Version - Find Broken Links, Errors & Redirects - Analyse Page Titles & Meta Data - Review Meta Robots & Directives - Audit hreflang Attributes - Discover Exact Duplicate Pages - Generate XML Sitemaps - Site Visualisations - Crawl Limit - Unlimited* - Scheduling - Crawl Configuration - Save & Open Crawls - JavaScript Rendering - Crawl Comparison - Near Duplicate Content - Custom robots.txt - Mobile Usability - AMP Crawling & Validation - Structured Data & Validation - Spelling & Grammar Checks - Custom Source Code Search - Custom Extraction - Custom JavaScript - Crawl with OpenAI & Gemini - Google Analytics Integration - Search Console Integration - PageSpeed Insights Integration - Accessibility Auditing - Link Metrics Integration - Forms Based Authentication - Segmentation - Looker Studio Crawl Report - Free Technical Support ## The SEO Spider Tool Crawls & Reports On... The **Screaming Frog** SEO Spider is an SEO auditing tool, built by real SEOs with thousands of users worldwide. A quick summary of some of the data collected in a crawl include - - **Errors** – Client errors such as broken links & server errors (No responses, 4XX client & 5XX server errors). - **Redirects** – Permanent, temporary, JavaScript redirects, meta & HTTP refreshes, redirect chains and loops. - **Blocked URLs** – View & audit URLs disallowed by the robots.txt protocol. - **Blocked Resources** – View & audit blocked resources in rendering mode. - **External Links** – View all external links, their status codes and source pages. - **Site Structure** – Analyse site architecture, indexability and crawl depth by directory. - **Internal Linking** – Analyse internal links, link counts, crawl depth and calculate internal Link Score. - **Anchor Text** – View aggregated and granular anchor text of all links. Identify non-descriptive anchor text to improve for users and search engines. - **Security** – Discover insecure pages, mixed content, insecure forms, missing security headers and more. - **URL Issues** – Non ASCII characters, underscores, uppercase characters, parameters, long URLs, repetitive paths or broken bookmarks. - **Page Titles** – Identify missing, duplicate, long, short or multiple titles. - **Meta Description** – Identify missing, duplicate, long, short or multiple descriptions. - **Headings** – View h1 and h2 headings, including if any are missing, duplicate, long, short, multiple or non-sequential. - **Content** – View word count, analyse readability and identify low relevance content that deviates from the average content focus of the site. - **Directives** – View directives in meta robots or the X-Robots-Tag header, including noindex, nofollow, none, nosnippet and more. - **Canonicals** – Analyse canonical link elements & canonical HTTP headers. - **Pagination** – View rel=”next” and rel=”prev” attributes, as well as common set up issues with paginated pages. - **hreflang Attributes** – Audit missing return tags, inconsistent & incorrect languages codes, non-200 response hreflang and more. - **Duplicate Content** – Discover exact, near duplicate and semantically similar pages using algorithmic checks and vector embeddings. - **Rendering** – Crawl JavaScript frameworks like AngularJS and React, by crawling the rendered HTML after JavaScript has executed. - **JavaScript** – Identify content, links, page titles, descriptions, headings and other key elements that rely on JavaScript. - **Images** – Find all images and discover those that are too large, missing alt text, background images, missing size attributes and more. - **Validation** – Find issues that can impact search bots from being able to parse and understand a page reliably. - **User-Agent** – Crawl as a search bot such as Googlebot or Bingbot, AI crawlers, or your own custom UA. - **Custom HTTP Headers** – Supply any header value in a request, from Accept-Language to cookie. - **Custom Source Code Search** – Search for anything in the source code of a website – such as analytics tracking tags, keywords or code. - **Custom Extraction** – Scrape any data from the HTML of a URL using XPath, CSS Path selectors or regex. - **Custom JavaScript** – Run custom JavaScript snippets while crawling, to extract data, trigger mouseover events, scroll a page or nearly anything else you’re able to do in the Chrome console. - **Google Analytics Integration** – Connect to the Google Analytics API and pull in user and conversion data directly during a crawl. - **Google Search Console Integration** – Connect to the Google Search Analytics and URL Inspection APIs and collect performance and index status data in bulk. - **PageSpeed Insights Integration** – Connect to the PSI API for Lighthouse metrics, speed opportunities, diagnostics and Chrome User Experience Report (CrUX) data at scale. - **Mobile Usability** – Use Lighthouse to check for common mobile usability issues. - **External Link Metrics** – Pull external link metrics from Majestic, Ahrefs and Moz APIs into a crawl to perform content audits or profile links. - **XML Sitemap Generation** – Create an XML sitemap and an image sitemap using the SEO spider. - **Custom robots.txt** – Download, edit and test a site’s robots.txt using the new custom robots.txt. - **Rendered Screen Shots** – Fetch, view and analyse the rendered pages crawled. - **Store & View HTML & Rendered HTML** – Essential for analysing the DOM. - **AMP Crawling & Validation** – Crawl AMP URLs and validate them, using the official integrated AMP Validator. - **XML Sitemap Analysis** – Crawl an XML Sitemap independently or part of a crawl, to find missing, non-indexable and orphan pages. - **Visualisations** – Analyse the internal linking and URL structure of the website, using the crawl and directory tree force-directed diagrams and tree graphs. - **Structured Data & Validation** – Extract & validate structured data against Schema.org specifications and Google rich result features. - **Spelling & Grammar** – Spell & grammar check your website in over 25 different languages. - **Website Accessibility** – Use the open-source AXE accessibility rule set for automated accessibility validation to test against Web Content Accessibility Guidelines (WCAG). - **AI Integration** – Set up custom AI prompts with OpenAI, Gemini, Ollama and Anthropic while crawling for insight. - **Crawl Comparison** – Compare crawl data to identify changes and track technical SEO progress. Compare site structure, detect changes in key elements and use URL mapping to compare staging Vs production. - **Data Studio Crawl Reports** – Set up automated Data Studio crawl reports to monitor site health and trends. ### About The Tool The **Screaming Frog** SEO Spider is an SEO audit tool trusted by thousands of professional SEOs, agencies and brands worldwide to perform technical SEO audits. It can help identify over [300 SEO issues](/content/seo-spider/issues/index.html), warnings and opportunities to improve website health and user experience. An in-app explanation of each issue and potential actions is provided. Each issue has a ‘type’ and an estimated ‘priority’ based upon the potential impact. The SEO Spider allows you to export key onsite SEO elements (URL, page title, meta description, headings etc) to a spread sheet, so it can easily be used as a base for SEO recommendations. Check our out demo video above. ### Crawl 500 URLs For Free The ‘lite’ version of the tool is free to download and use – without any sign up or email capture. The free version is restricted to crawling up to 500 URLs in a single crawl. The crawl configuration, saving of crawls, and advanced features such as JavaScript rendering, custom extraction, and API integrations are also restricted. For just £199 per year you can purchase a licence, which removes the 500 URL crawl limit, allows you to save crawls, and opens up the spider’s configuration options and advanced features.