Screaming Frog Guide For The SEO Spider
User Guide
For the SEO Spider
General
Installation
Crawling
- Saving, opening, exporting & importing crawls
- Configuration
- Scheduling
- Exporting
- Robots.txt
- User agent
- Memory
- Checking memory allocation
- Cookies
- XML sitemap creation
- Visualisations
- Reports
- Command line interface set-up
- Command line interface
- User Interface
- Search function
- Auto Updates
Configuration Options
Spider Crawl Tab
- Images
- Media
- CSS
- JavaScript
- SWF
- Internal hyperlinks
- External links
- Canonicals
- Pagination (rel next/prev)
- Hreflang
- AMP
- Meta refresh
- iframes
- Mobile alternate
- Uncrawlable Links
- Check links outside of start folder
- Crawl outside of start folder
- Crawl all subdomains
- Follow internal or external ‘nofollow’
- Crawl invalid links
- Crawl linked XML sitemaps
Spider Extraction Tab
Spider Limits Tab
- Limit crawl total
- Limit crawl depth
- Limit URLs per crawl depth
- Limit max folder depth
- Limit number of query strings
- Limit crawl total per subdomain
- Max redirects to follow
- Limit max URL length to crawl
- Max links per URL to crawl
- Max page size (kb) to crawl
- Limit by URL path
Spider Rendering Tab
- Rendering
- Rendered page screen shots
- JavaScript error reporting
- Flatten Shadow DOM
- Flatten iframes
- Archive website
- AJAX timeout
- Window size
Spider Advanced Tab
- Cookie storage
- Ignore non-indexable URLs for Issues
- Ignore paginated URLs for duplicate filters
- Always follow redirects
- Always follow canonicals
- Respect noindex
- Respect canonical
- Respect next/prev
- Respect HSTS policy
- Respect self referencing meta refresh
- Extract images from img srcset attribute
- Crawl fragment identifiers
- Perform HTML validation
- Green hosting carbon calculation
- Assume pages are HTML
- Response timeout
- 5XX response retries
Spider Preferences Tab
Other Configuration Options
- Content area
- Duplicates
- Spelling & grammar
- Embeddings
- Robots.txt
- URL rewriting
- CDNs
- Include
- Exclude
- Speed
- User agent
- HTTP header
- Custom search
- Custom extraction
- Custom link positions
- Custom JavaScript
- Google Analytics integration
- Google Search Console integration
- PageSpeed Insights integration
- Majestic
- Ahrefs
- Moz
- OpenAI
- Gemini
- Ollama
- Anthropic
- Authentication
- Segments
- Crawl analysis
- User Interface
- Language
- Proxy
- Storage mode
- Memory allocation
- Crawl Retention
- Trusted Certificates
- Notifications
- MCP Server
- Mode
Tabs
Top Tabs
- Internal
- External
- Security
- Response Codes
- URL
- Page titles
- Meta description
- Meta keywords
- h1
- h2
- Content
- Images
- Canonicals
- Pagination
- Directives
- hreflang
- JavaScript
- Links
- AMP
- Structured data
- Sitemaps
- PageSpeed
- Mobile
- Accessibility
- Custom search
- Custom extraction
- Custom JavaScript
- Analytics
- Search Console
- Validation
- Link Metrics
- AI
- Change Detection
Tutorials
- Getting Started Guide
- How To Find Broken Links Using The SEO Spider
- Site Architecture & Crawl Visualisations Guide
- How To Compare Crawls
- How To Crawl JavaScript Websites
- How To Crawl Large Websites
- How To Crawl A Staging Website
- How To Automate Crawl Reports In Data Studio
- How To Audit Core Web Vitals
- How to Identify Semantically Similar Pages & Outliers
- How to Run the Screaming Frog SEO Spider in the Cloud
- How To Crawl With AI Prompts
- How To Check For Duplicate Content
- How To Set Up Crawl Email Notifications & Export Attachments
- How To Perform A Web Accessibility Audit
- How To Find Missing Image Alt Text & Attributes
- Web Scraping & Custom Extraction
- Internal Linking Audit With the SEO Spider
- How to Work in Teams Using the SEO Spider
- How to Crawl with ChatGPT
- How To Test & Validate Structured Data
- How To Use Vector Embeddings for Redirect Mapping
- How To Use List Mode
- Spell & Grammar Check Your Website
- How To Audit Redirects In A Site Migration Using The SEO Spider
- How To Debug Missing Pages In A Crawl
- How To Use N-Grams
- How To Audit Canonicals
- How To Audit Mobile Usability
- How To Audit & Validate Accelerated Mobile Pages (AMP)
- How To Audit Hreflang
- How To Audit rel="next" and rel="prev" Pagination Attributes
- How To Audit XML Sitemaps
- How To Bulk Check Redirects
- How To Find Orphan Pages
- How To Use Custom Search
- HTTP Status Codes – Why Won’t My Website Crawl?
- Robots.txt Testing In The SEO Spider
- What Is Link Score?
- XML Sitemap Generator
- How To Use The SEO Spider In A Site Migration
- How To Bypass Geo IP Redirection In A Crawl
- How To Perform A Cookie Audit
- How To Find Broken Bookmarks
- How To Analyse Link Position
- How To Perform A Parity Audit
- How To Automate The URL Inspection API
- How To Test Readability
- How To Audit PDFs
- How To Use The SEO Spider For Broken Link Building
- How To Debug Invalid HTML Elements In The Head
- An SEOs Guide To Crawling HSTS
- Configure X Virtual Framebuffer
- Resolving Google Analytics / Google Search Console Connection Issues
- Crawling Password Protected Websites
- How to Debug Custom JavaScript Snippets
- Simulating User Interactions with Custom JavaScript