# User Guide ## For the SEO Spider ## General ### Installation - [Installation on Windows](/content/seo-spider/user-guide/general/#installation-on-windows/index.html) - [Installation on macOS](/content/seo-spider/user-guide/general/#installation-on-macos/index.html) - [Installation on Ubuntu](/content/seo-spider/user-guide/general/#installation-on-ubuntu/index.html) - [Installation on Fedora](/content/seo-spider/user-guide/general/#installation-on-fedora/index.html) ### Crawling - [Saving, opening, exporting & importing crawls](/content/seo-spider/user-guide/general/#saving-uploading-crawls/index.html) - [Configuration](/content/seo-spider/user-guide/general/#default-configuration/index.html) - [Scheduling](/content/seo-spider/user-guide/general/#scheduling/index.html) - [Exporting](/content/seo-spider/user-guide/general/#exporting/index.html) - [Robots.txt](/content/seo-spider/user-guide/general/#robots-txt/index.html) - [User agent](/content/seo-spider/user-guide/general/#user-agent/index.html) - [Memory](/content/seo-spider/user-guide/general/#memory/index.html) - [Checking memory allocation](/content/seo-spider/user-guide/general/#checking-memory-allocation/index.html) - [Cookies](/content/seo-spider/user-guide/general/#cookies/index.html) - [XML sitemap creation](/content/seo-spider/user-guide/general/#xml-sitemap-creation/index.html) - [Visualisations](/content/seo-spider/user-guide/general/#visualisations/index.html) - [Reports](/content/seo-spider/user-guide/general/#reports/index.html) - [Command line interface set-up](/content/seo-spider/user-guide/general/#command-line-interface-set-up/index.html) - [Command line interface](/content/seo-spider/user-guide/general/#command-line/index.html) - [User Interface](/content/seo-spider/user-guide/general/#user-interface-2/index.html) - [Search function](/content/seo-spider/user-guide/general/#search-function/index.html) - [Auto Updates](/content/seo-spider/user-guide/general/#auto-updates/index.html) ## Configuration Options ### Spider Crawl Tab - [Images](/content/seo-spider/user-guide/configuration/#check-images/index.html) - [Media](/content/seo-spider/user-guide/configuration/#check-media/index.html) - [CSS](/content/seo-spider/user-guide/configuration/#check-css/index.html) - [JavaScript](/content/seo-spider/user-guide/configuration/#check-javascript/index.html) - [SWF](/content/seo-spider/user-guide/configuration/#check-swf/index.html) - [Internal hyperlinks](/content/seo-spider/user-guide/configuration/#internal-hyperlinks/index.html) - [External links](/content/seo-spider/user-guide/configuration/#check-external-links/index.html) - [Canonicals](/content/seo-spider/user-guide/configuration/#crawl-canonicals/index.html) - [Pagination (rel next/prev)](/content/seo-spider/user-guide/configuration/#crawl-nextprev/index.html) - [Hreflang](/content/seo-spider/user-guide/configuration/#extract-hreflang/index.html) - [AMP](/content/seo-spider/user-guide/configuration/#extract-amp-links/index.html) - [Meta refresh](/content/seo-spider/user-guide/configuration/#meta-refresh/index.html) - [iframes](/content/seo-spider/user-guide/configuration/#iframes/index.html) - [Mobile alternate](/content/seo-spider/user-guide/configuration/#mobile-alternate/index.html) - [Uncrawlable Links](/content/seo-spider/user-guide/configuration/#uncrawlable-links/index.html) - [Check links outside of start folder](/content/seo-spider/user-guide/configuration/#check-links-outside-of-start-folder/index.html) - [Crawl outside of start folder](/content/seo-spider/user-guide/configuration/#crawl-outside-of-start-folder/index.html) - [Crawl all subdomains](/content/seo-spider/user-guide/configuration/#crawl-all-subdomains/index.html) - [Follow internal or external ‘nofollow’](/content/seo-spider/user-guide/configuration/#follow-internal-or-external-nofollow/index.html) - [Crawl invalid links](/content/seo-spider/user-guide/configuration/#crawl-invalid-links/index.html) - [Crawl linked XML sitemaps](/content/seo-spider/user-guide/configuration/#crawl-linked-xml-sitemaps/index.html) ### Spider Extraction Tab - [Page details](/content/seo-spider/user-guide/configuration/#page-details/index.html) - [URL details](/content/seo-spider/user-guide/configuration/#url-details/index.html) - [Directives](/content/seo-spider/user-guide/configuration/#directives-2/index.html) - [Structured data](/content/seo-spider/user-guide/configuration/#structured-data-2/index.html) - [HTML](/content/seo-spider/user-guide/configuration/#html/index.html) - [PDF](/content/seo-spider/user-guide/configuration/#pdf/index.html) ### Spider Limits Tab - [Limit crawl total](/content/seo-spider/user-guide/configuration/#limit-search-total-2/index.html) - [Limit crawl depth](/content/seo-spider/user-guide/configuration/#limit-search-depth/index.html) - [Limit URLs per crawl depth](/content/seo-spider/user-guide/configuration/#limit-urls-per-crawl-depth/index.html) - [Limit max folder depth](/content/seo-spider/user-guide/configuration/#limit-max-folder-depth/index.html) - [Limit number of query strings](/content/seo-spider/user-guide/configuration/#limit-number-of-query-strings/index.html) - [Limit crawl total per subdomain](/content/seo-spider/user-guide/configuration/#limit-crawl-total-per-subdomain/index.html) - [Max redirects to follow](/content/seo-spider/user-guide/configuration/#max-redirects-to-follow/index.html) - [Limit max URL length to crawl](/content/seo-spider/user-guide/configuration/#limit-max-url-length-to-crawl/index.html) - [Max links per URL to crawl](/content/seo-spider/user-guide/configuration/#max-links-per-url-to-crawl/index.html) - [Max page size (kb) to crawl](/content/seo-spider/user-guide/configuration/#max-page-size-kb-to-crawl/index.html) - [Limit by URL path](/content/seo-spider/user-guide/configuration/#limit-by-url-path/index.html) ### Spider Rendering Tab - [Rendering](/content/seo-spider/user-guide/configuration/#rendering/index.html) - [Rendered page screen shots](/content/seo-spider/user-guide/configuration/#rendered-page-screen-shots/index.html) - [JavaScript error reporting](/content/seo-spider/user-guide/configuration/#javascript-error-reporting/index.html) - [Flatten Shadow DOM](/content/seo-spider/user-guide/configuration/#flatten-shadow-dom/index.html) - [Flatten iframes](/content/seo-spider/user-guide/configuration/#flatten-iframes/index.html) - [Archive website](/content/seo-spider/user-guide/configuration/#enable-website-archive/index.html) - [AJAX timeout](/content/seo-spider/user-guide/configuration/#ajax-timeout/index.html) - [Window size](/content/seo-spider/user-guide/configuration/#window-size/index.html) ### Spider Advanced Tab - [Cookie storage](/content/seo-spider/user-guide/configuration/#cookie-storage/index.html) - [Ignore non-indexable URLs for Issues](/content/seo-spider/user-guide/configuration/#ignore-non-indexable-urls-for-on-page-filters/index.html) - [Ignore paginated URLs for duplicate filters](/content/seo-spider/user-guide/configuration/#ignore-paginated-urls-for-duplicate-filters/index.html) - [Always follow redirects](/content/seo-spider/user-guide/configuration/#always-follow-redirects/index.html) - [Always follow canonicals](/content/seo-spider/user-guide/configuration/#always-follow-canonicals/index.html) - [Respect noindex](/content/seo-spider/user-guide/configuration/#respect-noindex/index.html) - [Respect canonical](/content/seo-spider/user-guide/configuration/#respect-canonical/index.html) - [Respect next/prev](/content/seo-spider/user-guide/configuration/#respect-nextprev/index.html) - [Respect HSTS policy](/content/seo-spider/user-guide/configuration/#respect-hsts-policy/index.html) - [Respect self referencing meta refresh](/content/seo-spider/user-guide/configuration/#respect-self-referencing-meta-refresh/index.html) - [Extract images from img srcset attribute](/content/seo-spider/user-guide/configuration/#extract-images-from-img-srcset-attribute/index.html) - [Crawl fragment identifiers](/content/seo-spider/user-guide/configuration/#crawl-fragment-identifiers/index.html) - [Perform HTML validation](/content/seo-spider/user-guide/configuration/#perform-html-validation/index.html) - [Green hosting carbon calculation](/content/seo-spider/user-guide/configuration/#green-hosting-carbon-calculation/index.html) - [Assume pages are HTML](/content/seo-spider/user-guide/configuration/#assume-pages-are-html/index.html) - [Response timeout](/content/seo-spider/user-guide/configuration/#response-timeout/index.html) - [5XX response retries](/content/seo-spider/user-guide/configuration/#xx-response-retries/index.html) ### Spider Preferences Tab - [Page title & meta description width](/content/seo-spider/user-guide/configuration/#page-title-meta-description-width/index.html) - [Links preferences](/content/seo-spider/user-guide/configuration/#links-preferences/index.html) - [Other character preferences](/content/seo-spider/user-guide/configuration/#other/index.html) ### Other Configuration Options - [Content area](/content/seo-spider/user-guide/configuration/#content-area/index.html) - [Duplicates](/content/seo-spider/user-guide/configuration/#duplicates/index.html) - [Spelling & grammar](/content/seo-spider/user-guide/configuration/#spelling-grammar/index.html) - [Embeddings](/content/seo-spider/user-guide/configuration/#embeddings/index.html) - [Robots.txt](/content/seo-spider/user-guide/configuration/#ignore-robots-txt/index.html) - [URL rewriting](/content/seo-spider/user-guide/configuration/#url-rewriting/index.html) - [CDNs](/content/seo-spider/user-guide/configuration/#cdn/index.html) - [Include](/content/seo-spider/user-guide/configuration/#include/index.html) - [Exclude](/content/seo-spider/user-guide/configuration/#exclude/index.html) - [Speed](/content/seo-spider/user-guide/configuration/#speed/index.html) - [User agent](/content/seo-spider/user-guide/configuration/#user-agent-2/index.html) - [HTTP header](/content/seo-spider/user-guide/configuration/#http-header/index.html) - [Custom search](/content/seo-spider/user-guide/configuration/#custom-search/index.html) - [Custom extraction](/content/seo-spider/user-guide/configuration/#custom-extraction/index.html) - [Custom link positions](/content/seo-spider/user-guide/configuration/#custom-link-positions/index.html) - [Custom JavaScript](/content/seo-spider/user-guide/configuration/#custom-javascript/index.html) - [Google Analytics integration](/content/seo-spider/user-guide/configuration/#google-analytics-integration/index.html) - [Google Search Console integration](/content/seo-spider/user-guide/configuration/#google-search-console-integration/index.html) - [PageSpeed Insights integration](/content/seo-spider/user-guide/configuration/#pagespeed-insights-integration/index.html) - [Majestic](/content/seo-spider/user-guide/configuration/#majestic/index.html) - [Ahrefs](/content/seo-spider/user-guide/configuration/#ahrefs/index.html) - [Moz](/content/seo-spider/user-guide/configuration/#moz/index.html) - [OpenAI](/content/seo-spider/user-guide/configuration/#openai/index.html) - [Gemini](/content/seo-spider/user-guide/configuration/#gemini/index.html) - [Ollama](/content/seo-spider/user-guide/configuration/#ollama/index.html) - [Anthropic](/content/seo-spider/user-guide/configuration/#anthropic/index.html) - [Authentication](/content/seo-spider/user-guide/configuration/#authentication/index.html) - [Segments](/content/seo-spider/user-guide/configuration/#segments/index.html) - [Crawl analysis](/content/seo-spider/user-guide/configuration/#crawl-analysis/index.html) - [User Interface](/content/seo-spider/user-guide/configuration/#user-interface/index.html) - [Language](/content/seo-spider/user-guide/configuration/#language/index.html) - [Proxy](/content/seo-spider/user-guide/configuration/#proxy/index.html) - [Storage mode](/content/seo-spider/user-guide/configuration/#storage/index.html) - [Memory allocation](/content/seo-spider/user-guide/configuration/#memory-config/index.html) - [Crawl Retention](/content/seo-spider/user-guide/configuration/#crawl-retention/index.html) - [Trusted Certificates](/content/seo-spider/user-guide/configuration/#trusted-certificates/index.html) - [Notifications](/content/seo-spider/user-guide/configuration/#notifications/index.html) - [MCP Server](/content/seo-spider/user-guide/configuration/#mcp-server/index.html) - [Mode](/content/seo-spider/user-guide/configuration/#mode/index.html) ## Tabs ### Top Tabs - [Internal](/content/seo-spider/user-guide/tabs/#internal/index.html) - [External](/content/seo-spider/user-guide/tabs/#external/index.html) - [Security](/content/seo-spider/user-guide/tabs/#security/index.html) - [Response Codes](/content/seo-spider/user-guide/tabs/#response-codes/index.html) - [URL](/content/seo-spider/user-guide/tabs/#uri/index.html) - [Page titles](/content/seo-spider/user-guide/tabs/#page-titles/index.html) - [Meta description](/content/seo-spider/user-guide/tabs/#meta-description/index.html) - [Meta keywords](/content/seo-spider/user-guide/tabs/#meta-keyword/index.html) - [h1](/content/seo-spider/user-guide/tabs/#h1/index.html) - [h2](/content/seo-spider/user-guide/tabs/#h2/index.html) - [Content](/content/seo-spider/user-guide/tabs/#content/index.html) - [Images](/content/seo-spider/user-guide/tabs/#images/index.html) - [Canonicals](/content/seo-spider/user-guide/tabs/#canonicals/index.html) - [Pagination](/content/seo-spider/user-guide/tabs/#pagination/index.html) - [Directives](/content/seo-spider/user-guide/tabs/#directives/index.html) - [hreflang](/content/seo-spider/user-guide/tabs/#hreflang/index.html) - [JavaScript](/content/seo-spider/user-guide/tabs/#javascript/index.html) - [Links](/content/seo-spider/user-guide/tabs/#links/index.html) - [AMP](/content/seo-spider/user-guide/tabs/#amp/index.html) - [Structured data](/content/seo-spider/user-guide/tabs/#structured-data/index.html) - [Sitemaps](/content/seo-spider/user-guide/tabs/#sitemaps/index.html) - [PageSpeed](/content/seo-spider/user-guide/tabs/#pagespeed/index.html) - [Mobile](/content/seo-spider/user-guide/tabs/#mobile/index.html) - [Accessibility](/content/seo-spider/user-guide/tabs/#accessibility/index.html) - [Custom search](/content/seo-spider/user-guide/tabs/#custom/index.html) - [Custom extraction](/content/seo-spider/user-guide/tabs/#custom-extraction-tab/index.html) - [Custom JavaScript](/content/seo-spider/user-guide/tabs/#custom-javascript-tab/index.html) - [Analytics](/content/seo-spider/user-guide/tabs/#analytics/index.html) - [Search Console](/content/seo-spider/user-guide/tabs/#search-console/index.html) - [Validation](/content/seo-spider/user-guide/tabs/#validation/index.html) - [Link Metrics](/content/seo-spider/user-guide/tabs/#link-metrics/index.html) - [AI](/content/seo-spider/user-guide/tabs/#ai/index.html) - [Change Detection](/content/seo-spider/user-guide/tabs/#change-detection/index.html) ### Tutorials - [Getting Started Guide](/content/seo-spider/tutorials/quick-start-guide/index.html) - [How To Find Broken Links Using The SEO Spider](/content/seo-spider/tutorials/broken-link-checker/index.html) - [Site Architecture & Crawl Visualisations Guide](/content/seo-spider/tutorials/site-architecture-crawl-visualisations/index.html) - [How To Compare Crawls](/content/seo-spider/tutorials/how-to-compare-crawls/index.html) - [How To Crawl JavaScript Websites](/content/seo-spider/tutorials/crawl-javascript-seo/index.html) - [How To Crawl Large Websites](/content/seo-spider/tutorials/how-to-crawl-large-websites/index.html) - [How To Crawl A Staging Website](/content/seo-spider/tutorials/how-to-crawl-a-staging-website/index.html) - [How To Automate Crawl Reports In Data Studio](/content/seo-spider/tutorials/how-to-automate-crawl-reports-in-data-studio/index.html) - [How To Audit Core Web Vitals](/content/seo-spider/tutorials/how-to-audit-core-web-vitals/index.html) - [How to Identify Semantically Similar Pages & Outliers](/content/seo-spider/tutorials/how-to-identify-semantically-similar-pages-outliers/index.html) - [How to Run the Screaming Frog SEO Spider in the Cloud](/content/seo-spider/tutorials/seo-spider-cloud/index.html) - [How To Crawl With AI Prompts](/content/seo-spider/tutorials/how-to-crawl-with-ai-prompts/index.html) - [How To Check For Duplicate Content](/content/seo-spider/tutorials/how-to-check-for-duplicate-content/index.html) - [How To Set Up Crawl Email Notifications & Export Attachments](/content/seo-spider/tutorials/how-to-set-up-crawl-email-notifications-export-attachments/index.html) - [How To Perform A Web Accessibility Audit](/content/seo-spider/tutorials/how-to-perform-a-web-accessibility-audit/index.html) - [How To Find Missing Image Alt Text & Attributes](/content/seo-spider/tutorials/how-to-find-missing-image-alt-text/index.html) - [Web Scraping & Custom Extraction](/content/seo-spider/tutorials/web-scraping/index.html) - [Internal Linking Audit With the SEO Spider](/content/seo-spider/tutorials/internal-linking-audit-with-the-seo-spider/index.html) - [How to Work in Teams Using the SEO Spider](/content/seo-spider/tutorials/seo-spider-working-in-teams/index.html) - [How to Crawl with ChatGPT](/content/seo-spider/tutorials/how-to-crawl-with-chatgpt/index.html) - [How To Test & Validate Structured Data](/content/seo-spider/tutorials/structured-data-testing-validation/index.html) - [How To Use Vector Embeddings for Redirect Mapping](/content/seo-spider/tutorials/how-to-use-vector-embeddings-for-redirect-mapping/index.html) - [How To Use List Mode](/content/seo-spider/tutorials/how-to-use-list-mode/index.html) - [Spell & Grammar Check Your Website](/content/seo-spider/tutorials/spelling-grammar-checker/index.html) - [How To Audit Redirects In A Site Migration Using The SEO Spider](/content/seo-spider/tutorials/audit-redirects/index.html) - [How To Debug Missing Pages In A Crawl](/content/seo-spider/tutorials/how-to-debug-missing-pages-in-a-crawl/index.html) - [How To Use N-Grams](/content/seo-spider/tutorials/how-to-use-n-grams/index.html) - [How To Audit Canonicals](/content/seo-spider/tutorials/how-to-audit-canonicals/index.html) - [How To Audit Mobile Usability](/content/seo-spider/tutorials/how-to-audit-mobile-usability/index.html) - [How To Audit & Validate Accelerated Mobile Pages (AMP)](/content/seo-spider/tutorials/how-to-audit-validate-amp/index.html) - [How To Audit Hreflang](/content/seo-spider/tutorials/how-to-audit-hreflang/index.html) - [How To Audit rel="next" and rel="prev" Pagination Attributes](/content/seo-spider/tutorials/how-to-audit-pagination/index.html) - [How To Audit XML Sitemaps](/content/seo-spider/tutorials/how-to-audit-xml-sitemaps/index.html) - [How To Bulk Check Redirects](/content/seo-spider/tutorials/redirect-checker/index.html) - [How To Find Orphan Pages](/content/seo-spider/tutorials/find-orphan-pages/index.html) - [How To Use Custom Search](/content/seo-spider/tutorials/how-to-use-custom-search/index.html) - [HTTP Status Codes – Why Won’t My Website Crawl?](/content/seo-spider/tutorials/http-status-codes-when-crawling/index.html) - [Robots.txt Testing In The SEO Spider](/content/seo-spider/tutorials/robots-txt-tester/index.html) - [What Is Link Score?](/content/seo-spider/tutorials/link-score/index.html) - [XML Sitemap Generator](/content/seo-spider/tutorials/xml-sitemap-generator/index.html) - [How To Use The SEO Spider In A Site Migration](/content/seo-spider/tutorials/how-to-use-the-seo-spider-in-a-site-migration/index.html) - [How To Bypass Geo IP Redirection In A Crawl](/content/seo-spider/tutorials/bypass-geo-ip-redirection/index.html) - [How To Perform A Cookie Audit](/content/seo-spider/tutorials/how-to-perform-a-cookie-audit/index.html) - [How To Find Broken Bookmarks](/content/seo-spider/tutorials/find-broken-bookmarks/index.html) - [How To Analyse Link Position](/content/seo-spider/tutorials/how-to-analyse-link-position/index.html) - [How To Perform A Parity Audit](/content/seo-spider/tutorials/how-to-perform-a-parity-audit/index.html) - [How To Automate The URL Inspection API](/content/seo-spider/tutorials/how-to-automate-the-url-inspection-api/index.html) - [How To Test Readability](/content/seo-spider/tutorials/how-to-test-readability/index.html) - [How To Audit PDFs](/content/seo-spider/tutorials/how-to-audit-pdfs/index.html) - [How To Use The SEO Spider For Broken Link Building](/content/seo-spider/tutorials/broken-link-building/index.html) - [How To Debug Invalid HTML Elements In The Head](/content/seo-spider/tutorials/how-to-debug-invalid-html-elements-in-the-head/index.html) - [An SEOs Guide To Crawling HSTS](/content/seo-spider/tutorials/an-seos-guide-to-crawling-hsts/index.html) - [Configure X Virtual Framebuffer](/content/seo-spider/tutorials/configure-x-virtual-framebuffer/index.html) - [Resolving Google Analytics / Google Search Console Connection Issues](/content/seo-spider/tutorials/resolving-google-analytics-google-search-console-connection-issues/index.html) - [Crawling Password Protected Websites](/content/seo-spider/tutorials/crawling-password-protected-websites/index.html) - [How to Debug Custom JavaScript Snippets](/content/seo-spider/tutorials/how-to-debug-custom-javascript-snippets/index.html) - [Simulating User Interactions with Custom JavaScript](/content/seo-spider/tutorials/simulating-user-interactions-with-custom-javascript/index.html)