Site Architecture & Crawl Visualisations Guide

Introduction To Visualisations

The Screaming Frog SEO Spider has two types of interactive website visualisations – crawl visualisations, and directory tree visualisations.

The two types of visualisations are fundamentally different and are useful in understanding a site’s architecture in different ways. This guide will explain the differences and highlight how each can be utilised.

Each visualisation type also has 3 different formats:

• Force-Directed Diagram.
• 3D Force-Directed Diagram.
• Tree Graph.

The choice of format type can simply be down to personal taste, but the ‘best’ format to use can also depend on the size of the site, and situation.

While visualisations are beautiful and fun, they don’t provide any more data than is already available in a crawl. They are often not the best way to diagnose issues and don’t always tell the whole story.

However, the strength of visualisations is that they can help provide perspective, communicate ideas, or reveal underlying patterns that are harder to uncover in data, and spreadsheets.

Crawl Visualisations

The ‘force-directed crawl diagram’, ‘3D force-directed crawl diagram’ and ‘crawl tree graph’ visualisations provide a view of how the SEO Spider has crawled the site, by shortest path to a page.

If a page has multiple shortest paths (i.e, multiple links from the same lowest depth), the link that will be displayed is the one that was crawled first, often the first discovered in the source code.

The crawl visualisations are hierarchical by crawl depth, and the lines between URLs represent the shortest path. This makes the crawl visualisations useful when analysing site architecture, and internal linking.

Force-Directed Crawl Diagrams

The force-directed crawl diagram and 3D force-directed crawl diagram are like a heat-map, with the start URL represented by the darkest green, largest node (the circles) in the middle.

This is generally the homepage if you started the crawl there. The lines (known as ‘edges’) represent the link between one URL and another (by shortest path).

Indexable pages are represented by the green nodes, while the pastel red highlights URLs that are non-indexable. This makes it quite easy to spot problematic sections or pages of a website.

The ‘Information’ (i) menu icon provides a key with more information about the colours of nodes.

The 2D visualisation will show up to 10k URLs in the browser, but allow you to right-click and ‘focus’ to expand on particular areas of a site to show more URLs in that section (up to another 10k URLs at a time). You can use the browser as navigation, typing in a URL directly and moving forwards and backwards with ease.

When a visualisation has reached the 10k URL limit, it lets you know when a particular node has children that are being truncated, by colouring the nodes grey. You can then right-click and ‘explore’ to see the children.

The 3D force-directed crawl diagram will open in your default browser, rather than within our inbuilt browser.

You’re able to adjust node colours to be based upon response codes, or segments as well.

Crawl Tree Graph

The tree graph format visually is quite different to the force-directed crawl diagram, but fundamentally they represent websites in a similar way, with URLs represented by the circles and the shortest path by hyperlink, as the lines connecting them.

The crawl tree graphs will also show up to 10k URLs in the browser, but allow you to right-click and ‘focus’ to expand on particular areas of a site to show more URLs in that section (up to another 10k URLs at a time).

Directory Tree Visualisations

The ‘force-directed directory tree diagram’, ‘3D force-directed directory tree diagram’ and ‘directory tree graph’ visualisations show the URL architecture of a website.

The directory tree visualisations are hierarchical by URL component and path, and the lines between URLs represent the directory path. This makes the directory tree visualisations useful when analysing URL structure and general information architecture of a website.

Force-Directed Directory Tree Diagram

The force-directed directory tree diagrams are like a heat-map, with the protocol(s) represented by the darkest green, largest node (the circles) generally hanging from the very central node which is the homepage (the hostname).

The red nodes under the author directory, are all author pages, which are non-indexable with a meta ‘noindex’ tag.

As the directory tree view groups pages together by path which often share the same template, it often makes it easier to see these types of patterns and get an understanding of general, broad structure.

Directory Tree Graph

The directory tree graph format represents websites in a similar way to the force-directed directory tree diagram, with the URL components represented by the nodes, and the lines connecting them the URL path.

Displaying ‘All Links’ in Force-Directed Diagrams

Right-click on a node and select to ‘Show Inlinks’, ‘Show Inlinks to Children’ of a node, or perform the same for ‘Outlinks’.

This will update the visualisation to show all incoming links, which can be useful when analysing internal linking to a page or a section of a website.

Scaling Force-Directed Diagrams

Both the ‘force-directed crawl diagram’ and ‘force-directed directory tree diagram’ can be scaled by different metrics (by clicking on the cog icon).

However, both diagrams can be scaled by other items such as – unique inlinks, Link Score (PageRank), or external metrics like Moz Page Authority or Ahrefs URL Rating for example.

This means diagrams can be configured to visualise which URLs on a website have the highest authority, by using Ahrefs URL Rating, as an example.

Sharing Visualisations

To share a visualisation, you can click the ‘save’ file icon in the top right-hand corner of the visualisation.