How to Monitor AI Bots in the Log File Analyser

Introduction

Log file analysis is often overlooked when it comes to campaign strategy and is generally considered to be on the more advanced side of technical SEO. However, log files are an absolute gold mine of information and data, especially during an age where we’re seeing an explosion of LLMs and their AI bots.

While it may look intimidating, log file analysis isn’t as hard as it seems. The Screaming Frog Log File Analyser helps you to easily import your logs, verify search engine and AI bots, and analyse their behaviour.

What Are Log Files?

Before we dive into tracking AI bots, let’s quickly cover the basics. There are several types of log files that servers generate, including error logs, security logs, and access logs. For monitoring AI bot activity, we’re interested in access logs.

Access logs record every single request made to your website. They capture who visited, what they requested, when they requested it and how the server responded.

Each line in an access log represents a single request, containing information like:

• The IP address of the visitor
• The timestamp of the request
• The requested URL
• The HTTP status code that the server returned
• The user agent string (this is important, as it identifies the browser or bot)
• The referrer URL

While the raw data may look overwhelming, it’s incredibly valuable. You can see exactly how users, search engines and bots are engaging with your site at a very granular level, allowing you to carry out many different types of analysis.

How Can Log Files Be Used for Tracking AI Bots

Traditionally, log files are used to analyse search engine bot behaviour, looking at things such as response codes, crawl frequency, crawl budgets, and more. They also give you visibility into AI bot activity that you can’t find anywhere else.

Most AI chat and LLM platforms, such as Perplexity and OpenAI, have different bots for different purposes. The Screaming Frog Log File Analyser currently includes several presets for most major platforms:

As well as this, you’re also able to add your own custom user-agents.

This level of bot granularity unlocks even more insights, allowing you to identify what content is being used for model training, search/indexing or being surfaced in real-time to users within a chat.

Take a look at our introduction to log files for more information, as well as our guide to requesting logs from a server administrator.

1) Importing Your Logs into the Log File Analyser

Creating a new project and importing your logs is as simple as dragging and dropping them into the Log File Analyser.

When you do, you’ll be asked to create a new project.

You can click the ‘User Agents’ tab to configure the user-agents analysed in the project. By default, the Log File Analyser only analyses search engine bot events, so the ‘Filter User Agents (Improves Performance)’ box is ticked.

You can use the filters to select just AI Bots, either moving them all over or handpicking the bots of interest.

For this tutorial, we’re going to select the AI User-agents and use the arrow to move them across to the ‘Selected’ pane.

You’re also able to tick ‘Verify Bots When Importing Logs (Slows down Import)’.

Search engine bots are often spoofed, and this performs a lookup against publicly confirmed IP lists to confirm they are genuine.

2) Analysing the Data

Once the import is complete, you’ll see the above Overview tab. This provides a summary of the imported log file data based on the chosen time period and the user-agent(s) selected.

The Overview shows key metrics like total events, unique URLs crawled, average bytes per request, and response time data. The charts visualise crawl activity over time, letting you quickly spot patterns in how AI bots are interacting with your site.

You can filter by specific bots and change the dates using the dropdowns in the top right.

Understanding the Tabs

The Log File Analyser organises data across several tabs, each providing a different view of bot activity:

• URLs Tab – Shows every unique URL discovered in your logs, with metrics like request frequency, response codes, bytes transferred, and server response times per URL.
• Response Codes Tab – Breaks down HTTP status codes (2XX, 3XX, 4XX, 5XX) for each URL, making it easy to identify where bots are encountering issues.
• User Agents Tab – Aggregates data by bot type, showing total requests, unique URLs accessed, error rates, and response code distributions for each crawler.
• Referers Tab – Shows where traffic originated from based on the HTTP Referer header.
• Directories Tab – Aggregates data by directory path, revealing which sections of your site structure AI bots are exploring.
• IPs Tab – Shows activity by IP address, useful for investigating suspicious traffic or verifying bot authenticity.
• Countries Tab – Displays the geographic distribution of bot traffic based on IP addresses.
• Events Tab – The raw log data showing every individual request with all available attributes.

3) What to Look For

Response Codes and Errors

A good starting point is double-checking that AI bots are successfully accessing your content. High numbers of 4XX or 5XX errors indicate problems. For citation bots like ChatGPT-User or Perplexity-User, every error response is a missed opportunity.

Lastly, check the Response Codes tab for inconsistent responses. If the same URL returns 200 one day and 404 the next, investigate why.

Most Visited URLs

Sort the URLs tab by ‘Num Events’ ((total requests per URL) to see which pages AI bots are accessing most frequently.

You might find:

• Blog posts and guides getting heavy traffic from training bots
• Product pages being prioritised by citation bots

Bandwidth and Carbon Impact

The Total Bytes column shows how much data each AI bot is consuming. The Log File Analyser automatically calculates carbon footprint using the CO2.js library.

Aggressive Crawling Patterns

Check the Events chart on the Overview tab for unusual spikes in bot activity. Legitimate crawlers typically maintain steady, predictable patterns.

Crawl Depth and Coverage

Use the Directories tab to see how deeply AI bots are exploring your site structure.

4) Combining Crawl Data with Log File Analysis

For deeper insights, you can import data from a Screaming Frog SEO Spider crawl.

Importing SEO Spider Data

Summary

This guide should help you use the Log File Analyser to monitor how AI bots interact with your website, from importing logs and understanding the data, to identifying patterns and combining crawl data for deeper insights.