---
title: "Teardown: 30 Days of AI Crawler Logs Show Which Pages Get Fetched"
description: "A teardown of groas’s edge logs: 13,294 claimed AI bot hits on robots.txt, 3,730 verified requests, and one specific post that drew 330 live-answer fetches."
url: "https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/fae21441-cc0b-457f-834d-e523255cadbb.png"
published: "2026-10-15T05:32:45.778Z"
modified: "2026-10-15T05:32:45.848Z"
---

October 15, 2026 · 10 min read

# Teardown: 30 Days of AI Crawler Logs Show Which Pages Get Fetched

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Model town: three grand marble halls get inspected but never entered, while a tiny newsstand selling one dated paper is mobbed by couriers.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/fae21441-cc0b-457f-834d-e523255cadbb.png)

In this article

1. [The artefact: 30 days at the edge](#the-artefact-30-days-at-the-edge)
2. [Line 1: /robots.txt exposes the raw-count problem](#line-1-robotstxt-exposes-the-raw-count-problem)
3. [Line 2: the homepage gets fetched; the blog index barely does](#line-2-the-homepage-gets-fetched-the-blog-index-barely-does)
4. [Line 3: one dated post beats three category hubs on live retrieval](#line-3-one-dated-post-beats-three-category-hubs-on-live-retrieval)
5. [Line 4: a missing fetch is not a diagnosis](#line-4-a-missing-fetch-is-not-a-diagnosis)
6. [What I would keep from this teardown](#what-i-would-keep-from-this-teardown)

Our `/robots.txt` drew 13,294 requests claiming to be AI bots in 30 days. Only 3,730 passed IP verification. That gap is why I chose this artefact for a teardown: an edge server log export from groas.com, showing what reached our site before any dashboard tried to turn the traffic into an AI-visibility score.

The more useful finding sits further down the file. Three broad category hubs drew roughly 240 verified crawler fetches each and **zero live-answer fetches**. One dated post, `google-ads-updates-2026`, drew **330 verified live-answer fetches** from ChatGPT and Claude. If I had judged those pages by raw bot traffic or by how much work went into their structure, I would have missed the distinction that matters.

Marketing teams trying to appear in ChatGPT and Perplexity answers often start with a third-party dashboard. Those tools run synthetic prompts, check whether a brand appears, and identify potential content gaps. That can show you what happened in those tests. It cannot show which bots actually requested your pages. For that, I start at the edge. This teardown follows the export in the order a reader encounters it: the bot claims, the homepage, the blog index, the category hubs, the specific post, and finally the pages with no live-answer traffic at all.

## The artefact: 30 days at the edge

The export covers 30 consecutive days of HTTP requests to groas.com. It records requests declaring AI crawler or retrieval user-agents, then separates them into raw hits, verified hits, and live-answer hits. **Those are three different measurements**, not three names for AI citations.

Standard web analytics will miss much of this activity. As [Lemniscate Growth’s crawler-log analysis](https://lemniscategrowth.com/blogs/ai-crawler-log-file-analysis.html) explains, crawlers do not behave like visitors who execute client-side JavaScript and generate ordinary analytics sessions. A page can be fetched without appearing as a session in your usual marketing report. The access log tells you about the request; it does not tell you everything the requesting system subsequently did with the page.

### Three columns, three different claims

- **Raw hits** count requests carrying a declared AI user-agent, such as `GPTBot`. A user-agent is a header the requester supplies. It is easy to imitate.
- **Verified hits** count requests whose connecting IP matches the vendor’s published ranges. [OpenAI publishes IP prefix files](https://geoiphub.com/blog/gptbot-vs-oai-searchbot-vs-chatgpt-user) for its bots; the export uses the relevant published ranges to check claimed identities.
- **Live-answer hits** are verified requests attributed in this export to on-demand agents such as `ChatGPT-User` and `Claude-User`. They show that an agent fetched a URL in a live-retrieval context. They do **not**, by themselves, show the final answer, a clickable citation, or a resulting sale.

That last boundary matters throughout this teardown. I can compare which pages the agents requested. I cannot read an access-log line and declare that the page appeared in an answer. **Verdict on the columns: use them to narrow the question, not to skip straight to a citation claim.**

## Line 1: `/robots.txt` exposes the raw-count problem

The first entry worth stopping at is `/robots.txt`. Over the 30-day window, it received **13,294 requests declaring AI crawler user-agents**. That is an impressive number if the dashboard stops counting there. It becomes less impressive when I check the connecting IPs against the vendors’ published ranges: **3,730 requests verified; 9,564 did not**.

![Diagram comparing 13,294 claimed AI bot hits on robots.txt with 3,730 verified and 9,564 unverified requests.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/4b934d14-03fd-41cd-9422-faed0d510189.png)

An unverified request is not proof of a malicious scanner. It is proof that the declared bot identity did not pass this check. Some may be scrapers or other automated traffic using a borrowed user-agent; the log count alone does not classify every requester. [WebDecoy’s investigation of unverified crawler traffic](https://webdecoy.com/blog/which-ai-engines-read-your-site-verified/) describes why bot-name spoofing is a real issue, while [Centinel Analytica’s ClaudeBot tracking](https://www.centinelanalytica.com/bots/claudebot) reports requests claiming that identity which fail IP verification. Our own line makes the practical point without requiring a theory about every one of the 9,564 requests.

If an AI-monitoring report calls all 13,294 visits from authentic AI systems, its starting figure is wrong. And even the 3,730 verified requests are visits to a crawling-policy file, not evidence that a model read our content or cited groas. **Verdict on line 1: authenticate the requester before you interpret the volume.** A large raw count on `/robots.txt` is not an AI-presence strategy.

## Line 2: the homepage gets fetched; the blog index barely does

The next contrast is between `/` and `/blog`. In this export, **the homepage recorded 834 `ChatGPT-User` live-answer fetches; the main blog index recorded 4**. Both also received verified background crawling. The difference appeared when the on-demand agent selected a URL to fetch.

The homepage contains the core offer, pricing structure, customer case study metrics, and service architecture. The blog index is a directory of cards and excerpts. If the retrieval task concerns groas itself, the homepage gives the agent more material to work with in one request. The index asks it to choose another link before it reaches a complete article. That is a plausible explanation for the pattern, not a record of the prompts that caused it.

![Bar chart comparing 834 live-answer fetches on the homepage with 4 on the blog index and zero across three category hubs.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/571c5cdb-ba6a-4ff2-a90e-84333ee45112.png)

[OpenAI’s crawler documentation](https://developers.openai.com/api/docs/bots) distinguishes its bot roles, which is one reason I would not roll background indexing and `ChatGPT-User` requests into a single ‘AI visits’ chart. A background crawl can help a system discover pages. An on-demand fetch is a different event. Neither tells me, on its own, whether the final answer linked to the page.

**Verdict on line 2: the homepage is doing more work than the blog directory in this window.** I would keep the homepage’s direct, substantive answers. I would not pour effort into the index merely because it is the front door to the articles for a human reader.

## Line 3: one dated post beats three category hubs on live retrieval

This is the line that makes the export worth keeping. The three category hubs cover broad topics including search engine marketing, agency workflows, and Google Ads bidding. Each drew about **240 verified crawler fetches** during the window. Across all three, the export records **zero live-answer fetches** from ChatGPT and Claude. Meanwhile, `google-ads-updates-2026`, a specific post about recent Google Ads changes, recorded **330 verified live-answer fetches**.

I would not translate that into ‘330 citations.’ The log cannot support it. But I would not dismiss the contrast, either. The hubs were visited by verified crawlers, so a simple ‘the bots never found them’ explanation does not fit these lines. The dated post was fetched repeatedly in the live-answer category; the hubs were not.

### What the specific post offers that the hubs do not

The category pages organise links and summarise subjects. That is useful site architecture. It is less useful as a single document answering a narrow question. The updates post contains dates, named changes, bidding details, and cost-per-acquisition discussion. A request for recent Google Ads changes has a more direct match there than on a hub that points readers toward several other pages.

[DataDome’s description of Claude-User](https://datadome.co/bots/claude-user/) is a useful reminder that a user-initiated request can target a particular URL. It does not promise that an agent will follow every link on a category page to assemble a better answer. Our export cannot expose the agent’s selection process, but it shows the result at our edge: the specific document received live fetches, and the broad hubs did not.

![Technical cutaway showing a live AI retriever extracting a self-contained passage from a webpage rather than its generic introduction.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/cee70f77-c11a-4b8e-892a-c9212ce1e82b.png)

This is not an argument for deleting category pages. They still organise the site for people and crawlers. It is an argument against treating a well-structured hub as a substitute for a document that answers something. [Ten Point Labs’ review of Ahrefs search data](https://tenpointlabs.com/post/ai-seo-competitor-gap-analysis) makes a related point about AI Overview citations not being confined to the top ten organic results. I would not use that separate dataset to explain our individual requests. The log gives me a narrower, sturdier observation: **verified discovery did not translate into live retrieval for these hubs**.

**Verdict on line 3: keep the hubs as navigation; put the answer on the page you want fetched.** In this window, the dated, specific post gave live agents a reason to request it that the broad category pages did not.

## Line 4: a missing fetch is not a diagnosis

Further down the export, dozens of thought-leadership articles show **zero `ChatGPT-User` or `Claude-User` live-answer hits**. This is where a dashboard can make a routine absence sound like an emergency. Zero does not tell me the technical setup is broken. It does not tell me the page was penalised, either.

Perhaps no relevant live-retrieval request reached that article during these 30 days. Perhaps an answer did not require a web fetch. Perhaps another URL was selected. The access log records requests that arrived; it does not record all the questions users asked or every reason a system chose not to request a page. That limit is especially important when the sample is one site over one month.

Commercial _Generative Engine Optimization_ tools answer a different question. As [Scalenut’s evaluation of AI visibility platforms](https://www.scalenut.com/blogs/profound-vs-peec-ai-for-geo) describes, platforms such as Profound, Peec AI, Writesonic, and Otterly use synthetic prompts to check brand appearances. A missing appearance can be useful to investigate. It cannot, by itself, tell you whether a bot met a firewall challenge, whether the relevant page was fetched, or whether that particular answer used the live web at all.

**Verdict on line 4: investigate a zero; do not invent its cause.** A prompt test and a server log can inform the same decision, but neither is a transcript of the other.

## What I would keep from this teardown

I would keep the distinction the export forces on us: **claimed bot traffic, authenticated crawling, and live-answer fetches belong in separate columns**. Then I would use those columns in order, rather than buying a content-gap list and working backward to an explanation.

A practical log review starts with the fields this export needed:

1. Capture the incoming IP, user-agent, HTTP status code, and requested path in your edge access logs.
2. Check claimed OpenAI and Anthropic identities against their published IP ranges. Set unverified requests aside before comparing pages.
3. Separate background crawler requests from `ChatGPT-User` and `Claude-User` requests. Compare URLs within each group rather than combining them into a sitewide bot total.
4. Read the requested pages. If a hub receives background crawls but no live fetches while a specific document receives many, examine what each page actually answers. If relevant bots receive an error response, examine that before rewriting the copy.

For the content itself, I would keep the updates post’s specificity: a clear subject, a date, and material a reader can use without touring the rest of the site. I would keep direct commercial answers on pages such as the homepage, too. Someone looking for a price, a service boundary, or a technical detail should not have to extract it from three paragraphs of corporate philosophy. That is good writing for humans before it is any kind of retrieval tactic.

I would **not** keep the draft conclusion that a live-answer fetch proves a citation, or that zero such fetches proves a passage problem. If a verified agent fetches a page, I know it reached that page. If it does not, I need more evidence before choosing between content, technical access, and simple lack of demand. The log makes the next question sharper. It does not answer every question for me.

That distinction is also why I prefer execution records to an impressive-looking activity total. At [groas](https://groas.com/), the operating model pairs autonomous work across paid and organic search with a named human strategist who sets direction and owns accountability. Campaign and content actions are logged with their reasoning, with attention on attributable pipeline, closed-won revenue, and cost per acquisition rather than a chart of impressive but unqualified activity. The service runs on a flat monthly fee, with no setup fees and month-to-month cancellation; the [groas MCP connection for ChatGPT and Claude](https://groas.com/mcp) lets clients query campaign, citation, and crawler activity directly.

**Overall verdict:** this edge export is far more useful than a raw ‘AI bot visits’ total, and less conclusive than a citation report. Out of 13,294 claimed AI requests to `/robots.txt`, only 3,730 verified. Further down, three crawled category hubs drew no live-answer fetches while one specific, dated post drew 330. I would keep that post’s answer-first approach and keep checking the logs. I would stop calling a request a citation.

## Frequently Asked Questions

### Why do AI bot hit counts differ from verified bot requests in server logs?

Raw hit counts include every request that declares an AI user-agent, and that header is supplied by the requester and easy to imitate. Verified hits only count requests whose connecting IP matches the vendor's published IP ranges. In one 30-day export, /robots.txt received 13,294 claimed AI bot requests but only 3,730 passed IP verification.

### Are requests claiming to be AI bots always from real AI systems?

No. An unverified request is not proof of a malicious scanner, but it means the declared bot identity did not pass the IP check against the vendor's published ranges. Some may be scrapers or other automated traffic using a borrowed user-agent, and the log alone does not classify every requester.

### Does a live-answer fetch from ChatGPT mean my page was cited in an answer?

No. A live-answer hit only shows that a verified agent such as ChatGPT-User or Claude-User fetched a URL in a live-retrieval context. It does not show the final answer, a clickable citation, or a resulting sale, so a fetch should not be counted as a citation.

### Why did the homepage get many more AI fetches than the blog index?

In the 30-day export, the homepage recorded 834 ChatGPT-User live-answer fetches while the main blog index recorded 4. The homepage contains the core offer, pricing, case study metrics, and service architecture in one document, whereas the blog index is a directory of cards that asks the agent to follow another link before reaching a complete article.

### Do category hub pages get fetched by AI agents like ChatGPT and Claude?

They can be crawled but still not fetched for live answers. Three category hubs covering search engine marketing, agency workflows, and Google Ads bidding each drew about 240 verified crawler fetches, yet across all three there were zero live-answer fetches. A specific dated post about recent Google Ads changes drew 330 verified live-answer fetches in the same window.

### How should I structure content so AI agents fetch the page directly?

Put the answer on a specific page rather than relying on a hub that links to several others. The post that drew live fetches had a clear subject, a date, named changes, bidding details, and cost-per-acquisition discussion a reader could use without visiting other pages. Keep category hubs as navigation, but treat them as a substitute for nothing.

### If my page has zero AI agent fetches in the logs, is something wrong?

Not necessarily. Zero fetches could mean no relevant live-retrieval request arrived, the answer did not require a web fetch, or another URL was selected. The access log records requests that arrived, not the questions users asked or every reason a system chose not to request a page, so investigate a zero rather than inventing its cause.

### What fields should I capture in edge logs to analyse AI crawler traffic?

Capture the incoming IP, user-agent, HTTP status code, and requested path in your edge access logs. Check claimed OpenAI and Anthropic identities against their published IP ranges and set unverified requests aside. Also separate background crawler requests from ChatGPT-User and Claude-User requests, comparing URLs within each group rather than combining them into one sitewide total.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw#webpage","url":"https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw","name":"Teardown: 30 Days of AI Crawler Logs Show Which Pages Get Fetched","description":"A teardown of groas’s edge logs: 13,294 claimed AI bot hits on robots.txt, 3,730 verified requests, and one specific post that drew 330 live-answer fetches.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/fae21441-cc0b-457f-834d-e523255cadbb.png"},"breadcrumb":{"@id":"https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw#breadcrumb"},"dateModified":"2026-10-15T05:32:45.848Z","mainEntity":{"@id":"https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Teardown: 30 Days of AI Crawler Logs Show Which Pages Get Fetched","item":"https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw#article","url":"https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw","mainEntityOfPage":{"@id":"https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Teardown: 30 Days of AI Crawler Logs Show Which Pages Get Fetched","description":"A teardown of groas’s edge logs: 13,294 claimed AI bot hits on robots.txt, 3,730 verified requests, and one specific post that drew 330 live-answer fetches.","datePublished":"2026-10-15T05:32:45.778Z","dateModified":"2026-10-15T05:32:45.848Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/fae21441-cc0b-457f-834d-e523255cadbb.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"requests, pages, verified, teardown, crawler, live-answer, fetched, fetches","wordCount":2046,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw#faq","mainEntity":[{"@type":"Question","name":"Why do AI bot hit counts differ from verified bot requests in server logs?","acceptedAnswer":{"@type":"Answer","text":"Raw hit counts include every request that declares an AI user-agent, and that header is supplied by the requester and easy to imitate. Verified hits only count requests whose connecting IP matches the vendor's published IP ranges. In one 30-day export, /robots.txt received 13,294 claimed AI bot requests but only 3,730 passed IP verification."}},{"@type":"Question","name":"Are requests claiming to be AI bots always from real AI systems?","acceptedAnswer":{"@type":"Answer","text":"No. An unverified request is not proof of a malicious scanner, but it means the declared bot identity did not pass the IP check against the vendor's published ranges. Some may be scrapers or other automated traffic using a borrowed user-agent, and the log alone does not classify every requester."}},{"@type":"Question","name":"Does a live-answer fetch from ChatGPT mean my page was cited in an answer?","acceptedAnswer":{"@type":"Answer","text":"No. A live-answer hit only shows that a verified agent such as ChatGPT-User or Claude-User fetched a URL in a live-retrieval context. It does not show the final answer, a clickable citation, or a resulting sale, so a fetch should not be counted as a citation."}},{"@type":"Question","name":"Why did the homepage get many more AI fetches than the blog index?","acceptedAnswer":{"@type":"Answer","text":"In the 30-day export, the homepage recorded 834 ChatGPT-User live-answer fetches while the main blog index recorded 4. The homepage contains the core offer, pricing, case study metrics, and service architecture in one document, whereas the blog index is a directory of cards that asks the agent to follow another link before reaching a complete article."}},{"@type":"Question","name":"Do category hub pages get fetched by AI agents like ChatGPT and Claude?","acceptedAnswer":{"@type":"Answer","text":"They can be crawled but still not fetched for live answers. Three category hubs covering search engine marketing, agency workflows, and Google Ads bidding each drew about 240 verified crawler fetches, yet across all three there were zero live-answer fetches. A specific dated post about recent Google Ads changes drew 330 verified live-answer fetches in the same window."}},{"@type":"Question","name":"How should I structure content so AI agents fetch the page directly?","acceptedAnswer":{"@type":"Answer","text":"Put the answer on a specific page rather than relying on a hub that links to several others. The post that drew live fetches had a clear subject, a date, named changes, bidding details, and cost-per-acquisition discussion a reader could use without visiting other pages. Keep category hubs as navigation, but treat them as a substitute for nothing."}},{"@type":"Question","name":"If my page has zero AI agent fetches in the logs, is something wrong?","acceptedAnswer":{"@type":"Answer","text":"Not necessarily. Zero fetches could mean no relevant live-retrieval request arrived, the answer did not require a web fetch, or another URL was selected. The access log records requests that arrived, not the questions users asked or every reason a system chose not to request a page, so investigate a zero rather than inventing its cause."}},{"@type":"Question","name":"What fields should I capture in edge logs to analyse AI crawler traffic?","acceptedAnswer":{"@type":"Answer","text":"Capture the incoming IP, user-agent, HTTP status code, and requested path in your edge access logs. Check claimed OpenAI and Anthropic identities against their published IP ranges and set unverified requests aside. Also separate background crawler requests from ChatGPT-User and Claude-User requests, comparing URLs within each group rather than combining them into one sitewide total."}}]}]}
```