604 logged visits · 11 AI bots · 7 days
What AI Crawlers Actually Read: 7 Days of Data From 13 Real Marketplaces
We logged every AI crawler visit across 13 real online marketplaces for 7 days: 604 visits from 11 different bots, roughly one every 17 minutes. ClaudeBot was almost 4x more active than GPTBot, 40% of listings were read within the week, and listing age predicted being read far better than description quality. The full data, methodology, and findings.
AI crawlers from OpenAI, Anthropic, Amazon, Meta and Perplexity are reading websites every day - and almost nobody can see it. These bots never run the JavaScript that analytics tools depend on, so they are invisible in Google Analytics, Plausible, or any dashboard a normal site owner looks at. The only place they show up is the server, and most small-site owners have never seen their own server logs.
We run a marketplace platform, which means many stores sit behind one server layer. So on July 22, 2026 we started counting: every request from every AI crawler, on every store, every day. This page is the first-week dataset - 13 real marketplaces, 616 listings, 604 AI crawler visits in 7 days - published so that marketplace owners can see what these bots actually do, instead of guessing.
The counter is still running. This page will be updated as the dataset grows - the 30-day cut is next.
• 8 different AI bots arrived on day one; 11 distinct bots by end of week (from Anthropic, OpenAI, Amazon, Meta, Perplexity, ByteDance and Common Crawl).
• ClaudeBot was the most active crawler: 168 visits - almost 4x ChatGPT's GPTBot (46).
• 40% of all listings were read by at least one AI crawler within the week. Two stores had 100% of their catalog read; five stores had zero listings read.
• New listings got read fast: 84% of listings created during the week were read within days, most within about 24 hours.
• Listing age predicted being read. Description quality did not. Listings under a week old: 85% read. Older than three months: 17%.
• 74% of all visits went straight to listing (product) pages - 23% to homepages, 3% everything else.
Read the full video transcript▼
Real AI crawler logs (0:00)
This is ChatGPT's crawler reading a listing on a real marketplace last Thursday. Here it is back the next day. And the next day. Here's Claude's crawler - it came 168 times in a week. Here's Perplexity's. Here's Amazon's. The store owners had no idea this was happening. Almost no one does, because almost no one can see it. I can. And just a week ago, I started logging all AI crawlers on 13 live marketplaces. They showed up the first day, and they have not stopped coming since. Here's what they actually do.
Why nobody can see AI crawlers on their website (0:32)
In the last video, I gave you five moves to get your marketplace recommended by AI search. And the fair question in the comments was: how do you know? How does anyone know what these crawlers actually do? Who's guessing, and who's measuring it?
Here's the problem. If you run a small site, you've probably never seen your server logs. Your analytics can't see AI crawlers at all - they don't run the JavaScript that your analytics depends on. So the whole industry is giving advice on visitors nobody can see. But I'm in an unusual position: I run a marketplace platform, which means many stores, one server layer. So I did the obvious thing: I started counting every request from every AI crawler, every day. That was 7 days ago. 13 marketplaces, over 600 listings, one week of logs. So, no guessing from here on - just what they say. And I can already tell you now that I actually expected to wait weeks for enough data to make this video, but the bots actually gave it to me in days.
How I logged them (1:22)
So here's what I did. First of all, I identified what type of visitor it was. Was it a normal visitor - so a human - or was it a bot, like the ChatGPT bot, the ClaudeBot, the Facebook bot, the Perplexity bot, and so on. If it was a normal human, then of course they went straight to the store page. But if it was a bot, then I identified what bot it was. I then counted what they looked at.
Finding 1: the bots came immediately (1:52)
So here's what I found out. First of all, the bots - they came immediately, and they kept on coming. On average, there was a bot visiting every 17 minutes, around the clock - so day and night. And I was here tracking 13 active marketplaces with a total of 616 listings. On day one there were 27 bot visits, and in total, on day six, there had been 578 bot visits on these 13 marketplaces. And during this time, I identified 11 different AI bots.
The interesting thing here is that Anthropic and OpenAI have multiple different AI bots. Anthropic has the ClaudeBot, the Claude-SearchBot, and one called Claude-User. And OpenAI also has multiple ones. In this research I left out the Googlebot, because that is just the old one that we've been using for 20 years to index results - I only looked at 100% AI bots here. And the Anthropic bots - the Claude bots - were actually way more active than the ChatGPT ones. The Claude crawlers came 3.7 times more than the ChatGPT ones.
Finding 2: coverage was wildly uneven (2:58)
I then looked at what share of each marketplace's catalog got read during this week, and that was very interesting. Overall, of these 616 listings, 40% were read by these AI crawlers. There were two stores that had 100% of their catalog read, and five stores that had zero of their listings read by these AI crawlers during this week.
Finding 3: new listings get read within a day (3:21)
I then looked at the speed - how long did it take for the AI to read new listings created on these marketplaces? 84% of the 61 new listings created this week were read within days - 42 of them within 24 hours of going live. That is very fast, and it shows that new content does not really wait in line. These AI bots really like new stuff.
Finding 4: age predicted reads, description quality didn't (3:44)
It's all about the age, and not really that much about the content. The bots read what is new and linked. Listings that were less than one week old - 85% of them were read. Whereas the back catalog - listings more than 3 months old - only 17% of these were read.
I expected that a proper listing with a lot of content would get read more often than a thin listing. But at least this week, that was not the case. Comparing four different stores: one store's read listings had a median of 333 words versus 257 for skipped ones - but another store's read listings had 57 words while it skipped ones with 81. There's not really a consistent pattern, at least in this week's data.
Finding 5: 74% of visits went to listing pages (5:06)
Looking at the page types these AI crawlers visited: three out of four visits went straight to the listing pages. Basically, your product pages are your website. Listing pages took 74% of all AI crawler visits, and only 23% went to the homepages.
What this means for marketplace owners (5:28)
First: let the bots in. Add all your listings to your sitemap and do not block the bots in your robots.txt or on Cloudflare - on most platforms, including Prometora, this is done automatically. Second: give it something to read - AI does read listings above all, though longer descriptions did not earn more reads in this week's data. Writing quality is likely about being quoted, which crawl logs can't see. And the strongest signal in the data: the more new listings you get, the better. 85% of listings under a week old were read, versus 17% of those older than three months. Getting new listings all the time is what keeps AI reading your marketplace.
Seeing AI crawlers on your own marketplace (8:00)
As a marketplace owner on Prometora, you can go to the SEO & AI tab in your store settings and see the AI visibility of your marketplace: what share of your listings were read by AI, which listings have never been read, coverage by seller, and coverage by crawler. You can send a listing's link to the seller who owns it and ask them to improve it - or update it yourself from the managed seller area.
Why This Dataset Exists
Big publishers see crawler logs; small site owners almost never do. Client-side analytics can't help - AI crawlers don't execute JavaScript, so they leave no trace in any analytics tool. The result is an industry full of advice about AI search based on almost no observed behavior. A marketplace platform is one of the few places this data can exist at all: many independent stores, one server layer, one place to count.
Methodology
- Scope: 13 real online marketplaces running on the Prometora platform, 616 listings total, logged continuously from July 22 to July 28, 2026 (7 days).
- Where the counting happens: at the server middleware layer - the only place AI crawlers are visible, since they don't run client-side JavaScript.
- Bot identification: by declared user agent. Strictly speaking, every count is "requests identifying as" GPTBot, ClaudeBot, Claude-SearchBot, Claude-User, ChatGPT-User, OAI-SearchBot, PerplexityBot, Amazonbot, Meta-ExternalAgent, Bytespider, or CCBot.
- What was recorded: daily tallies per bot, per store, per page (listing page, homepage, or listings index). No visitor data, no personal data.
- What's excluded: Googlebot and Bingbot. Google's Gemini and Microsoft's Copilot read the web through the same crawlers that have indexed search for decades, so their AI reading can't be counted separately. Every bot in this dataset exists only for AI.
- Anonymization: stores are not named. Aggregates only.
- The hard limit: this data shows crawling - the machine reading. It cannot show whether any of it gets quoted or cited inside AI answers. Nobody outside the AI companies can see that, and this page never claims otherwise.
- The dataset: the day-by-bot aggregates behind this page are public - download the CSV (date, bot, page type, visit count; store-level data withheld for anonymity).
Finding 1: They Came Immediately - and Kept Coming
Eight different AI bots showed up on day one. Over the week the daily count grew almost every day: 27 visits on day one, 141 on day six. In total: 604 visits in 7 days - one AI crawler reading one of these stores roughly every 17 minutes, day and night.
Days 1-6 of logging · day 7 omitted (partial - the dataset was exported mid-day)
13 marketplaces · Jul 22-28, 2026 · 604 visits total
The surprise in the ranking: Anthropic's crawlers were far more active than OpenAI's. ClaudeBot alone made almost 4x as many visits as GPTBot, and the Claude family combined (265 visits) nearly doubled the OpenAI family (140). Count the companies on that list - Anthropic, OpenAI, Amazon, Meta, Perplexity, ByteDance - and the picture is clear: the whole AI industry is reading product pages, not just the company everyone optimizes for.
Finding 2: Coverage After One Week Was Wildly Uneven
In seven days, 40% of the 616 listings were read by at least one AI crawler. But the average hides the real story: on the same platform, in the same week, two stores had 100% of their catalog read - and five stores had zero listings read. Five of the thirteen stores received crawler visits only on their homepage: the bots knocked on the front door repeatedly and never stepped into the catalog. If your listings aren't linked and reachable from where the bots enter, nothing else about them matters.
Stores anonymized · sorted by coverage · first 7 days
Finding 3: New Listings Get Read Within a Day
61 new listings went live across these stores during the week. 51 of them - 84% - were read within days, and 42 within about 24 hours of going live. Several were read the same day they were created. The machine is not working through a backlog; new content gets picked up almost immediately, if it's reachable.
Finding 4: Age Predicted Being Read. Description Quality Did Not.
The expectation going in: skipped listings would be the thin ones - short descriptions, weak titles. The data disagreed. Comparing read versus skipped listings inside the same stores, description length showed no consistent pattern: one store's read listings had a median of 333 description words versus 257 for its skipped ones, while another store's read listings had 57 words versus 81 for skipped.
What did predict being read was age: listings under a week old were read 85% of the time; one to four weeks old, 70%; one to three months, 26%; older than three months, just 17%. The bots read what is new and what is linked - and a dormant back catalog quietly disappears from AI's view of the world.
616 listings across 13 marketplaces · first 7 days of logging
Finding 5: Three Out of Four Visits Went Straight to Product Pages
74% of all AI crawler visits went to listing pages, 23% to homepages, and 3% to everything else. As far as AI is concerned, a marketplace's product pages are the website. That's also the marketplace advantage in AI search: every listing is a page of specific, long-tail content - and sellers write new ones every day.
What This Means for Marketplace Owners
- The crawlers are already coming. They found these stores the first day the counter ran, and averaged about 86 visits a day across the network. This is not a trend to prepare for later.
- Freshness is the strongest lever in the data. New and updated listings keep the bots coming deeper into the catalog. An active store stays visible; a dormant one fades to that 17% read rate.
- Check whether bots get past your homepage. Five of thirteen stores only ever got front-door visits. Listings need to be linked and server-rendered from where crawlers enter.
- Keep the door open. Listings in the sitemap, no AI-crawler blocks in robots.txt or at the CDN. On most managed platforms - including Prometora - this is handled automatically.
- Write for the quote, not the crawl. Better descriptions didn't earn more reads in week one - but what the AI does with a page it has read almost certainly depends on what's written there. Crawl logs can't measure that, and nobody outside the AI companies can.
What This Data Can't Show
Two honest limits. First, referral traffic from AI answers can't be reliably measured - most AI-originated visits arrive with no label at all, so any tool promising a precise "ChatGPT revenue" number is guessing. Second, quoting is invisible: these logs show the machines reading at scale, every night. What they build from it - which stores get named inside AI answers - is something no crawl log can see. The reading comes first; the recommendations are built from it.
In this dataset they made 140 combined visits across 13 marketplaces in 7 days.
You can verify on your own site by checking server logs for those user agents - client-side analytics will never show them.
Frequency varied enormously by store - two stores had their entire catalog read within the week, while five stores only ever received homepage visits.
The only place they can be observed is the server (server logs or server middleware), which is where this dataset was collected.
The Claude family of crawlers combined (ClaudeBot, Claude-SearchBot, Claude-User) made 265 visits versus 140 for OpenAI's three bots. Amazonbot (89) and Meta-ExternalAgent (73) were also more active than GPTBot.
Many sites blocked these bots in 2023-2024 to protect content and locked themselves out of a growing discovery channel.
If specific content shouldn't be read, block selectively per bot in robots.txt rather than blanket-blocking.
That AI reading can't be counted separately from search indexing, and blocking it would block search itself - which is why this dataset only counts bots that exist purely for AI.
This research is part of a three-video series on marketplaces and AI search: building a marketplace with AI, the five moves to get recommended by AI search, and the crawler-log video above, which tests those five moves against real data. The counter is still running: this page gets its 30-day update next.
Related Guides
Keep going - these cover the next questions most marketplace founders have.
Tutorial
How to Build a Multi-Vendor Marketplace in 2026 (Step by Step)
From idea to first transaction without writing payment code or hiring a dev team
Marketplace Fundamentals
How to Start an Online Marketplace in 2026: 5-Step Playbook
The complete guide to going from idea to launch - without wasting months or money
Marketplace Payments
Stripe for Marketplaces: Everything You Need to Know in 2026
The complete guide to Stripe Connect for marketplace founders
Ready to build your marketplace?
Turn your marketplace idea into reality. No coding required.
Start Free Trial