Bots

Bots do not count

Crawlers, scripts, headless browsers and traffic from data centres (such as Google Cloud, Amazon and Microsoft) are recognised and kept out of your numbers, as is traffic that Cloudflare recognises as automated. The networks of Cloudflare, Akamai and Fastly do not count as data centres, because iCloud Private Relay and WARP run through them.

At the bottom of the Overview you see how many bots were filtered out, with a link to the Bots tab. No IP address is ever stored.

The Bots tab

The Bots tab (/app/<your-domain>/bots) starts with one sentence: how many bots came by, which share of all traffic that is, and what most of them are. Then a row of tiles, each with the change against the previous period and a small chart:

• Bot visits
• Useful: search engines, link previews, monitoring and archives.
• Commercial: SEO tools, AI crawlers and bots that introduce themselves by name.
• Suspicious: scripts, headless browsers, data centres and nameless bots.

Click a tile to see it per day (per week for long periods) with the previous period dashed. A rise in suspicious bots turns red.

API: GET /v1/sites/{site}/bots.

What each bot does

A table lists every bot, its kind and one sentence about what it does, for example “Reads your site to train OpenAI’s AI models.” Below it: the networks, countries and pages that bots visited.

The catalogue

Nvivio knows 104 bots by name: search engines, AI crawlers, SEO tools, link previews, monitoring, archives and scripts, each with an explanation. Search it, show only the ones that came by, and click a bot to read what it is. Bots that introduce themselves by a name Nvivio does not know are listed separately.

Bots that never run JavaScript

Many AI crawlers, such as GPTBot, ClaudeBot and CCBot, only fetch the HTML and never load the script. To see them too, let your server forward those requests: see Forward bots from your server. Once your server sends the first bot, the script only counts bots that pretend to be a browser, so nothing is counted twice.

For what AI crawlers read and what AI assistants send back, see AI traffic and AI crawlers.