AI Crawler Access for Manufacturer Websites
Which AI agents fetch a supplier site, what each one is for, why the training decision and the retrieval decision are different decisions sharing one file, and how to check what is actually being served.
AI agents fetching a site fall into three purposes: training, search indexing, and user-initiated retrieval on behalf of someone asking about the company right now. robots.txt controls all three in one file, and blocking the third is the expensive mistake.
The question usually arrives as a single decision: whether to let AI crawl the site. It is not one decision. It is three, they have different costs, and they are all made in one file, which is why they get made together and wrongly.
Three purposes, one file
Training crawlers collect text that may inform a future model. GPTBot, ClaudeBot, CCBot, Applebot-Extended, Google-Extended, Amazonbot, and Meta-ExternalAgent are in this group. What they take affects a model's parametric memory, which is the path a manufacturer cannot affect anyway and which the supplier search mechanics treat as weather.
Search indexing crawlers build the retrieval index an assistant queries when it browses. OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Bingbot are in this group. These decide whether a supplier is available to be found in the browsing path, which is the path a supplier can influence.
User-initiated fetchers retrieve a page because a person just asked. ChatGPT-User, Claude-User, and Perplexity-User are in this group. A request from one of these means a buyer is asking a question about this company or this capability at that moment.
The three groups are separated by user agent, which means they can be decided separately. Almost nobody does.
The expensive mistake
Blocking a user-initiated fetcher is the closest thing to blocking a visitor. The request exists because someone asked. Refusing it returns nothing to a buyer who was, at that moment, evaluating suppliers, and it produces no log entry anyone reads as a lost opportunity.
The second most expensive is blocking search indexing while intending to block training. The two look alike in a robots file and the outcomes are opposite: one withholds text from a corpus that may or may not ever matter, the other removes the company from the surface where constrained supplier prompts get answered.
If the training decision is the one a company genuinely cares about, it can be made without touching the other two: block GPTBot, ClaudeBot, CCBot, and Applebot-Extended, leave the search and user agents allowed. That preserves everything a supplier can act on.
The Google case is not symmetric
Blocking Google-Extended does not remove a site from AI Overviews. Google-Extended governs generative training and grounding for Gemini. AI Overviews are assembled from the ordinary search index built by Googlebot, so the only way to withhold content from them is to withhold it from Search.
The partial controls are nosnippet, max-snippet:0, and data-nosnippet, and they suppress the same text in ordinary results. There is no directive that removes a page from Overviews and leaves its blue-link snippet intact.
That constraint decides the question rather than complicating it. AI Overview presence reduces click-through rate on informational industrial queries, and the correct response is the one on the Overviews page: change which pages are written to be cited and which are written to be visited. Suppression is not on the menu, and being absent from the answer is worse than being summarised in it.
robots.txt is advisory, and this matters for one case
Compliance with robots.txt is voluntary. The major named agents document honouring it, and an agent that does not honour it has no reason to identify itself accurately either, which is why a share cannot be measured from one site's logs.
For ordinary marketing content this is fine. For one case it is not.
ITAR registration restricts publication of controlled technical data, and robots.txt is not a publication control. A drawing, tolerance table, or process detail that must not leave the country must not be on a public URL at all. A disallow line is a request, and a request is not a compliance boundary. Access control or removal is the answer, and the technical treatment covers where the boundary belongs.
The same logic applies more weakly to competitively sensitive capacity or pricing detail: if it must not be read, it must not be served.
Check what is served, not what is written
A robots file says what is permitted. It does not say what happens.
Bot management at the CDN or WAF blocks agents at the network layer, before robots.txt is consulted, and several platforms ship AI-agent rules enabled by default. A site can allow every agent in robots.txt and return 403 to all of them.
The check is a fetch, not a read. Request a real page with each agent's user-agent string and record the status code. A 200 with the page body means access. A 403, a challenge page, or a JavaScript interstitial means the agent gets nothing, whatever the robots file says.
Worth running against the user-initiated agents first, since those are the ones whose failures cost something immediately. Verifying access is also the precondition for entity work showing any result, because a site that cannot be fetched cannot be resolved.
What this site does and why
Every AI agent is allowed, and the allowlist is asserted on every deploy. The list lives in one module, robots.txt is generated from it, and the build fails if an agent drops out, so access cannot be lost by an edit nobody reviewed.
The reasoning is specific rather than general: blocking model access on a site whose business model is citation would be self-defeating. A manufacturer whose business model is quote requests is in a similar position for the retrieval agents and a different one for the training agents, which is why the recommendation here is per-purpose rather than a blanket allow.
The companion file, llms.txt, is generated the same way and grants nothing. Access is decided here.
Common questions
Should a manufacturer block AI training crawlers?
It is a defensible choice with a low cost, since the memory path cannot be influenced regardless. Block the training group only, and leave search and user agents allowed.
Does blocking crawlers protect content from being copied?
No. Compliant agents honour the file and nothing else is prevented. Content that must not be reproduced should not be public.
What about Bingbot?
It serves both classic Bing results and assistant surfaces built on that index. Blocking it costs classic visibility as well, which makes it a poor place to express an AI preference.
Do these agents identify themselves reliably?
The major ones publish user agent strings and IP ranges that can be verified by reverse DNS. That verification is worth doing before writing rules that depend on the string alone.
How often should this be checked?
At every platform or CDN change, and otherwise quarterly. The common failure is a default rule arriving with a vendor update rather than a decision anyone made.
Sources
- 01
Last reviewed . Published by ManufacturingSEO.ai.