Crawl Budget on Large Industrial Catalogs
A crawler with 100,000 URLs to consider spends most of its time on the wrong ones. What decides which pages get discovered, and why the answer is architectural rather than editorial.
A crawler allocates finite requests across a site, so a catalogue of 100,000 URLs has most of its pages considered rarely or never. The fix is reducing the URL inventory and shortening the path to the pages that matter, not asking for more crawling.
The premise this page works from, stated here rather than assumed: a part number page requires unique specification content to avoid duplication, and a distributor or manufacturer publishing at catalogue scale has tens of thousands of them. This page is about what happens to those pages before content quality is ever assessed.
The problem is arithmetic before it is quality
A crawler issues a finite number of requests to a site in a period. A catalogue of 100,000 URLs cannot be recrawled often at any plausible allocation, so the practical question is which subset gets attention rather than whether the pages are good.
That reorders the usual priorities. A page nobody crawls cannot be assessed, cited, or ranked, and improving its content changes nothing until the crawler reaches it.
The symptom is visible in server logs rather than in a report. Requests concentrated on parameter URLs, filtered views, and pagination while product pages go weeks between visits is the signature, and it is invisible from any interface that does not show raw requests.
What actually consumes the budget
Parameter URLs multiply the inventory without adding pages. Sort orders, filter combinations, session identifiers, and tracking parameters each generate a distinct URL for the same content, and a catalogue with six filters can generate more URLs than it has products.
Pagination deepens the path to every item. A category paginated fifty deep puts its last products fifty clicks from the category page, and internal link depth is one of the strongest predictors of whether a page is crawled at all.
Faceted navigation combines both problems, which is why it is the usual first place to look on an industrial catalogue.
Slow responses reduce the budget directly, since a crawler working to a time allocation gets fewer pages from a slow server than a fast one.
What changes it
Reduce the inventory first. Parameters that do not change content should not produce crawlable URLs, and filter combinations nobody searches should not be linked. This is the largest available gain and it is a configuration change rather than a content project.
Shorten the path second. Category structures that reach any product in three clicks, and internal links from high-authority pages to the products that matter, both move pages into the crawled set.
Then decide what deserves indexing at all. A catalogue where every variant has a URL is a catalogue asking to be crawled at ten times its useful size, and the publishing decision behind that is at part number page SEO.
The page-level treatment of what an individual item page needs is at part number page, and the wider cluster at technical SEO for manufacturers.
Why this is not a content problem
Editorial quality decides what happens after a page is fetched. Crawl budget decides whether it is fetched, and the two are independent enough that a site can have excellent product content and no visibility because the pages are never reached.
That independence is the reason this sits apart from the duplication question. Two pages can be perfectly distinct and both go uncrawled, and the argument for treating industrial search as an architecture problem before a content one runs through manufacturing SEO.
Common questions
How many URLs is too many?
There is no published threshold and it varies by site. The signal is in logs: product pages going weeks between crawls while parameter URLs are fetched daily.
Does submitting a sitemap solve it?
It helps discovery and does not increase allocation. A sitemap listing 100,000 URLs tells a crawler what exists rather than persuading it to fetch more.
Should unused filters be blocked in robots.txt?
Blocking prevents crawling and leaves the URLs discoverable. Not generating the links is better, and canonical handling is better than blocking where the URLs must exist.
Is this a problem for a machine shop with 40 pages?
No. It becomes real somewhere in the thousands, and below that the constraint is almost always content rather than crawling.
Last reviewed . Published by ManufacturingSEO.ai.