Faceted navigation and pagination hurt AI crawling when every filter, sort order and page number becomes its own URL. A big catalog then produces thousands of near duplicate pages. A crawler can spend its visits on those combinations while your real products wait unread. The fix is to tell crawlers which URLs to skip and which page is the real one.
This matters most on large catalogs. A store with 40 products and two filters rarely produces enough combinations to cause trouble. A store with 3,000 products and six filters can produce more filtered URLs than it has products.
Why do filters and page numbers confuse a crawler?
A crawler does not know in advance which filtered URL matters. It can follow a link to red, size 9, under $100, page 4, read a page that looks almost exactly like the plain collection, and move on to the next combination. Each sort order adds another copy, and each page number adds another URL for the same collection.
Structure problems are common. In our June 2026 study of 216 Shopify storefronts, 58% failed Mesh Integrity, which checks whether collections and products link in a way a machine can follow.
What controls keep a big catalog readable?
| Faceted-nav pattern | URLs it can spawn | The control |
|---|---|---|
| Each filter as a crawlable URL | One per filter combination. Thousands | Canonical to the base collection |
| Sort orders as separate URLs | A copy per sort option | noindex the sort variants |
| Deep numbered pagination | One URL per page, indefinitely | Robots rules / sensible limits |
Three tools do the work. Canonical tags point a filtered page, or one reordered by price, back at the plain collection. Robots.txt rules tell crawlers not to fetch certain URL patterns at all. Your sitemap lists the real pages and leaves the filtered ones out.
What does a Shopify store do with filter and sort URLs?
We checked our own Shopify store on September 26, 2026, with plain requests, the way a crawler fetches a page. The robots.txt it serves tells every crawler to skip collection URLs that contain sort_by and collection URLs that combine 2 or more filters. It also blocks tag combinations joined with a plus sign.
A URL with a single filter is not blocked, so a crawler can still reach it. On our store that page carries a canonical tag pointing at the plain collection, and the sort_by version does the same. Page 2 of a collection returns a normal page with a canonical pointing at itself, so each page number counts as its own URL with its own products. A store can customize its robots.txt and its theme can change these tags, so check yours instead of assuming it matches ours.
How do you check your own store in 4 steps?
- Open yourstore.com/robots.txt in a browser and look for the lines that mention sort_by and filter. If they are missing, someone has edited the robots.txt.liquid template, which you can see in Online Store > Themes > Edit code under Templates. Our guide to robots.txt in the AI era explains each line.
- Open your largest collection, apply 1 filter, and view the page source of the address you land on. Search it for canonical and confirm it points at the plain collection address.
- Sort the same collection by price and repeat the check. The canonical should again point at the plain collection.
- Open page 2 of the collection. Its canonical should point at page 2 itself, and it should list different products from page 1.
The knowledge base guide to inspecting canonical tags shows where to look in the page source.
How does an outdoor gear store stop its filters from making hundreds of boot addresses?
Take an outdoor gear store on Shopify with 2,500 products and filters for size, color, brand, price and activity. Its boots collection alone can produce hundreds of filtered addresses. Suppose an app replaced the theme's filter links with its own URL pattern. The default sort_by and filter rules in robots.txt may no longer match those addresses, and the canonical tags may point at themselves.
The owner checks one filtered boots page and finds a canonical pointing at the filtered address. The fix is in the filter app's settings or in the theme's collection template, so the canonical points back at /collections/boots. The filtered page now tells a crawler which page is the real one.
When should you get help?
If the filter app writes its own URLs, or your robots.txt.liquid template was edited and nobody remembers why, the fix sits in theme code. The Full Assessment and Roadmap reads up to 1,000 pages of your store and lists every fix in order, which on a large catalog shows which filtered pages need a canonical fix. For how a crawler moves through the rest of your store, see the Mesh site architecture.
Questions owners ask
Why is faceted navigation a problem for AI crawlers?
Faceted navigation is a problem when each filter combination creates its own URL, because the combinations multiply into thousands of near duplicate pages. A crawler can spend its visits on those pages before it reaches your real products and collections.
How does pagination affect crawlability?
Each page number in a collection is a separate URL. That is fine when each page has its own products and a canonical pointing at itself, and it becomes noise when page URLs also combine with filters and sort orders.
Do I need to worry about this on a small store?
A small store usually does not need to worry about this. A store with few products and few filters produces a manageable number of URLs. The problem grows with large catalogs and many filters.
Does Shopify block filter and sort URLs from crawlers?
On our own Shopify store, checked on September 26, 2026, the robots.txt blocks collection URLs with sort_by and URLs that combine 2 or more filters. A single filter URL stays open to crawlers, and its canonical tag points at the plain collection. Check your own robots.txt, since a store can customize it.