Which Pages Do You Audit Out of Thousands?

Say you run an accessibility checker. You feed it a site’s URL, and the checker has to decide: which pages should it actually look at?

Is the homepage enough? If so, you don’t have much basis for saying “this site is accessible” or “this site isn’t.” But checking every page isn’t realistic either — forums, blogs, and shopping sites can have hundreds or thousands of posts. Auditing all of them blows the time and cost budget.

I kept running into this question while building A11y Check, an automated accessibility auditing service. This post is a record of building an algorithm that deterministically picks the pages that represent a site, using only URL and sitemap signals — without downloading a single page body. I figured I might eventually split it into its own package, so from the start I designed it as a pure function with no I/O dependencies.

The Wheel Already Exists — WCAG-EM 2.0

Fortunately, this isn’t a problem I had to solve from scratch. The W3C’s WCAG-EM (Website Accessibility Conformance Evaluation Methodology) already defines how to sample and evaluate an entire site — and it was just revised to WCAG-EM 2.0 not long ago. The core idea comes down to two kinds of samples.

• Structured sample: common pages (home, login, contact, etc.) + a representative of each page type (layout) + key functional pages.
• Random sample: roughly 10% more pages picked at random, to verify the structured sample didn’t miss anything.

The essence of the methodology here is stratified sampling. It splits the population into “strata” of similar items and then samples from each stratum — the same technique pollsters use when they sample by age group or region. Here, the site gets split into strata by “page type,” and each stratum contributes a representative.

The catch is that this “splitting into types” has to be done automatically by code, not by a person. That’s where my job started.

The Limits of a Naive First Version

My first implementation classified pages like this:

function categorizePage(url) {
  if (/login|signin|로그인/.test(path)) return "login";
  if (/contact|문의/.test(path))        return "contact";
  if (/privacy|terms|약관/.test(path))  return "legal";
  // ...
  return "content"; // everything else
}

See the problem? Blog posts, products, news articles — everything gets flattened into a single content bucket. Even if a forum has 100 posts, the algorithm doesn’t see “100 posts of type content” — it just sees “100 miscellaneous pages.” So when it came time to sample, there was no guarantee that even one actual post would make it into the representative set.

And from a user’s perspective, individual posts on forums and blogs are exactly where accessibility problems pile up the most: body text typed into an editor, alt text on attached images, table structures — anything a human fills in by hand. Leaving that out of the sample means skipping the very thing you should be checking.

The goal became clear:

1. Recognize repeated content like forum posts, blog posts, and product pages as a single “type.”
2. Within that type, pull representative posts proportional to how many pages you’re checking, with a minimum of at least one.

Idea #1 — URL Template Clustering

The textbook way to detect repeated content is near-duplicate detection — techniques like shingling, SimHash, or MinHash that chop a document into pieces and compare fingerprints. But all of these require downloading the page body first. Pre-fetching hundreds of pages just isn’t feasible within a reasonable time budget.

But if you look closely, pages stamped out from the same layout share an identical URL structure.

/blog/123
/blog/456
/blog/hello-world

If you tokenize the path segments and replace the “variable parts” with placeholders, all of these collapse into one template key:

/blog/{slug}

Any set of URLs sharing this key is a single “repeated content cluster.” No page body is ever opened — the decision is made purely from the URL. (In the literature, this is called URL pattern learning — see, for example, “Learning URL Patterns for Webpage De-duplication” from WSDM 2010.)

Segment Normalization Rules

Each path segment is judged in this priority order:

function segmentToken(seg: string, index: number): string {
  const s = seg.toLowerCase();
  if (UUID_RE.test(s)) return "{uuid}";
  if (DATE_RE.test(s)) return "{date}";
  if (YEAR_RE.test(s)) return "{year}";   // ^(19|20)\d{2}$
  if (DIGITS_RE.test(s)) return "{n}";
  if (isHash(s)) return "{hash}";
  if (isSlug(s, index)) return "{slug}";
  return s; // lowercase literal if it's not a variable part
}

Three Rules to Prevent False Positives

The part that really matters here is minimizing false positives. Replace too eagerly with placeholders and unrelated pages get lumped into the same cluster, which breaks the whole stratification. These are the rules I ended up adding after getting burned a few times.

1) Never replace the first segment with a slug. If /blog/* and /products/* both flatten to /{slug}/*, two completely different sections end up in one cluster. So slug detection only kicks in from the second segment onward (index >= 1).

2) Keep short, dictionary-like segments as literals. /contact-us shouldn’t be read as a slug. So slug detection has a threshold: “length ≥ 8 chars AND (2+ hyphens OR mixed letters and digits).”

3) Validate years by range. 2024 is {year}, but 3000 is just {n}. ^(19|20)\d{2}$ only catches actual years.

Collapsing Pagination

Query parameters on listing pages, like ?page=2 or ?sort=desc, are different views of the same page. Counting each as a separate URL lets a single listing page pollute a cluster. So pagination and sort parameters get collapsed up front.

export const PAGINATION_PARAMS = new Set([
  "page", "p", "pg", "paged", "offset", "start", "sort", "order", // ...
]);

// /list?page=2 and /list?page=3 collapse into the same URL
normalizeForDedup("https://e.com/list?page=2"); // → "https://e.com/list"

Idea #2 — Stratification + Quota-Proportional Allocation

Once clusters are detected, the next problem is how to split up a limited budget (the number of pages you can check in one run). This happens in two rounds.

Round 1: Coverage (One Per Type)

Slots are filled one at a time in priority order:

Home → Login → Contact
     → One representative per repeated cluster (largest first)
     → Remaining common types (forms, help, terms, search)
     → Static content

Thanks to this order, even checking just 5 pages secures representatives from the two largest clusters. Login and contact come before clusters in priority, so they always survive.

Round 2: Proportional Allocation by Size (Largest Remainder Method)

Once Round 1 is filled, any remaining budget gets distributed proportionally to cluster size. This uses the largest remainder method (also known as the Hamilton method) — the same method used to allocate parliamentary seats in proportion to a party’s vote share. Each cluster first gets the integer part of its proportional share, then the leftover slots go one by one, in order of largest fractional remainder, to whichever clusters have the biggest leftovers.

To keep one cluster from hogging the whole sample, there’s a per-cluster cap = ceil(max/2), and any share that hits the cap gets redistributed to clusters with room to spare.

For example, say a forum has three types — products (120 posts), blog (30), and notices (12) — the allocation looks like this depending on how many pages you check per run:

Let’s trace the 20-page case. Round 1 gives each type one representative (product, blog, notice), leaving 9 slots. Split 120/30/12 proportionally, that’s 6.67 / 1.67 / 0.67. Give the integer parts first: 6 / 1 / 0 (sum 7). Distribute the remaining 2 slots by largest fractional remainder — product and blog each get one → 7 / 2 / 0. Add back the one representative each from Round 1, and you land on 8 Product · 3 Blog · 1 Notice.

What Picks the Representative Within a Cluster?

Within a cluster, which post gets picked as the representative? Sitemap signals decide this. A sitemap (sitemap.xml) is a file a site uses to tell search engines about its page list, and each URL can carry a last-modified date (<lastmod>) and a relative priority (<priority>).

Sort priority: most recent lastmod → highest priority → deterministic hash

So instead of just scanning <loc> values, the sitemap parser was rewritten to parse at the <url> block level, reading the sibling <lastmod> and <priority> elements along with it. Recent, higher-priority posts get picked as representatives first.

Determinism — the Same Site Always Yields the Same Sample

This turns out to matter more than it sounds. If auditing the same site twice gives a different sample each time, you can’t trust score trends at all. “It went up 3 points from last week” becomes meaningless — was that an actual improvement, or just a different sample?

So every sort’s final tiebreak is pinned to a content hash of the URL. It uses FNV-1a, a lightweight, fast hash function, seeded with the site’s address. The output looks random, but the same input always produces the same value, so it’s fully reproducible.

// Even with no lastmod/priority at all, the hash still picks deterministically → reproducibility guaranteed
return hashString(rootUrl + "|" + a.url) - hashString(rootUrl + "|" + b.url);

A Real-World Trap — Apex vs. www

Once this was built, I ran it against real sites — and, of all things, my own blog (codeslog.com) came back with just a single page: the homepage. I build an accessibility auditing algorithm, and the accessibility blog is the one site it returns zero samples for. I didn’t know whether to laugh.

Digging in, here’s what was going on:

• codeslog.com doesn’t redirect to www.codeslog.com. The apex domain (no www) just returns 200 directly.
• But the homepage HTML’s declared canonical address (og:url) points to https://www.codeslog.com/.
• And the sitemap’s actual content URLs were all on the www origin.

The crawler only follows same-origin (scheme + host + port) links, and since codeslog.com and www.codeslog.com have different hosts, it treated them as unrelated sites. There was already logic to follow redirects and settle on a canonical domain, but this site doesn’t redirect, so that logic never triggered.

Here’s the fix: when there’s no redirect, look at the canonical URL the root HTML itself declares (<link rel="canonical"> first, falling back to og:url), and only if it’s a www variant, swap the host to adopt it as the canonical root.

// If the redirect didn't resolve www, check the canonical/og:url declared in the HTML
if (canonicalRoot === rootUrl && rootHtml) {
  const declared = extractDeclaredCanonical(rootHtml); // <link canonical> → og:url
  const norm = declared ? normalizeUrl(declared, rootUrl) : null;
  if (norm && isWwwVariant(norm, rootUrl)) {
    const adopted = new URL(rootUrl);
    adopted.hostname = new URL(norm).hostname; // swap host only, keep the path
    canonicalRoot = adopted.toString();
  }
}

There’s a security concern to account for here. You can’t blindly trust whatever canonical a page declares. A malicious page could stuff evil.com into its og:url and drag the crawler off to somewhere it shouldn’t go. So the fix only accepts the declared canonical when it’s a www-prefix difference on the same registrable domain (isWwwVariant). Apex and www are, by definition, the same site, so this is safe.

Here’s the result after the fix, running codeslog.com again:

source = sitemap
clusters:
  /posts/{slug}      representative of 61
  /en/posts/{slug}   representative of 54
  /tags/{hash}       representative of 14
  /series/{slug}     representative of 10
  ...

Feed it just the apex address, and it now catches posts, tags, and series types living on www and pulls a representative from each.

Limitations and Next Steps

Being honest, a few things are still left on the table:

• Template false positives: percent-encoded Korean tags (/tags/%EC...) get classified as {hash}, while /tags/wcag-2.2 gets classified as {slug} — so the same “tag page” type ends up split into two clusters. That’s exactly what you can see in the codeslog result above, where /tags/{hash} and /tags/{slug} show up separately. It doesn’t break anything functionally, but there’s room to refine it.
• Sites with no sitemap: there’s an option to shallowly (2-hop) open listing pages to pull individual post links, but it’s off by default to protect the time budget.
• This doesn’t go as far as near-duplicate detection: URL signals alone are practical enough for most sites, but they’re weak on sites where the URL doesn’t carry meaning — like an old-school forum that identifies posts purely by a ?id= query string.

TL;DR

• The problem: you can neither check every page on a site nor rely on the homepage alone. You need to pick representative pages, and the criteria have to be decided automatically by code.
• The standard: the W3C’s WCAG-EM 2.0 stratified sampling — split the site into types and pick a representative from each.
• Idea #1: URL template clustering — replace the variable parts of a path with placeholders to classify repeated content by type from URL structure alone, with no page body required.
• Idea #2: Stratification + largest remainder (Hamilton) allocation — Round 1 secures one representative per type; Round 2 fairly splits the remaining budget proportional to size.
• The finish: a content-hash tiebreak guarantees determinism, so the same site always yields the same sample. The apex↔www trap got fixed by adopting a declared canonical only within a safe, narrow scope.

Good crawlers and parsers like crawlee and sitemapper are everywhere, but a tool that specifically handles “picking a representative sample for an accessibility evaluation” turned out to be surprisingly hard to find. So I built one, and I kept the core logic as pure functions with no I/O or framework dependencies, so it can eventually be split off into its own package.

A11y Check’s audit engine (@a11ychk/core) is open source under Apache-2.0. The implementation from this post lives at urlTemplate.ts and stratifiedSample.ts under packages/core/src/crawler in the GitHub repository. I hope this story — building an accessibility tool whose own crawler couldn’t read the accessibility blog — saves someone else a bit of sampling grief.

References

• WCAG Evaluation Methodology (WCAG-EM) 2.0 — W3C Group Note, stratified sampling methodology
• Learning URL Patterns for Webpage De-duplication — WSDM 2010, URL pattern learning
• A11y Check GitHub — audit engine (@a11ychk/core, Apache-2.0)