Contradicted
The ACR marks the criterion “Supports”, and we verified at least one failure of that criterion on an in-scope page.
The VPAT Accuracy Report
The pre-registered methodology for a study of 20 published SaaS accessibility conformance reports, tested against the vendors' own public pages. Published before the first scan ran, so the rules cannot move once the results exist.
The Study in Brief
A VPAT (Voluntary Product Accessibility Template) is the standard form a software vendor fills out to describe how its product conforms to WCAG. The completed document is an ACR (Accessibility Conformance Report). Procurement teams in government, education, and the enterprise rely on these documents to decide what to buy, and in most purchases nobody ever tests the claims inside them.
Those claims are testable. When an ACR marks a criterion “Supports”, it makes a falsifiable statement: one verified failure of that criterion on an in-scope public page contradicts it. This study collects the published ACRs of roughly 20 well-known SaaS vendors and tests their “Supports” claims against the vendors’ own publicly reachable pages, using automated scanning with manual verification of every flagged result. The asymmetry matters and the whole design honors it: automation can refute a conformance claim, but it can never confirm one.
This page is the pre-registered methodology, published before the first scan runs, so that the rules cannot drift to fit the results. There is precedent for both the question and the answer. DeLancey (Library Hi Tech, 2015) tested 17 published VPATs against their products and found discrepancies in 16 of the 17. The WebAIM Million (2026) found detectable WCAG 2 failures on 95.9% of the top one million home pages.
What We Test
WCAG 2.2 Level A and AA contains 55 success criteria. We classify all 55 into three evidence classes before any vendor is scanned. A criterion only enters the gap math if it sits in Class D or Class V, and only in the failing direction.
Class D Deterministic
Machine-decidable failures with near-zero false positives. If the scanner reports it, the failure condition is true by computation.
| Success criterion | Failure condition tested |
|---|---|
| 1.1.1 Non-text Content | Informative images with no alt text |
| 1.3.1 Info and Relationships | Form fields with no programmatic label |
| 2.4.2 Page Titled | Missing or empty document title |
| 2.4.4 Link Purpose (In Context) | Links with empty accessible names |
| 2.5.8 Target Size (Minimum) | Computed target under 24x24 CSS pixels |
| 3.1.1 Language of Page | Missing or invalid html lang attribute |
| 3.3.2 Labels or Instructions | Input with no accessible name |
| 4.1.2 Name, Role, Value | Buttons or controls with no accessible name; invalid ARIA |
Class V Verified
Machine-flagged, then human-reviewed. Every instance is inspected by a person before it counts. An unreviewed flag is discarded.
| Success criterion | What the machine flags for review |
|---|---|
| 1.4.3 Contrast (Minimum) | Computed contrast, measured on solid backgrounds only |
| 1.4.4 Resize Text / 1.4.10 Reflow / 1.4.12 Text Spacing | Zoom, reflow, and text-spacing probes |
| 1.4.11 Non-text Contrast | Boundaries of user interface components |
| 2.1.1 Keyboard | Interaction probes on actionable elements |
| 2.4.1 Bypass Blocks | No skip link, no landmarks, and no heading structure |
| 2.4.7 Focus Visible | No visible rendering change when an element receives focus |
| 3.3.8 Accessible Authentication (Minimum) | Paste blocked on credential fields, or a CAPTCHA with no alternative |
Class X Excluded
Criteria that are not machine-testable to our evidence standard. Roughly 30 of the 55 land here, including media alternatives (the 1.2.x series), the semantics of focus order, and the quality of error suggestions. This study renders no verdict on any Class X criterion, for any vendor.
The result is a deliberately narrow instrument. It covers roughly 20 of 55 criteria, in the failing direction only. That narrowness is the point: every gap this study can find sits in the band a vendor’s own QA tooling should have caught before the ACR was signed.
How We Define Accuracy
Every claim in scope resolves to exactly one of four verdicts. The vocabulary is fixed here, before any result exists.
Not contradicted
The ACR marks the criterion “Supports” and we found no failure. We never report this as confirmed or passed. A sample can refute a conformance claim; it cannot establish one.
Consistent (disclosed)
The ACR marks the criterion “Partially Supports” or “Does Not Support”. Whether or not we find failures, the vendor disclosed the limitation. This is the VPAT working as designed, and it is recorded as a point in the vendor’s favor.
Not claimed / Not testable
The criterion is absent from the ACR, out of scope for the product, or in Class X. Excluded from all gap arithmetic.
Gaps are counted as criterion-level booleans per vendor. A broken navigation pattern repeated on eight pages is one contradiction, never eight. Where a call could go either way, it goes to the vendor.
Three Claim Surfaces, Kept Separate
A conformance claim is only tested against pages it actually covers.
S1: The product ACR
Product ACR claims are tested only against publicly reachable product surfaces: login pages, help centers, and public product artifacts such as booking pages, surveys, and signing ceremonies.
S2: The website statement
Where a vendor publishes a website accessibility statement, its claims are tested against the marketing pages that statement covers.
S3: No claim made
Where no claim covers the website, results are reported as context only, benchmarked against the WebAIM Million. Nothing on this surface counts as a contradiction.
This separation answers the strongest rebuttal in advance. A vendor whose VPAT covers the product owes nothing for its homepage, and this design never bills them for it.
How Vendors Enter the Sample
Selection criteria are fixed before intake. We do not name candidates until publication.
- Public ACR. The document is downloadable without an NDA and without talking to sales.
- Recognizable. Forbes Cloud 100, a G2 category leader, or a public company.
- Web-delivered. The product has a public surface we can reach.
- Vertical spread. No vertical exceeds 4 of the 20, and at least 6 verticals are represented.
- Independence. Never a Harbor client, and not in any active conversation with us.
- Current ACR only. Whatever document is live on the intake date is the document tested.
Vendors whose ACR sits behind a request form are excluded from the sample but counted in the report. The access funnel is itself a finding: a conformance document that procurement cannot read without a sales conversation is doing a different job.
How We Keep This Honest
- Public pages only. robots.txt honored, crawls rate-limited, and an honest user agent that identifies the study.
- Full evidence per contradiction. Every contradiction carries a URL, a timestamp, the rule ID, the DOM selector, a screenshot, and a reviewer sign-off.
- Archived state. Page state is archived locally and at the Wayback Machine at scan time, so every finding can be re-examined later.
- Conservative by construction. Qualified claims are demoted, ambiguous results are dropped, and every Class V flag passes a human gate before it counts.
- 30-day notice. Every named vendor receives its full evidence packet 30 days before publication, with an offer to correct.
- Standing corrections page. A dedicated email address, dated corrections, and no silent deletions.
- Peer-reviewed precedent. DeLancey (Library Hi Tech, 2015) and Code4Lib’s “Trust, But Verify” (2020) established the method this study extends.
What We Will Measure
The headline numbers are committed here, in advance. The report will publish each of these whatever the values turn out to be.
- How many of the 20 vendors have at least one contradicted “Supports” claim.
- The median number of contradictions per vendor.
- The most over-claimed success criterion across the sample.
- Whether ACRs that disclose zero limitations are contradicted more often than ACRs that disclose some (the candor finding).
- The median ACR age, and the share of documents that predate WCAG 2.2.
- How many accessibility-statement pages themselves have detectable failures.
- How many issues are fixed during the 30-day notice window. If naming a gap gets it repaired before publication, the study has already done useful work.
For Vendors
Every named vendor receives its full evidence packet 30 days before publication: every contradiction, with the URL, timestamp, selector, and screenshot behind it. Responses will be quoted. Factual corrections will be incorporated. Fixes made during the window will be noted in the published report.
Questions and responses go to aszigety@harboraccessibility.com. Corrections follow a standing process: a dedicated email address, acknowledgment within 5 business days, and verified fixes confirmed by re-scan before the report is updated.
For Buyers: Verify a VPAT Today
The study will take months. The checklist below works now. Before you accept any vendor’s ACR, check these seven things.
- Dated within 18 months. Older documents describe an older product.
- VPAT 2.5 edition. The first edition to include WCAG 2.2.
- WCAG 2.2 coverage stated. The version evaluated should be named, and current.
- Evaluation methodology named. How the testing was done, in the document itself.
- Third-party author identified. Who evaluated the product, and what qualifies them.
- At least one “Partially Supports” entry. A document that discloses nothing is a candor signal in itself.
- Spot-check the login page yourself. Tab through it with a keyboard. Two minutes tells you a lot.
Harbor authors ACRs for a living, under the same falsifiability standard this study applies to others. Learn how Harbor authors VPATs.
Questions About the Study?
This methodology is public by design. If you have questions about the approach, want to discuss vendor participation, or need help verifying your own VPAT, reach out.
Contact Harbor