Testing Beyond the Scanner
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Welcome back to The Explainer. Today we are absolutely shattering the automated accessibility illusion. We're going to experience exactly how real people actually navigate digital products, using a keyboard, screen readers, 400% zoom and voice control, and learn how to engineer interfaces that work effortlessly for absolutely everyone.
Now that our pathway and operations desk apps carry meaningful image alternatives, semantic sortable tables, and fully synchronized web VTT captions, we have to verify our foundational code holds up to real human testing. We simply cannot afford to be fooled by vanity scanner badges that give us a false sense of security.
First, the automated scanner illusion. Second, auditing five hidden barriers. Third, the 15-minute manual pass. And finally, user research and triage.
Let's dive right in. The automated scanner illusion. Right, we need to confront a massive bit of cognitive dissonance in web development. It is entirely possible to get perfect scores on fundamentally broken pages.
Just look at this. You can have a brilliantly green 100 out of 100 Chrome Lighthouse badge and a terminal log showing absolutely zero AXCore violations on a seeded build-of-R Operations Desk approved enrollment drawer.
But at the exact same time, that page could literally match the UK government's famous benchmark for the world's least accessible web page. It sounds mad, right? But a deterministic engine absolutely will give a perfect score to a total nightmare of a UI.
The absolute crucial metric to remember here is 57%. Deterministic engines, things like Axe Core, Lighthouse, IBM Equal Access, or your Playwright integrations, they are brilliant tools for sure. They're incredibly reliable and give us near zero false positives, but they bracks out at catching, on average, just 57% of WACAG issues automatically.
And that leaves us with a seriously dangerous blind spot. Automation reliably catches stuff like missing labels or invalid ARIA attributes, but it silently passes catastrophic human-blocking failures. I'm talking about focus traps escaping behind modal backdrops, or completely useless alt text like img8842.jpg.
It ignores sticky toolbars that swallow up zoomed viewports, and mouse-only drag and drop traps. If you're relying purely on automated pipelines for your product governance, you are carrying a massive hidden risk.
Auditing five hidden barriers. To actually uncover those hidden barriers, we are going on a hands-on audit journey through our seeded pathway and operations desk app builds. We'll be manually testing five specific modalities: visual, keyboard, screen reader, 400% zoom, and mobile or voice.
This isn't just about reading code. It's a highly tactile, active process. We need to experience the interface exactly as our users do. First up, the visual modality.
Imagine a user activates a forced colours mode, like the night sky windows high contrast theme. In our so-called perfectly scored build, our custom enrolment badges are completely erased, and the submit error banner just gets pushed right off screen.
It's totally broken. The fix? We implement canvas text border tokens and inline field link errors. This ensures the UI actually respects user-defined system colours natively.
Next up is the keyboard, and honestly, this is a classic trap. If you press the tab key, focus completely escapes behind the operations desk's enrolment drawer and gets utterly lost in a 50-row cohort table. Meanwhile, hitting the escape key does absolutely nothing.
So, how do we fix it? We eradicate this dreaded keyboard trap by utilising native dialog APIs, specifically calling the.show modal method. This natively locks the focus, enables escape key dismissal, and seamlessly returns focus to the trigger element when you close it.
Now let's experience the screen reader reality. The broken version, which remember technically past our automated scanners, reads out img underscore 8842.jpg, graphic, edit text, blank button. I mean that is completely useless to a human being.
Now compare that to a truly accessible semantic version. Pathway certificate preview for cohort alpha, graphic. Learner email, required, invalid entry, enter a valid institutional email address, edit text.
Approve enrolment, button. That right there is the visceral difference between just passing a scanner and actually being accessible. Let's move on to 400% zoom.
Under WACAG reflow rules, we absolutely have to test this at a 320 CSS pixel width. In our broken build, Pathway's position sticky quiz toolbar swallows up an enormous 65% of the viewport height. It makes the page practically unreadable.
To fix this, we utilize a media query, specifically targeting a max height of 500 pixels and a max width of 480 pixels to unpin that toolbar. Making it static means the core content is finally actually visible. Finally, let's look at mobile, voice and switch controls.
Pathway's drag and drop card matcher is axe clean, sure, but it is mouse only. If you rely on voice control or a switch device, you are completely locked out. So to conform to WCAG 2.5.7, we fix this by engineering single tap and keyboard move item alternative menu actions, placing them right alongside that complex drag and drop gesture.
The 15-minute manual pass. So how do we actually operationalize all of this? We codify these fixes into a repeatable pre-pull request inspection workflow. In fact, I challenge every single engineer watching this to literally unplug your mouse.
Seriously, unplug it. Do a full traversal of your app using only tab, shift tab, enter, space, escape, and the arrow keys. Check your app at 320 CSS pixels for reflow.
Run the WCAG 1.4.12 Text Spacing Bookmarklet, and inspect the accessibility tree right there in Chrome DevTools. You need to feel the interface before you even think about submitting that PR. And when you do fire up a screen reader, you absolutely must use empirically matched pairings based on the WebAim survey.
That means using VoiceOver with Safari on Apple Mac OS and iOS devices, NVDA or Jaws with Chrome or Edge on Windows, and Talkback with Chrome on Android. Boot these up, use the web rotor or their NVDA elements list, and actually navigate your UI.
User research and triage. Because here's the thing: no matter how rigorous our 15-minute developer smoke tests are, they do not replace native daily assistive technology fluency. The W3C stresses that people with disabilities use highly personalized adaptive strategies to interact with the web.
We simply cannot just guess how they use our software. We must follow standards, like those from the W3C Wacky and the Gov.uk service manual, to ethically recruit, properly consent, and fairly compensate disabled contributors to test our products using their own real-world configurations.
When we conduct these paid sessions, we anchor the research in the official WCAGEM methodology. This means we strictly define the audit scope, explore the target application, select a representative sample of complex multi-view flows, audit those specific samples against the guidelines, and formally report the findings.
This transforms anecdotal feedback into highly structured actionable engineering data. Once you have that report, you have to triage the defects by their human impact, not just by technical deviation.
We rank these from blocker down to critical, degraded, and minor. Look, if a user cannot complete an enrolment flow because a focus trap physically prevents them from moving forward, that is a blocker. We are going to prioritize human blockers over a minor contrast failure on a decorative foot element every single time.
So I'll leave you with this final, rather provocative thought. If an automated scanner says your app is perfect, but a real user can't actually navigate it, is it really accessible? Think about that as you review your own products this week.
This perfectly tees up our core finale in our next explainer, where we'll tie all of this together and discuss building and maintaining an accessible product over the long term. Thanks for joining me, and I'll see you in the next one.