Your Agent's Layout Check Is Probably Measuring Nothing

The sweep came back clean. Six surfaces at 390×844, measured with a fine pointer and again with a coarse one, no overflowing controls, no unreachable content, a tidy table of green rows.

It had measured almost nothing. Three separate mechanisms produced that table and every one of them looked exactly like a pass.

That was version four of the check. It took five before it caught anything real, and the interesting part is not the bug I was hunting. It is that a layout check is itself a piece of software with its own bugs, and its bugs are silent by construction: a broken probe does not throw, it reports success. Here is each version and what broke it.

What the check was supposed to do

Context: an epic to make the app usable at phone width. Twenty-two routes, every marketing page, the blog, every dashboard page, the composer and onboarding, all at 390×844. Two classes of defect to catch. Content pushed out of reach, and controls too small to hit with a thumb.

An agent cannot look at a screen. It can run a script that measures the DOM and it can save a PNG that nobody opens unless the script says something is wrong. So the script is the eyes, and every property of a real reviewer that it does not have has to be written into it explicitly: knowing the page finished rendering, knowing which page it is on, knowing which elements belong to the product.

None of my first four versions had all three.

Attempt 1: the assertion everybody writes

The design document asked for document.documentElement.scrollWidth <= window.innerWidth. It is the check you find in every blog post about horizontal overflow, and in this app it is meaningless.

body carries overflow-x: hidden app-wide. CSS Overflow specifies what happens next: when html is visible and has a body child, the user agent applies the body's overflow values to the viewport, and "the element from which the value is propagated must then have a used overflow value of visible." Clipping moves to the viewport, body reports itself unclipped, and scrollWidth keeps returning the full content width for a page nothing can pan.

/posts reported 551px against a 390px viewport and was completely fine: the table scrolls inside its own card, and in a real touch context setting scrollLeft to 9999 leaves it at zero. Forcing window.scrollTo(9999, 0) in a desktop context does move the page, because overflow: hidden regions stay programmatically scrollable, so a second version of the same idea reported "can scroll sideways: true" for a page a user cannot pan at all.

Two more reasons I only found later. window.innerWidth includes the vertical scrollbar while scrollWidth excludes it, so on any vertically scrolling page the check silently tolerates about fifteen pixels of real overflow. And scrollWidth is an integer, so sub-pixel overflow vanishes. Three independent reasons, one popular one-liner.

Attempt 2: walk the elements instead

If the document-level number lies, measure each element. Walk everything, compare getBoundingClientRect() against the viewport, flag whatever sticks out. This is the other snippet everybody copies.

It flagged the marketing home with fourteen clipped elements. All fourteen were fine. They sat inside legitimate overflow-x-auto code samples, which scroll on purpose.

The walk had asked the wrong question. Being wider than the viewport is not a defect. Being unreachable is. Whether an element is reachable depends on the nearest ancestor that manages overflow, and my first walk broke on the first hidden or clip it found, ignoring auto and scroll on the way. Since body carries overflow-x: hidden, everything eventually walks up to body and reports as clipped.

Attempt 3: a fixed sleep

Meanwhile the shot script that fed all this waited for header and then measured 1200ms later. It reported the Dashboard as clean.

The Dashboard had a real bug: an action sidebar visible on desktop and completely absent on a phone, with no scrollbar, no affordance, and nothing on screen suggesting content was missing. The script missed it because 1200ms after header appeared, the posts table had not rendered, so it caught the page at its pre-content width.

A gate that samples before content settles produces confident false passes. That sentence went into my notes and then I proceeded to violate it twice more.

Attempt 4: wait for content, and three ways to still measure nothing

Version four waited for a real content selector, then swept six surfaces under fine and coarse pointer. Clean table. It was wrong three times over, and each failure had a different cause.

It waited on main. But the app shell supplies main before any route content exists, and the functions emulator cold-starts about ten seconds on its first call of a run. The composer, surface number one, was measured while it still read "Loading…". Both its samples reported a height of exactly 844, which is the viewport, and that number is the only reason I looked closer.

It measured a redirect. /settings had become a redirect to /design when the sidebar was reorganised, so the sweep measured Design twice and reported a seventh cleared surface. What gave it away was that the two screenshots were byte-identical.

Its selector was wider than the rule it was testing. The rule is scoped @media (pointer: coarse) { :is(.gs-app,.gs-tokens) :is(button,a[role=button],[role=tab],summary) }. The probe used a bare button, a[role=button], [role=tab], summary, which also collects the TanStack Router devtools. Those mount outside .gs-app, park about 2700px below the fold, and cannot be touched by a rule whose scope they are not in. They were most of the roughly ninety "controls" every page reported, and they alone produced /posts' apparent 349px growth under a coarse pointer: growth in a dev-only panel, read as growth in the product.

What I checked before writing version five

Briefly, because the answer is short. Playwright's ARIA snapshots capture role, name, state and hierarchy, and no geometry at all, so a snapshot passes identically whether a button is four pixels tall or off-screen. axe-core has exactly one geometric rule, target-size. Screenshot baselines would have caught the duplicate surface by accident, since a byte-identical baseline is the same tell I got lucky with, and nothing else. Vision models are the weakest option of the four: on VideoGameQA-Bench, across sixteen models, the best score at visual regression testing was 45.2%, worse than a coin flip.

One useful thing to steal instead of a tool: WCAG 1.4.10 requires content to be usable without scrolling in two dimensions at 320 CSS pixels wide, and it comes with explicit exceptions for maps, video, data tables and editing toolbars. Those are the exact cases my hand-rolled gate broke on. Where a standard already defines the criterion, use theirs.

Version five

Three changes, and the third is the one I would take to any other project.

Ask about reachability, not width. Walk to the nearest overflow-managing ancestor and let it decide, treating auto and scroll as reachable:

let unreachable = 0
for (const el of document.querySelectorAll(':is(.gs-app,.gs-tokens) *')) {
  const r = el.getBoundingClientRect()
  if (r.left < vw || r.width <= 30) continue
  let n = el.parentElement, ok = false
  while (n && n !== document.body) {
    const ox = getComputedStyle(n).overflowX
    if (ox !== 'visible') { ok = ox === 'auto' || ox === 'scroll'; break }
    n = n.parentElement
  }
  if (!ok) unreachable++
}

Wait for the numbers to stop moving, not for a clock. Poll the exact signature the probe is about to read, and require two identical samples in a row:

await page.waitForFunction(() => {
  const app = document.querySelector('.gs-app')
  const sig = `${Math.round(app?.getBoundingClientRect().height ?? 0)}:${
    document.querySelectorAll(':is(.gs-app,.gs-tokens) :is(button,a[role="button"],[role="tab"],summary)').length
  }`
  window.__stable = sig === window.__prev ? (window.__stable || 0) + 1 : 0
  window.__prev = sig
  return window.__stable >= 2
}, { timeout: 12000, polling: 400 })

Make the probe prove it looked. This is the part no tool I found does for you. Every run now returns evidence of observation alongside its verdict, and the evidence is printed in the results table rather than merely collected:

mainChars: (document.querySelector('main')?.innerText || '').trim().length,
stillLoading: /\bLoading[….]/.test(document.querySelector('main')?.innerText || ''),

Plus three rules that turn my lucky accidents into conditions:

Version five cleared all twenty-two routes for unreachable content and reported that the 44px coarse-pointer rule squeezes nothing anywhere in the app. The Dashboard sidebar bug was caught earlier, once the drawer script waited for the page's widest element instead of a clock. I believe both results now, because every run also prints how much rendered content each page had at the moment it was measured.

The general form

Mutation testing is the only established discipline aimed at this question, and where anyone has measured end-to-end suites the results are poor: a 2021 study using operators derived from 250 real bug reports found real suites killing about 20% of mutants, with event-handler mutations killed 0% of the time. Facebook ran 15,000 mutants against its full unit, integration and system suites and watched more than half survive. There is no term of art for a negative control in test engineering; search for one and you get "negative testing", which means feeding invalid input and is a different thing.

So the rule I now apply to any automated check, mine or bought: it must return a verdict and evidence that it observed the thing it judged. Every tool on the market returns the verdict and assumes the rest.

There is a 2026 paper on LLM judges called Reliability without Validity. It measures judges with test-retest reliability above 0.95 while their position bias exceeds 0.10. A judge can be perfectly stable and perfectly wrong.

My check said "all clean" three times. It was extremely reliable.