Skip to content

test(webui): wait for the shell's images before comparing baselines - #18

Closed
UncleMoc wants to merge 1 commit into
mainfrom
fix/responsive-baselines-wait-for-images
Closed

UncleMoc wants to merge 1 commit into
mainfrom
fix/responsive-baselines-wait-for-images

Conversation

@UncleMoc

Copy link
Copy Markdown
Collaborator

Change

responsive-baselines.spec.mjs captured the shell before its images had decoded. The composer's mascot is drawn from /assets/, and the screenshot compares pixels: a capture taken while that image is still loading shows the broken-image placeholder where the mascot belongs, and the comparison fails on pixels the code never drew.

Every other baseline in this directory already waits for the images before it captures. This one did not, so it captured whichever state the network happened to be in -- and on a loaded runner that state is sometimes "not finished".

How it was found

The Windows job of the WebUI CI on main failed once with 4197 differing pixels on the shell at 900px. Downloading the job's browser diagnostics artifact and comparing the two captures pixel by pixel:

  • expected and actual were both 894x900 -- the layout was identical;
  • all 5614 differing pixels fell inside the mascot's rectangle (x733..832, y671..745); nothing outside it differed;
  • sampling that rectangle: the committed baseline held the pink mascot (224, 131, 138), the capture held the placeholder (255, 255, 255) and (192, 192, 192).

So the baseline was right and the capture was taken too early. The fix is the wait the other specs already use: every /assets/ image reports complete with a natural width, so it has decoded rather than merely been requested.

What does not change

No assertion changes, no pixel budget changes, no baseline re-recorded. The comparison still fails on any real difference.

Verification

  • The full WebUI unit and integration suite passes on this branch (4485 passed, 0 failed).
  • The browser spec's behaviour on a machine whose scrollbar width differs from the recorded baseline is unchanged: the same 10 failures with the same 6px width difference and the same pixel counts before and after this change.
  • The Windows CI is the judge here -- it is the runner that loaded slowly enough to capture the placeholder.

The composer's mascot is drawn from /assets/, and the shell screenshots
compare pixels: a capture taken while that image is still loading shows the
broken-image placeholder where the mascot belongs, and the comparison fails
on pixels the code never drew.

Every other baseline in this directory already waits for the images before
it captures. This one did not, so it captured whichever state the network
happened to be in, and on a loaded runner that state is sometimes "not
finished". The Windows job failed this way once: expected and actual were
both 894x900, and every one of the 5614 differing pixels fell inside the
mascot's rectangle -- the baseline held the pink mascot, the capture held
the placeholder.

The wait below is the same one the other specs use: every /assets/ image
reports complete with a natural width, so it has decoded rather than merely
been requested. No assertion and no pixel budget changes.
@UncleMoc

Copy link
Copy Markdown
Collaborator Author

Superseded by #16, which fixes the same root cause and applies it to every baseline spec, not only the responsive ones: the shell's mascot image is a <picture> that starts on the poster PNG and upgrades to the WebP, so a frame taken inside that window shows the browser's broken-image placeholder instead of the picture. #16 also gives the browser-frame wait its own budget, which covers the other half of the macOS flake.

Closing this one so the tree keeps a single fix for the cause rather than two.

What was checked before closing: the combined #16 + #17 tree passes the WebUI unit and integration gate locally (4485 passed, 0 failed) and its browser suite fails on exactly the same 40 local-environment cases as main does (byte-identical failure lists), so the two PRs add no regression of their own. The three-platform CI on both #16 and #17 is green.

@UncleMoc UncleMoc closed this Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant