ci: keep visual-regression baselines refreshing and surface Cypress failures - #2669
Open
balzss wants to merge 5 commits into
Open
ci: keep visual-regression baselines refreshing and surface Cypress failures#2669balzss wants to merge 5 commits into
balzss wants to merge 5 commits into
Conversation
…ailures The baseline job ran Cypress without continue-on-error, so failing a11y/console assertions aborted the job before the "Push baselines" step — silently freezing the visual-baselines branch on every merge since the dark/light theme suite landed. Decouple baseline capture from the assertion gate: Cypress runs with continue-on-error, the screenshots always flatten and push (if: always()), and a trailing step fails the job for visibility if Cypress failed. Baselines now refresh on every merge regardless of the a11y/console outcome. Also add a "Cypress assertions" pass/fail line to the sticky PR comment so failures are visible even when the visual diff is clean, and remove the leftover TEMP diff-demo fixture step and regression-test/cypress/diff-demo/. Refs: INSTUI-5137 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
Contributor
Visual regression reportCypress suite: ❌ Failing — see the run logs. Visual diff:
Accessibility (axe):
📊 View full report — click a screenshot's ⚠ badge to see each violation boxed on the image, with the offending element named and contrast failures shown as color swatches. Diff images (32)alert.png — baseline no longer producedavatar.png — baseline no longer producedbadge.png — baseline no longer producedbillboard.png — baseline no longer producedbreadcrumb.png — baseline no longer producedbutton-and-derivatives.png — baseline no longer producedbyline.png — baseline no longer producedcalendar.png — baseline no longer producedcheckbox.png — baseline no longer producedcheckboxgroup.png — baseline no longer producedcolorpicker.png — baseline no longer producedcontextview.png — baseline no longer producedcustom-and-lucide-icons.png — baseline no longer produceddateinput-dateinput2.png — baseline no longer produceddatetimeinput.png — baseline no longer produceddrilldown.png — baseline no longer producedfiledrop.png — baseline no longer producedform-errors.png — baseline no longer producedheading.png — baseline no longer producedimg.png — baseline no longer producedlink.png — baseline no longer producedmenu.png — baseline no longer producedmetric-pill-tag-timeselect-text.png — baseline no longer producedoptions.png — baseline no longer producedpagination.png — baseline no longer producedprogressbar.png — baseline no longer producedselect-simpleselect.png — baseline no longer producedtable.png — baseline no longer producedtabs.png — baseline no longer producedtooltip.png — baseline no longer producedtreebrowser.png — baseline no longer producedview.png — baseline no longer producedBaselines come from the |
The regression suite already ran axe per page and theme, but the violations were only printed to the Cypress log and discarded. Persist them and render them in the report so designers can triage accessibility issues alongside the pixels. - regression-test: a new recordA11y Cypress task writes cypress/a11y.json keyed by "<slug>-<theme>" (mirroring meta.json); the spec records each violation's rule id, impact, help text, helpUrl, and offending node selector/summary. - ui-scripts visual-diff: a new --a11y flag reads that file and adds a per-row "a11y" badge, an "A11y" filter chip, a header count, and a collapsible list of violations (rule linked to its axe helpUrl, impact, selector, contrast summary) per screenshot. a11yPages/a11yViolations are added to summary.json. - visual-regression.yml passes --a11y cypress/a11y.json to the diff step. Additive: without --a11y the report is unchanged. Unit-tested esc() and a11yFor(), and verified end to end against a real suite run (15 flagged pages). Refs: INSTUI-5137 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… HTML view Builds on the a11y report data: make the violations visible without hunting. - Expand each row's violation list by default (Tier 1 was collapsed). - In the lightbox HTML view, outline the offending nodes inside the live iframe using the recorded axe selectors, and add a legend that lists each violation and scrolls/flashes its element on click. - Embed the a11y data as a JSON block for the client; same-origin iframe access means no pixel math. Runtime-generated selectors (Emotion hashes, generated ids) resolve because the iframe app and the axe run share one build; if they ever diverge the outline no-ops but the legend and text details still show. Refs: INSTUI-5137 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The a11y section of the visual-diff report was a nested list of axe rule ids, CSS selectors, and raw failureSummary prose, repeated per theme. Nothing was anchored to the image, so the artifact designers actually read - the screenshot - carried no accessibility information at all. Capture: while the page is still rendered, record each violating element's bounding box, a human-readable name, and the contrast numbers axe already computed. a11y.json gains a per-screenshot page size so boxes can be positioned as percentages and hold at any rendered scale. Report: numbered boxes over each offending element on the "Actual" image, colored and ordered by impact; a card per finding leading with plain language, the element's name, a suggested fix, and - for contrast - swatches with the measured and required ratios; a run-level overview grouped by rule that collapses the three themes into one line; impact and rule filters that narrow boxes and cards together; and a lightbox A11y mode pairing the full-size screenshot with its findings. Raw axe output moves behind a Technical details disclosure. The PR comment now names the failing rules in a table instead of pointing at the run logs. Reports generated from a11y.json captured before this change still render: the boxes are omitted, and contrast is recovered by parsing axe's prose.
Feedback on the first pass: the report showed the same findings three times over - boxes on every "Actual" thumbnail, a card list under each row, and outlines in the HTML view - which buried the image comparison the report exists for. Findings now live in one place, the lightbox's A11y view, reached by clicking a row's badge. The row is back to a three-up comparison plus a "N a11y" pill colored by worst impact. Removed: the a11y overlay and legend from the HTML view (it iframes the live page again, nothing more), the impact filter chips, the "Show markers on screenshots" toggle, and the embedded a11y JSON payload that only the HTML view's legend needed. Boxes and cards are rendered into an inert <template> per row and cloned when the A11y view opens, keeping one renderer. Markers carry their own rule so the overview's rule filter no longer needs a rendered card to consult. Also fixes the contrast violations the report surfaced in the test pages themselves - 403 violating elements down to 15. Two causes, both in the pages rather than the library: custom-icons styled its labels with Tailwind literals (text-gray-500 / text-gray-700) that ignore the theme and failed on light and dark alike, and eleven pages had bare text nodes inheriting the browser default black, unreadable on the dark theme's near-black surface. Both now use Text/Heading, which take their color from the active theme. The 15 remaining are deliberate: inverse/on-color text demos rendered on the page background, text on View's colored alert/success/danger/warning surfaces, a color="inherit" demo, a raw input in a FormField demo, and FileDrop's own renderLabel - that last one is a library finding, not a page one. These twelve pages will show as Changed on the next run; baselines refresh on merge.
balzss
force-pushed
the
fix/visual-regression-baseline-pipeline
branch
from
July 31, 2026 13:37
c85621b to
dc057df
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Repairs the self-hosted visual-regression pipeline and makes the report far more useful for triage. Two related concerns, in separate commits:
1. Baseline pipeline fix (
ci)visual-baselines.yml: decouple baseline capture from the a11y/console gate — Cypress runs withcontinue-on-error, screenshots always flatten + push (if: always()), and a trailing step fails the job for visibility if Cypress failed. Fixes the silent baseline freeze: thevisual-baselinesbranch had stopped updating on every merge since the dark/light theme suite landed (Cypress aborted the job before the push step).visual-regression.yml: the sticky PR comment now leads with a "Cypress assertions (a11y + console errors)" ✅/❌ line, so failures are visible even when the visual diff is clean.TEMP inject diff-demo fixturestep andregression-test/cypress/diff-demo/.2. A11y in the visual-diff report (
feat)recordA11yCypress task now persists them tocypress/a11y.json, keyed by<slug>-<theme>(mirrorsmeta.json).ui-scripts visual-diffgains a--a11yflag that renders a per-page⚠ N a11ybadge, an⚠ A11yfilter chip, a header count, and a collapsible violation list (rule linked to its axehelpUrl, impact, offending selector, contrast summary).a11yPages/a11yViolationsadded tosummary.json. Additive — without--a11ythe report is unchanged.--a11y cypress/a11y.jsoninto the diff step.Test Plan
visual-diffsuite (34 tests, incl. newesc/a11yFor).a11y.json(15 flagged page/theme entries) and the report renders badges/filter/list with the real axe contrast data.visual-baselineseven while the suite is red, and ends red with the::error::annotation.color-contrastviolations remain (tracked in INSTUI-5137) — the suite stays red-but-visible until those are fixed by design/component work.Fixes INSTUI-5137
🤖 Generated with Claude Code