Visual Regression Testing

Visual regression compares each step’s screenshot against a stored baseline during test execution. Steps that drift beyond configured thresholds mark the step and the run with a visual verdict (warn or fail), and a failing step aborts the rest of the run.

Setting it for a group

A new group starts with the screenshot check on: a step whose screenshot differs from its baseline by 30% or more gets a warn verdict, and no fail threshold is set. You can change the thresholds or switch the check off for each group.

Open a group on the Tests page and choose Group settings → Screenshot check. It has two independent checkboxes, each with a numeric percentage input:

• Warn if differs by more than — 30% when you tick it. When checked, any step whose screenshot differs from its baseline by at least this percentage gets a warn verdict.
• Fail if differs by more than — 50% when you tick it. When checked, any step that crosses this threshold gets a fail verdict; the step fails and the run aborts at that point.

If the warn threshold is set at or above the fail threshold, the dialog shows Warn threshold should be lower than fail threshold. — fix it before saving.

Unchecking both checkboxes switches visual regression off for the group. No baselines are written, no comparisons run, no cost is added at execution time.

A step can use the group default, turn the check off, or set its own thresholds: expand the step on the test page and choose under Screenshot check. See Per-step overrides.

Equivalent API call:

PATCH /api/v1/tests/suites/{suite_id}
{
  "visual_regression_warn_pct": 30,
  "visual_regression_fail_pct": 50
}

Set either field to null to disable that tier. The warn threshold must be lower than the fail threshold when both are on: the group settings dialog will not save otherwise.

Per-step overrides

Group thresholds are the default for every step, but individual steps can opt out of visual regression or use their own thresholds. This is the escape hatch for screens that never match a baseline (live video, channel feeds) or that vary within known bounds (timestamps, rotating banners) and need looser tolerances than the rest of the group.

Open a test case and expand a step and find Screenshot check. Its button reads Group default until you change it. Click it to open the per-step popover with three radio options:

• Use group defaults — the step inherits the group’s warn/fail thresholds. This is the default; no badge is shown. The popover hint displays the current group values (e.g. warn 30%).
• Disabled — visual regression is skipped entirely for this step. No baseline is written and no comparison runs. The button reads Screenshot check off, struck through.
• Custom thresholds — set this step’s own warn and/or fail percentages, using the same two checkboxes-with-percent inputs as the group-level controls. Enable at least one tier (custom mode with both disabled is rejected — use Disabled instead). If the warn threshold is at or above the fail threshold, Apply stays disabled until you fix it. The button reads Screenshot check 40/70% (a — means that tier is off).

A custom override wins even when the group has visual regression turned off — a single step set to Custom 10/25 is still evaluated while every other step is skipped. Conversely, a step set to Disabled is skipped even when the group has thresholds configured.

Read-only views (run history, the step timeline) show the badge only when a step carries a non-default override, so an all-default test stays uncluttered.

How the override is stored

The override is an optional visual_regression object on each step of a test, set through the API. A step with no visual_regression field inherits the group defaults.

{
  "action": "tap",
  "visual_regression": { "mode": "custom", "warn_pct": 40, "fail_pct": 70 }
}

Validation runs on POST/PATCH of a case (422 on a bad shape): mode must be one of the three values; warn_pct/fail_pct are only allowed when mode is custom; custom requires at least one threshold; and when both are set, warn_pct must be lower than fail_pct. Reads are forgiving — an unknown or malformed shape falls back to group defaults rather than erroring.

Overrides apply with every AI use: Smart, No AI and AI only.

Reordering or removing steps

A step’s visual regression baselines and its recording screenshot belong to the step itself, not to its position. Moving a step earlier or later, or inserting a new step before it, never changes which references it uses. Removing a step deletes its references. A new step starts without references.

How the drift percentage is computed

VibeView compares each step screenshot against the baseline in three tiers:

1. Perceptual hash — byte-identical or near-identical frames short-circuit to 0% instantly.
2. Alignment — the global offset between the two frames is estimated (scroll replays can land a few pixels off; TV rows scroll horizontally). For steps that performed a scroll or swipe, any offset is accepted; for all other steps the offset is only accepted up to ~12px — a larger shift means something moved the layout, and it is reported as drift.
3. Changed pixels — a pixel counts as changed if it can’t be explained by the other frame at an accepted alignment: it must fall outside the local texture envelope and match local average brightness. The local-average check catches different content occupying the same busy region (a different poster or paragraph), which texture tolerance alone can miss.

This means a replay that lands 4px lower on the page, a status-bar clock tick, or re-encoded image quality do not inflate the drift score. Real changes — a missing element, changed text, a moved tvOS focus ring — are still counted at their true size. The same logic applies to every platform (iOS, Android, tvOS, Android TV, Roku).

Dynamic content

Auto-advancing carousels, video thumbnails, and timestamps genuinely change between runs and will contribute real drift. Either keep your warn threshold above their typical contribution (the default 30% usually accommodates a banner carousel) or split volatile screens into their own steps with a per-step visual_regression threshold override.

How each step is classified

When either threshold is configured, the backend runs a comparison after each step. The changed_pixel_pct it returns is classified:

• If the fail threshold is configured and changed_pixel_pct >= fail_pct → fail.
• Else if the warn threshold is configured and changed_pixel_pct >= warn_pct → warn.
• Otherwise → ok.

Fail takes precedence when both thresholds would fire.

On fail, the step is marked failed, an error message is recorded, and the run aborts at that step. Remaining steps are skipped.

On warn, the step records the visual verdict and a warning but execution continues.

On ok, nothing user-visible happens.

Reference image selection

For each step, visual regression looks for a reference in this order:

1. Stored baseline — the baseline accepted for this step on this device (model, OS version and, on a foldable, posture). Created from a clean passing run when a threshold is enabled, or when you accept one.
2. Recording screenshot — the recording’s screen right after this step. Before any baseline exists for the device, it stands in as the reference, but only when the test was recorded on the same platform and a screen of the same shape (a phone recording is not a reference for a tablet).
3. Skip — if neither exists, the step records visual_verdict: null and execution continues normally. A test recorded on an iPhone and replayed on an Android phone therefore has no reference on the first Android run; the step shows no comparison until you accept a baseline for that device (Set as baseline on the run, or Accept as baseline on a step).

A baseline from another device or OS version is never used as a stand-in.

If a step’s screenshot came from a device with different dimensions than the baseline, the comparison resizes before comparing, so cross-device replay still works. Comparisons always run at the larger of the two image resolutions, so low-resolution recording references (stored at 328px width) don’t dilute sensitivity.

A recording reference taken on an iPhone or iPad while the device was turned to landscape is compared the right way up, even when it was recorded before recording screenshots were saved upright. A step that comes from a setup step is compared against the setup step’s own recording, and the run page names the device it was recorded on.

Once you accept a drifted step, the run page shows it as Accepted as baseline and stops offering Accept for it. A step that was not checked (for example in a No AI run) has no screenshot comparison and shows no drift.

On a foldable device (the iPhone Duo, or an Android foldable such as the Pixel 9 Pro Fold), each step is compared against — and, when accepted, stored as — the baseline for the posture the device was in when that step’s screenshot was taken. A test that captures a step folded, unfolds the device, and captures the next step keeps the two in their own posture’s baseline set.

The reference image on run pages

Every compared step keeps the exact images it was compared with: the reference, the live screenshot, the diff overlay (warn or fail only) and the recording screenshot. Accepting new baselines later does not change what a past run displays — the score, diff overlay and reference image always agree, whether the step passed or drifted.

Older runs, recorded before baselines followed their steps, kept a frozen reference only for steps that drifted. Their other steps show the current baseline, on the run’s device, of whichever step is now at that position in the test, which may be a different step if the steps were reordered since. The same goes for the recording screenshot shown beside a step of an older run: it is the one of the step now at that position, so after a reorder it can belong to another step.

How the run verdict aggregates

Every step’s verdict rolls up into a single visual_regression_status on the run, with this priority: fail > warn > ok > null.

• fail — at least one step hit the fail threshold.
• warn — no fails, but at least one step hit the warn threshold.
• ok — at least one comparison ran, and all were ok.
• null — no comparisons ran (thresholds disabled, or no reference screenshot existed for any step).

Group runs aggregate the same way one level up: a group run’s visual_regression_status is the max-severity verdict across its case runs (same fail > warn > ok > null priority), computed from the case runs at read time rather than stored. A group run whose tests all passed but where any test drifted shows the orange “passed” everywhere a group run appears, matching the test rows inside it.

Mid-run comparison

Every run (Smart, No AI and AI only) compares each step’s screenshot as soon as the step completes. That means:

• A step’s visual verdict is known before the next step starts.
• A fail verdict aborts the run immediately — no wasted time on the rest of the test.
• The step timeline on the run detail page shows the visual badge per step in real time.

Accepting a new baseline from run detail

When your UI intentionally changes, the baseline needs updating. There are two ways to do this from the run detail page:

• Per step — in the step timeline, a step with a warn or fail visual badge shows an Accept as baseline button beside the baseline and live screenshots. Clicking it asks you to confirm, then replaces only that step’s baseline with the screenshot from this run. Use when drift is localized. It works on the steps of a setup (Before this test) too: the baseline is saved on that setup test, so every test that uses the setup compares against it.
• Whole run (all steps) — the Set as baseline button in the bar at the top of the run detail page (shown only when the run passed) opens a confirmation. Confirming replaces the baseline of every step in this run with its screenshot from this run. Use when most of the test’s screens have changed, for example after a UI redesign.

Both per-step accepts happen inline: a drifted step opens with its baseline and live screenshots side by side and the accept button between them, so you can see what changed before you accept it.

Both actions are developer-role scoped and audit-logged. The underlying endpoints:

POST /api/v1/tests/runs/{run_id}/set-baseline                # bulk
POST /api/v1/tests/runs/{run_id}/set-baseline/drifted        # drifted steps only
POST /api/v1/tests/runs/{run_id}/set-baseline/{step_index}   # per step

The /drifted variant updates baselines only for the steps whose visual verdict was warn or fail, leaving clean steps untouched. In the per-step endpoint, step_index is the step’s position in that run.

Accepting works the same for test cases inside and outside a group. Accepting from a run always updates the step that produced the screenshot, even if you have since moved it. If that step has been removed from its test case, a per-step accept is refused with 409 and code step_removed; a whole-run or drifted accept skips removed steps. If the test case or setup step the step belongs to has been deleted, a per-step accept is refused with 409 and code case_removed; a whole-run or drifted accept skips those steps. Older runs, recorded before baselines followed their steps, can’t be accepted from: the request is refused with 409 and code run_predates_step_ids — re-run the test to accept from it. A step whose screenshot can no longer be read is refused with 422 and code unreadable_screenshot. Refusals carry detail: {"code": ..., "message": ...}; see Tests API for every code.

The tri-state banner on run detail

The run detail page shows a banner above the step timeline based on the run’s aggregated visual_regression_status:

• fail (red) — reads Visual regression, failed, followed by “A step differed past the fail threshold and the run stopped there.” The failing step is visible in the timeline below.
• warn (amber) — reads Visual regression, warn, followed by “Some steps drifted past the warn threshold.” The run completed; review the flagged steps and accept new baselines if the drift is expected.
• ok / null — no banner shown.

From either state, use the per-step Accept as baseline buttons or the whole-run Set as baseline action to update baselines.

Run status in lists

A run whose visual verdict is warn still shows as passed, in amber, in the run lists on the Tests page, so drift is visible without opening the run. A fail verdict marks the run failed. Open the run to see which steps drifted.