The Tests page

Tests in the sidebar opens your tests, one app at a time. Tests run your app on live devices: iPhone and iPad, Android phones and tablets, Apple TV, Android TV and Roku. AI testing is in beta.

Anyone in your organization can open tests, groups and runs. Running a test or group, cancelling a run, accepting baselines, generating a test from a recording, and creating, editing, reordering, copying, quarantining and deleting tests and groups needs the Developer or Admin role, so a Viewer doesn’t see those controls.

Pick an app

The app name at the top is a switcher. Pick an app to see its tests; the page remembers the last app you opened. An app’s own page has a Tests button that opens this page for that app. Tests and groups that belong to no app are under No app, which shows up in the switcher only when there are some.

Apps you have linked as the same product (see Apps) are one entry in the switcher, with their platforms listed under the name. The page then shows the product, not one of its apps: when the form factor you are viewing has more than one platform, Tests here run on iOS and Android (for example) under the product name names them, and the lists, runs, attention items, notes and test data cover all of them. A link to any of the linked apps opens the product. When a product has both phone and tablet apps and TV apps, Phone & tablet and TV switch between their tests. A test shows the platform of the app it belongs to when the list spans several apps. Its Last result shows one mark per platform it runs on, worked out per test.

Groups from different platforms are not merged: a group with the same name on iOS and on Android shows twice, side by side, each labelled with its platform. A test or group runs on every linked platform of its form factor; pick the devices in the Run dialog. Run all runs every test of the product for the form factor you are viewing. Each device installs its own platform’s app. Environments belong to the product, so the one you choose exists on every platform. Schedules saved from any linked app of the same form factor show across the product.

When VibeView finds apps that look like one product, a banner on the Tests page offers to link them; choose Not now to dismiss it.

Sections

• Tests — every test of the app, with a left column of views and groups.
• Runs — the last 30 days at a glance, then every run of the app’s tests.
• Schedules — the app’s scheduled runs: runs that start every day or on every new build.
• App notes & data — what the AI should know about the app, the values tests use, environments, and replay paths.
• CI — prompts for your coding agent, CLI commands and a GitHub Actions workflow.

The left column

• All tests — every test of the app except setup steps.
• Needs attention — tests where one of the last 3 runs failed because of the app or the test, and new routes waiting for your review. Cancelled runs, runs stopped for usage credit, and runs that ended because the device session ended don’t count. Each test shows why in one line.
• Not in a group — tests that are in no group.
• Groups — each group with its number of tests. Click + to make a new group.
• Setup steps — reusable tests that run before other tests, with how many tests use each one. See Setup steps.

The test list

Each row shows the test’s name, its groups (or why it needs attention), its last result on each platform, how it runs, its last five results, its average duration over 30 days and when it last ran. A test that is flaky or quarantined says so next to its name.

Last result has one pill per platform: ✓ when the last run on that platform passed, ✗ when it failed, and – when the test hasn’t run there yet. A run counts for the platform of the device it ran on, so an iOS test you ran on an Android phone shows under Android. Runs that were cancelled, not checked, hit an error, stopped for usage credit or ended with the device session don’t change the mark. Click a pill to open that run.

How it runs is one of:

• Recorded — it replays your recording; the AI takes over a step whose screen changed.
• Learned path — it replays the actions saved from a passing run; checks still use AI.
• AI steps — the AI works out every step until a passing run saves a path.

Above the list:

• Search — press / to jump to it.
• Status — the last run passed, failed, or the test hasn’t run yet.
• Tags — tags belong to groups, so a tag shows the tests in groups with that tag. Each tag’s number is how many tests in the current list it would show; tags that would show none are left out, and a long row folds behind +N more. With two or more tags picked, choose Any (a group has one of them) or All (a group has every one). A group’s own page has no Tags filter: its tags show next to its name, and you change them in the group’s settings.

All tests starts on Recent activity: the tests you last ran, created or edited come first, so a test you just wrote sits at the top even before its first run. A group lists its tests in the group’s order, and the other views list tests by name.

To sort the list, click the Test, Avg time or Last run heading. Click again to reverse the order, and a third time to go back to the view’s usual order. Last run shows the newest first, Avg time the longest first, and tests that haven’t run yet always come last. On a narrow window only the Test heading shows. While the list is sorted by a heading, a Sorted by chip above the list says how; click its × to go back to the usual order. The sort is kept in the page address, so a reload or a shared link shows the same order.

Hover a row and click the play button to open the Run dialog for that test. Tick rows to run them together with Run in the bar that appears, or to add them to a group with Add to group. Run all in the page header runs every test of the app.

Each row’s ⋯ menu has:

• Duplicate — a copy of the test in the same app and groups.
• Copy to another app — for another app of the same kind (phone and tablet, or TV), such as a white-label version. You can put the copy in one of that app’s groups (including those of apps linked to it as the same product) or a new one.
• Add to group and, in a group, remove it from that group.
• Quarantine — the test keeps running and recording results but leaves Needs attention and the flaky-test notification. End quarantine puts it back.
• Delete test — asks first; its run history goes with it. Runs of that test alone leave the Runs list, and a group run or a run of several tests keeps the other tests’ results and no longer lists the deleted one.

Create test in the header opens Describe (the AI drafts steps from a sentence), Record (Devices records you using the app) and Explore (the AI uses the app on its own and reports what looks broken; see Explore). Every way ends on Review steps, where you name the test and pick its groups, the tests to run before it and its start state. Inside a group, the new test joins that group unless you remove it. You can also create a new group right there. See Creating test cases.

On a product (apps you linked as one), Create test names the product. Record opens Devices with the product’s app for the platform of the device you last ran on from the Run dialog (for example Android after a run on a Pixel), and the test belongs to that app. Describe saves the test under the product’s iOS app (Apple TV for TV), or its first platform when it has no iOS (Apple TV) app; Record opens that same app when you haven’t run on a device of the product yet. The group pickers list the product’s groups for the form factor, each with its platform, and a test can join a group made from another of the product’s apps.

The Run dialog remembers the device you last ran on for the whole product, whichever of its apps you open it from. Before your first run there, it picks the device of the product’s most recent run, then the newest phone.

Groups

A group is a folder of tests. A test can be in several groups, with its own place in each. Deleting a group never deletes its tests: a test in no other group moves to Not in a group.

To put tests you already have in a group, open the group and click Add tests. Search by name, tick the tests you want and confirm: they go at the end of the group. Tests already in the group show ticked. On a product, the list holds every test of the form factor you are viewing, each with its platform.

Opening a group shows its start state, Run group (it opens the Run dialog), and a summary of the last 30 days: pass rate, runs, average duration, cost, tests and flaky tests. Cancelled runs don’t count, and a value with no data shows —. The group’s ⋯ menu has Group settings and Delete group.

Group settings (the group’s ⋯ menu, or the start state next to its name) has:

• Name, description and tags. CI can run every group with a tag.
• Start each test from:
  • Fresh install — deletes and reinstalls the app. Use for onboarding and first-launch tests.
  • Clean app data (default) — the app opens as if just installed: logged out and empty. Right for most tests.
  • Restart the app — keeps login and saved data. Use when tests assume you are already signed in.
  • Where the last test stopped — no reset. Tests run as one journey, so one failure can break the ones after it.
• Test order — move tests up or down. The order matters when tests start where the last one stopped.
• Screenshot check — on by default for a new group, warning when a step’s screenshot differs from its baseline by 30% or more. Change the warn and fail thresholds for every step of the group, or switch the check off (see Visual regression).
• Extra notes for the AI — added to the app notes for tests in this group.
• Run from CI — the command that runs this group.

A group with a schedule shows it next to its name, for example Daily 02:00; click it to open the schedules.

Runs

The Runs section opens with four tiles for the last 30 days, each compared with the 30 days before: pass rate, test runs (a group run counts once for each test it ran), tests failing now and average duration. Below them, a Runs per day chart shows passed and failed runs per day (failed bars are also striped); click a day to see only its runs. Needs attention sits next to the chart.

The run list shows every run of the app’s tests — whole groups, single tests and runs started from the Run dialog — with what ran, its status, tests passed, the platform it ran on (when the app is linked to apps on other platforms, or has a same-ID app on another platform), device, what started it, duration, cost, model and time. A run on several devices is one row, with the device column reading, for example, “2 devices”; click it to see each device. A run on several devices lists each platform, for example iOS · Android. The Trigger column reads Dashboard, CI, CLI, Coding agent, Daily, New build or Schedule (the last three are scheduled runs; Schedule is one started with Run now). Filter by status, group or trigger: Dashboard, CI, CLI, Coding agent, Schedule or API (any token). Click a run to open it; a test’s run page has Edit test, Re-run (the Run dialog with the same device, build and AI use) and, while the run is going, Cancel run; see Running tests. Explore runs are listed too, as Explore, with their findings count instead of tests passed; a status filter leaves them out, and they never count toward the tiles, the chart or Needs attention.

App notes & data

• App notes — what the AI should know about the app in every test, such as “a profile picker appears after login”.
• Test data — values your tests use, such as a test account. Write {{key}} in a step to use one. Mark a value as a secret and it is never shown again; runs, results and exports show it masked. A one-time code value holds the code’s setup key and types the current code. See Test data, secrets and environments.
• Environments — named sets of values that replace the app’s test data, such as a staging account.
• Replay paths — whether passing runs save the AI’s actions to replay next time, and how many steps and how much AI spend that saved in the last 30 days.

For a linked product, this section is one set for the whole product:

• Test data has a Shared value for each key and an override column for each platform of the form factor you are viewing. An empty override reads uses shared: fill one in only where a platform needs its own value. Saving the shared value into an override, or emptying it, makes that platform use the shared value again; for a secret, use Use shared. Delete removes a key on every platform. Shared values also apply on the product’s other form factor, which the section names.
• App notes has Every platform, then Only on each platform. A run gets the notes for every platform, then the notes of its device’s platform.
• Environments belong to the product: adding, renaming or deleting one does it on every platform, and its values have the same shared and override columns.
• Replay paths is one setting for every platform of the product, and its figures cover the product’s tests on the form factor you are viewing.

Schedules

The Schedules section lists the app’s Scheduled runs, with Run now, Pause, Edit and Delete; see Scheduled runs.

CI

The CI section points to Schedules and has prompts that ask your coding agent to write tests or set up CI, the CLI commands for this app’s tests and groups, and the GitHub Actions workflow template. See CI testing.

Old links

Links to a suite’s page open its group here, and links to an app’s old Test suites tab open this page for that app.