Run a test, a few tests you picked, a group, or every test of an app, on one or more live devices at once: iPhone, iPad and Android phones and tablets, or Apple TV, Android TV and Roku for TV apps. AI testing is in beta.
Start a run
On the Tests page, Developers and Admins can start a run from:
- the play button on a test’s row (hover the row), or Run test on the test’s own page;
- Run in the bar that appears when you tick several tests;
- Run group at the top of a group;
- Run all in the page header.
Each opens the Run dialog. Run starts the run and opens its page. You don’t need a session of your own: the run starts its own devices.
Devices
Pick one or more devices, up to 10. Devices are grouped by platform and form factor. A new dialog ticks one device for you: the one you last ran this app on, otherwise the newest phone. For apps you linked as one product, the dialog remembers one device for the whole product; before your first run from the dialog, it picks the device of the product’s most recent run. Tick more to run on several at once; the button then reads Run on N devices. Phone and tablet tests list phones and tablets; TV tests list TVs. A test never runs on the other form factor. If you have the same app on another platform of that form factor (the same bundle id or package name), that platform’s devices are listed too, and those devices run your app’s version for that platform.
Each device gets its own session, and the devices run at the same time. On each device the tests run one after another, in group order. A run of several groups runs each group in turn, and tests in no group come last.
Every device uses one of your plan’s concurrent sessions. When none is free, the device waits in the queue and starts as soon as one is. The dialog tells you how many devices will wait. A run uses streaming minutes for each device, like any session.
Build
Build is Latest by default: the newest build that can run on its own. Latest skips debug builds of a React Native app that load their JavaScript from a Metro dev server, because a test run has no Metro; the list marks them Debug · needs Metro. If every build of the app is one of these, Latest has nothing to run and the run is refused with a message asking you to upload a release build. You can still pick a debug build from the list yourself, but without Metro the app will most likely stop at its error screen. Pick an older build from the list instead if you need one. When you pick devices of more than one platform, each platform has its own Build list.
When
Now runs it at once. Every day and On every new build save a schedule instead, on paid plans: pick a time and time zone for a daily run, tick Also run it now for a first run straight away, and click Schedule. See Scheduled runs.
Environment
When the app has environments, Environment lets you run with one of them instead of the app’s own test data.
For an app linked across platforms, each device uses its own platform’s values (see Linked apps).
Start each test from
By default each test starts the way it is set to: its own setting, or else its group’s start state. Pick one here and it applies to every test of this run:
- Fresh install: deletes and reinstalls the app. Use for onboarding and first-launch tests.
- Clean app data: the app opens as if just installed, logged out and empty. Right for most tests.
- Restart the app: keeps login and saved data. Use when tests assume you are already signed in.
- Where the last test stopped: no reset. Tests run as one journey, so one failure can break the ones after it.
Advanced
AI use:
- Smart (the default): replays recorded and learned paths, and uses AI only where the screen changed or a step has no path yet.
- No AI: replays paths only and uses no usage credit. A step without a path is reported as not checked, and a changed screen fails the step.
- AI only: ignores recorded and learned paths and lets the AI drive every step. Use it when the app changed a lot, to find the new path.
The CLI and the API call these hybrid, replay and ai.
Stop on first failure: after a test fails on a device, the tests after it on that device are not run. They show as Not run.
Retry a failed test once (only when you save a schedule): a test that fails runs once more after a fresh start of the app, and a pass on the retry marks it flaky. Retries are off unless you tick it. See Scheduled runs.
Who pays
The last line of the dialog says how many runs you are starting, the AI model, and who pays: your usage credit, your own API key, or the free test. A No AI run uses no usage credit. When your organization can’t start an AI run (for example, no usage credit is left), the dialog says why and Run stays off until you pick No AI.
The run’s page
A test’s run page has the steps on the left and the device on the right.
- While the run is going, the device shows the live screen. You watch it; you can’t tap or type on it. Only the person who started the run (or an admin) can watch live; everyone else sees a message that the result appears when the run finishes. The page fills in as steps finish.
- After the run, the device shows the run’s video. It opens where step 1 starts; the recording begins a few seconds earlier, while the app comes to the front, and you can drag the video back to see it. A device that was turned during the run is shown upright, and the video fits the height of your window. Each step and each action under it shows when it started, and once the video is ready, click a step’s time (or its Step N button under the video) to play the video from there. Clicking the step row itself expands it. Videos are kept for 30 days. If a recording fails, the device shows the screen at the end of the run instead, and the run itself is unaffected.
- Logs shows the app’s log for the run. On iOS, Apple TV, Android and Android TV the operating system also writes messages from inside your app (network, launch measurements, media codecs, graphics and the like), often marked as errors; these system lines are hidden by default and Show system lines brings them back, dimmed. They never appear under a failed step’s “Logged during this step”. A run keeps up to 10,000 of your app’s lines, and system lines separately up to 2,000, so they never push your app’s lines out.
Each step opens into what it did: reading the screen (with how many elements it saw), taps, typing, scrolling, waiting for the screen to settle, and the check. Details on a step shows the raw data: for a replayed step, the recorded and the executed element and coordinates; for an AI step, its reasoning, each tool call with its arguments and result, and the tokens used. The steps of a setup test that ran before this test are grouped in a Setup row.
The steps count is out of all the steps in the test: a six-step test that stopped at step 2 reads 1/6, and the steps it never reached are listed greyed as Not run. When a failed test was retried, the retry reads Attempt 2 of 2 and links to the first attempt, which in turn links to its retry; the cost shown counts both attempts, and a group run’s table does the same.
The line at the top shows the build, what the test started from, steps, duration, how many steps were replayed and how many the AI ran, and the cost. When every replayed step was a learned path, it also shows what the same run would have cost with AI on every step (marked as an estimate when part of it ran on your own API key). A run started with a commit shows the commit, linked to GitHub when your organization connected a repository.
Edit test opens the test. Re-run (Developers and above, once the run has finished) opens the Run dialog with the same device, build and AI use filled in; when the earlier device or build isn’t available any more, the dialog says so and uses another. Cancel run (while the run waits or runs) stops it after its current step; both need the Developer role or above.
At the bottom, the run says whether it found issues, and asks Was the AI right? Answer with a thumbs up or down, and add a note with a thumbs down. Your answer helps us improve how checks are judged. Anyone in your organization, Viewers included, can answer.
A run on several devices, or of a whole group, has its own page: one row per test and one column per device, each cell linking to that test’s run (click anywhere in the cell, or its View run link). The header line shows the environment the run used, when it used one. Cancel run there stops every device: a device that is waiting leaves the queue, and one that is running stops after its current step. A run of one test on one device opens that test’s run page directly.
On the Runs section of the Tests page, a run on several devices is one row, with the device column reading, for example, “2 devices”. When the app runs on more than one platform, a Platform column shows where each run ran, and each test’s Last result shows its latest pass or failure on every platform.
When nothing was checked
No AI never uses AI, so it can only check a step that has a recorded or learned path on the device it runs on. What happens to the other steps depends on whether the test has any path at all:
- The test has no recording and no learned path. A No AI run of it is refused before it starts, with a message to run it with Smart or AI only instead (a test made only of Verify steps is the usual case). A group or multi-test run with No AI is refused the same way when none of its tests has a path; when only some have none, the run starts, and each of those tests ends with an error that says the same, without touching the device or using AI.
- The test has a path, but not for every step. Each step without a path on this device is reported as not checked. A run in which no step was checked (for example, when the test’s only learned path was saved on another platform or kind of device) ends Not checked, shown in grey: it is neither a pass nor a failure. The run page says so and offers Run with Smart, which can check those steps. A run where only some steps weren’t checked still passes, and says how many weren’t checked.
A not-checked run sends no failure notification, never stops a run under Stop on first failure, and doesn’t count as a pass in the test’s history. A group or multi-device run counts its tests not checked apart from the ones that passed (“2 of 4 passed · 2 not checked”); when every test was not checked, the whole run is Not checked.
When a run fails
The run page says what kind of failure it was (for example Check failed, Screenshot drifted, Couldn’t do this step, Setup failed, or Stopped by VibeView, not your app), shows the AI’s explanation and the app’s log lines from that step, and Show me jumps to the step. App crashed or froze appears only when the app actually stopped responding; when the AI reports a defect (for example a tap that did nothing visible), the heading reads Bug in the app. The actions depend on why it failed. Bug in the app copies a bug report (steps to reproduce, expected and actual, log lines and a link to the run) to paste into Jira, Linear or GitHub. Expectation changed, Test is wrong and Too strict open the test at that step. Changed on purpose accepts the new screenshot as this device’s baseline after asking you. AI misjudged it is coming soon. Run with Smart, Re-run, Change start state and Open the setup test appear where they fit the failure. Failures caused by VibeView (a device or session that stopped) never offer Bug in the app. A Verify step that checks a secret test data value fails with a message saying that Verify steps can’t check secrets, rather than a mismatch (see Secrets).
You get a notification, and for runs started from the Run dialog an email too, with a link to the run. A scheduled run tells its owner. See Notifications.
From the Devices page
To run a test on a device you already have open, use Run a test on the Devices page.
From the API, the CLI and coding agents
The Run dialog uses the same API your scripts can call, POST /api/v1/tests/runs, with the same fields as CI: commit SHA, metadata, model and provider, AI use, environment and build. See Tests API. Coding agents have the run_test, run_group and get_run tools over MCP; see AI agent control.