AI Testing (Beta)

VibeView’s AI testing system lets you define automated test cases that run on cloud iOS simulators and Android emulators, including Apple TV and Android TV, and on Roku devices. Tests can replay recorded interactions, use an AI agent that understands your app’s UI, or combine both approaches for maximum reliability.

Groups

A group is a folder of tests. A test can be in several groups, and groups live in the left column of the Tests page. Groups give the tests in them shared configuration.

Key properties

• App — Each group belongs to one app. The app determines which build is installed and run.
• Setup steps — Attach reusable tests (for example, a login flow) to a test so shared steps are defined once and reused everywhere. See Setup steps.
• Start each test from — Control how the app state is cleaned before each test, the first one included. The default, Clean app data, wipes app storage before each test, ensuring a consistent starting point.
• Extra notes for the AI — Free text added to the app’s notes for the tests in this group. It is included in the AI agent’s prompt, helping it make better decisions about how to interact with your app.
• Test data — Key-value pairs substituted into test steps, kept per app (or once per product for linked apps) on the Tests page’s App notes & data section, so every group of the app shares them. For different values per run (such as a staging account), use an environment (see Test data, secrets and environments).

Everything except test data is in the group’s Group settings, including the Screenshot check thresholds. See Visual Regression Testing for details on the threshold settings.

Test Cases

Each test case defines a sequence of steps to execute on the simulator. Every step is a natural language instruction (for example, “Tap the Sign In button” or “Scroll down until you see the Settings option”).

Recorded cases

A case made from a recording on the Devices page keeps the recording next to its steps. Its steps are generated from the recording (Generate with AI in the Recorder), and runs replay the recorded gestures first, which is fast and uses no AI, handing a step to the AI agent only when it can’t be replayed — see AI use below. A recording can’t be saved as a test on its own: without steps describing it, there is nothing to fall back on when the screen changes.

AI steps

For a step with no recorded gesture, the AI agent interprets the instruction by reading the current UI tree and screenshots, then decides which actions to perform.

A run must do something

A test needs at least one step (or a setup step): the editor won’t save a test with none, and the API refuses one. A step that holds no instruction fails the run before it starts, and a run in which no step did anything on the device — no tap, typing, replayed gesture, or check of the screen — is recorded as failed, never passed. A step that only verifies what is on screen, such as “Verify the home screen shows a welcome banner”, counts as a check when the AI agent looked at the screen and confirmed it. A green run always means the app was exercised or checked.

Creating test cases

Tests are created from the Tests page with Create test (Developers and Admins), and edited on each test’s page. Create test has three tabs:

• Describe — write what the test should check in a sentence or two, for example “Search for Kyoto, save it, then check that it is listed under Saved”, and click Draft steps. The AI drafts the steps; nothing runs until you review them. Drafting follows the app’s notes and knows the names of its test data, never the values: a step that needs one uses {{key}}. Write the steps yourself skips drafting and opens the review with one empty step.
• Record — opens Devices for the app with the Recorder on. Pick a device and start it; recording starts by itself once your app is open. Stop the recording and click Generate with AI: the generated steps open in the same review, and runs replay your gestures first.
• Explore — the AI uses your app on its own for 5, 10 or 20 minutes and lists what looks broken; any finding can be saved as a test. See Explore.

Under the Describe box the dialog names the AI model that drafts the steps and who pays: your usage credit, your own API key, or, on the Free plan with no usage credit, your organization’s free test. No price is shown before drafting. The line under it links the app’s notes and test data.

Review steps is the same however the test was started:

• Name — the test’s name (the only place you name it).
• Steps — edit, remove or add steps, and Add check for a Verify ... step. Each step is labelled Recorded (replays a recorded gesture), AI (worked out by AI on each run) or Check. A step’s key button, or typing «, inserts one of the app’s test data keys; it is saved as {{key}} and shown as a «key» chip.
• Before this test — reusable tests that run first, in the order shown (see Setup steps).
• Groups — the groups the test joins; none puts it under Not in a group. Click the field to open the list of the app’s groups, type to search it, and tick as many as you like; each chosen group shows as a chip you can remove. To put the test in a group that doesn’t exist yet, type its name and pick Create group, or use New group at the bottom of the list. A new group is marked New and is only created when you save the test; it starts from Clean app data until you change it in Group settings.
• Starts from — the group’s start state, or one you pick for this test.

Closing Review steps without saving asks you to confirm. The draft is kept, with your edits: while you stay on the Tests page, Create test offers Resume your draft until you start another test.

Save opens the new test’s page. Save and run saves it and opens the Run dialog on it: pick the devices and click Run. (In the Review steps on the Devices page, Save and run runs it on the session you are on.)

On the Free plan with no usage credit, Describe drafts are free while your organization’s free test is unused, up to a limited number of drafts, and Save and run runs the test as that free test (up to 8 steps, setup steps included). Once the free test is used, Describe asks you to upgrade your plan or add your own API key; Write the steps yourself stays available.

At the bottom of Create test, Write tests with your coding agent opens a prompt for your coding agent (Claude Code, Codex, Cursor or another) that creates tests with vibeview create-test.

Before your organization has any tests, the Tests page offers Developers and Admins Get the prompt: a prompt for your coding agent (Claude Code, Codex, Cursor or another) that finds your app’s main flows in the code and creates a test for each with vibeview create-test. Once you have tests, a one-line hint under the list offers a prompt that adds a CI workflow running them on every pull request (see CI testing).

AI use: Smart, No AI, AI only

Every run uses one of three AI uses, picked under Advanced in the Run dialog: Smart (the default), No AI and AI only. The CLI (--mode) and the API (mode) call them hybrid, replay and ai. The Run a test tab on the Devices page always uses Smart.

• Smart replays what a test already knows — its recording, or a learned path — and hands a step to the AI agent (beta) only when replay can’t follow it, for example because the screen changed. This gives you the speed of replay with the resilience of AI-driven testing.
• No AI only replays. It never calls the AI, uses no usage credit, and runs even when your usage credit is exhausted.
• AI only sends every step to the AI agent and ignores recordings and learned paths. Use it after a big change to your app.

A rotation you recorded replays too: the device turns to the orientation the recording turned it to (a phone or tablet ends in portrait or landscape, whichever way the run started — a device already there is left as it is), and the gestures recorded after it are adapted to the turned screen. The step fails, or hands over to the AI in a Smart run, if the device cannot be turned to that orientation. An app that stays upright when the device turns still passes the step, with a note that the app did not turn. On the run’s step list, a step that rotated the device shows the orientation it ended in, and a step that folded a foldable shows the posture it settled in (Closed, Partial or Open). A test run starts the device in the same orientation as a fresh session started from Devices (portrait on iPhone, iPad and Android phones; landscape on an Android tablet such as the Pixel Tablet, whose screen is landscape when upright), so if your test needs a particular orientation, record the rotation as a step.

Step ranges guide generation. When a recording contains user-defined step ranges (see Step Ranges in the Devices docs), the AI uses those ranges as deterministic anchors and only infers steps for the unbracketed gaps between them. This means you can isolate the exact step grouping you want — name it, lock its boundaries — and let the AI handle the rest. If you don’t define any ranges, the AI infers all step boundaries from scratch as before.

No AI never calls the AI, so such a run uses no usage credit. That also means:

• A case with no recording and no learned path has nothing to replay, so a No AI run of it is refused with a message saying so — run it with Smart or AI only instead. A group run with No AI is refused only when none of its tests has a recording or a learned path; otherwise each test without one fails with the same message and the rest run. A test whose learned paths were saved on a different device type, orientation or posture than the run uses has nothing to replay on this device: the run goes on and reports those steps as Not checked.
• Steps switched to Always AI are reported as Not checked in a No AI run instead of being handed to the AI, and the run goes on.
• A step with no recorded actions (typically a Verify ... step) has nothing to replay. A No AI run reports it as Not checked and goes on with the run — see Checking results with Verify steps.

A Smart run of a test without a recording is an AI only run, so it behaves exactly like AI only — including ignoring waits.

Checking results with Verify steps

A test checks its results with ordinary steps, written where the check belongs — in the middle of the flow or at the end:

Log in with username 'alice' and password 'secret'
Verify 'Welcome, Alice' is visible
Open Settings from the side menu
Verify the 'Dark mode' switch is on

When you generate a test from a recording with Generate with AI, the AI writes a few Verify ... steps inline where they matter. You can edit, add or remove them like any other step.

Write each check about the screen as it is at that point, in the present tense, and quote the text it is about: Verify 'Order placed' is visible is clearer than Verify the order went through, and Verify 'Welcome' is visible is clearer than Verify 'Welcome' appeared after logging in.

A Verify step has no recorded actions, so how it runs depends on the mode:

A Verify step never enters or changes what a field holds. To find what it checks, the AI agent can look at the screen, scroll, tap and move between screens, so a check can still change which screen the app shows; what it can’t do during a Verify step is type, clear a field, press keyboard keys or change a picker. Put anything the test must enter in its own step before the check. This applies to the steps the editor labels Check: a step that starts with “Verify” and has no recorded actions.

An empty text field counts as empty: the grey placeholder text it shows when nothing is typed is never read as what the field holds.

A Verify step can’t check a secret test data value; such a step fails with a message saying so. See Secrets.

A failed check fails the run, like any failed step. A replay run that otherwise passes still shows Passed, with a note saying how many steps were not checked (for example 2 steps not checked) on the run page and in the case’s run history, in the CLI (PASS (2 steps not checked)), and in the API (steps_not_checked). Each such step shows as Not checked in the step timeline. Run the case in hybrid mode to have them checked.

Assertions (deprecated). Test cases used to have a separate list of assertions checked after the last step. That list has been removed: runs no longer check it and the dashboard no longer shows it. For now the API and YAML import still accept an assertions field and save each one as a Verify ... step at the end of the case, so existing scripts keep working; use steps instead.

Per-step AI steps

In a Smart run, every step starts as Replay first: it replays the recorded gestures and falls back to the AI agent only when replay can’t follow. You can switch an individual step’s Replay first chip to Always AI to bypass recorded replay for that step and route it directly to the AI agent. A No AI run never uses AI, so it reports an Always AI step as Not checked and goes on with the run. The chip appears only on cases that have a recording: on a case without one, every step already runs through the AI agent. A step with no recorded action of its own (such as a written Verify ... check) shows a Run by AI chip instead: the AI agent does it on every Smart run, and a No AI run reports it as not checked. This is useful when a step requires the AI to actively verify screen content rather than blindly replaying recorded gestures.

When to switch a step to AI

• Verification steps — When you need the AI to confirm that specific content appears on screen (for example, checking that a profile name was updated correctly).
• Dynamic content — Steps where the UI varies between runs and recorded coordinates are unreliable.
• Checks on recorded steps — Steps that should evaluate the current state rather than perform a fixed interaction.

Notes for the AI

Each Always AI step can include an optional Notes for the AI — a short description of what the AI should verify. This context is appended to the step instruction so the AI knows the specific criteria to check.

For example, if your test records a profile name change from “abc” to “abpd”, you could switch the verification step to Always AI with these notes: profile name must be abpd, not abc. The AI will then read the screen and confirm the name matches, rather than replaying a tap on the old value.

Behavior

• Steps set to Always AI skip the replay block entirely and go straight to AI execution.
• Steps left at Replay first replay their recording, with the AI agent as a fallback.
• The Always AI setting works in single test runs and group runs.

Editing recorded actions

When a test case has a source recording (a recorded test), you can edit the underlying gestures attached to each step directly from the test case page.

How to edit

1. Open the test from the Tests page.
2. Click Edit to enter edit mode.
3. For any step with a recording segment, click the Actions toggle (▸ Actions) below the step to expand the gesture list.
4. Each gesture row shows a summary line (for example, tap @ (120, 340) or swipe ↓) and editable fields specific to its gesture type.
5. Click Save on the test case to persist your edits.

If the test’s steps were changed somewhere else (another tab or a teammate) since you started editing, saving is refused and you are offered a Reload to see the latest version; your unsaved edits are lost on reload.

What you can edit

• Taps and long presses — The text, accessibilityId, resourceId, and className element selectors that the replay engine uses to resolve the gesture’s target. Updating these changes how the replay engine finds the element on future runs, without changing the recorded coordinates.
• Swipes — Swipes replay the recorded finger motion itself, so they reproduce scrolls, flicks, and drags faithfully — including diagonal swipes and gestures that don’t land on a specific element. The motion is recorded relative to the screen, so a swipe recorded on one device replays at the proportional position on a different screen size, and works across platforms. (When replaying across platforms, the gesture is faithful but momentum scrolling can settle slightly differently because iOS and Android use different fling physics.)
• Device button presses — The button name (for example, home, back).
• Text input — The typed text.
• Delete — The X icon on each row removes the gesture from the recording. Deleting a swipe shows a confirmation prompt; other gestures are removed immediately.

Coordinates and gesture timing are not editable. To change them, record the flow again on the Devices page.

Removing a step that has recorded actions

Removing a step (the X next to it) that has recorded actions asks what to do with them:

• Move to step N — keeps the actions and hands them to the step directly before (or, if that step has no recorded actions, the step directly after), so they still replay as part of that step, in the same order.
• Delete actions — removes the actions from the recording along with the step.

If neither neighbouring step has recorded actions, only Delete actions is offered — moving the actions past a step the AI runs would change the order the test runs in. A step with no recorded actions is removed straight away. Nothing is saved until you click Save changes.

Reordering steps

In edit mode, the up and down arrows next to each step move it one place earlier or later. A step’s recorded actions move with it, so after the move the test replays them in the new step order. Steps with no recorded actions, such as Verify steps, move on their own. A test saved before steps were linked to their recorded actions can be reordered too, unless its recording has more actions than it has steps — then the arrows explain why the step can’t move. Nothing is saved until you click Save changes. Reordering keeps every step’s visual regression baselines and recording screenshot with the step — see visual-regression.md.

Step-to-recording validation

When you edit or delete gestures, VibeView revalidates the mapping between each step and its recording anchor. If a step’s source_action_index no longer points at a valid gesture after your edits, an inline error appears under that step explaining which anchor is missing. Fix the mapping (or restore the deleted gesture) before saving.

Wait between steps

Sometimes a step finishes an action (like tapping Submit) but the app is still loading the next screen. By default we wait for the UI to settle automatically before checking the result. For cases that don’t fully detect (such as a steady-state loading spinner whose layout doesn’t change while data loads), you can configure an explicit wait.

In the test editor, hover the gap between two steps and click + Add wait. Enter a duration (in seconds or milliseconds, up to 5 minutes) and save. The wait is applied after the step’s last action and before the next step starts.

The wait travels with the step it follows — moving or deleting the step also moves or deletes the wait. Waits apply only when a step is replayed from a recording, in Smart and No AI runs. AI only runs ignore them (the AI can call wait itself) — that includes a Smart run of a test that has no recording.

Before the first step of a run, VibeView waits for the app to be ready, so the step does not start against a launch screen. If the first step was recorded on an element, VibeView waits for that element to appear.

If the app still isn’t ready after the wait, the run goes on, and the first step’s result notes what the screen showed. When the first step then runs with AI and the app wasn’t ready, the AI takes one more look at the screen before a failure is recorded.

Learned replay paths

When a run passes, VibeView saves what the AI agent did in each step it ran as that step’s learned path. The next run replays the learned path the way it replays a recording, so those steps use no AI and no usage credit.

• A path is saved only from a run in hybrid or ai mode in which every step passed, every Verify step included, and no screenshot check failed. A replay run never saves paths: it has nothing new to learn.
• Steps that check the screen, such as Verify steps, always use the AI. Steps set to Always AI are never learned. A step with a recording keeps its recording; a learned path never replaces it, in ai runs too.
• A path applies only to the kind of device it was learned on: the same platform (iOS, Android or Roku), device type (phone, tablet or TV), the orientation the run started in, and, on a foldable, posture. A path learned on an iPhone is not used on an iPad, one learned in a run that started in landscape is not used in a run that starts in portrait (a step after a rotation keeps the orientation its run started in), and one learned with a foldable flat is not used when it is folded. The run page names each saved path’s device this way, for example iOS · Phone · starts portrait; a TV path is named for the TV (Apple TV, Android TV, Roku).
• Text the AI typed that matches one of your test data values is saved as the value’s name ({{password}}), never as the value itself, and each run types its own value. A step whose path would show a test data value on screen is not learned.
• Steps on a TV (Apple TV, Android TV and Roku) are learned too: the remote presses the AI made are replayed the way a recorded TV test replays them, following the focus to the same element.
• Some steps stay with the AI: steps that scroll until something appears, use a gesture preset, pick a picker value, answer a system alert, open a link, set the location, restart the app, rotate or fold the device, clear a field (including clearing it before typing), press a key combination, drag, or, on a phone or tablet, press a button other than Home, Back, Lock, Siri or Enter. After a run that passed, the run page lists each step that was not learned and why.
• A learned step finds its elements exactly as a recorded one does (see Element Matching). If the screen changed and it no longer fits, a hybrid run hands the step to the AI agent. When only the end of a path no longer fits, the steps that did replay are kept and the AI’s actions complete the path.
• When a learned path no longer fits and the AI completes the step in a way that can’t be learned, or the same way that just failed, the path is no longer used: the next run gives the step to the AI and can learn it again.
• Editing a step’s text forgets that step’s learned path.

When a route changes

If a learned path no longer fits and the AI finds another way through in a run that passes, the run page shows the step with three choices:

• Save the new route — replay the new route from now on.
• Keep the old path — go on replaying the old one.
• This is a bug — keep the old path: the change in your app was not intended.

A route you turn down with Keep the old path or This is a bug is remembered: a later run in which the AI takes that same route for the step on the same kind of device does not save it or ask about it again. Editing the step’s text clears this.

Choosing what is saved

Each test page has a Replay path card listing what is learned for each platform and device, with the Save new paths setting:

• Save automatically (default) — new and changed paths are saved, and the run page says so; you can still keep the old path there.
• Ask me first — a new or changed path is saved only when you choose to save it on the run page.
• Never — learned paths are neither saved nor replayed; steps without a recording always use the AI.

A test follows its app’s setting (App setting) until you pick one for the test. The app’s setting is Save new paths in the Replay paths card under App notes & data on the Tests page, with the same three choices (its Default option follows the default, Save automatically). Forget learned paths removes the test’s learned paths; the next passing run learns them again. Changing the setting and forgetting paths need the Developer or Admin role; a Viewer can read the card. For linked apps it is one setting for every platform of the product.

What it saves you

A run page shows how many steps were replayed and how many the AI ran (for example “1 replayed · 1 by AI”). When learned paths replayed and their steps passed, it also shows the AI spend avoided: what the AI cost for those steps in the run that learned them (an estimate when a path was learned with your own API key).

The run page’s top line also shows what the run would have cost with AI on every step, when every replayed step was a learned path (the figure comes from what learning those paths cost, never from a guess). At the bottom of the run, Was the AI right? records your answer and an optional note; it changes nothing in the run.

Replay Details & Debugging

Each step on the run page has a Details toggle. For a step that ran in a Smart or No AI run, it opens a comparison table with three columns — Property, Recorded, and Executed — showing how the original recording maps to what actually ran on the simulator. For a step the AI ran, it also shows the AI’s reasoning, each tool call with its arguments and result, and the tokens used.

The comparison covers:

• Gesture — The recorded gesture and the one that was executed.
• Element — The recorded element identifier (accessibility ID, text, or resource ID) next to how the replay engine resolved it.
• Coords — The original recorded coordinates alongside the coordinates that were actually executed after element resolution and adapting to the current device’s screen size.
• Resolved via — The resolution strategy used (coordinates, element, element_text, or not_found).

A retry badge appears next to the mode indicator when the replay engine retried a gesture. For example, “1 retry” means the first attempt failed and a second attempt succeeded.

When a replay step falls back to the AI agent, a warning in the expanded step says replay didn’t work and why, in plain words — for example, “The recorded target “Saved” isn’t on this screen.” A step that fails in a No AI run shows the same sentence as its error; its machine-readable kind (element_not_found, wrong_state, app_unresponsive or aborted) is unchanged.

Element Matching

When a step is replayed, VibeView looks for the element each recorded action was made on, and only accepts the same element:

• The same id — the element’s test id or accessibility id, whichever platform the recording was made on.
• The same text — when no element on screen has the recorded id (it is missing or has changed, or the recording has none), an element whose text is exactly the recorded text, ignoring capital letters and surrounding spaces. Sign in finds Sign In and Username finds Username, but ScriptX One does not find ScriptX, and Home does not find Loading Home.... A text field is known by its label or placeholder, never by what was typed into it: “Paris” typed into a search field does not make it the “Paris” element, and an empty field is still found by its label or placeholder.

When several elements share the recorded id or the recorded text, replay picks the one at the recorded position, and only when that position clearly sets it apart from the others. When the recorded position cannot tell them apart, or the recording has no position, replay does not guess, not even by their text. Replay never taps a near match. If neither finds the element, a Smart run hands the step to the AI agent, and a No AI run fails the step.

A tap is recorded on the element under your finger, never on the whole screen. When the only thing found at the tapped point is the screen itself — the app’s outermost frame, or a system layer lying over the app (on iPad, for example, the one iPadOS uses for Dock folders) — the recording stores the app element beneath it, or, when there is none, the tap’s position only, and replay taps that position. Taps recorded earlier on the app’s outermost frame or on such a system layer replay by position too.

A step only replays the actions recorded for it. A step with no recorded actions of its own never borrows another step’s taps — even in a test where no step is linked to the recording — so a No AI run reports it as Not checked.

Learned paths are matched the same way, with no exceptions.

Older test cases and API clients may still carry a fuzzy_match setting. It is accepted and ignored.

AI Execution

On iPhone, iPad, Android phones and tablets, and Android TV, a run first brings the test’s own app to the front — the app the test case belongs to, even if you installed a different app on the session since, or left the device on the Home screen. An app that is already running comes back where you left it. If that app is not installed on the session, the run fails straight away with a message naming the app; install it on the session and run the test again.

When the AI agent runs a test step, it follows a loop:

1. Read the UI tree — The agent requests the current accessibility tree from the simulator. The tree is pruned and filtered to show only interactive elements, each tagged with a short reference like @e1, @e2.
2. Take a screenshot — The agent captures the current screen to provide visual context alongside the structured tree.
3. Decide an action — Using a tool-calling LLM, the agent selects from the available tools based on the step instruction and current UI state.
4. Execute the action — The chosen tool runs on the simulator via the native automation layer on each platform.
5. Repeat — The agent continues observing and acting until the step objective is met or a failure condition is reached.

The agent has a default cap of 20 turns per step, which you can change (1–30) under Settings > AI configuration > Agent turns per step. If the goal is not reached within that many turns, the step fails with “This step needed more than 20 AI turns. Split it into smaller steps, or raise the limit in AI settings.” (with your own limit in place of 20); its machine-readable kind stays max_iterations. When the agent says a step is done but the screen looks unchanged since the step started, the step is not accepted yet and the agent is told so: it can finish the step with a reason (an action that changes nothing visible, such as pressing Left at the start of a list), act again, or fail the step. If it then keeps answering in text until its turns run out, the step fails saying exactly that instead of the turn-limit advice. Tight, specific step text (and relevant app notes) keeps the agent on the fast path.

Tap verification

After every tap, the agent compares the screen before and after. It first checks the whole screen for change; if that looks unchanged, it re-checks just the region around the tapped element. The two-stage check catches small state changes — a heart outline filling in, a checkbox toggling, a radio button selecting — that a whole-screen comparison would miss.

The result is surfaced on the tap response as one of:

• screen_changed — whole-screen diff detected movement.
• localized_change_detected — whole screen looked unchanged, but the local region around the tap moved.
• weak_change_detected — sub-threshold movement in the local region (common for small indicators); treated as a successful tap.
• unchanged — literally zero pixel change; the agent treats this as a missed tap and takes corrective action.

Scrolling

scroll_until_visible is the single scroll-search tool. Pass a text substring or accessibility id — it finds the right scrollable container, detects the scroll direction automatically, and fine-tunes until the element lands fully inside the viewport. To bring a previously-seen element back into view, pass its visible text or accessibility id. For lazy-loaded content that only appears in one direction, the tool falls back to bidirectional scrolling. It also detects end-of-scroll: when the scrollable content stops changing between passes, it stops early and the model receives a “likely not on this page” warning rather than looping indefinitely.

Hittability gate

On iOS, the agent checks that an element is actually hittable before tapping it. If the element is hidden behind something else or has no size, the tap is rejected with a not hittable error instead of firing at invisible UI — so the agent doesn’t silently tap covered elements.

Available tools

The agent has 26 tools on both iOS and Android touch devices. Two additional tools (tap_focused_tv, focus_element_tv) are available on Apple TV, Android TV and Roku sessions: they move focus to a target element (and, for tap_focused_tv, press SELECT) instead of chaining press_button(dpad_*) presses. On Apple TV they use the platform’s built-in focus navigation; on Android TV VibeView navigates the d-pad toward the target until focus lands. On Roku they reach list tiles by title. On Android TV they require an app whose focus is visible to the device’s accessibility layer (react-native-tvos / Leanback); custom-JS-focus apps fall back to plain d-pad navigation — see TV testing for the full picture.

On a foldable device — the iPhone Duo, or an Android foldable such as the Pixel 9 Pro Fold (beta) — the agent also gets set_posture, letting it fold or unfold the device to Closed, Partial, or Open as part of a test. It is not offered on any other device. A run that asks only for a phone or tablet never lands on a foldable; start the session on the foldable’s model, or pin the run to its device, to test on one. While the iPhone Duo is open or partly open, scroll_until_visible and select_picker_value are not available on its inner screen and say so when called; the agent scrolls with swipe there instead. Every other tool, and every screenshot, works on whichever screen the device is using.

Interaction:

Text and keys:

System and device:

A recording doesn’t capture an app restart, so a step that relaunches the app runs through the AI agent: run it with Smart or AI only, not No AI.

Query and checks:

Step completion:

App logs (beta): a step can ask the agent to check the app’s log — for example Verify the app logs show no JavaScript errors or Report the first crash line in the logs. The agent reads the most recent few hundred lines the session has captured (the same output as the session’s App Logs tab), filtered by text or minimum level, and judges them itself. If the session has no log output, the agent is told so rather than reading an empty log as “no errors”.

The agent uses element references (@e1, @e2, etc.) from the UI tree to target elements precisely. References are stable within a step — the same element keeps its ref even if the tree order shifts slightly between iterations.

When a check fails, the step and the run are marked as failed and the agent logs the reason along with a screenshot of the actual state.

Visual Regression During AI Runs

Every run (Smart, No AI and AI only) captures a baseline-comparison screenshot after every step when the group, or the step, has a screenshot check. Drift that exceeds the warn threshold is flagged on that step in the timeline; drift past the fail threshold fails the run. Configuration, thresholds, and baseline acceptance flow are documented in visual-regression.md.

Group runs

A group run runs every test of the group, in the group’s order. On each device you pick, the tests run one after another in one session; with several devices, each device runs the group at the same time in its own session (see Running tests).

• Each test starts from the group’s Start each test from setting unless the test sets its own (see Start each test from).
• A test’s setup steps run as part of that test.
• If one test fails, the next tests still run — unless Stop on first failure is on in the Run dialog.
• If preparing a test fails (for example the app’s data could not be cleared), that test is marked as an error with the reason and is not run on an unprepared app. The group goes on with the next test. A single test run behaves the same way.

Cancelling a group run

A run started from the Run dialog is cancelled from its run page. While a group is running on a session you opened from Devices, the Run a test tab shows two cancel buttons:

• Cancel this test — cancels the currently running test. It is recorded as cancelled, and the group continues with the next test. Use this when one test is stuck or known-bad and you want the rest of the group to run.
• Cancel the group run — cancels the entire group. The current test is cancelled, and all remaining tests are recorded as cancelled without executing.

An abort takes effect at the run’s next step, or between the AI’s actions during a long step, so a run can show running until its current step finishes (for a group, until the current test’s setup finishes). It then reads cancelled, and steps that already finished keep their results.

Between tests (during the app reset window), Cancel this test is briefly disabled — Cancel the group run is always available while the group is running.

When you run a single test (not a group), only one Cancel button is shown.

Running a test on the other platform

A test case belongs to one app, but you can run it on the other platform without copying it. Upload the other platform’s build as its own app (your iOS app and its Android build are two apps with the same bundle id / package name), then:

• Dashboard: open the Run dialog on the test and pick a device of the other platform. The dialog lists your apps with the same bundle id on the other platforms of the test’s form factor. A run started from the Tests page never opens Devices.
• CLI: vibeview run-test <test-id> --platform android (or run-suite), or --app <app_id> to name the app. See Run on another platform.
• API: pass platform (or app_id) to POST /api/v1/ci/run-test or POST /api/v1/ci/run-suite. See Cross-platform runs for how the app is found and what happens when none, or several, match.

The run is stored on the same test, next to its runs on the original platform. A phone test runs only on phone platforms and a TV test only on TV platforms: asking for the other form factor is refused with that reason.

The other platform’s app may have only debug builds that load their JavaScript from Metro (marked needs Metro on the app page). A test run has no Metro dev server, so the app would show an error screen: the run still starts, but the CLI and API warn you when it does (see Debug builds that need Metro). Upload a release build of that app first.

Keeping a copy per platform

To keep a separate test per platform (for example to edit the Android steps on their own), copy it to the other platform’s app:

1. On the Tests page, open the test’s row menu and click Copy to another app….
2. Pick the app under Choose an app, then a group of that app or No group (or name a new group).
3. Click Copy test. The copy keeps the name, or becomes <name> (copy) (then (copy 2) and so on) when the app already has a test with that name. It keeps the steps, the recording and its setup steps, which it shares with the original.

The API makes the same copy with POST /api/v1/tests/cases/{case_id}/copy (see Tests API).

What to expect:

• Runs and copies work between apps of the same form factor only: iOS and Android phone and tablet apps copy to each other, Apple TV and Android TV apps copy to each other, and a phone case can’t be copied to a TV app. For TV, see Cross-TV replay.
• AI steps are plain language, so they usually run unchanged. Recorded steps replay on the other platform when the elements carry the same accessibility identifiers or text on both; where they don’t, a Smart run falls back to the AI agent for that step.
• A copied test keeps its {{variables}} placeholders. When the target app is a different app, add the values to the target app’s test data, and copy any app notes the test relies on to the target app (see App notes and test data).
• A copy is independent: later edits to one test don’t reach the other.

Test Run Results

Each test run produces a detailed result record. Every run has a dedicated page at /tests/runs/:id that you can open directly, bookmark, or share with teammates. The URL works for single test runs and group runs.

Run detail page

The run detail page shows:

• Status and duration in a summary row under the run’s name.
• Model used and usage credit / cost statistics.
• Step timeline — a visual breakdown of each step with its status (passed, failed, or Not checked), screenshots, and duration. Click a screenshot to open it, then make it bigger with the scroll wheel, a pinch, the + and − buttons, or a double-click, and drag to move around it. Press Esc or click outside the image to close it. A run that stopped before its first step (for example because the app isn’t installed on the device) shows the reason in place of the timeline.
• Step start times — each step shows when it started, counted from the start of the run (for example 0:07), next to its duration.
• Logs tab — the app’s own log for the run (iOS, Android, Apple TV, Android TV and Roku), each line labelled with the step it was written in. Filter by level (All, Error, Warning, Info, Debug — each shows how many lines it has) or by text, and copy the lines on screen. The page says when lines were missed, when a very chatty run was cut off, when the app wrote nothing during the run, and when a run has no log (logs are kept for 30 days, and older runs have none).
• Logged during this step — on a failed step, up to three error and warning lines the app logged during that step appear under the error (never the operating system’s own messages), nearest to the failure first, so you often don’t need to open the Logs tab. They were logged while the step ran; that alone doesn’t prove they caused the failure.
• AI reasoning log — the agent’s decision-making trace, one line per turn, under the step it belongs to.
• Visual regression — side-by-side comparison with baseline screenshots (when available).

For in-progress runs, the page auto-refreshes every few seconds until the run reaches a terminal status (passed, failed, or error). You can also set a passing run as the new visual regression baseline directly from the detail page.

Status values

Result data

• Logs — up to 10,000 lines per run, kept for 30 days. If the app logs faster than lines can be collected, the Logs tab says how many were missed. Test data values marked as secret are masked in the logs.
• Duration — Total wall-clock time from start to finish.
• Screenshots — Captured at key moments: before and after actions, on failed checks, and at step completion.
• Reasoning log — The AI agent’s decision-making trace, showing what it observed and why it chose each action.
• Model used — Which LLM provider and model executed the run (Anthropic, OpenAI, Google, or OpenRouter). A run that never called the AI — every step replayed from its recording, as in a No AI run — shows no model.
• Token usage — Input and output token counts for the LLM calls (diagnostic; your bill is the Cost below, not the raw token count).
• Cost — for VibeView-provided runs, the usage credit actually deducted for this run (the real billed amount, in USD). Most runs cost less than a cent, so amounts under a cent show up to four decimal places (for example $0.0037) instead of rounding to $0.00. For BYOK runs, VibeView deducts nothing, so the Cost is shown as an estimate (~$X (est.)) computed from public list pricing — your provider bills you directly and the exact amount may differ. Estimated (BYOK) and actual (provided) costs are always shown separately, never combined into one figure. See Usage Credits.

App notes and test data

App notes

Each app has notes that the AI agent reads in every test of the app, when drafting steps and when running them: what the app does, its own words for things, and quirks to expect (for example “A profile picker appears after login”). A group can add its own notes, used for every test run as part of that group. Edit them under App notes & data on the Tests page (see The Tests page).

Test data

Usernames, passwords, PINs and one-time codes live on the app as test data and go into steps as {{key}} placeholders. For example, define username=testuser@example.com and write a step like Type {{username}} into the email field. At run time VibeView substitutes the value before the step executes. To override values for some runs, use an environment. A value can be marked secret: it is masked everywhere a run is shown and never put in the AI’s prompts or logs (a secret shown on screen can still appear in screenshots the AI sees). See Test data, secrets and environments.

• Substitution applies to every run of a test: a single test or a whole group, from the dashboard, the CLI, CI or the API, with any AI use.
• It covers the test’s step text, the verification context on Always AI steps, the steps of any setup steps attached to the test, and matching values in recorded actions (for example, text a recording typed).
• Your saved test is not changed — placeholders stay in the editor and are filled in fresh on every run.
• A placeholder with no matching variable is left as written.

YAML Export / Import

A group and its tests can be exported as a YAML file and imported back through the API. The dashboard has no YAML buttons. See YAML test format for the format and the two endpoints. An export includes the group’s settings, every test with its steps and source_action_index mappings, the recorded gestures and the source device size, and the app’s bundle ID so an import links the right app.

React Native Accessibility

The AI agent reads the iOS accessibility tree to understand your app’s UI structure. React Native apps can sometimes produce incomplete UI trees due to how iOS accessibility containers work.

The issue

When a React Native Pressable, TouchableOpacity, or TouchableHighlight wraps a TextInput, the touchable component acts as an accessibility container. iOS treats the entire container as a single element and hides its children from the accessibility tree. This means the native UITextField inside the wrapper becomes invisible to VibeView’s UI tree, and the input appears as a generic element rather than a text field.

This is a known iOS accessibility-container limitation that affects testing tools across the industry, including Appium, Detox, and Maestro.

How VibeView handles it

VibeView’s AI agent uses screenshots as its primary input, not just the UI tree. When the UI tree is incomplete (for example, showing generic elements instead of text fields), the agent falls back to visual understanding of the screenshot to identify inputs, buttons, and other interactive elements. It uses coordinate-based taps and text injection to interact with elements that are missing from the tree.

This means VibeView works with React Native apps even when the accessibility tree is incomplete, though providing better accessibility metadata will improve test reliability.

Recommended fix

For the best testing experience, set accessible={false} on wrapper components that contain interactive children:

{
  /* Before: TextInput hidden from accessibility tree */
}
<Pressable onPress={handleFocus}>
  <TextInput placeholder="Email" />
</Pressable>;

{
  /* After: TextInput visible as a proper text field */
}
<Pressable accessible={false} onPress={handleFocus}>
  <TextInput
    accessible={true}
    accessibilityLabel="Email"
    testID="email-input"
    placeholder="Email"
  />
</Pressable>;

This exposes the TextInput as a native text field in the accessibility tree, allowing the AI agent to identify it correctly and use optimized text input methods.

Tips

• Record stable, well-defined flows on the Devices page so runs replay them quickly. Write steps by hand, for the AI agent, for screens with dynamic content.
• Use Smart (the default) when you want replay speed with the AI agent as a safety net.
• Write specific, actionable step descriptions for AI only runs. “Tap the blue Submit button at the bottom of the form” works better than “Submit.”
• Keep app notes short and specific. The agent performs better with clear domain knowledge.
• Pick Clean app data as the start state when tests should not depend on what earlier tests left behind.
• Review the reasoning log when a test fails unexpectedly. It shows exactly what the agent saw and why it made each decision.
• For React Native apps, set accessible={false} on Pressable/TouchableOpacity wrappers around TextInput components to improve UI tree accuracy.