CI Testing with GitHub Actions

Run your VibeView test groups on every pull request. The workflow template on this page builds your app, runs the groups you tagged against that exact build on cloud iOS simulators and Android emulators, and reports the result on the pull request: one comment per platform, a job summary, and a job that fails when a test fails. The same jobs work for Apple TV, Android TV and Roku groups.

(The CLI and API call a group a suite: run-suites --tag runs every group with that tag.)

To build your app on every push to a branch instead (with an optional test run on each build), see Cloud builds in CI.

How it works

1. Build. iOS is built by cloud builds, so you don’t need a Mac runner. Android is built by your own job in the workflow. Either platform can use either option (see Customizing the workflow).
2. Upload. Each platform job ends up with the id of the build it just made, and every run is pinned to that build, never to “the latest upload”.
3. Run. vibeview run-suites runs every group with your tag on that platform, in hybrid mode (called Smart in the dashboard): a case’s recorded actions replay, and the AI agent (beta) handles only the steps that have no recording or whose replay fails.
4. Report. A report job writes a job summary and keeps one comment per platform up to date on the pull request, with a row per group and a link to every case’s run page.

If your organization has connected GitHub, every group also posts a commit status on the pull request’s head commit.

To run tests every day at a set time or on every new build without a workflow, use a schedule.

Before you start

1. Tag the groups the workflow should run, for example smoke, when you create a group or on its page (see the Tests page). A platform job runs the tagged groups of its own platform only.
2. Check the groups’ app. A run is pinned to the build the workflow made, so every tagged group of a platform must belong to the app that build is for. A group of another app fails the run before anything starts.
3. Create an API token for this repository. In Settings > API token, generate a token with the Developer role, CI only access, an expiry (for example 90 days) and a label naming the repository (creating tokens needs the Admin role; see API tokens). In GitHub, add it as the repository secret VIBEVIEW_API_TOKEN (Settings > Secrets and variables > Actions).
4. Set the tag. On the same GitHub page, open the Variables tab and add the repository variable VIBEVIEW_TAG with your tag.
5. Configure the iOS cloud build. The iOS job builds the ios entry of the vibeview.json in your repository (see Cloud Builds). For a React Native app, CI needs a build that embeds the JavaScript bundle: use the Release configuration. A Debug build waits for a Metro server that CI doesn’t have.
6. Replace the Android build step with your own build (see the comments in the workflow). The APK must include the x86_64 ABI, and a React Native APK must be a release build for the same reason as above.

Workflow template

Download vibeview-test.yml and save it as .github/workflows/vibeview.yml, or copy it from here:

# VibeView: run your tagged VibeView test groups against every pull request's build.
#
# Save this file as .github/workflows/vibeview.yml. Docs, and the latest version
# of this file: https://vibeview.io/docs/ci-testing
#
# One-time setup
#   1. In VibeView, tag the groups this workflow should run (for example "smoke").
#      Each platform job runs the groups with that tag for its own platform.
#      (The CLI calls a group a suite, so the command is run-suites. --mode hybrid is Smart in the dashboard.)
#   2. Create an API token for this repository only (VibeView > Settings >
#      API token, role Developer, access CI only, with an expiry) and save it
#      as the repository secret
#      VIBEVIEW_API_TOKEN (Settings > Secrets and variables > Actions).
#   3. Add the repository variable VIBEVIEW_TAG with that tag (same page, the
#      Variables tab).
#   4. iOS is built by VibeView cloud builds from the "ios" build block
#      in your vibeview.json. Android is built in this workflow: replace the
#      "Build" step of the build-android job with your own build.
#   5. Delete the jobs of a platform you don't ship, and remove them from the
#      report job's "needs".
#
# Pull requests from forks and from Dependabot are skipped: GitHub gives them
# no secrets. Never switch this workflow to pull_request_target.

name: VibeView

on:
  pull_request:
    paths-ignore:
      - '**/*.md'
      - 'docs/**'
  workflow_dispatch:

permissions:
  contents: read

defaults:
  run:
    # bash -eo pipefail: a failing command fails the step, also inside a pipeline.
    shell: bash

concurrency:
  group: vibeview-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

env:
  # The VibeView CLI version. Pinned: bump it deliberately.
  VIBEVIEW_CLI_VERSION: 0.28.1

jobs:
  ios:
    name: VibeView (iOS)
    if: >-
      github.event_name != 'pull_request' ||
      (github.event.pull_request.head.repo.full_name == github.repository &&
       github.event.pull_request.user.login != 'dependabot[bot]')
    runs-on: ubuntu-latest
    # Above the cloud build plus the CLI's own wait, so the CLI gives up first
    # and cancels the run on VibeView.
    timeout-minutes: 150
    env:
      VV_TAG: ${{ vars.VIBEVIEW_TAG }}
      HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
      PR_NUMBER: ${{ github.event.pull_request.number }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false

      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
          node-version: 22

      - name: Install the VibeView CLI
        run: npm install --global --no-audit --no-fund "vibeview@${VIBEVIEW_CLI_VERSION}"

      - name: Check the setup
        env:
          VIBEVIEW_API_TOKEN: ${{ secrets.VIBEVIEW_API_TOKEN }}
        run: |
          if [ -z "${VIBEVIEW_API_TOKEN}" ]; then
            echo "::error::The repository secret VIBEVIEW_API_TOKEN is not set."
            exit 1
          fi
          if [ -z "${VV_TAG}" ]; then
            echo "::error::The repository variable VIBEVIEW_TAG is not set."
            exit 1
          fi

      # Builds the "ios" block of vibeview.json on VibeView and adds the result
      # to your app's build history. For React Native, that build must embed
      # the JS bundle (a Release build): a Debug build needs Metro.
      - name: Build in the cloud
        id: build
        env:
          VIBEVIEW_API_TOKEN: ${{ secrets.VIBEVIEW_API_TOKEN }}
        run: |
          out="$RUNNER_TEMP/vibeview"
          mkdir -p "$out"
          status=0
          # The full build log is long: it is collapsed in the job log.
          echo "::group::Cloud build log"
          vibeview build --cloud --platform ios --json \
            | tee "$out/build.ndjson" \
            | jq --unbuffered -rj 'select(.event == "build_log") | .content' \
            || status=$?
          echo "::endgroup::"
          build_id=$(jq -rs '[.[] | select(.event == "build_finished" and .status == "succeeded")]
                             | last | .app_build_id // empty' "$out/build.ndjson")
          if [ "$status" -ne 0 ] || [ -z "$build_id" ]; then
            echo "The last 100 lines of the build log:"
            jq -rj 'select(.event == "build_log") | .content' "$out/build.ndjson" | tail -n 100
            jq -r 'select(.event == "error") | "error: \(.message)"' "$out/build.ndjson"
            echo "::error::The iOS cloud build failed. See the build log above (the full log is in the \"Cloud build log\" group)."
            exit 1
          fi
          echo "build_id=$build_id" >> "$GITHUB_OUTPUT"

      # A cancelled or timed-out job doesn't stop a cloud build by itself.
      - name: Cancel the cloud build
        if: cancelled()
        env:
          VIBEVIEW_API_TOKEN: ${{ secrets.VIBEVIEW_API_TOKEN }}
        run: |
          log="$RUNNER_TEMP/vibeview/build.ndjson"
          [ -s "$log" ] || exit 0
          for id in $(jq -r 'select(.event == "build_created") | .id' "$log"); do
            if ! jq -e --arg id "$id" 'select(.event == "build_finished" and .id == $id)' "$log" > /dev/null; then
              vibeview builds cancel "$id" || true
            fi
          done

      # exec: the CLI must be this step's own process, so a cancelled or
      # timed-out job stops the CLI and the CLI cancels the run on VibeView.
      # Exit code: 0 passed, 1 a test failed, 2 the run could not complete.
      - name: Run the tagged groups
        env:
          VIBEVIEW_API_TOKEN: ${{ secrets.VIBEVIEW_API_TOKEN }}
          BUILD_ID: ${{ steps.build.outputs.build_id }}
        run: |
          out="$RUNNER_TEMP/vibeview"
          jq -n --arg repo "$GITHUB_REPOSITORY" --arg sha "$HEAD_SHA" --arg pr "$PR_NUMBER" \
            --arg run "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" \
            '{github: {repository: $repo, sha: $sha, pull_request: $pr, workflow_run: $run}}' \
            > "$out/metadata.json"
          exec vibeview run-suites --tag "$VV_TAG" --platform ios --build "$BUILD_ID" \
            --mode hybrid --commit-sha "$HEAD_SHA" --metadata-file "$out/metadata.json" \
            --timeout 1800 --output json > "$out/result-$GITHUB_JOB.json"

      - name: Keep the result for the report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: vibeview-result-${{ github.job }}
          path: ${{ runner.temp }}/vibeview/result-${{ github.job }}.json
          if-no-files-found: ignore
          retention-days: 7

  # No VibeView secret in this job: the pull request's build scripts never see it.
  build-android:
    name: Build Android
    if: >-
      github.event_name != 'pull_request' ||
      (github.event.pull_request.head.repo.full_name == github.repository &&
       github.event.pull_request.user.login != 'dependabot[bot]')
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false

      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
          node-version: 22

      - uses: actions/setup-java@de7274f081f381c8f8158605e0321c36c376e2e6 # v6.0.1
        with:
          distribution: temurin
          java-version: 17

      # Replace with your own build (this one is a React Native project). The APK
      # must include the x86_64 ABI, and must run without a dev server: for
      # React Native, a release build embeds the JS bundle.
      - name: Build
        run: |
          npm ci
          cd android
          ./gradlew assembleRelease -PreactNativeArchitectures=x86_64,arm64-v8a
          cp app/build/outputs/apk/release/app-release.apk "$RUNNER_TEMP/app-android.apk"

      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: app-android
          path: ${{ runner.temp }}/app-android.apk
          retention-days: 3

  android:
    name: VibeView (Android)
    needs: build-android
    runs-on: ubuntu-latest
    # Above the CLI's own wait, so the CLI gives up first and cancels the run.
    timeout-minutes: 130
    env:
      VV_TAG: ${{ vars.VIBEVIEW_TAG }}
      HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
      PR_NUMBER: ${{ github.event.pull_request.number }}
    steps:
      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
          node-version: 22

      - name: Install the VibeView CLI
        run: npm install --global --no-audit --no-fund "vibeview@${VIBEVIEW_CLI_VERSION}"

      - name: Check the setup
        env:
          VIBEVIEW_API_TOKEN: ${{ secrets.VIBEVIEW_API_TOKEN }}
        run: |
          if [ -z "${VIBEVIEW_API_TOKEN}" ]; then
            echo "::error::The repository secret VIBEVIEW_API_TOKEN is not set."
            exit 1
          fi
          if [ -z "${VV_TAG}" ]; then
            echo "::error::The repository variable VIBEVIEW_TAG is not set."
            exit 1
          fi

      - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
        with:
          name: app-android
          path: ${{ runner.temp }}/app

      - name: Upload the build
        id: build
        env:
          VIBEVIEW_API_TOKEN: ${{ secrets.VIBEVIEW_API_TOKEN }}
        run: |
          out="$RUNNER_TEMP/vibeview"
          mkdir -p "$out"
          note="$GITHUB_REPOSITORY ${PR_NUMBER:+PR #$PR_NUMBER }${HEAD_SHA:0:7}"
          vibeview upload-app "$RUNNER_TEMP/app/app-android.apk" --note "$note" \
            --output json > "$out/upload.json"
          build_id=$(jq -er '.build_id' "$out/upload.json")
          echo "build_id=$build_id" >> "$GITHUB_OUTPUT"

      # exec: see the iOS job.
      - name: Run the tagged groups
        env:
          VIBEVIEW_API_TOKEN: ${{ secrets.VIBEVIEW_API_TOKEN }}
          BUILD_ID: ${{ steps.build.outputs.build_id }}
        run: |
          out="$RUNNER_TEMP/vibeview"
          jq -n --arg repo "$GITHUB_REPOSITORY" --arg sha "$HEAD_SHA" --arg pr "$PR_NUMBER" \
            --arg run "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" \
            '{github: {repository: $repo, sha: $sha, pull_request: $pr, workflow_run: $run}}' \
            > "$out/metadata.json"
          exec vibeview run-suites --tag "$VV_TAG" --platform android --build "$BUILD_ID" \
            --mode hybrid --commit-sha "$HEAD_SHA" --metadata-file "$out/metadata.json" \
            --timeout 1800 --output json > "$out/result-$GITHUB_JOB.json"

      - name: Keep the result for the report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: vibeview-result-${{ github.job }}
          path: ${{ runner.temp }}/vibeview/result-${{ github.job }}.json
          if-no-files-found: ignore
          retention-days: 7

  # Writes the job summary and one comment per platform on the pull request.
  # The only job that can write to the pull request, and it has no VibeView secret.
  report:
    name: VibeView report
    needs: [ios, android]
    if: >-
      !cancelled() &&
      (github.event_name != 'pull_request' ||
       (github.event.pull_request.head.repo.full_name == github.repository &&
        github.event.pull_request.user.login != 'dependabot[bot]'))
    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      pull-requests: write
    steps:
      - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
        continue-on-error: true
        with:
          pattern: vibeview-result-*
          path: ${{ runner.temp }}/results
          # One flat folder even when only one platform left a result (a single
          # matching artifact is otherwise unpacked straight into path/).
          merge-multiple: true

      - name: Summarize and comment
        env:
          GH_TOKEN: ${{ github.token }}
          NEEDS: ${{ toJSON(needs) }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
        run: |
          cat > "$RUNNER_TEMP/render.jq" <<'JQ'
          def clip($n): tostring | if length > $n then .[0:$n] + "..." else . end;
          # Names and messages come from your tests and your app: neutralize Markdown,
          # HTML and @mentions before they reach the comment.
          def esc: tostring | gsub("[\r\n\t]+"; " ")
            | gsub("&"; "&amp;") | gsub("<"; "&lt;") | gsub(">"; "&gt;")
            | gsub("(?<c>[\\\\`*_{}\\[\\]()#+!|~$])"; "\\\(.c)")
            | gsub("@"; "@​");
          def link($text; $url):
            if ($url | type) == "string" and ($url | test("^https://[^\\s()<>\\[\\]]+$"))
            then "[\($text)](\($url))" else $text end;
          def warned: .visual_regression_status == "warn" or .visual_regression_status == "fail";
          def icon: if .status == "passed" then (if warned then "⚠️" else "✅" end)
            elif .status == "cancelled" then "⚪"
            elif (.status == "queued" or .status == "pending" or .status == "running") then "⏳"
            else "❌" end;
          def note: if .status != "passed" and .error_message then ": " + (.error_message | clip(300) | esc)
            elif .status == "passed" and warned then ": passed with visual differences from its baseline"
            else "" end;
          def suite_note: if .status != "passed" and .error_message then ": " + (.error_message | clip(200) | esc) else "" end;
          def credit: if (.cost_micro | type) == "number"
            then "\nUsage credit: $\((.cost_micro / 100 | round) / 10000)\n" else "" end;
          def suites:
            "| Group | Result | Passed | Failed |\n| --- | --- | --- | --- |\n"
            + ([.suite_runs[] | "| \(icon) \(link(.suite_name // "Group" | clip(100) | esc; .url)) | \(.status | esc)\(suite_note) | \(.passed // 0) | \(.failed // 0) |"] | join("\n"))
            + "\n"
            + ([.suite_runs[] | .case_runs[]? | "- \(icon) \(link(.case_name // "Case" | clip(100) | esc; .url))\(note)"]
               | if length > 0 then "\n<details><summary>Test cases</summary>\n\n" + join("\n") + "\n\n</details>\n" else "" end);
          def steps:
            ([.steps[]? | "- \(if .status == "passed" then "✅" else "❌" end) \(.step | clip(200) | esc)"
              + (if .status != "passed" and .error then ": " + (.error | clip(300) | esc) else "" end)] | join("\n"));
          def tested: (if $sha | test("^[0-9a-f]{7,40}$") then "Commit `\($sha[0:7])` · " else "" end)
            + "[workflow run](\($run_url))\n\n";
          def header($icon; $status): "\($marker)\n### \($icon) VibeView (\($label)): \($status | esc)\n\n" + tested;
          if . == null then
            header("❌"; "no result")
            + (if $job_result == "skipped" then "The job did not run: a job it depends on failed or was skipped. "
               else "The job ended (\($job_result | esc)) before the tests reported a result. " end)
            + "See [the workflow run](\($run_url)).\n"
          elif .event? == "error" then
            header("❌"; "could not run") + (.message | clip(500) | esc) + "\n\n"
            + (if (.result | type) == "object" and (.result.suite_runs | type) == "array" then (.result | suites) else "" end)
            + (if .url then link("Open the run"; .url) + "\n" else "" end)
          elif (.suite_runs | type) == "array" then
            header(icon; .status) + suites + credit
          elif .type == "test_run" or (.steps | type) == "array" then
            header(icon; .status) + "\(link(.case_name // "Test" | clip(100) | esc; .url))\(note)\n\n" + steps + "\n" + credit
          else
            header("❔"; "unrecognized result") + "See [the workflow run](\($run_url)).\n"
          end
          JQ

          run_url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
          for job in $(jq -r 'keys[]' <<< "$NEEDS"); do
            case "$job" in
              ios) label="iOS" ;; android) label="Android" ;;
              tvos) label="Apple TV" ;; androidtv) label="Android TV" ;;
              *) label="$job" ;;
            esac
            job_result=$(jq -r --arg j "$job" '.[$j].result' <<< "$NEEDS")
            marker="<!-- vibeview-ci:$job -->"
            result="$RUNNER_TEMP/results/result-$job.json"
            body="$RUNNER_TEMP/comment-$job.md"
            args=(--arg marker "$marker" --arg label "$label" --arg job_result "$job_result" --arg run_url "$run_url"
                  --arg sha "$HEAD_SHA")
            if [ -s "$result" ] && jq -e 'type == "object"' "$result" > /dev/null 2>&1; then
              jq -r "${args[@]}" -f "$RUNNER_TEMP/render.jq" "$result" > "$body"
            else
              jq -r "${args[@]}" -f "$RUNNER_TEMP/render.jq" <<< 'null' > "$body"
            fi
            cat "$body" >> "$GITHUB_STEP_SUMMARY"

            [ -n "$PR_NUMBER" ] || continue
            comments="repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/comments"
            id=$(gh api --paginate "$comments" \
                   | jq -rs --arg m "$marker" '[.[][] | select(.user.login == "github-actions[bot]"
                       and (.body | startswith($m))) | .id] | first // empty')
            payload=$(jq -n --rawfile body "$body" '{body: $body}')
            if [ -n "$id" ]; then
              gh api --method PATCH "repos/$GITHUB_REPOSITORY/issues/comments/$id" --input - <<< "$payload" > /dev/null
            else
              gh api --method POST "$comments" --input - <<< "$payload" > /dev/null
            fi
          done

The workflow has four jobs:

Don’t ship one of the platforms? Delete its jobs and remove it from the report job’s needs.

Reading the results

• The platform jobs pass when every tagged group passed, and fail otherwise. vibeview run-suites exits 0 when the run passed, 1 when a test failed, and 2 when the run could not complete (no group matched the tag on that platform, a build that doesn’t belong to a group’s app, a timeout, or a cancelled run).
• The pull request comment, one per platform, is updated in place on every run. It names the commit it tested and links the workflow run, so a comment left from an earlier push (for example when a newer run was cancelled) is easy to spot. It has a row per group (linked to the group run), a list of the test cases (each linked to its run page, with the failure reason of a failed case), and the usage credit the run billed. A case that passed with visual differences from its baseline shows a ⚠️ instead of a ✅: this is informational and doesn’t fail the job.
• The job summary of the workflow run shows the same content, also for runs that aren’t on a pull request.
• Commit statuses, if your organization has connected GitHub: one per group, named vibeview/<suite name> (<platform>). See GitHub checks, which also covers which checks to make required.

Costs

• Each group run uses a live device session, which counts toward your organization’s concurrent sessions and streaming minutes like any other session (see Billing). When all the concurrent sessions your plan allows are in use, a run waits for one to free up instead of failing (up to 30 minutes, after which it ends in an error and never starts); the CLI shows it as queued. When the month’s streaming minutes are used up and overage is off (or your usage credit can’t cover overage), the run is refused with the same message the dashboard shows.
• In hybrid mode, recorded steps replay without the AI agent. Steps that use the AI agent (beta) are billed to usage credit, unless you use your own AI provider key (see Usage credit).
• The iOS cloud build is billed like any other cloud build.
• The workflow skips pull requests that change only Markdown files or the docs/ folder (paths-ignore), and a new push to a pull request cancels the run still going for the previous one (concurrency).

Retries

Retries are off unless you ask for them. To retry a failed test once, pass --retry to vibeview run-test, run-suite or run-suites, or retry_failed: true in the API (see CI API); a schedule can be saved with Retry a failed test once. The retry runs on the same device, after a fresh start of the app, and is its own run: the first attempt keeps its failure and the retry decides the result. A test that fails and then passes on the retry passes the run and is marked flaky. A retry never repeats a test that failed because its device session ended, or a quarantined test. A retried test uses AI and streaming minutes again, like any run. When vibeview run-test retried the test, its result links to the retry’s run page (Retry: <link>) and then to the first attempt’s (First attempt (failed): <link>); JSON output has the retry’s link in url and the first attempt under first_attempt, and JUnit output links the retry.

Without retries, a test is still marked flaky when both passes and failures show up among its recent runs (see The test list). --no-retry is still accepted, so existing scripts keep working, but it changes nothing.

Quarantined tests

A quarantined test keeps running and recording results, but its failure doesn’t fail a group run: a group run, run-suites and the dashboard’s group runs stay green. The failure still shows in the results as a quarantined failure: on the run page its row reads failed in amber with a Quarantined label, and the run’s test count, the run-suites summary (1 passed, 1 failed (quarantined)) and the MCP get_run result count it apart from real failures. JUnit output reports it as skipped. vibeview run-test of that one test reports it as FAIL (quarantined) (status: "failed" with quarantined: true in JSON output) and, like a group run, exits 0 for it, so it doesn’t fail the build. Earlier CLI versions exit 1 there.

Not checked and not run

A test in which no step was checked ends Not checked: it is neither a pass nor a failure. This happens when a No AI run (--mode replay), which never uses AI, finds no recorded or learned path for any of the test’s steps on its device (for example, when the test’s only learned path was saved on another kind of device). A test where only some steps weren’t checked still passes, and says how many weren’t. A test with no recording and no learned path at all is not run as Not checked: a No AI run of it is refused before it starts (see AI use).

• vibeview run-test, run-suite and run-suites print it as NOT CHECKED and exit 0 for it: it never fails the build, and it never counts as passed. JUnit output reports it as skipped (<skipped message="not checked: ...">).
• The CI API reports it with status: "passed" and not_checked: true, so scripts and earlier CLI versions that know only passed, failed, error and cancelled treat it as finished and not failing. Read not_checked to tell it apart (see CI API).
• GitHub gets a success status worded “not checked”, which doesn’t block a merge, and webhooks get run.not_checked instead of run.passed. No failure notice is sent.
• A group run in which every test was not checked is Not checked; one with at least one passed test and no failure passes, with the tests not checked counted apart.
• Under Stop on first failure, a not-checked test is not a failure, so the run goes on.

A test that didn’t run because an earlier test failed under Stop on first failure shows as Not run (“Not run: an earlier test failed, and this run stops at the first failure.”). The API keeps its status as cancelled and adds not_run: true; JUnit output reports it as skipped, not as an error.

Blocked runs

A run that couldn’t finish for a reason that isn’t your app (the device or session was lost, the app couldn’t be installed, or the run was stopped because usage credit ran out) is reported as blocked, not as a failure. GitHub gets an error status instead of failure, and the run.blocked webhook event is sent instead of run.failed. See Webhooks and Slack.

Security

The template is written to be safe on a public repository:

• Least privilege. The workflow token can only read the repository. Only the report job may write, and only to pull requests. The build job never sees the VibeView token, so a pull request’s build scripts can’t read it, and within the platform jobs only the steps that call VibeView get it.
• No expressions in scripts. Every value reaches a script through env:, never through ${{ }} inside run:. Test names, step text and error messages (which can quote text from your app’s screen) only ever reach the comment as escaped Markdown, built from files with jq and posted with gh api.
• Forks and Dependabot are skipped. GitHub gives their pull requests no secrets, so the jobs are skipped instead of failing. Never switch this workflow to pull_request_target to test them: that would run the pull request’s code with your token. To test a fork’s change, push it to a branch in your repository after review.
• Pinned versions. Each action is pinned to a full commit SHA (the version is in the comment next to it), and the CLI to an exact version (VIBEVIEW_CLI_VERSION). Let Dependabot bump the actions by adding the github-actions ecosystem to your .github/dependabot.yml, and bump the CLI version deliberately.
• A dedicated token. Give this repository its own Developer-role token with CI only access and an expiry, and store it only as a repository (or environment) secret. A CI-only token can upload builds, start and cancel test runs, and read their results, and nothing else; an expiring token stops working on its own if it’s forgotten. Before it expires, create a new one and update the secret. Delete it in Settings > API token if it leaks or when the repository no longer needs it. See CI-only tokens.

Customizing the workflow

Build iOS on your own macOS runner

Replace the cloud build with a build job of your own, like the Android one. Add this job:

  build-ios:
    name: Build iOS
    if: >-
      github.event_name != 'pull_request' ||
      (github.event.pull_request.head.repo.full_name == github.repository &&
       github.event.pull_request.user.login != 'dependabot[bot]')
    runs-on: macos-15
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false

      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
          node-version: 22

      # Replace with your own build: an iOS SIMULATOR build. For React Native,
      # the Release configuration embeds the JS bundle.
      - name: Build
        run: |
          npm ci
          (cd ios && pod install)
          xcodebuild -workspace ios/MyApp.xcworkspace -scheme MyApp -configuration Release \
            -sdk iphonesimulator -destination 'generic/platform=iOS Simulator' \
            -derivedDataPath build CODE_SIGNING_ALLOWED=NO
          # Workflow artifacts drop file permissions, so zip the .app bundle first.
          ditto -c -k --sequesterRsrc --keepParent \
            build/Build/Products/Release-iphonesimulator/MyApp.app "$RUNNER_TEMP/app-ios.zip"

      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: app-ios
          path: ${{ runner.temp }}/app-ios.zip
          retention-days: 3

Then, in the ios job:

1. Add needs: build-ios, and remove the if: (the build job already has it).
2. Remove the actions/checkout step and the Build in the cloud and Cancel the cloud build steps.
3. After Check the setup, copy the Android job’s actions/download-artifact and Upload the build steps, with app-android changed to app-ios and app-android.apk to app-ios.zip.

Build Android in the cloud

Delete the build-android job. In the android job, remove needs: build-android, add the ios job’s if:, and replace the actions/download-artifact and Upload the build steps with the ios job’s actions/checkout, Build in the cloud and Cancel the cloud build steps, with --platform ios changed to --platform android (and the error message to match). The android entry of your vibeview.json must have a build block.

Apple TV and Android TV

Copy a platform job (and its build job), rename it tvos or androidtv, and in Run the tagged groups change --platform to tvos or androidtv. The CLI picks a TV device for a TV group on its own. Add the new job to the report job’s needs: the report labels it Apple TV or Android TV. An Android TV APK needs both the x86 and x86_64 ABIs (see Preparing your build).

Roku

Roku channels have no cloud builds: package the channel in your own job (the zip you would sideload), upload it, and run the tagged Roku groups against that build:

BUILD_ID=$(vibeview upload-app ./out/channel.zip --output json | jq -er '.build_id')
vibeview run-suites --tag smoke --platform roku --build "$BUILD_ID" --commit-sha "$GIT_SHA"

Each session installs the channel before the first case. The group’s start state applies: Clean app data and Fresh install remove the channel and install it again before each case, because a Roku keeps a channel’s saved data for as long as it is installed (see TV devices). Build the channel with the VibeView Roku plugin so its tests replay by element.

Run on pushes to your main branch

Add a push trigger. The platform jobs then run on every push to main too, and post commit statuses on that commit (there is no pull request to comment on, so the results go to the job summary):

on:
  pull_request:
    paths-ignore:
      - '**/*.md'
      - 'docs/**'
  push:
    branches: [main]
  workflow_dispatch:

Mode and timeouts

• --mode hybrid (the default, called Smart in the dashboard) replays recorded steps and uses the AI agent (beta) for the rest. --mode replay (No AI) never uses AI, and refuses a group that has no recorded case. --mode ai (AI only) runs every step with the AI agent. See AI testing.
• POST /api/v1/tests/runs takes the same fields as the CI routes and also runs on several devices at once; see Tests API.
• --timeout 1800 is per group, and covers waiting for a free device as well as the run itself. The CLI’s total wait is that times the number of matched groups, capped at 2 hours. When it gives up, it cancels the run on VibeView before exiting. --timeout takes whole seconds from 1 to 7200, and --mode takes hybrid (Smart), ai (AI only) or replay (No AI); any other value stops the command with exit code 2 before a run starts.
• Keep each platform job’s timeout-minutes above the CLI’s total wait (plus the build, for a cloud build), so the CLI stops first and cleans up. A cancelled or timed-out job stops the CLI (the run step execs it for that reason), and the CLI then cancels the run.

AI steps from test-spec.json

You can also describe a test in plain English in .vibeview/test-spec.json and have the AI agent (beta) run it. This suits AI coding agents that write the test along with the change. Prefer tagged groups for regression testing: every run of a spec is a full AI run billed to usage credit, while recorded groups replay without AI.

{
  "version": 1,
  "test_name": "Login Flow",
  "steps": [
    "Launch the app",
    "Tap the Sign In button",
    "Enter 'testuser@example.com' in the email field",
    "Enter 'password123' in the password field",
    "Tap Submit",
    "Verify the dashboard screen loads with a welcome message"
  ],
  "context": "The app has a test account: testuser@example.com / password123. The Sign In button is on the first screen."
}

To run it, replace Run the tagged groups in a platform job with this step. It needs the app’s id: in the android job, add echo "app_id=$(jq -er '.app_id' "$out/upload.json")" >> "$GITHUB_OUTPUT" to Upload the build; in the ios job, add echo "app_id=$(jq -rs '[.[] | select(.event == "build_finished" and .status == "succeeded")] | last | .app_public_id' "$out/build.ndjson")" >> "$GITHUB_OUTPUT" to Build in the cloud.

      - name: Run the AI test from test-spec.json
        env:
          VIBEVIEW_API_TOKEN: ${{ secrets.VIBEVIEW_API_TOKEN }}
          BUILD_ID: ${{ steps.build.outputs.build_id }}
          APP_ID: ${{ steps.build.outputs.app_id }}
        run: |
          out="$RUNNER_TEMP/vibeview"
          spec=.vibeview/test-spec.json
          if [ ! -f "$spec" ]; then
            echo "::error::$spec not found. Is it committed? A .gitignore line that ignores the whole .vibeview/ directory keeps it out of git: see Keep test-spec.json in git."
            exit 1
          fi
          if ! jq -e '.version == 1 and (.steps | type == "array" and length > 0)' "$spec" > /dev/null; then
            echo "::error::$spec needs \"version\": 1 and a non-empty \"steps\" array."
            exit 1
          fi
          jq '.steps' "$spec" > "$out/steps.json"
          name=$(jq -r '.test_name // "CI Test"' "$spec")
          context=$(jq -r '.context // ""' "$spec")
          args=(--app "$APP_ID" --steps-file "$out/steps.json" --if-exists update --output json)
          if [ -n "$context" ]; then args+=(--context "$context"); fi
          vibeview create-test "$name" "${args[@]}" > "$out/test.json"
          test_id=$(jq -er '.test_id' "$out/test.json")
          exec vibeview run-test "$test_id" --build "$BUILD_ID" --commit-sha "$HEAD_SHA" \
            --timeout 1800 --output json > "$out/result-$GITHUB_JOB.json"

This step needs actions/checkout in the job (the android job doesn’t have it yet). The report shows the test’s steps with the failure reason of a failed step. With the GitHub integration connected, the run posts a commit status named vibeview/<test name> (<platform>).

Keep test-spec.json in git

vibeview dev keeps local session state in the same .vibeview/ folder. So that state isn’t committed while test-spec.json still is, it writes a .vibeview/.gitignore inside the folder:

*
!test-spec.json

It doesn’t add lines to your project’s own .gitignore (older CLI versions added .vibeview/* and !.vibeview/test-spec.json there, which still work). A line that ignores the whole folder (.vibeview/, which even older versions added) keeps test-spec.json out of git: Git can’t re-include a file inside an ignored folder. The next vibeview dev replaces that line with .vibeview/* and !.vibeview/test-spec.json; you can also edit it yourself. Check with git check-ignore -v .vibeview/test-spec.json, which prints nothing when the file isn’t ignored. If the file is missing in CI, the step above fails with “not found”.

Keep in mind:

• The spec is read from the pull request, so anyone who can open a pull request in the repository can change the steps. They reach the CLI as a file and as quoted arguments, never as shell code.
• --if-exists update makes each run overwrite the app’s test with this name that is in no group (or, with --group, in one of those groups), using the spec’s steps. Give it a name no other test of the app uses: an existing one with the same name, recorded or not, is replaced. Two pull requests running at the same moment share the case, and the one that updates it last decides the steps both run. Without --if-exists, create-test reuses an identical case and refuses a different one.

Other CI systems

On GitLab CI, CircleCI, Jenkins or any other system, install the CLI, set VIBEVIEW_API_TOKEN as a masked secret variable, and run the same commands:

npm install --global vibeview@0.28.1

# Upload your build and pin the run to it (a zipped simulator .app, or a
# .app directory, for iOS; an .apk for Android).
BUILD_ID=$(vibeview upload-app ./build/app-release.apk --output json | jq -er '.build_id')

vibeview run-suites --tag smoke --platform android --build "$BUILD_ID" \
  --commit-sha "$(git rev-parse HEAD)" --output junit > vibeview-results.xml

Or build in the cloud and take the build’s id from the build_finished event:

BUILD_ID=$(vibeview build --cloud --platform ios --json \
  | jq -rs '[.[] | select(.event == "build_finished" and .status == "succeeded")] | last | .app_build_id // empty')

--output junit writes a JUnit XML report most CI systems can display, with a link to each run. --output json writes the result as one JSON document on stdout (progress goes to stderr). On an error, the JSON is a single {"event": "error", "message": "..."} object, and the JUnit report has one test case with an <error>. The exit codes are the same as in Reading the results. If the build a run installs is a debug build that needs a Metro dev server, the command prints a warning when the run starts (and adds build_warning to the JSON), because the run will most likely stop at the app’s error screen: see Builds that need a Metro dev server.

Pinning a specific build

Every run command takes --build <id>, and the workflow template always passes it. Without it, a run uses the app’s latest build that doesn’t need a Metro dev server (if every build needs Metro, the run still starts on the latest one, with a warning), which may not be the build your pipeline just made. The id is the numeric build id that upload-app --output json returns as build_id and a cloud build reports as app_build_id. You can also find it on the app’s page under Build history, or with vibeview list-builds <app>. A --build value that isn’t a build id (a typo such as 12a) stops the command with exit code 2; it never falls back to the latest build.

vibeview run-suites --tag smoke --platform ios --build 142
vibeview run-suite <suite-id> --build 142
vibeview run-test <test-id> --build 142

The CI API’s run endpoints (POST /api/v1/ci/run-test, /run-suite and /run-suites) take the same value as an optional build_id integer in the JSON body. When the case or group has no app attached, sending build_id returns HTTP 400.

run-test and run-suite also take --platform to run a recording on another platform of the same form factor: the run uses the app you linked to the test’s app as the same product (see Same app on other platforms), or, if none is linked, the app with the same bundle id or package. The run uses the test data, notes and environment values of the platform it runs on (see Linked apps). Pass --app <id> to name the app yourself. A phone test never runs on a TV. (run-suites --platform is different: it only filters which tagged groups run.)

Reading run results

Run commands (run-test, run-suite, run-suites) are async: the server accepts the request and returns its id immediately, and the CLI polls until the run completes. run-test and run-suite poll GET /api/v1/ci/runs/{run_id} every 3 seconds; run-suites polls GET /api/v1/ci/multi-suite-runs/{id} every 2 seconds. This section documents the runs payload for programmatic consumers who call the endpoint directly.

Response shapes

The poll response has two shapes depending on whether the run is a single test or a group.

type: "test_run" — a single test case run. Top-level run fields plus a steps[] array (present once the run reaches a terminal status):

{
  "run_id": "tr_abc123",
  "type": "test_run",
  "status": "failed",
  "case_name": "Login Flow",
  "duration_ms": 8420,
  "run_url": "https://vibeview.io/tests/runs/tr_abc123",
  "model_used": "claude-4-sonnet",
  "progress": null,
  "steps": [
    {
      "step": "Tap the Sign In button",
      "status": "failed",
      "error": "Element not found: Sign In button",
      "error_kind": "element_not_found",
      "error_kind_description": "The element the step needed was not present or could not be located on screen.",
      "duration_ms": 3100,
      "screenshot_url": "/api/v1/tests/test-runs/tr_abc123/screenshots/step_1.jpg",
      "screenshot_signed_url": "https://vibeview.io/api/v1/public/screenshots/tr_abc123/step_1.jpg?exp=1746835200&sig=abc123",
      "iteration_screenshots": []
    }
  ]
}

While the run is still in flight, steps is absent and progress carries the live step-progress events instead; progress is null once the run is terminal.

A finished test_run also carries steps_not_checked: how many steps a replay run could not check (they have no recorded actions, typically Verify steps, and a No AI run never uses AI). Such a step’s status in steps[] is not_checked, with a note saying why; it is neither a pass nor a failure. Treat a passed run with steps_not_checked > 0 as not fully checked — the CLI prints PASS (N steps not checked) for it, and its JUnit report keeps the case passed with a steps_not_checked property. A run in which no step was checked is not a pass: it carries not_checked: true (see Not checked and not run). Each test_runs[] entry of a group run carries the same count. assertion_results and assertions_summary are deprecated and always null.

type: "suite_run" — a group run. Top-level run fields plus a test_runs[] array of per-case entries. total_cases is the group’s full case count — test_runs[] only contains cases that have started, so use total_cases as the denominator for “N/M complete”. The top-level run_url points at this group run’s own results page (/tests/runs/{run_id}); each per-case entry’s url points at that case run’s own page.

{
  "run_id": "sr_def456",
  "type": "suite_run",
  "status": "failed",
  "suite_public_id": "suite_abc123",
  "suite_name": "Login Suite",
  "duration_ms": 24800,
  "run_url": "https://vibeview.io/tests/runs/sr_def456",
  "total_cases": 4,
  "test_runs": [
    {
      "public_id": "tr_ghi789",
      "case_name": "Login Flow",
      "status": "failed",
      "duration_ms": 8200,
      "error_message": "Visual mismatch on home screen",
      "error_kind": "visual_regression",
      "error_kind_description": "The screen differed from its visual baseline beyond the allowed threshold.",
      "screenshot_signed_url": "https://vibeview.io/api/v1/public/screenshots/tr_ghi789/step_3.jpg?exp=1746835200&sig=xyz789",
      "url": "https://vibeview.io/tests/runs/tr_ghi789"
    }
  ]
}

Both shapes also carry started_at, finished_at, and error_message at the top level. While a run is waiting for a device (status: "queued"), the payload includes a queue block with entry_id, position, specific_device_id, and reason: why it waits (devices_busy, devices_updating or plan_limit), or null when VibeView can’t tell.

screenshot_url vs screenshot_signed_url

Each failed step or group case includes two screenshot fields:

• screenshot_url — a relative API path (/api/v1/tests/test-runs/{run_id}/screenshots/{file}). Resolve it against your VibeView API host and send an Authorization: Bearer <token> header to fetch it. A CI-only token can fetch it too, for runs in its own organization.
• screenshot_signed_url — a signed, time-limited public URL. No auth header needed. Returns image/jpeg directly (suitable for embedding in Slack messages, PR comments, or email). Expires after 7 days; returns HTTP 403 if expired or the signature is invalid.

error_kind values

The error_kind field classifies why a step or group case failed. It is populated for failed or errored steps/cases and null for passed ones. Every payload that carries error_kind also carries error_kind_description — a human-readable sentence explaining the classification.

AI-supplied — the AI agent picks these when it calls step_failed. A step that fails in a No AI run also uses element_not_found, wrong_state or app_unresponsive when that is why the recorded action could not be replayed:

System-supplied — set in code when the cause is known:

Fallback:

For webhook payloads that also carry error_kind, see the error_kind table in the Webhooks & Slack guide.

Filtering groups by tag

Tag groups in VibeView (see the Tests page), then have CI run a tag-filtered set instead of hard-coding group ids. In a monorepo, different teams can own different tags.

Discovering tags

vibeview list-suites --tag profiles

Lists every group tagged profiles, so you can confirm what would run.

Running by tag

vibeview run-suites --tag profiles --platform ios --build "$BUILD_ID" --commit-sha "$GIT_SHA"

Runs every matching group in turn, one device session per group, and reports one aggregate result. Exit codes: 0 the aggregate passed, 1 a group failed, 2 no match, a timeout, an error or a cancelled run.

--platform (ios, android, tvos, androidtv or roku) keeps only the matching groups of that platform, and lists the others as excluded. A group’s platform is its own, or else its app’s. Use --match all when a group must have every listed tag: --tag smoke --tag checkout --match all runs only the groups tagged both smoke and checkout.

To pin a tagged run to one device, list the devices with vibeview list-devices and pass --device.

Running a tag only when certain files change

Give that tag its own workflow, triggered by the paths it covers. For example, a copy of the template whose trigger is:

on:
  pull_request:
    paths:
      - 'src/profiles/**'

and whose VIBEVIEW_TAG is replaced by the literal tag in its jobs’ env: (VV_TAG: profiles). Change the comment marker too if both workflows comment on the same pull request: the report finds its comment by the <!-- vibeview-ci:<job> --> marker, so rename the jobs (for example ios-profiles).

Version History