Run from CI

Tracks fits in a CI pipeline: start a run on the commit under test and fail when it doesn’t complete, or launch an experiment and fail when its pass rate drops.

1. Give CI an API key

In the web app, open API keys and create a key named after where it’s used, such as CI. Save it as a secret named TRACKS_API_KEY in your CI: in GitHub, Settings → Secrets and variables → Actions → New repository secret. The CLI reads TRACKS_API_KEY from the environment. It needs Node.js 24 or later, and runs without installing with npx @alpic-ai/tracks. A CLI error exits with code 1 and prints <error>: <message> on stderr. A run or an experiment that didn’t turn out as required exits with code 2. Either fails the step.

2. Gate on a run

RUN=$(npx --yes @alpic-ai/tracks runs create "Run the tests, fix any that fail and explain the cause" \
  --harness claude-code --model claude-sonnet-5-5 \
  --repository https://github.com/alpic-ai/apps-sdk-template \
  --setup "npm install" --budget 5 \
  --idempotency-key "$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" | jq -r .id)

npx --yes @alpic-ai/tracks runs follow "$RUN" > /dev/null

--idempotency-key makes a retried step safe: creating a run with a key already used in your organization returns that run instead of starting another. Use one key per run you mean to start.
tracks runs follow waits until the run ends, then exits with code 2 when it ended before its agent finished, printing how, such as RunUnfinished: Run … ended with budget_exceeded before its agent finished. tracks runs get shows its failure when it failed.

completed means the agent stopped, not that it succeeded. To fail the build on what the agent actually did, judge its work with an experiment.

--repository checks out the repository’s default branch, and only works with public repositories. To work on the commit under test, fetch it in the setup script, as the workflow below does. For a private repository, see Private repositories.

3. Gate on an experiment

Create the experiment once, in the web app or with tracks experiments create, and keep its id. Then in CI:

npx --yes @alpic-ai/tracks experiments launch "$EXPERIMENT_ID" --min-pass-rate 80

--min-pass-rate waits until no run is pending, printing progress on stderr every 10 seconds, then prints the execution. It exits with code 2 when the pass rate is below 80, or when no run was judged, and retries a check that fails on the network a few times:

PassRateTooLow: The execution's pass rate is below 80%: 75%

An experiment’s runs use the repositories and agent tools from your library, not your pull request’s code. Use it to check what agents do with your MCP server or skill once deployed, see Measure your MCP server or skill. To keep the experiment’s definition in your repository, replace it before each launch with tracks experiments update "$EXPERIMENT_ID" experiment.json.

Every launch starts a run for every scenario, variant, agent and repetition, and pays for them. Launch on pushes to your main branch, or on demand, rather than on every pull request, unless the experiment is small.

A complete GitHub Actions workflow

Save it as .github/workflows/tracks.yml. It needs the TRACKS_API_KEY secret and, for the experiment, a TRACKS_EXPERIMENT_ID variable in Settings → Secrets and variables → Actions → Variables.

.github/workflows/tracks.yml

name: Tracks

on:
  pull_request:
  push:
    branches: [main]

defaults:
  run:
    shell: bash

env:
  TRACKS_API_KEY: ${{ secrets.TRACKS_API_KEY }}

jobs:
  agent-run:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      COMMIT: ${{ github.event.pull_request.head.sha }}
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 24

      - name: Start a run on this commit
        id: run
        run: |
          RUN=$(npx --yes @alpic-ai/tracks runs create \
            "Run the tests, fix any that fail and explain the cause" \
            --harness claude-code --model claude-sonnet-5-5 \
            --repository "https://github.com/$GITHUB_REPOSITORY" \
            --env GITHUB_REPOSITORY --env COMMIT \
            --setup 'git fetch -q --depth 1 "https://github.com/$GITHUB_REPOSITORY" "$COMMIT" && git checkout -q FETCH_HEAD && npm install' \
            --budget 5 --time-limit 1500 \
            --idempotency-key "$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" | jq -r .id)
          echo "id=$RUN" >> "$GITHUB_OUTPUT"

      - name: Wait for it to end, fail unless its agent finished
        run: npx --yes @alpic-ai/tracks runs follow "${{ steps.run.outputs.id }}" > /dev/null

  experiment:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    timeout-minutes: 120
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 24

      - name: Launch the experiment and require an 80% pass rate
        run: npx --yes @alpic-ai/tracks experiments launch "${{ vars.TRACKS_EXPERIMENT_ID }}" --min-pass-rate 80

The run’s setup script is in single quotes: the sandbox expands $GITHUB_REPOSITORY and $COMMIT, passed with --env, not the CI shell. --time-limit 1500 ends the run before the job’s 30-minute timeout. The last step fails when the run ended before its agent finished, and says how.

Label runs with your commit

Labels attach your own keys and values to a run, such as the commit it tested, to find it again. Add --label KEY:VALUE to tracks runs create, once per label:

npx --yes @alpic-ai/tracks runs create "Run the tests, fix any that fail and explain the cause" \
  --harness claude-code --model claude-sonnet-5-5 \
  --repository https://github.com/alpic-ai/apps-sdk-template \
  --label "commit:$GITHUB_SHA" --label workflow:ci

tracks runs list --label "commit:$GITHUB_SHA" then lists that commit’s runs, like GET /runs?label=commit:<sha>. A run takes up to 19 labels of yours: Tracks adds started_by, the API key that created the run.

Get started

Concepts

Guides

1. Give CI an API key

2. Gate on a run

3. Gate on an experiment

A complete GitHub Actions workflow

Label runs with your commit

​1. Give CI an API key

​2. Gate on a run

​3. Gate on an experiment

​A complete GitHub Actions workflow

​Label runs with your commit

1. Give CI an API key

2. Gate on a run

3. Gate on an experiment

A complete GitHub Actions workflow

Label runs with your commit