> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tracks.alpic.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run from CI

> Start runs and experiments from your CI, and fail the build on how they end

Tracks fits in a CI pipeline: start a run on the commit under test and fail when it doesn't complete, or launch an experiment and fail when its pass rate drops.

## 1. Give CI an API key

In the [web app](https://tracks.alpic.ai), open **API keys** and create a key named after where it's used, such as `CI`. Save it as a secret named `TRACKS_API_KEY` in your CI: in GitHub, **Settings** → **Secrets and variables** → **Actions** → **New repository secret**.

The CLI reads `TRACKS_API_KEY` from the environment. It needs Node.js 24 or later, and runs without installing with `npx @alpic-ai/tracks`.

A CLI error exits with code 1 and prints `<error>: <message>` on stderr. A run or an experiment that didn't turn out as required exits with code 2. Either fails the step.

## 2. Gate on a run

```bash theme={null}
RUN=$(npx --yes @alpic-ai/tracks runs create "Run the tests, fix any that fail and explain the cause" \
  --harness claude-code --model claude-sonnet-5-5 \
  --repository https://github.com/alpic-ai/apps-sdk-template \
  --setup "npm install" --budget 5 \
  --idempotency-key "$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" | jq -r .id)

npx --yes @alpic-ai/tracks runs follow "$RUN" > /dev/null
```

* **`--idempotency-key`** makes a retried step safe: creating a run with a key already used in your organization returns that run instead of starting another. Use one key per run you mean to start.
* **`tracks runs follow`** waits until the run ends, then exits with code 2 when it ended before its agent finished, printing how, such as `RunUnfinished: Run … ended with budget_exceeded before its agent finished`. `tracks runs get` shows its `failure` when it failed.

`completed` means the agent stopped, not that it succeeded. To fail the build on what the agent actually did, judge its work with an experiment.

> [!NOTE]
>
> `--repository` checks out the repository's default branch, and only works with public repositories. To work on the
> commit under test, fetch it in the setup script, as the workflow below does. For a private repository, see [Private
> repositories](https://docs.tracks.alpic.ai/guides/private-repositories.md).

## 3. Gate on an experiment

Create the experiment once, in the web app or with `tracks experiments create`, and keep its id. Then in CI:

```bash theme={null}
npx --yes @alpic-ai/tracks experiments launch "$EXPERIMENT_ID" --min-pass-rate 80
```

`--min-pass-rate` waits until no run is pending, printing progress on stderr every 10 seconds, then prints the execution. It exits with code 2 when the pass rate is below 80, or when no run was judged, and retries a check that fails on the network a few times:

```text theme={null}
PassRateTooLow: The execution's pass rate is below 80%: 75%
```

An experiment's runs use the repositories and agent tools from your library, not your pull request's code. Use it to check what agents do with your MCP server or skill once deployed, see [Measure your MCP server or skill](https://docs.tracks.alpic.ai/guides/measure-your-mcp-server.md).

To keep the experiment's definition in your repository, replace it before each launch with `tracks experiments update "$EXPERIMENT_ID" experiment.json`.

> [!WARNING]
>
> Every launch starts a run for every scenario, variant, agent and repetition, and pays for them. Launch on pushes to
> your main branch, or on demand, rather than on every pull request, unless the experiment is small.

## A complete GitHub Actions workflow

Save it as `.github/workflows/tracks.yml`. It needs the `TRACKS_API_KEY` secret and, for the experiment, a `TRACKS_EXPERIMENT_ID` variable in **Settings** → **Secrets and variables** → **Actions** → **Variables**.

```yaml .github/workflows/tracks.yml theme={null}
name: Tracks

on:
  pull_request:
  push:
    branches: [main]

defaults:
  run:
    shell: bash

env:
  TRACKS_API_KEY: ${{ secrets.TRACKS_API_KEY }}

jobs:
  agent-run:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      COMMIT: ${{ github.event.pull_request.head.sha }}
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 24

      - name: Start a run on this commit
        id: run
        run: |
          RUN=$(npx --yes @alpic-ai/tracks runs create \
            "Run the tests, fix any that fail and explain the cause" \
            --harness claude-code --model claude-sonnet-5-5 \
            --repository "https://github.com/$GITHUB_REPOSITORY" \
            --env GITHUB_REPOSITORY --env COMMIT \
            --setup 'git fetch -q --depth 1 "https://github.com/$GITHUB_REPOSITORY" "$COMMIT" && git checkout -q FETCH_HEAD && npm install' \
            --budget 5 --time-limit 1500 \
            --idempotency-key "$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" | jq -r .id)
          echo "id=$RUN" >> "$GITHUB_OUTPUT"

      - name: Wait for it to end, fail unless its agent finished
        run: npx --yes @alpic-ai/tracks runs follow "${{ steps.run.outputs.id }}" > /dev/null

  experiment:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    timeout-minutes: 120
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 24

      - name: Launch the experiment and require an 80% pass rate
        run: npx --yes @alpic-ai/tracks experiments launch "${{ vars.TRACKS_EXPERIMENT_ID }}" --min-pass-rate 80
```

The run's setup script is in single quotes: the sandbox expands `$GITHUB_REPOSITORY` and `$COMMIT`, passed with `--env`, not the CI shell. `--time-limit 1500` ends the run before the job's 30-minute timeout. The last step fails when the run ended before its agent finished, and says how.

## Label runs with your commit

Labels attach your own keys and values to a run, such as the commit it tested, to find it again. Add `--label KEY:VALUE` to `tracks runs create`, once per label:

```bash theme={null}
npx --yes @alpic-ai/tracks runs create "Run the tests, fix any that fail and explain the cause" \
  --harness claude-code --model claude-sonnet-5-5 \
  --repository https://github.com/alpic-ai/apps-sdk-template \
  --label "commit:$GITHUB_SHA" --label workflow:ci
```

`tracks runs list --label "commit:$GITHUB_SHA"` then lists that commit's runs, like `GET /runs?label=commit:<sha>`. A run takes up to 19 labels of yours: Tracks adds `started_by`, the API key that created the run.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.