Skip to main content
An experiment gives the same scenarios, tasks you write, to several agents in several setups, and repeats them. After each run, a model checks it against yes-or-no criteria you wrote. Judging is free: it is not billed and doesn’t count against any run’s budget. Use one to:
  • Measure your product with agents: do they succeed more often with your MCP server, skill or CLI than without?
  • Compare agents and models on your own tasks rather than on public benchmarks.
  • Catch regressions in CI, by failing a build when the pass rate drops.

​
What an experiment holds

Scenarios

What the agents are asked to do: a prompt each, and the criteria a run must meet.

Agents

The agents and models that try every scenario.

Variants

The setups each scenario is tried in: a repository and a set of agent tools.

Run settings

What every run shares: repetitions, time limit, budget, scripts and variables.

​
Scenarios

A scenario is one task given to every agent:
  • Prompt: what the agent is asked to do.
  • Files: up to 20 files copied into each run’s sandbox.
  • Evaluations: the criteria, each with a name, a yes-or-no question for the judge, and a weight (1 unless you set one).
  • Pass rule: either Every evaluation passed (the default), or a Passing score out of 100, weighted by the evaluations’ weights.
The judge reads only what the run recorded: the agent’s messages, tool calls and their results. An agent saying it succeeded is not enough. Write questions that what the agent did can answer, such as “Did the agent run the tests and did they pass?”. See Write evaluations. A scenario without evaluations still runs, but its runs are not judged.

​
Agents

Each agent is an agent program and a model, such as Claude Code with claude-sonnet-5-5. Each pair appears once. See Agents and models.

​
Variants

A variant is one setup a scenario is tried in: a repository, or none, and one set of agent tools. Tracks makes a variant of every repository with every set of agent tools, minus the combinations you exclude.
  • Repositories come from your library. Without one, runs start in an empty folder.
  • Agent tools are the skills, MCP servers, command-line tools and browser given to the agent. Name each set, such as “With docs MCP”.
Keep one set of agent tools with nothing in it, named for example “No agent tools”. It is the baseline: without it you can’t tell whether your tools helped.
For example, one repository and these two sets make two variants:

​
Run settings

Agents don’t do the same thing twice. Use several repetitions to tell a real difference from chance: see Reading results.

​
The library

Repositories, skills, MCP servers and command-line tools are added once to your organization’s Library in the web app, then picked by any experiment.
  • Repositories: a public one by URL, with the branch, tag or commit to check out. For a private one, see private repositories.
  • Skills: uploaded, or added from a public GitHub URL.
  • MCP servers: with no sign-in, a bearer token, or a connection to your account on that server. One that needs you to connect must be connected before you launch.
  • Command-line tools: a name, which is the command the agent runs, and an install script.
tracks library lists them with the ids the API and CLI use.

​
Creating an experiment

In Experiments, click New experiment:
  • From scratch opens an empty experiment to fill in: Scenarios, Agents, Variants and Run settings.
  • From a document writes scenarios and their evaluations from a test plan, a spec, a checklist or an evals.json you paste or upload. It keeps your wording; review the result before launching.
Changes are saved as you go.

​
Launching

Launching starts an execution: one launch of the experiment, with a run for every scenario, variant, agent and repetition.
3 scenarios, 2 variants, 2 agents and 3 repetitions make 36 runs.
  • At most 500 runs. An experiment planning more can’t be launched.
  • Maximum spend is shown before you launch: the number of runs times the budget per run. 36 runs at $1 can spend up to $36. If your balance reaches $0 first, the remaining runs are canceled.
  • Executions are numbered #1, #2, #3… in launch order, and all are kept.
  • Each execution keeps the experiment as it was when launched. Editing the experiment, or a skill in your library, afterwards changes none of its runs.
The runs start in the background right after launch.
With --min-pass-rate, the command waits for every run to be judged and exits with an error below that pass rate: see Run from CI.

​
Slots

Slots are one row per run of the execution: one scenario, variant, agent and repetition number each. A slot is either:
  • not started yet;
  • refused: its run could not start, for example because your balance was $0 or an MCP server was disconnected since launch. A refused slot is never retried;
  • or holding its run, with its verdict once judged.
Read what each result means in Reading results.