> ## Documentation Index > Fetch the complete documentation index at: https://docs.tracks.alpic.ai/llms.txt > Use this file to discover all available pages before exploring further. # Experiments > Give the same scenarios to several agents, with and without your tools, and see which ones succeed An experiment gives the same scenarios, tasks you write, to several agents in several setups, and repeats them. After each run, a model checks it against yes-or-no criteria you wrote. Judging is free: it is not billed and doesn't count against any run's budget. Use one to: * **Measure your product with agents**: do they succeed more often with your MCP server, skill or CLI than without? * **Compare agents and models** on your own tasks rather than on public benchmarks. * **Catch regressions** in CI, by failing a build when the pass rate drops. ## What an experiment holds > **Scenarios** > > What the agents are asked to do: a prompt each, and the criteria a run must meet. > **Agents** > > The agents and models that try every scenario. > **Variants** > > The setups each scenario is tried in: a repository and a set of agent tools. > **Run settings** > > What every run shares: repetitions, time limit, budget, scripts and variables. ### Scenarios A scenario is one task given to every agent: * **Prompt**: what the agent is asked to do. * **Files**: up to 20 files copied into each run's sandbox. * **Evaluations**: the criteria, each with a name, a yes-or-no question for the judge, and a weight (1 unless you set one). * **Pass rule**: either **Every evaluation passed** (the default), or a **Passing score** out of 100, weighted by the evaluations' weights. The judge reads only what the run recorded: the agent's messages, tool calls and their results. An agent saying it succeeded is not enough. Write questions that what the agent did can answer, such as "Did the agent run the tests and did they pass?". See [Write evaluations](https://docs.tracks.alpic.ai/guides/write-evaluations.md). A scenario without evaluations still runs, but its runs are not judged. ### Agents Each agent is an agent program and a model, such as Claude Code with `claude-sonnet-5-5`. Each pair appears once. See [Agents and models](https://docs.tracks.alpic.ai/concepts/agents.md). ### Variants A variant is one setup a scenario is tried in: a repository, or none, and one set of agent tools. Tracks makes a variant of every repository with every set of agent tools, minus the combinations you exclude. * **Repositories** come from your library. Without one, runs start in an empty folder. * **Agent tools** are the skills, MCP servers, command-line tools and browser given to the agent. Name each set, such as "With docs MCP". > [!TIP] > > Keep one set of agent tools with nothing in it, named for example "No agent tools". It is the baseline: without it you > can't tell whether your tools helped. For example, one repository and these two sets make two variants: | Agent tools | Skills | MCP servers | Command-line tools | | - | - | - | - | | No agent tools | none | none | none | | With docs MCP | none | `docs` | none | ### Run settings | Setting | Meaning | Web app default | | - | - | - | | **Repetitions** | Identical runs started for each scenario, variant and agent. | 1 | | **Time limit** | Longest each run may last, up to 12 hours. | 10 minutes | | **Budget per run** | Most each run may spend, up to \$10,000. | \$1 | | **Setup script** | Run in each sandbox after the repository is cloned, before the prompt. | none | | **Teardown script** | Run in each sandbox after the run ends. | none | | **Environment variables** | Set in every process of the agent. | none | Agents don't do the same thing twice. Use several repetitions to tell a real difference from chance: see [Reading results](https://docs.tracks.alpic.ai/concepts/reading-results.md#why-repetitions-matter). ## The library Repositories, skills, MCP servers and command-line tools are added once to your organization's **Library** in the web app, then picked by any experiment. * **Repositories**: a public one by URL, with the branch, tag or commit to check out. For a private one, see [private repositories](https://docs.tracks.alpic.ai/guides/private-repositories.md). * **Skills**: uploaded, or added from a public GitHub URL. * **MCP servers**: with no sign-in, a bearer token, or a connection to your account on that server. One that needs you to connect must be connected before you launch. * **Command-line tools**: a name, which is the command the agent runs, and an install script. `tracks library` lists them with the ids the API and CLI use. ## Creating an experiment **Web app** In **Experiments**, click **New experiment**: * **From scratch** opens an empty experiment to fill in: **Scenarios**, **Agents**, **Variants** and **Run settings**. * **From a document** writes scenarios and their evaluations from a test plan, a spec, a checklist or an `evals.json` you paste or upload. It keeps your wording; review the result before launching. Changes are saved as you go. **CLI** Write the experiment as JSON, then create it: ```bash theme={null} tracks experiments create experiment.json ``` `tracks experiments get ` prints an experiment in the same format. Edit it and replace the whole experiment with `tracks experiments update experiment.json`. Repetitions, time limit and budget have no defaults here: set them in `run_settings`. See [Measure your MCP server](https://docs.tracks.alpic.ai/guides/measure-your-mcp-server.md) for a complete file. ## Launching Launching starts an **execution**: one launch of the experiment, with a run for every scenario, variant, agent and repetition. ``` runs = scenarios × variants × agents × repetitions ``` 3 scenarios, 2 variants, 2 agents and 3 repetitions make 36 runs. * **At most 500 runs.** An experiment planning more can't be launched. * **Maximum spend is shown before you launch**: the number of runs times the budget per run. 36 runs at $1 can spend up to $36. If your balance reaches \$0 first, the remaining runs are canceled. * **Executions are numbered** #1, #2, #3… in launch order, and all are kept. * **Each execution keeps the experiment as it was** when launched. Editing the experiment, or a skill in your library, afterwards changes none of its runs. The runs start in the background right after launch. ```bash theme={null} tracks experiments launch tracks experiments launch --wait --min-pass-rate 80 # wait, then fail below 80% tracks executions cancel # start no more runs, cancel the others ``` With `--min-pass-rate`, the command waits for every run to be judged and exits with an error below that pass rate: see [Run from CI](https://docs.tracks.alpic.ai/guides/run-from-ci.md). ## Slots Slots are one row per run of the execution: one scenario, variant, agent and repetition number each. A slot is either: * **not started** yet; * **refused**: its run could not start, for example because your balance was \$0 or an MCP server was disconnected since launch. A refused slot is never retried; * or holding its **run**, with its verdict once judged. ```bash theme={null} tracks executions get # where its runs stand, its pass rate and cost tracks executions slots # every slot, its run, verdict and score ``` Read what each result means in [Reading results](https://docs.tracks.alpic.ai/concepts/reading-results.md). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.