Reading results

Once an execution’s runs end, each one is judged on its own: one yes-or-no answer per evaluation of its scenario, combined into a score and a verdict. The execution then adds them up into a pass rate.

Verdict and score

Each evaluation of a run is passed or failed. From those:

The score is the weighted share of evaluations passed, out of 100: the weights of the passed evaluations divided by the weights of all of them.
The verdict is passed or failed, by the scenario’s pass rule: every evaluation passed (the default), or a score at least equal to its passing score.

For example, a scenario with a passing score of 80 and three evaluations:

Evaluation	Weight	Result
The app builds	2	Passed
The toggle switches themes	2	Passed
The choice survives reload	1	Failed

The score is (2 + 2) / (2 + 2 + 1) = 80, so the run passed. Under “every evaluation passed”, the same run failed. The score is never rounded before the pass rule is applied. The web app shows it only for scenarios with a passing score.

Pass rate

The pass rate is the runs that passed out of the runs judged, as a whole percent.

Only judged runs count. A run that was not evaluated says nothing about the agent’s work, so it counts neither as passed nor as failed.

Say an execution has 10 runs: 6 passed, 2 failed, and 2 reached their time limit. Its pass rate is 6 out of 8, 75%. The 2 others are listed apart: if many runs end the same way, fix the cause, here a time limit too short, and launch again.

Not evaluated

A run is not evaluated when it can’t be judged. Only runs that ended completed are judged.

Reason in the API	Shown in the web app as	What to do
`never_started`	The slot’s state, such as Refused or Canceled	See refused and not started.
`no_evaluations`	No evaluations	Add evaluations to the scenario, then launch again.
`run_not_completed`	How the run ended, such as Time limit reached or Failed	Open the run: raise the time limit or budget, or fix the setup.
`agent_failed`	Agent error	The agent stopped with an error, for example when its model provider couldn’t be reached.
`evidence_too_long`	Too much to judge	The run recorded more than the judge can read. Ask for less, or split the scenario.
`too_many_attempts`	Judging failed	Tracks retried judging on its own and gave up after 20 failed attempts.

Refused and not started

A slot whose run hasn’t started yet is not started. If its execution is canceled first, it never starts. A slot is refused when its run couldn’t start when its turn came. It is never retried. The web app says why:

your organization’s balance was $0, or your organization is blocked;
the agent’s model is no longer offered;
one of its MCP servers was disconnected, lost its bearer token, or its connection can’t be used.

Fix the cause, then launch a new execution.

Why repetitions matter

The same agent, on the same prompt, does different things from one run to the next. With one run per combination, a single lucky or unlucky run decides the result. Set Repetitions to 3 or more when you compare agents or variants. Each one adds a run per scenario, variant and agent, so check the maximum spend before launching. Treat a gap of one or two runs between groups as noise, not as a winner.

In the web app

Open an experiment, then one of its executions.

Read the summary

The top shows the Pass rate with how many judged runs passed and why others were not judged, the Runs ended so far, and the Total spend against the maximum.Below it, bars compare pass rates By agent tools, By agent, By repository and By scenario, for each of them that has more than one value.

Group the results

In Results, pick what goes in Rows and Columns: scenario, agents, repository or agent tools. Each cell shows a pass rate and how many judged runs passed. Narrow it down with Filters.

Open the runs

Click a cell to list its runs, filtered by Passed, Failed, Pending or Not evaluated. Open a run to read its Conversation, the result of each evaluation in Evals, and the tools it used in Agent Tools.

Pending means the run hasn’t ended yet, or is waiting for its evaluation results.

Agent tool use

Agent tool use answers the question behind most experiments: did the agent actually use your MCP server, skill or command-line tool? A low pass rate with a tool nobody called is a discovery problem, not a quality problem. For each skill, MCP server, command-line tool and browser a run was given, Tracks counts how many of the agent’s tool calls touched it, and how many of those failed:

Item	Counted as used when the agent
Skill	loads it, or reads or runs one of its files
MCP server	calls one of its tools
Browser	calls one of its tools
Command-line tool	runs a shell command with the tool’s name

It is recorded for every run once it ends, however it ended, so even a run that was not evaluated has it. It is free. Where to read it:

Execution page: under Results, an Agent tool use section shows, for each item and each column, how many of the runs given it used it, such as “4 of 6 runs used it”, and how many calls failed. A dash means that group was never given the item.
Run page: the Agent Tools tab lists each tool the run was given, with its number of calls and failed calls, or Not used.

Agent tool use is shown only in the web app. The API, CLI and MCP server don’t return it yet.

With the CLI or API

tracks executions get <execution-id>

{
  "id": "…",
  "experiment_id": "…",
  "number": 3,
  "pending": false,
  "slots": { "total": 12, "running": 0, "judging": 0, "judged": 10, "passed": 7 },
  "pass_rate": 70,
  "cost_usd": 4.82
}

pending is false once every run has ended and been judged: the pass rate is then final. pass_rate is absent until a run has a verdict.

tracks executions slots <execution-id>

Each slot gives its scenario_id, repository_id, agent_tools_id, agent and repetition, then:

Field	Meaning
`outcome`	`running`, `judging`, `judged` or `not_evaluated`.
`verdict`	`passed` or `failed`, once judged.
`score`	Out of 100, once every evaluation result is in.
`evaluation_results`	Each evaluation judged so far, and whether the run passed it.
`not_evaluated_reason`	Why it won’t be judged. See Not evaluated.
`refusal`	Why its run couldn’t start.
`run_id`	The run, once started. Read what the agent did with `tracks runs events <run-id>`.

Get started

Concepts

Guides

Reading results

Verdict and score

Pass rate

Not evaluated

Refused and not started

Why repetitions matter

In the web app

Agent tool use

With the CLI or API

​Verdict and score

​Pass rate

​Not evaluated

​Refused and not started

​Why repetitions matter

​In the web app

​Agent tool use

​With the CLI or API

Verdict and score

Pass rate

Not evaluated

Refused and not started

Why repetitions matter

In the web app

Agent tool use

With the CLI or API