> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tracks.alpic.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Write evaluations

> Write yes-or-no criteria that judge an experiment's runs reliably, and weigh them

Evaluations decide whether a run passed its scenario. Each one is a yes-or-no question, answered for every run of the scenario. Good questions give pass rates you can trust; vague ones give noise.

## How a run is judged

Once a run ends `completed`, a model reads the scenario's prompt and what the run recorded: the agent's messages, its reasoning, every tool call and command it ran, and what they returned. It answers each evaluation from that record alone.

This has three consequences:

* **The judge sees only what the agent did.** It doesn't open the repository afterwards or run anything. To check that tests pass, the agent must have run them.
* **Claims don't count.** "All tests pass", written by the agent, is not proof. The output of the command that ran them is.
* **Unclear means no.** When the record doesn't show a criterion was met, the evaluation fails.

A run that ended any other way, such as `failed` or `time_limit`, is not judged. Neither is one whose record is too long to read. Judging is free: it doesn't count against a run's budget.

## Write good criteria

* **One thing per evaluation.** "Did the agent add the tool and is the build green?" fails for two different reasons; split it in two.
* **Ask a yes-or-no question.** Start with "Did the agent…" or "Does…".
* **Name what to look for**: the file, the command, the tool, the exact value.
* **Check something visible in the record**: a command and its output, a file written, a tool called.
* **Judge the result, not the style.** "Is the code clean?" gets a different answer every time.

| Avoid | Write instead |
| - | - |
| Is the code good? | Did the agent's last `npm run build` finish without errors? |
| Did it work? | Did the agent register a tool named `get-weather` in `src/server.ts`? |
| Did the agent use our docs well? | Did the agent call the `search` tool of the `docs` MCP server before editing a file? |
| Are the tests passing and is the README updated? | Two evaluations: one for the tests, one for the README. |
| Did the agent say it was done? | Did the agent's final message list the files it changed? |

Keep how a run is judged out of the prompt: the prompt says what to do, evaluations say what done looks like.

## Weights and passing score

Each evaluation has a weight, 1 unless you set one. A run's score is the weight of the evaluations it passed out of the total weight, times 100.

With three evaluations weighted 2, 1 and 2, a run that passes the first and last scores 4 out of 5: 80.

A scenario's pass policy turns the score into a verdict:

* **Every evaluation passed**, the default: the run passes only when it passes all of them. Leave out `passing_score`.
* **Passing score**: the run passes when its score reaches it, from 0 to 100. Set `passing_score`.

```json theme={null}
{
  "id": "weather-view",
  "name": "Add a weather view",
  "prompt": "Add a tool named get-weather that takes a city and shows its current temperature in a new view. Then run npm run build and fix any error.",
  "evaluations": [
    {
      "id": "tool-registered",
      "name": "Tool registered",
      "judging_instructions": "Did the agent register a tool named get-weather in src/server.ts?",
      "weight": 2
    },
    {
      "id": "view-added",
      "name": "View added",
      "judging_instructions": "Did the agent add a new view file under src/views that shows a temperature?"
    },
    {
      "id": "build-passes",
      "name": "Build passes",
      "judging_instructions": "Did the last npm run build the agent ran finish without errors?",
      "weight": 2
    }
  ],
  "passing_score": 80
}
```

Give the higher weights to what the task can't do without, and use a passing score when the rest is nice to have.

## Import them from a document

Already have a test plan, a spec or a checklist? In the web app, a model can write scenarios and evaluations from it, in its words:

* **Experiments** → **New experiment** → **From a document** creates an experiment with the scenarios and evaluations the document describes.
* **Import** in an experiment's scenarios adds scenarios to it; **Import** in a scenario's evaluations adds evaluations to that scenario.

Paste or drop any text, up to 40,000 characters. An evaluation gets a weight only where the document says how much it counts. What the model writes is added to the experiment and saved: review it and edit it. The document itself isn't kept, and importing is free. Import is in the web app only.

## Check your evaluations

Launch a small execution first, one repetition, and read a few runs' evaluation results:

```bash theme={null}
tracks executions slots <execution-id> | jq '.data[] | {run_id, verdict, score, evaluation_results}'
```

When an evaluation's result surprises you, read what the agent did with `tracks runs events <run-id>`, or open the run in the web app, and reword the question. See [Reading results](https://docs.tracks.alpic.ai/concepts/reading-results.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.