Write evaluations

Evaluations decide whether a run passed its scenario. Each one is a yes-or-no question, answered for every run of the scenario. Good questions give pass rates you can trust; vague ones give noise.

How a run is judged

Once a run ends completed, a model reads the scenario’s prompt and what the run recorded: the agent’s messages, its reasoning, every tool call and command it ran, and what they returned. It answers each evaluation from that record alone. This has three consequences:

The judge sees only what the agent did. It doesn’t open the repository afterwards or run anything. To check that tests pass, the agent must have run them.
Claims don’t count. “All tests pass”, written by the agent, is not proof. The output of the command that ran them is.
Unclear means no. When the record doesn’t show a criterion was met, the evaluation fails.

A run that ended any other way, such as failed or time_limit, is not judged. Neither is one whose record is too long to read. Judging is free: it doesn’t count against a run’s budget.

Write good criteria

One thing per evaluation. “Did the agent add the tool and is the build green?” fails for two different reasons; split it in two.
Ask a yes-or-no question. Start with “Did the agent…” or “Does…”.
Name what to look for: the file, the command, the tool, the exact value.
Check something visible in the record: a command and its output, a file written, a tool called.
Judge the result, not the style. “Is the code clean?” gets a different answer every time.

Avoid	Write instead
Is the code good?	Did the agent’s last `npm run build` finish without errors?
Did it work?	Did the agent register a tool named `get-weather` in `src/server.ts`?
Did the agent use our docs well?	Did the agent call the `search` tool of the `docs` MCP server before editing a file?
Are the tests passing and is the README updated?	Two evaluations: one for the tests, one for the README.
Did the agent say it was done?	Did the agent’s final message list the files it changed?

Keep how a run is judged out of the prompt: the prompt says what to do, evaluations say what done looks like.

Weights and passing score

Each evaluation has a weight, 1 unless you set one. A run’s score is the weight of the evaluations it passed out of the total weight, times 100. With three evaluations weighted 2, 1 and 2, a run that passes the first and last scores 4 out of 5: 80. A scenario’s pass policy turns the score into a verdict:

Every evaluation passed, the default: the run passes only when it passes all of them. Leave out passing_score.
Passing score: the run passes when its score reaches it, from 0 to 100. Set passing_score.

{
  "id": "weather-view",
  "name": "Add a weather view",
  "prompt": "Add a tool named get-weather that takes a city and shows its current temperature in a new view. Then run npm run build and fix any error.",
  "evaluations": [
    {
      "id": "tool-registered",
      "name": "Tool registered",
      "judging_instructions": "Did the agent register a tool named get-weather in src/server.ts?",
      "weight": 2
    },
    {
      "id": "view-added",
      "name": "View added",
      "judging_instructions": "Did the agent add a new view file under src/views that shows a temperature?"
    },
    {
      "id": "build-passes",
      "name": "Build passes",
      "judging_instructions": "Did the last npm run build the agent ran finish without errors?",
      "weight": 2
    }
  ],
  "passing_score": 80
}

Give the higher weights to what the task can’t do without, and use a passing score when the rest is nice to have.

Import them from a document

Already have a test plan, a spec or a checklist? In the web app, a model can write scenarios and evaluations from it, in its words:

Experiments → New experiment → From a document creates an experiment with the scenarios and evaluations the document describes.
Import in an experiment’s scenarios adds scenarios to it; Import in a scenario’s evaluations adds evaluations to that scenario.

Paste or drop any text, up to 40,000 characters. An evaluation gets a weight only where the document says how much it counts. What the model writes is added to the experiment and saved: review it and edit it. The document itself isn’t kept, and importing is free. Import is in the web app only.

Check your evaluations

Launch a small execution first, one repetition, and read a few runs’ evaluation results:

tracks executions slots <execution-id> | jq '.data[] | {run_id, verdict, score, evaluation_results}'

When an evaluation’s result surprises you, read what the agent did with tracks runs events <run-id>, or open the run in the web app, and reword the question. See Reading results.

Get started

Concepts

Guides

Write evaluations

How a run is judged

Write good criteria

Weights and passing score

Import them from a document

Check your evaluations

​How a run is judged

​Write good criteria

​Weights and passing score

​Import them from a document

​Check your evaluations

How a run is judged

Write good criteria

Weights and passing score

Import them from a document

Check your evaluations