Measure your MCP server or skill

Does your MCP server, or your skill, make agents better at a task? Give the same task to agents with it and without it, several times each, and compare how often they pass and whether they used it at all. This guide builds one experiment: one scenario on the demo repository, two agents, a baseline variant with no agent tools and a variant with your MCP server, three repetitions each. That makes 12 runs.

1. Add your tools to the library

Experiments pick what they give agents from your organization’s library. In the web app, open Library:

Repositories → Add repository: URL https://github.com/alpic-ai/apps-sdk-template, ref main. Repositories in the library must be public.
MCP servers → Add MCP server: its URL, the name the agent knows it by, and its authentication:
- None when it asks for nothing.
- Bearer token when it takes a key. Tracks never shows the token again once saved.
- OAuth when it needs an account: click Connect and sign in once, and runs act as that account.
Skills → Add skill, to measure a skill instead: from a public GitHub URL, or uploaded or pasted.

Then list their ids:

tracks library

{
  "repositories": [
    { "id": "...", "name": "apps-sdk-template", "url": "https://github.com/alpic-ai/apps-sdk-template", "ref": "main" }
  ],
  "skills": [],
  "mcp_servers": [{ "id": "...", "name": "docs", "url": "https://docs.example.com/mcp", "connected": true }],
  "command_line_tools": []
}

An MCP server with connected: false needs Connect in the web app first: an experiment using it cannot launch.

2. Write the experiment

Save this as experiment.json, with your repository and MCP server ids in place of <repository-id> and <mcp-server-id>:

experiment.json

{
  "name": "Weather view with and without our docs",
  "scenarios": [
    {
      "id": "weather-view",
      "name": "Add a weather view",
      "prompt": "Add a tool named get-weather that takes a city and shows its current temperature in a new view. Then run npm run build and fix any error.",
      "evaluations": [
        {
          "id": "tool-registered",
          "name": "Tool registered",
          "judging_instructions": "Did the agent register a tool named get-weather in src/server.ts?",
          "weight": 2
        },
        {
          "id": "view-added",
          "name": "View added",
          "judging_instructions": "Did the agent add a new view file under src/views that shows a temperature?"
        },
        {
          "id": "build-passes",
          "name": "Build passes",
          "judging_instructions": "Did the last npm run build the agent ran finish without errors?",
          "weight": 2
        }
      ],
      "passing_score": 80
    }
  ],
  "agents": [
    { "harness": "claude-code", "model": "claude-sonnet-5-5" },
    { "harness": "codex", "model": "gpt-5.5" }
  ],
  "repositories": ["<repository-id>"],
  "agent_tools": [
    {
      "id": "baseline",
      "name": "No agent tools",
      "skills": [],
      "mcp_servers": [],
      "command_line_tools": []
    },
    {
      "id": "with-docs-mcp",
      "name": "With our docs MCP server",
      "skills": [],
      "mcp_servers": ["<mcp-server-id>"],
      "command_line_tools": []
    }
  ],
  "run_settings": {
    "repetitions": 3,
    "time_limit_seconds": 1800,
    "budget_usd": 5,
    "setup": "npm install"
  }
}

What each part does:

scenarios: the task. The prompt doesn’t mention your MCP server, so you also learn whether agents find it on their own. Each evaluation is a yes-or-no question; a run passes with a weighted score of 80 or more out of 100. See Write evaluations.
agents: a harness and a model each. Every agent gets every scenario in every variant.
repositories and agent_tools: every combination is a variant, here two: the repository with no agent tools, the baseline, and the repository with your MCP server. To measure a skill, put its id in skills instead.
run_settings: shared by every run. repetitions runs each combination 3 times, so chance evens out. budget_usd and time_limit_seconds cap each run.

The ids you choose for scenarios, evaluations and agent tools are how results refer to them.

3. Launch it

EXPERIMENT=$(tracks experiments create experiment.json | jq -r .id)
tracks experiments launch "$EXPERIMENT" --wait

Launching starts an execution: here 1 scenario × 2 variants × 2 agents × 3 repetitions = 12 runs, each judged once it ends. At most it costs 12 runs × $5 = $60; judging is free. With --wait, the command prints progress every 10 seconds until no run is pending, then prints the execution, here trimmed. Its pass_rate covers all runs:

{
  "id": "...",
  "pending": false,
  "slots": { "total": 12, "running": 0, "judging": 0, "judged": 12, "passed": 9 },
  "pass_rate": 75,
  "cost_usd": 9.84
}

If launching fails because something is missing, the error lists what to fix.

4. Compare the variants

The execution’s pass rate mixes both variants. Split it per variant and agent from its slots, one per run:

tracks executions slots <execution-id> | jq '.data
  | group_by([.agent_tools_id, .agent.harness])
  | map({
      agent_tools: .[0].agent_tools_id,
      agent: .[0].agent.harness,
      passed: map(select(.verdict == "passed")) | length,
      judged: map(select(.outcome == "judged")) | length
    })'

If with-docs-mcp passes more often than baseline, your MCP server helps. Each slot also has its score, the result of each evaluation and its run_id: read what an agent did with tracks runs events <run-id>. A run that didn’t end completed is not judged, and counts neither way in the pass rate: its slot’s not_evaluated_reason says why.

Did agents use it?

A better pass rate means little if agents never called your server. Open the execution in the web app: under Results, Agent tool use shows, for each MCP server, skill or command-line tool, how many of the runs given it used it, and how many of their calls failed. Each run’s Agent Tools tab lists its calls one by one. See Reading results for every number on that page.

The same in the web app

Web app
CLI

Create the experiment

Open Experiments, then New experiment → From scratch.

Scenarios

Write the prompt, add the evaluations and their weights, and pick a pass policy: every evaluation passed, or a passing score.

Agents

Add Claude Code with claude-sonnet-5-5 and Codex with gpt-5.5.

Variants

Pick the repository. Click Add agent tools twice: leave the first empty as the baseline, pick your MCP server in the second.

Run settings

Set 3 repetitions, a time limit, a $5 budget and npm install as the setup script.

Launch

Click Launch. The results page fills in as runs end and are judged.

Steps 2 to 4 above. tracks experiments get <id> prints an experiment made in the web app in the same JSON shape, to keep in your repository and update with tracks experiments update <id> experiment.json.

To block a change when the pass rate drops, launch the experiment from CI: see Run from CI.

Get started

Concepts

Guides

Measure your MCP server or skill

1. Add your tools to the library

2. Write the experiment

3. Launch it

4. Compare the variants

Did agents use it?

The same in the web app

​1. Add your tools to the library

​2. Write the experiment

​3. Launch it

​4. Compare the variants

​Did agents use it?

​The same in the web app

1. Add your tools to the library

2. Write the experiment

3. Launch it

4. Compare the variants

Did agents use it?

The same in the web app