> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tracks.alpic.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Measure your MCP server or skill

> Find out whether your MCP server or skill helps coding agents, with an experiment run with and without it

Does your MCP server, or your skill, make agents better at a task? Give the same task to agents with it and without it, several times each, and compare how often they pass and whether they used it at all.

This guide builds one experiment: one scenario on the [demo repository](https://github.com/alpic-ai/apps-sdk-template), two agents, a baseline variant with no agent tools and a variant with your MCP server, three repetitions each. That makes 12 runs.

## 1. Add your tools to the library

Experiments pick what they give agents from your organization's library. In the [web app](https://tracks.alpic.ai), open **Library**:

* **Repositories** → **Add repository**: URL `https://github.com/alpic-ai/apps-sdk-template`, ref `main`. Repositories in the library must be public.
* **MCP servers** → **Add MCP server**: its URL, the name the agent knows it by, and its authentication:
  * **None** when it asks for nothing.
  * **Bearer token** when it takes a key. Tracks never shows the token again once saved.
  * **OAuth** when it needs an account: click **Connect** and sign in once, and runs act as that account.
* **Skills** → **Add skill**, to measure a skill instead: from a public GitHub URL, or uploaded or pasted.

Then list their ids:

```bash theme={null}
tracks library
```

```json theme={null}
{
  "repositories": [
    { "id": "...", "name": "apps-sdk-template", "url": "https://github.com/alpic-ai/apps-sdk-template", "ref": "main" }
  ],
  "skills": [],
  "mcp_servers": [{ "id": "...", "name": "docs", "url": "https://docs.example.com/mcp", "connected": true }],
  "command_line_tools": []
}
```

An MCP server with `connected: false` needs **Connect** in the web app first: an experiment using it cannot launch.

## 2. Write the experiment

Save this as `experiment.json`, with your repository and MCP server ids in place of `<repository-id>` and `<mcp-server-id>`:

```json experiment.json theme={null}
{
  "name": "Weather view with and without our docs",
  "scenarios": [
    {
      "id": "weather-view",
      "name": "Add a weather view",
      "prompt": "Add a tool named get-weather that takes a city and shows its current temperature in a new view. Then run npm run build and fix any error.",
      "evaluations": [
        {
          "id": "tool-registered",
          "name": "Tool registered",
          "judging_instructions": "Did the agent register a tool named get-weather in src/server.ts?",
          "weight": 2
        },
        {
          "id": "view-added",
          "name": "View added",
          "judging_instructions": "Did the agent add a new view file under src/views that shows a temperature?"
        },
        {
          "id": "build-passes",
          "name": "Build passes",
          "judging_instructions": "Did the last npm run build the agent ran finish without errors?",
          "weight": 2
        }
      ],
      "passing_score": 80
    }
  ],
  "agents": [
    { "harness": "claude-code", "model": "claude-sonnet-5-5" },
    { "harness": "codex", "model": "gpt-5.5" }
  ],
  "repositories": ["<repository-id>"],
  "agent_tools": [
    {
      "id": "baseline",
      "name": "No agent tools",
      "skills": [],
      "mcp_servers": [],
      "command_line_tools": []
    },
    {
      "id": "with-docs-mcp",
      "name": "With our docs MCP server",
      "skills": [],
      "mcp_servers": ["<mcp-server-id>"],
      "command_line_tools": []
    }
  ],
  "run_settings": {
    "repetitions": 3,
    "time_limit_seconds": 1800,
    "budget_usd": 5,
    "setup": "npm install"
  }
}
```

What each part does:

* **`scenarios`**: the task. The prompt doesn't mention your MCP server, so you also learn whether agents find it on their own. Each evaluation is a yes-or-no question; a run passes with a weighted score of 80 or more out of 100. See [Write evaluations](https://docs.tracks.alpic.ai/guides/write-evaluations.md).
* **`agents`**: a harness and a model each. Every agent gets every scenario in every variant.
* **`repositories`** and **`agent_tools`**: every combination is a variant, here two: the repository with no agent tools, the baseline, and the repository with your MCP server. To measure a skill, put its id in `skills` instead.
* **`run_settings`**: shared by every run. `repetitions` runs each combination 3 times, so chance evens out. `budget_usd` and `time_limit_seconds` cap each run.

The ids you choose for scenarios, evaluations and agent tools are how results refer to them.

## 3. Launch it

```bash theme={null}
EXPERIMENT=$(tracks experiments create experiment.json | jq -r .id)
tracks experiments launch "$EXPERIMENT" --wait
```

Launching starts an execution: here 1 scenario × 2 variants × 2 agents × 3 repetitions = 12 runs, each judged once it ends. At most it costs 12 runs × $5 = $60; judging is free.

With `--wait`, the command prints progress every 10 seconds until no run is pending, then prints the execution, here trimmed. Its `pass_rate` covers all runs:

```json theme={null}
{
  "id": "...",
  "pending": false,
  "slots": { "total": 12, "running": 0, "judging": 0, "judged": 12, "passed": 9 },
  "pass_rate": 75,
  "cost_usd": 9.84
}
```

If launching fails because something is missing, the error lists what to fix.

## 4. Compare the variants

The execution's pass rate mixes both variants. Split it per variant and agent from its slots, one per run:

```bash theme={null}
tracks executions slots <execution-id> | jq '.data
  | group_by([.agent_tools_id, .agent.harness])
  | map({
      agent_tools: .[0].agent_tools_id,
      agent: .[0].agent.harness,
      passed: map(select(.verdict == "passed")) | length,
      judged: map(select(.outcome == "judged")) | length
    })'
```

If `with-docs-mcp` passes more often than `baseline`, your MCP server helps. Each slot also has its `score`, the result of each evaluation and its `run_id`: read what an agent did with `tracks runs events <run-id>`.

A run that didn't end `completed` is not judged, and counts neither way in the pass rate: its slot's `not_evaluated_reason` says why.

### Did agents use it?

A better pass rate means little if agents never called your server. Open the execution in the web app: under **Results**, **Agent tool use** shows, for each MCP server, skill or command-line tool, how many of the runs given it used it, and how many of their calls failed. Each run's **Agent Tools** tab lists its calls one by one.

See [Reading results](https://docs.tracks.alpic.ai/concepts/reading-results.md) for every number on that page.

## The same in the web app

**Web app**

**1. Create the experiment**

Open **Experiments**, then **New experiment** → **From scratch**.

**2. Scenarios**

Write the prompt, add the evaluations and their weights, and pick a pass policy: every evaluation passed, or a passing score.

**3. Agents**

Add Claude Code with `claude-sonnet-5-5` and Codex with `gpt-5.5`.

**4. Variants**

Pick the repository. Click **Add agent tools** twice: leave the first empty as the baseline, pick your MCP server in the second.

**5. Run settings**

Set 3 repetitions, a time limit, a \$5 budget and `npm install` as the setup script.

**6. Launch**

Click **Launch**. The results page fills in as runs end and are judged.

**CLI**

Steps 2 to 4 above. `tracks experiments get <id>` prints an experiment made in the web app in the same JSON shape, to keep in your repository and update with `tracks experiments update <id> experiment.json`.

To block a change when the pass rate drops, launch the experiment from CI: see [Run from CI](https://docs.tracks.alpic.ai/guides/run-from-ci.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.