Tools - Tracks

create_run

Start a coding agent on a prompt in a fresh sandbox. Returns the run right away, while it is still preparing; check on it with get_run. Runs start only while the organization’s balance is above $0.

Parameter	Type	Required	Description
`agent`	`object`	yes	The coding agent and the model it uses.
`agent.harness`	`string`	yes	The coding agent to run: `claude-code` (Claude Code), `claude-desktop` (Cowork, the agent workspace of Anthropic’s Claude desktop app, for documents, research and longer tasks), `codex` (OpenAI’s Codex CLI), `codex-app` (the Codex mode of OpenAI’s ChatGPT desktop app, with its browser, computer-use and document tools) or `cursor` (Cursor’s CLI).
`agent.model`	`string`	yes	Model the agent uses, such as `claude-sonnet-5-5`, or a Cursor model id such as `gpt-5.6-luna-low`.
`prompt`	`string`	yes	What you ask the agent to do first.
`input_files`	`array`	no	Up to 20 uploaded file ids, staged in the sandbox before setup and referenced in the first prompt.
`environment`	`object`	yes
`environment.repository`	`string`	no	URL of a Git repository to clone before the agent starts.
`environment.variables`	`object`	no	Environment variables set in every process of the agent, by name. Their values are never returned. Names match `^[A-Za-z_][A-Za-z0-9_]*$`; at most 100 variables of up to 32768 characters each. `PATH`, `HOME` and names starting with `ANTHROPIC_`, `CLAUDE_`, `OPENAI_`, `CODEX_` or `AGENT_RUNNER_` are reserved for running the agent.
`environment.setup`	`string`	no	Shell script run with bash in the repository directory after checkout and before the first prompt, with the run’s variables set, while the run is `preparing`. A non-zero exit fails the run. It may run again after a restart on our side: make it idempotent. At most 32768 characters.
`environment.teardown`	`string`	no	Shell script run with bash in the repository directory after the run ends and before its sandbox is destroyed, with the run’s variables set. Its exit code doesn’t change how the run ended. It may run again after a restart on our side: make it idempotent. At most 32768 characters.
`environment.skills`	`array`	no
`environment.skills[].name`	`string`	yes	Name of the skill, unique in the run, matching `^[a-z0-9]+(-[a-z0-9]+)*$` and up to 64 characters, such as `release-notes`. `plugin-management` is reserved for running the agent.
`environment.skills[].files`	`array`	yes
`environment.skills[].files[].path`	`string`	yes
`environment.skills[].files[].content`	`string`	yes	Text of the file. A file starting with `#!` is made executable.
`environment.mcp_servers`	`array`	no
`environment.mcp_servers[].name`	`string`	no	Name the agent knows the MCP server by, unique in the run, matching `^[A-Za-z0-9_-]{1,64}$`, such as `linear`. `chatgpt_plugins` is reserved for running the agent.
`environment.mcp_servers[].url`	`string`	no
`environment.mcp_servers[].headers`	`object`	no	Headers we add to every request to the MCP server, typically its API key, such as `{ "Authorization": "Bearer …" }` or `{ "x-api-key": "…" }`. They never enter the sandbox and their values are never returned. Names match `^[A-Za-z0-9-]{1,64}$`; at most 20 headers of up to 8192 characters each, on one line.
`environment.mcp_servers[].connection_id`	`string`	no	An `active` connection: the agent uses its MCP server as the user who consented.
`environment.browser`	`boolean`	no
`labels`	`object`	no	Labels you attach to the run to find it again, as keys and values, such as `{ "ticket": "ENG-42" }`. We never read them. Keys match `^[A-Za-z0-9_.-]{1,64}$`; at most 20 labels with values of up to 256 characters.
`budget_usd`	`number`	no	Maximum the run may spend, in US dollars. Defaults to a standard budget when omitted. The run ends with `budget_exceeded` once it is reached.
`one_shot`	`boolean`	no	Whether the run ends as soon as its first prompt finishes. Defaults to `true`. Set it to `false` to send follow-up prompts while the run is `idle`.
`idle_timeout_seconds`	`integer`	no	How long the run waits for a follow-up prompt before it ends with `idle_timeout`, in seconds, from 120 to 900. Defaults to 900. Only allowed when `one_shot` is `false`.
`time_limit_seconds`	`integer`	no	Longest the run may last, in seconds from when it starts, from 60 to 43200. Defaults to 7200. Once reached, the run ends with `time_limit`, even in the middle of a prompt.

get_run

Show a run and where it stands: preparing, working, idle or ended.

Parameter	Type	Required	Description
`run_id`	`string`	yes	Unique identifier of the run.

send_prompt

Ask the agent of an idle run to continue where its last prompt stopped.

Parameter	Type	Required	Description
`run_id`	`string`	yes	Unique identifier of the run.
`text`	`string`	yes	What you ask the agent to do next.
`model`	`string`	no	Model for this prompt only. Defaults to the run’s model.

list_prompts

List a run’s prompts in order. To get the next page, pass the index of the last prompt received as after.

Parameter	Type	Required	Description
`run_id`	`string`	yes	Unique identifier of the run.
`after`	`integer`	no	Return only items after this position.
`limit`	`integer`	no	Maximum number of items to return, from 1 to 1000. Defaults to 100.

list_frames

List the agent’s output, line by line, in order. To get the next page, pass the seq of the last line received as after.

Parameter	Type	Required	Description
`run_id`	`string`	yes	Unique identifier of the run.
`after`	`integer`	no	Return only items after this position.
`limit`	`integer`	no	Maximum number of items to return, from 1 to 1000. Defaults to 100.
`prompt`	`integer`	no	Position of the prompt in its run, starting at 1.

list_events

List what the agent did, in order: each prompt, then its messages, reasoning, tool calls and subagents, then a stop. To get the next page, pass the seq of the last event received as after.

Parameter	Type	Required	Description
`run_id`	`string`	yes	Unique identifier of the run.
`after`	`integer`	no	Return only items after this position.
`limit`	`integer`	no	Maximum number of items to return, from 1 to 1000. Defaults to 100.
`prompt`	`integer`	no	Position of the prompt in its run, starting at 1.

get_run_usage

Show what a run has consumed so far, per model and in sandbox time, and what it cost.

Parameter	Type	Required	Description
`run_id`	`string`	yes	Unique identifier of the run.

end_run

End a run once its current prompt finishes, then destroy its sandbox.

Parameter	Type	Required	Description
`run_id`	`string`	yes	Unique identifier of the run.

cancel_run

Stop a run right away, even mid-prompt, then destroy its sandbox.

Parameter	Type	Required	Description
`run_id`	`string`	yes	Unique identifier of the run.

connect_mcp_server

Connect to an MCP server that needs the user’s account on the MCP server, such as Linear or Sentry, so runs can use it as the user. Show the returned consent_url to the user and ask them to open it, sign in, approve, then give you the code the page shows. Call confirm_connection with that code. Not needed for an MCP server without an account: it is refused, saying so.

Parameter	Type	Required	Description
`server_url`	`string`	yes

confirm_connection

Finish a connection with the code the user got after approving on the MCP server, within 10 minutes. The connection is then active: list its MCP server’s URL in create_run’s environment.mcp_servers and the run’s agent uses it as the user.

Parameter	Type	Required	Description
`connection_id`	`string`	yes	Unique identifier of the connection.
`confirmation`	`string`	yes	The code the user gave you, such as K7QM-4XPD.

list_connections

List the connections to MCP servers, to find which ones are active.

list_experiments

List the experiments, each with its scenarios, agents, variants and run settings.

get_experiment

Show an experiment: its scenarios and their evaluations, agents, variants and run settings.

Parameter	Type	Required	Description
`experiment_id`	`string`	yes	Unique identifier of the experiment.

create_experiment

Create an experiment: scenarios given to several agents in several variants, each run judged on the scenario’s evaluations. Pick repositories, skills, MCP servers and command-line tools by their ids from get_library. Launch it with launch_experiment.

Parameter	Type	Required	Description
`name`	`string`	yes	Name of the experiment.
`scenarios`	`array`	yes
`scenarios[].id`	`string`	yes	Identifier you choose for the scenario, unique in the experiment, up to 128 characters, such as `checkout`. Results refer to it.
`scenarios[].name`	`string`	yes	Short name of the scenario.
`scenarios[].prompt`	`string`	yes	What the agent is asked to do.
`scenarios[].input_files`	`array`	no	Up to 20 uploaded file ids, given to each of the scenario’s runs. Kept for good once saved. Defaults to none.
`scenarios[].evaluations`	`array`	yes
`scenarios[].evaluations[].id`	`string`	yes	Identifier you choose for the evaluation, unique in its scenario, up to 128 characters, such as `tests-pass`. Results refer to it.
`scenarios[].evaluations[].name`	`string`	yes	Short name of what is checked.
`scenarios[].evaluations[].judging_instructions`	`string`	yes	The yes-or-no question a judge answers from what the agent did, such as `Do the tests pass?`.
`scenarios[].evaluations[].weight`	`integer`	no	How much it counts in the run’s score. Defaults to 1.
`scenarios[].passing_score`	`integer`	no	Score out of 100, weighted by the evaluations’ weights, a run needs to pass. Omit it so a run passes only when it passes every evaluation.
`agents`	`array`	yes	Agents every scenario is given to, each once.
`agents[].harness`	`string`	yes	The coding agent to run: `claude-code` (Claude Code), `claude-desktop` (Cowork, the agent workspace of Anthropic’s Claude desktop app, for documents, research and longer tasks), `codex` (OpenAI’s Codex CLI), `codex-app` (the Codex mode of OpenAI’s ChatGPT desktop app, with its browser, computer-use and document tools) or `cursor` (Cursor’s CLI).
`agents[].model`	`string`	yes	Model the agent uses, such as `claude-sonnet-5-5`.
`repositories`	`array`	no	Ids of repositories from your library, as listed by `GET /library`, each once, tried in turn with every agent tool configuration. Defaults to none: runs start without a repository.
`agent_tools`	`array`	yes
`agent_tools[].id`	`string`	yes	Identifier you choose for the agent tool configuration, unique in the experiment, up to 128 characters, such as `with-docs-mcp`. Results refer to it.
`agent_tools[].name`	`string`	yes	Short name of the agent tools, such as `No agent tools` for the baseline.
`agent_tools[].skills`	`array`	yes	Ids of skills from your library, as listed by `GET /library`.
`agent_tools[].mcp_servers`	`array`	yes	Ids of MCP servers from your library, as listed by `GET /library`.
`agent_tools[].command_line_tools`	`array`	yes	Ids of command-line tools from your library, as listed by `GET /library`.
`agent_tools[].browser`	`boolean`	no
`excluded_variants`	`array`	no	Combinations of repository and agent tools not to run. Every other combination is a variant, run for each scenario and agent. Defaults to none.
`excluded_variants[].repository_id`	`string`	no	Id of one of the experiment’s repositories. Omit it for no repository.
`excluded_variants[].agent_tools_id`	`string`	yes	Id of one of the experiment’s agent tool configurations.
`run_settings`	`object`	yes
`run_settings.repetitions`	`integer`	yes	Identical runs started for each scenario, variant and agent, to compare results beyond chance.
`run_settings.time_limit_seconds`	`number`	yes	Longest each run may last, in seconds, above 0 and at most 43200.
`run_settings.budget_usd`	`number`	yes	Most each run may spend, in US dollars, above 0 and at most 10000.
`run_settings.setup`	`string`	no	Shell script run in each run’s sandbox after checkout and before the prompt. At most 32768 characters.
`run_settings.teardown`	`string`	no	Shell script run in each run’s sandbox after the run ends. At most 32768 characters.
`run_settings.variables`	`array`	no	Environment variables set in every process of each run’s agent, each name once. Defaults to none.
`run_settings.variables[].name`	`string`	yes
`run_settings.variables[].value`	`string`	yes

update_experiment

Replace a whole experiment: send it complete, as get_experiment returns it. Executions already launched keep the experiment as it was.

Parameter	Type	Required	Description
`experiment_id`	`string`	yes	Unique identifier of the experiment.
`name`	`string`	yes	Name of the experiment.
`scenarios`	`array`	yes
`scenarios[].id`	`string`	yes	Identifier you choose for the scenario, unique in the experiment, up to 128 characters, such as `checkout`. Results refer to it.
`scenarios[].name`	`string`	yes	Short name of the scenario.
`scenarios[].prompt`	`string`	yes	What the agent is asked to do.
`scenarios[].input_files`	`array`	no	Up to 20 uploaded file ids, given to each of the scenario’s runs. Kept for good once saved. Defaults to none.
`scenarios[].evaluations`	`array`	yes
`scenarios[].evaluations[].id`	`string`	yes	Identifier you choose for the evaluation, unique in its scenario, up to 128 characters, such as `tests-pass`. Results refer to it.
`scenarios[].evaluations[].name`	`string`	yes	Short name of what is checked.
`scenarios[].evaluations[].judging_instructions`	`string`	yes	The yes-or-no question a judge answers from what the agent did, such as `Do the tests pass?`.
`scenarios[].evaluations[].weight`	`integer`	no	How much it counts in the run’s score. Defaults to 1.
`scenarios[].passing_score`	`integer`	no	Score out of 100, weighted by the evaluations’ weights, a run needs to pass. Omit it so a run passes only when it passes every evaluation.
`agents`	`array`	yes	Agents every scenario is given to, each once.
`agents[].harness`	`string`	yes	The coding agent to run: `claude-code` (Claude Code), `claude-desktop` (Cowork, the agent workspace of Anthropic’s Claude desktop app, for documents, research and longer tasks), `codex` (OpenAI’s Codex CLI), `codex-app` (the Codex mode of OpenAI’s ChatGPT desktop app, with its browser, computer-use and document tools) or `cursor` (Cursor’s CLI).
`agents[].model`	`string`	yes	Model the agent uses, such as `claude-sonnet-5-5`.
`repositories`	`array`	no	Ids of repositories from your library, as listed by `GET /library`, each once, tried in turn with every agent tool configuration. Defaults to none: runs start without a repository.
`agent_tools`	`array`	yes
`agent_tools[].id`	`string`	yes	Identifier you choose for the agent tool configuration, unique in the experiment, up to 128 characters, such as `with-docs-mcp`. Results refer to it.
`agent_tools[].name`	`string`	yes	Short name of the agent tools, such as `No agent tools` for the baseline.
`agent_tools[].skills`	`array`	yes	Ids of skills from your library, as listed by `GET /library`.
`agent_tools[].mcp_servers`	`array`	yes	Ids of MCP servers from your library, as listed by `GET /library`.
`agent_tools[].command_line_tools`	`array`	yes	Ids of command-line tools from your library, as listed by `GET /library`.
`agent_tools[].browser`	`boolean`	no
`excluded_variants`	`array`	no	Combinations of repository and agent tools not to run. Every other combination is a variant, run for each scenario and agent. Defaults to none.
`excluded_variants[].repository_id`	`string`	no	Id of one of the experiment’s repositories. Omit it for no repository.
`excluded_variants[].agent_tools_id`	`string`	yes	Id of one of the experiment’s agent tool configurations.
`run_settings`	`object`	yes
`run_settings.repetitions`	`integer`	yes	Identical runs started for each scenario, variant and agent, to compare results beyond chance.
`run_settings.time_limit_seconds`	`number`	yes	Longest each run may last, in seconds, above 0 and at most 43200.
`run_settings.budget_usd`	`number`	yes	Most each run may spend, in US dollars, above 0 and at most 10000.
`run_settings.setup`	`string`	no	Shell script run in each run’s sandbox after checkout and before the prompt. At most 32768 characters.
`run_settings.teardown`	`string`	no	Shell script run in each run’s sandbox after the run ends. At most 32768 characters.
`run_settings.variables`	`array`	no	Environment variables set in every process of each run’s agent, each name once. Defaults to none.
`run_settings.variables[].name`	`string`	yes
`run_settings.variables[].value`	`string`	yes

delete_experiment

Delete an experiment. Its executions, their runs and results stay.

Parameter	Type	Required	Description
`experiment_id`	`string`	yes	Unique identifier of the experiment.

launch_experiment

Launch an experiment as it is now: a run for every scenario, variant, agent and repetition, started in the background and judged once ended. Returns the execution; check on it with get_execution until it is no longer pending, then read each run’s verdict with list_slots. If refused as incomplete, fix what the error lists with update_experiment.

Parameter	Type	Required	Description
`experiment_id`	`string`	yes	Unique identifier of the experiment.

list_executions

List an experiment’s executions, the latest launched first, with their status and pass rate.

Parameter	Type	Required	Description
`experiment_id`	`string`	yes	Unique identifier of the experiment.

get_execution

Show an execution: whether runs are still pending, and its pass rate.

Parameter	Type	Required	Description
`execution_id`	`string`	yes	Unique identifier of the execution.

list_slots

List every run an execution plans, its scenario, variant and agent, its run_id once started, its outcome and its verdict once judged. Read what a run’s agent did with list_events.

Parameter	Type	Required	Description
`execution_id`	`string`	yes	Unique identifier of the execution.

cancel_execution

Cancel an execution: start none of its runs left and cancel those started.

Parameter	Type	Required	Description
`execution_id`	`string`	yes	Unique identifier of the execution.

get_library

List the repositories, skills, MCP servers and command-line tools an experiment can pick, with the ids to pick them by.

​create_run

​get_run

​send_prompt

​list_prompts

​list_frames

​list_events

​get_run_usage

​end_run

​cancel_run

​connect_mcp_server

​confirm_connection

​list_connections

​list_experiments

​get_experiment

​create_experiment

​update_experiment

​delete_experiment

​launch_experiment

​list_executions

​get_execution

​list_slots

​cancel_execution

​get_library