Docs | Searches API

Searches are read and changed through these endpoints, under /api/teams/{teamId}. They answer JSON, and errors are application/problem+json as on the resource API. {lineId} and {stationId} are as in the Replays API.

Endpoints

Method and path	Returns
`GET /lines/{lineId}/stations/{stationId}/searches`	The station's searches, newest first, each a search.
`POST /lines/{lineId}/stations/{stationId}/searches/estimate`	`200` and an estimate. Nothing is written.
`POST /lines/{lineId}/stations/{stationId}/searches`	`201` and `{ "id", "seedReplayId", "objective", "warnings" }`: starts a search. See Start a search.
`GET /searches`	The team's searches, newest first, a page at a time.
`GET /searches/{searchId}`	One search with its split, iterations, candidates and frontier. See Search detail.
`POST /searches/{searchId}/stop`	`204`: stops the search. Its agent's run is withdrawn or stopped, its replays stop, and nothing more starts.
`POST /searches/{searchId}/candidates/{candidateId}/pick`	`201` and a config pull request when picked, or picked again after a refusal; `200` and the one it has when picked already.
`GET /config-pull-requests/{pullRequestId}`	A config pull request.
`GET /search-runs/{runId}`	A search's agent run: its search and iteration, the agent it ran, its steps, and what it spent.

GET /searches takes:

Parameter	Description
`limit`	At most this many. Default 50, at most 200.
`before`	Only searches started before this one, by its id. A search the team does not hold answers an empty page.

A search's agent proposes candidates at POST /api/runs/{runId}/search-candidates, which only the runner holding the run may call, with its runner token. The agent reaches it through spotlight candidate. From a reflection's run it is refused with 409.

A reflection's agent records a cause at POST /api/runs/{runId}/search-causes, on the same terms, through spotlight cause. The body is { "taskId", "cause", "evidence", "suggestion" }, suggestion optional; it answers 201 and { "taskId", "cause", "replayRunId" }.

Status	Cause
`404`	The run is no search run the runner holds, the search no longer holds it, or the task is none the reflection looks into.
`409`	The run's iteration proposes candidates.
`422`	A `cause` other than `config`, `task`, `judge`, `check` or `infra`; no `evidence`, or more than 4,000 characters of it; a `suggestion` over 4,000 characters.

Start a search

{
  "setId": "<replay set id>",
  "budget": {
    "iterations": 3,
    "perIteration": 2,
    "ceiling": { "amount": 20, "currency": "USD" },
    "concurrency": 4,
    "repeats": 1
  },
  "agent": { "harness": "claude-code", "model": "claude-opus-4-5", "effort": "high" },
  "objective": "success"
}

Field	Type	Default	Description
`setId`	string	required	A replay set at the station, covering runs of at least 5 different tasks.
`budget.iterations`	number	required	How many times the search runs its agent, 1 to 10.
`budget.perIteration`	number	required	How many candidates each agent run may propose, 1 to 5.
`budget.ceiling`	object	required	`{ amount, currency }`: the most the whole search may spend, above 0 and at most 1,000, in a three-letter currency.
`budget.concurrency`	number	`4`	How many of its replay runs run at once, 1 to 10, apart from the team's own limit.
`budget.repeats`	number	`1`	How many times each of its replays replays each run, 1 to 5: the live config's, every candidate's and the held-out replays. The runs and the share of the ceiling each needs grow with it.
`agent.harness`	string	required	The harness the agent runs on. One a runner can start.
`agent.model`	string or null	none	The agent's model, 1 to 60 characters. Without it, the harness's default.
`agent.effort`	string or null	none	`low`, `medium`, `high`, `xhigh` or `max`. Without it, the harness's default.
`objective`	string or null	from the set's health	`success` ranks candidates by how often they succeed; `cost` ranks those that succeed as often as the live config, within noise, by what a run costs. Left out, `cost` when the live config succeeds on more than 95% of the set, else `success`. See Aims.

Starting takes a snapshot of the station's live config, draws the held-out split, and starts a replay of the live config on every run of the set: seedReplayId. That replay is the set's new baseline, with a ceiling from the search's budget. objective is the aim it keeps, as asked or as the set's health suggests. warnings are as for an estimate; the search starts all the same.

Estimate a search

The body is setId, budget and objective, as above. The answer:

Field	Type	Description
`searchedTasks`	number	Tasks its agent sees.
`heldOutTasks`	number	Tasks it holds out.
`runs`	object	The most runs it may start: `seed` (the live config, one per run of the set and copy), `searchReplays` (every candidate it may propose, on the searched runs), `heldOutReplays` (the best candidates, on the held-out runs), `agent` (one per iteration) and `total`. Every replay count includes the copies `budget.repeats` asks for.
`ceiling`	object	`{ amount, currency }`.
`perReplayAllotments`	object	The ceilings each replay would get: `seed`, `searchReplay`, `heldOutReplay`, and `agentRun`, what each agent run is expected to spend.
`estimatedDuration`	number or null	Roughly how long it takes at its concurrency, in milliseconds, from the average time of the set's runs. Null when none was timed.
`minGain`	number	The smallest gain over the live config the line counts as real: `replay_check.min_gain`, or `0.05` when the line has none.
`objective`	string	`success` or `cost`: what the search would aim for, as asked or as the set's health suggests.
`warnings`	array	What makes the search's results hard to trust. Empty when nothing does. See Warnings.

Warnings

Each warning is an object whose kind says which it is and which other fields it has.

`kind`	When	Other fields
`noise-unmeasured`	No finished replay of the live config, at its current config fingerprint, replayed the set's runs more than once. See Noise.	None.
`noisy`	The set's success moves between copies so much that `budget.repeats` copies cannot bring it under `minGain`.	`spread`: how far the set's success moved between copies. `minGain`: the line's, as in the estimate. `neededRepeats`: the copies it would take, the spread divided by `minGain`, squared and rounded up, however many; more than 5 means no search of the set can tell such a gain from noise. `suggestedRepeats`: `neededRepeats`, at most 5.
`saturated`	The live config succeeds on more than 95% of the set. See Room to improve.	`rate`: the live config's success on the set.
`judge-unstable`	The set's newest check found the judge passing or failing the same change alike less than 80% of the time.	`agreement`: that share. `compared`: how many changes it judged twice.
`flaky-checks`	The set's newest check found the station's check passing once and failing once on some changes.	`flaky`: how many changes it flipped on. `rechecked`: how many changes it checked twice.
`no-headroom`	The newest replay with the answer was compared with the live config and did not do clearly better, so the set leaves little room for a better config to show.	`gain`, `low`, `high`: the gain the answer brought over the live config and its 95% interval. `answered`: how many runs were handed an answer.

The estimate assumes the choice of held-out tasks whose runs cost most.

Search

Each item of a list, and the first fields of a search's detail:

Field	Type	Description
`id`, `lineId`, `stationId`, `setId`	string
`state`	string	`seeding`, `searching`, `holding-out`, `finished` or `stopped`. See Search states.
`ending`	string or null	Why it stopped searching. See Search endings.
`agent`	object	`{ harness, model, effort }`.
`budget`	object	`{ iterations, perIteration, ceiling, concurrency, repeats }`.
`objective`	string	`success` or `cost`: what it aims for, which orders its frontier and its best. See Aims.
`spent`	object	`{ amount, currency }`: its replays and its agent's runs so far, in the ceiling's currency.
`iterationsRun`	number	How many times its agent ran, or is running.
`proposed`	number	How many candidates its agent proposed.
`best`	object or null	`{ candidateId, name, seed, searchScore, searchInterval, heldOutScore, heldOutInterval, costPerRun, onlyOnSeenTasks }`: the best candidate, with fields as on a candidate. `seed` is `true` when it is the live config.
`startedAt`, `endedAt`	string, string or null	RFC 3339 times.

Search detail

Field	Type	Description
`split`	object	`searchedTasks`, `heldOutTasks`, and `searched` and `heldOut`, each a list of `{ runId, taskId }` from the set.
`iterations`	array	Each `{ number, kind, runId, state, attempts, proposed, agentSpent, reflection }`. `kind` is `propose` or `reflect`. `state` is `wanted`, `running`, `ended` or `failed`. `attempts` counts its proposals, refused ones included; `proposed` lists the candidates it admitted, by id. `agentSpent` is what its agent run spent, once known. `reflection` is a reflection for a `reflect` iteration, else null.
`seed`	object	The live config the search started from, as a candidate. Its search score counts the searched runs only, its held-out score the held-out runs.
`candidates`	array	Every candidate its agent proposed.
`frontier`	array of strings	The candidates on the frontier, by id, cheapest first, the seed always among them. With `objective` set to `cost`, only candidates whose `searchedGain.verdict` is `better` or `within-noise`.
`onlyOnSeenTasks`	number	How many frontier candidates' `onlyOnSeenTasks` is `marked`. See Only on seen tasks.
`minGain`	number	The smallest gain over the seed the line counts as real: `replay_check.min_gain`, or `0.05` when the line has none.

Reflection

What an iteration that looked into why the search stalled looked into and found.

Field	Type	Description
`standing`	string or null	The candidate whose failures it looked into, by id: the seed's when nothing beat the live config. Null until it looked.
`tasks`	array	The searched tasks that config still failed, each `{ taskId, originalRunId, replayRunId }`: the task, its run in the set, and the replay run that failed it. Never a held-out task.
`causes`	array	One per task given a cause, each `{ taskId, originalRunId, replayRunId, cause, evidence, suggestion, recordedAt }`. `cause` is `config`, `task`, `judge`, `check` or `infra`; see Causes. `suggestion` is a string or null.
`outcome`	string or null	Once it ended: `went-on`, `no-config-cause`, `no-failures`, `did-not-run` or `out-of-budget`. See Reflection outcomes.
`endedSearch`	boolean	Whether the search stopped searching when it ended: `true` for every `outcome` but `went-on`.

Candidate

Field	Type	Description
`candidateId`	string
`name`	string	The slug the agent named it by, or `seed` for the live config.
`seed`	boolean	Whether it is the live config.
`iteration`	number or null	The iteration that proposed it. Null for the seed.
`hypothesis`	string or null	What its agent expected it to improve, and why.
`fingerprint`	string or null	Its config fingerprint.
`changes`	array	How it differs from the live config, each `{ part, name, kind, was, now }`, as on a replay candidate.
`replayId`, `replayState`	string, string or null	Its replay on the searched runs; for the seed, its replay of the whole set.
`heldOutReplayId`, `heldOutReplayState`	string or null	Its replay on the held-out runs, when it had one.
`searchScore`	number or null	Its success on the searched runs, 0 to 1.
`searchInterval`	object or null	`{ rate, low, high }`: `searchScore` with its 95% interval.
`searchScored`	number	How many searched replay runs, copies included, were scored.
`searchedGain`	object or null	Its gain over the seed on the searched runs, paired task by task, as a replay's gain. Null for the seed.
`heldOutScore`	number or null	Its success on the held-out runs, 0 to 1, once that replay finished.
`heldOutInterval`	object or null	`heldOutScore` with its 95% interval.
`heldOutGain`	object or null	Its gain over the seed on the held-out runs, once both its held-out replay and the seed's replay finished. Null for the seed.
`searchedPairs`	object or null	`{ wins, losses, ties }`: the searched tasks whose change the judge, comparing it side by side with the seed's replay, found better, worse or as good. Null for the seed.
`heldOutPairs`	object or null	The same on the held-out tasks. Null for the seed and before its held-out replay.
`costPerRun`	number or null	What it spent per scored searched run. Null while any of it went unpriced or was in another currency.
`spent`	object	`{ amount, currency }`.
`onFrontier`	boolean	Whether it is on the frontier.
`onlyOnSeenTasks`	string or null	When `searchedGain.verdict` is `better`: `marked` when `heldOutGain.verdict` is not, over at least 5 tasks scored on both sides, so its gain showed only on the tasks the search could see; `too-few-held-out` when `heldOutGain.tasks` is under 5. Null otherwise, until `heldOutGain` is known, and for the seed. See Only on seen tasks.
`pullRequest`	object or null	`{ id, state, url, reason }` of the config pull request picking it opened. Null until a person picks it, and always for the seed.

A searched candidate is not a replay candidate resource: it is not listed, imported, read or withdrawn there.

Config pull request

Field	Type	Description
`id`	string
`state`	string	`opening`, `opened` or `refused`.
`url`	string or null	The pull request on GitHub, once opened.
`reason`	object or null	Why it was refused: `{ code, message, attribute, expression, wanted }`. `code` is one of the config pull request reasons. `attribute` is the station attribute it is about, `expression` the Terraform expression that computes it, and `wanted` the value to set by hand.
`teamId`, `searchId`, `candidateId`, `lineId`, `stationId`	string
`candidateName`, `stationKey`	string
`repository`, `path`	string or null	The line's `source` when it was picked. Null when the line named none.
`branch`	string	The branch Spotlight commits to.
`attempts`	number	How many times a person picked it.
`createdAt`, `updatedAt`	string	RFC 3339 times.

Picking a candidate whose pull request is opening or opened answers that one. Picking a refused one tries again, reading the line's source afresh.

Who may read and change

Reading, and estimating, need a person on the team or an owner or admin of its workspace, or a workspace token of the team's workspace with the read scope. Anyone else is answered 404. Starting and stopping a search and picking a candidate also need plan: a token with read alone is refused with 403. See Team outcomes.

Starting a search is recorded as a configuration change of the kind replay-search with the action created, and stopping it as updated. Picking a candidate is recorded as config-pull-request, keyed by its branch: created, and updated when picked again after a refusal.

Errors

Status	Cause
`401`	No session or token, or an unknown token.
`403`	A change by a caller who may read the team but not plan.
`404`	The team, line, station, set, search, candidate or pull request is not one the caller may see, as in `Search <searchId> not found.` A set at another station than the one in the path.
`409`	Starting a search of a station that is being searched (`Station <stationId> is already being searched. Please stop that search first.`), or over a set whose station has left its line. Stopping a search that has ended (`Search <searchId> has ended.`). Picking the seed (`The seed is already the station's live config. Please pick a candidate the search proposed.`), or a candidate whose replay on the searched runs has not finished (`Candidate <name> has no finished replay on the searched tasks. Please pick it once its replay finishes.`).
`422`	A missing or invalid field, at its path, such as `budget.iterations`. An `objective` other than `success` or `cost` (`Objective must be success, cost or left out.`). A count out of bounds (`Search iterations must be from 1 to 10.`). An agent the runners cannot start (`Harness <harness> is not one that a runner can start. Harnesses: <harnesses>`). An agent model or effort that is empty or longer than 200 characters (`Search agent model must be 1 to 200 characters, or left out.`, and the same for `effort`). A model of 61 to 200 characters (`Model must be 1 to 60 characters, or empty.`), or an effort that is not a known level (`Effort must be one of low, medium, high, xhigh, max, or empty.`): these checks of the form are the same for every harness. A set covering too few tasks (`A search needs at least 2 held-out tasks and 3 searched tasks. The replay set covers <n> tasks. Held out: <m>`). A ceiling too low (`Search ceiling must allow at least 0.01 <currency> for each replay run. Please raise the ceiling, or lower the iterations or candidates per iteration.`). `budget.repeats` not a number (`Repeats must be a number.`) or out of range (`Search repeats must be from 1 to 5.`).

Searches API

EndpointsCopy Anchor Link

Start a searchCopy Anchor Link

Estimate a searchCopy Anchor Link

WarningsCopy Anchor Link

SearchCopy Anchor Link

Search detailCopy Anchor Link

ReflectionCopy Anchor Link

CandidateCopy Anchor Link

Config pull requestCopy Anchor Link

Who may read and changeCopy Anchor Link

ErrorsCopy Anchor Link

On this page