Replayed candidates API
Candidates, on the Stations tab of
a team's Insights page, is read through this endpoint. It answers JSON, and errors are
application/problem+json as on the
resource API.
GET /api/teams/{teamId}/replayed-candidates?period=30dIt answers the station configs the team replayed on past tasks in the period,
by station and by replay set, each set with its
baseline: the live config replayed. A
success rate here is a replay's, scored by the candidate's own checks and judge
on past tasks, and says nothing of how live runs went. A replay set holds runs,
and a task may have more than one run in it. Success rates, their intervals
and gains are worked out over the set's runs, each run counted once however
many copies of it were replayed (see Success).
compared and regressions count the set's runs, and scored counts replay
runs, copies included.
An interval is { rate, low, high }: the success rate with its 95% interval.
A gain is { gain, low, high, verdict, minGain, tasks }, as in the
Replays API.
Parameters
| Parameter | Description |
|---|---|
period | 7d, 30d or 90d: whole UTC days up to and including today. Default: 30d. |
A candidate is answered when it was replayed in the period:
kind | Answered when |
|---|---|
searched | Its search started in the period. |
proposed | A replay of it over the set started in the period. With more than one, its newest finished replay in the period is answered, or else its newest. |
Candidates of replay checks and of
replays with the answer
are left out, and the live config is answered only as each set's baseline. A set's
health and baseline are as they stand now, whatever the period.
The endpoint reads the team's newest 500 replays started in the period. When
there were more, it answers truncated: true and leaves out the oldest.
Answer
| Field | Type | Description |
|---|---|---|
period | object | { name, from, to }: the period's name, the start of its first UTC day, and the start of the UTC day after today, which it stops short of. RFC 3339 times. |
truncated | boolean | Whether the team replayed more in the period than one answer covers. |
stations | array | Each work station with a candidate replayed in the period, the newest replayed first: { lineId, stationId, sets }. A station removed from its line is still answered. |
Set
Each item of a station's sets, the newest replayed first:
| Field | Type | Description |
|---|---|---|
setId | string | |
name | string | |
stationGone | boolean | Whether its station has left its line. |
guardsChanges | boolean | Whether it guards its station's changes. |
health | object | { tasks, runs, oldestRunEndedAt, newestRunEndedAt, lastAddedAt, noise, saturation, judge, checks, oracle, minGain }. runs counts the runs the set holds now, and tasks the different tasks they worked. The three times are when the oldest and the newest of its runs ended, and when a run was last added to it; each is null while there is none. noise, saturation, judge, checks and oracle are the set's health, as in the Replays API; minGain the smallest gain its line counts as real. |
baseline | object or null | Its newest baseline that is queued, running or finished, whatever the period. Null while it has none. See Baseline. |
chartCurrency | string or null | The currency its cost and success chart is drawn in: the baseline's, or else that of its newest replayed candidate. Null with neither. |
candidates | array | The candidates replayed over it in the period, the greatest againstLive.gain first and those without one last, the newest replayed first among equals. See Candidate. |
Baseline
| Field | Type | Description |
|---|---|---|
replayId | string | |
state | string | queued, running, finished, stopped or over-budget. |
fingerprint | string or null | The config fingerprint of the live config it replayed. |
scored | number | How many of its runs were scored. |
successRate | number or null | Its success over the whole set, 0 to 1. Null while none was scored. |
interval | object or null | successRate with its 95% interval. |
costPerRun | number or null | What it spent per scored run. Null while any of its spend went unpriced or was in another currency than its ceiling's. |
currency | string | Its ceiling's currency, which costPerRun is in. |
endedAt | string or null | |
charted | string | Whether it is drawn on the set's chart. See Charted. |
Candidate
| Field | Type | Description |
|---|---|---|
candidateId | string | |
name | string | |
kind | string | searched or proposed. |
hypothesis | string or null | What the search's agent expected it to improve, and why. Null unless searched. |
fingerprint | string or null | Its config fingerprint. |
changes | array | How it differs from the live config, each { part, name, kind, was, now }, as on a replay candidate. |
replayId | string | Its replay: on the searched tasks when searched, over the set when proposed. |
replayState | string or null | That replay's state, as for a baseline. |
replayedAt | string or null | When that replay started. Null while a searched candidate's replay cannot be found. |
tasks | number | How many different tasks its replay covers: the searched ones for a searched candidate. |
runs | number | How many runs of the set its replay covers, each once however many copies. |
successRate | number or null | Its replay success, 0 to 1. For a searched candidate, its search score, as its search page gives it; for a proposed one, its replay's success. |
interval | object or null | successRate with its 95% interval. |
scored | number | How many of its runs were scored. |
heldOut | object or null | { replayId, state, score, interval, tasks, runs }: its replay on the tasks its search held out, kept apart from successRate. score is its success there, 0 to 1, once that replay finished, else null; tasks is how many different tasks the search held out, and runs how many held-out replay runs, copies included, it was scored on. Null for a candidate not replayed there, and for every proposed candidate. |
costPerRun | number or null | What its replay spent per scored run. Null while any of its spend went unpriced or was in another currency. |
currency | string | The currency costPerRun is in: its search's ceiling's, or its replay's. |
live | object or null | { successRate, interval, costPerRun, currency }: how the set's baseline did on the compared runs alone. currency is the baseline's, and may differ from the candidate's. Null without a baseline. |
successRateOnCompared | number or null | The candidate's own success on the compared runs alone, 0 to 1, to set beside live.successRate. Null without a baseline. |
intervalOnCompared | object or null | successRateOnCompared with its 95% interval, to set beside live.interval. Null without a baseline. |
costPerRunOnCompared | number or null | What the candidate spent per run on the compared runs alone, in its own currency, to set beside live.costPerRun. Null without a baseline, and while any of that spend went unpriced. |
regressions | number or null | How many of the compared runs it regressed on, as its replay page counts them. Null without a baseline. |
compared | number | The compared runs: how many of its runs the baseline replayed too. |
againstLive | object or null | Its gain over the baseline on the compared runs, paired run by run, against the line's min_gain. Null without a baseline. |
onFrontier | boolean or null | Whether it is on its search's frontier. Null unless searched. |
search | object or null | { searchId, state, objective, startedAt, seed }: the search that proposed it. state is seeding, searching, holding-out, finished or stopped. objective is success or cost, the search's aim, which decides onFrontier. seed is { successRate, interval, costPerRun, scored, charted }: the live config the search started from, scored on the same searched runs as its candidates, as the search page gives the live config's search score, with its cost per run in the candidate's currency. Its charted counts it as finished. Null unless searched. |
pullRequest | object or null | { id, state, url, reason }: the config pull request picking it opened. Null until a person picks it, and for every proposed candidate. |
charted | string | Whether it is drawn on the set's chart. See Charted. |
shown | object | The figures to set beside the live config. See Shown. |
Charted
charted says whether a baseline, a candidate or a search's seed is drawn
on its set's chart, in the set's chartCurrency:
charted | When |
|---|---|
unfinished | Its replay has not finished. |
unscored | It has no success rate. |
unpriced | It has no cost per run. |
other-currency | Its cost per run is in another currency than chartCurrency. |
priced | Otherwise: it is drawn. |
The first that holds is answered.
Shown
shown is { basis, costBasis, successRate, interval, liveSuccessRate, runs, costPerRun, liveCostPerRun }:
| Field | Type | Description |
|---|---|---|
basis | string | compared when the candidate has a live and compared is above 0, else overall. |
costBasis | string | compared when basis is and live.currency is the candidate's currency, else overall. |
successRate, interval | number or null, object or null | successRateOnCompared and intervalOnCompared on the compared basis, else successRate and interval. |
liveSuccessRate | number or null | live.successRate on the compared basis, else null. |
runs | number | compared on the compared basis, else scored. |
costPerRun | number or null | costPerRunOnCompared when costBasis is compared, else costPerRun. |
liveCostPerRun | number or null | live.costPerRun when costBasis is compared, else null. |
Who may read
A person on the team or an owner or admin of its workspace, signed in, or a
workspace token of the team's workspace
with the read scope, sent as Authorization: Bearer <token>. Anyone else is
answered 404. See
Team outcomes.
Errors
| Status | Cause |
|---|---|
401 | No session or token, or an unknown token. |
404 | The team is not one the caller may read: Team <teamId> not found. |
422 | period is not 7d, 30d or 90d, or is given twice: Period must be 7d, 30d or 90d. |