Why changes are replayed on past tasks
A team changes a work station to make its runs better, with a sharper prompt or another model. Whether the change helped shows in the runs after it, but slowly and with noise. Each run works a different task, so a run that went badly after the change may have had a harder task. People act on a run hours or days after it ends, and a pull request merges later still. By the time enough runs have been recorded to say anything, the station may have been changed again.
A replay answers the question sooner, on fixed ground. It takes runs the station has already done and runs their tasks again under a changed config, called a candidate. The tasks are the same for every candidate, so the results can be set side by side, task by task, before the change reaches the team's real work.
The same start, a different config
For a comparison to mean anything, the only thing that differs between a replay and the run it repeats has to be the config. So a replay starts where the original run started, as closely as Spotlight can rebuild it:
- the same workspace, restored from the exact archive the original restored, with each repository checked out at the commit the original started from;
- the task as it was briefed to the original: its title, description, notes, attachments and decisions then, not as they read now;
- the team memories the original was briefed with, and only memories that stood when it started;
- the station's history and outcomes, as
spotlight historyreads them, as they stood when the original started.
What changes is the station block: the candidate's prompt, harness, model, effort, harness profile, skills, check and judge, compiled into the line as it is now.
What a replay may not touch
A replay works a real task in a real repository, and its agent believes it is doing the work. Anything it did outside the sandbox would act on the task a second time: a push to the task's branch, a comment on its pull request, a question in someone's inbox, a memory proposed to the team. So a replay is a run of its own kind. It never holds the task, and it leaves nothing behind but its own result:
- Its git access is fetch-only. A push is refused.
- It is given no tools and no MCP servers, and so wears no secrets.
- No hook runs for it.
- It cannot answer a pull request's remarks, propose a memory or read a task's trace.
- Its change stack is drafted and checked, and never sent to GitHub.
- It never asks a person. A replay that would put the task to a person, by handing it over to one or asking permission, ends there and counts as asked.
- It is not counted among the station's outcomes or in the Insights figures of the team's runs, and no memories are drafted from it. Insights shows replays apart, under Candidates on its Stations tab.
The sandbox's network is not narrowed further. Egress lets a replay reach the hosts the host's allow-list admits, as for any run; see Limit where sandboxes can connect.
Scored against the station as it is now
A replay is scored by what the candidate itself asks for. If the candidate's
station drafts change stacks (diffs: true) and has a check, its change must
pass that check. If it has a judge, the judge scores the change against its
rubric. Where people accepted the original's change, by closing the task as
the first thing they did on it after the run, the judge is also shown that
change and scores the replay against it. A run passes the judge on a yes, on 4 or more on a 1 to 5
rubric, or, for a judge with claims, on meeting at least its pass share of
them. With both a check and a judge, a run must pass both.
A replay's judge scores never count toward the judge's agreement with people, its standing, alerts or the Insights figures of the team's runs. They are only compared between replays.
Some runs cannot be scored. A run that changed nothing gives the check nothing
to run on; a check that could not run, or judging that could not finish, says
nothing either way. Those runs are unscored and left out of the success rate,
each with its reason. So are runs that could not be replayed or started, runs a
replay's stop or ceiling cut short, runs Spotlight abandoned with no failure
code, and runs that failed for a reason outside the candidate: a workspace that
could not be made ready before the agent began (unprepared), no runner
provisioned (unprovisioned), hooks unanswered (hooks-unanswered), a lost
runner (runner-lost) or superseded.
Only some runs count against the candidate as failures:
a run that failed with exited, with timed-out (it kept running past its
host's time limit), with refused (Spotlight would not start it as configured,
as for a missing harness profile or tools, MCP servers or secrets a replay is
not given) or with no code; a run that asked a person; and a run that failed
its check or judge. In
the docs' view, this keeps a failure to measure from reading as a failure of
the candidate.
Every candidate is compared with the station's live config replayed on the same runs: most often the set's baseline, replayed at about the same time, and for a replay with the answer, the finished live replay that started it. The original runs are not the comparison. In the docs' view, they ran under conditions a replay cannot bring back, such as an earlier model release or a dependency that has since moved, and a baseline replayed now meets the same conditions the candidate meets. A task regresses when the baseline succeeded on it and the candidate did not, or when the judge scored the candidate lower. With more than one copy of each run, the rule changes a little; see Copies of the same run.
Side by side with the baseline
A judge's score is absolute: it says how good one change is, read on its own. Where there is a replay to compare with, the judge can instead be shown both changes and asked which is better, which makes its verdict steadier and easier to check. In the docs' view, two changes can both pass while one is plainly closer to what the task needed, and scores given to each on its own may not show it.
So once a replay has finished, on each task where both it and the replay it is compared with changed something, and people accepted the original's change, the judge is shown the two changes together, labelled A and B, beside the accepted one, and asked which is better. It is not told which change is the candidate's, so it cannot favour it. It reads the pair twice, once each way round, and when the two readings disagree, the task counts as a tie. In the docs' view, a model can lean towards whichever change it saw first, and a preference that flips with the order was never a preference.
The wins, losses and ties sit beside the success rate and never change it. The success rule, the gain and its verdict stay as they are. In the docs' view, a comparison that counts only tasks both sides changed and people accepted covers too few tasks to replace the rate, but where the rate reads Within noise, it can say which way the candidate leans.
A replay other than a baseline keeps one counterpart, chosen when it starts: a proposed candidate the baseline standing then, a search's candidates the live config the search replayed when it started, and a replay with the answer the live replay whose finishing started it. Its task-by-task rows and its gain use that one replay, and so do its wins and losses, where it has them; a replay with the answer is never set side by side. A baseline started later does not change what an earlier replay was compared with. In the docs' view, figures read against two different replays could not be read together. Pairs judged under different rubric versions are left out; in the docs' view, comparing them would mix two questions.
The comparisons cost model calls. They count against the replay's ceiling, with room for two tries set aside before each is asked, and a pair that would pass the ceiling is not asked. They do not count toward the replay's spend or its cost per run, since they measure the change, not what running the candidate costs; that reason is the docs' own.
Every result comes with a range
An agent does not do the same thing twice. Replayed twice on the same task under the same config, a run can pass the first time and fail the second. So a replay's success rate is partly chance, and so is the difference between a candidate and its baseline. On a set of 20 tasks, one task going the other way moves the rate by 5 points.
So every success rate comes with a range: a 95% interval, drawn by resampling the replay's tasks 2,000 times and reading where the middle 95% of the resampled rates fall. A candidate's gain over the baseline is worked out task by task, as the candidate's success on each task less the baseline's on the same task, and its range is resampled the same way. In the docs' view, pairing the tasks this way keeps how hard each task is out of the gain, since the candidate and the baseline faced the same tasks. Here, and wherever results count tasks, a task is one run of the set: a set that holds two runs of the same task counts them as two.
A gain then reads one of three ways. It is better when its whole range lies
above no change and it reaches the smallest gain the line counts as real,
min_gain in the line's replay_check, 5 points unless set. It is worse when
its whole range lies below no change and the loss reaches the same size.
Anything else is within noise. With fewer than 5 tasks scored on both sides,
every gain is within noise. In the docs' view, a gain can be clearly real and
still too small to be worth a change, and min_gain is where a team says how
small that is.
A replay check reads its gain the same way, but the gain does not decide the check: a pull request still passes or fails on its count of regressions.
Copies of the same run
A replay can replay each run up to 5 times. Each task's success is then the share of its copies that succeeded, so one task can count as two thirds of a success. The rate is averaged over tasks, not over copies, so a task replayed more often does not weigh more, and the range is drawn over tasks. More copies make each task's score steadier, which narrows the range. They add no tasks: a set that is too small, or whose tasks are too alike, stays so.
With copies, a task regresses only once every copy on both sides has settled. Counted by success, it regresses when the share of the baseline's copies that succeeded is higher than the candidate's share by more than 50 points. One copy failing where the baseline succeeded every time is not enough. With one copy each, this is the rule above: the baseline succeeded and the candidate did not. A judge's scores are compared as the average over each side's copies.
The baseline a replay is compared with must have at least as many copies of each run. When it has fewer, a new baseline starts beside the replay with the same number of copies.
How noisy a set is
Replaying the live config with copies shows how far a set's success moves when nothing changed: the spread between the copies' success rates, and which tasks went both ways. That spread is the set's noise, and the replay set's page shows it once measured. It is measured on the station's config as it is now, from the newest finished replay of the live config with at least 2 copies. Once a change gives the station a new config fingerprint, the noise is unmeasured again, since the new config may be steadier or less so.
A search reads the noise before it starts. With none measured, it warns that a
small gain may be luck. With a spread larger than min_gain allows at the
number of copies asked for, it warns and suggests more: the copies needed are
the spread divided by min_gain, squared and rounded up, and it suggests at
most 5. That rule takes the spread to shrink with the square root of the
number of copies, as an average of independent tries does. More than 5 needed
means no search of that set, at any number of copies, can tell such a gain
from noise. A search starts all the same; the warning is advice.
Whether a set can tell configs apart
Noise is one way a set can fail to tell a better config from a worse one. There are others, and on any of them a search would chase something other than a better config. A set's page reads each of them for the live config as it is now, as its health, and a search warns about each before it starts.
- Room to improve. When the live config already succeeds on more than 95% of the set, a candidate has almost no tasks left to win. A search of such a set aims, unless told otherwise, for the same success for less: it ranks the candidates whose gain over the live config is not Worse by what a run costs. A set can also leave little room well below 95%, when no config could do better on its tasks; see Why a set is replayed with the answer.
- A judge that disagrees with itself. A judge is a model, and asked twice about the same change it can answer differently. When it passes or fails the same change alike less than 80% of the time, part of every score is chance, and a search would chase luck.
- Flaky checks. A test that passes on one run and fails on the next, on the same change, can flip a candidate's result by chance. Running the check a second time on the same change shows which tasks do it.
What the set holds matters as well. A set whose original runs were nearly all accepted, or nearly all sent back, says little about how runs that turned out other ways would go. In the docs' view, a candidate that wins on such a set may lose on the runs it lacks. So a set's page counts how its original runs turned out, where their tasks came from and when they ended, and warns when one outcome covers more than two thirds of a set of 6 runs or more. While you add runs, it suggests recent ones of the outcomes the set is short of, but never adds them itself. In the docs' view, which tasks a set should stand for is the team's choice.
Nobody has to ask for these readings. The first baseline of the live config as it is now checks the set: it replays each run at least 3 times, which measures the noise, and once it finishes, each change is judged and checked a second time. Its ceiling grows with its copies, so it costs about three times a single replay. In the docs' view, a set is only worth reading once it has been checked, and the first baseline has to replay the live config anyway. A set of fewer than 5 runs is never checked, since it is too small to measure noise on. A search's starting replay and a pull request check's baseline are never the check. The second judgings cost model calls, so they count against the check's ceiling and stop once it would be spent; the rest are not judged again. A second judging or check never changes a run's result. In the docs' view, it measures the judge and the check, not the run. The second judgings also do not count toward what the check's runs cost.
Why a set is replayed with the answer
A set is for telling a better harness from a worse one: a better prompt, skills, tools or setup. It should reward a better config around the model, not a better model. So the question for a set is how much a better config could gain on its tasks.
Spotlight answers it with an upper bound. After a replay of the live config over the whole set finishes, it replays the live config again with each run's answer added to its briefing: for a run people accepted, the change they accepted; for a run a person sent back, what they wrote when they sent it back. Runs that failed, asked a person or turned out another way have no answer, so they are left out, and both sides are compared on the same runs. The gain the answer brings is the most a better config could gain on those runs. It is never the gain to expect: no config hands the agent the answer.
When even the answer does not do clearly better, the live config already does about as well on these tasks as any config could show, and a search on the set would find little. What helps is adding tasks the live config fails. A search warns about it before it starts.
The replay with the answer is a replay like any other. It has its own ceiling, the same as the live replay's, its own runs and its own row on the set's page. The set's health, the replay's own page and the API all read its gain against the live replay that started it, task by task and with a range. Its changes are never judged blind beside that replay's. In the docs' view, an upper bound that skipped the usual scoring would not be comparable with the gains it bounds.
Spotlight starts one for each set and each config of the station, and only while the station still runs the config that was replayed. If it is stopped or runs over its ceiling, the next replay of the live config over the whole set starts another.
What it costs
A replay spends what its runs cost, and what drafting and judging their changes cost. Each copy of a run is a run of its own, so a replay with 3 copies starts three times the runs. A person sets a ceiling on each replay, and once a replay's spend reaches it, nothing more of it starts. Runs already running finish, so a replay can end a little over its ceiling, and its page says by how much. The baseline started beside a proposed candidate has a ceiling of its own, the same as the replay that started it, so the two can spend twice that ceiling. When that baseline also checks the set, its ceiling can be larger, so at 1 or 2 repeats the two can spend more; see Checking the set. A replay of the live config becomes the baseline itself when the set has none, or when the set's runs changed since the last one. Costs in another currency than the ceiling's, and costs never reported, do not count toward it. The replay page shows them apart from the spend the ceiling counts.
Replays take runners the team's real work needs. At most 2 of a team's replay runs run at once, apart from a search's, which keep to a limit the search sets, and replay runs wait behind every waiting task run and chat run. In the docs' view, a replay answers a question about the future, and should never hold up work that people are waiting on now.
What a replay can start from
A run can be replayed only if Spotlight kept where it started. Each task run records the workspace archive it restored and the commit each repository started from. Runs that ended before Spotlight began recording this cannot be replayed.
Spotlight keeps that archive for 30 days after the run started, and for as long as the run is in a replay set. After that the archive goes, and the run can no longer be replayed. A run is also refused if its task was deleted, if a file its task was briefed with is gone, or if a repository it worked in has left the line. Why a run cannot be replayed lists each reason.
Replays beside live runs
Once a team has replayed a few candidates, the question is which of them, if any, to apply. The Candidates section of the Stations tab on the team's Insights page sets every candidate replayed in a period beside the live config replayed on the same replay set, station by station, and its Config changes section can draw a station's replayed candidates on the same chart as the setups its live runs used.
The two kinds of point measure different things. A live setup's height is what happened to real runs: whether a person accepted the work, sent it back, or merged its pull request. A replayed candidate's height is replay success: the share of replayed past runs that passed the candidate's own check and judge. The setups chart under Config changes draws replayed points apart, as a series of their own, and says what their height is, so neither is read as the other.
A candidate is read against the live config replayed on the same runs, not against the live runs, because only that replay worked the same tasks at about the same time. A search's candidates are set beside the live config on that search's own runs, and a proposed candidate beside the set's baseline. A search's held-out score is never drawn as a point. In the docs' view, it comes from other tasks than the search score, so one point cannot stand for both, and it is the score to trust, so it is given as a figure of its own. The candidates of a pull request's replay check are left out: their verdict is on the pull request and on the replay set's page.
A replay changes nothing
A replay never changes the station, however well its candidate does. A person reads the result and makes the change in Terraform, the same way as any other change to a line. A search, which replays many candidates an agent proposes, goes no further: picking one of its candidates opens a pull request, which the team reviews and applies. A line can also have its CI replay each pull request's station changes before they merge. Why configuration lives in Terraform gives the reasons. Once applied, the change's own runs are recorded under its new config, and the record of each run's config shows whether it held up on the team's real work.