Replay past tasks against a change to a station

Use this to try a change to a work station, such as a new prompt or another model, on tasks the station has already worked, before you apply it. A replay starts each chosen task again from where its run started, under the changed config, and scores what it does. Nothing a replay does reaches the task, GitHub or anyone's inbox. You need:

• To be on the team, or an owner or admin of the workspace.
• Runs at the station that can still be replayed. Spotlight keeps where a run started for 30 days. See What a replay can start from.
• A host serving the workspace with the station's harness. Replays run on the workspace's runners, after any task run that is waiting. See Install a host.
• A workspace token with the plan scope, and the spotlight binary to propose the change. You don't need a host on your own computer; see Install.
• The team's configuration in Terraform, to apply the change afterwards. See Manage Spotlight with Terraform.

Pick the tasks to replayCopy Anchor Link

A replay set is a list of the station's past runs that you replay together, up to 50.

Add runs the station did well as well as runs it did badly. While a run is in a set, Spotlight keeps the workspace it started from.

Above the set's runs, a bar shows how they turned out. Hover over the ⓘ beside Runs for where their tasks came from and when they ended. If a warning sign shows at the right of the panel's title, most of the set's runs turned out one way: add runs that turned out other ways. See What a set is made of.

Propose the changeCopy Anchor Link

A candidate is the station's config as you want to try it. You write it as JSON, in the shape of one entry of a line's stations, and propose it with spotlight replay propose. Spotlight checks it as it would check the line with the station replaced, and never applies it.

export SPOTLIGHT_URL=https://app.usespotlight.ai
export SPOTLIGHT_API_TOKEN=<workspace token>

WORKSPACE=$(curl -s -H "Authorization: Bearer $SPOTLIGHT_API_TOKEN" \
  "$SPOTLIGHT_URL/api/resources/workspaces" | jq -r '.items[0].id')

curl -s -H "Authorization: Bearer $SPOTLIGHT_API_TOKEN" \
  "$SPOTLIGHT_URL/api/resources/workspaces/$WORKSPACE/lines/import?address=core/delivery" \
  | jq '.definition.stations[] | select(.key == "implementation")' > implementation.json

spotlight replay propose --station core/delivery/implementation \
  --name "Higher effort" implementation.json

If Spotlight refuses the block, each message names its field under station, as in station.effort. The command refuses a .tf or .hcl file: give the block as JSON. See spotlight replay propose.

Start a replayCopy Anchor Link

A replay you start is compared with the set's baseline: the station's live config replayed on the same runs. If the set has no baseline covering all its runs, because it was never replayed or runs were added since:

• Under Live config, the replay becomes the baseline.
• Under a proposed candidate, a baseline starts beside it, with a ceiling of its own the same as yours, so the two may spend up to twice the ceiling. At 1 or 2 repeats, they may spend more when that baseline also checks the set, as below.

If the set has a baseline covering its runs, with at least as many copies of each as you asked for, a Live config replay runs beside it, and no new baseline starts. Otherwise a new baseline starts as above, with your number of repeats.

The first baseline of the live config as it is now also checks the set, when the set holds 5 runs or more. At fewer than 3 repeats, it replays each run 3 times, and its ceiling can be larger than yours. Once it finishes, each change is judged and checked a second time, within its ceiling. A set of fewer than 5 runs is never checked. Before you set the ceiling, see Checking the set for that baseline's ceiling, and Replay set page for what the replay and its baseline may spend together and which notes the dialog shows.

At most 2 of a team's replay runs run at once. See Limits and timeouts.

Follow the replayCopy Anchor Link

The replay's page shows its success, its gain over the replay it is compared with, and what it has spent of its ceiling. From the command line:

spotlight replay watch --team core <replay id>

It prints progress every 15 seconds until the replay and the replay it is compared with have ended, then the result for each task. It exits 2 if the candidate regressed on any task, so a CI job that runs it fails on a regression. See spotlight replay watch.

To stop a replay, select Stop, then Stop replay, on its page, or run spotlight replay stop --team core <replay id>. Nothing more of it starts, and its running runs are stopped. A replay that spends its ceiling ends as Over budget: nothing more starts, and runs already running finish.

Read the resultCopy Anchor Link

Success gives the replay's success rate; hover over its ⓘ for the range it likely falls in. Gain gives its gain in points over the replay it is compared with, as in +12 pts, with a verdict of Better, Worse or Within noise and that replay's success on the tasks both scored, as in Better · baseline 50%. Its ⓘ gives the gain's range. See Gains for when each verdict shows.

Act on Better or Worse. If a result you expected to matter reads Within noise, add runs of more tasks to the set, or replay with more repeats. To change the smallest gain that counts, see Set how large a gain must be.

Under Tasks, the replay's page lists each task, regressions first. A regressed task has a red warning sign beside its title. See Regressions for when a task regresses. Hover over a result for its reasons, such as Checks failed. With more than one copy, a task shows how many copies succeeded; hover over it for each copy's result.

Select a task to open its details beside the list: each run's result, why it succeeded, failed or went unscored, its checks and what it changed. Select Open the run to open that replay run, or Original run for the run it replayed. See Results for every result, Unscored included, and every reason.

If the station has a judge, a Side by side tile shows its wins, then its losses and ties, once the replay has finished and any task was compared. Under Tasks, the Closer column says which change the judge found closer to the accepted change: This one, the replay it is compared with, as in The baseline, or Neither. Select the task to read the judge's reasons in its details. Read the wins and losses beside the success rate. See Side by side for how the judge compares the changes.

If the candidate did worse, propose another candidate and replay the same set. To see what a replay with a gain is compared with, hover over its Gain on the set's page. Once that replay's success is known, the hover names it, as in Baseline: 50% or Earlier baseline: 50%. A replay without a gain reads –.

Read the set's healthCopy Anchor Link

Read whether a set can tell a better config from a worse one before you read small gains on it or search on it. There is nothing to start: the set's first baseline checks it, then Spotlight replays the live config with each run's answer.

On the set's page, read the rows under Health. Hover over the ⓘ beside a reading for its advice. A reading marked with a warning sign found a problem:

• Noise more than the gain the line counts as real: replay candidates with more repeats, add runs of more tasks, or raise min_gain.
• Room to improve reads Little or Almost none: add runs of tasks the live config fails, or search for the same success for less. See Read the room to improve.
• Judge agreeing with itself below 80%: make the judge's rubric more specific. See Set a station's judge and alerts.
• Flaky checks lists tasks whose check flipped: fix the flaky tests, or take those tasks' runs out of the set.

The readings belong to the station's config as it is now. Once an applied change gives the station a new config fingerprint, they read Not measured or Not checked until the first baseline of the new config checks the set. A replay of the live config with 2 or more repeats, started any other way, measures the noise too once it finishes. See Health for every reading.

Read the room to improveCopy Anchor Link

There is nothing to start. Once a replay of the live config over the whole set finishes, such as the first baseline, Spotlight replays the live config once more on the set's accepted and sent-back runs, handed each run's answer: the change people accepted, or what a person said when they sent it back. It does this once for each config the station has. The replay is named With the answer and its date, and shows under Replays like any other. It has a ceiling of its own, the same as the live replay's.

While it runs, Room to improve reads Measuring with the answer. Once it finishes:

• If it reads Little, add runs of tasks the live config fails, or search for the same success for less.
• If it reads Too few runs to tell, add runs that people accepted or sent back.
• If it reads Couldn't measure with the answer because the replay with the answer was stopped or ran over its ceiling, replay the live config over the whole set again.

Hover over the ⓘ for the live config's success and the success with the answer. See Room to improve for what each reading means.

See Why a set is replayed with the answer.

Set how large a gain must beCopy Anchor Link

If a gain of a few points is not worth a change to your team, raise the smallest gain the line's replays, searches and checks count as real. Give the line's replay_check block a min_gain, as a share from above 0 to 1, and apply it:

replay_check {
  most_regressions = 0
  ceiling          = 10
  currency         = "USD"
  min_gain         = 0.1
}

Without it, a gain counts as real from 0.05, which the app shows as +5 pts. The same block is what lets spotlight replay check check the line's pull requests; see Check Terraform pull requests for station regressions.

Compare the station's candidatesCopy Anchor Link

Use the team's Insights page to set every candidate replayed at the station, yours and those of any search, beside the live config.

See Candidates for every figure in the section.

Make the changeCopy Anchor Link

A replay never changes the station. To keep a change that did better:

Runs with the new config start a new group under Runs by config on the station's view on Insights. See Check whether it helped.

To have every pull request that changes the station replayed on a set before it merges, turn on Guards config changes on the set's page. See Check Terraform pull requests for station regressions.

To let go of a set you no longer need, select Delete on its page. Its replays go with it, and Spotlight stops keeping its runs' workspaces for it.

RelatedCopy Anchor Link

• Why changes are replayed on past tasks
• Replays
• spotlight replay
• Replay candidates
• Improve a station from its runs
• Search for a better station config