Search for a better station config

Add a source block to the line's spotlight_line resource, naming the repository and the .tf file, from the repository's root, that holds the resource:

resource "spotlight_line" "core_delivery" {
  # ...

  source {
    repository = "acme/platform"
    path       = "teams/core/delivery.tf"
  }
}

Run terraform apply.

On the line's board, select the station's name. On the Station page, open the Improve tab. On the Searches card, select Start a search.

If the station has no replay set and no searches yet, the card reads No searches yet and shows no Start a search: create a replay set first. If the button is greyed out, hover over it to see why, such as No replay set covers 5 different tasks yet. Add runs of more tasks to a set, or wait for the station's search under way to end.

Under Search on, choose the replay set. A set with runs of fewer than 5 different tasks can't be chosen: a search keeps about 30% of the tasks back to check its results on, and needs some left to search on. See Why a search holds tasks back.

Set the budget:

• Iterations: how many rounds, so how many times the agent runs, from 1 to 10.
• Candidates per iteration: how many candidates each round may propose, from 1 to 5.
• At once: how many of the search's replay runs run together, from 1 to 10. They run beside the team's other replays, below every task run.
• Repeats: how many times each of its replays replays each run, from 1 to 5. The live config's replay, every candidate's and the held-out replays all take this many copies. More copies narrow the range around each score, and the runs and spend grow with them.
• Ceiling: the most the whole search may spend, its replays and its agent's runs, and its currency, as in 20.00 and USD.

Under Aim for, choose what counts as better:

• More success: candidates that succeed more often.
• The same success for less: candidates that succeed as often as the live config, within noise, and cost less a run.

If the live config already succeeds on more than 95% of the set, The same success for less is chosen for you, with a note saying so. You can change it.

Under Its agent, choose the harness, and the model and effort if you want other than the harness's own defaults. This is the agent that proposes candidates, not the station's own.

Read the estimate under the fields. It gives how many tasks are searched and held out, how many copies of each run, the most runs the search may start, what each replay may spend, and roughly how long it takes, from how long the set's runs took. If Ceiling says the ceiling must allow at least 0.01 for each replay run, raise the ceiling or lower the iterations, candidates or repeats.

Read any warning under the estimate:

• If nobody has measured how much the set's success moves by chance, start a replay of the set first, so its first baseline checks it, or set Repeats above 1.
• If the set's success moves more than the line counts as real, select the button under the warning, as in Replay each run 3 times, to set Repeats to what it suggests, and raise the ceiling if the estimate asks. If the warning says even 5 copies won't bring it under, read small gains with care, or raise the line's min_gain.
• If the live config already succeeds on almost all of the set, aim for the same success for less, or add runs of tasks the live config fails.
• If the judge gave the same verdict too rarely when it judged the same change twice, make its rubric more specific before you search.
• If the check flipped on changes it ran twice, fix the flaky tests, or take those tasks' runs out of the set. The set's page lists them.
• If the live config, handed each run's answer, gained little on the set, a better config has little room to show there. Add runs of tasks the live config fails. See Read the room to improve.

The search starts with a warning all the same, but a gain it finds may be chance.

Select Start search. The search's page opens.

Open Inbox. A notice about the search reads, for example, The search for a better Build config ended with nothing the config can fix. Open it, and it moves under Resolved. Under Details, it says what happened, then names each task with what to fix. Select Open the search.

On the search's page, under Why it stalled (iteration <n>), the tasks are grouped by cause. Each shows what in the runs showed the cause and what the agent suggests. Select Open the failed replay run to check it yourself.

Fix what each group asks:

• The task itself: take the task's run out of the replay set, or reword the task. See Pick the tasks to replay.
• The judge: fix the judge's rubric. See Set a station's judge and alerts.
• The check: fix the broken or flaky test the station's check runs.
• Around the work: replay the task again. If it keeps failing, fix what the run needs, such as a tool or a service.
• The station's config: nothing. The search keeps trying configs for these.
• No cause recorded: open the failed replay run and decide for yourself.

If the search ended, start a new one once the fixes are in.

Select the candidate's name. In its details, select Pick. You can pick any candidate whose replay on the searched tasks has finished, while the search runs or after, and more than one. If Pick is greyed out, the text beside it says why.

If the candidate is marked Only on seen tasks, a warning above Pick says its gain may not carry over to new work. You can still pick it.

The candidate's details read Opening a pull request…, then Pull request opened. Select it to open the pull request on GitHub. Its row under Pull request reads Opened, linked to it too.

Review the pull request. Its description has the agent's hypothesis, the candidate's scores beside the live config's with their ranges, whether each gain is more than noise, and its side-by-side wins, losses and ties when it has any. For a candidate marked Only on seen tasks, a paragraph under the opening one warns that its gain showed only on the searched tasks and gives both gains. It says instead when too few held-out tasks were scored to tell, or when the held-out replays had not finished as the pull request opened. The plan of the change runs in your Terraform workflow as for any other pull request; if the line checks pull requests, that check runs too.

Merge it and apply it as you would any change to the line.