Why a search holds tasks back
A replay answers one question: does this change do better than the station as it is, on these tasks? A search asks it many times over. An agent reads how the station's past runs went, proposes a changed config, sees how that config did when replayed, and proposes again. Each round learns from the one before.
That loop has a known way of going wrong, and most of a search's design is there to catch it.
Fitting the tasks it sees
An agent that sees which tasks a candidate failed, and why, will propose the next candidate to pass those tasks. After a few rounds the best candidate may mention a file name from one task, or a test from another, and score well on exactly the tasks it was tuned on. On the next real task it does no better than before. This is overfitting: a result that describes the tasks it was measured on, and nothing else.
The more rounds a search runs, the more its scores on the tasks the agent saw flatter its candidates. Those scores still rank the candidates against each other, but they can no longer say how a candidate does on work the station has not seen.
Tasks the agent never sees
So a search splits its replay set before it starts. About 30% of the set's tasks are held out: the agent never sees them, and the candidates are never replayed on them while the search runs. The rest are the searched tasks, the ones the agent studies and every candidate is replayed on.
The agent is not told which tasks are held out, only how many. spotlight history in its run leaves out the held-out tasks' runs and their results,
and it cannot read their traces. When a stalled search looks into why its
tasks fail, it looks only at searched tasks. Team memories stay out of a search
altogether: its agent can neither recall nor propose one.
Once the search stops searching, the best candidates are replayed on the
held-out tasks, and those scores are the ones to trust. Each score, and each
candidate's gain over the live config, comes with a range; see
Every result comes with a range.
A few choices follow from this.
- The split is by task, not by run. If a set holds two runs of one task, both land on the same side, so nothing about a held-out task leaks through a run of it the agent saw.
- The split is drawn from the search's own id, so each search draws its own, and the same search always draws the same one.
- A search needs at least 2 held-out tasks and at least 3 searched ones, so a set needs runs of at least 5 different tasks. With fewer, the held-out score would rest on a single task, or the agent would have almost nothing to learn from.
- Only the best candidates are replayed on the held-out tasks, up to as many as one iteration may propose. In the docs' view, replaying every candidate there would cost the most, and a score on held-out tasks is only worth having for a candidate someone might pick.
- The station's live config is replayed on the whole set at the start, so it has a held-out score to compare with from the beginning. The same replay becomes the set's baseline.
When a candidate's gain over the live config reads Better on the searched tasks but not on the held-out ones, the search marks it "Only on seen tasks". That is the pattern overfitting leaves: the gain held on the tasks the agent tuned against and did not reach the tasks it never saw. A marked candidate can still be picked. A held-out gain rests on few tasks and can read Within noise for a candidate that does help, so in the docs' view the person who picks should weigh it, with the mark in front of them on the search page and in the pull request. With fewer than 5 held-out tasks scored on both sides, the held-out gain cannot tell either way, so the search says that instead of marking the candidate.
A frontier of candidates
A candidate can do better by succeeding more often or by costing less. A bigger model may succeed on more tasks and cost far more a run. Which matters more depends on the team, and a search cannot know.
So a search keeps the frontier: the candidates for which no other succeeds as often for less. Everything off the frontier is beaten on both counts by something on it. The live config is always on the frontier, so every candidate is read against the station as it is. The search page names a best candidate, but a person picks, and may pick a cheaper one that succeeds almost as often.
What the search names as best depends on what it aims for, which a person chooses when it starts. Aiming for more success, the best is the candidate that succeeds most, held-out score first. Aiming for the same success for less, the search keeps out of its frontier, its held-out replays and its best every candidate whose gain over the live config does not read Better or Within noise, and ranks the rest by what a run costs. Only candidates cheaper than the live config are then replayed on the held-out tasks: in the docs' view, a candidate that costs as much has nothing to offer such a search. The best is then the cheapest, and the page calls it that.
A search aims for the same success for less unless told otherwise when the live config already succeeds on more than 95% of the set: there, more success has almost nowhere to come from. Its agent is told the same, and is pointed at what the station may spend without needing to, such as a cheaper model, a lower effort or a shorter prompt.
What a search may change
A candidate may change only the station's prompt, harness, model, effort, harness profile, skills, tools, decisions, memories and description. A candidate that changes anything else is refused before it is replayed.
What it may not change falls into three groups.
- How it is scored. A replay scores a candidate by the station's own check and judge. A search that could loosen the check or rewrite the judge's rubric could pass by lowering the bar. In the docs' view, a search must be held to a bar it cannot move.
- How tasks flow. Which stations a run may hand a task on to and whether it may finish a task shape the whole line, and the station's key, name and kind identify it there. A change to them is a change to the line, which in the docs' view a person should design.
- Where it runs and what is watched. The station's runners and its alerts are left as they are: they say where work is placed and what the team is told, and neither makes a run better.
A search starts from the station's live config when the search began, and every candidate is a change to that block.
When the config is not the problem
Some tasks fail for reasons no station config can reach: the task is unclear or asks for something impossible, the judge's rubric fails good work, the check is flaky, or the run broke around the work. In the docs' view, more iterations against such tasks spend the budget and find nothing, and an agent pushed to beat them anyway is pushed toward fitting them.
So when two iterations in a row find nothing better than the best config before them, a search with room left spends its next iteration looking instead of proposing. Its agent gives each task the best config still fails one cause. Only a cause in the config keeps the search going, and the next agent is told what was found. The other causes go to the person who started the search as an inbox notice, since fixing a task, a judge or a check is outside what a search may change. See Stalls.
Budgets
A search can start many runs: the live config on every task, each candidate on every searched task, the best on every held-out task, and the agent once per iteration. With more than one copy of each run, every one of those replays starts that many copies. A person sets one ceiling for all of it, and the search plans its spending before it starts.
A tenth of the ceiling is set aside for the agent's runs, shared across the iterations. The rest is divided into an even share per replay run, sized so that every replay the budget allows has room, with a margin of as many extra runs as the search runs at once. Each replay gets a ceiling of its own from that share: the live config's covers every run in the set, each candidate's covers the searched runs, and each held-out replay the held-out runs. If the share would be under 0.01 in the ceiling's currency, the search is refused before it starts.
Before each iteration, the search checks that what it has spent, plus what the held-out replays will need, plus one more iteration, still fits under the ceiling. If not, it stops searching and moves on to the held-out replays. So a search stops early rather than overspend, and the held-out check it ends with has its share set aside from the start.
Some spend comes back without a price, as for a harness that reports no cost, or in another currency than the ceiling's. A search cannot leave that out without letting unpriced runs spend for free, so it counts each such replay run as its full share, and an unpriced agent run as its full share of the agent's tenth. An agent run is not cut off at its share, so a search can end a little over its ceiling.
Picking is still a review
A search changes nothing on its own. When a person picks a candidate,
Spotlight edits the station's block in the line's Terraform, commits it to a
branch of its own, and opens a pull request. The change goes through the same
review and the same terraform apply as any other change to the line, and if
the line checks its pull requests,
that check replays it again. Why configuration lives in Terraform
gives the reasons the repository stays the record.