Searches

A search has an agent propose changed configs for a work station, called candidates, and replays each on a replay set's searched tasks. The best are then replayed on the set's held-out tasks. Picking a candidate opens a pull request in the line's Terraform repository. The same data is read and written through the Searches API. A search's candidates also show under Candidates on the Insights Stations tab, beside the team's other replayed candidates.

Only people on the team, the workspace's owners and admins, and workspace tokens with the read scope can read a team's searches. Anyone else is answered 404, and the web app reads Only team members can see its searches. Starting and stopping a search and picking a candidate also need plan. See Team outcomes.

Station pageCopy Anchor Link

A work station's "Improve" tab has a "Searches" card under "Try a change", beside "Replay sets", at #searches. It lists the station's searches, newest first, one row each:

"Start a search" opens the Start a search dialog. It is refused, with the reason on hover, while a search of the station is under way (A search of this station is underway.), when the station has no set (Create a replay set first.), or when no set covers runs of 5 different tasks (No replay set covers 5 different tasks yet.). While the station's sets load, it is disabled with Loading the replay sets…, and when they cannot be read, with Sorry, the replay sets couldn't be loaded. Please try again later.

With no searches, the card reads "No searches yet" and An agent tries configs on a replay set to find a better, cheaper one., with "Start a search". With no searches and no replay set, it reads Searches need a replay set to try configs on., with no action.

Start a searchCopy Anchor Link

The dialog's description reads An agent proposes configs for this station. Each config replays on most of the set's tasks. Then the best configs replay on held-out tasks to check that they hold up. The station doesn't change until you pick a config.

A count out of bounds reads Please enter a whole number from <least> to <most>. A ceiling of 0 or less reads Please enter a ceiling above 0., a ceiling over 1,000 Maximum ceiling: 1,000, and a currency that is not three capital letters Please enter a three-letter currency code, like USD.

"Aim for" is preselected until a person chooses: "The same success for less" when the newest finished replay of the live config, as it is now, over every run of the set, other than a search's, succeeded on more than 95% of it, and "More success" otherwise. This can differ from the set's room to improve, which a replay with the answer decides once there is one. Once the estimate arrives, its aim is preselected instead. Until a person chooses, such a set shows Suggested: the live config already succeeds on <share> of this set. The set has little room for more success. under the choices. The aim is sent with the search only once a person picks one or the estimate has arrived; otherwise the search chooses it as it starts.

Until a set and a valid budget are given, the estimate reads Choose a replay set and budget to see the estimated runs and spend. Then it shows what the estimate endpoint answers:

The estimate always holds out as many tasks as the search will, and assumes the choice of tasks whose runs cost most, so the search's own split costs as much or less. Its counts of runs and its allotments count every copy. Under it: An agent run may slightly overspend. The search may end a little over its ceiling. When the endpoint refuses, the estimate shows the refusal, or Please fix the marked field to see an estimate. when every problem it names is at a field.

Under the estimate, the dialog may warn, from the set's health and the line's min_gain:

Under either Success moved warning, unless "Repeats" is the suggestion already, Replay each run <n> times sets "Repeats" to the copies needed, at most 5. A warning does not stop the search from starting. "Start search", which cannot be selected until the harnesses are known, reads "Starting…" while it is sent, then opens the new search's page. A refusal is shown at its field, such as a ceiling too low to give each replay run at least 0.01 of its currency.

Search pageCopy Anchor Link

/w/<slug>/team/<teamId>/lines/<lineId>/stations/<stationId>/searches/<searchId>

To someone who may read the team's searches, an unknown search, or one of another station, reads "Search not found" and No search at this address. It may belong to another station., with "See the station's searches", which opens the station's Searches card. When the search cannot be loaded for another reason, the page reads The search could not be loaded., with "Try again".

The header shows Search <n>, or "Search" while its number is not known, and, for some endings, a pill, such as "Stalled", whose hover gives the ending in words. Under it are the replay set's name, linked to its page ("Its replay set" until the name is loaded), and how long ago the search ended, or started while it has not ended. Hovering over that shows Started <when>, ended <when>, or Started <when>. While the search is under way, "Stop" opens Stop this search?: The search agent stops and no more candidates replay. Finished replays keep their scores, and you can still pick them., with "Keep it searching" and "Stop search", which reads "Stopping…" while it is sent.

Below the header are its tiles, a "Why it stalled" card per reflection, the "Candidates" card and the "Success against cost" card.

TilesCopy Anchor Link

An ⓘ beside a tile's label shows its note on hover.

Why it stalledCopy Anchor Link

One card per reflection, newest first, under the tiles, headed Why it stalled (iteration <n>). The page opened with #why-it-stalled-<n> scrolls to it.

Each task shows its title in the set, or Task <first 8 characters of its id>, "Open the failed replay run", linked to the Run page of the replay run that failed it, the agent's evidence, and Suggested: <suggestion> when it gave one.

CandidatesCopy Anchor Link

The "Candidates" card's note reads Configs that the search agent proposed, by iteration. Pick a candidate to see its reasoning, replays and pull request. It is one table:

The first row, "Started from", holds the live config. Then each iteration has a row Iteration <n>, followed by its candidates. An iteration's row reads · Looked for why it stalled for a reflection, its state unless it ended, and "Agent run", linked to the agent run's Run page, titled Iteration <n> of Search <m>. Hovering over Iteration <n> shows <state>. The search took <n> of <m> proposals. The search agent spent <spent>.: refused proposals count among the second, a reflection has no count of proposals, and what its agent spent shows once known.

An iteration with no candidates reads Nothing proposed yet. while its agent run waits or runs, and The search agent proposed nothing replayable. after. A reflection reads This iteration proposes nothing. The search agent is looking into why the remaining tasks fail. until it ends, then This iteration proposed nothing. The search agent looked into <n> tasks that the best config still failed. Causes recorded: <n>. Both are followed by "See why it stalled", which scrolls to its card.

With no iteration yet, the card reads "No candidates yet" and The live config replays first. The search agent proposes candidates after that replay finishes. while the search is seeding, Spotlight couldn't replay the live config. The search agent never ran. once it ended seed-failed, or else The search agent proposed no candidates., with "Watch the live config's replay".

The page opened with #candidate-<candidateId> scrolls to that candidate's row. A searched candidate's "Changes" link on the Insights Candidates section opens it so, unless picking it opened a pull request.

Candidate detailsCopy Anchor Link

Selecting a candidate opens a side pane, which can be resized and closed. Its heading is the candidate's name, "On the frontier" for a frontier point, its reading when it has one, and its replay's state until that replay has finished. Under it:

• its agent's hypothesis: what the candidate should improve, and why;
• for the live config, Station as configured when the search started.; for a candidate, Changed from live: with one phrase per change, as Config words them, or Same as the live config.;
• "Searched", its score with its range and of <n> runs scored, or "Not scored yet"; "Held out", its held-out score with its range, or as in the table; and "Cost per run";
• once it has them, Gain over the live config, searched and Gain over the live config, held out, each its gain with its range, as in +12 pts (+3 to +21), its verdict, "Better", "Worse" or "within noise", and its side-by-side wins, losses and ties;
• "Replay", linked to its replay on the searched tasks ("Replay of the set" for the live config), and "Held-out replay" when it has one;
• for a candidate, what picking it does: see Picking a candidate.

Success against costCopy Anchor Link

The card's note says what the frontier joins for the search's aim: Success on the searched tasks and cost per run of each candidate. Frontier: candidates that no other candidate beats for less. The live config is always on the frontier. For the same success for less, the second sentence reads Frontier: candidates that match the live config's success, within noise, and that no other candidate matches for less. Its header has a "Chart" and "Table" switch, on "Chart" at first.

The chart draws each candidate with a search score whose searched replay has finished, and the live config once it has a search score: its search score up the side, its cost per run along the bottom. Points on the frontier are joined by a dashed line, and the live config is drawn as a diamond. The legend reads "On the frontier", "Other candidates" and "Live config". Hovering over a point shows its name, <share> succeeded · <cost> a run, and on the frontier for a frontier point. The live config is always on the frontier, so its point reads where it started · on the frontier.

When any frontier candidate is marked, a note above the chart, shown on "Chart" only, reads <n> frontier candidates gained only on searched tasks, not on held-out tasks. ("candidate" for 1). With no priced point, the chart reads No candidate has a priced score yet. Show the table to see the rest. Points not priced are listed under the chart after Hidden, not priced:, each as its name with its search score and range.

"Table" lists every candidate instead:

Search statesCopy Anchor Link

A search starts seeding: the station's live config is replayed on every run of the set, as many times as "Repeats" asks, and this replay becomes the set's baseline. Once it finishes, the search is searching, and wants its first iteration. Each iteration runs its agent once; the agent proposes up to "Candidates per iteration" candidates, each replayed on the searched runs. An iteration closes once its agent run has ended, its spend is known, and its candidates' replays have ended. Then the next iteration starts, unless the search has run its iterations, stalled, or cannot afford one more. It then holds out: holding-out while the best candidates are replayed on the held-out runs, then finished. A search with no candidate whose replay finished goes straight to finished.

Search endingsCopy Anchor Link

The pill shows in the search page's header. Hovering over it, or over the search's state on the station page, shows the ending in words.

ceiling-reached also ends a search whose replay of the live config ran over its ceiling. stalled ends a search that stalled with no room to look into why, or whose reflection found nothing the config can fix. A stopped search withdraws its agent's run and stops its replays.

Iteration statesCopy Anchor Link

Each iteration is of one kind: propose, whose agent proposes candidates, or reflect, a reflection, whose agent proposes nothing and records causes. A reflection counts toward "Iterations" like any other.

An agent run no runner takes within 60 minutes is withdrawn, and its iteration fails. An agent run whose spend is not known 15 minutes after it ended is counted as unpriced.

The search's agent runCopy Anchor Link

Each iteration's agent run is a run of its own kind:

StallsCopy Anchor Link

A propose iteration found something better when one of its candidates' replays on the searched tasks finished with a gain, paired task by task on the searched runs against the best config of the iterations before it, that the search's aim counts:

The best before is the first in the order of the aim among the live config and the candidates of earlier iterations whose replays finished. An iteration whose agent run failed, or none of whose candidates' replays finished, found nothing better.

When an iteration closes, the search ends iterations-spent once it has run every iteration. Otherwise, if its last two iterations were both propose iterations that found nothing better, it has stalled:

• With at least 2 iterations left, and room under the ceiling for what it has spent, what the held-out replays need, one more iteration and one more agent run, its next iteration is a reflection.
• Otherwise it ends stalled, and its starter gets a notice.

A reflection is not a propose iteration, so after one that went on, two more propose iterations must find nothing better before the search stalls again.

ReflectionsCopy Anchor Link

A reflection looks into the failures of the best config so far: the first in the order of the search's aim among the live config and every candidate whose searched replay finished. It looks into each searched run of the set on which that config's success, the share of its scored copies that succeeded, is below one half and at least one copy failed. A held-out task is never among them. The first failed copy is the replay run it links to.

With no such task, the reflection ends at once, without an agent run, and the search ends stalled. Otherwise its agent records one cause per task with spotlight cause. Recording a task's cause again replaces the one before. Once its agent run ends, or fails:

CausesCopy Anchor Link

Only config lets the search go on.

Reflection outcomesCopy Anchor Link

NoticesCopy Anchor Link

The person who started the search, or for a search started with a workspace token, the person who issued the token, gets an inbox notice linked to the search, "Open the search", while they are a member of the team's workspace:

<search> is The search for a better <station> config, or The search for a better station config when the station has no name to give. The first point says what happened. After it, one point per cause other than config, as <what to do>, for "<task title>": <suggestion>, or the evidence without a suggestion: "Take the task out of the replay set, or reword it", "Fix the judge's rubric", "Fix the station's check" or "Fix what the run needed around the work". A task with no title reads "one of the searched tasks". A point is cut at 1,000 characters; past 20 points, the last reads The search page shows <n> more. A reflection that went on with only config causes sends no notice. Each reflection sends at most one.

ScoresCopy Anchor Link

Results and their reasons are a replay's: see Results.

GainsCopy Anchor Link

Each candidate other than the live config reads its gain over the live config's replay twice: on the searched tasks, and on the held-out tasks once both its held-out replay and the live config's replay have finished. Each is paired task by task, with its range and a verdict of "Better", "Worse" or "within noise", as a replay's gain is, against the line's min_gain. The frontier and the best candidate do not depend on it.

Side by side with the live configCopy Anchor Link

Each candidate's replays, on the searched and on the held-out tasks, are set side by side with the search's replay of the live config once they finish, when the candidate has a judge. Each reads Side by side with the search's starting config: <n> wins, <n> losses, <n> ties, counting the tasks of that replay, once any was compared. They never change its score, its gain, the frontier or the best candidate.

Only on seen tasksCopy Anchor Link

Once a candidate's held-out gain is known, a candidate whose searched gain reads "Better" gets a reading from it:

In the "Best" tile and the candidates table, the reading is a warning icon, amber when marked, named by the reading. In the candidate's details and the frontier table, a marked candidate shows an "Only on seen tasks" pill, and the other reading its words. Hovering over or focusing the reading opens its note. A candidate whose searched gain does not read "Better" has no reading, nor does one before its held-out gain is known, nor the live config. Only marked candidates are counted above the chart and warned about when picked. A marked candidate stays on the frontier, can still be the best candidate, and can still be picked.

AimsCopy Anchor Link

A search aims for one of two things, chosen when it starts and kept for its life:

Left unset, a search aims for the same success for less when the newest finished replay of the live config, as it is now, over every run of the set, other than a search's, succeeded on more than 95% of it, and for more success otherwise.

FrontierCopy Anchor Link

The frontier is every candidate with a finished searched replay and a search score that no other candidate beats on both counts: succeeding at least as often for no more cost per run, and better on one. An unpriced cost counts as the highest. The live config is always on the frontier. When the search aims for the same success for less, only candidates whose gain on the searched tasks reads "Better" or "within noise" can be on it.

When a search stops searching, the frontier's candidates, the live config aside, are replayed on the held-out runs, as many as "Candidates per iteration", in the order of its aim. When it aims for the same success for less, only candidates that cost less per run than the live config are.

Best candidateCopy Anchor Link

The candidate first in the order of the search's aim: by held-out score and held-out gain once any candidate has a held-out score, else by search score and searched gain. When it aims for more success, that is the one that succeeds most, then the cheaper; when it aims for the same success for less, the cheapest that holds its success, read "Cheapest". When the live config comes first, the best reads "Live config".

Spend and ceilingCopy Anchor Link

A search's spend is what its replays and its agent's runs spent, in the ceiling's currency. A tenth of the ceiling is for the agent's runs; the rest gives each replay its own ceiling. A replay run whose cost was not reported or is in another currency counts as the full share a replay run is given, and an agent run whose spend is unpriced as the full share an agent run is given. A new iteration starts only while what was spent, what the held-out replays need and one more iteration fit under the ceiling. An agent run may go over its share, so a search can end a little over its ceiling.

A search's replays run up to "At once" at a time, apart from the team's own limit of 2 replay runs. See Limits and timeouts.

Picking a candidateCopy Anchor Link

A candidate is picked from its details. It can be picked once its replay on the searched tasks has finished, while the search runs or after. The live config's details have no "Pick".

While the pick is sent, the button reads "Picking…". "Retry" tries again, reading the line's source afresh.

Until its replay on the searched tasks has finished, "Pick" cannot be selected, and the reason is beside it:

The pull requestCopy Anchor Link

Spotlight reads the file named by the line's source on the repository's default branch, edits the station's block, commits to its own branch and opens the pull request against the default branch.

The paragraph on the held-out result says one of:

Any other candidate's body has no such paragraph.

The edit finds exactly one resource "spotlight_line" whose key is the line's, written as a plain string, preferring the one whose team_id refers to the team's spotlight_team, and in it exactly one station block whose key is the station's. It changes only the attributes the candidate changes:

Before it writes, the edit checks that every attribute the candidate may change, the station's check and passes_to, and its judge block hold the values the search started from, and refuses with drifted otherwise. An attribute it cannot read and the candidate does not change is listed in the pull request as not checked. An attribute read through file() with a path not built on ${path.module} cannot be followed, so changing it is refused with computed-attribute. A file() path outside the repository is refused with file-missing. The edited file is read back, and refused with edit-failed if it does not parse or does not hold the candidate's values.

GitHub refusing a request with a 4xx status, other than for a rate limit, refuses the pull request at once with github-refused. An attempt that fails for another reason, such as GitHub not answering or a rate limit, is tried again after 30 seconds, 2 minutes, 8 minutes and 32 minutes, and then refused with github-refused.

Config pull request reasonsCopy Anchor Link

Each comes with Spotlight's message, such as Terraform computes `prompt` from `local.build_prompt`. Please change it by hand.. block-not-found also covers a station no longer on its line, and file-missing a file a file() reads that the repository does not hold. file-unwritable is a file that is a symbolic link in the repository. branch-taken means the branch holds commits Spotlight did not write.

Guarding a station's changesCopy Anchor Link

A replay set's page has a "Guards config changes" switch, at the bottom left of its "Checks" panel. When it is on, spotlight replay check replays a planned change of the station on this set. A station has at most one such set: turning it on while another set guards the station is refused. The switch cannot be changed once the set's station has left its line. Turning it off leaves checks already started running.

The "Checks" panel's ⓘ reads spotlight replay check replays a pull request's change of this station on this set. One guarding set per station., then the line's replay_check, read-only: Fails past <n> regressions. Ceiling per check: <ceiling> · Minimum real gain: <min_gain>, or, for a line without one, Line sets no replay_check. Its pull requests aren't checked.

The panel lists the checks replayed on the set, as Change, Verdict, Regressions and When. Change is the pull request (Pull request #<n>) or commit (Commit <sha>) that planned the change, or "A planned change", with attempt <n> from the second attempt on; the row opens the check's replay. Verdict is the check's verdict in words, such as Undecided: replay reached its ceiling. With none, it reads "No checks yet" and This set checks pull requests that change this station., or The set doesn't guard config changes. when the switch is off. "Show older checks" loads more.

Search replaysCopy Anchor Link

A search's replays are replays of its set, and each replay page and replay run reads as any other replay. A candidate's replay covers only the searched runs, and a held-out replay only the held-out runs. The replay page's header, and the Run page of each of its replay runs, read candidate <name> in its search, or the live config it started from in its search, linked to the search.

While a search is under way, its set takes no runs, gives none up, and cannot be deleted. The set's page disables "Add runs" and the X beside each run, with A search of this set is running. Add or take out runs after the search ends or a person stops it. on hover, and "Delete", with A replay or search of this set is running. Please delete the set after it ends. on hover.