spotlight replay
spotlight replay proposes replay candidates
and starts, follows and stops replays from any
computer, through the resource API and the Replays API.
spotlight replay check checks the station changes in a Terraform plan,
through the Replay checks API.
It is part of the spotlight binary; see
The spotlight command.
Synopsis
spotlight replay propose --station <team/line/station> --name <name> <station.json>
spotlight replay candidates --station <team/line/station>
spotlight replay show --station <team/line/station> <candidate name or id>
spotlight replay sets --station <team/line/station>
spotlight replay start --station <team/line/station> --set <name or id> --candidate <name, id or live> --ceiling <amount> [--currency USD] [--repeats 1]
spotlight replay status --team <key> <replay id>
spotlight replay watch --team <key> <replay id> [--every 15s]
spotlight replay stop --team <key> <replay id>
spotlight replay check [--json] [--timeout 60m] [--every 15s] [--commit <sha>] [--pull-request <n>] [--repeats 1] <plan.json | ->Every subcommand also takes --json. Flags may come before or after the
argument.
Install
spotlight replay needs only the spotlight binary, not a host. The
install script sets up a host, so
download the binary from the release instead:
<os> | <arch> |
|---|---|
linux or darwin | amd64 or arm64 |
curl -fsSL -o spotlight https://github.com/usespotlightai/spotlight/releases/latest/download/spotlight-<os>-<arch>
curl -fsSL -o spotlight.sha256 https://github.com/usespotlightai/spotlight/releases/latest/download/spotlight-<os>-<arch>.sha256Compare the file's SHA-256 with spotlight.sha256, then make it executable
with chmod +x spotlight. For a given release, replace latest/download with
download/<version>.
Environment
| Variable | Description |
|---|---|
SPOTLIGHT_URL | Required. The Spotlight address, as https://app.usespotlight.ai. |
SPOTLIGHT_API_TOKEN | Required. A workspace token. read is enough for candidates, show, sets, status and watch; propose, start and stop need plan. |
Without either, every subcommand but check fails with
set SPOTLIGHT_URL and SPOTLIGHT_API_TOKEN to replay, followed by the usage.
check reads more; see Environment for check.
The token must see exactly one workspace; otherwise the command fails with
the token sees <n> workspaces; use a workspace token, which acts in exactly one.
Common behaviour
| Value | |
|---|---|
| Request timeout | 2 minutes |
| Client header | Spotlight-Client: cli/<release> |
| An error | spotlight replay: <error> on standard error. A refusal from Spotlight is printed as its detail, or its title, followed by each field error as <path>: <message> on a line of its own. |
--json | Prints Spotlight's answer as indented JSON in place of the text output. |
--station names a station as <team key>/<line key>/<station key>, as in
core/delivery/implementation. Another form fails with
--station is a team, line and station key, as in core/delivery/build, not "<value>",
and a line without that station with
line <team>/<line> has no station <key>; it has <keys>.
--team names a team by its key. A set or candidate is named by its id or its
exact name. A name no set or candidate has fails with
<station> has no <kind> "<name>"; it has <names>, and a name more than one
has with <station> has <n> of <kind> named "<name>"; name one by its id: <ids>.
Exit codes
| Code | Meaning |
|---|---|
0 | Done. For status and watch, the candidate regressed on no task. |
1 | A usage error, a refusal, or Spotlight could not be reached. |
2 | status or watch only: the candidate regressed against its counterpart on at least one task. The message reads the candidate regressed against its counterpart on <n> of <m> tasks. |
check has exit codes of its own; see
spotlight replay check.
spotlight replay propose
spotlight replay propose --station <team/line/station> --name <name> <station.json>Proposes a candidate for the station from a file holding its station block as
JSON, in the shape of one entry of a line's stations. Give - to read the
block from standard input. Nothing is applied.
| Flag | Required | Description |
|---|---|---|
--station | yes | The station. |
--name | yes | The candidate's name. See Fields. |
The request carries a new Idempotency-Key. If Spotlight cannot be reached or
answers with a 5xx status, the request is sent once more with the same key,
so the retry never proposes the candidate twice.
Proposed core/delivery/implementation/Higher effort (<id>); nothing is applied.
What it changes from the live config <fingerprint>:
effort high → xhighThe fingerprint is shown to 12 characters. A candidate that changes nothing
prints It changes nothing from the live config <fingerprint>.
| Message | Cause |
|---|---|
--name tells the candidate from the station's others | No --name, or a blank one. |
name one file holding the station block, as the line resource holds a station, or - to read it from stdin | No file, or more than one. |
<path> is HCL; Spotlight writes HCL but never reads it, so give the station block as JSON, as the line resource holds it | The file ends in .tf or .hcl. |
<path> holds no JSON object; it holds the station block, as the line resource holds a station | The file is not a JSON object. |
propose <name>: <refusal> | Spotlight refused the candidate. See Checks. |
spotlight replay candidates
spotlight replay candidates --station <team/line/station>Lists the station's proposed candidates as a table with the columns NAME,
CHANGES (how many), PROPOSED and ID. With none, it prints
<station> has no candidates.
spotlight replay show
spotlight replay show --station <team/line/station> <candidate name or id>Prints one candidate: its address and when it was proposed, its id, its
fingerprint, and what it changes from the live config, as propose does.
spotlight replay sets
spotlight replay sets --station <team/line/station>Lists the station's replay sets as a table with the columns NAME, RUNS,
REPLAYABLE (runs that can still be replayed), CREATED and ID. With none,
it prints <station> has no replay sets; pick one on the station's page.
Sets are opened and filled on the Station page.
spotlight replay start
spotlight replay start --station <team/line/station> --set <name or id> --candidate <name, id or live> --ceiling <amount> [--currency USD] [--repeats 1]Starts a replay of the set under the candidate.
| Flag | Required | Default | Description |
|---|---|---|---|
--station | yes | The station. | |
--set | yes | The replay set. | |
--candidate | yes | A proposed candidate, or live for the station as it is configured now. | |
--ceiling | yes | The most the replay may spend, a number above 0. Spotlight takes at most 1,000. | |
--currency | no | USD | The ceiling's currency, as a three-letter code. |
--repeats | no | 1 | How many times to replay each run, 1 to 5. The ceiling covers every copy. |
Started replay <id> of <set> under <candidate>, spending at most <ceiling>.
It replays each run <n> times, to measure how much its results move by chance.
Its baseline, the live config, replays beside it as <id>, with a ceiling of its own.
Follow it with: spotlight replay watch --team <team> <id>The second line appears only with --repeats above 1, and the third only when
a baseline starts beside the replay.
| Message | Cause |
|---|---|
--set and --candidate name what to replay, and under what | --set or --candidate is missing. |
--ceiling is the most the replay may spend, as a number above 0, not "<value>" | --ceiling is missing, not a number, or 0 or less. |
--repeats is how many times to replay each run, from 1 to 5, not <n> | --repeats is below 1 or above 5. |
start a replay of <set>: <refusal> | Spotlight refused to start it. See Errors. |
spotlight replay status
spotlight replay status --team <key> <replay id>Prints the replay as it stands, and the replay it is compared with, its
counterpart, and exits 2 if it
regressed on any task.
Replay <id> under <candidate> (<kind>): <state>, <spent> of <ceiling> spent
It replays each run <n> times.
Succeeded: <n> of <scored> (<share>%); <share>% (<low>–<high>%) over <n> of <m> tasks, <n> unscored
Compared with the baseline, <state>: <n> of <scored> (<share>%); <share>% (<low>–<high>%) over <n> of <m> tasks, <n> unscored
On the tasks both scored: <share>% (<low>–<high>%)
Within noise of the baseline: +4 points (-8 to +16) over 12 tasks, not clearly past the 5-point gain that counts as realWhen runs already running took the replay past its ceiling, the first line
ends with , over the ceiling by <amount>, spent by runs already running when it was reached. <kind> is the candidate's kind: proposed, live for a replay
of the live config, searched for a search's candidate, planned for a
replay check's planned change, or oracle
for a replay with the answer.
A replay whose candidate is withdrawn reads under a candidate since withdrawn.
The second line appears only when the replay replays each run more than once.
On the Succeeded and Compared with lines, <n> of <scored> counts copies,
and the success with its 95% range
follows, over the tasks with a scored copy of the tasks replayed.
The Compared with line names the counterpart:
| Counterpart | Named |
|---|---|
| The set's baseline now | the baseline |
Another, for an oracle replay | the live config |
Another, for a searched replay | the search's starting config |
| Another, for any other replay | an earlier baseline |
The On the tasks both scored line gives the counterpart's success on the
tasks both replays scored, the tasks the gain pairs, with its range. It
appears only when that success is known and differs from the one on the
Compared with line.
For the set's baseline, the Compared with lines read
It is the set's baseline. A replay compared with nothing reads
It is compared with nothing: <reason>.:
| Reason | Cause |
|---|---|
the live replay it was compared with is gone | An oracle replay whose live replay is gone. |
the set has no baseline | Any other replay, while the set has no baseline. |
the replay it was set beside is gone | Any other replay, while the set has a baseline: its counterpart is gone. |
When there is a gain, the line after the Compared with lines reads the
gain over the counterpart, in points, with
its range and the tasks both scored:
| Verdict | Line |
|---|---|
better | Better than <counterpart>: <gain> points (<low> to <high>) over <n> tasks, past the <min_gain>-point gain that counts as real |
worse | Worse than <counterpart>: <gain> points (<low> to <high>) over <n> tasks |
within-noise | Within noise of <counterpart>: <gain> points (<low> to <high>) over <n> tasks, not clearly past the <min_gain>-point gain that counts as real |
<counterpart> is named as on the Compared with line.
A table follows with one row per task, regressions first: a REGRESSED mark,
TASK, CANDIDATE and COMPARED WITH (each a result: pending, succeeded,
failed or unscored, or none; with copies, each copy's result joined by
/, as in succeeded/failed/succeeded), COST Δ, STEPS Δ and SCORE Δ,
each averaged over the copies. A difference not known reads -.
With --json, it prints { "replay", "comparedWith" }, each as the
Replays API reads a replay. comparedWith is
the counterpart, or null without one.
If the counterpart cannot be read, the command fails with
read the replay it is compared with, <id>: <error>.
spotlight replay watch
spotlight replay watch --team <key> <replay id> [--every 15s]Reads the replay every --every until both it and its counterpart have ended,
then prints what status prints and exits as status does. A replay compared
with nothing is waited for alone. Until then it prints one line each time:
<state>: <n> of <m> settled, <spent> spentWith --json, it prints only the final answer.
| Flag | Default | Description |
|---|---|---|
--team | required | The team's key. |
--every | 15s | How often to read the replay, as a Go duration such as 30s or 1m. At least 1s. |
A read that fails because Spotlight could not be reached, or answered with a
5xx status, is tried again. The command prints
could not ask (<error>); asking again, unless --json is given, and waits
twice --every after the first failure in a row, doubling after each, at most
2 minutes. After 5 failures in a row it ends with
gave up after 5 failed tries in a row: <error>. Any other refusal ends the
command at once. An --every under 1s fails with
--every is at least 1s, not <value>.
spotlight replay stop
spotlight replay stop --team <key> <replay id>Stops the replay: nothing more of it starts, and its running runs are
stopped. It then reads the replay and prints
Stopped replay <id>, now <state>: nothing more of it starts, and its running runs stop.
With --json, it prints the replay as read after stopping, as the
Replays API reads one. A replay that has already
ended is refused with 409.
spotlight replay check
spotlight replay check [--json] [--timeout 60m] [--every 15s] [--commit <sha>] [--pull-request <n>] [--repeats 1] <plan.json | ->Checks each work station a saved Terraform plan changes against the replay set
that guards the station's changes, under the live line's
replay_check, and waits
for the verdicts. Give it the output of terraform show -json <planfile>, or
- to read it from standard input. Spotlight never runs Terraform. See
Check Terraform pull requests for station regressions.
| Flag | Default | Description |
|---|---|---|
--json | Prints the report as JSON in place of the text. | |
--timeout | 60m | How long to wait for the checks, as a Go duration, above 0. A check still running then keeps running in Spotlight. |
--every | 15s | How often to read the checks, as a Go duration, at least 1s. |
--commit | none | The commit that plans the change, recorded on each check. |
--pull-request | none | The pull request that plans the change, recorded on each check and used to name its candidate. |
--repeats | 1 | How many times each check replays each run of the set, 1 to 5. The line's ceiling covers every copy. |
Environment for check
| Variable | Description |
|---|---|
SPOTLIGHT_URL | Required. The Spotlight address. |
SPOTLIGHT_API_TOKEN | A workspace token with plan. Used in place of a trust when both are set. |
SPOTLIGHT_OIDC_TRUST | The id of a workspace token trust that grants plan. The command exchanges the job's OIDC token through it for a short-lived workspace token, exchanges again before that token expires, and once more on a 401. |
SPOTLIGHT_OIDC_TOKEN | The OIDC token to exchange. Without it, the command asks GitHub Actions for one, through ACTIONS_ID_TOKEN_REQUEST_URL and ACTIONS_ID_TOKEN_REQUEST_TOKEN, which need permissions: id-token: write. |
SPOTLIGHT_OIDC_AUDIENCE | The audience to ask the OIDC token for. Default spotlight. Must equal the trust's audience. |
GITHUB_STEP_SUMMARY | When set, a Markdown summary is appended to the file it names. |
Credentials are read only when the plan has a station to check.
What it checks
The command reads every managed spotlight_line of the provider
usespotlightai/spotlight that the plan updates, and compares each station's
block before and after. A work station whose block changes, or whose line's
repositories change, is checked: its planned block, and the planned
repositories with their image and setup, are sent to Spotlight. Each is
named by the line's address and the station's key, as in
core/delivery/build.
Some changes are not checked, and are printed as notes:
| Note | Cause |
|---|---|
<resource> is a new line, with no live stations to check a change against | The plan creates the line. |
<resource> is replaced, so its stations start anew, with no live config to check a change against | The plan replaces the line. Each of its work stations is also reported UNDECIDED. |
<resource> is removed, so there is nothing of it to check | The plan deletes the line. |
<resource> has stations known only after apply, so none is checked | The line's stations are unknown until apply. |
<resource> removes station <key>, which leaves nothing to replay | A station is removed. |
<resource> adds station <key>, with no live config to check it against | A station is added. If the plan also removes a work station of the line, an added work station is also reported UNDECIDED. |
<resource> has a station whose key is known only after apply, so it is not checked | A station's key is unknown until apply. |
<resource> changes station <key>, where nothing runs, so there is nothing to replay | A backlog or completed station changes. |
A station is reported UNDECIDED without being checked when:
| Message | Cause |
|---|---|
not checked, since the plan replaces its line, so it starts with no live config to check it against | The plan replaces its line. |
not checked, since the plan removes <keys> and adds this station, which may be a station renamed; a renamed station starts with no live config to check it against | The plan removes a work station of the line and adds this one. A changed station key is planned this way. |
not checked, since Terraform knows its <attributes> only after apply | Part of the station, or the line's repositories, is known only after apply. |
not checked, since the plan also changes the <resources> its runs depend on, which a check cannot replay before apply | The plan also changes a spotlight_skill it names, the spotlight_harness_profile it names, a spotlight_integration whose tools it uses, a spotlight_hook that matches it, or any spotlight_secret. A changed secret makes every work station of the lines in the plan undecided. |
Each check is started with an Idempotency-Key made from the line, the
station, its planned block, the planned repositories, --commit,
--pull-request and --repeats, so running the job again on the same plan answers the same
check. Starting and reading are tried again when Spotlight cannot be reached or
answers with a 5xx status, waiting twice --every after the first failure
and doubling, at most 2 minutes, and given up after 5 failures in a row.
Verdicts
Each station ends with one status:
| Status | Meaning |
|---|---|
PASSED | The change regressed on no more tasks than the line allows, and enough tasks were scored. |
FAILED | It regressed on more tasks than the line allows. |
UNDECIDED | The check could not decide, or the station was not checked. |
RUNNING | The check was still running at --timeout: still running after <timeout>; it keeps running, so follow it at its link. |
UNCHANGED | The plan changes nothing that changes how the station's runs go, so there is nothing to replay. |
REFUSED | Spotlight refused the check, or reading it failed. The message says why. See Errors. |
An undecided check prints why:
undecidedBecause | Printed after undecided: |
|---|---|
stopped | the replay was stopped |
over-ceiling | the replay reached its ceiling before every task ran |
baseline-unfinished | the baseline replay of the live config did not finish |
too-few-scored | fewer than half the tasks were scored on both sides |
replay-gone | the replay is gone |
no-baseline | no baseline replay of the live config the check started against is left, and none can come, since the live config or its set moved on |
Output
note: <note>
core/delivery/build: FAILED
1 of 6 tasks regressed (at most 0 may), replaying set "Queue fixes": 6 tasks, 6 scored on both sides
candidate 4 of 6 succeeded, 67% (33–100%) over 6 tasks, baseline 5 of 6 succeeded, 83% (50–100%) over 6 tasks
Within noise of the baseline: -17 points (-50 to +0) over 6 tasks, not clearly past the 5-point gain that counts as real
spent 3.2 USD of a 10 USD ceiling
scored with the live check, judge, which the plan also changes
REGRESSED <task id> "<title>": failed where the baseline succeeded, check-failed: npm test
<link to the check's replay>With --repeats above 1, a line each run replayed <n> times comes before
the candidate line. The candidate and baseline line counts copies, then gives each
side's success with its range over the tasks scored, and the next line the
candidate's gain over the baseline, worded as for
status. The gain does not decide the status.
REGRESSED lines give each copy's result joined by /.
A plan that changes no live station prints
The plan changes no live station of a line, so there is nothing to check.
With --json, it prints { "stations", "notes", "exitCode" }, each station as
{ station, resource, status, message, url, check }, check being the
check as Spotlight read it last.
The step summary is headed Spotlight replay check, with any notes, then a
table with the columns Station, Verdict, Regressed, Candidate, Baseline, Gain,
Spent and Replay, then, for each station with a message, a gain or
regressions, the message, the gain line and each regressed task. A gain reads
as in -17 points (-50 to +0), within noise, or - without one.
Exit codes for check
| Code | Meaning |
|---|---|
0 | Every station PASSED or UNCHANGED, or there was nothing to check. |
1 | A usage error, a plan it cannot read, missing credentials, or any station REFUSED. |
2 | No station REFUSED, and at least one FAILED. |
3 | No station REFUSED or FAILED, and at least one UNDECIDED or RUNNING. |
| Message | Cause |
|---|---|
name the plan, the output of `terraform show -json <planfile>`, or - to read it from stdin | No plan, or more than one. The usage follows. |
this is the log `terraform plan -json` streams, which holds no station blocks; save the plan with `terraform plan -out=<planfile>`, then give this the output of `terraform show -json <planfile>` | The file is the streamed log of terraform plan -json. |
the plan is not JSON; give this the output of `terraform show -json <planfile>` | The file is not JSON. |
this is not a saved plan shown as JSON; give this the output of `terraform show -json <planfile>` | The JSON has no format_version. |
set SPOTLIGHT_URL, and SPOTLIGHT_API_TOKEN or SPOTLIGHT_OIDC_TRUST, to check a plan | No SPOTLIGHT_URL. The usage follows. |
set SPOTLIGHT_API_TOKEN, a workspace token that may plan, or SPOTLIGHT_OIDC_TRUST, a workspace token trust to exchange the job's OIDC token through | Neither credential. The usage follows. |
SPOTLIGHT_OIDC_TRUST needs an OIDC token: set SPOTLIGHT_OIDC_TOKEN, or run in GitHub Actions with permissions: id-token: write | A trust, but no OIDC token to exchange. |
the workspace token trust <trust> did not admit the OIDC token: <refusal> | The trust refused the exchange. |
--timeout is how long to wait, above 0, not <value> | --timeout is 0 or less. |
--every is at least 1s, not <value> | --every is under 1s. |
--repeats is how many times to replay each run, from 1 to 5, not <n> | --repeats is below 1 or above 5. |
Each is printed after spotlight replay check: on standard error.