Check Terraform pull requests for station regressions

Use this when your team changes its lines through pull requests to a Terraform repository, and you want each pull request that changes a work station to be tried on the station's past tasks before it merges. A CI job runs spotlight replay check on the saved plan. For each work station the plan changes, Spotlight replays a set of the station's past tasks under the planned change, compares the results with the station as it is now, and the job fails if the change did worse on more tasks than the line allows. You need:

• The line's Terraform planned in GitHub Actions. The workflow below builds on Run Terraform from GitHub Actions without a stored token.
• For each station to check, a replay set of its past runs.
• A host serving the workspace with the station's harness. Checks run on the team's runners, after task runs.
• A way for the job to call Spotlight with the plan scope: a workspace token trust the job's OIDC token is exchanged through, or a workspace token.

Choose the tasks each check replaysCopy Anchor Link

Each station has at most one replay set that guards its changes. A check of the station replays that set.

While a check is running, it keeps replaying the set as it was when the check started. A later change to the set makes the next check start afresh.

Set the threshold on the lineCopy Anchor Link

Add a replay_check block to the line's spotlight_line resource, and apply it:

resource "spotlight_line" "core_delivery" {
  # ...

  replay_check {
    most_regressions = 0
    ceiling          = 10
    currency         = "USD"
    min_gain         = 0.05
  }
}

• most_regressions: how many tasks a change may regress on and still pass. A task regresses when the live config succeeded on it and the change did not, or the station's judge scored the change lower.
• ceiling and currency: the most each replay a check starts may spend. A check always replays the change, and also replays the live config when the set has no baseline for it yet, so one check spends at most twice the ceiling, and a little over if runs are still going when it is reached. A check that reaches its ceiling before it fails is undecided. See Ceiling.
• min_gain: the smallest gain in success that counts as more than noise, as a share above 0 and at most 1, 0.05 unless set. The output says whether the change's gain over the live config reaches it. It does not decide whether the check passes. The line's replays and searches read their gains against it too.

A line without replay_check is not checked: the job fails for its stations and says so. Once it is applied, hover over the ⓘ beside Checks on the set's page to read the line's threshold. The threshold always comes from the line as it is applied, never from the plan, so a pull request cannot loosen its own check. In the same way, a change to a station's check, judge, diffs or runners is replayed with the live station's, and the output says which it kept. See replay_check.

Let the job call SpotlightCopy Anchor Link

Give the job SPOTLIGHT_URL as well: https://app.usespotlight.ai, or your deployment's address on a self-hosted Spotlight.

Add the check to the workflowCopy Anchor Link

spotlight replay check reads a saved plan as terraform show -json prints it. It has no install of its own apart from a host, so the job downloads the release binary.

Replace the plan job of .github/workflows/terraform.yml with:

  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    env:
      SPOTLIGHT_OIDC_TRUST: ${{ vars.SPOTLIGHT_PLAN_TRUST }}
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_wrapper: false
      - run: terraform init -input=false
      - run: terraform plan -input=false -out=tfplan
      - run: terraform show -json tfplan > plan.json
      - name: Download spotlight
        run: |
          base=https://github.com/usespotlightai/spotlight/releases/latest/download
          curl -fsSL -o spotlight "$base/spotlight-linux-amd64"
          curl -fsSL -o spotlight.sha256 "$base/spotlight-linux-amd64.sha256"
          echo "$(cut -d' ' -f1 spotlight.sha256)  spotlight" | sha256sum -c -
          chmod +x spotlight
      - run: >-
          ./spotlight replay check
          --pull-request ${{ github.event.pull_request.number }}
          --commit ${{ github.event.pull_request.head.sha }}
          plan.json

terraform_wrapper: false keeps the JSON free of the wrapper's own output. Give the plan from terraform show -json: the log that terraform plan -json streams holds no station blocks, and the command refuses it. To pin a release, replace latest/download with download/<version>. See Install.

To replay each run of the set more than once, add --repeats and a number up to 5. Each copy is a run, and the line's ceiling covers them all, so raise it to match. See Copies of the same run.

The command waits for the checks, up to 60 minutes unless you pass --timeout. A check still running then keeps running in Spotlight, and the job ends with exit code 3 and the check's link.

Read the resultCopy Anchor Link

The job's summary gets a table with one row per station: its verdict, how many tasks regressed, how the change and the live config did, each with its range, the change's gain over the live config with better, worse or within noise, what the check spent, and a link to its replay in Spotlight. Each regressed task is listed under it, with why. The log prints the same.

Some changes are not checked, and the output notes why: a new line, a line that is removed, a station that is added or removed, or a station where nothing runs. A station is undecided, exit code 3, when its line is replaced, when the plan removes a work station and adds another (which may be a renamed station), when Terraform knows part of it only after apply, or when the plan also changes a skill, harness profile, integration, hook or secret it depends on. A changed secret makes every work station undecided.

Running the check again on the same change, against the same live config, the same runs of the set, the same threshold and the same scoring, answers with the earlier check while it is running, passed or failed. Each check also shows on the guarding set's page, under Checks. Verdicts lists every verdict and why a check can be undecided.

RelatedCopy Anchor Link

• spotlight replay check
• Replay checks API
• replay_check
• Replay past tasks against a change to a station
• Search for a better station config