Set a station's judge and alerts
Use this to have a share of a work station's runs scored against what a good run looks like, or to set thresholds on its sent-back rate, cost and the other measures. You need the line in Terraform. See Manage Spotlight with Terraform.
Set a judge
Write the rubric, what a good run at the station looks like, in a Markdown file
such as judges/core/delivery/implementation.md. Keep it under 20,000
characters. On its first line, name the scale the judge scores on:
scores: yes-no
The run does what the task asks, and its tests cover the change.Write scores: 1-5 for a score from 1 to 5. Without the line, the judge
answers yes or no.
Add a judge block to the station's block in the spotlight_line resource:
station {
key = "implementation"
name = "Implementation"
harness = "claude-code"
# ...
judge {
rubric = file("${path.module}/judges/core/delivery/implementation.md")
sample = 0.2
trust = 0.6
}
}Set sample to the share of runs to judge, and trust to the agreement with
people the judge needs before its scores show, both from 0 to 1. Agreement is
Cohen's kappa on a yes-or-no rubric and on claims, and Spearman's ρ on a 1 to 5
rubric, so 0.6 is not 60% of runs.
Run terraform apply.
A judge is part of the station's config, so the next run starts a new group named Judge added under Runs by config on the station's view on Insights.
Judge by claims
Use claims in place of a rubric's single score when you can say what a good run does as statements the judge can check one by one.
Write 1 to 20 claims, each a statement of at most 500 characters that holds
for a good run, such as The change adds a test that fails without it.
Put them in the station's judge block as claims. Leave out rubric, or
keep it to give the judge more context:
judge {
claims = [
"The change adds a test that fails without it.",
"The change touches only what the task names.",
]
pass_share = 1
sample = 0.2
trust = 0.6
}A rubric beside claims needs no scores: line.
If a replayed run may pass without meeting every claim, set pass_share to the
share it must meet, such as 0.5. Leave it out to require every claim.
Run terraform apply.
The judge then marks each claim met or not, with its own reasons. Changing a
claim, its order or pass_share starts a new rubric version, and the judge's
agreement with people is measured again from nothing.
Set alerts
Add one alert block per threshold to the station's block. Each names a
measure, how many runs it compares in over, and exactly one of rises_by,
falls_by, above and below:
alert {
measure = "sent_back_rate"
rises_by = 0.2
over = 20
}
alert {
measure = "cost_per_run"
above = 5
over = 10
}For a rate, write shares from 0 to 1, so 0.2 is 20 percentage points. For
cost_per_run, write an amount in the workspace's currency. Give a measure at
most one alert of each kind of threshold, such as one rises_by and one
above; a second of the same kind is refused. Run terraform apply.
Check the station
Open the line's board and select the station's name. On the Overview tab, check the share judged and any claims under Judge, and select Show rubric to read the rubric. Check each alert under Alert rules; if they do not all fit, use the arrows to page through them. Only people on the team and the workspace's owners and admins can see the judge's agreement. See Judge and alerts for what each part shows.
If Terraform refuses the line, the message names the field. Judge fields and Alert fields list each message. A judge or alert on the backlog or the completed station is refused, because nothing runs there.
When an alert goes off, the task of the run that crossed it gets an inbox item that reads Station crossed an alert threshold. It reaches only people on the team and the workspace's owners and admins. See Answer the inbox.
Once the judge has scored runs, check its agreement with people. See Check a station's judge against people.