Check a station's judge against people
Use this once a work station has a judge, to get its scores shown or to find out why they are not. To give a station a judge, see Set a station's judge and alerts.
Only people on the team and the workspace's owners and admins can grade runs and see the judge's agreement.
Grade judged runs
Some judged runs reach the inbox as A judged run needs your grade. Grades check the judge. They reach the task's assignee, or everyone on the team when the task has no assignee. Open one. It shows the run's conversation, then a card with the question and, when the judge has a rubric, the rubric itself.
Under Does this run meet the rubric?, select Yes or No. On a 1 to 5 rubric, the question reads How well does this run meet the rubric, from 1 to 5?: select a number. Without a rubric, the questions read Is this a good run? and How good is this run, from 1 to 5?. Your grade saves as soon as you choose. To change it, select another answer.
On a judge with claims, the question reads Does the run meet each claim?. Select Yes or No beside every claim. Your grade saves once every claim is answered. To change it, change any answer.
Nothing waits on your grade.
Read the judge's agreement with people
Open the line's board and select the station's name. On the Overview tab, under Judge, select How it agrees with people. The station's view on Insights opens at How far to trust the judge.
If the judge has judged runs under more than one rubric version, pick the one to read under Rubric.
Read the first tile, which says whether the judge is trusted:
- If it reads Hidden until more runs have scores from both the judge and people, grade more of the station's runs. At least 10 need both scores.
- If it reads Hidden. Agreement can't be measured while all scores are identical., grade more runs until both scores vary.
- If it reads Hidden. followed by the measure and Minimum required, follow Change the rubric or the trust.
- If it reads Hidden. The rubric names a scale that cannot be read.,
correct the rubric's first line to
scores: yes-noorscores: 1-5.
On a judge with claims, look under By claim. If one claim's kappa is far below the others, reword that claim.
Agreement says what each tile, statistic and card shows.
Change the rubric or the trust
Under Score comparison, compare the judge's scores, in rows, with people's, in columns. If the counts off the diagonal lean one way, the judge is stricter or looser than people:
- To make the judge read the station as people do, edit the rubric file, such
as
judges/core/delivery/implementation.md, or reword the claim people disagree with it on, and runterraform apply. The new rubric version starts with no runs that have both scores. Once the judge has judged runs under it, it is the Current version under Rubric. - To show the scores at a lower agreement, lower
trustin the station'sjudgeblock and runterraform apply.