Insights
The Insights page sums up how a team's runs, tasks and pull requests went over
a period, against the period before it. It is "Insights" in the team's
sidebar, at /w/<slug>/team/<teamId>/insights. The same figures are read
through the Insights API, and the Candidates
section of the Stations tab through the
Replayed candidates API.
Only people on the team, the workspace's owners and admins, and workspace
tokens with the read scope can read Insights, on every tab. Anyone else is
answered 404, and the page reads "Only team members can see Insights". See
Team outcomes.
The tabs, in order:
| Tab | ?tab= |
|---|---|
| Overview | overview, the default |
| Cost | cost |
| People | people |
| Productivity | productivity |
| Stations | stations |
| Tools and skills | tools |
| Runs | runs |
| Traces | traces |
Period
The period picker, at the top of the page, offers "Past 7 days", "Past 30
days" and "Past 90 days", and starts at 30 days. A period is whole UTC days,
up to and including today. The period before is as long and ends where the
period starts. The page's address keeps the tab and the period as
?tab=<tab>&period=<7d|30d|90d>, leaving out either while it is the default.
Changing the period keeps the tab and its filters.
A figure with a change against the period before shows it beside the figure, with "Against the previous period" on hover:
| Change | Reads |
|---|---|
| A count, time or amount | A whole percent, as in +12% or -8%. |
| A rate | Points, as in +5 pts, with one decimal under 1 point, as in +0.4 pts. |
| None | No change. |
| From nothing | Up from 0. |
An arrow beside a change is green when the change is good and red when it is bad. A change in "Spend" and "Cost", in a station's runs, and in a station's tasks is neither. A change in spend is shown only when both periods are priced in the same currency.
Each tab's figures sit in cards. An ⓘ beside a card's title describes it on hover, and a card with a table under its chart has a "Chart" and "Table" switch. On People, Productivity, Stations and Tools and skills, the figures are grouped in sections whose heading folds them away.
What is counted
A run is counted in a period when it ended in it. Except on the Runs tab, only runs that ran are counted: every run except those withdrawn before they started. Only task runs at a station are recorded; chat runs are not.
| Term | Meaning |
|---|---|
| p50 | The median. With an even count, the rounded mean of the two middle values. |
| p75, p90, p95 | The nearest-rank 75th, 90th and 95th percentiles. |
| Next act | The first thing a person did with the task after the run: the act names the run, or else it follows the task's last run at that station. It is one of Started, Parked, Closed, Answered and Sent back. Of acts at the same moment, the first of Sent back, Closed, Answered, Parked and Started counts. Approving or refusing an approval is never a next act. |
| Accepted | A run whose next act was Closed. |
| Sent back | A run whose next act was Sent back. |
| Person act | A person moving a task, answering, approving, refusing or sending a run back. |
| Activity | A person opening a task, acting on one, or starting an agent chat on the team. |
Outcomes are recorded as runs end and as people act. Nothing that happened before recording began is counted. A deleted task's runs, pull request and acts are removed.
The figures on every tab except Productivity's pull request figures, Stations'
Candidates, and Runs are worked out from the runs that ended in the period
and the period before, or the last 30 days where that reaches further: at most
the 20,000 of them that started latest. The team's task pull requests, tasks people acted on, tasks
with failures and agent chats over that time are each counted up to 20,000
too. When any of them was more, Overview, Cost, People, "Where work waits",
"Runs at each station", "Config changes", "Skills and tools" and "Memory" each
show Only the latest 20,000 runs over this period and the one before are counted.
A station that no longer exists shows as "A removed station", a person without an account as "A former member", and a task no person opened as "No one".
With nothing to count in the period, these read "No runs in this period", with "Create a task", which opens the team's first line, and a line of their own:
| Tab or section | Line |
|---|---|
| Overview | Cost and delivery show here once stations run tasks. |
| Cost | Cost by harness, model, station and repository shows here once stations run tasks. |
| People | Load on people shows here once they open tasks, answer the inbox or chat with the agent. |
| Where work waits | Where tasks wait shows here once they move through the stations. |
| Runs at each station | Each station's runs show here once stations run tasks. |
| Config changes | Config changes and results by harness, model and effort show here once stations run tasks. |
| Skills and tools | Skills read and MCP servers called show here once stations run tasks. |
| Runs | Runs that end in this period show here, newest first. |
A part that could not be read reads <part> could not be loaded., with
"Try again".
Figures open the Runs tab, narrowed by the pills listed under Runs, or narrow the Productivity tab's pull request list:
| Selecting | Opens |
|---|---|
| A tile's figure | The Runs, Cost or People tab, or the pull request list, narrowed to what the tile counts where it can. |
A day's "Runs" in "Output against cost", <n> runs and Runs with config on Overview, a cost slice, a person's name in "What each person did", "Tasks" in "Where tasks wait", "Runs" and "Failed" in "By station", a failure reason, a skill's week, "Given", "Read" or station, an MCP server's counts, and a refused tool's count | The Runs tab, narrowed to those runs. |
| A count in "Spotlight's share", "Week by week", "By repository", "Spotlight against people", "What Spotlight ships" and "Review depth" | The pull request list, narrowed to those pull requests. |
Overview
| Figure | Counts | A rise is |
|---|---|---|
| Spend | The cost of the runs, when every priced run is in one currency, with Over <n> runs. Otherwise it reads –, with No priced runs or Prices in more than one currency. | neither |
| PRs merged | Task pull requests merged in the period, with <n> closed unmerged. | good |
| Merge rate | Merged, of the task pull requests merged or closed unmerged in the period. Its change is in points. | good |
| Cost per merged PR | Spend divided by PRs merged. | bad |
| Task to merge | For pull requests merged in the period, the p50 time from the task opening to the merge, and the p90. | bad |
| Human touches per task | Person acts in the period on the tasks the period's runs worked, per task, with <acts> over <tasks> tasks. | bad |
| Active people | People with an activity in the period. | good |
"Spend" and "PRs merged" each draw their figure day by day under it. A tile's figure opens the Runs, Cost or People tab, narrowed to what it counts where it can be.
"Output against cost" charts spend and merged pull requests per UTC day. Spend
is charted only when every priced run in the period is in one currency;
otherwise the card's description reads
Merged pull requests per day. Spend not shown. <reason>., with the reason
No priced runs or Prices in more than one currency. Hovering over a day
shows <spend> spent over <n> runs, <n> PRs merged and
Config changed at <station> for each config change that day. Its table lists
"Day", "Runs", "Spend", "PRs merged" and "Config changes".
A diamond under a day, and a dashed line across it, marks a config change: a
station's run that started with a config fingerprint other than that of the
station's run started before it. Runs are taken in the order they started,
and the diamond is dated from when the run started. A station's first config
is not marked, and a return to an earlier config is marked like any other
change. A run that never started marks nothing. Hovering over a diamond shows
Config changed on <day> and, for each station, its name, linked to the
station, "Impact" (see Impact), and
Runs with config <fingerprint>, the first 8 characters, which opens the Runs
tab at the station's runs with that config.
"Stations worth improving" lists stations with at least 3 runs in the period where a signal is worse than the station's own in the period before, or worse than all stations' together in the period:
| Signal | Rate |
|---|---|
| Sent back | Runs whose next act was Sent back, of runs. |
| Checks failed | Failed checks, of passed and failed checks. |
| Run failures | Failed runs, of runs. |
| Task failures per run | Task failures raised at the station, per run. |
| Cost per run | Spend, per run. Not compared across currencies. |
A signal is worse when it is more than 1.25 times the value it is compared with and, for every signal but cost, also at least 0.05 higher: 5 percentage points for a rate, 0.05 failures per run for task failures. Stations are ordered by how many signals are worse, then by how far the worst exceeds its comparison, then by runs.
Each station's row shows its name, linked to the station, its line,
<n> runs, which opens the Runs tab at the station, and "Improve", which
opens the Agent page with the station attached to a new chat. Unfolding a row
lists each worse signal:
| Column | Shows |
|---|---|
| Signal | The signal. |
| Compared | A dot for this period, a ring for the period before, a line for all stations, and a dashed line where the station gets flagged. |
| This period | The station's value. |
| Period before | The station's value in the period before. |
| All stations | The value over all stations in the period. |
| Flagged above | The lowest line the signal crossed: 1.25 times the value it is compared with, or, for every signal but cost, 0.05 above it where that is higher. |
A station whose spend came in more than one currency adds
Cost isn't compared. Prices are in more than one currency. With none listed, the
card reads No station stands out this period.
Cost
| Figure | Counts |
|---|---|
| Spend | The cost of the runs, when every priced run is in one currency. Otherwise –, with No priced runs or Prices in more than one currency. |
| Runs with no price | Runs whose harness reported no cost. |
| Cache share | Cache-read tokens, of input, cache-read and cache-write tokens together, over the runs whose harness reported input and cache-read tokens. Otherwise not reported. |
While the spend is in one currency, "By harness", "By model", "By station" and
"By repository" split it, highest first, and "Where the spend goes" follows it
from model to station to repository. A run on several repositories splits its
cost evenly among them. "Read from cache" gives each station's "Cache share",
"Cache read", "Cache write" and "Input" tokens, or reads
No station reported tokens in this period. See
Spend for which
harnesses report tokens.
People
"What the work asks of people", over the tasks the period's runs worked:
| Figure | Counts | A rise is |
|---|---|---|
| Inbox asks per task | Times a station asked a person in the period, per task, with <n> asks. | bad |
| Time to answer | The p50 time from an ask to its answer, with the p90 and <n> unanswered. | bad |
| Sent-back rate | Runs whose next act was Sent back, of runs, with <n> runs sent back. Its change is in points. | bad |
| Refused approvals | Approvals a person refused in the period, tool approvals and harness permissions alike. It opens the Runs tab at the tasks with a refusal, when there are at most 200. | bad |
"Who is using Spotlight":
| Figure | Counts |
|---|---|
| Active people | People with an activity in the period. |
| New adopters | Active people whose first activity on the team falls in the period. |
| Weekly active | People with an activity in the last 7 days, whatever the period. |
| Monthly active | People with an activity in the last 30 days, whatever the period, and the share of them active weekly. |
"What each person did" lists each active person's "Tasks opened", "Acts", "Chats" (agent chats they started), "First seen" and "Last active". A new adopter is marked "New".
Productivity
The Productivity tab, ?tab=productivity, has the sections "Where work
waits", "What the agents ship", "Review and approval" and "Pull requests".
Where work waits
"Where tasks wait" shows, for each station in the order its first run ended, the p50 of:
| Wait | From | To |
|---|---|---|
| Queued | A run is queued. | The run starts in its image on a runner. |
| Running | The run starts in its image on a runner. | The run ends. |
| In the inbox | The station asks a person. | The first person act on the task after the ask, at any station, before the task's next ask. |
Its table adds the p90 of each on hover, "Tasks" with its change, "Runs", "Asks" (times the station asked a person) and "Unanswered".
"Past the last station" follows each task's pull request:
| Figure | Counts | Time shown |
|---|---|---|
| PRs opened | Pull requests opened in the period. | Task to PR: from the task opening to the pull request opening. |
| First reviews | First reviews by a person in the period. Reviews by bots are not counted. | PR to review: from the pull request opening to its first review. |
| PRs merged | Pull requests merged in the period. | Review to merge: from the first review, or from the opening where no person reviewed it, to the merge. |
Times show as p50 and p90. A task tracks one pull request at a time. A task that opens another pull request is tracked on the new one.
What the agents ship
This section and "Review and approval" count the pull requests described under
Pull requests, at most the newest 20,000 over the period and
the one before, together with the team's runs. When the pull requests or the
runs passed 20,000, the section shows
Only the newest 20,000 pull requests in this period are counted. When none
of Spotlight's pull requests and none of people's alone merged in the period,
none waited on a review and no person reviewed one, it reads "No pull requests in this period", with "Connect a repository", which
opens the team's Integrations settings, and "Review and approval" does not
show.
Spotlight's pull requests are set against people's alone: those people opened with no Spotlight review task, in the repositories Spotlight merged into in the period. Bots' pull requests are left out of both.
"Flow through the pipeline", "Time to merge", "Time to merge by PR type" and "Time to first review" each have a percentile picker, "p50", "p75", "p90" or "p95", which starts at "p75". Each time shown is that percentile of its own times.
Flow through the pipeline
"Flow through the pipeline" follows Spotlight's pull requests merged in the period whose task opened in the period, the period before, or the last 30 days where that reaches further, from the task opening to the merge:
| Stage | From | To |
|---|---|---|
| Coding | The task opens. | The pull request opens. |
| A station | The time the task spent at the station: the time its ended runs there were queued and running, and its waits in the inbox there, from ask to answer. | |
| Awaiting review | The pull request is ready for review. | Its first review by anyone but its author, bots' included. Time as a draft is left out. |
| In review | That first review. | Its first approval that was not dismissed, or its closing where that came first. |
| Awaiting merge | The first approval, or the closing. | The merge. |
"Show coding as" picks "Stations", the default, which shows each station the
tasks ran at, in the order they were first used, in place of Coding; or
"Stages", which shows Coding as one stage. The card's header reads
<n> PRs merged. The legend lists each type of pull request; picking one
shows that type's times alone and reads <n> <type> PRs merged, and picking
it again shows every type.
A stage's colour runs from "flows freely" to "jammed": its time against the
longest stage's, at the percentile, for the same type. The longest stage is
the worst jam: its name is red, a badge under it reads
Worst jam | <time>, with mostly <part> at a station, and in the table a
red dot marks it, Worst jam: the stage with the longest wait on hover. Hovering over a station's name shows its
time queued, running and in the inbox. "Play" sets the chart moving, and
"Pause" stops it. Its table groups the stages under "At the stations" and "On
the pull request", with "Stage", "Time", and for stations "Queued", "Running"
and "In the inbox".
Time to merge
"Time to merge" gives the time from opening to merge of Spotlight's pull requests, "Spotlight's", and of people's alone, "People's alone", merged in the period, each with its change against the period before, where a fall is good. The chart draws both day by day. Its "Smoothing" switch picks "7-day rolling", the default, which takes the pull requests merged in the 7 days ending each day, or "Daily", those merged that day. "Time to first review" has the same switch.
"Time to merge by PR type" lists each type with a pull request
merged, as <n> merged over both groups, with Spotlight's time and people's,
slowest first by Spotlight's time. A type Spotlight has no time for comes last.
Review depth as throughput climbs
"Review depth as throughput climbs" gives, for each week from Monday, UTC, the share of the two groups' pull requests merged that week in the period that took "No review", "One round", "Two rounds" and "Three or more" review rounds, with "Merged that week" as a line. Beside the chart, each bucket's count and share over the period.
Review and approval
"Review and approval" counts the team's pull requests, less those written by
bots; a pull request a bot opened that has a Spotlight review task is
counted:
| Card | Shows |
|---|---|
| Time to first review | Day by day, the time from a pull request being ready for review to its first review by anyone but its author, bots' included, less any time as a draft, for those first reviewed that day ("Daily") or in the 7 days ending that day ("7-day rolling", the default). |
| PRs awaiting first review | At the end of each day, the pull requests ready for review, not drafts and not closed, with no review yet by anyone but their author. |
| AI review agents | Each bot that commented on or reviewed the team's pull requests, those bots opened included, that were open at any time in the period, most comments first: "Agent", "Comments" (inline and in the conversation, less those deleted), "Reviews", and "Resolved", the share of the review threads it started that are now resolved. |
| Human reviews | Reviews people submitted each day. |
Pull requests
Spotlight records every pull request in the repositories the GitHub App is
installed on, as GitHub reports it opening, changing, being reviewed and
closing. What a report says about a pull request is ignored when the pull
request's address disagrees with the report's repository or number. The
section counts the pull requests a task of the team opened, and those in a
repository one of the team's lines names. At most the newest 20,000 in a
period are counted; past that, the section shows
Only the newest 20,000 pull requests in this period are counted. With none,
it reads "No pull requests in this period", with "Connect a repository".
Each pull request is written by one of:
author | "Written by" | The pull request |
|---|---|---|
spotlight | Spotlight | A task opened it. |
touched | Others, with a Spotlight review task | Someone else opened it, and a Spotlight review task was opened on it. |
bots | Bots alone | A bot opened it, with no Spotlight review task. |
people | People alone | Any other. |
Spotlight's share
| Figure | Counts |
|---|---|
| PRs merged | Pull requests merged in the period, bots' included. |
| Made by Spotlight | Of those, the ones a task opened. |
| Reviewed by Spotlight | Of those, the ones others opened with a Spotlight review task. |
| Spotlight's share | Made or reviewed by Spotlight, of all merged. |
"Week by week" splits the merged pull requests by who wrote them, for each week from Monday, UTC. "By repository" gives Spotlight's share week by week for each repository the team's lines name, and any other its tasks merged into, most merged first, with "Merged" and "Share" over the period.
Spotlight against people
"Spotlight against people" sets Spotlight's pull requests merged in the period
against people's alone, merged in the same period in the repositories
Spotlight merged into, which its description names. Bots' pull requests and
those with a Spotlight review task are left out of people's. With nothing
Spotlight merged, its description reads
Spotlight merged no pull requests in this period. There is nothing to compare people's pull requests with.
| Row | Measures |
|---|---|
| Merged | How many merged. |
| Time to merge | From opening to merge: p50, with the p90 on hover. |
| Time in review | From the first review by a person to merge: p50, with the p90 on hover. Pull requests no person reviewed are left out. |
| Review rounds | Mean rounds per pull request. |
| Review comments | Mean inline review comments by people per pull request, less those deleted. |
| Revert rate | Pull requests since reverted, of those merged. |
Each approval or request for changes by a person is a review round. Reviews by people after the last of them, such as reviews that only comment, make one more round, so a pull request with only comments has 1. Reviews by bots and dismissed reviews do not count. "Review depth" gives how many merged pull requests took "No rounds", "1 round", "2 rounds" and "3 or more rounds", for the same two groups.
A pull request is a revert when its description has a line
Reverts <owner>/<repository>#<number> naming a pull request of the same
repository, or else when its title is Revert "<title>". When a revert merges,
the pull request it names, by number or by the latest merged one with that
title, is marked reverted, and the list shows its state as
<state>, reverted.
Types
"What Spotlight ships" counts merged pull requests by type, Spotlight's against
people's alone, with "Spotlight spent": what all the runs of the tasks behind
Spotlight's pull requests cost, or – with the reason on hover. People's here
are counted only in the repositories Spotlight merged into, as in "Spotlight
against people" and "Review depth".
A pull request's type is the first of these that gives one:
-
For Spotlight's, its task's latest change stack: each
changecohort is typed by its title, or else its intent, by the title rules below, and the most common type wins. A tie goes to the earlier of Feature, Fix, Refactor, Tests, Chore and Other. -
Its title, by a conventional prefix such as
fix:orfeat(api)!::Prefix Type feat,featureFeature fix,bugfix,hotfixFix refactor,perfRefactor test,testsTests chore,build,ci,docs,style,deps,revertChore -
Its title, by its first words, tried in this order:
First words Type revert,bump,chore,upgrade,release,doc,docsChore updatethen a word startingdependenc,depsorlockChore add,writeorcover, then optionallymore, then optionallyunitore2e, thentestortests, as inadd unit testsTests test,testsTests fix,fixes,fixed,hotfix,patch,resolve,correct,repairFix refactor,restructure,rename,extract,simplify,split,inline,tidy,clean up,moveRefactor add,implement,introduce,support,allow,create,build,enable,show,let,offerFeature -
Its branch, by a prefix followed by
/or-:Prefix Type feat,feature,featuresFeature fix,bugfix,hotfix,bugFix refactor,refactoringRefactor test,testsTests chore,deps,dependabot,renovate,release,docs,ci,buildChore -
Otherwise, Other.
Prefixes and words are matched without regard to case.
The list
"Pull requests" lists every pull request opened or closed in the period, or still open, the latest first, 50 to a page:
| Column | Shows |
|---|---|
| Pull request | Its title and #<number>, linked to it on GitHub. |
| Repository | Its repository. |
| Written by | "Spotlight", linked to the task, with "Runs", which opens the Runs tab at the task; or its author, or "Someone", with "Spotlight review", linked to the review task, when it has one. |
| Type | Its type. |
| State | "Merged", "Closed unmerged" or "Open". |
| Opened | When it opened. |
| To merge, In review | The time to merge and in review. |
| Rounds, Comments | Its review rounds and comments. |
| Size | +<added> −<deleted>. |
| Cost | For Spotlight's, what its task's runs cost. |
"Repository" ("Every repository"), "Written by" ("Anyone"), "Type" ("Every
type") and "State" ("Every state") narrow it. Following a figure elsewhere on
the tab narrows it further, shown as a pill: "Week of", "Reverted", "Review
rounds" (as above) or "Repositories" ("Where Spotlight merged"). Narrowed to
nothing, the list reads No pull requests match., with "Show every pull
request"; with none in the period, No pull requests in this period.
Retention
A pull request is forgotten 180 days after it closed, or, while open, 180 days after it last changed. One already past that when first heard of is not recorded. Spotlight's pull requests, and those with a Spotlight review task, stay counted after that. Removing the GitHub App's installation forgets every pull request of the workspace in the same way, trying again for any that could not be forgotten at once. Closing a workspace disconnects it from GitHub before its pull requests are removed.
A forgotten pull request is recorded again only when GitHub reports it updated later than the last update Spotlight had before forgetting it. Spotlight keeps that last update until 210 days after it.
Stations
The Stations tab, ?tab=stations, has the sections "Runs at each station",
"Config changes" and "Candidates". With &stationId=<stationId> it shows
one station instead.
Runs at each station
"The whole team" sums up the team's runs, counted as for station outcomes:
| Tile | Counts | A rise is |
|---|---|---|
| Runs | Runs that ran, drawn day by day. The other figures count only these. | neither |
| Failed | Runs that ended Failed, with <share> of runs, drawn day by day. | bad |
| Accepted | Runs whose next act was Closed. Shown when above 0 in the period or the period before. | good |
| Sent back | Runs whose next act was Sent back. Shown in the same way. | bad |
| Merged | Tasks among the runs whose pull request merged, at any time. Shown in the same way. | good |
| Cost | The runs' cost, when all are in one currency. | neither |
| Median run | The median time the runs took, with the p90. | bad |
"By station" lists each station with a run ending in the period, in the order each one's first run ended, and each station with task failures and no runs. A station's name opens its own view. "Runs per day" draws its runs per day, failures in red, beside the change in its runs. Then:
| Column | Counts |
|---|---|
| Runs | Runs that ran. |
| Failed | Runs that ended Failed. |
| Stopped | Runs a person stopped. |
| Checks passed, Checks failed | Runs whose check passed or failed. |
| Accepted | Runs whose next act was Closed. |
| Sent back | Runs whose next act was Sent back. |
| Handed on | Runs that passed their task to another station. |
| Merged | Tasks among the runs whose pull request merged, at any time. |
| Withdrawn | Runs withdrawn before they ran. |
| Cost | The runs' cost, when all are in one currency. |
| Median, p90 | The time the runs took. |
A column from "Failed" to "Withdrawn" shows only when the whole team's count
in the period is above 0. "Runs" and "Failed" open the Runs tab at the
station's runs, or its failed ones. Each row has the anchor
#station-<stationId>, which a station alert's "Open in Insights" opens.
One station
?tab=stations&stationId=<stationId> shows one station, under the breadcrumb
"Stations" and the station's name, with its line beside it. It is opened from
a station's name under "By station", from judge links on this page, and from
the station's page: "See in Insights", "History", and the judge's "How it
agrees with people", which opens it at #judge.
| Part | Shows |
|---|---|
| Over this period | The tiles of "The whole team", for the station's runs alone. "Runs" and "Failed" open the Runs tab at the station's runs, or its failed ones. With task failures and no runs, the tiles read 0. With neither in the period, No runs at this station in this period. |
| Why runs failed | Shown when the station has failed runs or task failures in the period. See below. |
| Runs by config | The station's latest 50 runs, whatever the period, grouped by config. See below. |
| How far to trust the judge | Shown when the station has a judge, at #judge. See below. |
"Why runs failed" lists the station's failed runs by reason, the reason or
No reason given, with how many runs, most first; and its task failures by
source (run or hook:<key>) and code, as <source> · <code>, with how many
times, and · <n> in a row when one task raised it more than once in a row.
With both, they are headed "Run failures" and "Task failures". A reason with
one run opens that run; another opens the Runs tab at the station's failed
runs. A task failure that one task raised opens the Runs tab at that task. A
hook the loop guard stopped shows only among task failures.
"Runs by config" groups runs that started one after another with the same
config, the newest group first; a config that returns after another starts a
new group. A group is named by the first of its
changes from the last
config that ran before it, with +<n> more, or Config <fingerprint>, the first 8 characters;
the group of runs before any recorded config reads
Before the first run with a config. A group whose runs never started on its
config is marked "Never started". Each group shows "Checks" as
<share> passed, "Next" as <share> accepted, "Delivery" as
<share> merged, "Took" (the median), "Cost" and "Ended", and "Impact" when
it has changes (see Impact). Unfolding it lists its runs, with
"Run", "Round", "Checks", "Next", "Delivery", "Took", "Cost" and "Ended". With
no runs, the card reads "No runs yet".
"How far to trust the judge" shows the judge's measure of agreement with
people, with Trusted. Minimum required: <trust> or why its scores are hidden, then how
many of the station's last runs have "Both scores", "Judge only", "People
only" and "Neither". When the judge has judged runs under more than one rubric
version, "Rubric" picks the version. Each version shows "Statistics"; "Score
comparison" and "Distribution" when it has runs with both scores; "Trend over
time" when it has a trend to draw; and "By claim" for a judge with claims. With no judged runs, it reads "No judged runs yet". See
Agreement for how
agreement is measured and when a judge is trusted.
Config changes
"Config changes" lists the times a station started a new config in the period, by the same rule as the diamonds under "Output against cost", the latest first:
| Column | Shows |
|---|---|
| First run | The day of the change's first run. Selecting it shows the whole row. |
| Line | The station's line, linked, or "Removed line". |
| Station | The station, linked. |
| What changed | The changes from the config of the station's previous run, as a plan: + added, - removed, ~ changed. Otherwise Config not on record. |
| Runs before, Runs after | The runs counted under Impact. |
Each row ends with "Impact". With none, the card reads
No station changed its config in this period.
Impact
"Impact", on a row of "Config changes", a diamond on Overview or a group of
"Runs by config", opens a panel beside the page, titled <station>: impact.
It stays open across tabs until it is closed with its close button ("Close
the change"), and closes when the Traces tab opens.
The panel counts the station's runs that ended in the period and the period before, or the last 30 days where that reaches further, ordered by when each started, or ended where it never started:
| Window | Runs |
|---|---|
| Before | Runs from the station's previous config change, or its earliest counted run, up to this change, with the config the station ran last before it. |
| After | Runs from this change up to the station's next config change, with the new config. A run with the new config that never started counts here even when it ended before the change. |
"What changed" shows the changes as under "Config changes", or
Config not on record. "Runs before and after" gives "Before", "After" and
"Change" for:
| Row | Counts |
|---|---|
| Runs | Runs that ran. The other rows count only these. |
| Sent-back rate | Runs whose next act was Sent back, of the runs whose next act is known. |
| Accepted rate | Runs whose next act was Closed, of the runs whose next act is known. |
| Checks-failed rate | Failed checks, of passed and failed checks. |
| Failed rate | Runs that ended Failed, of runs. |
| Merged rate | Tasks whose pull request merged, of the tasks of the runs whose pull request merged or closed unmerged. |
| Cost per run | The cost of the priced runs, per priced run. |
| Judge score | Only while the station's judge is trusted: the mean judge score on each side, or –. Hovering shows Over <n> judged runs before and <m> after. Opens how often the judge agrees with people., and the label opens the station's judge. |
Each row but "Runs" counts, on each side, the runs settled for it, as for
station alerts.
"Change" is green when the row got better and red when it got worse. A rate's
change is in points and cost per run's a whole percent; with none, it reads
No change. It reads Too few runs when either side has fewer than 5 settled
runs, and for cost per run when the two sides are priced in different
currencies. The judge score's change is in points, green when it rose.
A judge score is the mean of the judge's scores on the station's current rubric, as a share of its scale: Yes is 100% and No 0%; on 1 to 5, 1 is 0% and 5 is 100%; on claims, the share of claims met.
The panel ends with "Open the station" and "Improve". A change no longer in
the period reads This change is outside the period., with "Open the
station".
Harness, model, effort and profile
This card sets how runs turned out against what they cost, for each setup: a harness, model, effort and harness profile together, as each run recorded them. It counts the runs that ran and ended in the period; a run with no recorded setup is left out. Its description names the outcome charted, the fewest runs a setup needs for the frontier, and the size rules. The filters sit in a menu, "Filter the setups", which counts those applied.
| Filter | Narrows to |
|---|---|
| Outcome | The outcome charted: "Accepted", the default, "Sent back", "Handed on", "Checks passed", "Merged", or "Judge score" when any setup has judged runs. |
| Station | "Every station", or the runs at one work station. |
| Task size | "Every size", or the runs of one size. |
| Outcome | Of a setup's runs |
|---|---|
| Accepted | Runs whose next act was Closed, of the runs whose next act is known. |
| Sent back | Runs whose next act was Sent back, of the runs whose next act is known. Lower is better. |
| Handed on | Runs that passed their task to another station, of the runs with a next act or a hand-on. |
| Checks passed | Passed checks, of passed and failed checks. |
| Merged | Tasks whose pull request merged, of the tasks of the runs whose pull request merged or closed unmerged. |
| Judge score | The mean judge score, as under Impact. |
Judge scores are counted only with a station picked, and only while that
station's judge is trusted. With a station picked that has a judge and runs
in the period, the card adds Judge scores count only for judges that agree with people often enough.
and names the station, linked to its judge, as trusted or hidden.
A setup's cost per run is the cost of its priced runs, per priced run, when they are all in one currency. The setups are compared in the currency most of them are priced in, or the first in alphabetical order on a tie.
For each outcome, a setup is on the frontier when it has at least 3 runs, a cost per run in that currency and a value, and no other such setup costs no more and does as well or better, with either a lower cost or a better value.
The chart plots each setup priced in that currency with a value for the
outcome, cost per run across and the outcome up, as "On the frontier" or
"Other setups". Each frontier setup is labelled with its name. A dot's size
is its runs, with a "Runs" legend of sizes under the chart. Hovering over a
setup shows its name, <outcome> <value> · <cost> a run, and <n> runs, with
· on the frontier for a frontier setup. With no runs, the card reads
No runs match these filters in this period., and with no priced setup that
has a value for the outcome,
No priced runs to chart. Show the table for outcomes alone.
The table groups the setups by harness, each as model, <effort> effort and
<profile> profile, as far as each is set, with "Runs", "Cost per run" and
each outcome; a trophy, "Best value", marks a setup on the frontier of the
charted outcome.
A run's size is told by the lines it changed: the lines taken out and put in across its file changes, each file compared from before its first change to after its last. A run that changed no lines is told by its steps, over the attempts it kept.
| Size | Lines changed | Steps, for a run that changed none |
|---|---|---|
| Small | Up to 50 | Up to 20 |
| Medium | Up to 400 | Up to 80 |
| Large | More | More |
A run whose cost was never worked out has no size and is left out when a size is picked.
Replayed candidates beside the setups
With a station picked, the chart also draws the station's replayed candidates,
the same points Candidates draws, hollow, for the replay sets
charted in the currency the setups are compared in; with no such currency,
none are drawn. The legend names them "Searched", "Proposed" and "Live config,
replayed", and the side of the chart reads <outcome> · replay success. A replayed point's height
is its replay success, scored by the candidate's own checks and judge,
whatever outcome the card charts; it is not the live outcome the setups show.
Hovering over one shows it as under Chart. "Task size" does not
narrow replayed points, and they are not joined to the setups' frontier.
| Note under the chart | When |
|---|---|
Replayed points show replay success on up to <n> runs of past tasks, not live outcomes. Each candidate's own checks and judge score it. The task-size filter doesn't apply. | Replayed points are drawn. <n> is the most runs any of them was scored on. |
Replayed points show success only. They're hidden while the chart shows Sent back. | "Sent back" is charted. |
Replayed candidates aren't drawn. No live setup has a priced outcome. The table lists them. | The station has replayed points to draw, and the setups have a currency, but no priced setup has a value for the outcome charted. |
The replayed candidates could not be loaded., with "Try again" | Reading them failed. The rest of the tab still shows. |
While replayed points are drawn, the card's description adds
Replayed points show replay success on past tasks. Whenever the station has
replayed points and the setups have a currency, except while "Sent back" is
charted, the table adds a second table, one group per replay set reading
<set> replayed, with "Config", "Source", "Runs scored", "Cost per run",
"Replay success" and "Held out".
Candidates
"Replayed candidates" sets the station configs the team replayed on past tasks against the station's live config, replayed on the same tasks, before any of them is applied. It counts replays, and it has no filters but the period.
A replay set holds runs, and a task may have more than one run in it. Success rates average over the set's runs, and comparisons count them. Scored counts count replay runs, so a replay that replays each run 3 times scores up to three times the set's runs.
Which candidates show
| Candidate | Shows when |
|---|---|
| Searched | Its search started in the period. |
| Proposed | A replay of it started in the period. With more than one over a set, the newest finished one shows, or else the newest. |
The live config shows only as the replays it was compared with, and the candidates of replay checks do not show: a check's verdict is on its pull request and on its replay set page. A set's health and baseline are as they stand now, whatever the period.
The section reads the newest 500 replays started in the period. When there
were more, it shows Only the latest replays in this period are counted.
With none, it reads "No candidates replayed in this period", with "Open the line", which opens one of the team's lines.
Stations and sets
The card starts on "Table". It is one table: a row for each work station with a candidate in the period, the station with the newest replay first, named with its line, or "A removed station". Unfolding a station lists each replay set its candidates were replayed over, the newest replayed first:
| Part | Shows |
|---|---|
| Name | The set's name, linked to its page. |
| Size | <n> tasks, the different tasks its runs worked, with (<m> runs) when it holds more runs than tasks, or "No tasks yet". |
| Warning icon | When a health reading is a warning, or a trust check was not run. Hovering lists each warning as <reading>: <what it says>., then <n> of <m> trust checks not run: <readings>. |
| Shield icon | When the set checks the station's pull requests. Hovering explains it, starting Checks pull requests. |
| Station gone | When its station has left the line. |
| No baseline | The live config hasn't been replayed on this set. Nothing is compared with live. when the set has no baseline. |
Candidate rows
Unfolding a set lists its candidates, the greatest gain over the live config first, those with none last, then "Live config" when the set has a baseline. Each row ends with a "Changes" link, described below the table:
| Column | Shows |
|---|---|
| Config | The candidate's name, a trophy ("Best value") when it is on its search's frontier, and its replay's state until that replay has finished. Selecting it shows the whole row. |
| What changed | How its config differs from the live config, as a plan. |
| Source | "Search", "Proposed", or "Live" for the live config. Hovering over the heading explains each, and the trophy. |
| Replay success | A bar from 0% to 100%: the candidate's success, with its 95% range on hover, against a mark for the live config's, coloured by the verdict against live. |
| Candidate, Live | The candidate's success and the live config's, as percentages. |
| Against live | Its gain over the live config in points, as in +16 pts, with the verdict and the set's minimum real gain on hover; or "Not compared". |
| Cost per run, Live cost | What each spent per run. When the live config is priced in another currency, the candidate's cost is over all its scored runs, as its hover says, and "Live cost" reads –. |
| Held out | Only when a candidate has a held-out replay: its held-out score, "Replaying" while that replay is queued or running, or –. |
| Runs | The runs the figures cover. |
Once a candidate has runs compared with the baseline, "Replay success",
"Candidate", "Live", "Cost per run" and "Runs" count those runs alone. Until
then they count every run it was scored on, and "Live" reads –. A searched
candidate's success is its search score;
a proposed candidate's is its replay's
success over the set.
| "Changes" opens | When |
|---|---|
| The pull request on GitHub | Picking the candidate opened a pull request. |
| Its search page, at the candidate | A searched candidate, otherwise. |
| Its replay page | A proposed candidate, otherwise. |
| The baseline's replay page | The "Live config" row. |
The "Live config" row shows the baseline's success under "Live", its cost under "Live cost", and the runs it was scored on.
Chart
Choosing "Chart" draws one chart for each set in place of its candidate rows, cost per run along the bottom and "Replay success" up the side:
| Shape | Point | Height |
|---|---|---|
| Diamond | Live config, replayed (the search's tasks): the live config a search started from. One per search; with more than one search over the set, Live config, replayed (tasks of the search on <day>). | Its search score: the live config's success on that search's searched runs. |
| Diamond | Live config, replayed (whole set): the set's baseline. Drawn only when the set has a proposed candidate. | Its success over the whole set. |
| Circle | "Searched". | Its search score, on its search's searched runs. |
| Square | "Proposed". | Its replay's success over the set. |
A searched candidate on its search's frontier is highlighted, as "On the
search's frontier" in the legend; which candidates are on it follows the
search's aim. A search's
Live config, replayed diamond is always counted as finished and always
marked on its search's frontier. Each point is labelled with its name, the
baseline's as "Live config".
Hovering over a point shows Replayed <name>,
Replay success <share> of <n> scored runs · <cost> a run, and the set as
over <set>, with its held-out score as
held out <share> of <n> held-out runs once it has one, and
on the search's frontier for a frontier candidate.
The chart is in the set's currency: its baseline's, or else that of its newest
candidate. Only points whose replay has finished, has a score, and is priced in
that currency are drawn, the search diamonds counted as finished. With nothing to draw, the chart reads
Nothing on this set has a priced score yet. Show the table to see the replays.
Tools and skills
The Tools and skills tab, ?tab=tools, has the sections "Skills and tools"
and "Memory". "Skills and tools" counts the runs that ran and ended in the
period. With no skills, unused skills, MCP servers or refusals to list, it
reads "No skills or MCP servers used in this period", with "See the team's
skills".
Skills
A run is given a skill when its station's config gave it. A run reads a skill when one of its tool calls does either of these:
- Reads, searches or runs a command on a path inside the skill's folder,
.claude/skills/<key>/or.spotlight/skills/<key>/, whatever the run's harness. Listing that folder counts; a path to the skills folder itself, writing or editing a skill's file, and MCP calls do not. - Asks Claude Code's
Skilltool for the skill by its key.
Spotlight's built-in skills are not counted. A key the team does not define is counted, and shown without a link.
"Skills" lists each skill, most read first, with "Read per week" (the share of all runs that read it, week by week from Monday, UTC), "Given", "Read" and "Trend"; its "Runs" is left empty. Unfolding a skill lists each station with "Runs", "Given", "Read" and "Trend", and no chart. A skill's trend compares the share of all runs that read it in the last 2 weeks with the weeks before:
| Trend | When |
|---|---|
| Spreading | The share rose by 20 points or more. |
| Fading | The share fell by 20 points or more. |
| Steady | Neither. |
A skill has no trend when the period touches 2 weeks or fewer, or when either side has fewer than 5 runs. A station's trend is worked out the same way from its own runs.
"Unused skills" lists the skills the team defines that no run read in the period. Each links to the skill on the team's Skills page.
MCP servers
A tool call belongs to an MCP server when Claude Code names the tool
mcp__<server>__<tool>, or, for a harness that does not name its tools, when
the call's title is Tool: <server>/<tool> or mcp__<server>__<tool>. The
title of a call that runs a shell command is not read for a server. The tools
Claude Code names mcp__acp__Read, mcp__acp__Write, mcp__acp__Edit,
mcp__acp__MultiEdit, mcp__acp__Bash, mcp__acp__BashOutput and
mcp__acp__KillShell belong to no server.
| Column | Counts |
|---|---|
| Runs | Runs that called the server. |
| Calls per run | Calls to it, per run that called it. |
| p50, p95 | How long a finished call took, from when it started working until it completed or failed, so a wait for approval is left out, and so is any time the run was paused. Approximate: each call is counted in one of fixed time ranges, from under 10 ms to 10,000 seconds and over, and the percentile is given as the geometric middle of its range; in the first range, under 10 ms, as 5 ms, and in the last as 10,000 seconds. |
| Error rate | Failed calls, of calls. A call that never finished counts as failed. |
| Failing runs | Runs that failed where the run's last call to the server failed or never finished. |
| Refused | Runs in which a person refused to approve one of its tools. |
Refused calls are left out of every column but "Refused". With no server
called, the card reads No run called an MCP server in this period.
A call is refused when the harness asked permission for it and Spotlight's answer was anything but allow: a refusal, the call being put to a person, or no answer because the control plane could not be reached. What the tool's output says does not make a call refused.
Refused approvals
"Refused approvals" counts, for each tool and server, under "Tool", "Server"
and "Refused", the runs that ran in the period in which a person refused to
approve it. A tool is named by its own name, as in linear.comment, or for a
harness permission that names no tool, as <harness> <kind>. A tool of no
server reads – under "Server".
Memory
| Tile | Counts |
|---|---|
| Standing memories | The memories the team keeps now, whatever the period. |
| Briefed | Memories briefed to the runs that ended in the period, withdrawn runs included, each counted once, with to <n> runs, the runs that ran they were briefed to. |
| Read | Memories the runs that ended in the period read with spotlight recall, withdrawn runs included, each counted once, with by <n> runs, the runs that ran that read them. |
| Flagged wrong | Memories a person retired in the period. A memory replaced by a correction is not counted. |
| Solved with memories | –, with No benchmark yet. |
"Runs with memories and without" sets the period's runs that were briefed or
read a memory against its other runs, under "Without", "With" and "Change",
with the rows of Impact but "Judge score". Only runs that ran are
counted. "Change" reads as under Impact, Too few runs and No change
included.
"Flagged wrong" lists each memory flagged wrong, the latest first: "Memory",
its handle, linked to the memory, and its text; "Reason", or –; "Flagged
by", or "A former member"; and "Date". With none, it reads
No memory flagged wrong in this period.
With no standing memories, and none briefed, read or flagged wrong in the period, the section reads "No memories yet", with "Open the team's memory".
Runs
The Runs tab, ?tab=runs, lists every run that ended in the period, withdrawn
runs included, newest first, 50 to a page:
| Column | Shows |
|---|---|
| Outcome | Completed, Failed, Stopped, Abandoned, Withdrawn or Superseded, linked to the run. |
| Failure | The run's failure, or –. |
| Station | The station. |
| Person | Who opened the task, "No one", or "A former member". |
| Setup | The harness and model, as <harness> · <model>, or <harness> · Harness default for a run with no model set. |
| Took, Cost, Ended | How long the run took, what it cost, and when it ended. |
"Station" (work stations only) and "Outcome" narrow the list. Following a figure on another tab narrows it further, shown as a pill that can be removed:
| Pill | Narrows to runs |
|---|---|
| Line | On the line. |
| Pull request | Whose task's pull request is now Merged, Closed unmerged or Open. |
| Then | Whose next act was Started, Parked, Closed, Answered or Sent back. |
| Asked a person | Of tasks that asked a person in the period. |
| Harness | Run by the harness. |
| Model | Run with the model. "Harness default" is a run with no model set. |
| Repository | On the repository. "No repository" is a run on none. |
| Config | With the config fingerprint, shown by its first 8 characters. |
| Task, Tasks | Of the task, or of up to 200 tasks. |
| Person | Of tasks the person opened. |
| Day | That ended on the UTC day. |
| Week of | That ended in the week starting that Monday, UTC. |
| Given skill, Read skill | Whose station gave the skill, or that read it. |
| Called | That called the MCP server. |
| A call failed on | With a failed call to the MCP server. |
| Last call failed on | That failed where their last call to the MCP server failed. |
| Refused, Refused on | Where a person refused the tool, or a tool of the MCP server; "No server" for a tool of none. |
Every filter narrows the others. The skill, MCP server and refusal filters
leave out withdrawn runs. The address keeps the filters and the page. Under
the list, the pager reads <first>–<last> of <total>, as in 1–50 of 230.
Narrowed to nothing, the list reads No runs match., with "Show every run".
Traces
The Traces tab, ?tab=traces, lists the team's task
traces, one for each task with a run created,
that is queued, in the past 7, 30 or 90 days, as the period picker says. The span counts back
from now rather than in whole UTC days, and has no period before it. The
traces are not part of the Insights API.
A chart of "Traces" and "Faults" over the span sits above the list. Each row
shows "Started", "Stations", "Task", "Asked", "Answered", "Faults", "Took" and
"Cost", and opens the task's trace at /w/<slug>/team/<teamId>/tasks/<taskId>/trace.
Each row starts with a checkbox. Ticking rows shows a bar reading
<n> selected, with "Discuss with the agent", which opens the Agent page with
those traces attached to a new chat, and "Clear selection".
With nothing traced in the span, the tab reads "No traces" and
No traces in the <span>. Create a task to leave one., as in
No traces in the past 30 days., with "Create a task", which opens the
team's first line.