Tasks, runs and the inbox
A task stands at one station of a line. Runs work it; people answer it from the inbox. The terms are defined in the glossary.
Tasks
Fields
| Field | Description |
|---|---|
| Title | Up to 200 characters. Fixed once the task is created. |
| Description | Markdown, up to 20,000 characters. Optional. Fixed once the task is created. |
| Station | The station the task stands at. |
| Assignee | The person whose inbox the task's conversations reach. A task created by a person is assigned to them. |
| Raised by | The person who created the task, if a person did. |
| Attachments | Files given with the task. See Attachments. |
| Pull request | The pull request the task opened, or the one it reviews. |
| Marks | The handles the task is known by outside Spotlight. See GitHub events. |
| Revision | A counter. It goes up when the task moves to another station and when it is finished. |
| Branch | spotlight/<taskId>. The branch a run pushes the task's work to. |
A task created on the board is placed at the back of the station it is created at. A task created at a station with a harness starts a run at once. A task created at the backlog does not.
Task states
A task has no stored status. Its state follows from its notes, its run and whether it is finished.
| State | Condition | Board label |
|---|---|---|
| Awaiting review | The latest asked note has no later answered or passed note. | Waiting for you |
| Running | A run holds the task, queued or claimed. | In progress |
| Finished | The task was moved to the completed station, or its run declared done. | none |
| Idle | None of the above. | none |
A task that is created, moved, answered or reached by a happening, and is then idle at a station with a harness, is handed to a new run at once. A finished task is unfinished again when it moves to a station other than completed, or when a person or a happening adds a note to it.
Notes
A task keeps a trail of notes. Each new note is added at the end of the trail, and a note's words never change.
| Kind | Written when |
|---|---|
asked | A station puts the task to a person: a review handover, a failed or stopped run, an approval to decide, a request to connect; or a happening stops the run (see Moves). |
passed | The task moves to another station or is finished. It closes the open conversation. |
answered | A person answers in the inbox or on the run's page, or a happening with words answers the open conversation. |
said | A person says something to a running run, a happening arrives that leaves the task where it is, or a different failure arrives while the task awaits review. |
Each note has a voice, station, person or world (a happening), and a
form: words, setup, settled, approval, failure or connection.
A conversation runs from an asked note to the next passed note. It is open
until that passed note is written.
Handover
A task's run ends with a declaration, made with spotlight handover --next
(see Agent commands). Keys are compared
without regard to case.
| Declaration | Allowed when | Effect |
|---|---|---|
| a station key | The station is on the line, is not the current station, has a harness, and the current station passes to it. | The task moves to the back of that station with a passed note, its revision goes up, and a run starts there. |
review | Always. | An asked note is written and the task awaits review. An unassigned task is assigned to the person who raised it. |
done | The current station has ends: true. | The task is finished at the station where it stands, with a passed note, and its revision goes up. |
Any other ending is handled as review, with one of these notes:
| Ending | Note |
|---|---|
| No declaration | The run finished without saying where the task should go next. |
| No outcome reported | The run ended without reporting an outcome. |
| A key that is not a station on the line | The run asked for station "<key>". This line has no such station. |
| The station the task stands at | The run asked for station "<key>". The task already stands there. |
| A station with no harness | The run asked for station "<key>". Nothing runs there to take up the task. |
| A station the current one does not pass to | The run asked for station "<key>". Work does not pass there from the task's station. |
done where the line does not end | The run said the work is done. The task is not at the end of its line. |
| The run failed | The run failed: <message>, with a failure. See Failure codes. |
| The runner was lost, or the host stopped the sandbox at its run timeout, on the last attempt | Spotlight gave up on the run after <n> attempts. The run did not finish., with the failure runner-lost or timed-out. |
| The run was stopped | A person stopped the run. The task is back with you. |
A hook on handover.giving refused it | The hook's reason, then The run said: <note>. See spotlight_hook. |
If a reaction was held back while the run held the task, every ending other
than a station key moves the task to the latest held-back station instead of
the ending above. So does a hook stopped for looping while a run holds the
task: the run is let go, and the passed note carries the hook's failure. See
Moves.
Moves
| Mover | How | Placed | Effect |
|---|---|---|---|
| Person | Dragging on the board, the selection bar's "Move to", or the task's "Station" property. Only from the backlog to a work station, from a work station to the backlog, from anywhere to the completed station, or within one station. Any other move is refused with 422 and Task cannot move from <from> to <to>. <reason>, where <reason> is A completed task stays completed. or Only runs move a task between work stations. | Below the task it is dropped under (afterTaskId), or at the top. | A passed note Moved to <Station>. is written. |
| Reaction | A happening that a line's reaction names, when no run holds the task. See spotlight_line. | At the top of the station. | A passed note in the world voice, carrying what the happening said. |
| Held-back reaction | The run that held the task when a reaction was held back ends other than by passing the task on. | At the top of the station. | A passed note in the station voice, Moved to <Station> when the run ended. An event arrived while the run worked., followed by the run's note if it gave one; for a failure or a lost runner, that ending's note with its failure; for a missing outcome, that ending's note; for a console stop, A person stopped the run. |
A move to another station:
- raises the task's revision;
- ends the task's run, for a person's move: a queued run is withdrawn, a claimed run is asked to stop, and a waiting or parked run is completed;
- finishes the task if the station is completed, and unfinishes it otherwise;
- starts a run if the task is idle at a station with a harness.
A person who moves an unassigned task from the backlog to a work station becomes
its assignee. A person's move names the task's current revision; a stale
revision is refused with 409. A move to the station the task already stands
at only reorders it.
When a happening finds a task, the line decides one of these outcomes:
| Outcome | Condition | Effect |
|---|---|---|
| nothing | The task stands at the completed station, or none of the other outcomes applies. | None. |
| moves | A reaction for the happening, from this station or from any, leads to another station, and no run holds the task. | The task moves there. |
| held back | As for moves, but a run holds the task, queued, claimed, waiting or parked, and the station is not the completed station. | A said note Heard while a run held the task. If the run does not pass the task on, the task moves to <Station> when the run ends., followed by what the happening said. The run is told, and keeps the task. A later passed note, or a later asked note other than an approval, drops held-back moves; a later stays outcome does not. |
| interrupts | As for moves, but a run holds the task and the station is the completed station, and the task does not await a person's answer to a question (a pending approval does not count). | The run ends as for a person's move, and its later handover is ignored. The task is put to its assignee, or else the person who raised it, with an asked note in the world voice: <What happened> while <Station> was working. Close the task, or reply to carry on., where <What happened> is the happening's text without its final full stop, and its station takes the place of <Station> when the station is not known. |
| notes while waiting | As for interrupts, but the task already awaits a person's answer to a question. | A said note in the world voice, <What happened> while <Station> was waiting for an answer. Close the task, or reply to carry on., added to the open conversation, with its station when the station is not known. The run keeps the task, and earlier held-back moves are cancelled. |
| answers | No reaction leads to another station, the task awaits review, and the happening carries a person's words. | An answered note; the run that asked carries on, or else a run starts at the same station. |
| stays | A reaction applies and leads to the station the task stands at, and the happening does not answer. | A said note. |
Runs
A run is one execution of a harness, for a task at a station or for a chat. A replay run, which runs a past task run again under a candidate config, holds no task and is described in Replays. A search's agent run, which proposes candidates, holds no task either; see The search's agent run.
Run states
| State | Meaning | Final |
|---|---|---|
queued | Waiting for a runner. | no |
claimed | A runner holds it under a 60-second lease that the runner renews. | no |
waiting | The run asked a person and waits for the answer. Its runner still holds it and keeps the sandbox, for up to the host's SPOTLIGHT_ANSWER_WAIT. | no |
parked | The run asked a person, and its runner let the sandbox go; no runner holds it. An answer puts it back in the queue. | no |
completed | Ended without a failure. A stopped run ends here. | yes |
failed | Ended with a failure code. | yes |
superseded | Withdrawn before it started, or failed after its task had moved on. | yes |
On the Runs list and on a task's "Activity", a queued run reads "Queued", a
superseded run "Superseded", and a claimed run shows that it is in progress. A
run reads "Waiting for you" while it is waiting or parked, and while the
asked note it wrote has no later answered or passed note. On the Runs
list, that label links to the run's inbox item. On a task's "Activity", such a
run is led by a row that reads "Waiting for you", or "Answered" once a later
answered or passed note is written, linked to the inbox item. The Runs
list's "Waiting for you" view shows only the runs that read "Waiting for you";
with none, it reads Nothing is waiting for you. Switch to All to see every run.
The Run page of a waiting or parked run shows "Waiting for you" with "Answer
in the inbox", or "Open in the inbox" when the open question is an approval.
An approval can also be decided on the Run page itself, on the same card as
in the inbox: copy the call, edit a tool call's inputs, "Approve", or
"Dismiss", which first asks for a reason to tell the agent, Tell the agent why (optional). See Approvals.
Transitions
| From | To | When |
|---|---|---|
queued | A task or chat is handed to a run. | |
queued | claimed | A runner that meets the run's placement claims it, and no other run for the same task is claimed or waiting. |
queued | failed | The station gives what cannot be given: a catalog item that does not exist or that the plan does not include, or a harness profile the team does not hold (refused). A push pool could not provision a runner (unprovisioned). |
queued | superseded | The task moved to another station or was deleted. |
claimed | queued | The lease lapsed, or the host stopped the sandbox at its run timeout (timed-out), no stop was asked for, and the run has had fewer than 3 attempts. Or its run.starting hooks could not be asked, and the run is handed back to wait. |
claimed | waiting | The turn ended cleanly with the task put to a person, and the runner reports the workspace stored for the turn. Only a task's run waits; a chat run never does. |
waiting | claimed | The person answers while the sandbox is kept, in the inbox, in the task's Slack thread or on the run's page. The answer goes to the same session, in the same attempt. A run asked to stop is not taken up. |
waiting | parked | SPOTLIGHT_ANSWER_WAIT passes without an answer, at once when no host started the runner, or the lease lapses. |
claimed | queued | An answer arrives while the runner is setting the run aside after its harness has closed. The run is queued as its next attempt. |
parked | queued | The person answers. The run is queued as its next attempt, and restores the stored workspace wherever it is claimed. |
waiting, parked | completed | The task is moved or deleted, or the run is stopped. |
claimed | completed | The runner reports an outcome without a failure, or the run is stopped, or its lease lapses after it was asked to stop. |
claimed | failed | The harness failed (exited), or the run failed before its harness started (unprepared). A hook on run.starting refused it (refused). Its hooks could not be asked after the last hand-back (hooks-unanswered). Its lease lapsed on the last attempt, and no stop was asked for (runner-lost). Its host stopped it at its run timeout on the last attempt, and no stop was asked for (timed-out). |
failed | superseded | The run's task has moved on. |
Words said to a run at the station where its task stands idle, when that
station has a harness, start a new run there that resumes the station's
session. Words said to a run whose task has left its station are refused with
409, and words from anyone but the task's assignee with 404.
Only a claimed, waiting or parked run can be stopped. The web app offers Stop
only while a run is claimed. Attempts a run spent parked do not count toward
the 3. Only the task's assignee may stop a run or say
something to it. See Steer or stop a run.
Run page composer
| When | Placeholder | Accessible name |
|---|---|---|
| The run waits on a person's answer to a question that is not an approval. | Answer the question to continue the run… | Answer <Station> |
| The task stands at the run's station with nothing running and no open question, and the station has a harness. Words start a new run there. | Say what to change. <Station> starts again… | Send it back to <Station> |
| Otherwise. | Say something… | Say something to <Station> |
The "+" button, "Attach files", adds files, and so does dropping or pasting them on the composer. Images show above the text; other files show by name in the text. The composer shows the station's harness, model and effort and the line's repositories, and none of them can be changed there. While a runner holds the run and the composer is empty, "Send" becomes "Stop".
Once the task has left the run's station, the composer is replaced by
The task has moved to another station. Messages here reach no one. A replay
run and a search's agent run have the composer turned off, with
The agent runs autonomously. It can't receive messages.
Failure codes
| Code | Cause | Retried |
|---|---|---|
exited | The runner reported that the run failed once its harness had started, or the harness did not start. | no |
unprepared | The run failed before its harness started: its workspace, checkouts, attachments, skills or harness files could not be made ready, or its host could not start a sandbox with the line's image and its egress. | no |
refused | The run could not be given what its station names, or a hook refused its start. | no |
unprovisioned | A push pool did not provision a runner for it. | no |
hooks-unanswered | Its run.starting hooks could not be asked. | yes |
runner-lost | Its runner stopped renewing the lease. | yes |
timed-out | Its host stopped its sandbox for running past SPOTLIGHT_RUN_TIMEOUT. | yes |
superseded | Its task had moved on. | no |
A run whose run.starting hooks could not be asked is handed back and tried
again, 6 attempts in all, waiting 5 seconds and doubling to at most 2 minutes.
A run whose runner is lost, or that timed out, is queued again at once, 3
attempts in all, counted together.
A failed run puts the task to a person with a failure of source run and the
run's code. A failing hook reports a failure of source hook:<key>; see
spotlight_hook for its codes. A failure with the same source and
code as one already open on the task raises that failure's count instead of
asking again. runner-lost and timed-out count as one repeating failure: the
count carries on across both, and the failure is named for the latest code.
Every other code, and each hook's failures, repeat only with themselves.
Run failure messages
A failed run's note reads The run failed: <message>. Where a run is refused
for more than one reason, the reasons are joined with ; .
| Message | Cause |
|---|---|
Runner image <image> can't be pulled. Error: <error> | The host could not pull the line's image and holds no copy of it. Unreadable registry credentials fail the pull even when the host holds a copy, unless the image is named by digest or the copy has no registry digest. Code unprepared. |
Runner image <image> is refused. <reason> | The host does not serve the image. The reasons are those the Runners page shows for the image. Code unprepared. |
Runner image <digest> no longer holds the <harness> harness. | The image changed and lost the station's harness. Code unprepared. |
The sandbox's egress didn't start. Error: <error> | See Egress start failures. Code unprepared. |
The runner couldn't prepare the run. Error: prepare: git clone <path>: <error>: <output> | A repository could not be cloned. Code unprepared. |
The runner couldn't prepare the run. Error: prepare: preparing <path> did not finish within 10m0s | The checkouts took longer than their time limit. Code unprepared. |
The runner couldn't prepare the run. Error: clear an earlier run's outcome: <error> | .spotlight/outcome.json from an earlier run could not be removed. Code unprepared. |
The runner couldn't prepare the run. Error: fetch attachment <id>: <error> | An attachment could not be written into the workspace. Code unprepared. |
The runner couldn't prepare the run. Error: hand over workspace: <error> | /workspace could not be given to the agent user. Code unprepared. |
The runner couldn't start the agent. Error: start agent: <error> | The harness did not start. Code exited. |
The runner couldn't start the agent. Error: resume the session: <error> | The harness could not reopen the task's earlier conversation. Code exited. |
The runner couldn't start the agent. Error: open a session: <error> | The harness could not open a new conversation. Code exited. |
harness_profile_unknown: Harness profile <profile> is not in this team. Station: <Station> | The station's profile names no profile of the team. Code refused. |
catalog_item_unknown: <reason> | The station names a catalog item that does not exist. Code refused. |
catalog_item_wrong_kind: <reason> | The station names a catalog tool as a skill, or a skill as a tool. Code refused. |
not_entitled: <reason> | The workspace's plan does not include a catalog item the station gives. Code refused. |
entitlement_lapsed: <reason> | The plan that included a catalog item the station gives has lapsed. Code refused. |
No runner was provisioned after <n> attempts. Last failure: <failure> | No push pool started a runner. See Provision call failures. Code unprovisioned. |
Hooks did not answer after <n> tries. Last error: <reason> | Its run.starting hooks went unanswered on every attempt. Code hooks-unanswered. |
A run whose runner is lost, or that times out, on its last attempt is noted
Spotlight gave up on the run after <n> attempts. The run did not finish., with the
failure runner-lost or timed-out, as its last attempt ended. The run's own
outcome reads No runner finished the run after <n> attempts., or
Sandbox stopped at the host's time limit. No runner finished the run after <n> attempts.
for a timeout, where <n> leaves out the attempts that were parked. See Inside a sandbox for
the phase each runner message comes from.
Config
When a task's run is queued at a station with a harness, it records the station's config as it stands then. A replay run records its candidate's config. A chat's run records none.
| Part | Holds |
|---|---|
| Prompt | The station's prompt. |
| Harness, model, effort | As the run was given them. |
| Profile | The harness profile's key, rules, MCP servers and native settings. Each MCP server's environment values are kept as SHA-256 digests. |
| Image | The line's runner image. |
| Repositories | Each repository of the line, with its setup command. |
| Skills | Each team and catalog skill given, with a SHA-256 of its files. Built-in skills are left out. |
| Tools | The tools given. |
| Decisions, memories | Whether the station drafts them. |
| Judge | The SHA-256 of the judge's rubric, its sample and its trust, when the station has a judge. The station's alerts are not part of the config. |
The run's config fingerprint is a SHA-256 of these parts. The task's title, description and marks are not part of it.
The config in force at a station shows under "Config" on the station's
"Overview" tab, and the station's view on Insights groups its runs by config
under "Runs by config"; see Station outcomes.
GET /api/teams/{teamId}/runs/{runId}/config returns a run's config and its
fingerprint, and, when the station's previous run with a recorded config ran
another config, that run under changedSince, with the changes since it.
The web app words each change as one phrase, in the names of the groups under "Runs by config" and in what a search candidate changes:
| Change | Phrase |
|---|---|
| Prompt | Prompt changed, Prompt added, Prompt removed |
| Harness, model, effort, image, profile | <Part> <was> → <now>, <Part> set to <now>, <Part> <was> removed |
| Profile settings | Profile <key> settings changed |
| Repository | Repository <name> added, Repository <name> removed, Repository <name> setup changed |
| Skill | Skill <key> added, Skill <key> removed, Skill <key> changed |
| Tool | Tool <name> added, Tool <name> removed |
| Decisions, memories | Decisions on, Decisions off, Memories on, Memories off |
| Judge | Judge added, Judge removed, Judge rubric changed, Judge sample <was> → <now>, Judge trust <was> → <now>, with sample and trust as percentages |
For example: Effort high → xhigh.
Starting point
Once its checkouts are ready, a task's run records where it started: the
workspace archive it restored, if any, and the commit each repository's
checkout is at. A later attempt of the same run does not replace what the
first attempt recorded. If the starting point cannot be recorded, the run
carries on and its start step reads
Where the run started was not recorded: <error>. A run with no starting point
cannot be replayed.
The archive is kept for 30 days after the run started; see
Workspace archives.
Spend
Once a run has ended and its journal has been digested, Spotlight records
what it spent. GET /api/teams/{teamId}/runs/{runId}/conversation returns it
under spend:
| Field | Holds |
|---|---|
cost, currency | What the run cost. |
inputTokens | Fresh input tokens. |
outputTokens | Output tokens. |
cacheReadTokens | Input tokens read from the prompt cache. |
cacheWriteTokens | Input tokens written to the prompt cache. |
Each count is added up over every turn of the attempts the run kept. A count
or cost the harness did not report is unknown, not zero, and is null.
The task trace shows a run's cost as "Cost" in the run's details, as $1.87,
or to four decimals under $0.01.
The Insights "Cost" tab adds the runs' costs up and gives each station's
"Cache read", "Cache write" and "Input" tokens under "Read from cache", where
a count the harness did not report reads not reported; see
Cost. There, a count under 1,000 is shown
whole, a larger one in thousands (3.1k) or millions (2.4M) to one decimal.
In the task trace, a span that reports its own token use shows "Tokens", its
input and output tokens together.
What each harness reports:
| Harness | Reports |
|---|---|
claude-code | All four counts. |
codex | Input, output and cache read, from the last request of each turn only, so its counts are lower than what the turn used. Cache write is not reported. |
cursor | Not confirmed. |
Judging
When a task's run ends at a station with a judge, the run is judged if all of these hold:
- It ended Completed or Stopped. Failed, abandoned, withdrawn and superseded runs are not judged.
- Its task still exists.
- It falls in the judge's
sample. Whether a run falls in it is worked out from the run's id, so it never changes. Asampleof 1 judges every run, and 0 none. - The rubric's scale can be read. A judge with claims always scores claim by claim.
- It has not been judged before. A run is judged once.
At most 4 runs are judged at once. The judge reads the task's title (up to 300 characters) and description (up to 4,000), the run's steps from every attempt, and the hunks of its file changes, each file compared from before its first change to after its last, with 3 lines of context.
| Evidence | Kept |
|---|---|
| Steps | The latest, up to 80,000 characters together. A step's text is cut at 4,000 characters, and what a tool was given and returned at 2,000 each. |
| Hunks | The first that fit in 60,000 characters together. A hunk is cut to its first 200 lines, followed by … <n> more lines, and each line at 500 characters. A hunk that does not fit is skipped, and the ones after it are still tried. A hunk is named <path>#<n>, counting from 1. |
The judge is told how many steps and hunks were left out.
The judge answers with a score on the rubric's scale and 1 to 5 reasons. Each reason is up to 1,000 characters and cites 1 to 10 steps, by their place in the run's journal, or hunks, by name. A reason citing a step or hunk the judge did not read is dropped.
A judge with claims answers each claim, in order, met or not, with 1 to 3 reasons of its own in the same form. It is told that a claim the steps and hunks do not show to hold is not met. Its score is the share of claims met, and each claim keeps its mark and its reasons.
A score is recorded against the rubric version: the SHA-256 of the rubric, the claims and the pass share together.
| Failure | When | Tried again |
|---|---|---|
no-evidence | The run left no steps and no hunks. | no |
refused | The model refused to answer. | up to 4 attempts, from 30 seconds to 10 minutes apart |
unreadable | The answer could not be read, gave more than 5 reasons, or a score off the scale. With claims: it did not answer each claim once, in order, or gave a claim no reasons or more than 3. | up to 3 attempts, from 30 seconds to 5 minutes apart |
ungrounded | Half the reasons or fewer cited what the judge read. With claims: half or fewer of any one claim's reasons. | up to 3 attempts, from 30 seconds to 5 minutes apart |
errored | Another error. | up to 4 attempts, from 30 seconds to 10 minutes apart |
A run whose judging failed shows no score.
Agreement
A judged run's people's score is the latest grade a person gave it on the judge's scale. A grade counts only for the rubric version it was given under: the version of the run's judge score, or else the station's rubric at the time. On a yes-or-no scale, a run no one graded takes its people's score from its next act: Yes when it was Closed, No when it was Sent back; failing those, Yes when its pull request merged and No when it closed unmerged. Approving or refusing an approval is never a next act. A 1 to 5 scale and claims use grades only.
Agreement is measured from the station's latest 500 judge scores, across rubric versions, grouped by version, from the runs with both scores. The measure decides whether the judge is trusted; the statistics are shown under "Statistics" on the station's view on Insights (see One station):
| Scale | Measure | Statistics |
|---|---|---|
| Yes or no | Cohen's kappa | "Cohen's kappa"; "F1 score": twice the runs both scored Yes, over the runs the judge scored Yes plus the runs people scored Yes; "Overall agreement": the share of runs where both scores are the same. |
| 1 to 5 | Spearman's ρ, with tied ranks averaged | "Pearson correlation"; "Spearman correlation"; "Mean absolute error": the mean absolute difference between the two scores, in points on the scale; "Root mean square error": the square root of the mean squared difference. |
| Claims | Cohen's kappa over every claim's marks, pooled across claims | "Cohen's kappa", "F1 score" and "Overall agreement", over the pooled claim marks, with Met in place of Yes. For each claim, its own Cohen's kappa, agreed share and count of runs. |
On claims, a run counts once toward the runs with both scores, however many
claims it has. Statistics are shown to two decimals, and "Overall agreement" as
a percentage. A statistic that cannot be worked out reads –.
Cohen's kappa, each claim's kappa and the two correlations each carry a reading, shown on hover with what the statistic measures. "F1 score", "Mean absolute error" and "Root mean square error" have no reading.
| Cohen's kappa | Reads |
|---|---|
| 0.81 or more | Almost perfect agreement |
| 0.61 to below 0.81 | Substantial agreement |
| 0.41 to below 0.61 | Moderate agreement |
| 0 to below 0.41 | Fair to slight agreement |
| Below 0 | Less agreement than chance |
| Pearson, Spearman | Reads |
|---|---|
| 0.9 or more | Very strong correlation |
| 0.7 to below 0.9 | Strong correlation |
| 0.5 to below 0.7 | Moderate correlation |
| 0 to below 0.5 | Weak correlation |
| Below 0 | Inverse correlation |
The judge's scores show only when the station's current rubric is trusted:
| Standing | When | Reads |
|---|---|---|
| Trusted | At least 10 runs have both scores, and the measure is at least the judge's trust. | Trusted. <measure>: <value> over <n> runs. Minimum required: <trust> |
| Hidden: too few runs | Fewer than 10 runs have both scores. | Hidden until more runs have scores from both the judge and people. Runs so far: <n> |
| Hidden: no agreement | The measure cannot be worked out, as when every score so far is the same. | Hidden. Agreement can't be measured while all scores are identical. |
| Hidden: below trust | The measure is below trust. | Hidden. <measure>: <value>. Minimum required: <trust> |
| Hidden: unreadable rubric | The rubric's scale cannot be read. | Hidden. The rubric names a scale that cannot be read. |
The measure reads "Cohen's kappa" or "Spearman's ρ", and the value and trust
are shown to two decimals.
While the judge is not trusted, a run's score and its reasons are both hidden. A score given under an earlier rubric version stays hidden, whatever the standing. A new rubric version starts with no runs that have both scores.
The station's view on Insights shows, under "How far to trust the judge", the
measure with the standing, then how many of the station's latest 200 ended
runs, less those withdrawn, have "Both scores", "Judge only", "People only"
and "Neither", each with <share> of the last <n>. A trusted judge's standing there reads
Trusted. Minimum required: <trust>. With no judged runs, it reads "No judged
runs yet" and
Judge agreement with people shows here per rubric once the judge scores runs.
To a person who may not read it, Only team members can see judge agreement.
Each rubric version then shows:
| Card | Shows |
|---|---|
| Statistics | The statistics above. Hovering over one shows what it measures, with its reading when it has one. |
| Score comparison | Runs counted by the judge's score, in rows, against people's, in columns. On claims, it counts claim marks. Shown when runs have both scores. |
| Distribution | How often each score was given, by the judge and by people, over runs with both scores. On claims, it counts the pooled claim marks. Shown when runs have both scores. |
| Trend over time | The judge's and people's scores by day, as share scored yes, by day, mean score, by day or share of claims met, by day. The judge's line shows only on the current rubric while it is trusted; otherwise its legend reads "Judge hidden". |
| By claim | For a judge with claims: "Claim", "Cohen's kappa", "Agreed" and "Runs" for each claim. On an earlier rubric version, a claim reads Claim <n>. |
"Rubric" picks the version when the judge has judged runs under more than one:
"Current version", or Version until <day>.
Judge and alerts
A work station's "Overview" tab shows its
judge under "Judge" and its
alerts under
Alert rules · <n>, read-only. "Judge" has the anchor #judge.
Alert rules · <n> shows only when the station has alerts.
Only people on the team, the workspace's owners and admins, and workspace
tokens with the read scope can read the judge's agreement with people.
Anyone else is answered 404. See Team outcomes.
A station with no judge reads "No judge" and Only people score runs here.
Otherwise "Judge" shows <sample>, as a percentage, with "of runs judged".
Then, when the agreement value and trust are both known, the measure and its
value on a bar marked at trust; see Agreement. Under it, a trusted
judge reads Trusted above <trust> over <n> runs, and any other judge its
standing. When the agreement cannot be read, the part reads
Trusted from <trust> agreement. A judge with claims lists them under
"Claims", numbered, then A run passes when it meets every claim., or
A run passes when it meets at least <share> of its claims. "Show rubric"
opens the rubric, and "Hide rubric" closes it, when the judge has one. "How it
agrees with people" opens the station's view on Insights at #judge; see
One station.
Alert rules · <n> lists the alerts in the order written, on a track that
scrolls sideways. When they do not all fit, arrows and dots page through them.
Each alert shows the measure's label, then the threshold: +<change> for
risesBy, −<change> for fallsBy, > <amount> for above and
< <amount> for below, with a change in a rate as <n> pts. Under it, how
many runs it compares: over <over> runs. With over set to 1, this reads
from one run to the next for risesBy and fallsBy, and in a single run
for above and below. See Station alerts for when an
alert goes off.
| Measure | Label |
|---|---|
sent_back_rate | Sent-back rate |
accepted_rate | Accepted rate |
checks_failed_rate | Checks-failed rate |
failed_rate | Failed rate |
merged_rate | Merged rate |
cost_per_run | Cost per run |
Each alert also has a one-line description: the accessible name of its slide,
and the title of the chart on its inbox item. It reads
Alerts when the <measure>, with the measure's label in lower case, then:
| Threshold | Reads |
|---|---|
risesBy | rises <change> or more over <over> runs |
fallsBy | falls <change> or more over <over> runs |
above | goes above <amount> over <over> runs |
below | drops below <amount> over <over> runs |
In the description, a change in a rate is in points, as 20 points, and an
amount of a rate is a percentage. Cost is a number, to two decimals, or four
under 0.01, with no currency on the slide and in the currency of the
station's runs on the inbox item. With over set to 1, the description ends
with from one run to the next or in a single run, as on the slide. For
example: Alerts when the sent-back rate rises 20 points or more over 20 runs.
Judge score
A judged run's score shows in the task trace.
The judge's score is Yes, No, <n> of 5 or <n> of <m> claims met, and it
is shown when the station's current rubric is trusted and the score was given
under it. While it is shown, the run's row in the trace carries it. Selecting
the row shows "Judged" in the run's details, then the judge's score, or
score hidden. People's grades follow as People said <grades>, with (you)
beside the reader's own. While the judge is not trusted, its standing is shown beneath. A
shown score lists its reasons under it, each with the steps and hunks it
cites, as Step <n> or the file's name and hunk header. Selecting a step shows
Step <n> with its tool or kind, and its text; selecting a hunk shows its
lines. On claims, each claim is listed with "Met" or "Not met" and its own
reasons under it.
A run's judge score and people's grades can be read only by people on the
team, the workspace's owners and admins, and workspace tokens with the read
scope. Anyone else is answered 404.
A run that was not judged shows none of this.
Station outcomes
Spotlight records how each task run at a work station turned out, as runs end and as people act. A run that ended before recording began has no outcome. Replay runs have none; see Replays.
Only people on the team, the workspace's owners and admins, and workspace
tokens with the read scope can read a station's outcomes. Anyone else is
answered 404. See Team outcomes.
The outcomes of the station's latest 50 ended runs, in the order they started, are shown in these places:
| Where | Shows |
|---|---|
| "Latest runs on this config", on the station's "Overview" tab | Tiles for "Runs", "Accepted", "Sent back", "Failed", "Cost" and "Median", over the config in force: the newest config that ran, in its newest group and in earlier groups of it that only groups that never started separate from it. The newest group of another config that ran ends it and is not counted. "Accepted", "Sent back" and "Failed" add of <n> runs and a bar of that share. "Runs" draws the endings of the newest group with a recorded config as a strip, oldest first; "Cost" and "Median" draw that group's costs and durations run by run, when they vary. The tiles are not shown until a run has run a recorded config, or when the outcomes cannot be read. |
| "Recent runs", on the station's "Overview" tab | The latest 8 runs of the newest group with a recorded config, whether its runs started or not, one row each. With none, "No runs on this config yet" and Runs on earlier configs are under All runs., or, when the station has no runs, "No runs yet" and Latest runs show here once tasks reach this station. To a person who may not read the outcomes, Only team members can see this station's runs. "All runs" opens the Insights Runs tab at the station. |
| "Version", in the station's "Config" on the "Overview" tab | The newest group with a recorded config, whether its runs started or not: Since <day>, the day its first run ended, or "In force", with Version <fingerprint>, the first 8 characters. "View pull request" opens the pull request a picked search candidate opened for that config, when there is one, and "History" opens the station's view on Insights. Before any such group, "None yet" and The station's first run sets the first version.; when the outcomes cannot be read, "Unknown" and Sorry, the station's runs couldn't be loaded. |
| "Runs by config", on the station's view on Insights | Every group, newest first. With no runs, "No runs yet" and Runs show here by config once tasks reach this station. To a person who may not read the outcomes, Only team members can see this station's runs. See One station. |
Runs are grouped by config, in the order they started: each group holds runs
that started one after another with the same fingerprint. A config that
returns after another starts a new group. A group is named by the first of its
changes from the last config that ran before it, or
Config <fingerprint>, the fingerprint's first 8 characters. The group of
runs before any recorded config reads Before the first run with a config. A
group whose runs never started on its config, because they failed before
starting or were withdrawn, is marked "Never started".
A run's outcome:
| Part | Values | Meaning |
|---|---|---|
| Ending | Completed, Failed, Stopped, Abandoned, Withdrawn, Superseded | How the run ended. Stopped: ended after it was asked to stop. Abandoned: given up after its runner was lost on the last attempt. Withdrawn: ended before it started. Superseded: failed after its task had moved on. |
| Failure | A failure code | Shown beside a Failed or Superseded ending. |
| Round | 1, 2, … | 1 plus the number of the task's earlier runs at the station that completed or failed. |
| Checks | Checks passed, Checks failed, Checks skipped | The station's check on the run's change. See Checks. |
| Next | Started, Parked, Closed, Answered, Sent back | The first thing a person did with the task in answer to the run, or after it. See the table below. |
| Delivery | PR open, Merged, PR closed | The state of the task's pull request. |
| Took, Cost | How long the run took and what it cost, from its journal. |
| Next | The person |
|---|---|
| Started | Moved the task to a work station. |
| Parked | Moved the task to the backlog. |
| Closed | Moved the task to the completed station. |
| Answered | Answered the conversation without a decision. |
| Sent back | Said something to a run of the task while no run held it, other than an answer to an open question that asks for no approval. |
Only Closed accepts a run, and only Sent back sends it back. Approving or refusing an approval is never a next act: it does not decide whether a run was accepted or sent back. When two acts come at the same moment, the first of Sent back, Closed, Answered, Parked and Started counts.
A tally counts:
| Figure | Counts |
|---|---|
| Runs | The group's runs that ran. The other figures, except Withdrawn, count only these. |
| Failed | Runs that ended Failed. |
| Stopped | Runs that ended Stopped. |
| Checks passed, Checks failed | Runs whose check passed or failed. |
| Accepted | Runs whose next act was Closed. |
| Sent back | Runs whose next act was Sent back. |
| Handed on | Runs that passed their task to another station. |
| Merged | Tasks the group's runs worked whose pull request merged. |
| Withdrawn | Runs withdrawn before they started. |
| Cost | The runs' cost together, when all are in one currency. |
| Median, p90 | The time the runs took. |
"Latest runs on this config" shows "Runs", "Accepted", "Sent back", "Failed",
"Cost" and "Median". "Runs by config" shows each group's shares, median, cost
and days. The other figures are returned only by
GET /api/teams/{teamId}/lines/{lineId}/stations/{stationId}/outcomes; the
Insights Stations tab
counts them in the same way.
On "Runs by config", each group shows "Checks" as <share> passed, of the
runs whose check passed or failed; "Next" as <share> accepted and
"Delivery" as <share> merged, of its runs; "Took", the median; "Cost"; and
"Ended", the day or days its runs ended. Each run row shows the ending,
linked to the run, then the failure, "Round", "Checks", "Next", "Delivery",
"Took", "Cost" and "Ended". A part with no value shows –.
The inbox
A person's inbox holds every conversation, open or resolved, on every task in the workspace that is assigned to them, and the notices put to them. Items are ordered by their latest note, notices by when they were raised, newest first, in the sections "Open" and "Resolved".
| Flag | Set when |
|---|---|
| resolved | A passed note has closed the conversation. |
| working | The conversation is open, a run holds the task, and the task does not await a person. |
| unread | An asked note in the conversation has not been read. Opening the item reads it. |
| failure | The conversation holds a failure. Shown as ×N once it has occurred N times, N being 2 or more. |
Only the task's assignee can answer an item, and only an open conversation can
be answered. An answer writes an answered note. The run that asked carries
on; if no run holds the task and the station has a harness, a run starts there,
taking up the harness session where the last run left it.
A resolved item shows no answer box. In its place it reads
The task is settled here. Nothing is left to answer. while the task stands at
the item's station, or The task moved on to <Station>. Nothing is left to answer.
Notices
A notice tells one person about their team's work, outside any task. Spotlight raises one when a search stalls. Nothing waits on it, and it cannot be answered.
| Field | Shows |
|---|---|
| Form | ended: work the person started has ended. advice: it carries on, with something they could fix. |
| Headline | What it is about, at most 200 characters. |
| Points | 1 to 20, of at most 1,000 characters each: what happened, then what it suggests. |
| Link | A page of its team, with a label naming it. |
| Flag | Set when |
|---|---|
| unread | Its person has not opened it, and it is not settled. |
| resolved | Its person opened it, which settles it. A settled notice shows under "Resolved" for 30 days after it was settled, then leaves the inbox. |
Its row shows a lightbulb, the headline, an unread dot, the day it was raised,
and the first point. Its page, /w/<slug>/inbox/notices/<noticeId>, shows when
it was raised, the headline, the points under "Details", and a button that
opens the link.
A notice reaches only its person, and only while they are a member of its
team's workspace. To anyone else, and once it has left the inbox, its page
reads No longer waiting for you. Settling is
POST /api/workspaces/{slug}/inbox/notices/{noticeId}/settle, answering
204, or 404 for a notice not put to the caller. Disbanding
a team forgets its notices.
Grades
When a judge score is recorded, 1 in 5 judged runs is also put to people to
grade, chosen from the run's id. The item reads
A judged run needs your grade. Grades check the judge. and, once graded,
Run graded.
It reaches the inbox of the task's assignee, or of everyone on the team when the task has no assignee. It never reaches a person who is neither on the team nor an owner or admin of the workspace. The item shows the run's conversation, then a card with the question, the judge's rubric when the station's judge has one, and the answers:
| Scale | Question | Answers |
|---|---|---|
| Yes or no, with a rubric | "Does this run meet the rubric?" | Yes, No |
| Yes or no, without one | "Is this a good run?" | Yes, No |
| 1 to 5, with a rubric | "How well does this run meet the rubric, from 1 to 5?" | 1 to 5 |
| 1 to 5, without one | "How good is this run, from 1 to 5?" | 1 to 5 |
| Claims | "Does the run meet each claim?" | Yes, No beside each claim |
A grade saves as soon as it is chosen, and on claims once every claim is
answered. While it saves, the card reads "Saving…". A claim whose words changed since the
judge marked it reads Claim <n>. The judge's score is not shown on it.
- Any person on the team, and any owner or admin of the workspace, can grade
a run. Anyone else is answered
404, and the card readsOnly team members can grade this run. One grade settles the item for everyone. - A person's later grade of the same run replaces their earlier one.
- A run is graded on its judge's scale, else on the scale of its station's rubric, else yes or no. On claims, a grade marks every claim met or not, and counts as the share met.
- Nothing waits on a grade. The task moves on as it would without one.
Memory reviews
An item is raised for a run when it proposes at least one
memory: drafted after the run at a station
with memories: true, or proposed by the agent with
spotlight remember.
The item reads A run proposed memories for the team. It settles, and reads
Proposed memories decided., once no memory of the run is left to decide:
every one is approved or rejected, or a later run's drafting has dropped the
undecided ones.
It reaches the inbox of the task's assignee, or of everyone who can see the
team when the task has no assignee. The item shows the run's conversation,
then a summary line that folds the cards and starts open. <proposed> is how
many memories the run proposed, and <open> how many are still undecided:
<agent> proposed <proposed> memories · <open> to review, or<agent> proposed 1 memory · 1 to review;<agent> proposed <proposed> memories · all decided, or<agent> proposed 1 memory · all decided.
<agent> is the run's harness, or "The agent" when it is not known. Each
memory is a card titled "Waiting for your approval" while undecided, then
Memory approved by <name> or Memory rejected by <name>, or
Memory approved or Memory rejected when the person who decided is not
known:
| Row | Shows |
|---|---|
| Memory | The memory. |
| Kind | The kind, with what it means on hover. |
| Applies to | "The whole team", "This line" or "This station". |
| Files | <paths> in <repository>, or the repository alone for a memory about the whole repository. Only for a memory kept for a repository. |
| Source | The notes and steps it was drawn from: each note with a person's name, "A teammate" or "From another tool", and each step with the run's harness or "The agent". |
An undecided card has a copy icon, an edit icon, "Reject" and "Approve". The edit icon changes the memory in place, at most 1,000 characters. While it is edited, "Reject" becomes "Cancel", which puts the memory back, and "Approve", which keeps it as edited, is off while the text is empty. A decided card keeps the copy icon, with "Reject" and "Approve" off.
- An approved memory stands, with a new handle. If it corrects a standing memory, that one is retired in its favour.
- A rejected memory is kept and never briefed. When a station drafts
memories, the drafter is shown the 20 newest memories the team rejected in
the last 90 days that apply where the run ran.
spotlight rememberdoes not check rejected memories. - A memory from outside the team cannot be approved together with other memories in one request. The web app approves one memory at a time.
- When a later run's drafting finds something to draw on, it drops every
undecided drafted memory of the task's earlier runs, whether or not it
proposes any itself. Memories proposed with
spotlight rememberstay. - Nothing waits on a review. The task moves on as it would without one.
Station alerts
Spotlight checks a work station's alerts when one of its task runs ends or settles, when the pull request of a task it worked closes, and when its line is changed or retired. A run counts toward a measure once it has settled for that measure:
| Measure | A run has settled when |
|---|---|
sent_back_rate, accepted_rate | Its next act is known. |
checks_failed_rate | Its check passed or failed. |
failed_rate | It ended. |
merged_rate | Its task's pull request merged or closed unmerged. |
cost_per_run | Its cost was worked out. |
Only runs that ran count, ordered by when they ended. An alert reads its
measure over the latest over settled runs and, for risesBy and fallsBy,
over the over settled runs before those. Settled runs are looked for among the
station's latest runs, 4 times as many as its alerts need at most. Until enough
runs have settled, or while the measure cannot be worked out, as for cost with
no priced runs, the alert is not checked.
| Threshold | Goes off when |
|---|---|
risesBy | The measure over the latest runs, less the measure over the runs before, is at least risesBy. |
fallsBy | The measure over the runs before, less the measure over the latest runs, is at least fallsBy. |
above | The measure is higher than above. |
below | The measure is lower than below. |
An alert goes off once per crossing. While it is open, later checks do not
raise it again. It recovers once it has stayed within its threshold for over
further settled runs: each check whose newest settled run is a new one, and
finds the alert no longer crossed, counts one, and a check that finds it
crossed again starts the count over. For over settled runs after it
recovers, a crossing reopens the same alert, and its inbox item, instead of
raising a new one. After that, a crossing raises a new alert.
If an open alert's amount or over is changed, the next check judges it by
the new values: crossed, it stays open with the new reading; otherwise the
check counts toward its recovery. An open alert whose measure or threshold is
removed from the station recovers at once, at the station's next check, and a
crossing after that raises a new alert. Changing or retiring a line checks
each of its stations, those it had and those it has. Deleting a task forgets
the alerts raised on it.
When an alert goes off, the task of the run that set off the check gets an
inbox item: the run that ended or settled, or, when a pull request closed, the
task's latest run at the station. The item reads
Station crossed an alert threshold. and, once the alert recovers,
Station back within its alert threshold. A reopened alert opens its item again. It reaches the task's assignee, or
everyone on the team when the task has no assignee. It never reaches a person
who is neither on the team nor an owner or admin of the workspace, and to
anyone else it is not found: its card reads
Only team members can see this alert. Nothing waits on it.
The item shows:
| Part | Shows |
|---|---|
| Heading | Alert raised at <station>, with the day it went off. |
| Reading | What the runs did, then the threshold, with cost in the currency of the station's runs. For a rate, <value> of the last <over> runs <did>., where <did> is "were sent back", "were accepted", "were merged", "failed" or "failed their checks". For cost, The last <over> runs cost <value> each on average. For risesBy and fallsBy, the comparison follows before the full stop: , up from <value> in the <over> before, or down from or the same as. Then The alert is set at a <change>-point rise. (or fall), a rise of <amount> for cost, or The alert is set at above <amount>. (or below). With over set to 1, the last run and the one before, with "was sent back", "was accepted", "was merged", "failed" or "failed its checks", and The last run cost <value> for cost. |
| Recovery | The station has been back under the threshold since <day>., or over in place of under for fallsBy and below, once the alert has recovered. |
| Chart | The measure over the over settled runs up to each run, from 2 × over runs before the run that crossed it to over runs after. Its legend names "Alert level" with the amount, for above and below; "Before" with the measure over the runs before, for risesBy and fallsBy; and "The run it went off at". It is drawn from the station's latest 500 runs; without that run among them, it reads There is no chart to show. The runs behind this alert are deleted. |
| Links | "Open the station", which opens its "Overview" tab; "Open in Insights", which opens the Stations tab at the station's row, #station-<stationId>, over the shortest period that covers the day it went off; and "Improve", while the station is on one of the team's lines, which opens the Agent page with the station attached to a new chat. |
Requests to connect
When a run calls a tool of an integration whose connection.holder is
requester, and the person who started the task has not connected it or the
host refused to renew their connection, nothing is sent. The task is assigned
to that person and put to them with an asked note of form connection:
I need to use `<tool>`, which acts as you in <integration>, and you haven't connected <integration> yet.
Connect <integration> and I will carry on. Nothing has been sent yet.For a refused renewal, <integration> refused to renew your connection, so it has to be made again takes the place of you haven't connected <integration> yet. <integration> is the integration's name.
Below the note, a card titled Connect <integration> offers "Connect" to that
person only; anyone else sees Waiting for <person> to connect <integration>.
Once they connect, the card reads Connected <integration>. Connecting answers it
with Connected <integration>., in their voice, on every task waiting on that
connection in the team; the run resumes as for any answer. A request is never
met with the team's connection. See
Holder.
Approvals
A run asks for approval when a tool with approval in its integration is
called, when the person whose connection a tool acts with set it to "Ask me
first", or when the harness profile's ask rules match a harness action. See
spotlight_integration,
Person rules and
spotlight_harness_profile.
| Standing | Meaning |
|---|---|
asked | Waiting on a person, as an inbox item. |
granted | Approved. The grant is spent by the call it allows. |
refused | Dismissed. The dismissal stands. |
In the inbox, the card's title reads "Waiting for your approval" while it is
asked, then "Request approved", "Request approved with changes" or "Request
dismissed". An undecided card offers copy and, for a tool call with inputs,
edit, beside "Dismiss" and "Approve". For a call that asks because of a
person's own rule, everyone else sees Only <Name> can approve this, with
"Approve" turned off and edit hidden.
When a run asks, Spotlight looks for the latest approval of the same subject at the same station. The subject is the tool with the same inputs, or the same harness permission. An approval granted with changed inputs also matches a call with the changed inputs.
| Latest approval | Result |
|---|---|
refused | The call fails with <tool> was dismissed, so nothing was sent., followed by the recorded reason. Nothing is asked. |
granted, not spent | The call goes ahead, and the grant is spent. Where the person changed the inputs, the tool runs with the changed inputs, and the tool.using hook and the run's steps see them. |
| none, or spent | The task is put to a person with an approval. A tool call fails with <tool> needs a person's approval before it runs, and they have been asked in their inbox. Nothing was sent. Stop here and end your turn: their answer comes back to you in this conversation. A harness permission request cancels the turn, and the run waits for the decision. |
- A grant is spent by one call. The next call asks again.
- An approval asked at one station does not count at another.
- Only a person can approve or dismiss. A workspace token cannot.
- A call that asks because of a person's own tool rule can be approved only by that person. See Person rules.
- An approval is decided once. A second decision is refused with
409. - An answer without a decision answers the conversation and leaves the approval undecided.
- A decision without words is recorded as
Approved.orDismissed..
Person rules
A tool of an integration whose connection.holder is requester acts with
the connection of the person who started the task, and follows that person's
rule for it as well as the team's. Of the two, the stricter wins:
| Team | Person | Result |
|---|---|---|
| Runs without asking | Allow | The call runs. |
| Runs without asking | Ask me first | The call waits, and only that person can approve it. The ask reads <Name> asks to approve each call made with their <Integration> connection., with The person when the person has no name. |
| Needs approval | Allow | The call waits, and the task's assignee can approve it. |
| Needs approval | Ask me first | The call waits with the team's reason, and only that person can approve it. |
| Either | Block | The call never runs. The call fails with <tool> was not used, so nothing was sent: <person> blocks it on calls made with their connection., or the person it acts as in place of <person> when the person has no name. |
- When a call waits because of a person's rule, the task is put to that person: they become its assignee.
- Approving it by anyone else is refused with
409. Whoever may answer the task can still dismiss it. - A grant from someone else does not cover such a call. It asks that person again.
- A person's rule applies only to calls made with their own connection, never to calls made with the team's. It stays when they disconnect and connect again.
- The run's list of tools shows the team's rule. The person's rule takes effect when the tool is called.
See Let a person ask first or block a tool.
Changed inputs
A person may approve a tool call with changed inputs, given as inputs with
approve: true when answering the inbox item.
- The changes name exactly the inputs the call was asked with. Inputs cannot be added or removed.
- Each value has the type of the input it replaces: any text for a
string, a finite number for anumber,trueorfalsefor aboolean. - A changed list or object is checked against the input's schema, and refused naming where it does not fit. See Input schemas.
- A value may not be blank where the call was asked with one.
- A harness permission and a dismissal take no changes.
- Changes that leave every input as it was are recorded as no change.
A refused answer is 409. See
Answering an approval.
The answer the agent reads is the person's words, or Approved., followed by
I changed what it runs with. Call it again as you asked and it runs with these instead: and every input of the changed call. A one-line value reads
<input>: `<value>` ; a longer one follows its name in a code block. Values
over 400 characters are cut, ending in ….
What the agent wants
A tool that requires approval also takes intent, given as --intent to
spotlight <tool>: what
the agent wants to do, as a phrase that completes "Spotlight would like to …".
It is kept with the approval and never passed to the tool.
- Spaces and line breaks become single spaces, and final full stops are dropped.
- It is cut to 200 characters, ending in
…. - A tool that requires approval may not declare an input named
intent. - A call made while a person still owes an answer keeps the first intent.
- A harness permission has none.
| Approval | Shown above the call |
|---|---|
| With an intent | Spotlight would like to <intent>. |
| Without one | Spotlight would like to use <name>. <because> |
| Granted with changed inputs | Spotlight asked to <intent>. You edited its inputs., with use <name> in place of a missing intent. |
Chats
A chat belongs to a team and to the person who started it. Only that person can list or read it. Each chat names its harness, and optionally a model, an effort, repositories and a work station it is improving. The harness and the repositories are fixed when the chat starts. A message may name another model or effort. The agent answers that message, and the messages after it, with the model and effort it names. A chat message holds up to 20,000 characters and at most 20 traces.
The Agent page's composer reads "What do you want to plan?", and a chat's
composer reads "Reply…". One picker in the composer names the harness, then the
model and effort when they are set. Under it, "Model" reads "Default" when no
model is set, and is turned off for a harness that offers no models. "Effort"
reads "Default" when no effort is set, and is left out for a harness that
offers no efforts. A new chat takes the harness picked last, or the
first one offered. In a chat, only the model and effort can be changed, and
the web app sends them with each reply. Repositories are picked below the
composer, which reads "No repositories", the repository's name, or
<n> repositories; in a chat they are read-only. Files are attached as on the
Run page. While the agent works, the chat shows
"The agent is working…", and with the composer empty "Send" becomes "Stop".
A chat started from a station's "Improve", "Ask the agent" or "Review with
the agent" names that station. Its agent is briefed to write the station's runs with
spotlight history and
to propose one change at a time, as a unified diff of the station's block in
the line's spotlight_line resource and of its prompt file,
prompts/<team>/<line>/<station>.md. The chat's first message names the
station as <station> on <line>, linked to the station. If the station has left the team's lines,
the agent is told so.
A setup chat, started from the setup task's "Set up configuration as code"
(see Setup task),
names a configuration repository and a folder in it. The repository joins the
chat's repositories. The folder is 1 to 200 characters, from the repository
root, each part made of letters, digits, ., _ and -, and never . or
... Its agent is briefed to survey the repositories and the team's tools and
to open a pull request with the team's Terraform there, with
setup-pull-request.
The chat keeps the pull request. Its runs reach Spotlight's API with
the run's Spotlight token.
Chat runs have priority over task runs. Which run goes first gives the reasons.
Proposals
The agent proposes a task with the
propose-task tool,
naming a line and a station by key.
| State | Card title | Next |
|---|---|---|
open | Waiting for your approval | accepted, or dismissed |
accepted | Creating the task | created, or back to open if the task cannot be created |
created | Task created | |
dismissed | Dismissed |
Each proposal shows in the chat as a card titled by its state. "Task" holds
the proposed title, linked to the task once it is created. "Station" reads
<Line> › <Station>, or "Station deleted" once the station is gone. "Details"
holds the description, when there is one. "Dismiss" and "Create task" act
only on an open proposal, and "Create task" is off while the station is
deleted. Copy <title> copies the title and the description.
An accepted proposal creates the task at the proposed station. A task created at a station with a harness starts a run at once.
Attachments
| Property | Value |
|---|---|
| Largest file | 25 MiB |
| Given with | A task, an answer, a message to a run, a chat message. |
| In the run | .spotlight/attachments/<id>/<name> in the run's workspace. |
| In the briefing | Listed under Attached, in the workspace:. |