Troubleshoot a run
Some of the fixes below need the host's log, others the line's Terraform, so have access to both.
The task shows In progress but nothing happens
Open the task. Under Activity, the latest run reads Queued: it is waiting for a runner to claim it. Open Settings from the workspace menu, then Runners, and look for the first of these that applies:
- No host is listed, or every host reads Not answering: no machine is taking work. Run a host, or read the log of the one that stopped answering.
- A host reads
Running <n> of <capacity>with every slot taken: it is full, and the run waits for a free slot. Raise the host's capacity or add a host. - No host's Harnesses include the station's harness (the Run page's composer shows that harness): use a runner image that includes it. See Harnesses.
- The host's Images reads
<n> refused: the host will not start runs from those images. Hover over it for each image and its reason, and fix the reason as in the table below. - The line or station names
runnersor an image: only a host that matches that placement can claim the run. Compare the names on the Runners page (Pool <key>,Trust <key>, the images) with the line. See Send a line's work to particular runners.
| Refusal reason | What to change |
|---|---|
The host's SPOTLIGHT_RUNNER_IMAGE doesn't admit this image. | Add the image, or a prefix that covers it, to SPOTLIGHT_RUNNER_IMAGE on the host. See Run a line in your own runner image. |
Image can't be pulled. Error: … | Check the image reference and the host's registry access. See Pull runner images from a private registry. |
Image can't be inspected. Error: … | Check that the image is built on the Spotlight runner image. See Run a line in your own runner image. |
Runner is on release <release>. Minimum version required: <release> | Rebuild the image on a newer runner release. |
Runner doesn't report its release number. Minimum version required: <release> | Rebuild the image on the Spotlight runner image, at that release or a newer one. |
The run failed with a message
A failed run puts an item in the assignee's inbox that starts "The run failed:" and gives the reason, and its Run page shows the same note. When the same failure happens again while the item is open, no new note is written: the item's count goes up instead. If such a run never reached the agent, its Run page reads "The run failed before the agent started.", and the message is on the inbox item. Run failure messages lists every message and its code.
| Message | What to change |
|---|---|
Runner image <image> can't be pulled. Error: … | The image reference or the host's registry credentials. See Pull runner images from a private registry. |
Runner image <image> is refused. … | Fix the refusal reason as in the table above. |
Runner image <digest> no longer holds the <harness> harness. | Put the harness back in the image, or change the station's harness. |
The sandbox's egress didn't start. Error: … | Read the host's log. See Egress. |
The runner couldn't prepare the run. Error: prepare: git clone <path>: … | Check that the GitHub App can read the repository and that the line names it correctly. See Point a line at your repositories. |
The runner couldn't prepare the run. Error: prepare: preparing <path> did not finish within 10m0s | Check the host's network to GitHub. See Limits and timeouts. |
The runner couldn't start the agent. Error: start agent: …, The runner couldn't start the agent. Error: open a session: … | If the rest of the message names authentication, check the harness key the host gives sandboxes. See Give hosts your harness keys. |
The runner couldn't start the agent. Error: resume the session: … | The harness could not reopen the task's earlier conversation. |
The runner couldn't prepare the run. Error: fetch attachment <id>: … | One of the task's attachments could not be fetched into the sandbox. |
No runner was provisioned after N attempts. Last failure: … | Check the push pool's provision endpoint. See Provision call. |
Hooks did not answer after N tries. Last error: … | Check the hook named in the message. See spotlight_hook. |
For a message that names neither a setting nor a guide, read the rest of it, then the host's log.
When you have fixed the cause, reply to the item to start a new run at the same station, as in Answer the inbox.
The run refused to start
A station's run fails before it starts when the station asks for something the team or workspace does not have. The message after "The run failed:" says which:
harness_profile_unknown: Harness profile <profile> is not in this team. Station: <Station>: create that harness profile for the team, or remove it from the station. See Limit what agents may do.not_entitled: …orentitlement_lapsed: …: the station gives a catalog item that the workspace's plan does not include, or that its plan included before the plan lapsed. Remove the item from the station.catalog_item_unknown: …orcatalog_item_wrong_kind: …: the station names a catalog item that does not exist, or names it as the wrong kind. Correct the reference on the station.
Refused runs are not retried. After changing the station, reply to the item.
The run was given up on
"Spotlight gave up on the run after N attempts. The run did not finish." means runners
claimed the run and then stopped reporting, or ran it past the host's run time
limit, N times in a row. Spotlight puts such a run back in the queue by itself
before it gives up. On Insights, the Runs tab shows its code under
Failure: runner-lost or timed-out.
Read the log of the host that claimed it around the time of each attempt.
Look for the host stopping or restarting, and for the warning
the sandbox ran past its time, which the host logs when it stops a sandbox
at its run time limit. See Limits and timeouts.
The inbox item says it failed N times
×N on an inbox row (the tooltip reads "Failed N times") means the same
failure happened again in the same conversation. A lost runner and a run time
limit count as the same failure here, named for the latest. Find the message in the
tables above and fix what it names before you reply again.
The agent worked without the repository's setup
A failing setup command does not fail the run; the agent starts without it.
On the Run page, open the Ran setup step. Its output ends with
Setup failed for <path>. Error: …. Fix the repository's setup command in the
line, then reply to the item, or send the work back to the
station to run it
again.
The agent ignored, missed or was misled by what the team knows
Open the Run page. Below the run's last step, select the Remembered icon in the toolbar. The toolbar shows once the run stops working, and the icon only when the run was given or proposed memories. The pane lists:
- Given to the agent: the memories the run was briefed with before it
started, and the ones the agent looked up with
spotlight recallor read from.spotlight/memory. Eachspotlight recallis also aspotlight.recallspan in the task's trace, with the words or handle it asked for and how many it found. - Proposed by this run: what the run put to the team to keep, and where each stands.
Then read it against what went wrong:
| What you see | Cause | What to change |
|---|---|---|
| The memory it needed is not in the pane, and not on the Memory page | Nobody kept it. | Write it on the team's Memory page, or turn on memories at the station. |
| It is on the Memory page but not in the pane, and it applies to this run's line, station, repository and files | The briefing holds at most eight memories, or 3,000 characters, and memories that are more specific, closer to the task's words, or warnings filled it. The agent did not look it up in .spotlight/memory. | Retire memories that no longer help, narrow where others apply, or say the memory in words the task will use. |
| It is on the Memory page but not in the pane, and it does not apply to this run | It doesn't reach this run's line, station, repository or files. | Correct it to apply more widely. |
| It is under Given to the agent and the agent acted against it | The agent didn't follow it. | Say it more plainly in a correction, or in the station's prompt. |
| A listed memory is itself wrong | The memory is wrong. | Select This was wrong beside it, say why it no longer holds, and select Retire. Later runs aren't told it. |
Each memory's handle links to it on the team's Memory page, which also shows how many runs were briefed with it or recalled it. A run counts as briefed once it is queued with the memory, even if it is withdrawn before it starts.
Related
- Run failure messages and Failure codes, with which codes are retried.
- Host settings