acceptodds
Under review as a conference paper at ICLR 2027

StopLoss: Teaching Agents to Predict Failure and Stop Early

Abstract

Coding agents keep working on runs that will ultimately fail verification, and asking a frontier model to watch for this is expensive in its own right. We introduce StopLoss, a small external monitor that learns failure recognition from completed trajectories and their verifier outcomes. Final outcomes label earlier local windows, so a small fine-tuned language model can score the most recent tool steps in a single forward pass, and a threshold calibrated on successful runs turns that score into an abort decision without changing the executor. Trained on GPT-family SWE-rebench trajectories and evaluated on SWE-bench Verified and a SWE-bench Pro subset with GPT and Claude executors, StopLoss is compared against six general-purpose judges reading identical windows. Outcome supervision lifts failure prediction from near chance to a useful signal at the first check, at one to two orders of magnitude lower cost per check than any tested judge. This translates into savings: relative to completing every task, StopLoss lowers cost per retained solve by 4.5% on Verified while preserving 99% of successful runs, and by up to 31% on the costlier Pro subset, and in all four settings its cost per retained solve is below that of every judge policy. Ranking transfers less well to the unseen Claude executor, and a threshold calibrated on one benchmark does not carry its false-kill rate to another. Outcome-trained small monitors can thus make failure recognition cheap enough for early stopping to pay for itself.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Coding Agents, Failure Prediction, Early Stopping, Agent Monitoring, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_svxKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces StopLoss, a small external monitor that predicts whether a coding-agent run will eventually fail and stops it early. LoRA-tuned 7–8B models are trained on SWE-rebench trajectory windows labeled by the final verifier outcome. At scheduled checkpoints, StopLoss reads an ABORT probability from the output logits and stops the run when a threshold calibrated on successful runs is crossed. The paper evaluates StopLoss on SWE-bench Verified and a three-repository SWE-bench Pro subset, with GPT and Claude executors. It compares against six frontier-model judges using cost per retained solve (CPS), which charges both monitoring cost and lost successes. StopLoss has lower CPS than every judge policy in all four settings and reduces CPS relative to full execution by 4.5% on Verified/Luna and by 24–31% on Pro. The paper also reports weak transfer to the Claude executor and poor threshold transfer across benchmarks.

Critical strengths

1. A cost-aware evaluation of early stopping.

CPS charges each policy for the checks it makes and for the successful runs it terminates. The break-even condition (Eq. 2) makes clear when stopping pays for itself. This leads to a useful observation that ranking metrics alone would miss. On Verified/Sonnet, Opus 5 has the best AUC in the study, yet it raises cost per solve by 34.7% because its monitoring bill exceeds the execution it saves (Table 2). This framing is likely to be useful beyond this paper.

2. Learning early failure signals from outcome supervision.

StopLoss uses final verifier outcomes to supervise short windows of recent agent actions, without requiring manual step-level annotations or access to the executor's internals. The learned signal is already useful at the first checkpoint: at tool step 5, AUC reaches 0.642 on Verified/Luna and 0.645 on Pro/Luna, compared with 0.494 and 0.450 for the untuned base. Training also improves prediction across all three model backbones (Table 3). The authors are also transparent about weaker transfer to Sonnet and the failure of thresholds to generalize across benchmarks.

Critical weaknesses

1. The incremental benefit of learned stopping over simple budget-based policies remains unclear.

Missing zero-cost stopping policies. The paper names step and spend caps as one of the two natural monitor families (Section 1). Appendix B.1 says that "step and spend caps remain as simple controls", but no cap policy appears in Table 2. Step count enters only as a ranking score in Table 3. That score is evaluated with tier AUC, which compares windows within quartiles of spent cost, so much of the length signal is removed by construction.

A calibrated cap is a strong alternative here for two reasons:

  • Failed runs are longer under both executors (Section 5.2).
  • The savings are concentrated in a few expensive failures. In the full Verified/Luna cohort, the three largest correct stops account for 65.8% of avoided execution cost (Section 5.1).

For reference, the concurrent FailFast–RestartSmart work (arXiv, August 2026) includes such a control: a step-count policy calibrated to the same 5% false-positive budget. That control recovers a substantial share of the learned monitor's savings (11.4% vs. 20.4% of tokens on its main policy). Without a calibrated step or spend cap, or the untuned base used as a stopping policy, it is unclear how much of StopLoss's CPS reduction comes from recognizing failure rather than from cutting off long runs.

Judge comparison. The advantage over judges appears to be mainly a cost effect. In three of the four settings, all three judges exceed StopLoss in tier AUC, and on Verified/Luna two of them match or exceed it (Table 2). All judges are run at high reasoning effort, which is their most expensive configuration (Section 3.2). The claim that StopLoss is more cost-effective than judging would be more convincing with at least one low- or no-reasoning judge.

2. The economic claim is broader than the per-setting evidence.

Per-setting evidence. The claim that StopLoss beats every judge "in all four settings" combines very different cases (Table 2):

  • Verified/Luna: the 4.5% saving comes from six correct stops and disappears once the monitoring charge reaches about 3.8× its estimate (Table 9).
  • Verified/Sonnet: StopLoss makes no stops at all. It beats the judges only by spending less on monitoring.
  • Pro/Luna: the 24.2% saving comes with 27.8% of successful runs killed, against a nominal 5% target.

Pro/Sonnet provides the strongest evidence of substantial cost savings with relatively high success retention, while the other settings show either modest gains, no actual stopping, or a large loss of successful runs.

Reported uncertainty. The introduction and conclusion state that StopLoss's advantage is supported by "simultaneous intervals excluding zero in nine of twelve comparisons." However, these intervals do not appear in any table or figure. Section 3.4 also promises a task-cluster sensitivity analysis for the Pro subset, which has only three repository clusters, in Appendix B, but I could not find it.

Both concerns appear addressable. The cap and untuned-base policies can be evaluated by replaying the existing trajectories, and the interval results presumably already exist.

Questions for authors
  1. Simple stopping-policy baselines (additional analysis; W1). A comparison with calibrated step and spend caps would help establish how much learned failure recognition contributes beyond simple budget limits. These policies could use the same ρ = 5% calibration rule and pools as StopLoss, with evaluation performed by replaying existing trajectories. Including the untuned base as a stopping policy would further clarify the benefit of training. If StopLoss achieves lower CPS at comparable success retention, this would substantially address my main concern.

  2. Judge configuration (additional analysis; W1). All reported judges use high reasoning effort, making monitoring relatively expensive. Could the authors include at least one lower-cost judge configuration, using low or no reasoning effort, to assess whether StopLoss's CPS advantage persists?

  3. Reported intervals (clarification; W2). The paper cites simultaneous intervals excluding zero in nine of twelve comparisons, but the intervals themselves are not shown. The task-cluster sensitivity analysis promised in Section 3.4 also appears to be missing. Please indicate where these results can be found or provide them.

  4. Pro/Luna operating point (clarification; W2). On Pro/Luna, StopLoss stops 40 of 144 successful runs (27.8%), despite the nominal 5% calibration target. It would be helpful to clarify how this higher false-kill rate affects the practical interpretation of the 24.2% CPS reduction, and whether benchmark-specific calibration on held-out Pro trajectories could retain meaningful savings at a lower stopping risk.

6 comments

Q1 / W1 — Calibrated caps and the untuned base as stopping policies

At ≥95% success retention StopLoss has lower cost per retained solve than a calibrated step cap and a calibrated spend cap in all four settings, by 2–5 points against the step cap, and the untuned base is significantly worse.

Thank you for the precise and constructive review. We replayed a calibrated step cap, a calibrated spend cap and the untuned base on the existing trajectories, with the same ρ = 5% calibration rule and pools as StopLoss. Under that rule the policies land at different realized retention, so their ΔCPS cannot be compared directly. StopLoss against the step cap: −4.5% at 99.0% retention against −5.2% at 94.8% on Verified/Luna, +0.2% at 100% against −4.1% at 98.8% on Verified/Sonnet, −24.2% at 72.2% against −10.1% at 93.1% on Pro/Luna, and −30.6% at 91.6% against −20.9% at 94.4% on Pro/Sonnet. Table A therefore compares all four policies at comparable success retention, the criterion you name:

Table A. Lowest ΔCPS (%) among operating points that retain at least 95% of successes. StopLoss performs the best.

SettingStopLossStep capSpend capUntuned base
Verified / Luna−9.6−5.5−6.3−3.9
Verified / Sonnet−10.4−6.8−10.0−0.7
Pro / Luna−10.0−7.8−9.4+0.2
Pro / Sonnet−24.3−19.0−20.0−5.5

The same picture holds without any test-set tuning, using thresholds from the original calibration pools at other nominal targets. On Verified/Luna, StopLoss at nominal 10% saves 9.2% at 95.8% retention, against 5.2% at 94.8% for the step cap. On Pro/Sonnet, StopLoss at nominal 5% saves 30.6% at 91.6% retention, against 24.7% at 87.9% for the step cap at nominal 10%. On Verified/Sonnet the two lie on roughly the same curve.

What we conclude, and will state in the paper:

  • A calibrated step cap recovers 57–78% of StopLoss's saving at ≥95% retention. This matches the Duration control in FailFast–RestartSmart, which recovers 56–75% of the learned monitor's token savings across its four policies. So we agree: most of the saving available at high retention comes from cutting long runs. The remaining 22–43% is what the learned signal adds.

  • The learned part is real. The untuned base, with the same readout and calibration, is significantly worse than StopLoss.

What StopLoss brings beyond length:

  • It acts where a cap cannot. At a fixed step every run has the same length, so step count carries no information, yet at step 5 the trained ensemble reaches ordinary AUC 0.642 / 0.604 / 0.645 / 0.645 (untuned base: 0.494 / 0.470 / 0.450 / 0.442).

  • It stops earlier. On Verified/Luna at comparable false-kill rates, StopLoss correctly stops 12 failed runs at a median of step 24 with 66% of the run's cost still unspent; the cap stops 9 at step 35 with 44% unspent. On Pro/Sonnet at the frozen thresholds, StopLoss correctly stops 21 failed runs at a median of step 23 with 79% of cost unspent, and the cap 18 at step 50 with 49% unspent; of the 14 failed runs both stop, StopLoss stops 13 earlier.

Q2 / W1 — Judge configuration

Measured low-reasoning judges. We ran the cheapest judge of each family, GPT-5.6 Luna and Claude Sonnet 4.6, with reasoning turned off on every checkpoint window of the stopping cohort and on the calibration windows. To avoid any impact from time-related factors on the API, we reran all the experiments here.

Table B. Frozen nominal-5% policies, whole stopping cohort. ↓ lower is better, ↑ higher is better.

SettingPolicyCost per check, ×10⁻⁴ $ ↓Net ΔCPS ↓Success retained ↑Failure capture ↑
SWE-bench Verified / GPT-5.6 LunaStopLoss2.4−4.6%99.0%5.0%
GPT-5.6 Luna judge, reasoning off3.2−4.6%89.5%12.6%
SWE-bench Pro / GPT-5.6 LunaStopLoss2.4−24.2%72.2%43.5%
GPT-5.6 Luna judge, reasoning off3.4−8.7%88.9%18.8%
SWE-bench Verified / Claude Sonnet 4.6StopLoss2.4+0.2%100%0.0%
Claude Sonnet 4.6 judge, reasoning off51+2.7%98.5%0.0%
SWE-bench Pro / Claude Sonnet 4.6StopLoss2.4−30.6%91.6%19.8%
Claude Sonnet 4.6 judge, reasoning off51−18.7%92.5%15.1%

Table C compares them at equal retention with the method of Table A.

Table C. Lowest ΔCPS (%) among operating points that retain at least 95% / 90% of successes. ↓ lower is better.

SettingStopLoss ↓Judge, reasoning off ↓
SWE-bench Verified / GPT-5.6 Luna−9.6 / −17.8−1.7 / −3.1
SWE-bench Pro / GPT-5.6 Luna−10.0 / −11.9+0.1 / +0.1
SWE-bench Verified / Claude Sonnet 4.6−10.4 / −14.5−3.7 / −3.7
SWE-bench Pro / Claude Sonnet 4.6−24.3 / −33.7−7.4 / −19.5

With reasoning turned off, the Luna and Sonnet judges still save less than StopLoss at equal success retention in all four settings, by 7–17 points at ≥95% retention.

For Luna this is not a price effect. With reasoning off a Luna check costs $3.2–3.4×10⁻⁴, close to the $2.4×10⁻⁴ of a StopLoss check, and it still saves less at equal retention. A Sonnet check with reasoning off costs $51×10⁻⁴, about 21 times a StopLoss check.

At equal success retention, even a free GPT judge would not beat StopLoss. As a check that does not depend on any price, we set the cost of every GPT judge call to zero and kept the real cost of StopLoss. We used the judge scores of the submission, took the best of the three GPT judges (Sol, Terra, Luna) in each setting, and compared at the same success retention (≥95% / ≥90%) as in Table C. On Verified/Luna the best free judge saves 8.8% / 10.3% and StopLoss saves 9.6% / 17.8%. On Pro/Luna the best free judge saves 4.4% / 9.5% and StopLoss saves 10.0% / 11.9%. So on GPT trajectories price is not what holds the judges back: a cheaper judge configuration can overtake StopLoss only if its scores are better than those of the high-reasoning judges.

Q3 / W2 — Reported intervals

Table D gives the twelve paired comparisons with simultaneous intervals: nine of the twelve cost differences exclude zero, and only Claude Opus 5 ranks better than StopLoss. We apologize for this oversight. The intervals and the task-cluster analysis for Pro are both below and will be in the appendix.

Table D. The twelve paired comparisons. Each row compares one judge with StopLoss in one setting. Every entry is judge minus StopLoss with a simultaneous 95% interval (within each metric family, as in Section 3.4). A positive Δ tier AUC means the judge ranks failing runs better than StopLoss; a positive ΔCPS difference means the judge policy costs more per retained solve than StopLoss. An interval that excludes zero means the difference holds at the 95% level jointly over the twelve comparisons; an interval that includes zero means the judge and StopLoss cannot be told apart on that metric. All twelve ΔCPS differences are positive, so every judge policy costs more per retained solve than StopLoss.

SettingJudgeΔ tier AUCΔCPS difference (points)
Verified / LunaGPT-5.6 Sol+0.011 [−0.044, +0.066]+121.3 [+85.6, +157.0]
GPT-5.6 Terra+0.001 [−0.054, +0.056]+43.1 [+22.1, +64.1]
GPT-5.6 Luna−0.022 [−0.085, +0.041]+12.8 [+2.8, +22.8]
Verified / SonnetClaude Opus 5+0.186 [+0.110, +0.262]+34.5 [+16.1, +53.0]
Claude Opus 4.6+0.041 [−0.026, +0.107]+11.5 [+9.1, +13.9]
Claude Sonnet 4.6+0.041 [−0.014, +0.095]+10.1 [+1.2, +19.1]
Pro / LunaGPT-5.6 Sol+0.043 [−0.022, +0.109]+83.5 [+74.3, +92.7]
GPT-5.6 Terra+0.037 [−0.040, +0.115]+49.1 [+41.2, +57.0]
GPT-5.6 Luna+0.025 [−0.060, +0.111]+24.2 [+10.1, +38.4]
Pro / SonnetClaude Opus 5+0.116 [+0.022, +0.210]+4.1 [−3.6, +11.8]
Claude Opus 4.6+0.036 [−0.051, +0.122]+13.3 [−2.5, +29.1]
Claude Sonnet 4.6+0.020 [−0.053, +0.092]+1.6 [−2.0, +5.2]

Task-cluster sensitivity on Pro, promised in Section 3.4. Pro has only three repositories, so we repeat the six Pro comparisons with tasks as the resampling unit. Ranking: Sol +0.043 [−0.021, +0.107], Terra +0.037 [−0.028, +0.103], Luna +0.025 [−0.043, +0.093]; Opus 5 +0.116 [+0.038, +0.195], Opus 4.6 +0.036 [−0.046, +0.117], Sonnet 4.6 +0.020 [−0.063, +0.102]. The conclusion is unchanged: only Opus 5 ranks better than StopLoss.

Reviewer_2YntOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes to train a light weight monitor to run parallel to the actual coding agent, and predict eminent task failure and then intervene to stop the run - saving costs. The paper constructs the training data for the monitor by taking offline trajectories with final task success labels - and considering short (overlapping) windows within them. The process then goes as follows:

(i) all windows from failed runs receive a label "ABORT"

(ii) all windows in successful runs (expect the two windows ending in last two steps) get the label of "CONTINUE".

(iii) Then, a SLM, takes as input the short, 3-length windows, and predicts the label.

During inference, the agent is stopped when the predicted failure score (or p("ABORT"|window)) exceeds a threshold calibrated on successful runs. Then, the paper shows results on SWE-bench verified and Pro and compares its failure prediction and cost savings with larger LLM-based judges

Critical strengths
  • A low cost monitor is highly beneficial in the real-world - as is proposed in this work.
  • A simple algorithm is proposed, making it easy to use for different domains.
Critical weaknesses
  • Errors might brew across large number of past observations, thus such a small window of only three most recent action-observation pairs within a 1600-token limit is limiting. The reason its limiting is that such a small past horizon may omit decisions needed to distinguish an unsuccessful trajectory from one that can still recover. No analysis is available to indicate why this past horizon length is sufficient.
  • Failure capture seems to be main contribution of the paper but these numbers are not great for the proposed monitor. Table 2 reports failure capture of only 5% and 0% on SWE-Bench verified for luna and sonnet. These results indicate the lack of effectiveness of the monitor.
  • The paper misses a lot of relevant works like AgentTracer [1], Aegis[2], "Efficient Forecasting of Task Failures in LLM Agents" [3] etc. Please discuss these works and clarify how the proposed method differs from them. This is a major drawback of this work - novelty of a work's method and findings can only be established in context to existing literature.

[1] AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?

[2] Aegis: Automated Error Generation and Attribution for Multi-Agent Systems

[3] Efficient Forecasting of Task Failures in LLM Agents through Adaptive Fault Injection

Questions for authors
  • why not use the complete word probability? e.g. lets say ABORT tokenizes to AB, OR, T. Currently, the paper (sec 2.2), computes the failure score by using the first token of each candidate answer. If the candidate answer spans multiple tokens, why not use the complete word probability for computing the failure score? Is there any empirical argument for it too?
  • In section 2.3, why start the checkpoint counter at 5?
  • Why have the authors limited their experiments to coding benchmarks? Is there any particular reason? The title of the work and intro is fairly general, but the experiments are only catering to the coding domain - making the claims of the work too broad for what is shown in experiments.
  • See the weakness for more of my comments on what needs to be addressed.
3 comments

Thank you for the review. We answer each point in order.

W1 — Is a three-step, 1,600-token window sufficient?

We keep 1,600 tokens because a 4,096-token window nearly doubles the cost of every check and brings no consistent improvement.

Table A. 1,600-token against 4,096-token window on the test runs. ↓ lower is better, ↑ higher is better. The 1,600-token rows use four seeds, so they differ slightly from Table 2.

SettingWindowMonitoring ($) ↓Tier AUC ↑Success retained (%) ↑ΔCPS (%) ↓
SWE-bench Verified / GPT-5.6 Luna1,600 tokens0.350.62398.4−5.1
4,096 tokens0.660.61599.0−3.0
SWE-bench Verified / Claude Sonnet 4.61,600 tokens0.350.556100.0+0.1
4,096 tokens0.640.587100.0+0.2
SWE-bench Pro / GPT-5.6 Luna1,600 tokens0.160.61672.9−25.7
4,096 tokens0.270.61872.9−30.6
SWE-bench Pro / Claude Sonnet 4.61,600 tokens0.240.57691.6−32.1
4,096 tokens0.410.59088.8−39.0

How the window is built (as already explained in Appendix A). The window is not a plain cut at 1,600 tokens. Each of the three most recent steps keeps the first 700 characters of its action, which is the agent's reasoning followed by the commands it sent, and the first 900 characters of its observation, which is what the tools returned. This way every step keeps part of its action and part of its observation, and one long tool output cannot crowd out the other steps. The 1,600-token limit comes on top of this and rarely applies: it is reached in 52 of the 8,051 test windows (0.6%).

Larger windows bring higher costs. We trained the same model in the same way with larger limits: 1,600 and 2,800 characters per action and observation, and 4,096 tokens overall. With the larger ones the monitor reads about twice as many tokens per check (2,153 on average instead of 1,139). The larger window makes monitoring 70–87% more expensive in every setting, and nothing improves consistently in return. Tier AUC changes by less than the 0.03 line of Section 3.4 in three of the four settings, and with single seeds the larger window is ahead for some seeds and behind for others in every setting.

Why a short window. It keeps the cost of a check fixed however long the run is; a monitor that reads the whole history pays more at every step. The window is also not all the monitor sees: its header gives the step count, and each action contains the agent's own reasoning, which carries forward part of what it has found so far.

W2 — Failure capture of 5% and 0% on Verified

We want to clear up a misunderstanding and clarify the claim of the paper. The paper does not aim to catch as many failures as possible. We aim to balance catching failures against the cost of catching them and the successful runs lost, with a monitor that is cheap to run. Cost per retained solve measures this balance, and it is lower for StopLoss than for every judge in all settings.

Stopping a failed run saves the rest of its cost. But every check costs money, and every successful run stopped by mistake is a solved task thrown away. Cost per retained solve therefore falls only if the execution cost avoided is larger than the monitoring cost plus the share of successful runs lost. Failure capture counts only the first of these three, so it can be raised simply by stopping more runs.

SWE-bench Verified leaves less to stop. We report Verified because it is the benchmark the community knows best. It is also old and public. OpenAI reported in February 2026 that frontier models can reproduce its reference fixes from memory (https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/). We saw this leakage ourselves with Claude Opus 5. As a judge it reads only the last three steps of a run, with no task statement and no issue number. For a failed Django run it still named the issue number of the task and the exact condition that the reference patch adds. A number cannot be worked out from the code, so the model recalled this task from memory. On Verified fewer runs fail (23–28% against 32–50% on Pro), and failed runs take about a third of all execution cost (32–36% against 45–65%), so there is less for any monitor to stop.

Low capture on Verified is a cautious setting, not a limit of the monitor. We set the threshold to protect successful runs. With a looser threshold the same monitor stops 10.1% of the failed runs on Verified/Luna (and 4.2% of the successful ones), or 34.5% (and 13.7%), and cost per retained solve falls by 9.2% and 19.5%.

On the newer benchmark the same policy stops far more failures. On SWE-bench Pro, which is newer and harder, the frozen policy stops 43.5% of the failed runs with Luna and 19.8% with Sonnet, and lowers cost per retained solve by 24.2% and 30.6%.

W3 — Related work

We want to clear up misunderstandings and clarify the claim of the paper.

  • AgenTracer (Zhang et al., arXiv 2509.03312) and Aegis (Kong et al., arXiv 2509.14295) address post-hoc failure attribution in multi-agent systems: given a complete, already-failed trajectory, they identify the responsible agent and step (AgenTracer) or agent and error mode (Aegis). Neither predicts the outcome of a run in progress, makes a stop decision, or accounts for cost. They answer "who and when, after failure"; we answer "will this run fail, and is stopping it now worth it".

  • Efficient Forecasting of Task Failures in LLM Agents through Adaptive Fault Injection (Sengar et al., 2026) is the closest. It uses the forecast to trigger extra compute, and the run always continues: on SWE-bench success rises from 0.67 to 0.69 while total cost rises from $8.48 to $10.17. StopLoss uses the forecast to stop the run and save its remaining cost. StopLoss is also simpler to build and cheaper per check: its labels are the verifier outcomes of ordinary completed runs, with no fault injection or rollouts, and it reads a short window instead of the whole history. This paper is also concurrent with our work: the conference version will appear at NeurIPS 2026 in December.

The submission's related work already covers the nearest neighbors in online early stopping, such as hidden-state probes, AgentStop, PrefixGuard, FailFast–RestartSmart, BAGEN and EET.

Q1 — First token versus the complete word

We tested the complete-word probability: it stops exactly the same runs at the same steps in all four settings, and it costs about 50% more per check. So we keep the first token.

Why the two agree. For our models each answer word is two tokens (CONT+INUE, DEL+IVER, AB+ORT), and the three first tokens are different, so the first token already tells which word the model chose. The model was trained to write the whole word, so after "AB" it continues with "ORT" almost surely: with probability 0.99997 on average, and never below 0.993. The probability of the complete word is essentially the first-token probability times this number, which is why the two are practically equal.

What we measured. We scored every window used in the paper with both readouts, for all five seeds. The two scores differ by at most 0.00003 and tier AUC by at most 0.00002. The thresholds are the same to five decimals, and the same runs are stopped at the same steps, so retention, false kills and failure capture in Table 2 are identical. This also holds at a stricter and a looser target (2% and 10%).

Why the first token. It needs one forward pass and no decoding. The complete word needs a second step, which adds 49.5% compute per check on the same GPU and would make ΔCPS 0.04–0.8 points worse, without changing any decision.

Q2 — Why the first check is at step 5

By step 5 the signal has formed and at most 11% of a failing run's cost has been spent.

Table B. Ordinary failure AUC at early steps, and cost already spent.

Step 2Step 3Step 4Step 5Step 8
AUC, Verified / Luna0.6040.6090.6180.6420.627
AUC, Pro / Luna0.5640.5860.5910.6450.625
AUC, Verified / Sonnet0.6010.6150.6380.6040.578
AUC, Pro / Sonnet0.5850.6030.6250.6450.641
Failed-run cost already spent (range over settings)1–3%1–4%—4–11%8–23%

In three of the four settings AUC rises through step 5. At step 5 at least 98.6% of runs are still executing and 89–96% of the cost of failing runs is still ahead; by step 8 that is 77–92%. The first steps are mostly repository exploration, and the window needs three completed steps of content. Moving the first check to step 8 or 11 leaves the frozen-policy result on Verified unchanged and costs 1–3 points on Pro/Luna.

Q3

We want to clarify that the method is general and nothing in it is specific to code. We ran the experiments on coding because that is where every run gets an automatic verdict and where failed runs are long and expensive.

The core of the method. StopLoss needs three things: completed runs recorded as steps of action and observation, a final verdict of success or failure for each run, and a threshold set on successful runs. The final verdict labels every earlier window of the run, a small model learns from these labels to score the most recent steps, and the run is stopped when the score passes the threshold. The monitor reads the steps as plain text and never looks inside the executor. Any agent whose runs are logged step by step and judged at the end can therefore be monitored in the same way, for example agents that browse the web, operate a computer or analyze data.

Why coding for the experiments. Training on outcomes needs a reliable automatic verdict for thousands of runs. Coding benchmarks provide one through hidden tests, at a scale we could afford. The abstract and the introduction of the submission already present the study as one on coding agents, and Section 7 states that the approach is general and that evaluation in further settings is the next step.

Reviewer_MPK6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents StopLoss, a small external monitor that decides whether to stop a coding agent’s run early if that run will ultimately fail verification. Previous works either focus on cutting off by budget or relying on an expensive general model for failure prediction. The authors argue that a trained small model can predict failure cheap while allows expensive yet successful run to complete. The small model is trained by completed trajectories and their verifier outcomes. Specifically, earlier local windows are supervised by final outcomes where the model learns to abort windows for failed outcome, and continue or deliver for successful outcomes.

The evaluation uses SWE-bench Verified and a subset of SWE-bench Pro, with GPT-5.6 Luna and Claude Sonnet 4.6 as executors. StopLoss is compared with general-purpose judges reading identical window, and its cost per retained solve is below that of every judge policy.

Critical strengths
  1. Construction useful stopping signals for local windows from completed trajectory outcome is a good idea, which is the key reason of success behind StopLoss.
Critical weaknesses
  1. One concern is the argument of lowering cost per retained solve is incomplete. The authors trained multiple models including Qwen2.5-7B-Instruct, Qwen3-8B, and Qwen2.5-Coder-7B-Instruct but they are all similar size of 7-8B. To complete the argument of having a small model monitor for cost efficiency, the authors should train even smaller models from 4B to 0.6B and understand the lowest cost possible with this method. It is possible that a very small model say 0.6B cannot accurately predict the failure after training, but this is a question that should be studied.

  2. Another concern is that the executors are very cheap models to be monitored (especially Luna). In this sense, the metric of cost per retained solve is essentially strongly penalizes expensive judges instead of a good balance between judge cost and saved executor costs. The authors may consider if the executors are expensive, whether it is worth it to have stronger expensive judge.

Questions for authors
  1. Could training smaller models from 4B to 0.6B further lower cost per retained solve? At what model size the failure prediction would be too low to be useful?

  2. Is the metric of cost per retained solve fair for a cheap executor like Luna? It might strongly penalize expensive judges instead of a good balance between judge cost and saved executor costs. Could the authors try stronger executors and see if stronger judges would make senes?

4 comments

Thank you for the two concrete questions. We designed the following experiment:

W1 / Q1 — Smaller monitors

With the same recipe, the 4B monitor is close to the 8B one except on Pro/Sonnet and the 0.6B monitor ranks clearly worse, while the lower price of a smaller model is worth at most 1.5 points.

How much cost is left to remove. With the five-member 7B ensemble, monitoring already costs 1.6% of execution on Verified/Luna, 0.9% on Pro/Luna, and 0.1–0.2% on the two Sonnet settings. If monitoring were free, ΔCPS would move from −4.5% to −6.0% on Verified/Luna, from −24.2% to −25.4% on Pro/Luna, and would not change on the Sonnet settings. So through its own price, a smaller model can lower cost per retained solve by at most 1.5 points.

Prediction degrades gradually with size, and within the range we tested we did not reach a size at which it stops being useful. At 4B it is close to 8B except on Pro/Sonnet. At 0.6B it is clearly weaker but still works: on GPT trajectories its tier AUC is 0.55–0.59, above chance, and on Pro it still saves 8–21%. Within the discussion period we could train three models (two seeds at 0.6B, one at 4B). The revision will add more seeds and model sizes to locate the point where prediction becomes too weak.

Table A. Prediction degrades gradually with size. Each column is one model with one seed, from large to small. Tier AUC is computed on all valid runs; ΔCPS is the change in cost per retained solve relative to full execution, including monitoring cost, at the frozen nominal-5% threshold. ↓ lower is better, ↑ higher is better.

SettingMetricQwen3-8B, seed 0Qwen3-8B, seed 1Qwen3-4B, seed 0Qwen3-0.6B, seed 0Qwen3-0.6B, seed 1
All settingsCost per check, ×10⁻⁴ $ ↓0.580.570.390.120.12
SWE-bench Verified / GPT-5.6 LunaTier AUC ↑0.6020.6040.5920.5760.586
ΔCPS (%) ↓−4.4−2.8−4.9−0.8−1.4
Success retained (%) ↑98.098.096.7100.099.7
SWE-bench Verified / Claude Sonnet 4.6Tier AUC ↑0.5490.5400.5370.5070.523
ΔCPS (%) ↓−1.3−1.8+0.0+0.0+0.0
Success retained (%) ↑99.799.799.7100.0100.0
SWE-bench Pro / GPT-5.6 LunaTier AUC ↑0.6210.6080.6190.5470.570
ΔCPS (%) ↓−31.9−13.7−34.8−9.4−16.4
Success retained (%) ↑74.377.875.787.581.9
SWE-bench Pro / Claude Sonnet 4.6Tier AUC ↑0.6100.5940.5820.5300.568
ΔCPS (%) ↓−50.0−30.7−21.1−8.1−21.3
Success retained (%) ↑86.089.794.498.196.3

A smaller model is cheaper per check, but this brings no clear further reduction in cost per retained solve. The 4B monitor ranks slightly below the same 8B seed and saves about as much in three settings; on Pro/Sonnet it saves less (21% against 50%). The 0.6B monitor ranks worse in every setting and stops fewer runs: it saves almost nothing on Verified, and 8–21% on Pro against 14–50% for the two 8B seeds.

W2 / Q2 — Is cost per retained solve fair with a cheap executor?

The GPT judges do not predict failure better than StopLoss, so a more expensive executor does not make them the better choice.

The aim of the paper is a monitor cheap enough to watch every run: StopLoss is a small trained model, and at the executors' real prices it has a lower cost per retained solve than all six frontier-model judges in all four settings (Table 2). You are right that with a cheap executor an expensive judge can hardly pay for itself, so this verdict depends on the executor's price. To test this we kept all recorded stops and changed only that price (Table B): ×1 is the paper.

Table B. ΔCPS (%) when the executor is made more expensive. ↓ lower is better. Stops and false kills are those of the paper at the frozen nominal-5% thresholds; only the executor's price changes.

SettingPolicy (false kills %)×1 (paper)×5×20
Verified / LunaStopLoss (1.0)−4.5−5.7−6.0
GPT-5.6 Sol (8.2)+116.8+15.1−3.9
GPT-5.6 Terra (10.5)+38.6−1.9−9.5
GPT-5.6 Luna (9.2)+8.4−4.6−7.0
Verified / SonnetStopLoss (0.0)+0.2+0.0+0.0
Claude Opus 5 (4.3)+34.7−2.8−9.9
Claude Opus 4.6 (0.9)+11.6+1.1−0.9
Claude Sonnet 4.6 (2.5)+10.3−0.7−2.8
Pro / LunaStopLoss (27.8)−24.2−25.1−25.3
GPT-5.6 Sol (13.2)+59.3−2.4−14.0
GPT-5.6 Terra (20.8)+24.9−3.5−8.8
GPT-5.6 Luna (12.5)+0.0−8.0−9.5
Pro / SonnetStopLoss (8.4)−30.6−30.6−30.6
Claude Opus 5 (4.7)−26.5−33.0−34.2
Claude Opus 4.6 (2.8)−17.2−21.7−22.5
Claude Sonnet 4.6 (16.8)−29.0−33.3−34.1

A stronger, more expensive judge is therefore worth it only when two things hold: runs are expensive, and the judge really predicts failure better than the small monitor. The GPT judges do not meet the second condition. Where their ΔCPS drops below that of StopLoss on Verified/Luna, they stop 8–10% of the successful runs against 1%; at equal success retention they stay behind even if their checks cost nothing. Opus 5 meets both conditions on Claude runs that cost a few dollars. For scale, Claude Fable 5.2, one of the most capable models available today, costs 2.7 times Claude Sonnet 4.6 per token. Even with an executor that expensive, Opus 5 would still cost more per solve than StopLoss on Verified, and would be cheaper only on Pro. We re-priced the recorded runs rather than run a new executor. A new executor would change the runs themselves, how long they are and how many succeed, together with the price, so the effect of price alone could not be seen. Re-pricing keeps everything else fixed and changes only the price.

Thank you for new the results and comparison. I do not have further questions now.

Thank you very much for your thoughtful review and constructive feedback. We sincerely appreciate the time you took to consider our response, and we are grateful that it addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.