- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces StopLoss, a small external monitor that predicts whether a coding-agent run will eventually fail and stops it early. LoRA-tuned 7–8B models are trained on SWE-rebench trajectory windows labeled by the final verifier outcome. At scheduled checkpoints, StopLoss reads an ABORT probability from the output logits and stops the run when a threshold calibrated on successful runs is crossed. The paper evaluates StopLoss on SWE-bench Verified and a three-repository SWE-bench Pro subset, with GPT and Claude executors. It compares against six frontier-model judges using cost per retained solve (CPS), which charges both monitoring cost and lost successes. StopLoss has lower CPS than every judge policy in all four settings and reduces CPS relative to full execution by 4.5% on Verified/Luna and by 24–31% on Pro. The paper also reports weak transfer to the Claude executor and poor threshold transfer across benchmarks.
Critical strengths
1. A cost-aware evaluation of early stopping.
CPS charges each policy for the checks it makes and for the successful runs it terminates. The break-even condition (Eq. 2) makes clear when stopping pays for itself. This leads to a useful observation that ranking metrics alone would miss. On Verified/Sonnet, Opus 5 has the best AUC in the study, yet it raises cost per solve by 34.7% because its monitoring bill exceeds the execution it saves (Table 2). This framing is likely to be useful beyond this paper.
2. Learning early failure signals from outcome supervision.
StopLoss uses final verifier outcomes to supervise short windows of recent agent actions, without requiring manual step-level annotations or access to the executor's internals. The learned signal is already useful at the first checkpoint: at tool step 5, AUC reaches 0.642 on Verified/Luna and 0.645 on Pro/Luna, compared with 0.494 and 0.450 for the untuned base. Training also improves prediction across all three model backbones (Table 3). The authors are also transparent about weaker transfer to Sonnet and the failure of thresholds to generalize across benchmarks.
Critical weaknesses
1. The incremental benefit of learned stopping over simple budget-based policies remains unclear.
Missing zero-cost stopping policies. The paper names step and spend caps as one of the two natural monitor families (Section 1). Appendix B.1 says that "step and spend caps remain as simple controls", but no cap policy appears in Table 2. Step count enters only as a ranking score in Table 3. That score is evaluated with tier AUC, which compares windows within quartiles of spent cost, so much of the length signal is removed by construction.
A calibrated cap is a strong alternative here for two reasons:
- Failed runs are longer under both executors (Section 5.2).
- The savings are concentrated in a few expensive failures. In the full Verified/Luna cohort, the three largest correct stops account for 65.8% of avoided execution cost (Section 5.1).
For reference, the concurrent FailFast–RestartSmart work (arXiv, August 2026) includes such a control: a step-count policy calibrated to the same 5% false-positive budget. That control recovers a substantial share of the learned monitor's savings (11.4% vs. 20.4% of tokens on its main policy). Without a calibrated step or spend cap, or the untuned base used as a stopping policy, it is unclear how much of StopLoss's CPS reduction comes from recognizing failure rather than from cutting off long runs.
Judge comparison. The advantage over judges appears to be mainly a cost effect. In three of the four settings, all three judges exceed StopLoss in tier AUC, and on Verified/Luna two of them match or exceed it (Table 2). All judges are run at high reasoning effort, which is their most expensive configuration (Section 3.2). The claim that StopLoss is more cost-effective than judging would be more convincing with at least one low- or no-reasoning judge.
2. The economic claim is broader than the per-setting evidence.
Per-setting evidence. The claim that StopLoss beats every judge "in all four settings" combines very different cases (Table 2):
- Verified/Luna: the 4.5% saving comes from six correct stops and disappears once the monitoring charge reaches about 3.8× its estimate (Table 9).
- Verified/Sonnet: StopLoss makes no stops at all. It beats the judges only by spending less on monitoring.
- Pro/Luna: the 24.2% saving comes with 27.8% of successful runs killed, against a nominal 5% target.
Pro/Sonnet provides the strongest evidence of substantial cost savings with relatively high success retention, while the other settings show either modest gains, no actual stopping, or a large loss of successful runs.
Reported uncertainty. The introduction and conclusion state that StopLoss's advantage is supported by "simultaneous intervals excluding zero in nine of twelve comparisons." However, these intervals do not appear in any table or figure. Section 3.4 also promises a task-cluster sensitivity analysis for the Pro subset, which has only three repository clusters, in Appendix B, but I could not find it.
Both concerns appear addressable. The cap and untuned-base policies can be evaluated by replaying the existing trajectories, and the interval results presumably already exist.
Questions for authors
-
Simple stopping-policy baselines (additional analysis; W1). A comparison with calibrated step and spend caps would help establish how much learned failure recognition contributes beyond simple budget limits. These policies could use the same ρ = 5% calibration rule and pools as StopLoss, with evaluation performed by replaying existing trajectories. Including the untuned base as a stopping policy would further clarify the benefit of training. If StopLoss achieves lower CPS at comparable success retention, this would substantially address my main concern.
-
Judge configuration (additional analysis; W1). All reported judges use high reasoning effort, making monitoring relatively expensive. Could the authors include at least one lower-cost judge configuration, using low or no reasoning effort, to assess whether StopLoss's CPS advantage persists?
-
Reported intervals (clarification; W2). The paper cites simultaneous intervals excluding zero in nine of twelve comparisons, but the intervals themselves are not shown. The task-cluster sensitivity analysis promised in Section 3.4 also appears to be missing. Please indicate where these results can be found or provide them.
-
Pro/Luna operating point (clarification; W2). On Pro/Luna, StopLoss stops 40 of 144 successful runs (27.8%), despite the nominal 5% calibration target. It would be helpful to clarify how this higher false-kill rate affects the practical interpretation of the 24.2% CPS reduction, and whether benchmark-specific calibration on held-out Pro trajectories could retain meaningful savings at a lower stopping risk.
6 comments
Q2 / W1 — Judge configuration
Measured low-reasoning judges. We ran the cheapest judge of each family, GPT-5.6 Luna and Claude Sonnet 4.6, with reasoning turned off on every checkpoint window of the stopping cohort and on the calibration windows. To avoid any impact from time-related factors on the API, we reran all the experiments here.
Table B. Frozen nominal-5% policies, whole stopping cohort. ↓ lower is better, ↑ higher is better.
| Setting | Policy | Cost per check, ×10⁻⁴ $ ↓ | Net ΔCPS ↓ | Success retained ↑ | Failure capture ↑ |
|---|---|---|---|---|---|
| SWE-bench Verified / GPT-5.6 Luna | StopLoss | 2.4 | −4.6% | 99.0% | 5.0% |
| GPT-5.6 Luna judge, reasoning off | 3.2 | −4.6% | 89.5% | 12.6% | |
| SWE-bench Pro / GPT-5.6 Luna | StopLoss | 2.4 | −24.2% | 72.2% | 43.5% |
| GPT-5.6 Luna judge, reasoning off | 3.4 | −8.7% | 88.9% | 18.8% | |
| SWE-bench Verified / Claude Sonnet 4.6 | StopLoss | 2.4 | +0.2% | 100% | 0.0% |
| Claude Sonnet 4.6 judge, reasoning off | 51 | +2.7% | 98.5% | 0.0% | |
| SWE-bench Pro / Claude Sonnet 4.6 | StopLoss | 2.4 | −30.6% | 91.6% | 19.8% |
| Claude Sonnet 4.6 judge, reasoning off | 51 | −18.7% | 92.5% | 15.1% |
Table C compares them at equal retention with the method of Table A.
Table C. Lowest ΔCPS (%) among operating points that retain at least 95% / 90% of successes. ↓ lower is better.
| Setting | StopLoss ↓ | Judge, reasoning off ↓ |
|---|---|---|
| SWE-bench Verified / GPT-5.6 Luna | −9.6 / −17.8 | −1.7 / −3.1 |
| SWE-bench Pro / GPT-5.6 Luna | −10.0 / −11.9 | +0.1 / +0.1 |
| SWE-bench Verified / Claude Sonnet 4.6 | −10.4 / −14.5 | −3.7 / −3.7 |
| SWE-bench Pro / Claude Sonnet 4.6 | −24.3 / −33.7 | −7.4 / −19.5 |
With reasoning turned off, the Luna and Sonnet judges still save less than StopLoss at equal success retention in all four settings, by 7–17 points at ≥95% retention.
For Luna this is not a price effect. With reasoning off a Luna check costs $3.2–3.4×10⁻⁴, close to the $2.4×10⁻⁴ of a StopLoss check, and it still saves less at equal retention. A Sonnet check with reasoning off costs $51×10⁻⁴, about 21 times a StopLoss check.
At equal success retention, even a free GPT judge would not beat StopLoss. As a check that does not depend on any price, we set the cost of every GPT judge call to zero and kept the real cost of StopLoss. We used the judge scores of the submission, took the best of the three GPT judges (Sol, Terra, Luna) in each setting, and compared at the same success retention (≥95% / ≥90%) as in Table C. On Verified/Luna the best free judge saves 8.8% / 10.3% and StopLoss saves 9.6% / 17.8%. On Pro/Luna the best free judge saves 4.4% / 9.5% and StopLoss saves 10.0% / 11.9%. So on GPT trajectories price is not what holds the judges back: a cheaper judge configuration can overtake StopLoss only if its scores are better than those of the high-reasoning judges.
Q3 / W2 — Reported intervals
Table D gives the twelve paired comparisons with simultaneous intervals: nine of the twelve cost differences exclude zero, and only Claude Opus 5 ranks better than StopLoss. We apologize for this oversight. The intervals and the task-cluster analysis for Pro are both below and will be in the appendix.
Table D. The twelve paired comparisons. Each row compares one judge with StopLoss in one setting. Every entry is judge minus StopLoss with a simultaneous 95% interval (within each metric family, as in Section 3.4). A positive Δ tier AUC means the judge ranks failing runs better than StopLoss; a positive ΔCPS difference means the judge policy costs more per retained solve than StopLoss. An interval that excludes zero means the difference holds at the 95% level jointly over the twelve comparisons; an interval that includes zero means the judge and StopLoss cannot be told apart on that metric. All twelve ΔCPS differences are positive, so every judge policy costs more per retained solve than StopLoss.
| Setting | Judge | Δ tier AUC | ΔCPS difference (points) |
|---|---|---|---|
| Verified / Luna | GPT-5.6 Sol | +0.011 [−0.044, +0.066] | +121.3 [+85.6, +157.0] |
| GPT-5.6 Terra | +0.001 [−0.054, +0.056] | +43.1 [+22.1, +64.1] | |
| GPT-5.6 Luna | −0.022 [−0.085, +0.041] | +12.8 [+2.8, +22.8] | |
| Verified / Sonnet | Claude Opus 5 | +0.186 [+0.110, +0.262] | +34.5 [+16.1, +53.0] |
| Claude Opus 4.6 | +0.041 [−0.026, +0.107] | +11.5 [+9.1, +13.9] | |
| Claude Sonnet 4.6 | +0.041 [−0.014, +0.095] | +10.1 [+1.2, +19.1] | |
| Pro / Luna | GPT-5.6 Sol | +0.043 [−0.022, +0.109] | +83.5 [+74.3, +92.7] |
| GPT-5.6 Terra | +0.037 [−0.040, +0.115] | +49.1 [+41.2, +57.0] | |
| GPT-5.6 Luna | +0.025 [−0.060, +0.111] | +24.2 [+10.1, +38.4] | |
| Pro / Sonnet | Claude Opus 5 | +0.116 [+0.022, +0.210] | +4.1 [−3.6, +11.8] |
| Claude Opus 4.6 | +0.036 [−0.051, +0.122] | +13.3 [−2.5, +29.1] | |
| Claude Sonnet 4.6 | +0.020 [−0.053, +0.092] | +1.6 [−2.0, +5.2] |
Task-cluster sensitivity on Pro, promised in Section 3.4. Pro has only three repositories, so we repeat the six Pro comparisons with tasks as the resampling unit. Ranking: Sol +0.043 [−0.021, +0.107], Terra +0.037 [−0.028, +0.103], Luna +0.025 [−0.043, +0.093]; Opus 5 +0.116 [+0.038, +0.195], Opus 4.6 +0.036 [−0.046, +0.117], Sonnet 4.6 +0.020 [−0.063, +0.102]. The conclusion is unchanged: only Opus 5 ranks better than StopLoss.
Q1 / W1 — Calibrated caps and the untuned base as stopping policies
At ≥95% success retention StopLoss has lower cost per retained solve than a calibrated step cap and a calibrated spend cap in all four settings, by 2–5 points against the step cap, and the untuned base is significantly worse.
Thank you for the precise and constructive review. We replayed a calibrated step cap, a calibrated spend cap and the untuned base on the existing trajectories, with the same ρ = 5% calibration rule and pools as StopLoss. Under that rule the policies land at different realized retention, so their ΔCPS cannot be compared directly. StopLoss against the step cap: −4.5% at 99.0% retention against −5.2% at 94.8% on Verified/Luna, +0.2% at 100% against −4.1% at 98.8% on Verified/Sonnet, −24.2% at 72.2% against −10.1% at 93.1% on Pro/Luna, and −30.6% at 91.6% against −20.9% at 94.4% on Pro/Sonnet. Table A therefore compares all four policies at comparable success retention, the criterion you name:
Table A. Lowest ΔCPS (%) among operating points that retain at least 95% of successes. StopLoss performs the best.
The same picture holds without any test-set tuning, using thresholds from the original calibration pools at other nominal targets. On Verified/Luna, StopLoss at nominal 10% saves 9.2% at 95.8% retention, against 5.2% at 94.8% for the step cap. On Pro/Sonnet, StopLoss at nominal 5% saves 30.6% at 91.6% retention, against 24.7% at 87.9% for the step cap at nominal 10%. On Verified/Sonnet the two lie on roughly the same curve.
What we conclude, and will state in the paper:
A calibrated step cap recovers 57–78% of StopLoss's saving at ≥95% retention. This matches the Duration control in FailFast–RestartSmart, which recovers 56–75% of the learned monitor's token savings across its four policies. So we agree: most of the saving available at high retention comes from cutting long runs. The remaining 22–43% is what the learned signal adds.
The learned part is real. The untuned base, with the same readout and calibration, is significantly worse than StopLoss.
What StopLoss brings beyond length:
It acts where a cap cannot. At a fixed step every run has the same length, so step count carries no information, yet at step 5 the trained ensemble reaches ordinary AUC 0.642 / 0.604 / 0.645 / 0.645 (untuned base: 0.494 / 0.470 / 0.450 / 0.442).
It stops earlier. On Verified/Luna at comparable false-kill rates, StopLoss correctly stops 12 failed runs at a median of step 24 with 66% of the run's cost still unspent; the cap stops 9 at step 35 with 44% unspent. On Pro/Sonnet at the frozen thresholds, StopLoss correctly stops 21 failed runs at a median of step 23 with 79% of cost unspent, and the cap 18 at step 50 with 49% unspent; of the 14 failed runs both stop, StopLoss stops 13 earlier.