acceptodds
Under review as a conference paper at ICLR 2027

When Passing Checks Are Not Enough: Separating Evidence Reports from Outcome Judgments in Coding Agents

Abstract

Coding agents can pass their own tests and still falsely claim completion. We distinguish reporting verification events accurately from judging whether those events justify confidence in success. We introduce an E/J completion interface and verdict training on agent-generated trajectories. Evidence reports (E) are supervised by execution records, while outcome verdicts (J) predict hidden evaluation outcomes. On SWE-bench Verified, an SFT baseline with this interface performs verification but remains overconfident in its verdicts. We train J on the baseline's trajectories with two targets: Hard-J uses observed outcomes, and Soft-J blends outcomes with a difficulty-conditioned estimate of evidence reliability. The two targets have complementary strengths. On fixed development histories, Hard-J raises AUC from 0.590 to 0.787, while Soft-J reduces expected calibration error from 0.294 to 0.083. On 72 held-out tasks, Hard-J makes the fewest false-positive verdicts after passing checks (28%, versus 39% for Soft-J and 83% for SFT), while Soft-J has lower calibration error and Brier score than independently calibrated Hard-J on every trajectory source, with comparable discrimination among passing-check episodes. Ablations and report interventions show that verdicts condition on reported evidence, and directly incentivizing positive self-reports induces reporting hallucinations, reinforcing the need to supervise process fidelity separately from outcome judgment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coding agents, agent self-evaluation, confidence calibration, uncertainty quantification, process supervision, outcome prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nhqHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates evidence reporting from outcome judgment in coding agents through an E/J interface. It trains verdicts using observed outcomes in Hard-J and a mixture of outcomes and estimated evidence reliability in Soft-J. Experiments on SWE-bench Verified examine report accuracy, calibration, false completion claims, and failure screening.

Critical strengths

The motivation is reasonable: passing the checks selected by an agent does not necessarily mean that the task is complete. The authors conduct training experiments and ablations to study this distinction, with improvements in completion judgments over the SFT baseline.

Critical weaknesses
  1. The contribution is incremental, and the added value of Soft-J is not sufficiently established. The method combines structured reporting with outcome supervision and a soft training target. Comparisons with existing confidence learning methods are limited, and the training experiments cover only one model family and one benchmark. Although the paper reports failure screening results, Table 17 omits Hard-J with independent calibration. This leaves the practical advantage of Soft-J over this stronger baseline unclear.

  2. The writing is simplistic, and the paper is not developed into a complete, well-written study. Key design choices receive limited explanation, especially why the model should generate evidence fields that can already be extracted from execution logs. The paper needs a clearer and more substantive account of how the interface, training objective, and practical benefits fit together.

Questions for authors

Why does Table 17 omit Hard-J with independent calibration? On the same trajectories and at the same false-positive rate, can Soft-J detect more failures than this baseline?

2 comments
AuthorsOpenReview ↗

Response to Reviewer nhqH

We thank the reviewer for the clear questions.

Table 17 and Hard-J with independent calibration. We added Hard-J to the failure-screening results (R5). Independent calibration is a two-parameter Platt transform of Hard-J's margin. It preserves the ranking, so at any fixed false-positive rate it flags the exact same episodes as Hard-J and has identical catch rates. We evaluate screening in two complementary settings, and at the same false-positive rate Soft-J catches more failures than calibrated Hard-J in both:

  • Same trajectories (the scorer in isolation). On the 90 fixed development contexts, every scorer reads identical histories and evidence prefixes, which answers the reviewer's question directly. Soft-J detects more failures at both budgets: 0.32 vs 0.23 at a 10% false-positive rate, and 0.51 vs 0.42 at 20%.
  • Each agent on its own rollouts (the deployment setting). The verdict is not an external monitor applied to arbitrary trajectories: the same agent generates the execution history, reports its evidence, and then judges its own outcome. The end-to-end question is therefore whether an agent flags failures in the trajectories it produces itself. Scoring another policy's trajectories isolates the scorer, as the fixed contexts already do, but does not measure this use. On the 72 held-out tasks, Soft-J leads again: 0.45 vs 0.27, and 0.59 vs 0.42.

Added value of Soft-J. Beyond failure screening, Soft-J:

  • is significantly better calibrated than independently calibrated Hard-J on the pooled held-out contexts (ECE +0.041 [+0.006, +0.075], Brier +0.018 [+0.003, +0.036]; R1), without requiring the separate calibration set (311 contexts from 600 additional rollouts) that Hard-J needs;
  • outperforms generic softening toward the base rate on all five metrics, on both development and held-out data (R3);
  • achieves lower ECE than Hard-J at every training checkpoint (R4).

Comparison with confidence-learning methods. Our comparators cover the main families:

  • prompting-based confidence (P(True) and verbalized confidence; Table 2);
  • post-hoc recalibration (Platt scaling of Hard-J, and logistic recalibration of SFT with parser features);
  • label smoothing toward the base rate (R3);
  • early stopping (R4).

Soft-J has the lowest ECE of all of these on the development set and of all those evaluated on the held-out contexts, and the lowest held-out Brier score among the cross-scored comparators.

Scope. Training experiments use Qwen3-32B on SWE-bench Verified, where hidden tests make false completion claims measurable. The diagnosis covers two model families (GLM-5.2 and Qwen). The revised Section 9 states this scope explicitly.

Why generate evidence fields that the parser can extract.

  1. Auditability of agent honesty (the goal): Running the parser gives E* (what commands ran), but only comparing the agent's own Ê against E* reveals whether the agent reports faithfully—for instance, exposing the all-positive over-claiming induced by positive-label supervision on E instead of burying it inside J.
  2. No cost to verdict quality (the check): Replacing Ê with E* (J|E*, R2) never improves any verdict model and significantly reduces Soft-J's held-out discrimination (ΔAUC −0.023, ΔGreen AUC −0.042).

To make this logic explicit in the paper, the revision opens Section 4 with a dedicated design-rationale paragraph connecting the two-claim interface, matched supervision signals, Soft-J calibration, auditability, and practical gains in false-claim reduction and failure screening.

Thank you for the detailed response and the revision. I have read the rebuttal carefully. However, my core concerns have not been resolved, and I believe the paper does not yet meet the bar for acceptance at ICLR. I will therefore maintain my score.

Reviewer_6tywOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies coding agents that pass their own checks yet falsely claim task completion. It proposes an E/J completion interface. At the end of an episode the agent emits four binary evidence tokens (test run, pre-edit reproduction, passing check after the last edit, check after the first edit), supervised by a deterministic execution-log parser, followed by a binary verdict token J that predicts the hidden SWE-bench Verified outcome (Sec. 4, pp. 3-4). A Qwen3-32B LoRA SFT baseline is trained with this interface. Training then continues only at the J position, on fixed SFT-policy trajectories, with one of two targets: hard outcome labels (Hard-J), or 0.3y + 0.7q (Soft-J), where q is a difficulty-conditioned evidence-reliability estimate built from leave-one-out same-issue success rates plus a cross-fitted evidence adjustment (Eq. 1, p. 4; Eqs. 3-5, p. 21). Main results: on 90 fixed development contexts (22 issues), Hard-J raises AUC from 0.590 to 0.787 and Soft-J lowers ECE from 0.294 to 0.083 (Table 2). On fresh rollouts for 72 held-out issues, false-positive generated verdicts among unresolved passing-check episodes fall from 25/30 (SFT) to 7/25 (Hard-J) and 11/28 (Soft-J), with similar task success (Table 5). In cross-scoring, Soft-J has lower ECE and Brier than independently calibrated Hard-J on all three trajectory sources (Table 3). A motivating diagnosis (App. B) finds that most false claims by a strong agent follow a passing post-edit check, and that prompting for more testing does not reduce them. Report interventions, target/interface ablations, and an incentive experiment complete the study.

Critical strengths
  1. A well-motivated decomposition backed by a convincing diagnosis. Of GLM-5.2's false completion claims, 79.5% and 88.3% follow a passing post-edit check (p.3; Table 6, p.17). Adding testing skills raises pre-edit checking by +22.0 pp [19.3, 24.8] but changes neither task success (+1.4 [−0.5, 3.4]) nor false claims per episode (−0.8 [−2.7, 1.1]) (Table 7, issue-clustered bootstrap). This supports the premise that more checking alone does not fix false completion claims and that the outcome judgment needs its own supervision.

  2. Careful evaluation, and a main effect that holds up. The 72-task set is issue-disjoint from the SFT and verdict data and is not used for tuning or checkpoint selection (pp.5, 15); cross-scoring holds the trajectory source fixed (Tables 3, 13); and the paper reports the cases where Hard-J wins (pp.6, 15). On-policy verdict training clearly improves over SFT: FC_gen falls from 25/30 to 7/25 and 11/28 (Fisher p ≤ 0.001 by my computation, ignoring clustering) at unchanged task success (54/53/52 of 216; Table 5), and Soft-J's own-rollout ECE falls from 0.203 to 0.111 (Table 12).

Critical weaknesses
  1. The paper's distinctive comparative claims lack statistical support and the simplest controls.

No confidence intervals are reported for Tables 2-5, 12 or 13, and every verdict variant is trained once (seed 0, p.20). Several of the comparisons that define the contribution are within noise:

  • Hard-J vs Soft-J FC_gen is 7/25 vs 11/28 (Fisher p ≈ 0.56).
  • Soft-J and independently calibrated Hard-J differ by 0.02-0.05 in ECE and 0.002-0.027 in Brier on held-out trajectories (Tables 3, 13). The three trajectory sources share the same 72 issues and scorers, so they are not independent replications.
  • The difficulty-conditioning ablations differ by a few of the 29 failed-green development contexts (roughly 5/29 vs 8/29 vs 10/29), and all soft-target variants have ECE between 0.083 and 0.098 on n = 90 (Table 4). At this sample size even a perfectly calibrated predictor has an expected 10-bin ECE of about 0.10 (roughly 0.07-0.14 between the 10th and 90th percentiles in a quick simulation), so these values cannot be told apart.

There is also no generic-softening control, such as constant-prior smoothing t = 0.3y + 0.7π or an early-stopped Hard-J. Since the paper itself attributes the calibration gain to soft targets in general (p.8), it is unclear what the difficulty-conditioned q adds. The separate calibration set behind the key "Hard-J + independent calibration" comparator gets one sentence on p.4 and one on p.21; its source issues, size, generating policy, base rate and calibrator form are not given. λ = 0.3 and α = 2 have no stated selection procedure or sensitivity analysis.

  1. The value of having the model self-report E is not isolated.

E* is computed from the observable history alone (pp.4, 18), so it is available at inference time, and the auditability argument already assumes the parser exists (p.9). Yet there is no control in which J is conditioned on parser-injected E*. J-first (p.8) serves as a no-E control, but it and the natural-language variant start from an SFT model trained only on the E-first token format, with frozen embeddings and 300 steps (Table 4 caption; p.20), so the interface comparison is confounded by initialization. Even so, the natural-language variant is best on No-green AUC (0.902) and within-issue ranking (92%) (Table 11). E quality also comes entirely from SFT: Soft-J has zero true-positive Erepro reports on the 72 tasks (12/12 over-claims, 15/15 under-claims; Table 8), and flipping a single E field from negative to positive still raises Soft-J's pJ by 0.12-0.16 with the history held fixed (Table 14). Without a J|E* control, the first contribution (separately self-reported evidence) and its claimed benefit for the verdict remain untested.

Questions for authors

Clarifications (answers could change my assessment):

  1. What exactly is the "separate calibration set" used for Hard-J + independent calibration: source issues, number of contexts, generating policy, base rate and calibrator features? Does it overlap the 333 verdict-training contexts?
  2. How were λ = 0.3, α = 2 and the 300-step checkpoint chosen? Were all of them chosen on the same 90 development contexts?
  3. For the 72-task Hard-J population, how many contexts, resolved episodes and failed-green episodes underlie Tables 12 and 13? The reported FC values (0.31, and 0.29 on the Soft-J source) appear inconsistent with the denominators in Tables 5 and 10.
  4. What is the training objective behind "positive token rewards" in the incentive experiment (RL with a token reward, or SFT on all-positive labels)? Were the two incentive conditions trained with matched budgets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 6tyw (1/2): weaknesses

We thank the reviewer for a careful and quantitative review. We address each point with new analyses (R1–R5 in the general response).

Statistical support. The revision reports 95% CIs for Tables 1–5, 8, 11–16 (Appendix E.8 for main-text tables) and Catch@20% in Table 17, plus paired issue-cluster bootstrap intervals for key comparisons (R1, Table 18). Each verdict variant has one run, initialized from the same 800-step SFT checkpoint with frozen embeddings and trained on the same contexts with the same seed, so each comparison changes a single factor. Our two main conclusions also hold across independently trained models:

  • Soft targets calibrate the verdict. All five separately trained soft-target models (Soft-J, the three target ablations in Table 4, and the constant-prior control in R3) have dev ECE in 0.083–0.101, inside the 10th–90th percentile range (0.07–0.14) from the reviewer's simulation for a calibrated predictor at n = 90; Hard-J's 0.208 lies far outside it. Held out, the two soft-target models evaluated there reach 0.123 and 0.145, versus 0.261 for Hard-J.
  • Verdict training reduces generated false completions. Both verdict-trained models reduce them relative to SFT (25/30 to 7/25 and 11/28), and each reduction is significant (R1).

Soft-J's lower ECE also holds at every checkpoint (R4).

Hard-J vs Soft-J generated FC. We agree 7/25 vs 11/28 does not separate the two (Δ +0.113 [−0.153, +0.349]) and do not claim it does. Both reduce generated false positives significantly relative to SFT: +0.553 [+0.307, +0.744] (Hard-J) and +0.440 [+0.208, +0.628] (Soft-J).

Soft-J vs Hard-J+cal on shared issues. R1 resamples the 72 issues jointly across the three trajectory sources, accounting for shared issues. On pooled held-out contexts, Soft-J's advantage is significant for ECE (+0.041 [+0.006, +0.075]) and Brier (+0.018 [+0.003, +0.036]), and also significant on SFT and Soft-J trajectories separately.

Ablations at n = 90, and what q adds. We agree n = 90 cannot separate the soft-target variants. We therefore tested what difficulty-conditioned q adds over generic softening on the 346 held-out contexts using the exact control proposed (R3: t = 0.3y + 0.7π, π = 139/333). Soft-J wins on all five metrics in both populations (held-out ECE 0.123 vs 0.145, Brier 0.212 vs 0.225, Green AUC 0.678 vs 0.643, failed-green FC 17/82 vs 22/82), with the largest gain on failed attempts that pass checks.

Early-stopped Hard-J (R4). Soft-J has lower ECE than Hard-J at every checkpoint on both dev and held-out sets. Selecting by dev ECE gives held-out ECE 0.173 for Hard-J (100 steps) vs 0.123 for Soft-J; Hard-J+cal (0.164) already beats every early-stopped Hard-J checkpoint.

Calibration set and hyperparameters. See Q1 and Q2 in part 2.

Isolating the value of self-reported E (J|E* and initialization).

  1. Direct control without initialization confound (R2). Keeping the model, format, and history fixed and teacher-forcing parser E* in place of Ê improves no verdict model, and Soft-J's held-out discrimination drops significantly (ΔAUC −0.023, ΔGreen AUC −0.042). By design, verdict training updates only J, so Ê inherits SFT's errors, such as Erepro over-claims; even so, Ê is at least as useful to J as E*.
  2. Format-ablation confound. We agree J-first and Natural language share the token-format SFT initialization, so we rely on J|E*—not those rows—to isolate the source of E. The natural-language variant's advantages (No-green AUC, within-issue ranking) are on secondary metrics; under the matched budget it trails on the main ones: ECE 0.155 vs 0.076, Green AUC 0.593 vs 0.688, and failed-green FC 25% vs 14% (Table 4).
  3. Why self-report E at all. E* only records what commands ran; it cannot audit whether the agent's own account is honest. Emitting Ê makes unfaithful claims (like E = (y,y,y,y) under positive-label supervision) visible rather than buried inside J.
  4. Sensitivity to flipped E fields. A verdict should respond to evidence; the question is how much. On the test, green and regression fields, Soft-J's increase is about a third of SFT's (+0.123 vs +0.399, +0.130 vs +0.348, +0.157 vs +0.441; Table 14). In the natural setting of R2, parser evidence raises Soft-J's held-out failed-green false positives only from 17 to 19, versus 64 to 72 for SFT.
AuthorsOpenReview ↗

Response to Reviewer 6tyw (2/2): questions

Q1. The separate calibration set contains 311 completion contexts from 600 new rollouts of the frozen SFT policy (6 per issue) on 100 SFT-demonstration issues that were not used in any development evaluation or checkpoint selection. Its base rate is 132/311 = 0.424, close to the development (0.411) and held-out (0.451) rates. These issues are disjoint from the 89 verdict-training issues (333 contexts), the 22 development issues, and the 72 held-out tasks. The calibrator is two-parameter Platt scaling on the verdict margin of the frozen 300-step Hard-J checkpoint, p = σ(a·m + b), fitted to binary outcomes, so ranking metrics are unchanged. We use a separate set because the 333 verdict-training contexts are Hard-J's own training data. The post-hoc baseline fitted on them (a logistic model on the margin plus the four E* features) is reported separately in Tables 2 and 4. These details are now in Appendix D.

Q2. Neither λ nor α was searched, on the 90 development contexts or on any other set.

  • λ = 0.3 was chosen once, without a search over λ. It puts most of the target weight on the reliability estimate q while keeping a minority weight on the observed outcome.
  • α = 2 treats the base-rate prior as two additional rollouts of the issue. Training issues have 1–8 verdict contexts (median 3), so a typical issue has 2–4 sibling rollouts after leaving one out. With α = 2, its success rate is shrunk by one third to one half toward the base rate, and issues without siblings fall back to the base rate.
  • Training length. All verdict continuations use the same 300-step training budget.

The available endpoints show that our conclusions do not hinge on these values:

  • λ. q alone (λ = 0) is worse than λ = 0.3 on all five metrics. The outcome alone (λ = 1, Hard-J) shows the expected trade-off: stronger discrimination but much worse calibration (ECE 0.208 vs 0.083).
  • Difficulty component. Both removing it (the no-difficulty variant, Table 4) and replacing the target with the base rate (the constant prior, R3) are worse than Soft-J on every metric.
  • Training length. Soft-J has lower ECE than Hard-J at every checkpoint (R4).

Finally, the main conclusions are confirmed on the 72 held-out tasks, which were never used for any selection.

Q3. The Hard-J probability population has 115 contexts: 53 resolved and 62 unresolved, of which 25 are failed-green and 37 failed-no-green. Hard-J's thresholded FC is 8/25 = 0.32; "0.31" in Tables 12 and 13 was a typo. On the Soft-J source, Hard-J+cal is 7/27 = 0.26, not 0.29. We have added the Hard-J row to Table 10 and corrected these values. The pooled values and all comparisons are unchanged.

Q4. Both incentive conditions are SFT, not RL. Their budgets are not matched, and the mismatch strengthens our conclusion:

  • Positive-label E supervision applies cross-entropy at the four E positions toward the positive label for every field, regardless of parser labels, for 300 steps at effective batch 8 (2,400 sequences; Table 9).
  • Trajectory reweighting is SFT on full assistant turns (3,233 rows; 400 steps at effective batch 16 = 6,400 sequences, ~2 epochs; LR 2e-5; max length 4,096) that upweights successful trajectories with parser-verified checks while keeping report labels truthful.
  • Both start from the same 800-step SFT checkpoint and train LoRA adapters only, with embeddings frozen, evaluated on the same 22 dev issues (6 rollouts each, 132 episodes).
  • Even with the smaller budget, positive-label supervision collapsed to E = (y, y, y, y) in all 86 emitted reports; even with 2.7× more updates, trajectory reweighting kept over-claim rates at or below 8% on audited fields.

The revision renames "positive token rewards" to "positive-label supervision on E" to avoid suggesting RL.

Thank you for the additional experiments and clarifications. The paired analyses and added controls address my main concerns. I will raise my rating to weak acceptance.

Reviewer_H6mJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the challenge of distinguishing evidence reports from outcome judgments in coding agents. The paper studies how coding agents can report verification evidence faithfully and express confidence in its task outcome. The authors introduce a completion interface for agent-generated trajectories that can be used to separately supervise process reports and outcome judgments, and the papers reports findings from experiments when training models on agent-generated trajectories. The key finding from this paper is an empirical observation that scientists should consider supervising process fidelity separate from outcome judgment in order to improve coding agents.

Critical strengths

The paper has a fairly rigorous setup for evaluation that uses fixed trajectories and cross-scores each policy's trajectories. This design is used to isolate improvements in verdict prediction from differences in agent behavior. The decision to compare Soft-J (combining outcomes with an estimate of evidence reliability) against an independently calibrated Hard-J (using observed outcomes directly) makes the evaluation setup stronger.

The motivation for the work is clearly explained (high clarity in the related works and motivation sections, which is a great strength).

Critical weaknesses

There have been several studies with agents and agent training before that have separated process evidence from outcome judgments. This paper would be stronger if the authors articulated the key novelty of the work. For example, is the key novelty the E/J interface? the SFT and controlled study on new kinds of information enabled by that interface?

The paper does not conclude (Section 9) with clear takeaways or findings for the research community. The conclusion emphasizes that the paper highlights the need to investigate process evidence in addition to outcome correctness, but this is already a well known distinction in the field. How would the E/J interface, and the training this enables, improve coding agents? The paper's takeaways would be stronger if there were a clear demonstration of how the findings led to practical improvements in coding agents.

The paper reports improvements without discussing any notion of statistical significance. It is currently difficult to interpret the differences reported in Table 3 and make judgments about SFT vs Soft-J vs Hard-J with the different scorers, since the numbers are quite similar across experimental settings.

Questions for authors

Table 1 mentions that the evidence reports can be incorrect. For example, Erepro has a 87% over claim rate. Reports are emitted in 51.6% of episodes. How does this low reliability in evidence reporting impact the quality of the downstream completion judgments?

How would Soft-J compare against an even simpler predictor that might use the prevalence of task success or other simple features based on the evidence? For example, would a parser operating on the evidence be able to create features that reach the performance of Soft-J? It is hard to assess the value of the proposed training method without simpler baselines as well.

What are the key takeaways from this paper for the research community? Based on the new empirical findings from this paper, when should practitioners use Hard-J vs Soft-J?

How sensitive is the performance of Soft-J to the number of rollouts available?

3 comments
AuthorsOpenReview ↗

Response to Reviewer H6mJ (1/2): novelty, takeaways and significance

We thank the reviewer for the positive assessment of our evaluation design and motivation.

Novelty. Distinguishing process from outcome is not new. Our core contribution is twofold: a trainable, auditable completion interface (E/J) inside the acting agent, and a controlled study of how separate supervision affects verification fidelity and outcome calibration. Concretely, this yields four pieces:

  1. The E/J interface. The agent reports four evidence tokens, supervised by a deterministic execution-log parser, and then a verdict token, supervised by hidden-test outcomes. Each claim is trained on the signal that can actually verify it.
  2. A diagnosis. Most false completion claims follow a passing post-edit check (79.5–88.3% for GLM-5.2), and adding testing skills raises pre-edit checking by 22 points but leaves false claims unchanged (−0.8 points [−2.7, +1.1]; Table 7).
  3. Soft-J. A verdict target that blends the observed outcome with a difficulty-conditioned estimate of evidence reliability.
  4. A decoupled evaluation. Fixed contexts and cross-scoring isolate verdict quality from differences in agent behavior.

Takeaways and practical value. The revised Section 9 is organized around four findings:

  1. More testing does not fix false completion claims (Table 7).
  2. Supervising the verdict separately cuts false completion claims after passing checks from 83% (25/30) to 28% (Hard-J, 7/25) and 39% (Soft-J, 11/28), at unchanged task success (25.0%, 24.5%, 24.1%); both reductions are significant (R1).
  3. Self-reported evidence is not a liability. Replacing it with parser evidence does not improve any verdict model (R2).
  4. Rewarding reports directly backfires. Supervising E toward positive labels makes every report claim full verification, whereas reweighting verified trajectories keeps over-claiming at or below 8%.

In deployment, an autonomous coding agent is only as trustworthy as its stopping signal: when its own checks pass but the task is unsolved, the SFT agent still declares "done" 83% of the time, so users cannot rely on that signal. Verdict training cuts these false declarations without sacrificing solve rate (finding 2), improves failure screening (at a 20% false-alarm budget, Soft-J flags 59% of failed held-out episodes versus 45% for SFT and 42% for Hard-J+cal; R5), and provides the calibrated p_J needed for downstream escalation or retry controllers.

Statistical significance (Table 3). R1 resamples the 72 issues jointly across trajectory sources. On the pooled held-out set, Soft-J's ECE and Brier advantages over independently calibrated Hard-J are significant (ECE +0.041 [+0.006, +0.075]; Brier +0.018 [+0.003, +0.036]), and they are also significant on SFT and Soft-J trajectories separately. Green AUC is statistically indistinguishable (+0.012 [−0.051, +0.083]). This matches our claim: comparable discrimination, better calibration for Soft-J. The revision reports 95% CIs across Tables 1–5, 8, 11–16 (Appendix E.8 for main-text tables) and Catch@20% in Table 17, plus paired CIs in Table 18.

AuthorsOpenReview ↗

Response to Reviewer H6mJ (2/2): questions

Q1: How do unreliable reports affect the verdict? They do not degrade it:

  1. Error-free parser evidence does not help J. In the J|E* control (R2), replacing each self-reported E with the parser's E* under a fixed history improves no verdict model, and every significant change is a degradation. For SFT, filling in under-reported passing checks raises held-out failed-green false positives from 64 to 72 of 82; for Soft-J, only from 17 to 19, because Soft-J learns to discount reported evidence (Table 14).
  2. Erepro's 87% over-claim has limited impact on J. Toggling Erepro shifts Soft-J's p_J by only +0.055 (versus +0.123 to +0.157 for the other fields; Table 14), and refitting the reliability target q without Erepro yields nearly identical targets (r = 0.99; Table 15).
  3. Low coverage (51.6%) reflects budget exhaustion, not unreliable reporting. Missing reports occur primarily when episodes time out or exhaust the step budget (46.0% timeout rate on the 104-task set, Appendix D; on the 72-task set, 179 of 200 missing reports across SFT and Soft-J hit the 80-step limit, with zero resolved episodes). Such episodes emit no completion claim and are excluded from report-conditioned verdict evaluation.

Q2: Simpler predictors. Soft-J outperforms both simple baselines the reviewer suggests:

  • Prevalence only. A constant predictor at the training base rate (0.417) is calibrated in aggregate but has no discrimination (AUC 0.5). Its Brier score is 0.242 on the development set and 0.249 held-out, versus 0.205 and 0.212 for Soft-J.
  • Parser features. Logistic regression on the four parser features E* plus the SFT verdict margin, fitted on the 333 training contexts (Table 2, "+ post-hoc calibration"), reaches AUC 0.569, ECE 0.131 and Brier 0.251 on the development set, versus 0.730, 0.083 and 0.205 for Soft-J. Post-hoc recalibration can only rescale what the SFT verdict already encodes: adding the four parser flags to SFT's margin does not raise its AUC (0.590 for SFT alone), whereas verdict training improves the verdict itself.

We also trained with softening toward the base rate as the target (constant prior, R3); Soft-J beats it on all five metrics.

Q3: When to use Hard-J vs Soft-J.

  • Hard-J gives a conservative binary verdict. It has the strongest discrimination on fixed contexts (AUC 0.787) and the fewest generated false positives (7/25), but it under-declares success: its positive-verdict rate among passing-check reports is 36%, against a 53% success rate.
  • Soft-J is the choice when the probability itself is used, for example to decide whether to escalate to a human or to screen failures at a fixed false-alarm budget. Its positive-verdict rate matches the success rate (52% vs 53%), it has the best held-out calibration (ECE 0.123 vs 0.164 for Hard-J+cal), and it catches more failures at fixed false-alarm rates (R5).
  • Extra data. Hard-J needs a separate calibration set (311 contexts from 600 additional rollouts) to become competitive on calibration; Soft-J does not.

Q4: Sensitivity to the number of rollouts. Soft-J needs no extra rollouts at inference: it scores a single trajectory. Rollouts matter only when building training targets, where each issue's leave-one-out success rate is shrunk toward the base rate with α = 2. Training issues have 1–8 verdict contexts (median 3), and issues without siblings fall back to the base rate, so q is defined for every context. Removing the per-issue component entirely (the no-difficulty variant, which uses zero sibling rollouts) is worse than Soft-J on every metric (Tables 4 and 11).

AuthorsOpenReview ↗

Follow-up on Rebuttal and Clarifications

Thank you again for your valuable feedback. We would greatly appreciate it if you could revisit our rebuttal and consider whether our clarifications adequately address your concerns and warrant an updated assessment. We would be happy to address any remaining questions or concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response (to all reviewers)

We thank the reviewers for detailed and constructive reviews. The main concerns were statistical support, missing controls (generic softening, parser-provided evidence, calibrated Hard-J in failure screening), and missing details. We address all of them; key results (R1–R5) are below. Hard-J+cal = Hard-J + independent calibration; dev = the 90 fixed development contexts; held-out (346) = the pooled cross-scored contexts on the 72 held-out tasks; FP = false positives on failed-green contexts (failed attempts whose own checks passed).

R1. Paired differences with 95% issue-cluster bootstrap intervals. Issues are resampled jointly across trajectory sources (2,000 resamples), keeping matched scorer outputs together; intervals are percentiles of each paired difference.

ComparisonPopulationΔ95% CI
ECE: SFT − Soft-JDev+0.211[+0.092, +0.318]
Generated FC: SFT − Hard-J72 tasks+0.553[+0.307, +0.744]
Generated FC: SFT − Soft-J72 tasks+0.440[+0.208, +0.628]
Generated FC: Soft-J − Hard-J72 tasks+0.113[−0.153, +0.349]
ECE: Hard-J+cal − Soft-JHeld-out (346)+0.041[+0.006, +0.075]
Brier: Hard-J+cal − Soft-JHeld-out (346)+0.018[+0.003, +0.036]
Green AUC: Hard-J+cal − Soft-JHeld-out (346)+0.012[−0.051, +0.083]

By trajectory source, the ECE and Brier advantages are also significant on SFT and Soft-J trajectories (Table 18).

R2. J|E* control: parser evidence in place of self-reported evidence. Each completion context is scored twice with the history fixed: once with the model's own evidence Ê, and once with the parser's E* teacher-forced in its place. Δ = (with E*) − (with Ê); * marks a 95% paired bootstrap CI that excludes 0. HO = held-out (346), with 82 failed-green contexts.

ScorerDev ΔAUCDev ΔECEHO ΔAUCHO ΔGreen AUCHO ΔECEHO FP
SFT+0.018+0.006+0.013+0.032−0.00664 → 72*
Hard-J+0.022−0.006−0.004−0.014+0.020*20 → 19
Soft-J+0.009−0.015−0.023*−0.042*−0.00317 → 19

No verdict model improves significantly with parser evidence; every significant change is a degradation (filling in missed passing checks makes SFT over-declare more).

R3. Generic-softening control (constant prior). Identical to Soft-J except that q is replaced by the base rate: t = 0.3y + 0.7π, π = 139/333 = 0.417. Soft-J is better on every metric in both populations.

Population · targetAUC ↑Brier ↓ECE ↓Green AUC ↑FP ↓
Dev · Soft-J0.7300.2050.0830.6765/29
Dev · const. prior0.7140.2160.1010.6106/29
Held-out · Soft-J0.7560.2120.1230.67817/82
Held-out · const. prior0.7310.2250.1450.64322/82

R4. Early-stopped Hard-J. Soft-J has lower ECE than Hard-J at every checkpoint (100 / 200 / 300 steps): on dev, 0.111 / 0.096 / 0.083 vs 0.156 / 0.215 / 0.208; held out, 0.163 / 0.139 / 0.123 vs 0.173 / 0.275 / 0.261.

R5. Failure screening including Hard-J. @k = fraction of failed episodes caught when the threshold allows k% false alarms on successful ones. Independent calibration is an order-preserving Platt transform of Hard-J's margin, so at any false-alarm rate it flags exactly the same episodes as Hard-J. On dev, every scorer reads the same trajectories; on the 72 held-out tasks, each model scores its own rollouts.

ScorerDev @10%Dev @20%72 tasks @10%72 tasks @20%
SFT0.280.340.380.45
Hard-J = Hard-J+cal0.230.420.270.42
Soft-J0.320.510.450.59

Revisions (marked in red in the updated PDF).

  1. R1–R4 are Tables 18–21 (Appendix E.7), and R5 is added to Table 17. 95% CIs now cover Tables 1–5, 8 and 11–17; for main-text Tables 1–4 they are in Appendix E.8 (Tables 22–25).
  2. Two thresholded FC values are corrected (Tables 12–13), and Table 10 adds a Hard-J row.
  3. Appendix D details the calibration set, the choice of λ and α, and the 300-step budget.
  4. Section 4 opens with a design rationale; Section 9 states explicit takeaways and the scope.
  5. "Positive token rewards" is renamed "positive-label supervision on E".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.