- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper separates evidence reporting from outcome judgment in coding agents through an E/J interface. It trains verdicts using observed outcomes in Hard-J and a mixture of outcomes and estimated evidence reliability in Soft-J. Experiments on SWE-bench Verified examine report accuracy, calibration, false completion claims, and failure screening.
Critical strengths
The motivation is reasonable: passing the checks selected by an agent does not necessarily mean that the task is complete. The authors conduct training experiments and ablations to study this distinction, with improvements in completion judgments over the SFT baseline.
Critical weaknesses
-
The contribution is incremental, and the added value of Soft-J is not sufficiently established. The method combines structured reporting with outcome supervision and a soft training target. Comparisons with existing confidence learning methods are limited, and the training experiments cover only one model family and one benchmark. Although the paper reports failure screening results, Table 17 omits Hard-J with independent calibration. This leaves the practical advantage of Soft-J over this stronger baseline unclear.
-
The writing is simplistic, and the paper is not developed into a complete, well-written study. Key design choices receive limited explanation, especially why the model should generate evidence fields that can already be extracted from execution logs. The paper needs a clearer and more substantive account of how the interface, training objective, and practical benefits fit together.
Questions for authors
Why does Table 17 omit Hard-J with independent calibration? On the same trajectories and at the same false-positive rate, can Soft-J detect more failures than this baseline?
2 comments
Thank you for the detailed response and the revision. I have read the rebuttal carefully. However, my core concerns have not been resolved, and I believe the paper does not yet meet the bar for acceptance at ICLR. I will therefore maintain my score.
Response to Reviewer nhqH
We thank the reviewer for the clear questions.
Table 17 and Hard-J with independent calibration. We added Hard-J to the failure-screening results (R5). Independent calibration is a two-parameter Platt transform of Hard-J's margin. It preserves the ranking, so at any fixed false-positive rate it flags the exact same episodes as Hard-J and has identical catch rates. We evaluate screening in two complementary settings, and at the same false-positive rate Soft-J catches more failures than calibrated Hard-J in both:
Added value of Soft-J. Beyond failure screening, Soft-J:
Comparison with confidence-learning methods. Our comparators cover the main families:
Soft-J has the lowest ECE of all of these on the development set and of all those evaluated on the held-out contexts, and the lowest held-out Brier score among the cross-scored comparators.
Scope. Training experiments use Qwen3-32B on SWE-bench Verified, where hidden tests make false completion claims measurable. The diagnosis covers two model families (GLM-5.2 and Qwen). The revised Section 9 states this scope explicitly.
Why generate evidence fields that the parser can extract.
To make this logic explicit in the paper, the revision opens Section 4 with a dedicated design-rationale paragraph connecting the two-claim interface, matched supervision signals, Soft-J calibration, auditability, and practical gains in false-claim reduction and failure screening.