- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether coding-agent patches should be accepted, reviewed, or withheld before benchmark correctness is known. OpenCoderX combines candidate-bound uncertainty, provenance, and verification signals with a weighted risk score, isotonic calibration, and fixed routing thresholds. Experiments cover 128 candidates across two coding clients. Guard effects are statistically inconclusive, while all 60 calibration candidates receive the same risk score, preventing selective routing.
Critical strengths
- The distinction between failure prediction, selective routing, and intervention effects is useful: it clarifies which conclusions different experiments can support.
- The authors transparently report negative results and emphasize binding assessments to exact candidates before observing correctness. This supports an auditable evaluation protocol.
Critical weaknesses
- The motivation and added value of the proposed layer are insufficiently established.
- Coding agents already use tests to assess patches. The limitations of these checks do not by themselves justify an additional risk-scoring layer. The paper lacks a concrete case showing how this layer improves acceptance decisions over directly using verification results.
- The approach combines a manually weighted score, standard isotonic calibration, and fixed thresholds. Although intended as a diagnostic baseline, it is not compared with verification-only routing or alternative pre-outcome predictors, leaving its incremental contribution unclear.
- The thresholds are not justified through review or error costs, and benchmark correctness alone does not establish deployment-level trust. The practical benefit of the three-way policy therefore remains uncertain.
- The empirical evidence does not adequately explain the failure or establish broader conclusions.
- All 60 calibration candidates receive risk 0.35, yielding AUROC 0.5 and all-abstain routing. Section 5 suggests fallback patterns as a cause, but provides no component distributions, missingness rates, fallback frequencies, or ablations. The elementary observation that calibration cannot separate ties does not explain why they arose.
- Each client has only 17 matched tasks, with McNemar p-values of 1.00 and 0.50. Although the authors acknowledge this uncertainty, opposite point estimates do not establish the client-dependent effects suggested by the Section 4.1 heading.
- Calibration is fitted and evaluated on the same candidates. This supports the tied-score illustration, but does not validate prospective routing. The distinction between freezing raw scores and applying a label-fitted calibration map needs clarification.
- Signal normalization, fallback values, Guard triggers, and the execution consequences of routing decisions are inadequately specified. Concrete traces and quantitative audits of candidate binding and treatment delivery are also missing, limiting reproducibility and interpretation.
Questions for authors
- What evidence shows that the proposed risk score adds value beyond directly routing by test and verification results? Without such a comparison, why is an additional scoring layer warranted, rather than merely another way to aggregate the same evidence?
- Why do all 60 calibration candidates receive exactly 0.35? Do the recorded component values and fallback frequencies establish whether this is a feature-extraction or missing-data artifact? Without identifying the cause or comparing alternative predictors, what generalizable insight does this failure provide beyond the elementary fact that calibration cannot separate tied scores?
- The calibration map is fitted using the labels of the same candidates whose calibrated routes are reported. How is this consistent with the claim that routing is frozen before correctness is observed? Which reported routing results, if any, were produced without using the evaluated candidates’ labels to fit the calibration map?
- The paper lacks the AI use statement required by the ICLR 2027 author guidelines. Could the authors provide this statement?