acceptodds
Withdrawn from ICLR 2027

Trust Before Autonomy: Pre-Outcome Selective Routing for Coding Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding agents act before their final correctness is known. We study pre-outcome selective autonomy: whether an exact software change should proceed automatically, be reviewed, or be withheld using only evidence available before grading. OpenCoder X binds the candidate to uncertainty, evidence provenance, verification status, and run identity while enforcing the boundary y not in z between pre-outcome information and the eventual benchmark label. A frozen policy then maps that record to ACCEPT, REVIEW, or ABSTAIN. Across 128 labeled SWE-bench Verified candidates, two failure modes emerge. In a matched 68-candidate development pilot, Guard changes resolved-task rate from 12/17 to 13/17 for the Codex client but from 13/17 to 11/17 for the Claude Code client; paired inference is inconclusive, and the effects differ in sign. On a disjoint 60-candidate calibration split, both clients resolve 20/30 tasks, yet every candidate receives raw failure risk 0.35 (AUROC 0.50; Brier 0.2225). Isotonic calibration maps the single support point to failure probability 1/3, after which the frozen policy abstains on every candidate. The failure is therefore not merely miscalibration: the score contains no instance-level ordering to exploit. More generally, deterministic scalar calibration cannot separate candidates that are tied by the underlying score; K distinct score values yield at most K + 1 threshold-defined empirical coverage regimes. The contribution is a leakage-resistant protocol for coding-agent authorization together with an empirical diagnosis of when selective routing fails because the pre-outcome representation itself lacks discrimination.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
coding agents, selective autonomy, selective prediction, uncertainty quantification, calibration, abstention, trustworthy AI, LLM agents

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_HCiQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether coding-agent patches should be accepted, reviewed, or withheld before benchmark correctness is known. OpenCoderX combines candidate-bound uncertainty, provenance, and verification signals with a weighted risk score, isotonic calibration, and fixed routing thresholds. Experiments cover 128 candidates across two coding clients. Guard effects are statistically inconclusive, while all 60 calibration candidates receive the same risk score, preventing selective routing.

Critical strengths
  1. The distinction between failure prediction, selective routing, and intervention effects is useful: it clarifies which conclusions different experiments can support.
  2. The authors transparently report negative results and emphasize binding assessments to exact candidates before observing correctness. This supports an auditable evaluation protocol.
Critical weaknesses
  1. The motivation and added value of the proposed layer are insufficiently established.
  • Coding agents already use tests to assess patches. The limitations of these checks do not by themselves justify an additional risk-scoring layer. The paper lacks a concrete case showing how this layer improves acceptance decisions over directly using verification results.
  • The approach combines a manually weighted score, standard isotonic calibration, and fixed thresholds. Although intended as a diagnostic baseline, it is not compared with verification-only routing or alternative pre-outcome predictors, leaving its incremental contribution unclear.
  • The thresholds are not justified through review or error costs, and benchmark correctness alone does not establish deployment-level trust. The practical benefit of the three-way policy therefore remains uncertain.
  1. The empirical evidence does not adequately explain the failure or establish broader conclusions.
  • All 60 calibration candidates receive risk 0.35, yielding AUROC 0.5 and all-abstain routing. Section 5 suggests fallback patterns as a cause, but provides no component distributions, missingness rates, fallback frequencies, or ablations. The elementary observation that calibration cannot separate ties does not explain why they arose.
  • Each client has only 17 matched tasks, with McNemar p-values of 1.00 and 0.50. Although the authors acknowledge this uncertainty, opposite point estimates do not establish the client-dependent effects suggested by the Section 4.1 heading.
  • Calibration is fitted and evaluated on the same candidates. This supports the tied-score illustration, but does not validate prospective routing. The distinction between freezing raw scores and applying a label-fitted calibration map needs clarification.
  • Signal normalization, fallback values, Guard triggers, and the execution consequences of routing decisions are inadequately specified. Concrete traces and quantitative audits of candidate binding and treatment delivery are also missing, limiting reproducibility and interpretation.
Questions for authors
  1. What evidence shows that the proposed risk score adds value beyond directly routing by test and verification results? Without such a comparison, why is an additional scoring layer warranted, rather than merely another way to aggregate the same evidence?
  2. Why do all 60 calibration candidates receive exactly 0.35? Do the recorded component values and fallback frequencies establish whether this is a feature-extraction or missing-data artifact? Without identifying the cause or comparing alternative predictors, what generalizable insight does this failure provide beyond the elementary fact that calibration cannot separate tied scores?
  3. The calibration map is fitted using the labels of the same candidates whose calibrated routes are reported. How is this consistent with the claim that routing is frozen before correctness is observed? Which reported routing results, if any, were produced without using the evaluated candidates’ labels to fit the calibration map?
  4. The paper lacks the AI use statement required by the ICLR 2027 author guidelines. Could the authors provide this statement?
Reviewer_gKJjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates pre-outcome selective autonomy: deciding whether a coding agent's candidate patch should be accepted automatically, reviewed, or withheld before its final correctness is known. The authors motivate this problem by noting that a plausible patch may violate a repository invariant or pass only incomplete checks, while a correct patch may be supported by weak evidence. They propose OPENCODER X, a trust layer separated from generation that can intervene during the agent's workflow (this mode is called Guard). The system also uses isotonic calibration to map raw risk scores to estimated failure probabilities for routing. The evaluation covers 128 candidate patches from SWE-bench Verified: 68 in a matched intervention pilot and 60 in a separate calibration study. Guard shows no consistent correctness benefit, while the raw scores in the calibration study collapse to a constant, preventing useful selective routing even after calibration.

Critical strengths
  1. I appreciate the importance of studying pre-outcome authorization to support fully autonomous coding agents. The separation between prediction, selection, and intervention is also worthwhile. However, the system is not described in sufficient detail to assess its design, and the results are not encouraging.
Critical weaknesses
  1. The paper is hard to read. It spends considerable space describing high-level concepts without providing enough concrete explanation of the architecture and methodology. The abstract, which should be "catchy", does the opposite: it throws concepts such as Guard and isotonic calibration at the reader without properly explaining their roles. This makes the relationship between the proposed system and the experiments difficult to follow.
  2. The overall evaluation size seems small. The intervention pilot contains only 17 paired tasks per client, and the reported comparisons are inconclusive. The paper does not report repeated runs, so the changes observed with Guard could very well be due to non-determinism rather than to the intervention itself. The calibration study includes 30 candidates per client, with calibration fitted and evaluated on the same split. Moreover, the error-AUROC of 0.346 is computed on only 4 failures out of 17 tasks, which is too few to interpret. These experiments provide limited support for broader conclusions about selective autonomy.
  3. The most important weakness is the lack of detail about the experimental framework and system implementation. The high-level description makes it difficult to assess whether the design of each component is justified. In particular, the paper does not adequately explain how the inputs to the risk score are computed and normalized, which signals were actually available, how missing information was handled in practice, or how Guard intervenes during execution. For example, Equation 7 specifies a weighted combination but does not justify the weights, nor why these signals and components were chosen. These omissions limit reproducibility and make it difficult to build on the work.
  4. Results are not encouraging, but what is missing is a detailed analysis of why the experiments are not successful. Guard does not demonstrate a consistent correctness benefit, and the raw risk score assigns every candidate in the calibration study the same value. While I appreciate the authors' transparency, the paper provides insufficient analysis of why the collapse occurs, and its terminology is vague and does not lead to an immediate understanding of the cause. Without this analysis and a more explicit design description, the negative result offers limited transferable insight.
  5. The experimental section lacks baselines and ablations. There is no comparison with alternative risk signals and no ablation of the components of the risk score, so it is unclear which parts of the design matter or whether a different signal would avoid the collapse.
Questions for authors

Questions:

  • Could the authors describe the system’s implementation in greater detail, particularly how the risk components are computed, how missing signals are handled, and how Guard intervenes during execution? Also, it's not clear how the agents are called (ie: prompt) and what kind of output they are producing.
Reviewer_7ZSTOpenReview ↗
Rating
1(−1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The paper presents OpenCoder X, a framework that routes coding-agent patches to acceptance, review, or abstention using evidence collected before benchmark grading. It binds each assessment to the exact patch and separates failure prediction, selective routing, and generation-time intervention.
  2. The evaluation covers 128 candidates. A 68-candidate pilot finds inconclusive Guard effects. On a separate 60-candidate calibration split, every candidate receives risk 0.35. Calibration maps this to 1/3, and the chosen thresholds reject autonomous execution for all candidates. The main finding is that scalar calibration cannot separate candidates with identical raw scores.
Critical strengths
  1. The protocol addresses concrete evaluation problems. Binding assessments to exact patches and checking actual Guard delivery help prevent stale assessments and misleading intervention comparisons. These are useful practices for coding-agent evaluation.
  2. The negative results are reported carefully. The paper acknowledges inconclusive paired tests and zero autonomous coverage. It also correctly distinguishes probability calibration from failure discrimination, rather than treating calibration alone as evidence of useful routing.
Critical weaknesses
  1. The central failure diagnosis is incomplete. Section 4.2 shows that the final scalar score is constant. This does not establish that the underlying evidence lacks useful information. The collapse could arise from missing telemetry, fixed fallbacks, or the aggregation rule in Eq. 7. The paper provides no component-level distributions, missingness statistics, or ablations to distinguish these explanations. Without this analysis, the main finding remains a failure of one hand-designed scorer.
  2. The experimental description is insufficient to assess the protocol fully. Sections 2.2 and 3 do not specify the component normalization rules, fallback values, exact client configurations, or what Guard changes during execution. Excluded incidents and treatment-delivery failures are also not quantified. In addition, calibration and routing are assessed on the calibration-fitting split, as acknowledged in Section 7. This supports an in-sample diagnostic, but not independent validation of selective routing.
  3. The main text is brief, but the concern is missing substance rather than length itself. Important implementation details and diagnostic analyses are absent, while several figures repeat information already given in the text or tables. And the quality of the picture is not good.
Questions for authors
  1. What are the distributions and missingness rates of the three score components? Which specific calculation causes every calibration candidate to receive 0.35? Does any individual component distinguish successful and failed candidates?
  2. What exactly does Guard change during execution? Please provide the client configurations, task-selection procedure, and counts of excluded incidents or failed treatment delivery.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.