acceptodds
Under review as a conference paper at ICLR 2027

EDN-R/CODE: A RELIABILITY RUNTIME FOR MODEL-HETEROGENEOUS SOFTWARE ENGINEERING AGENTS

Abstract

Repository-level coding agents modify software through models that differ in out-put protocol, instruction sensitivity, failure modes, and provider behavior. Re-placing a model can therefore change how execution fails, not just task accu-racy. We present EDN-R/Code, a runtime that treats models as proposal pro-ducers rather than execution authorities. Adapters lower native responses into ProposalIR; each attempt runs in an isolated workspace from an immutable base snapshot. The runtime authorizes mutations, binds veriffcation evidence to candidate state, classiffes failures, retries or rebinds producers under explicit bud-gets, and persists attempt state for recovery. DeepSeek’s native JSON-schema output and GLM’s native tool calls enter the same runtime. Targeted kill tests ex-ercise malformed-output, timeout, stale-evidence, capability, critical-information, and crash/recovery boundaries. On six frozen held-out micro-repositories, ho-mogeneous DeepSeek redundancy and heterogeneous GLM-to-DeepSeek each achieve 6/6 success. All 12 runs accept candidates passing visible and hidden tests and pass replay checks; no false acceptance or unsafe execution is observed, and all eight required multi-ffle scope checks pass. Heterogeneity provides one real cross-producer rescue but no accuracy gain: it consumes 10,971 versus 5,106 tokens and 185.39 versus 60.19 seconds. These controlled ffndings position het-erogeneous producers as a recovery resource; reliability rests on stable execution semantics, not assumed stability of model behavio

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Semantic Authorization, ProposalIR, Failure-Aware Rebinding

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_KjLPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates what helps a language model after a failed program repair. This paper proposes a controlled framework for post-failure measurement that. In two prospective DeepSeek populations, BLIND best-of-3 recovers 52.4% and 49.0% of failures, STRUCTURE does not improve on it, and STRONG reaches 83.5% and 82.8%.

Critical strengths
  1. This paper shows clear definition of what post-failure recovery measures. For example, Eq. 1 conditions on the screening failure. Excluding the screening candidate from the BLIND samples prevents BLIND from reusing the event that selected the task, and resetting the buggy state keeps the conditions independent.

  2. Two independently screened prospective populations agree on the qualitative pattern. In both P1 and P2supp, BLIND recovers about half of the failures, and TRUCTURE does not exceed BLIND.

Critical weaknesses
  1. The historical and prospective runs differ in decoding temperature and screening configuration as well as in the failure population, so the paper's selection-effect claim is not isolated.

The abstract states that the historical experiments, with "a much lower blind baseline and an apparent structure effect", demonstrate "that failure-selected populations are themselves part of the measurement". Section 7 repeats this as "Failure populations are experimental objects". However, the historical DeepSeek and Xing interventions use temperature 0.0, and the prospective runs use 0.3 (Table 8, B.3). Also, the model configuration of the 919-cell historical screen "is not established by the supplied record" (section 3.2). Therefore, the two populations can also come from different screening models.

The historical GLM-5.3 Flash runs use temperature 0.3 and also show BLIND at 19.0%, which suggests that selection matters as well. However, DeepSeek, Xing and GLM-5.3 Flash all recover exactly 30/158 under BLIND (Table 14). The paper should explain this before the result can be read as evidence.

The selection effect is the paper's main methodological message. However, its main evidence, DeepSeek historical vs. prospective, changes temperature, screening configuration, and population at once.

  1. The evidence cannot be checked from the paper, and one reported test result cannot be reproduced from the reported counts.

By the paper's own account, the prompts for the five conditions, the task manifests, screening outputs, generated candidates, evaluator traces, and statistical scripts are "not available during editing" (Appendix E). Section 8 also states that "Exact prospective per-condition numerators and confidence intervals were not supplied". Without the STRUCTURE prompt, a reader cannot tell whether it is a non-diagnostic control or contains misleading content. This matters because STRUCTURE falls below BLIND in both prospective populations (−7.8 and −8.5).

Then, for P2supp, BLIND recovers 145/296 and STRUCTURE 120/296 (A.2). However, across every possible pairing of tasks, an exact McNemar test on these best-of-3 outcomes gives at most p≈0.14. An unpaired Fisher test gives p≈0.047. So the reported p=0.302 (Tables 4 and 11) matches neither, and the paper does not name the test.

Also, the results are inconsistent. The P2supp rates are "recomputed with the stated cohort denominator of 296 rather than the inconsistent denominator of 300 implied by the originally supplied percentages" (Table 10). But the "about 81-82%" in section 1 and Figure 2 matches the denominator of 300.

With the prompts for the five conditions missing, a key test result not reproducible, and the originally supplied percentages known to be inconsistent, a reader cannot verify the central "STRUCTURE does not help" result or the size of any effect.

Given that this paper's main contributions are empirical, current results are unable to support the paper's main claims and contributions.

Questions for authors
  1. Could you show the results at the matched configuration, so the comparison of historical and prospective populations changes only the failure population?

  2. Is BLIND run separately for each model in the historical five-condition study? How do DeepSeek, Xing and GLM-5.3 Flash, at two different temperatures, each recover exactly 30/158?

  3. Which test produces p=0.344 and p=0.302 for STRUCTURE vs. BLIND? Can you report per-condition counts with confidence intervals for both prospective populations to validate your reported scores, which are currently unreproducible from results in the paper?

  4. Which model and decoding configuration produce the 919-cell historical screen? How do the P2supp counts relate to the denominator of 300?

  5. How do the results relate to Olausson et al. [1], who compare self-repair with resampling under matched budgets, and to EvalPlus [2], which shows that HumanEval's tests miss many incorrect solutions?

[1] Is Self-Repair a Silver Bullet for Code Generation? ICLR 2024.

[2] Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. NeurIPS 2023.

Reviewer_cf4oOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies what helps an LLM repair code after an initial failure: additional sampling, prompt restructuring, or valid diagnostic information. It compares five conditions, ranging from blind retries to an oracle that identifies and displays the faulty source line.

The main experiments prospectively select two cohorts of 103 and 296 failures, using the same model configuration for screening and subsequent interventions. Each condition receives three fresh attempts. Blind retries recover 52.4% and 49.0% of failures, while the strongest diagnostic condition reaches 83.5% and 82.8%. A diagnostic-shaped prompt without useful information provides no observed improvement. Historical experiments show different response patterns, illustrating dependence on the selected tasks and model settings.

The contribution is an empirical framework for separating retry and information effects. A correctness audit also illustrates why passing tests and matching a reference implementation provide incomplete evidence of correctness.

Critical strengths
  1. The prospective protocol controls important sources of confusion. Each condition receives the same number of fresh attempts, excludes the screening attempt, and starts from the same buggy program. The nondiagnostic prompt control is particularly useful for distinguishing improvements from prompt formatting versus actual defect information. (§3)

  2. The paper treats correctness evidence carefully. The authors explicitly acknowledge that their strongest hints provide oracle information. They also investigate candidate–reference disagreements using independent checks, including a case where the candidate is correct and the reference is wrong. This strengthens the paper’s methodological message. (§6)

Critical weaknesses
  1. The central empirical results are difficult to verify. Appendix E lists essential records unavailable during manuscript preparation, including exact prompts, task manifests, paired outcomes, and statistical scripts. There is also a concrete statistical concern: P2 reports 145 versus 120 successes out of 296, but gives p = 0.302 for the paired comparison. That value is incompatible with a standard exact paired-binary test of those counts; a different test or analysis unit would need explanation. These gaps are particularly consequential for a paper whose contribution is careful empirical measurement. (Appendices A and E)

  2. The conclusions have limited generality and explanatory reach. The main experiments use localized synthetic defects and privileged diagnostic information. Moreover, historical and prospective experiments differ in task populations and sampling temperature, so their contrasting response patterns cannot isolate the effect of failure selection. The authors acknowledge these limitations, but controlled comparisons and realistic diagnostics would make the findings more actionable. (§8; Table 8)

Questions for authors
  1. Can you provide the exact prompts, cohort manifests, paired outcomes, and statistical scripts, and reconcile the reported p = 0.302?
  2. How do P1 and P2 differ, and are multiple mutations of the same underlying program treated as independent observations?
  3. Do the gains persist with automatically generated, potentially incorrect diagnostics on real bugs, under a matched total inference budget?

Comments on the paper

1 comment

Dear Reviewers,

Please review the authors' rebuttal and participate in the discussion to address any remaining questions before the period closes on Nov 18 AoE.

Thank you!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.