acceptodds
Desk rejected at ICLR 2027

FrontierRefactor: Benchmarking Multi-Scale, Performance-Oriented Codebase Refactoring

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Autonomous codebase refactoring requires coding agents to restructure working software while strictly preserving externally observable behavior, a process driven in production by execution performance and compute efficiency rather than stylistic preference. However, existing benchmarks evaluate these requirements in isolation: refactoring benchmarks evaluate functional correctness alone while overlooking severe runtime regressions, whereas code performance benchmarks target isolated micro-kernels that ignore codebase-scale complexity. We observe that software refactoring scale is governed by the scope of the observable contract, the caller-visible interface that fixes behavioral invariants while freeing private implementations for optimization. Building on this insight, we introduce FrontierRefactor, a benchmark of 60 qualified engineering tasks spanning four scales from computational kernels to full repositories across in-place optimization and cross-language rewriting. Evaluating frontier coding agents through a reference-free cascaded protocol across 1,080 execution runs reveals that correctness and speedup are largely decoupled: 41.2% of candidate submissions satisfy functional equivalence yet fail to deliver verified performance improvements, with 19.3% suffering active performance regressions and 21.9% trapped in measurement noise. Furthermore, refactoring paradigms reveal orthogonal engineering bottlenecks: while cross-language rewrites secure substantial speedup dividends from native compilation upon satisfying functional checks, agents struggle with contract fidelity, dropping from a 65.0% pass rate on modules to 32.6% at package scale; conversely, in-place optimization reliably preserves behavioral contracts at 86.5% but rarely achieves acceleration without physical performance profiling. Trajectory analysis demonstrates that agents frequently fall victim to uncalibrated micro-benchmarking illusions, an operational blind spot that multi-turn execution timing feedback directly resolves, raising strict success from 16.0% to 40.0%.

Primary area:
datasets and benchmarks
Keywords:
Codebase Refactoring, Benchmark, Coding Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pdfWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces FrontierRefactor, a benchmark of 60 performance-refactoring tasks organized by four scales from Kernel to Repository and two forms, in-place optimization and cross-language rewriting. Submissions are judged without reference patches by three gates in sequence: compliance, functional equivalence against the original program, and a worst-case speedup that must clear a noise-calibrated threshold. Evaluating six frontier agents, the authors find that functional correctness often does not come with real speedup, that in-place and cross-language tasks fail in different ways, and that repeated measurement and timing feedback help agents succeed.

Critical strengths

The benchmark asks the right question by requiring both behavior preservation and measurable speedup, and it extends this to cross-language rewriting under a single reference-free protocol. The timing methodology is also careful: noise-calibrated thresholds, worst-case aggregation over workloads, and explicit anti-cheating safeguards address well-known reliability problems in speedup benchmarks.

Critical weaknesses
  1. The scale framing does not hold up. Scale is the benchmark’s main organizing axis, yet the task definitions contradict it: half of the Kernel tasks are full cross-language rewrites. Results do not follow scale either, since Repository tasks are the easiest and task form dominates every comparison. The task set is also small at the higher scales and includes many textbook algorithm exercises, so it does not convincingly measure real-world, multi-scale refactoring.

  2. The evaluation cannot fully back the conclusions. Most findings depend on the functional-equivalence gate, but only its false rejections are audited, so how often incorrect programs pass is unknown. Cross-language tasks also gain speedup simply by moving to a compiled language, which makes results pooled with in-place tasks hard to interpret.

Questions for authors
  1. Can you estimate how often incorrect programs pass the functional-equivalence gate, for example by re-testing the functionally passing submissions against a much larger differential or fuzzed test suite, broken down by scale and form? A low false-acceptance rate would substantially strengthen my confidence in the paper's conclusions.

  2. How do the main findings change if cross-language and in-place tasks are analyzed separately, and how exactly is average speedup computed in each table?

2 comments
AuthorsOpenReview ↗

Summary of Responses to Reviewer pdfW's Follow-up Points

Dear Reviewer pdfW,

Thank you again for your careful review and detailed follow-up, which helped us strengthen the paper. We appreciate your confirmation that the per-form breakdown and speedup definitions address Q2, that the 645,228-input audit is what you asked for in Q1, and that the Direct Port control is a sound way to address W2. As the discussion period is closing, we would like to briefly summarize how we have addressed each of your three follow-up points, in case it is helpful for your final assessment:

  • Reporting under the expanded gate: as you suggested, results are now also reported under the 645,228-input expanded gate, with all 91 discrepant submissions counted as failures. Overall strict success is 24.4% (264/1,080), and cross-language Repository strict success is 31.5%, below Module (44.4%) and Kernel (33.3%) (Sections 3.2 and 4.4, Appendix Tables 11–12).
  • Differences previously treated as non-semantic: none of the 41 runs is counted as a pass, and only the unconditional pass rate of 88.2% (683/774) is reported. The comparator rules are now stated: float64 outputs use an absolute tolerance of 10−1210^{-12} and a relative tolerance of 10−910^{-9}, and Repository tasks check exact digests of exit_code, stdout, stderr, and the file tree (Section 3.2, Appendix A.9).
  • Interpreting cross-language speedup: the harness times the external invocation boundary, including JSON pipe round-trips and CLI process launch, which compresses lighter calls toward the 1.090× median for a direct port, while compute-heavy tasks reach 2.05×–5.33×. Following your suggestion, the two forms are reported separately and cross-language strict success is also scored against the direct port, giving 9.6% (52/540) (Section 4.4, Appendix A.6, Appendix Tables 11–12).

We hope these revisions resolve your remaining concerns, and we would be sincerely grateful if you could take them into account in your final assessment. A summary of the full discussion and revisions is posted in a separate comment, and we would be glad to clarify anything further before the discussion closes.

Thank you again for your time and consideration.

Best regards, The Authors

AuthorsOpenReview ↗

Thank You and a Kind Request for Your Feedback

Dear Reviewer pdfW,

Thank you very much for your careful review and detailed follow-up, which helped us strengthen the paper. We understand that this is a busy period, and we apologize for the additional message.

As the discussion period closes today, we would like to kindly follow up on the three points you raised (reporting under the expanded gate, the comparator rules, and cross-language scoring). Each of them is addressed in the revision, as summarized point by point in our previous comment, and we hope this resolves your concerns. If so, we would be sincerely grateful if you would consider updating your assessment. Should you have any remaining questions, we would be glad to respond right away.

We truly appreciate the time and thought you have devoted to our submission.

With sincere thanks, The Authors

Reviewer_doydOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents 60 tasks across 7 programming languages that tests the ability of coding agents to write performance refactors. These refactors are governed by different granularities while still adhering to contracts. The authors additionally study settings where the underlying programming language is (1) strictly the same and (2) can be changed. The emperical results show that performance optimization which maintaining contract obligations is a challenge for coding agents, and current frontier agents are still far from ensuring idea speed ups.

Critical strengths

The paper presents a benchmark spanning 60 tasks across four refactoring granularities and two operational forms: in-place optimization and cross-language rewriting. The authors analyze why agents often fail to achieve reliable performance improvements. Mainly this is due to not targeting relevant bottlenecks and insufficient timing. The paper further shows that providing explicit runtime feedback in a multi-turn setting can recover performance losses and substantially improve strict success compared with blind retries or agents relying on their own ad hoc timing measurements.

Critical weaknesses
  • The paper has a limited number of repository-level tasks (only 9). Out of these 9, 3 are multilingual tasks. My concern is that this is a narrow selection to make general deductions on the capabilities of agents to perform performance refactorings.
  • Skew in programming languages - The top 3 languages cover 86.6% of the benchmark
    • Python: 40%
    • JavaScript: 28.3%
    • TypeScript: 18.3%
Questions for authors

Clarifications:

  • Why do you not try writing cross-module refactors into C ? Would that have provided substantial gains?
  • Cross-language and in-place tasks may not be perfectly comparable because compilation itself might improve performance? How can you separate algorithmic improvements from optimizations done by compilation
3 comments
AuthorsOpenReview ↗

Summary of Responses to Reviewer doyd

Dear Reviewer doyd,

Thank you again for your thoughtful review and for your positive assessment of the paper's presentation and soundness. As the discussion period is closing, we would like to briefly summarize how we have addressed each concern you raised, in case it is helpful for your final assessment:

  • Limited number of repository tasks: multi-file refactoring is covered by the Package and Repository tiers together: 24 tasks (13 in-place, 11 cross-language), or 40.0% of all evaluations, with a median of 1,420 LoC for the 15 Package tasks and 14,280 LoC for the 9 Repository tasks. The core finding is the same on these 24 tasks (41.4% of runs pass functional checks without verified speedup), on the benchmark without the 9 Repository tasks (41.1%), and on the full benchmark (41.2%) (Sections 3.3 and 4.4).
  • Language skew: the 86.6% figure counts source languages; with rewrite targets included, Rust and Go appear in 56.7% of tasks (Section 3.3). The finding also holds outside Python/JS/TS: on the compiled source languages (C++, Go, Java, Rust; 144 runs), 86.1% of runs pass functional checks while only 20.1% achieve strict speedup (Section 4.4).
  • Compilation vs. algorithmic gains: a new Direct Port control with 249 matched comparisons measures the two separately: the port alone yields a 1.090× median speedup, and agent submissions add a further 1.002× (Section 4.4, Appendix A.6).
  • C as a target language: C would also yield native speedups on small routines; we chose Rust and Go because they provide standard build tooling at package and repository scale and memory-safe, deterministic grading (Part 2 of our response).

We hope these additions resolve your concerns, and we would be sincerely grateful if you could take them into account in your final assessment. We would of course be glad to clarify anything further before the discussion closes.

Thank you again for your time and consideration.

Best regards, The Authors

I acknowledge the response by the authors. Thank you for the additional experiments. I believe I have no further questions.

AuthorsOpenReview ↗

Thank You to Reviewer doyd

Dear Reviewer doyd,

Thank you very much for taking the time to review our responses and the additional experiments, and for updating your score. We sincerely appreciate your constructive questions, which helped us strengthen the paper.

Best regards, The Authors

Reviewer_ay6dOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FrontierRefactor, a benchmark of 60 performance refactoring tasks. Half ask for in-place optimization and half for a rewrite into Rust or Go, spread over four scales from computational kernels to full repositories. A task was kept from 184 candidates only if the targeted code dominates runtime, its tests are deterministic, and its noise is low. A submission must pass a dependency and symbol check, then every hidden differential probe, and then beat a per-task noise threshold in worst-case speedup over held-out workloads. Six agent setups attempt every task three times, for 1,080 runs. Strict success stays low for every agent, and many runs that preserve behavior still fail to make the code faster. The paper also finds that in-place edits and cross-language rewrites fail in different ways, and that multi-turn timing feedback raises strict success on a 25-task subset.

Critical strengths
  1. The benchmark fills a real gap in how coding agents are evaluated. Refactoring and migration benchmarks reward preserved behavior alone, while performance benchmarks stay at a single scale. FrontierRefactor brings the two together, asking for unchanged behavior and a verified speedup from single kernels up to whole repositories, and for cross-language rewrites as well. This matches why code is refactored in practice, as Section 1 argues.

  2. The performance gate makes a new kind of failure measurable. Each task has its own noise threshold, taken from repeated runs of the original code against itself, so a correct submission can be told apart as truly faster, within noise, or actually slower. Figure 2 shows that both failure groups are large for every agent.

Critical weaknesses
  1. My main concern is that the paper's own control does not support its claim about timing feedback. The abstract and conclusion say timing feedback resolves the profiling blind spot, raising strict success from 16% to 40%. Table 10 also runs Blind Retry, which gives no timing information at all, and it reaches 32% by the third round. Scalar feedback adds only two tasks on top of that. Noise-Aware Timing, the arm with the most timing detail, ends below Blind Retry. The reported McNemar test compares rounds within the scalar arm and never compares it with Blind Retry.

  2. The counterfactual results in Table 9 come from runs the paper never describes. Section 3.2 states that no submission in the 1,080 runs was rejected by the compliance gate, so every failure came from the functional or performance gates. Yet Table 9 reports that switching the compliance gate off would lift Claude Opus 5 from 34.4% to 67.9% strict success. Rescoring the same submissions without that gate could not change anything, so these numbers must come from separate runs. The paper gives no tasks, trial counts or instructions for them, yet Appendix A.9 uses these numbers to show that the safeguards are necessary.

Questions for authors

I have no questions beyond the concerns raised in the weaknesses above.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and ACs

We sincerely thank all reviewers for their careful and constructive feedback. We appreciate your recognition of the evaluation gap filled by uniting behavior preservation with verified multi-scale speedup and our noise-calibrated performance gate (Reviewer ay6d), the clear presentation, sound methodology, and multi-granularity analysis across in-place and cross-language settings (Reviewer doyd), and our reference-free protocol with worst-case aggregation and anti-cheating safeguards (Reviewer pdfW).

Our contribution goes beyond combining test suites with runtime timing. (i) FrontierRefactor formalizes multi-scale performance refactoring via the observable contract (OO), defining scale by the caller-visible interface (Kernel, Module, Package, Repository) orthogonally to implementation form (In-Place optimization vs. Cross-Language rewriting). (ii) We pair a three-stage Amdahl qualification funnel with a reference-free cascaded gate enforcing compliance, differential equivalence, and two-tier noise-calibrated worst-case speedup, separating genuine acceleration from noise-bound edits and regressions without reference patches. (iii) Evaluating six frontier agents across 1,080 runs shows that correctness and speedup are largely decoupled (41.2% pass functional checks without speedup) and exposes orthogonal bottlenecks between contract fragility in cross-language rewrites and bottleneck-targeting failures in in-place edits.

Summary of new experiments and analyses:

  1. Expanded differential and fuzzing audit of the functional gate (Reviewer pdfW). Across all 774 passing runs on 54 tasks, we ran 645,228 new checks: 1,000 new inputs per Kernel/Module/Package task and 100 new CLI scenarios per Repository task. Across all 645,228 evaluations, 99.27% (640,508/645,228) of test cases pass (99.60% in-place, 98.82% cross-language). At the submission level, 88.2% (683/774) pass all 1,000/100 new inputs with zero discrepancies, rising to a 93.5% (724/774) core semantic pass rate after accounting for standard float tolerances (10 runs), 10−410^{-4} round() tie-breaks on one task (16 runs), and stderr-only diagnostic text matching exit codes and stdout (15 runs)—with zero semantic bugs across all 255 Kernel and Module in-place runs. Even under a zero-defect filter across all 645,228 checks, 264/329 strict successes survive and non-accelerating passing runs rise from 57.5% to 61.3% (Section 3.2, Appendix A.9).

  2. Algorithm-preserving direct-port control for cross-language tasks (Reviewer doyd; Reviewer pdfW). To separate compiled-language translation dividends from agent algorithmic optimization, we built algorithm-preserving Direct Ports (PP) in Rust and Go matching the control flow, data structures, and complexity of the Source programs (SS). Across 27 validated ports and 249 matched Source (SS), Direct Port (PP), and Agent (AA) pairs timed in identical sandboxes across three batches, task-equal medians are 1.090×1.090\times for S→PS \to P, 1.103×1.103\times for S→AS \to A, and 1.002×1.002\times for P→AP \to A, with 249-pair means of 1.441×1.441\times, 1.531×1.531\times, and 1.027×1.027\times, showing that most cross-language speedup comes from direct translation while marginal agent gain is minimal (Section 4.4, Appendix A.6).

  3. Stratified robustness across scales, forms, and source languages (Reviewer doyd; Reviewer pdfW). Excluding all 9 repository tasks leaves our core findings unchanged across the remaining 51 tasks and 918 runs: 41.1% pass functionally without speedup, and strict success is 14.1% in-place versus 39.7% cross-language. Restricting to the 24 multi-file Package and Repository tasks likewise yields 41.4% non-accelerating runs across 432 trials. On the eight compiled source-language tasks in C++, Go, Java, and Rust (144 runs), agents reach an 86.1% functional pass rate but only 20.1% strict success, leaving 66.0% non-accelerating. Separating all six models across the 540 In-Place and 540 Cross-Language runs confirms that DeepSeek-V4.1-Flash ranks #1 in both forms at 25.6% and 54.4% (Section 4.4, Appendix A.6).

  4. Multi-turn feedback controls and counterfactual protocol clarifications (Reviewer ay6d; Reviewer pdfW). We clarify that all three feedback arms share functional correctness feedback after each round while Blind Retry withholds only runtime telemetry, report the Round-3 paired comparison against Blind Retry (40.0% vs. 32.0%, p=0.73p=0.73), and show how timing telemetry guides post-repair conversion (5/5 vs. 1/4) and active-slowdown recovery (vs. 0/5 for Blind Retry) (Section 4.6, Appendix A.10). We also detail the separate-run protocol, 60-task ×\times 3-trial setup, prompt/verifier changes, and valid-run Pass@1 definition for Table 9 (Section 3.2, Appendix A.9), clarify Avg. Speedup in Tables 3 & 6, and relabel Table 10 to Median Speedup.

For your convenience, all revisions in the updated PDF are highlighted in blue.

AuthorsOpenReview ↗

Summary of Discussion and Revisions

We sincerely thank all three reviewers and the Area Chair for the careful and constructive discussion, which has substantially strengthened the paper. For convenience, we summarize below how each reviewer's concerns have been addressed, with all changes marked in blue in the revised PDF.

Reviewer ay6d. We are grateful to Reviewer ay6d for updating the score, finding the paper's contributions "leaning more toward the acceptance", and for confirming that the concern on the compliance gate and the verifier is resolved. Following the suggestions from this in-depth discussion, the revision distinguishes the prompt compliance rules Π\Pi from the out-of-band verifier check Gcomp(S′)G_{\text{comp}}(S'), reports the rules/verifier 2×22 \times 2 breakdown (Section 3.2, Appendix A.9), and clarifies the multi-turn feedback analysis (Section 4.6, Appendix A.10).

Reviewer doyd. We are grateful to Reviewer doyd for updating the score after our responses and additional experiments, and for the positive assessment of the paper's presentation and soundness. In response to the questions raised, the revision shows that the core finding also holds on the 24 multi-file Package and Repository tasks (40.0% of all evaluations) and on the compiled source languages (Sections 3.3 and 4.4), and adds a Direct Port control with 249 matched comparisons that separates compilation gains from algorithmic gains (Section 4.4, Appendix A.6).

Reviewer pdfW. We thank Reviewer pdfW for the careful review and detailed follow-up, and for acknowledging that the per-form breakdown addresses Q2, that the 645,228-input audit is what was asked for in Q1, and that the Direct Port control is a sound way to address W2. Each of the three follow-up points is addressed in the revision:

  1. Expanded gate: results are also reported under the expanded functional gate, with all 91 discrepant submissions counted as failures, in the main text (Sections 3.2 and 4.4) and in full by scale, form, and model in Appendix Tables 11–12. Under this gate, cross-language Repository strict success is 31.5%, below Module (44.4%) and Kernel (33.3%).
  2. Comparator rules: the rules shared by both gates, including float tolerances and stderr checking, are stated explicitly (Section 3.2, Appendix A.9).
  3. Cross-language scoring: the two forms are reported separately, and cross-language strict success is scored against both the source program and an algorithm-preserving direct port, with the timing boundary specified (Section 4.4, Appendix A.6, Appendix Tables 11–12).

The revision also states contract scale and implementation form as orthogonal axes (Section 3.3).

We are sincerely grateful for the reviewers' time and effort, and remain happy to clarify any remaining point before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.