- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces FrontierRefactor, a benchmark of 60 performance-refactoring tasks organized by four scales from Kernel to Repository and two forms, in-place optimization and cross-language rewriting. Submissions are judged without reference patches by three gates in sequence: compliance, functional equivalence against the original program, and a worst-case speedup that must clear a noise-calibrated threshold. Evaluating six frontier agents, the authors find that functional correctness often does not come with real speedup, that in-place and cross-language tasks fail in different ways, and that repeated measurement and timing feedback help agents succeed.
Critical strengths
The benchmark asks the right question by requiring both behavior preservation and measurable speedup, and it extends this to cross-language rewriting under a single reference-free protocol. The timing methodology is also careful: noise-calibrated thresholds, worst-case aggregation over workloads, and explicit anti-cheating safeguards address well-known reliability problems in speedup benchmarks.
Critical weaknesses
-
The scale framing does not hold up. Scale is the benchmark’s main organizing axis, yet the task definitions contradict it: half of the Kernel tasks are full cross-language rewrites. Results do not follow scale either, since Repository tasks are the easiest and task form dominates every comparison. The task set is also small at the higher scales and includes many textbook algorithm exercises, so it does not convincingly measure real-world, multi-scale refactoring.
-
The evaluation cannot fully back the conclusions. Most findings depend on the functional-equivalence gate, but only its false rejections are audited, so how often incorrect programs pass is unknown. Cross-language tasks also gain speedup simply by moving to a compiled language, which makes results pooled with in-place tasks hard to interpret.
Questions for authors
-
Can you estimate how often incorrect programs pass the functional-equivalence gate, for example by re-testing the functionally passing submissions against a much larger differential or fuzzed test suite, broken down by scale and form? A low false-acceptance rate would substantially strengthen my confidence in the paper's conclusions.
-
How do the main findings change if cross-language and in-place tasks are analyzed separately, and how exactly is average speedup computed in each table?
2 comments
Thank You and a Kind Request for Your Feedback
Dear Reviewer pdfW,
Thank you very much for your careful review and detailed follow-up, which helped us strengthen the paper. We understand that this is a busy period, and we apologize for the additional message.
As the discussion period closes today, we would like to kindly follow up on the three points you raised (reporting under the expanded gate, the comparator rules, and cross-language scoring). Each of them is addressed in the revision, as summarized point by point in our previous comment, and we hope this resolves your concerns. If so, we would be sincerely grateful if you would consider updating your assessment. Should you have any remaining questions, we would be glad to respond right away.
We truly appreciate the time and thought you have devoted to our submission.
With sincere thanks, The Authors
Summary of Responses to Reviewer pdfW's Follow-up Points
Dear Reviewer pdfW,
Thank you again for your careful review and detailed follow-up, which helped us strengthen the paper. We appreciate your confirmation that the per-form breakdown and speedup definitions address Q2, that the 645,228-input audit is what you asked for in Q1, and that the Direct Port control is a sound way to address W2. As the discussion period is closing, we would like to briefly summarize how we have addressed each of your three follow-up points, in case it is helpful for your final assessment:
float64outputs use an absolute tolerance of 10−12 and a relative tolerance of 10−9, and Repository tasks check exact digests ofexit_code,stdout,stderr, and the file tree (Section 3.2, Appendix A.9).We hope these revisions resolve your remaining concerns, and we would be sincerely grateful if you could take them into account in your final assessment. A summary of the full discussion and revisions is posted in a separate comment, and we would be glad to clarify anything further before the discussion closes.
Thank you again for your time and consideration.
Best regards, The Authors