- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve the runtime efficiency of LLM-generated programs at inference time while preserving functional correctness. The authors propose EffiPair, which first generates multiple candidates, selects the fastest correct program as a reference, and chooses a slower correct candidate whose AST similarity exceeds a threshold. Both programs and their execution feedback are then provided to the model for refinement. The new program is tested, timed, and added to the candidate pool, while previous candidates are retained. The authors evaluate EffiPair on EvalPerf, Mercury, and ENAMEL. The full method achieves the highest reported efficiency scores across nine model–benchmark combinations, with no decrease in Pass@1 from initial candidate selection to two rounds of refinement. Component ablations examine the roles of pairing and execution feedback, while repository-level experiments explore different sources of comparison programs.
Critical strengths
-
The complete pipeline consistently improves efficiency across multiple settings and has practical value. In Table 1, the official efficiency scores after two refinement rounds exceed those of the initial candidate selection in all nine settings. Pass@1 remains unchanged in six settings and improves in three. The authors separate internal candidate-selection timing from official evaluation, reducing the bias associated with using the same timing measurements for both selection and reporting. They also report token costs. These results support a practical approach to improving the efficiency of generated programs through a small candidate pool, execution feedback, and iterative refinement.
-
The ablations address the core design choices and provide a useful basis for evaluating pairing. Table 2 compares isolated refinement, random pairing, and similarity-based pairing under the same initial candidate pool and correctness gate, and separately examines the effect of execution feedback. This is more informative than comparing only against direct generation. The appendix also provides code changes and execution results across refinement rounds, allowing readers to examine how optimization occurs rather than relying solely on aggregate metrics.
Critical weaknesses
-
The methodological contribution depends mainly on the value of structural pairing. Candidate generation, correctness checks, and execution-guided refinement are already present in prior work [1–3]. EffiPair adds structurally matched comparison programs to this pipeline. This is a reasonable extension, but its significance depends on a reliable advantage over single-program feedback and random pairing, which the current results do not yet establish.
-
The experiments provide limited support for the independent contribution of differential pairing. Section 4.3 attributes the improvement from to to DEF, but this comparison also introduces refinement and execution feedback, so it does not isolate the effect of pairing. In the more direct ablations in Table 2, the full method improves over isolated refinement with execution feedback by only 0.16–1.16 points, and over random pairing with execution feedback by only 0.07–1.27 points. The repository-level results are also close: same-site pairing, best seed, and solo profiling achieve mean gains of 1.24%, 1.21%, and 1.19%, respectively. Thus, although the complete pipeline is effective, the results do not yet convincingly establish how much practical value differential pairing itself adds.
[1] Y. Peng, A. D. Gotmare, M. R. Lyu, C. Xiong, S. Savarese, and D. Sahoo, “PerfCodeGen: Improving performance of LLM generated code with execution feedback,” in Proceedings of the 2025 IEEE/ACM 2nd International Conference on AI Foundation Models and Software Engineering (FORGE), 2025.
[2] D. Huang et al., “EffiLearner: Enhancing efficiency of generated code via self-optimization,” in Advances in Neural Information Processing Systems, 2024.
[3] S. Waghjale, V. Veerendranath, Z. Z. Wang, and D. Fried, “ECCO: Can we improve model-generated code efficiency without sacrificing functional correctness?” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Questions for authors
-
What information does DEF add to the refinement prompt? Appendix A.3.2 provides a template but leaves {guidance} unspecified. Please show a filled example and clarify whether DEF explicitly computes code or performance differences, or supplies two programs and their respective profiles for the model to compare.
-
Where does structural matching provide additional value over random pairing and single-program feedback? The gains in Table 2 are relatively small. Which types of tasks benefit most from structural matching? With only three initial candidates by default, how many tasks have multiple eligible partners? Please clarify the random-pair selection rule, the behavior when no eligible neighbor exists, the meaning of “±,” and the number of runs.
-
How do differences in experimental conditions affect the baseline comparisons? Table 1 reports averages over three runs for EffiPair but single runs for the baselines. Do the gaps to the closest baselines persist across repeated runs? EffiLearner uses a memory-integral-based acceptance criterion, and test access also differs across methods. How do the authors distinguish the effects of these settings from those of the proposed method?
-
What conclusions are supported by the repository-level experiments? The mean gains of same-site pairing, best seed, and solo profiling are very close. On which instances does pairing provide meaningful benefits? Please specify the model and refinement configuration, explain why the other comparison groups have different numbers of valid instances, and clarify whether the conclusions hold when evaluated on the same instance set.