acceptodds
Withdrawn from ICLR 2027

EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) can generate functionally correct programs that differ substantially in execution efficiency. Existing inference-time optimization methods typically refine each candidate using pointwise runtime or profiling feedback, which identifies how costly an implementation is or where the cost arises, but offers limited guidance on how the computation should change. We introduce DIFFERENTIAL EXECUTION FEEDBACK (DEF), which compares implementations that are nearby in structural program space but separated in performance, turning their execution and implementation differences into directional optimization evidence. We demonstrate DEF in EffiPair, a training-free, test-time framework that pairs structurally similar programs with different efficiencies, distills their relative execution behavior into compact feedback, and iteratively refines a candidate pool. Across EvalPerf, Mercury, and ENAMEL, using GPT-4o mini, DeepSeek-V4.1 Flash, and GPT-5 mini, EffiPair achieves the highest value on each benchmark’s official efficiency metric in all nine model–benchmark settings under matched evaluation conditions. Moreover, two contrastive refinement rounds improve efficiency over the selected initial draft in every setting while preserving or improving Pass@1. These results demonstrate the effectiveness of relational execution feedback as a lightweight signal for test-time code optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Code Efficiency, Code Generation, Differential Execution Feedback, Test-Time Optimization, Program Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BuMYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve the runtime efficiency of LLM-generated programs at inference time while preserving functional correctness. The authors propose EffiPair, which first generates multiple candidates, selects the fastest correct program as a reference, and chooses a slower correct candidate whose AST similarity exceeds a threshold. Both programs and their execution feedback are then provided to the model for refinement. The new program is tested, timed, and added to the candidate pool, while previous candidates are retained. The authors evaluate EffiPair on EvalPerf, Mercury, and ENAMEL. The full method achieves the highest reported efficiency scores across nine model–benchmark combinations, with no decrease in Pass@1 from initial candidate selection to two rounds of refinement. Component ablations examine the roles of pairing and execution feedback, while repository-level experiments explore different sources of comparison programs.

Critical strengths
  1. The complete pipeline consistently improves efficiency across multiple settings and has practical value. In Table 1, the official efficiency scores after two refinement rounds exceed those of the initial candidate selection in all nine settings. Pass@1 remains unchanged in six settings and improves in three. The authors separate internal candidate-selection timing from official evaluation, reducing the bias associated with using the same timing measurements for both selection and reporting. They also report token costs. These results support a practical approach to improving the efficiency of generated programs through a small candidate pool, execution feedback, and iterative refinement.

  2. The ablations address the core design choices and provide a useful basis for evaluating pairing. Table 2 compares isolated refinement, random pairing, and similarity-based pairing under the same initial candidate pool and correctness gate, and separately examines the effect of execution feedback. This is more informative than comparing only against direct generation. The appendix also provides code changes and execution results across refinement rounds, allowing readers to examine how optimization occurs rather than relying solely on aggregate metrics.

Critical weaknesses
  1. The methodological contribution depends mainly on the value of structural pairing. Candidate generation, correctness checks, and execution-guided refinement are already present in prior work [1–3]. EffiPair adds structurally matched comparison programs to this pipeline. This is a reasonable extension, but its significance depends on a reliable advantage over single-program feedback and random pairing, which the current results do not yet establish.

  2. The experiments provide limited support for the independent contribution of differential pairing. Section 4.3 attributes the improvement from r0r_0 to r2r_2 to DEF, but this comparison also introduces refinement and execution feedback, so it does not isolate the effect of pairing. In the more direct ablations in Table 2, the full method improves over isolated refinement with execution feedback by only 0.16–1.16 points, and over random pairing with execution feedback by only 0.07–1.27 points. The repository-level results are also close: same-site pairing, best seed, and solo profiling achieve mean gains of 1.24%, 1.21%, and 1.19%, respectively. Thus, although the complete pipeline is effective, the results do not yet convincingly establish how much practical value differential pairing itself adds.

[1] Y. Peng, A. D. Gotmare, M. R. Lyu, C. Xiong, S. Savarese, and D. Sahoo, “PerfCodeGen: Improving performance of LLM generated code with execution feedback,” in Proceedings of the 2025 IEEE/ACM 2nd International Conference on AI Foundation Models and Software Engineering (FORGE), 2025.

[2] D. Huang et al., “EffiLearner: Enhancing efficiency of generated code via self-optimization,” in Advances in Neural Information Processing Systems, 2024.

[3] S. Waghjale, V. Veerendranath, Z. Z. Wang, and D. Fried, “ECCO: Can we improve model-generated code efficiency without sacrificing functional correctness?” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.

Questions for authors
  1. What information does DEF add to the refinement prompt? Appendix A.3.2 provides a template but leaves {guidance} unspecified. Please show a filled example and clarify whether DEF explicitly computes code or performance differences, or supplies two programs and their respective profiles for the model to compare.

  2. Where does structural matching provide additional value over random pairing and single-program feedback? The gains in Table 2 are relatively small. Which types of tasks benefit most from structural matching? With only three initial candidates by default, how many tasks have multiple eligible partners? Please clarify the random-pair selection rule, the behavior when no eligible neighbor exists, the meaning of “±,” and the number of runs.

  3. How do differences in experimental conditions affect the baseline comparisons? Table 1 reports averages over three runs for EffiPair but single runs for the baselines. Do the gaps to the closest baselines persist across repeated runs? EffiLearner uses a memory-integral-based acceptance criterion, and test access also differs across methods. How do the authors distinguish the effects of these settings from those of the proposed method?

  4. What conclusions are supported by the repository-level experiments? The mean gains of same-site pairing, best seed, and solo profiling are very close. On which instances does pairing provide meaningful benefits? Please specify the model and refinement configuration, explain why the other comparison groups have different numbers of valid instances, and clarify whether the conclusions hold when evaluated on the same instance set.

Reviewer_tkV4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes EffiPair, a test-time method for making LLM-generated code faster by contrasting two functionally correct programs that are structurally similar but have different execution speeds. Given an exsiting program, EffiPair shows LLMs a faster, structurally related implementation and uses their execution differences to guide optimization. EffiPair achieves the higher efficiency scores across several benchmarks.

Critical strengths

The use of AST-based structural similarity to identify contrasting programs is an interesting idea. Pairing structurally similar implementations with different execution efficiencies offers a lightweight and potentially useful way to construct comparative feedback for code optimization.

Critical weaknesses
  • The central claim is that contrasting structurally similar programs with different execution efficiencies provides directional guidance for further optimization. However, a performance difference between two programs primarily explains why one is faster than the other, rather than revealing how to further improve the faster program. The slower candidate may contain trivial inefficiencies that the faster candidate already avoids, while remaining optimization opportunities may lie in entirely different aspects of the implementation. Therefore, it is unclear why the selected contrast should provide actionable guidance for further optimization. This challenges the paper's core motivation and claimed novelty.
  • The paper assumes that high AST similarity implies localized implementation differences, but its AST representation is based on node-type counts and cosine similarity, which do not guarantee small or semantically meaningful changes. Moreover, independently sampling candidates and selecting pairs based on structural similarity and runtime differences does not ensure that the resulting contrasts are informative. A more direct alternative would be to construct pairs through controlled LLM-generated edits to an existing program, allowing differences to be explicitly constrained. The paper does not sufficiently justify its pairing strategy or compare it against such alternatives.
  • Although the experiments demonstrate improvements from paired refinement, they do not convincingly establish that these improvements arise from directional execution feedback. The gains could also result from providing additional implementation examples or from the LLM's independent optimization capabilities. In particular, the paper does not investigate whether successful refinements actually target the differences between paired programs. Additional ablations examining trivial versus meaningful contrasts, already-optimized versus unoptimized aspects, and the relationship between pair differences and subsequent code changes would be necessary to substantiate the claimed mechanism.
Questions for authors

Can the authors provide comparisons under matched total computational budgets, including LLM inference, correctness testing, and profiling overhead? Additionally, what is the end-to-end optimization cost relative to the runtime savings achieved, and under what circumstances would these savings amortize the additional optimization overhead?

Reviewer_K7ZeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a method for optimizing generated code via Differential Execution Feedback (DEF). The system first generate several candidates and evaluate them, the system then uses pairs of similar slow/fast programs with their execution feedback to further improve the generated code. The experiments showed that their method achieves competitive performance against other systems for code optimization across tasks and is training free/model agnostic.

Critical strengths

The overall method is straightforward and requires no training, making it easy to work with any LLM that is suitable for a given workload. With a default budget of 3 candidates and 2 refinement rounds, the method is overall token efficient as well.

The paper evaluated on 3 different tasks and different open/closed-LLMs, showing consistent improvement across different settings.

Critical weaknesses
  • There are some prior works which works on very similar problems, making the novelty of this work limited. RAS/AEGIS [1] uses slow-fast program examples to guide iterative optimization, and SBLLM [2] use multiple generations and their measured performance. These methods are not discussed or evaluated in this paper, which makes it hard to assess how much novelty does EffiPair contributes beyond pre-existing comparison based program optimization methods. EffiPair does use same-task pairs and paired profile runs, but the overall framing that comparisons provide better optimization seems to be less novel.

  • The proposed AST similarity is a cosine between bags of node-type counts, which throws away structural relationships like nesting/ordering etc. It is unclear if this similarity metric accounts for local differences reliably. Comparisons against code or text embedding models would be useful to see if the pairing heuristic in this paper is appropriate.

  • I'm not fully convinced on the usefulness of DEF as a whole, from table 2, it seems like the full DEF method only improves over isolated refinement + execution feedback (EF) by 0.16-1.16 eff@1 points, and improves over random pair + EF by 0.07-1.27 eff@1 points. The differences seems small over the variability reported, which seems like the 'differential' part in the DEF part is not contributing much. It would be useful to compare against RAS/SBLLM under similar evaluation budgets, and see whether using differential EF provide gains to existing frameworks.

  • The main results in table 1, the baselines only uses one sample, while EffiPair's r_0 already selects the best@3. For example on Mercury with gpt-4o-mini, the instruction baseline is 81.06, r_0 gets to 86.32 and r_2 is 86.62. Most of the improvement here is before DEF happened. Table 2 did do best-of-5 ablations, but is limited to the ENAMEL set, and uses a different set of language models. It would make the assessment of DEF gains clearer if these are also ran on EvalPerf and Mercury with the same LLMs.

[1] Anupam et al., LLM Program Optimization via Retrieval Augmented Search

[2] Gao et al., Search-Based LLMs for Code Optimization

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.