acceptodds
Withdrawn from ICLR 2027

Learning to Generate Efficient Code with Privileged Efficiency Context and AST-Guided Supervision

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) have achieved remarkable success in code generation, yet functionally correct programs often present sub-optimal inefficiency in runtime efficiency and memory usage. Recently, there are some method have attempted to alleviate this problem. While effective, these methods only provide sparse feedback at the program level after generation is complete, making it difficult to provide fine-grained supervision during generation. To address this limitation, we propose EffiCode-OPD, an on-policy distillation (OPD)-based framework for efficient code generation to provide efficiency-aware token-level supervision during training. Specifically, EffiCode-OPD samples on-policy rollouts from the student model and uses a stronger teacher model to provide token-level supervision conditioned on an efficiency context that includes reference code, runtime, memory usage, and AST information. In addition, we design an AST-guided token weighting scheme that focuses the distillation objective on efficiency-relevant code segments. Experiments on representative code efficiency benchmarks show the effectiveness of EffCode-OPD for generating efficient code. For example, EffiCode-OPD reduces average runtime by up to 17.9% and memory usage by up to 19.4% relative to the strongest baseline on EFFIBENCH-X, while preserving functional correctness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Code Efficiency, On-Policy Distillation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ituvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to train code generation models to produce functionally correct, faster, and more memory-efficient programs in a single generation. EffiCode-OPD first samples programs from the student and retains trajectories that pass all tests and have a more efficient reference implementation. A teacher provides next-token distributions conditioned on the student-generated prefixes and privileged context, including reference code, runtime and peak-memory measurements for both programs, and AST structural differences. The student sees only the problem and its own prefix. Training minimizes the reverse KL divergence from the student distribution to the teacher distribution, with larger weights assigned to tokens in selected AST regions, such as loops, calls, and container operations. At inference time, the student generates programs without the teacher, reference implementations, or execution feedback. The authors evaluate the method on four code-efficiency benchmarks and multiple model families against SFT, PPO, and several inference-time optimization methods, reporting improvements in correctness and efficiency. They also provide ablations of the privileged context, AST weighting, and reference quality.

Critical strengths
  1. The method shifts the additional cost of efficiency optimization to training, offering practical deployment value. At inference time, the student requires only a single generation, without code execution, teacher queries, or candidate reranking. The end-to-end inference measurements in Appendix E also show substantially lower inference time than the evaluated inference-time optimization methods.

  2. The ablations provide some evidence that the runtime and memory signals have distinct effects. In Table 2, removing runtime information primarily affects runtime metrics, while removing memory information primarily affects memory metrics. Removing AST weighting while retaining the full privileged context also degrades the reported performance. Table 6 further examines individual information sources.

Critical weaknesses
  1. The efficiency comparisons do not establish gains on the same tasks. Appendix D averages efficiency metrics over each method’s own correctly solved problems, so lower averages may reflect differences in task composition. Reference normalization does not resolve this issue. The abstract’s 17.9% runtime and 19.4% memory reductions come from CodeLlama-7B on 104 Effibench-X problems, with only 14.5% Pass@1. Paired comparisons are needed to determine how much of these gains comes from better programs.

  2. The experiments provide limited support for the claimed token-level mechanism. Appendix G assigns weights through fixed syntactic rules, while the teacher receives reference code and whole-program measurements. The ablations show that weighting helps in the tested setting, but do not explain why the selected AST regions are particularly useful. The component ablations cover only CodeLlama-7B on EffiBench and use the same conditional efficiency metrics. Unclear annotation definitions in Figures 1 and 5 further weaken the mechanism evidence.

Questions for authors
  1. Can the efficiency gains be confirmed on the same problems? Please report efficiency metrics and paired runtime/memory ratios on the intersection of correctly solved problems, together with the intersection size and counts solved exclusively by either method. Please also clarify how Pass@1 is sampled and aggregated.

  2. Do the component gains generalize across models and benchmarks? The ablations in Tables 2 and 6 cover only CodeLlama-7B on EffiBench, leaving it unclear whether the same components contribute consistently in other settings.

  3. How are the token-level diagnostics in Figures 1 and 5 defined? In Figure 1, how is the reference’s next token aligned with a student prefix that uses different code? Is the probability measured from the student or the context-conditioned teacher, on training or held-out problems? In Figure 5, how is the “first inefficient token” identified?

Reviewer_2qEAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EffiCode-OPD, whose main idea is to use a powerful teacher model with access to privileged information, including efficient reference code, runtime measurements, memory usage, and Abstract Syntax Tree (AST) analysis, to supervise a smaller student model at the token level, using on-policy distillation. The results show that EffiCode-OPD outperforms several baselines on efficiency and pass@1 in some efficient code generation benchmarks.

Critical strengths
  • The use of AST structural comparisons to enrich the teacher's efficiency context and AST-guided token weighting to emphasize efficiency-relevant code regions is an interesting design. This provides more targeted token-level supervision, with ablation studies supporting its effectiveness.
  • EffiCode-OPD consistently improves execution time, memory efficiency, and functional correctness across multiple benchmarks and model families. Notably, these improvements require no additional inference-time computation, demonstrating the practical effectiveness of the approach.
Critical weaknesses
  • The core motivation and effectiveness of the proposed privileged efficiency context are not sufficiently established. Since the teacher is directly provided with an efficient reference solution, it is unclear whether the resulting token-level supervision genuinely captures efficiency-specific knowledge from runtime, memory, and AST information, or primarily reflects the teacher's imitation of the reference solution. Although the proposed method differs mathematically from standard supervised fine-tuning (SFT) by operating on student-generated prefixes and matching teacher distributions, the underlying learning signal may still be largely driven by the reference code. The existing ablations do not adequately disentangle these possibilities. In particular, removing the reference code from the teacher prompt does not eliminate reference-derived information, since the AST structural contrast is still computed against the reference solution. Furthermore, the paper does not directly analyze how privileged efficiency information changes the teacher's token distributions or whether these changes correspond to genuinely more efficient coding decisions. Consequently, the necessity of the proposed context and its substantive advantage over reference-guided on-policy distillation remain unclear. More targeted controlled experiments and token-level analyses are needed to substantiate the claimed methodological contribution.
  • The paper samples (K=5) student rollouts per programming problem, but the necessity and effectiveness of this design choice are insufficiently justified. According to the proposed formulation, each eligible rollout is independently paired with its own execution profile and privileged efficiency context, and the resulting distillation losses are averaged. Therefore, the five rollouts do not jointly contribute to teacher supervision or explicitly exploit relative efficiency differences among candidate solutions. From an expectation perspective, averaging independently sampled rollout losses does not change the underlying expected objective compared with using a single rollout; it primarily reduces sampling variance and increases the probability of obtaining eligible samples after correctness filtering. Nevertheless, the paper does not investigate whether these benefits justify the additional rollout generation, execution profiling, and teacher computation. In particular, no ablation varying (K), or comparison under a fixed sampling or computational budget, is provided. It is therefore unclear whether the observed improvements arise from the proposed efficiency-aware distillation mechanism or simply from collecting more training trajectories. A systematic comparison of different rollout counts, including (K=1), together with training costs, eligible-sample rates, and downstream performance, is necessary to establish the effectiveness and computational efficiency of this design.
Questions for authors

The proposed method excludes student rollouts when no more efficient reference solution is available. What proportion of student rollouts is discarded by this criterion across different benchmarks and model sizes? Given that some trained students achieve better average efficiency than the reference solutions, how does the method continue improving when the available reference is no longer superior? Could this filtering strategy limit further improvements for stronger student models?

Reviewer_VktZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EffiCode-OPD, a novel technique for training LLMs to generate more efficient code through on-policy distillation. Specifically, EffiCode-OPD samples correct and more efficient code rollouts from the student model, then uses a teacher model with access to privileged efficiency context to provide dense token-by-token supervision. Furthermore, EffiCode-OPD assigns higher loss weights to tokens deemed more important for code efficiency based on AST analysis. Results across different LLMs and benchmarks show that EffiCode-OPD can improve code efficiency while maintaining or even improving code correctness.

Critical strengths
  • The paper explores an interesting and important research direction.
Critical weaknesses
  • The models used in the main experiments (Table 1) are outdated, making it unclear whether EffiCode-OPD’s effectiveness generalizes to more recent models such as Qwen 3.8 27B.
  • The heuristics used to incorporate AST information into training are not sufficiently justified. For example, the AST analysis covers five types of code structures that may be relevant to runtime and memory usage, but the effect of each type on training remains unclear.
  • The paper lacks case studies or in-depth analysis showing how different types of privileged efficiency context influence the teacher model’s logits for tokens that are important for code efficiency.
Questions for authors
  • Do the efficiency gains achieved by EffiCode-OPD generalize to more recent settings or benchmarks, such as SWE-Bench Verified?
  • The citation format in the paper is incorrect; \citep should be used instead of \citet.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.