- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to train code generation models to produce functionally correct, faster, and more memory-efficient programs in a single generation. EffiCode-OPD first samples programs from the student and retains trajectories that pass all tests and have a more efficient reference implementation. A teacher provides next-token distributions conditioned on the student-generated prefixes and privileged context, including reference code, runtime and peak-memory measurements for both programs, and AST structural differences. The student sees only the problem and its own prefix. Training minimizes the reverse KL divergence from the student distribution to the teacher distribution, with larger weights assigned to tokens in selected AST regions, such as loops, calls, and container operations. At inference time, the student generates programs without the teacher, reference implementations, or execution feedback. The authors evaluate the method on four code-efficiency benchmarks and multiple model families against SFT, PPO, and several inference-time optimization methods, reporting improvements in correctness and efficiency. They also provide ablations of the privileged context, AST weighting, and reference quality.
Critical strengths
-
The method shifts the additional cost of efficiency optimization to training, offering practical deployment value. At inference time, the student requires only a single generation, without code execution, teacher queries, or candidate reranking. The end-to-end inference measurements in Appendix E also show substantially lower inference time than the evaluated inference-time optimization methods.
-
The ablations provide some evidence that the runtime and memory signals have distinct effects. In Table 2, removing runtime information primarily affects runtime metrics, while removing memory information primarily affects memory metrics. Removing AST weighting while retaining the full privileged context also degrades the reported performance. Table 6 further examines individual information sources.
Critical weaknesses
-
The efficiency comparisons do not establish gains on the same tasks. Appendix D averages efficiency metrics over each method’s own correctly solved problems, so lower averages may reflect differences in task composition. Reference normalization does not resolve this issue. The abstract’s 17.9% runtime and 19.4% memory reductions come from CodeLlama-7B on 104 Effibench-X problems, with only 14.5% Pass@1. Paired comparisons are needed to determine how much of these gains comes from better programs.
-
The experiments provide limited support for the claimed token-level mechanism. Appendix G assigns weights through fixed syntactic rules, while the teacher receives reference code and whole-program measurements. The ablations show that weighting helps in the tested setting, but do not explain why the selected AST regions are particularly useful. The component ablations cover only CodeLlama-7B on EffiBench and use the same conditional efficiency metrics. Unclear annotation definitions in Figures 1 and 5 further weaken the mechanism evidence.
Questions for authors
-
Can the efficiency gains be confirmed on the same problems? Please report efficiency metrics and paired runtime/memory ratios on the intersection of correctly solved problems, together with the intersection size and counts solved exclusively by either method. Please also clarify how Pass@1 is sampled and aggregated.
-
Do the component gains generalize across models and benchmarks? The ablations in Tables 2 and 6 cover only CodeLlama-7B on EffiBench, leaving it unclear whether the same components contribute consistently in other settings.
-
How are the token-level diagnostics in Figures 1 and 5 defined? In Figure 1, how is the reference’s next token aligned with a student prefix that uses different code? Is the probability measured from the student or the context-conditioned teacher, on training or held-out problems? In Figure 5, how is the “first inefficient token” identified?