- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how much LLM should learn from its self-generated corrections based on execution feedback on code generation tasks. The authors propose Effective-Evidence Self-Distillation (EESD). It separates the relative support for different execution outcomes from the strength of the supporting evidence. It uses a Dirichlet posterior to estimate correction utility and uncertainty, which are then used to weight the training sample/loss. The authors evaluate the method on multiple models and coding tasks and show improvements in outcome prediction and downstream code generation.
Critical strengths
-
The paper motivates a important problem in self-improvement: how much a model should learn from its own corrections. Separating relative execution support from evidence strength is an intuitive idea.
-
The paper provides theoretical guarantees. It includes preservation of category ordering and an upper bound on the supervised weight relative to fixed-mass weighting.
-
The experiments cover multiple models and coding domains, and examine both outcome prediction and downstream code generation.
Critical weaknesses
-
RQ3 experiments compare EESD update with no update, but do not include matched correction SFT, fixed-mass training, or ablations of the uncertainty penalty. Thus, it is unclear whether the observed gains come from their method or simply from training on the corrected programs.
-
Results are a bit mixed. For RQ3, Table 7 reports four positive, one unchanged, and seven negative changes in Pass@1. In Table 10, the NLL benefit of effective mass also changes across different initialization seeds.
-
In Sections 3.1–3.2, the authors use execution relevance scores to compute transition support and effective evidence mass. However, the paper provides little evidence the resulting mass can identify corrections that are more reliable or useful for training. appendix B.3 explains the relevance scores are computed using a frozen lexical similarity between the code diff and execution descriptions, but the effectiveness of this relevance measure is not evaluated. Moreover, much of the improvement from additional execution history in RQ1 is also achieved by the fixed-mass baseline (Table 8). Although lower NLL indicates better outcome prediction, it does not directly validate the proposed uncertainty-based training weights or demonstrate their benefit for correction learning.
Questions for authors
-
Could the authors specify the source-level split and any overlap between correction collection, hyperparameter selection, and downstream evaluation (Appendix B.1)? For RQ1 and RQ2, please also clarify what “future outcome” refers to and how the predicted outcomes are separated from the visible executions.
-
I am confused about how the revised program (a_1) is generated and selected in Section 3.1. Are revisions that still fail some public tests included in training? If so, why is the entire revised program used as the SFT target despite its remaining errors? This matters because the utility vector assigns the same value to PRESERVE and UNRESOLVED. eg, with four uniformly weighted executions, "one FIX and three UNRESOLVED" and "one FIX and three PRESERVE" receive the same training weight, although the revised programs pass one and four tests, respectively.
-
For the execution relevance scores, could the authors provide more details about the text features and hashing procedure used by the 256-dimensional encoder in Appendix B.3? does each execution descriptor contain the results before correction, after correction, or both?