- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
EGAD is a training-free framework for improving code generation through entropy-guided revision. After producing an initial solution, it partitions Python code into AST-derived blocks, averages next-token entropy over lines and blocks, and flags lines whose entropy exceeds 1.5 times their enclosing block mean. A Suggestion LLM assesses whether flagged locations contain logical errors, and a Code LLM performs targeted revisions using the original problem and code context. The paper evaluates the framework on HumanEval, LiveCodeBench and a lite variant, and four MLE-bench tasks. It reports accuracy gains over direct generation, a sampling baseline, and an entropy-free reflection ablation, together with changes in validation-test agreement and runtime.
Critical strengths
- Practical design. The training-free framework uses model uncertainty to guide targeted code revision.
- Selective intervention. A separate diagnosis step avoids treating every high-entropy location as an error.
Critical weaknesses
- Unreliable empirical comparisons. Scores differ across the abstract, Figure 1, and Table 1, and the difficulty breakdowns do not consistently match the totals. The DeepConf implementation and warm-up accounting are also unclear, weakening both accuracy and efficiency claims.
- Insufficient validation of entropy localization. The ablation changes intervention scope and timing alongside the entropy signal, preventing clear attribution. The stated AST partition also creates single-function and single-line block limitations whose impact is not evaluated.
Questions for authors
- Which benchmark versions and problem sets were used, and how can the inconsistent scores and difficulty totals be reconciled?
- Which DeepConf components were implemented, and does EGAD retain its advantage after accounting for warm-up, token usage, and latency?
- Does AST-relative entropy outperform random targeting and whole-solution normalization under the same revision budget?
- Are held-out test scores excluded from revision decisions, and do improvements in MLE-bench validation–test agreement accompany better test performance?