- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies code-style evaluation and fine-tuning for LLM-generated Python. It introduces paired human/LLM datasets, an augmented-AST source classifier, and a procedure that selects unit-test-passing candidates using classifier scores. It also proposes an information-theoretic account of a Function-Style Trade-off. The evaluation includes classifier comparisons, cross-model style scoring, node-level perturbations, and fine-tuning experiments on HumanEval+ and CCE.
Critical strengths
- Evaluating documentation, code organization, and readability alongside functional correctness is a worthwhile direction.
- The paired-data construction and test-first candidate selection form a concrete pipeline. Appendices E and J provide useful architecture and training details.
Critical weaknesses
- The experiments do not sufficiently support the paper’s central claims. The introduction attributes the Function-Style Trade-off to competition for finite representational capacity, but Appendix J.5 explains the main SFT failures through output-format mismatch and distribution shift. These effects are not separated from the proposed capacity mechanism, and Appendix F.17 acknowledges that the theory itself does not derive a negatively sloped trade-off. The observed metric changes therefore do not establish the explanation offered by the paper. The interpretation of style improvement also needs more evidence: lower P(LLM) shows greater similarity to the classifier’s human-source distribution, but its relationship to readability or maintainability has not been independently validated. These gaps affect the main contribution: the experiments do not yet establish either the proposed mechanism or the broader improvement in code quality.
- Inconsistent evaluation conditions and missing controls make the reported gains difficult to interpret. Appendix J.2 uses the same 164 HumanEval+ tasks for functional training and final evaluation, and RFT selects training targets from those tasks. The increase from 53.8% to 82.9% therefore does not demonstrate improvement on unseen problems. There is no matched RFT baseline that selects passing candidates without style ranking. E6 already uses the classifier, so E6 versus E7 measures the effect of adding CCE, not the contribution of style ranking. The CCE comparisons have a separate inconsistency: Table 13 marks E1–E3 as using a 256-token limit, while later runs use 512 tokens, which affects syntax validity and the composite quality score. Together with using the same classifier for candidate selection and final style evaluation, these design choices leave the benefits of the proposed style signal insufficiently isolated. Task-disjoint evaluation, a passing-candidate control without style ranking, and comparable evaluation conditions are needed to substantiate the headline result.
Questions for authors
- Clarification of the trade-off claim. Is the capacity-based account intended as an empirically supported mechanism or a conceptual interpretation? Appendix J.5 attributes the main SFT failures to output-format mismatch and distribution shift. Which reported experiment distinguishes these effects from the proposed capacity mechanism? This distinction matters for assessing the paper’s main theoretical contribution.
- Evidence for the code-quality claim. Is there evaluation independent of the classifier used for candidate selection showing that the resulting code is more readable or maintainable, with functional correctness controlled? The feature analysis helps explain differences between human and LLM code, but the connection from source-distribution similarity to improved code quality remains unclear.
- Evidence for the functional gain. Are results available on tasks excluded from both fine-tuning and RFT candidate selection? Is there also a matched RFT control that selects passing candidates without style ranking? E6 already uses the classifier, so E6 versus E7 does not isolate the contribution of style ranking. These comparisons would determine whether the reported gain generalizes and whether the proposed style signal contributes beyond correctness-filtered self-training.
- Clarification of the evaluation protocol. Table 13 marks E1–E3 as using a 256-token CCE limit, while later runs use 512 tokens. Which comparisons have been rerun under a common generation budget, and do they support the same conclusions about the trade-off and joint improvement? This matters because truncation affects syntax validity and the composite quality score.