acceptodds
Under review as a conference paper at ICLR 2027

The Function-Style Trade-off: Understanding LLM Code Generation from a Novel Perspective

Abstract

LLMs have made significant progress on code generation, yet their outputs often suffer from poor readability, inconsistent naming, missing documentation, and jumbled control flow. Current benchmarks overemphasize execution correctness while lacking scalable code-style metrics, hindering the development of style-aware code generation. To fill this gap, we construct a fine-grained code-style evaluation benchmark and propose a graph-based framework using Augmented AST that captures structural style features beyond the token level and enables node-level attribution of style decisions. Experiments reveal a Function-Style Trade-off: improving style naturalness degrades functional performance. We provide an information-theoretic interpretation of this phenomenon, framing fine-tuning as reallocation of representational capacity between natural language and code distributions. Extensive experiments demonstrate that the proposed structural style signal transforms the Function-Style Trade-off from an uncontrollable statistical fluctuation into an analyzable and optimizable engineering problem. This work underscores the necessity of incorporating software quality into LLM code evaluation.

Primary area:
interpretability and explainable AI
Keywords:
LLM Interpretation, Agent Coding, Function-Style Trade-off, Augmented AST

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_Z682OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies code-style evaluation and fine-tuning for LLM-generated Python. It introduces paired human/LLM datasets, an augmented-AST source classifier, and a procedure that selects unit-test-passing candidates using classifier scores. It also proposes an information-theoretic account of a Function-Style Trade-off. The evaluation includes classifier comparisons, cross-model style scoring, node-level perturbations, and fine-tuning experiments on HumanEval+ and CCE.

Critical strengths
  1. Evaluating documentation, code organization, and readability alongside functional correctness is a worthwhile direction.
  2. The paired-data construction and test-first candidate selection form a concrete pipeline. Appendices E and J provide useful architecture and training details.
Critical weaknesses
  1. The experiments do not sufficiently support the paper’s central claims. The introduction attributes the Function-Style Trade-off to competition for finite representational capacity, but Appendix J.5 explains the main SFT failures through output-format mismatch and distribution shift. These effects are not separated from the proposed capacity mechanism, and Appendix F.17 acknowledges that the theory itself does not derive a negatively sloped trade-off. The observed metric changes therefore do not establish the explanation offered by the paper. The interpretation of style improvement also needs more evidence: lower P(LLM) shows greater similarity to the classifier’s human-source distribution, but its relationship to readability or maintainability has not been independently validated. These gaps affect the main contribution: the experiments do not yet establish either the proposed mechanism or the broader improvement in code quality.
  2. Inconsistent evaluation conditions and missing controls make the reported gains difficult to interpret. Appendix J.2 uses the same 164 HumanEval+ tasks for functional training and final evaluation, and RFT selects training targets from those tasks. The increase from 53.8% to 82.9% therefore does not demonstrate improvement on unseen problems. There is no matched RFT baseline that selects passing candidates without style ranking. E6 already uses the classifier, so E6 versus E7 measures the effect of adding CCE, not the contribution of style ranking. The CCE comparisons have a separate inconsistency: Table 13 marks E1–E3 as using a 256-token limit, while later runs use 512 tokens, which affects syntax validity and the composite quality score. Together with using the same classifier for candidate selection and final style evaluation, these design choices leave the benefits of the proposed style signal insufficiently isolated. Task-disjoint evaluation, a passing-candidate control without style ranking, and comparable evaluation conditions are needed to substantiate the headline result.
Questions for authors
  1. Clarification of the trade-off claim. Is the capacity-based account intended as an empirically supported mechanism or a conceptual interpretation? Appendix J.5 attributes the main SFT failures to output-format mismatch and distribution shift. Which reported experiment distinguishes these effects from the proposed capacity mechanism? This distinction matters for assessing the paper’s main theoretical contribution.
  2. Evidence for the code-quality claim. Is there evaluation independent of the classifier used for candidate selection showing that the resulting code is more readable or maintainable, with functional correctness controlled? The feature analysis helps explain differences between human and LLM code, but the connection from source-distribution similarity to improved code quality remains unclear.
  3. Evidence for the functional gain. Are results available on tasks excluded from both fine-tuning and RFT candidate selection? Is there also a matched RFT control that selects passing candidates without style ranking? E6 already uses the classifier, so E6 versus E7 does not isolate the contribution of style ranking. These comparisons would determine whether the reported gain generalizes and whether the proposed style signal contributes beyond correctness-filtered self-training.
  4. Clarification of the evaluation protocol. Table 13 marks E1–E3 as using a 256-token CCE limit, while later runs use 512 tokens. Which comparisons have been rerun under a common generation budget, and do they support the same conclusions about the trade-off and joint improvement? This matters because truncation affects syntax validity and the composite quality score.
Reviewer_bFhjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies code style alongside functional correctness. It introduces CCE-Full and CCE-Worse, two paired human/LLM code datasets, and an Augmented AST classifier with structural and semantic graph channels. The classifier's source probability is combined with documentation, complexity, and syntax metrics into a style-quality score. The authors use this signal for sample selection and preference-based fine-tuning of Qwen2.5-Coder-1.5B-Instruct. They report better style scores and a HumanEval+ pass-rate increase from 53.8% to 82.9%, and propose an information-theoretic interpretation of the relationship between function and style. My main concerns are the training/evaluation overlap behind that gain and whether the proposed metric measures code quality rather than code origin.

Critical strengths
  1. Evaluating generated code beyond test passing is a worthwhile problem. The paired datasets cover industrial and educational/competition code at several granularities, and the breakdown into documentation, complexity, and structural features is useful. This could be a valuable resource for studying differences between human code and regenerated code, even if the stronger quality claims need further validation.

  2. The graph representation makes the classifier more inspectable than a single text-level score. The architecture ablations and the 100-snippet perturbation study in Appendix H.2.4 provide concrete evidence about which input features affect its decisions. I also appreciate that the appendices disclose limitations of the proxy, output-format problems, and weaker performance on rewritten code. Those details are important for interpreting the results.

Critical weaknesses
  1. The main fine-tuning result is not a held-out evaluation, and the comparisons do not isolate the effect of style supervision. Appendix J.2 explicitly says that all 164 HumanEval+ tasks are used for functional SFT and that evaluation uses the full same set. Appendix J.3 samples candidates for these tasks, filters them with their unit tests, and repeats selected solutions 8 or 15 times for training. The 53.8% to 82.9% result therefore measures improvement on tasks exposed during adaptation; it does not establish better code generation on unseen problems. The disjoint CCE evaluation split is useful, but its syntax rate cannot replace held-out functional tests. There is also a substantial format confound. Appendices J.3 and J.5 attribute the SFT collapse to training on function bodies while RFT uses complete functions. That comparison cannot establish that a function-style trade-off caused the collapse. E6 already uses the style classifier for RFT selection, so E6 versus E7 tests adding CCE data, not adding style selection versus correctness-only selection. A matched correctness-only RFT baseline and an SFT baseline with the same output format are needed. CCE comparisons also mix 256- and 512-token protocols for some rows in Table 13. These are central experimental issues, not small reporting omissions.

  2. The evidence does not yet validate the proposed style-quality measure or the claimed explanation. The classifier is trained to recognize human versus model origin. Appendix M argues that professional human code makes this a useful quality proxy, but no independent developer-rating study establishes that lower P(LLM) means more readable or maintainable code. Weak functional equivalence here means addressing the same specification, not passing the same tests (Appendix B.3). The classifier could therefore exploit missing functionality, repository conventions, or generator-specific patterns. Optimizing against this classifier and then reporting improvement with it does not resolve that concern. The heuristic quality score still assigns over 60% of its weight to classifier mean and variability (Appendix D). The generalization evidence is also narrower than Section 5.2 suggests: Appendix K obtains 83.12% after training on an 80/20 split of HumanEval+ and EvoEval, rather than transferring a frozen CCE classifier to those datasets. Table 3 gives 79.41% for the selected SlotGCN, versus 80.15% for CodeBERT with metadata, so the structural benefit needs clearer evidence than overall accuracy. Finally, Equation (1) is an information identity under matched joint information; it does not force a trade-off unless the interaction term is constrained. Appendix F.17 correctly acknowledges this. The experiments do not establish the information-level matching or report the capacity-matched control result described in F.13. The introduction's intrinsic-trade-off claim is therefore stronger than either the theory or experiments support. Taken together, these issues make me recommend rejection in the current form.

Questions for authors
  1. Is there any task-disjoint functional evaluation beyond the protocol in Appendix J.2? Please report the fine-tuned models on tasks whose prompts, solutions, and tests were not used for SFT, rejection sampling, or preference construction. This is necessary to assess the headline functional improvement.

  2. With training tasks, output format, sample budget, repetitions, and evaluation limits held fixed, how do correctness-only RFT and style-selected RFT compare? Does the SFT collapse persist when canonical targets are presented as complete functions? These controls would distinguish the proposed trade-off from format and data-selection effects.

  3. Does the style score agree with blinded developer judgments on functionally correct code, including unseen repositories and generators? Please also clarify whether the CCE split groups paired examples and nested class/function/line snippets by source repository or file. A frozen-classifier transfer test would help separate general style assessment from source recognition.

  4. Which experiment verifies the matched information-level assumption, and where are the results for the capacity-matched non-structural control in Appendix F.13? If those quantities cannot be tested here, please narrow the theory to a conceptual interpretation and reconcile the introduction's intrinsic-trade-off claim with the conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.