acceptodds
Withdrawn from ICLR 2027

How code helps different tasks? A decompositional lens on LLM post-training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating code data as a single corpus can obscure which types of code data benefit which models and downstream tasks. Effective data selection requires understanding both the benefits of individual categories and whether these benefits persist when categories are combined. We introduce a decompositional lens for studying these effects in LLM post-training. We first decompose an execution-verified code corpus into interpretable categories based on the computational patterns of its solutions. Through controlled fine-tuning experiments, we compare individual categories with a balanced mixture across instruction-tuned models on question answering, mathematics, and code generation. The resulting response maps reveal recurring gains in average question-answering performance, while the same category can improve one model or task and degrade another. The best-performing category also varies with the starting model and target task. We then compose compact mixtures guided by these results and examine whether benefits observed in individual categories persist under joint training. On selected model–task pairs, mixtures whose constituents each improve the target task outperform both their best constituent and full-corpus training while using roughly 10–15% of the full corpus. These exploratory findings illustrate a less is more pattern and highlight how the value of code data in post training depends on which categories are combined for which model and task.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM post-training, Code data, Data selection, Data mixtures, Cross-task transfer, Data efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Nd5VOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors post-train open models in the 3B-9B range on code, and evaluate performance change on downstream tasks. The code corpus is split by an LLM annotator into ten classes, such as "hashing" or "sorting". Post-training on code generally helps on many tasks. There's a complex and hard-to-describe-succinctly pattern of which of the ten classes help which downstream task. Finally, the authors show that it is possible to select just a subset of the examples and reach comparable performance, shortening the amount of time required for fine-tuning (though the fine-tuning dataset is pretty small to begin with).

Critical strengths
  1. The study is extensive: a lot of experiments are reported.
  2. The question of how and why code improves performance on non-code tasks is scientifically interesting (it's an empirical finding that we still don't really understand).
Critical weaknesses
  1. The presentation of the quantitative results is weak. The y axes are all over the place. There are no error bars or significance tests, which makes it hard to know what to make of the improvement (or degradation) of performance resulting from post-training.
  2. There are a lot of results but it is hard to know what insight to derive from them. Section 6, for example, is basically a list of empirical findings that go in a lot of different directions, duplicating the figures. The paragraph titles are all things like "results depend on everything, sometimes negative, sometimes positive", with no insight into the overall trends of why particular code categories help particular downstream tasks. There is no real attempt at analysis or synthesis beyond the raw outcomes. For ICLR 2025, I would say just the empirical contribution could justify considering this paper seriously, but nowadays when the kind of experiments described in the paper are feasible to complete in a few days with a good coding agent, I think our bar should be higher.
Questions for authors
  1. How were the ten classes determined?
  2. Was there human validation of the AI category annotations?
Reviewer_k9n3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how different computational categories of code data affect downstream performance when used for post-training instruction tuning. The authors partition an execution-verified Python corpus into ten interpretable categories (e.g., dynamic programming, hashing/sets, string parsing) and conduct controlled LoRA fine-tuning across six instruction-tuned backbones on eleven benchmarks spanning QA, math, and code generation. The central findings are that (i) balanced all-category code post-training consistently improves QA means but has mixed effects on math and code, (ii) the best single category depends on both the backbone and target task with frequent ranking reversals, and (iii) compact two- or three-category mixtures, composed from individually helpful categories, can outperform both their best constituent and full-corpus training on selected model–task pairs using small amount of the data.

Critical strengths
  1. Evaluation across multiple models. The paper evaluates six instruction-tuned backbones from several model families, providing broader evidence than a single-model analysis.

  2. Valuable findings on code-to-QA transfer. Improved average QA performance across all six backbones following code-only fine-tuning is a meaningful empirical contribution. The category-level analysis further illustrates how different types of code supervision affect QA performance.

Critical weaknesses

Major Weaknesses

  1. Categories are not necessarily orthogonal. A solution may involve multiple computational patterns, yet each example receives only one primary category. These categories may therefore conflate multiple skills, making it difficult to attribute effects to a specific pattern. Quantifying category overlap would strengthen the analysis.

  2. Token exposure is not controlled. Single-category runs match examples and optimization steps, but not input or supervised-token counts. Longer solutions may therefore provide more supervision, confounding comparisons. Appendix D.4 reports some token counts, but broader reporting and token-matched comparisons would help assess this issue.

  3. Generalizability is limited. The taxonomy and results are specific to Python examples derived from KodCode. Whether the findings extend to other languages or non-synthetic code sources remains unclear. Validation on another language or source would strengthen the broader conclusions.

  4. Limited explanation of the observed effects. The paper emphasizes empirical answers to its research questions but offers less insight into why code training improves QA performance. Appendix D explores possible explanations, but these remain inconclusive. A more focused discussion connecting the gains to plausible mechanisms and alternative explanations would improve the paper.

  5. Appendix organization and reporting need improvement. Descriptions of experimental settings are interspersed with references to specific files and checkpoints, making parts of the appendix difficult to follow. Reporting is also incomplete in places, for example, Appendix D.3 introduces language-specific rules (L) and other/ambiguous questions (O), but does not report their separate counts or performance results. Clearer organization and more complete reporting would improve the paper’s readability and transparency.

Minor Weaknesses and Suggestions

  1. Limited label validation. Category labels are assigned and reviewed by AI, without independent human validation or an independent inter-annotator agreement assessment. Human validation of a subset, especially ambiguous examples, would increase confidence in the partition.
  2. Missing details on category assignment. Reporting the model names and prompts used for annotation and review would improve reproducibility.
Questions for authors
  1. Could you provide token-matched comparisons, or at least extend the token-exposure statistics in Appendix D.4 to all categories and backbones? Reporting total input and supervised-token counts separately would help assess whether differences in token exposure contribute to the observed effects.

  2. How are the auxiliary tags assigned, and what purpose do they serve? Since they do not affect group assignment or training-subset selection, please clarify their role and provide examples pairing primary labels with auxiliary tags.

Reviewer_RRTEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how different code patterns help with diverse tasks. The authors decompose code data into categories based on computational patterns, fine-tune multiple LLMs on each category and their mixture. They find that the best performing category is dependent on models and tasks, and that compact mixture can outperform full training corpus tuning.

Critical strengths

The experiments are extensive spanning multiple categories, models, and tasks.

Critical weaknesses
  1. Overlapping categories make the analysis confounded. And there lacks human evaluation on the AI-assigned category labels.
  2. The novelty is limited beyond what is established in Waheed et al (2025), and the findings are expected and not informative enough, such as that a category's benefits depend on the task and the model itself.
Questions for authors

How did the authors come up with the ten categories? Have you evaluated how much the categories overlap with each other?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.