- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors post-train open models in the 3B-9B range on code, and evaluate performance change on downstream tasks. The code corpus is split by an LLM annotator into ten classes, such as "hashing" or "sorting". Post-training on code generally helps on many tasks. There's a complex and hard-to-describe-succinctly pattern of which of the ten classes help which downstream task. Finally, the authors show that it is possible to select just a subset of the examples and reach comparable performance, shortening the amount of time required for fine-tuning (though the fine-tuning dataset is pretty small to begin with).
Critical strengths
- The study is extensive: a lot of experiments are reported.
- The question of how and why code improves performance on non-code tasks is scientifically interesting (it's an empirical finding that we still don't really understand).
Critical weaknesses
- The presentation of the quantitative results is weak. The y axes are all over the place. There are no error bars or significance tests, which makes it hard to know what to make of the improvement (or degradation) of performance resulting from post-training.
- There are a lot of results but it is hard to know what insight to derive from them. Section 6, for example, is basically a list of empirical findings that go in a lot of different directions, duplicating the figures. The paragraph titles are all things like "results depend on everything, sometimes negative, sometimes positive", with no insight into the overall trends of why particular code categories help particular downstream tasks. There is no real attempt at analysis or synthesis beyond the raw outcomes. For ICLR 2025, I would say just the empirical contribution could justify considering this paper seriously, but nowadays when the kind of experiments described in the paper are feasible to complete in a few days with a good coding agent, I think our bar should be higher.
Questions for authors
- How were the ten classes determined?
- Was there human validation of the AI category annotations?