- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how different synthetic transformations and context-packing formats of GitHub commits affect continued pretraining ("mid-training") of code LLMs under a fixed 1B-token budget. Using gpt-oss-120b on Python commits from CommitPack, the authors construct SynthCode (5.1M documents from 6.4M commits) and evaluate 11 training strategies: four single-artifact formats (RAW, CLEAN, NOTEBOOK, BLOG) and seven multi-artifact packed formats combining pre-commit code x_old, post-commit code x_new, and human or synthetic commit messages, plans, or code reviews. Each strategy is trained for 1B tokens across six base models from two families (Qwen3-4B/8B/14B and Gemma4-12B/26B/31B) under two instruction-replay mixtures: weak (10% general code QA from Evol-CodeAlpaca and Magicoder OSS-Instruct) and strong (adding ~7% whole-file edit instructions derived from EDIT-Bench's training split). Additional experiments on Qwen3-8B and Gemma4-12B examine warmup-stable-decay (WSD) annealed checkpoints at 250M/500M/1B tokens, equal-token strategy mixtures, and a 0.5B-token source-overlap study (DISJOINT, SHARED, REPEAT). Checkpoints are evaluated on EDIT-Bench (108 problems, 1 sample/problem) and LiveCodeBench v6 (131 problems, 5 samples/problem).
Critical strengths
None
Critical weaknesses
1: The tasks the paper targets, EDIT-Bench and LiveCodeBench, are already saturated. They were interesting coding benchmarks in 2025, but models in 2026 has already surpassed this stage and are targeting more complex agentic coding workflows.
2: The method proposed in paper is hard to generalise. It basically offers a dataset specific to these obsolete coding tasks. Also for table 3, the official instruction tuning version of the models perform scales better(0.6-0.8 range for livecodebench) than the reported numbers(0.1-0.2 range).
Questions for authors
I do not have questions for authors. The paper has fundamental flaw.
Response to Reviewer 2kuw
Thank you for the review. We believe both weaknesses come from comparing our checkpoints with post-trained, instruction-tuned releases, whereas the paper studies data choices in a 1B-token mid-training stage on base checkpoints. We clarify that scope below.
1. Benchmark saturation (W1). The benchmarks are not saturated for the models we study: no checkpoint in the paper exceeds .66 on EDIT-Bench or .23 pass@1 on LiveCodeBench. We agree that agentic workflows are where coding models are now evaluated. However, a base checkpoint after mid-training does not yet follow instructions or call tools, so an agentic score would mainly measure the post-training that follows rather than the data choice we study. We release the corpus and checkpoints so that this follow-up can be run. The closest proxy in the revision is the large-file test (Reviewer 1, topic 2): the strategies differ most when the edit sits inside a 14k-character file.
2. Generality and Table 3 (W2). The SynthCode corpus is not built for these benchmarks: it is derived from general Python commits in CommitPack, with no benchmark-specific construction. The revision tests generality directly with a third model family (OLMo-2 7B), a second teacher, two further editing benchmarks and a large-file test. The 0.6-0.8 figures are not comparable with Table 3: they belong to post-trained, instruction-tuned releases, whereas every model in the paper is a base checkpoint plus 1B tokens of mid-training, evaluated on the 131 LiveCodeBench problems dated February-April 2025, chosen to postdate the base models' training data. On that window the base checkpoints themselves score .12-.21 (Table 3, first row).
3. What a reader learns. (i) A real commit packed as an edit is at least as good as every generated rewriting of it (Reviewer 1, topics 1 and 2). (ii) What that packing teaches is correct editing inside large files; raw code does not teach it, and edit-instruction data teaches it only in part (Reviewer 1, topic 2). (iii) One- and two-line commits should be kept (Reviewer 2, topic 4). (iv) Two training runs of one strategy differ by about .02 on EDIT-Bench, as much as most differences between strategies, so data comparisons at this scale must be reported against it (Reviewer 1, topic 3).
We hope the added model family, second teacher, two editing benchmarks and large-file test address the concern about generality, and that you will reconsider your assessment.