acceptodds
Under review as a conference paper at ICLR 2027

SynthCode: Not All Synthetic Code Is Worth the Same Tokens

Abstract

Synthetic data now underpins code pretraining, yet the decisions that produce it are made largely by intuition: which transformation to apply to a source file, whether to train on the generated artifact alone or pack it with its source, and whether a fixed token budget is better spent on new documents or on new views of the same ones. Because these choices are rarely compared at equal cost, their relative value is unknown. We introduce SynthCode, a 5.1M-document corpus distilled from 6.4M Python commits with gpt-oss-120b, together with a token-matched protocol for isolating the value of each generation decision. The protocol covers sixteen recipes: single-transformation distillation, five context-packing formats, equal-token mixtures, and a three-way source-overlap study that prices a second view of one document against a first view of two. Every recipe is trained for the same number of tokens under identical hyperparameters, so differences are attributable to the data rather than to compute or tuning. We then train the best-performing recipe into six base models spanning two families and 4B–31B parameters (Qwen3-4B/8B/14B, Gemma4-12B/26B/31B). We release the corpus, the generation and filtering pipeline, and all checkpoints.

Primary area:
datasets and benchmarks
Keywords:
coding LLMs, synthetic-data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2kuwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how different synthetic transformations and context-packing formats of GitHub commits affect continued pretraining ("mid-training") of code LLMs under a fixed 1B-token budget. Using gpt-oss-120b on Python commits from CommitPack, the authors construct SynthCode (5.1M documents from 6.4M commits) and evaluate 11 training strategies: four single-artifact formats (RAW, CLEAN, NOTEBOOK, BLOG) and seven multi-artifact packed formats combining pre-commit code x_old, post-commit code x_new, and human or synthetic commit messages, plans, or code reviews. Each strategy is trained for 1B tokens across six base models from two families (Qwen3-4B/8B/14B and Gemma4-12B/26B/31B) under two instruction-replay mixtures: weak (10% general code QA from Evol-CodeAlpaca and Magicoder OSS-Instruct) and strong (adding ~7% whole-file edit instructions derived from EDIT-Bench's training split). Additional experiments on Qwen3-8B and Gemma4-12B examine warmup-stable-decay (WSD) annealed checkpoints at 250M/500M/1B tokens, equal-token strategy mixtures, and a 0.5B-token source-overlap study (DISJOINT, SHARED, REPEAT). Checkpoints are evaluated on EDIT-Bench (108 problems, 1 sample/problem) and LiveCodeBench v6 (131 problems, 5 samples/problem).

Critical strengths

None

Critical weaknesses

1: The tasks the paper targets, EDIT-Bench and LiveCodeBench, are already saturated. They were interesting coding benchmarks in 2025, but models in 2026 has already surpassed this stage and are targeting more complex agentic coding workflows.

2: The method proposed in paper is hard to generalise. It basically offers a dataset specific to these obsolete coding tasks. Also for table 3, the official instruction tuning version of the models perform scales better(0.6-0.8 range for livecodebench) than the reported numbers(0.1-0.2 range).

Questions for authors

I do not have questions for authors. The paper has fundamental flaw.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2kuw

Thank you for the review. We believe both weaknesses come from comparing our checkpoints with post-trained, instruction-tuned releases, whereas the paper studies data choices in a 1B-token mid-training stage on base checkpoints. We clarify that scope below.

1. Benchmark saturation (W1). The benchmarks are not saturated for the models we study: no checkpoint in the paper exceeds .66 on EDIT-Bench or .23 pass@1 on LiveCodeBench. We agree that agentic workflows are where coding models are now evaluated. However, a base checkpoint after mid-training does not yet follow instructions or call tools, so an agentic score would mainly measure the post-training that follows rather than the data choice we study. We release the corpus and checkpoints so that this follow-up can be run. The closest proxy in the revision is the large-file test (Reviewer 1, topic 2): the strategies differ most when the edit sits inside a 14k-character file.

2. Generality and Table 3 (W2). The SynthCode corpus is not built for these benchmarks: it is derived from general Python commits in CommitPack, with no benchmark-specific construction. The revision tests generality directly with a third model family (OLMo-2 7B), a second teacher, two further editing benchmarks and a large-file test. The 0.6-0.8 figures are not comparable with Table 3: they belong to post-trained, instruction-tuned releases, whereas every model in the paper is a base checkpoint plus 1B tokens of mid-training, evaluated on the 131 LiveCodeBench problems dated February-April 2025, chosen to postdate the base models' training data. On that window the base checkpoints themselves score .12-.21 (Table 3, first row).

3. What a reader learns. (i) A real commit packed as an edit is at least as good as every generated rewriting of it (Reviewer 1, topics 1 and 2). (ii) What that packing teaches is correct editing inside large files; raw code does not teach it, and edit-instruction data teaches it only in part (Reviewer 1, topic 2). (iii) One- and two-line commits should be kept (Reviewer 2, topic 4). (iv) Two training runs of one strategy differ by about .02 on EDIT-Bench, as much as most differences between strategies, so data comparisons at this scale must be reported against it (Reviewer 1, topic 3).

We hope the added model family, second teacher, two editing benchmarks and large-file test address the concern about generality, and that you will reconsider your assessment.

Reviewer_EyuJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically investigates how code rewriting, the organization of editing context, and source diversity affect the training utility of synthetic code under a fixed token budget. The authors construct the SynthCode corpus and conduct budget-matched comparisons of training recipes across six models, providing an empirical testbed for synthetic code data design. The results indicate that the benefits of different recipes depend on the downstream task and editing instruction data, providing empirical guidance for selecting data generation and organization strategies according to the target task.

Critical strengths

1.The research question has practical relevance. The paper directly addresses resource allocation in training with synthetic code: whether a limited token budget should be allocated to code rewriting, editing context, or a broader set of sources. This can inform practical dataset construction.

2.The study offers a useful incremental perspective. Building on existing research on code rewriting and commit-based edit supervision, the paper compares code representations, editing context, and instruction-data proportions under a common training budget. Its novelty lies primarily in examining the interactions among these data design factors and assessing the applicability of the findings across models and tasks, constituting a practically relevant empirical contribution.

3.The experimental comparisons are systematic. For each model, the training token budget and hyperparameters are held fixed while comparing raw code, cleaned code, notebooks, blog posts, and multiple editing-context recipes. This improves comparability across strategies.

4.Model and task coverage is reasonably broad. The main text covers six models of different sizes from two model families and evaluates both code editing and code generation. This enables comparisons across models and captures differences in task-specific preferences.

5.Some conditional findings are supported by direct experimental evidence. Table 2 shows that, under the strong condition, the mean editing scores across the six models are approximately 0.569, 0.571, 0.568, and 0.566 for Raw, Clean, Notebook, and Blog, respectively, while the commit-message group performs better on average. This suggests that the value of single-file rewriting depends substantially on editing instruction data, and that instruction-data proportions and task requirements should be considered when selecting synthetic data recipes.

Critical weaknesses

1.The correspondence between core strategy names and reported results is inconsistent. Section 5 attributes the Qwen3-4B score of 0.657 to Old+TerseCM, whereas Table 2 assigns this result to OldFormatCM and reports 0.556 for OldTerseCM. Moreover, Section 4.1 does not define OldFormatCM. The text also ranks the human-message strategy third, while the table ranks it second. These inconsistencies prevent an unambiguous mapping from the best-performing recipe to its generation method.

2.Matching the token budget does not adequately isolate the effects of data representation. Section 4.1 claims that “source content is fixed,” but the Limitations section explicitly acknowledges that “the kept source files are not identical across strategies.” Different filtering rules change the retained sources and length distributions, so performance differences may reflect both representation and sample selection. Recommendation: restrict the conclusions to comparisons of complete recipes and add controlled comparisons on a common subset of source files for the key strategies to isolate the contribution of data representation.

3.The conditioning information used to generate Plan and Review is inconsistent with their intended research objectives. Section 3.1 describes plans as specifying changes from old to new code and reviews as identifying problems in the old code and recommending fixes. However, Section 4.1 states that plans are generated from “xnew alone” and that reviews are also based solely on the new code. A post hoc description of the target program does not guarantee editing guidance aligned with the actual change. These weaker results are therefore insufficient to establish that appropriately generated edit plans or code reviews offer no benefit.

4.The rankings of the main strategies lack supporting uncertainty quantification. Figure 1 specifies “one run per recipe,” and Section 4.4 states that EDIT-Bench contains 108 problems, with one sample generated per problem. In Table 2, the leading synthetic-message recipe has an average advantage of approximately 2.3 percentage points over human messages across models. Some differences correspond to only a few problems, but the main text nevertheless recommends a default strategy based on these results.

5.Structural fidelity metrics are insufficient to support claims of behavioral preservation and executability. Section 3.2 attributes the benefits of Clean to “the added explanation,” ruling out changes to the underlying program as a source of the effect. However, Table 1 primarily examines the preservation of definitions, imports, and source code. These metrics do not establish semantic equivalence. Similarly, successful parsing alone does not justify the claim that the transformation “preserves executability and content.”

6.The composition and training budgets of the instruction-only baselines are insufficiently specified. Section 4.2 states that all recipes allocate 10% of training tokens to general QA instructions, with additional editing instructions in the strong setting. However, Table 2 describes the strong instruction-only baseline as using editing instructions alone, while Section 5 states that the comparisons use “the same instruction data.” The main text also does not specify the training token counts or the number of passes over the data for these baselines, making it difficult to assess the incremental contribution of synthetic code.

Questions for authors

1.Section 4.4 indicates that evaluation scores are sensitive to the output protocol, while Figure 3 illustrates how some recipes enable the repair of code defects not explicitly mentioned in the instruction. Could the authors use failure-mode analysis or targeted controlled comparisons to distinguish the contributions of format compliance, full-file output, and actual edit correctness to the main gains? How would this decomposition affect the interpretation that “context organization matters more than code rewriting”?

2.Section 4.1 claims that “source content is fixed,” but the Limitations section acknowledges that different strategies retain different source files. Do the authors have comparisons on a common subset of source files, or other evidence, showing that the main gains come from representation rather than filtering-induced changes in source composition and length distributions?

3.Section 4.1 describes Plan and Review as generated from the new code, including the specification “xnew alone.” How do the authors verify that these texts accurately capture the actual editing intent and defects in the old code? What conclusions about the value of edit plans and code reviews are supported by the current experiments?

4.Figure 1 specifies “one run per recipe,” and EDIT-Bench uses only one sample per problem. Could the authors provide paired performance differences based on the per-problem results, together with confidence intervals, for comparisons among the leading synthetic-message strategy, the human-message strategy, and Raw? This would help assess the robustness of the main advantages and distinguish improvements from sampling variability.

5.What exactly does “the same instruction data” in Section 5 refer to? The strong training recipes contain both QA and editing instructions, whereas Table 2 describes the corresponding baseline as using editing instructions alone. Please clarify the data composition, decontamination rules, training token counts, and number of passes over the data for each baseline, and explain how these differences affect the assessment of the incremental contribution of synthetic code.

1 comment
AuthorsOpenReview ↗

Response to Reviewer EyuJ

Thank you for the careful review; we adopted each recommendation.

1. Naming (W1). Fixed; Section 4.1 defines each strategy. Old+TerseCM (formatted synthetic message) is Table 2's OldFormatCM, the 0.657; Old+VerboseCM (intent-only) is its OldTerseCM, the 0.556.

2. Representation or commit selection (W2, Q2). As recommended, the main tables are now described as comparisons of complete recipes, and we ran the common-subset control: eight strategies rebuilt from one shared set of commits (strong instruction set, two budgets; Qwen3-8B, Gemma4-12B and a new family, OLMo-2 7B; 48 checkpoints). Mean score:

StrategyEDIT-BenchCanItEditHumanEvalFixInside a 14k-character fileSame, tests only
Raw.532.321.550.279.301
Clean.531.322.542.274.291
Blog.498.309.568.276.308
Old+CM.524.319.576.366.375
Old+VerboseCM.530.324.562.301.311
Plan+Old.528.309.558.291.305
Plan+CM+Old.524.322.569.302.324
Plan.526.322.564.295.318
Run s.d..010.004.009.009.009

With sources fixed, the eight strategies are level on the three benchmarks (largest difference from Raw: Blog -.034 [-.079, +.011]). The representation effect is in the last two columns: the same CanItEdit and HumanEvalFix problems and tests, each program inside 14k characters of unrelated functions (Qwen3-8B, Gemma4-12B). Column four requires those functions to be returned; column five uses the original tests alone. Old+CM beats Raw by +.086 [+.055, +.118] and +.074 [+.041, +.106], is first in all four runs and ahead of every other strategy.

3. Format, full-file output, correctness (Q1). Failure-mode analysis: every EDIT-Bench return was classified (six paper models, weak setting):

ReturnsUsable whole-file editFragment onlyFile unchangedOtherPass given a usable edit
Raw45%28%10%16%.36
Old+CM57%17%12%14%.38

Full-file output accounts for two thirds of Old+CM's gain (+.043 of +.064); a usable edit passes equally often. The difference sits where more than 6.3k characters surround the edit (Reviewer 1, topic 2). With the strong instruction set every strategy returns usable whole-file edits (86%), and they converge. Controlled comparison: the 14k-file test (topic 2) changes only the file around the program; on the original tests alone, which a correct fragment passes, Old+CM is still ahead (+.074 [+.041, +.106]), so its edits are more often correct, not only more often complete. So "context organization matters more than code rewriting" holds in a sharper form: a real commit packed as an edit teaches correct editing inside a large file.

4. Uncertainty and the default (W4, Q4). Tables 2 and 3 now report means and std over three evaluation runs; intervals include training-run variance, measured from 54 same-strategy run pairs. Paired per-problem differences, EDIT-Bench, six paper models (models ahead in brackets):

DifferenceWeak settingStrong setting
Old+CM - Raw+.064 [+.022, +.106] (6/6)+.019 [-.015, +.053] (4/6)
Old+TerseCM - Old+CM-.055 [-.093, -.017] (1/6)-.004 [-.041, +.033] (2/6)
Old+TerseCM - Raw+.009 [-.030, +.048] (2/6)+.015 [-.021, +.052] (4/6)

With run variance included, the synthetic message has no advantage over the human one: level in the strong setting, behind in the weak setting, partly through commit selection (the synthetic-message strategies dropped commits of two lines or fewer, and keeping them is worth +.049 [+.007, +.090] on small fixes). On shared commits, Old+CM and the synthetic-message strategy tested there (Old+VerboseCM) are level on the three benchmarks, and the human message is ahead inside the 14k file (+.065 [+.034, +.096]). We therefore recommend the simpler recipe: real commits packed as edits, small commits kept.

5. Baselines and instruction data (W6, Q5). All strategies share the QA instruction data (10% of tokens); the strong setting adds the edit instruction set (about 7%); "the same instruction data" refers to these two sets. Against the no-code baseline trained on the same QA data (~102M tokens; Table 2's weak "instruction tuned" row), Old+CM is +.047 [+.006, +.088] (6/6 models). The decontamination filter removes benchmark test files by exact match. The models see the data only once during training.

6. Plans and reviews (W3, Q3). We scope the claim to post-hoc descriptions of the target file. They do not improve editing over raw code, in either setting. We no longer conclude that plans or reviews conditioned on the change would not help.

7. Fidelity (W5). "Preserves executability" is replaced by what we measured: 61% of cleaned files match the source's AST once comments and docstrings are removed; the rest contain small edits (median six lines), so the Clean effect is no longer attributed to the explanation alone.

We hope this resolves your concerns and that you will consider raising your score.

Reviewer_pV1UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how different synthetic code formats affect continued pretraining. The authors generate synthetic data from Python commits using gpt-oss-120b, including cleaned code, notebooks, blog posts, planning documents, and commit messages. They compare different training strategies under a fixed token budget across six models from the Qwen3 and Gemma4 families, evaluating performance on EDIT-Bench and LiveCodeBench. They conclude that synthetic code transformations can outperform raw code and that commit-message-based formats are particularly effective for code editing.

Critical strengths

The authors conduct a wide range of experiments, evaluating multiple strategies across six models, providing useful empirical comparisons.

Critical weaknesses
  1. The experiments are not conducted under sufficient control. The strategies differ not only in data format but also in the amount of original code, the number of unique source documents, and the proportion of code versus natural language. The filtering procedures also produce different source distributions. Therefore, matching the token budget alone does not isolate the effects of the proposed transformations.

  2. The main findings may largely reflect alignment between training and evaluation formats. In Appendix F, the authors acknowledge that the target format has a larger impact than other data construction choices. Table 10 shows that changing the target format can affect EDIT-Bench accuracy by up to 6.5 percentage points. It is therefore unclear whether the proposed strategies improve general code-editing capabilities, or mainly teach models to produce outputs in the format expected by the benchmark.

  3. The empirical evidence is insufficient to establish reliable strategy rankings. Many differences are small. For example, on Qwen3-8B with strong instructions (Table 2), OLD+FormatCM achieves 0.620 on EDIT-Bench, compared with 0.593 for CLEAN and 0.583 for the instruction-tuned baseline. These differences correspond to only three and four additional solved problems, respectively, on a benchmark containing 108 problems. Although the appendix reports statistical tests for selected comparisons, the paper lacks systematic significance analysis and training-seed variation to support its overall rankings.

  4. The claimed value of multiple synthetic views is not sufficiently supported. In Table 6, DISJOINT and SHARED differ by only one solved problem on Qwen3-8B and perform identically on Gemma4-12B. Nevertheless, the authors claim that a second view retains 74-77% of the value of a new commit, based on small differences in held-out bits-per-byte. It is unclear whether this ratio meaningfully measures the value of synthetic data for downstream tasks.

Questions for authors
  1. How do the proposed strategies compare with training on the original, unmodified commit data? Is the benefit due to synthetic generation or simply the inclusion of edit-related context?

  2. How can the authors establish that the gains come from the synthetic transformations rather than differences in source coverage, code exposure, or output format?

  3. Do the improvements transfer to code-editing benchmarks with different output formats or task distributions?

  4. How is the correctness of teacher-generated code and annotations evaluated? Could differences in generation quality explain the strategy rankings?

1 comment
AuthorsOpenReview ↗

Response to Reviewer pV1U

Thank you for your comments on the usefulness of the empirical comparisons.

1. What differs between strategies (W1, Q2). As our limitations section states, strategies differ in the source files they keep and in code share; Section 4.1 now reports both. We report means over models in the weak setting: EDIT-Bench, the paper's six models (Qwen3-4B/8B/14B, Gemma4-12B/26B/31B) plus a new family, OLMo-2 7B; LiveCodeBench, the paper's six models; the two added benchmarks, Qwen3-8B, Gemma4-12B and OLMo-2 7B.

StrategyCode tokensEDIT-BenchCanItEditHumanEvalFixLiveCodeBench
Raw100%.223.292.510.156
Clean100%.231.279.534.151
Notebook72%.229.277.497.146
Blog24%.259.269.523.154
Old+CM99%.284.300.536.153
Old+TerseCM98%.230.277.468.150
Old+VerboseCM97%.244.297.455.149
Plan+Old55%.228.270.512.109
Plan+CM+Old55%.234.276.506.152
Old+Review66%.244.283.509.148
Plan47%.238.291.493.131
Run s.d..009.006.013.003

Code share does not explain the EDIT-Bench result. Blog has the least code (24%) yet ranks second, and code share and score are essentially uncorrelated (-.06). Code share is also part of a strategy: a fixed budget asks whether a plan is worth the code it displaces. Source coverage: with commits fixed (Reviewer 2, topic 2) the eight strategies are level on the three benchmarks; what survives is editing inside a large file (topic 2 below).

2. Format alignment, unmodified commits, transfer (W2, Q1, Q3). The unmodified-commit baseline is in the paper: Old+CM is the untouched CommitPack commit, Raw its post-commit file. No generated strategy improves on Old+CM, which has the highest mean on all three editing benchmarks above. Its gain over Raw on EDIT-Bench (+.061 [+.023, +.100], 7 of 7 models) grows with the amount of unchanged code around the edit (problems requiring an edit):

Unchanged code around the editRawOld+CMAhead in
None or no region (31 problems).186.2104 of 7
Up to 2.2k characters (15).190.2034 of 7
2.2k to 6.3k (19).103.2217 of 7
More than 6.3k (15).086.2067 of 7

CanItEdit and HumanEvalFix programs are short (median 1.0k and 0.6k characters), hence the small margin there. We placed each of those programs inside 14k characters of unrelated functions, problems and tests unchanged (Qwen3-8B, Gemma4-12B). Raw / Old+CM:

SettingProgram aloneInside the 14k fileOrig tests only
Weak setting.484 / .512.000 / .106.232 / .333
Strong setting, shared commits.547 / .551.279 / .366.301 / .375

“Inside the 14k file” requires the surrounding functions to be returned. “Original tests only” scores just the original tests, which a correct fragment also passes, so it does not depend on output format. In the strong setting every strategy is trained on the same 18k whole-file edit instructions, yet the gap remains. On shared commits Old+CM leads by +.086 [+.055, +.118] and +.074 [+.041, +.106], and it ranks first of eight strategies in all four runs (two models × two budgets). Format is part of the EDIT-Bench gain, as Appendix F suggests: two thirds of it comes from returning the whole file (Reviewer 2, topic 3). The gap inside the 14k file, however, persists when format is taken out of the scoring. It reflects a capability, making a correct edit inside a large file, which raw code does not teach and the Old+CM packing does.

3. Reliability of rankings (W3). Tables 2 and 3 now report means and std over three evaluation runs, with paired contrasts and Holm-corrected tests. We measured training-run variance from 54 pairs of separately trained runs of the same strategy and model; our intervals include it (first table, last row). While we expected one strategy to come out ahead, we found no strategy uniformly beats the others. For this reason, the paper no longer ranks within groups.

4. Generation quality and teacher (Q4). Of the cleaned files, 61% have exactly the source’s AST once comments and docstrings are removed; the rest contain small edits (median six lines). A second teacher (Gemma4-31B-it; 42 checkpoints, strong setting) gives the same result. Its best strategy is again commit-message packing (Old+VerboseCM), and none of its seven strategies is significantly above Raw or the human-message Old+CM. The strongest strategy, Old+CM, contains no generated text, so generation quality cannot explain the main result.

5. Second-view claim (W4). We agree that the 74–77% figure is a language-modeling measure and now state it as one. Bits-per-byte orders DISJOINT, SHARED and REPEAT identically in both families. EDIT-Bench shows the same direction but cannot separate SHARED from DISJOINT. We report bits-per-byte as a complement: it reads the mid-trained model directly, while benchmarks need one that follows instructions.

We hope this resolves your concerns and that you will consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We appreciate the time and effort that the reviewers dedicated to providing feedback.

Naming. Our submission file had a rendering problem, pointed out by Reviewer 2, that made strategy names in the text disagree with the tables. The manuscript is fixed, with changes highlighted. Names used below:

  • Old+CM: old file + human commit message → new file (nothing generated)
  • Old+TerseCM: formatted synthetic message (OldFormatCM in some tables)
  • Old+VerboseCM: intent-only synthetic message (OldTerseCM in some tables)

New experiments. During the discussion period we trained 176 additional checkpoints and added a third model family (OLMo-2 7B) and two editing benchmarks (CanItEdit, HumanEvalFix). We answer each reviewer individually and refer to shared experiments by reviewer and topic. The revision supports three findings:

  • When the strong instruction set is not present, packing the commit with its message (Old+CM) beats training on just the code in all seven models on EDIT-Bench (+.061 [+.023, +.100]) and is the only strategy significantly above its no-code instruction baseline (+.047 [+.006, +.088]). It also has the highest mean of the eleven strategies on the two added editing benchmarks, there within run variance (Reviewer 1, topics 1 and 2).
  • What the packing teaches is more than an output format. On EDIT-Bench, two thirds of its gain comes from returning the whole file. To separate format from correctness, we placed the same benchmark problems inside a 14k-character file. There Old+CM is ahead of raw code in both instruction settings. On a shared set of commits it ranks first of eight strategies in every run. It stays ahead when only the original tests are scored, which a correct fragment also passes, so inside a large file its edits are more often correct, not just more often complete. On short programs the strategies converge (Reviewer 1, topic 2; Reviewer 2, topics 2 and 3).
  • The EDIT-Bench score of one checkpoint varies by about .02 (s.d.) between training runs of the same strategy, and the means in our tables by about .01; we measured this and report every result against it (Reviewer 1, topic 3).

The added controls changed one recommendation, and we think the paper is more useful for it. With training-run variance included, the synthetic commit message has no advantage over the human one, so we recommend real commits packed as edits, with small commits kept (Reviewer 2, topic 4). We also no longer rank strategies within groups (Reviewer 1, topic 3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.