acceptodds
Under review as a conference paper at ICLR 2027

Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

Abstract

Synthetic rewrites, where a language model rewrites a natural document into a target style, are now a standard ingredient of pretraining corpora. Yet critical aspects of their use remain poorly understood, such as how much of the training data should be rewritten, given that generating rewrites costs compute that could instead train the model, and whether a model trained on rewrites can outperform the rewriter. We study these two questions by proposing a rewriting scaling law that predicts loss from model size, training tokens, the fraction of training tokens that are rewrites, and the number of epochs over the natural data, charging the compute spent generating rewrites. We fit it in two settings. In the compute-limited setting, natural data is plentiful and rewrites come from a third-party rewriter trained with more compute than our largest run. We find that the estimated optimal rewrite fraction increases with compute, from under at FLOPs to at FLOPs. Our scaling forecast extrapolates in FLOPs to train a 22B-parameter model, whose final loss is within of our prediction. This model achieves a better BPB than its rewriter on several web domains at one-tenth the rewriter's training compute. In the data-limited setting, natural data is a fixed seed set and the rewriter is a model of the same size as the target, pretrained on that seed set. Again the optimal rewrite fraction increases with the compute budget. We also demonstrate that a model trained on rewrites of its own pretraining data, generated by a rewriter of the same size, beats the best run on natural data alone at every model size up to 3B. Our results show that synthetic rewriting can be studied through the scaling law lens: that optimal methodological choices are predictable and scaling does not necessarily break down beyond the rewriter's capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling laws, synthetic rewriting, pretraining, LLMs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QBNXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a novel rewriting scaling law that integrates synthetic data mixture ratios into the standard Chinchilla-style framework by modeling rewrite-dependent irreducible loss and data repetition penalties. This formulation allows for the joint optimization of model size, training tokens, and compute-adjusted rewrite fractions, transforming synthetic data allocation from a heuristic task into a predictable, loss-minimizing objective.

Critical strengths
  • The authors extend scaling by explicitly modeling generation costs as a compute constraint. This provides a rigorous, actionable framework to optimize the trade-off between model scaling and synthetic pre-training overhead. This may shift synthetic data allocation from heuristics to predictable optimization for training stage.
Critical weaknesses
  • The paper frames the proposed rewriting framework as a "Scaling Law," but the law’s predictive power appears heavily reliant on empirical curve-fitting constants (λn,λs,ρn,ρs\lambda_n, \lambda_s, \rho_n, \rho_s) that are specific to a given rewriter-dataset pair. Since the irreducible loss term E(R)E(R) is a function of synthetic data quality, which is a highly stochastic and difficult-to-control variable, the current framework lacks a theoretical mechanism to account for variations in rewriter capability or output distribution. Consequently, the framework serves more as an a posteriori descriptive tool for a specific experiment rather than a predictive a priori scaling law. The paper does not adequately address whether the functional form of E(R)E(R) remains stable under significant distribution shifts or when using rewriters that introduce novel, higher-order reasoning capabilities not captured by the current four FinePhrase styles.
Questions for authors
  • The rewriting scaling law is empirically fitted using a specific rewriter (SmolLM2-1.7B in the compute-limited setting). Since the irreducible loss term E(R)E(R) relies on parameters that are contingent on the rewriter's output quality, how robust is this scaling law to changes in the rewriter's capability? Specifically, if the rewriter is replaced with a significantly more powerful model (e.g., a frontier-scale LLM), would the current scaling law merely require a re-fitting of these parameters, or would the functional form of E(R)E(R) likely need to be revised to account for the qualitative shift in synthetic data density?
2 comments

Thank you for the review! We address the predictivity concern and your question below.

On fitted constants and predictivity. We agree that the fitted constants are setup-specific. This is common to scaling laws: the Chinchilla coefficients [1] are likewise specific to their own corpus and architecture. What transfers is the functional form of the scaling law, calibrated at small scale and validated on held-out larger scales. Our own setup illustrates this in that our compute-optimal allocation is roughly 4 tokens-per-parameter even on natural data alone, while Chinchilla's is roughly 20.

A main contribution of our paper is that the rewrite-dependent component of loss is modelable in a scale-invariant way, and we test this directly: 82 IsoFLOP-optimal observations across a 3,333×3{,}333\times compute range collapse onto a single curve with RMSE 0.00440.0044 BPB (Appendix N.1.1, Figure 17). We then validate its predictive consequences out of sample: fitting only through 3×10203 \times 10^{20} FLOPs predicts the held-out 102210^{22}-FLOP hero run (33×33\times extrapolation) prior to training within 1.11.1\\% in loss, and all four 102210^{22} runs within 1.41.4\\% (Figure 3, Table 8); data-limited fits through W=18W = 18 predict held-out W=21W = 21 and 2424 within 0.740.74\\% at the best observed RR (Table 6).

On robustness to the rewriter. In every setting we have tested, changing the rewriter or the seed corpus required refitting constants, not revising the functional form. The same form fits an external rewriter trained at more than 102310^{23} FLOPs and four of our own rewriters from 550M–3B parameters, and DCLM as a second seed corpus (Appendix N.7). The derivation of E(R)E(R) (Appendices L and N.1.2) uses nothing specific to rewriting: it describes how the loss floor changes when training on a convex combination of two text distributions, with constants λ\lambda and ρ\rho set by how much of the evaluation distribution each component distribution covers. A different rewriter or corpus changes those distributions, and hence the constants, but not the functional form. The modeling assumption that rewrites affect only this loss floor is supported by the scale-invariant collapse in Figure 17. Finally, in the scenario of a much stronger rewriter, the FinePhrase study underlying our rewrites [2] sweeps generator capacity and finds downstream quality saturates near 1B parameters, with "negligible gains" beyond it for most rephrasing prompts, suggesting limited sensitivity to further rewriter scaling.

[1] https://arxiv.org/abs/2203.15556

[2] https://arxiv.org/abs/2604.13977

As the discussion period closes on Nov 18, we wanted to check whether our response addresses your question about refitting versus revising the functional form. We would be happy to clarify anything further, and we would appreciate it if you would consider updating your assessment if your concerns have been resolved. Thank you!

Reviewer_ZaHQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a scaling law that takes the rewrite fraction R — the share of training tokens that are LLM rewrites of natural documents — as an input and predicts held-out loss, then uses it to pick the loss-minimizing R at a fixed compute budget while charging the FLOPs spent generating the rewrites. The law is Chinchilla's A/N^α + B/D^β plus a rewrite-fraction-dependent irreducible term E(R), derived as an approximation to the KL between training and evaluation mixtures, plus a repetition penalty borrowed from Lovelace et al. for the epoching case.

For the two settings. In the compute-limited one, natural data is effectively unbounded and the rewriter is a third party (SmolLM2-1.7B-Instruct) trained with more compute than any target model. In the data-limited one, the seed set is a fixed 5B tokens of FineWeb-Edu and the rewriter is a same-size model the authors pretrain on that same seed set.

For the two claims: the law predicts the optimal rewrite fraction and its loss (RQ1), and including rewrites still helps when the target is meant to exceed the rewriter (RQ2). For RQ1 they fit at budgets up to 9×10^19, extrapolate 33× to 10^22, and report the 10^22 run at the predicted optimum R = 0.32 landing within 1.1% of predicted loss and lowest among the four 10^22 runs they train. For RQ2 they report that the 22B/76B hero run beats SmolLM2-1.7B-Instruct on 10 of 16 Paloma sources at roughly a tenth of its training compute, and that in the data-limited setting mixtures beat natural-only epoching at every size and budget. Roughly 671 runs total, with a DCLM corpus control in the appendix.

Critical strengths

The load-bearing assumption of the whole framework is well tested, making the 33× extrapolation credible. The main point rests on E(R) being scale-invariant — the shape of loss-versus-rewrite-fraction has to be the same at every budget, so that all the predicted scale dependence of R* comes from generation cost amortizing against a growing training cost. Appendix N.1.1 checks this directly: after subtracting each budget's R = 0 value, 82 IsoFLOP-optimal observations from nine budgets collapse onto a single curve with RMSE 0.0044 BPB. Without that collapse the extrapolation would be an article of faith; with it, the forecast at 10^22 is a real prediction, and the held-out runs at that budget are realistic.

The scale and the candor together is comprehensive. 671 runs, 11 rewrite fractions, genuinely held-out validation runs, per-source and per-task breakdowns, and — unusually — explicit reporting of how much the extrapolated quantities depend on fit specification: the token multiplier at 10^22 is 5.01× under one specification and 12.17× with a [6.39, 45.01] bootstrap interval under another, and the divergence point moves from 3.6×10^26 to 1.8×10^24 FLOPs. Most scaling-law papers report one fit and stop. They also ran a corpus control and reported the result even though it cuts against them, which brings me to the weaknesses.

Critical weaknesses

The RQ2 result rests on comparing against the instruction-tuned checkpoint, and the base model of the same rewriter beats the hero run on BPB metrics in the paper. Table 11 is unambiguous: SmolLM2-1.7B base reaches 0.943 Paloma macro BPB against the hero run's 0.983, and also wins on QA BPB (0.667 vs 0.700), Math (0.707 vs 0.842) and Code (0.399 vs 0.587). The Instruct checkpoint used as the reference sits at 0.972, worse than its own base model on raw web text — which is the ordinary and well-documented cost of alignment, not evidence about what the rewriter knows. So "reaches a lower Paloma BPB than its rewriter on 10 of 16 sources" is substantially a statement about instruction tuning degrading language modeling. Several of those ten wins are also very narrow — TwitterAAE is 2.453 against 2.455, mC4 0.825 against 0.828, M2D2 S2ORC 0.760 against 0.764 — margins well under the paper's own ~1% forecast error, with no seed variance reported anywhere. The one place the hero run genuinely clears both checkpoints is QA accuracy, 0.622 against the base model's 0.617, which is half a point. I think the honest version of RQ2 is "a model trained on 32% rewrites comes within 4% of a rewriter trained on 11× the compute and matches it on QA accuracy," which is still interesting, rather than "scaling does not break down beyond the rewriter's capabilities." To the authors' credit the base-model row is right there in Table 11; it just doesn't reach the abstract.

The headline rewrite fractions appear to be a property of FineWeb-Edu, and the paper's own corpus control shows the effect largely disappearing on a better seed set. Appendix N.7 swaps in DCLM and finds a fitted floor-optimal fraction of 0.078 with a gain over R = 0 of 0.00577 BPB, against FineWeb-Edu's 0.328 and 0.0825 BPB — roughly 4x lower a fraction and 14x smaller a benefit, on a corpus the same appendix shows beating FineWeb-Edu in all 77 matched width–depth cells. And DCLM is itself a long way from a frontier pretraining mixture. That reframes the paper's practical message considerably: the recommendation isn't "rewrite about a third of your tokens at 10^22 FLOPs," it's "rewrite about a third of your tokens if your corpus is a filtered educational subset, and perhaps a twentieth if it isn't." Two sentences in the Limitations section acknowledge this, but the abstract, the introduction and Figure 2 all report the FineWeb-Edu numbers without it. Given that the corpus-dependence is arguably the most decision-relevant finding in the paper, it should be a headline rather than an appendix.

Questions for authors

The one that would most change my reading: please put the SmolLM2-1.7B base comparison in the main text alongside Instruct, and restate the 10-of-16 count against the base model. How many sources does the hero run win on then, and what does the macro average look like? If the answer is that the base model is still ahead, I'd ask you to reframe RQ2 rather than drop it, since the QA-accuracy result and the 11× compute gap are both worth reporting on their own terms.

3 comments

Thank you for the thorough and constructive review! We agree with both of your points and have revised the paper accordingly.

On the base model comparison. We have updated Table 1 in the main text to include the base model results. We restate the comparison: against SmolLM2-1.7B base, the hero run wins on 3 of 16 sources (M2D2 Wikipedia, WikiText-103, Penn Treebank), and the macro average is 0.9830.983 vs. 0.9430.943, within 4.24.2\\% at 1/111/11th the training compute, while clearing both the base and Instruct checkpoints on QA accuracy (0.6220.622 vs. 0.6170.617 and 0.6060.606; Table 11). We agree these margins are comparable to the narrow Instruct wins you flagged, so the revision emphasizes the overall pattern and notes the lack of multi-seed variance at this scale.

On RQ2. We kept the question, and the answer now names both references: the Instruct checkpoint generated every rewrite token, so that comparison tests whether a student can beat the generator of its synthetic data; the base checkpoint is, as you argue, the right reference for underlying pretraining capability (within 4.24.2\\% BPB and slightly exceeding QA accuracy at 1/111/11th the compute). In the data-limited setting, mixtures consistently outperform both the matched rewriter and the best natural-only runs across sizes (Figure 5). The closing claim now specifies both references.

On corpus dependence. We agree that the optimal rewrite fraction depends strongly on the seed corpus. The abstract now conditions the headline fractions on the seed corpus, the introduction and Figure 2 reference the DCLM control (Appendix N.7), and its key result (R^∞∗=0.078\hat{R}^*_\infty = 0.078 vs. 0.3280.328) now appears in the main text.

As the discussion period closes on Nov 18, we wanted to check whether our response and the updated manuscript address your concerns. In particular, Table 1 now includes the base model, and the abstract and statements in the introduction are now conditioned on the seed corpus. We would be happy to clarify anything further. Thank you again for your detailed review!

Reviewer_ZaHQOpenReview ↗

Response to rebuttal

Thanks for your clarification. I decide to maintain my score.

Reviewer_jmX3OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a rewriting scaling law, which, building on the classic Chinchilla scaling law, explores the scaling law of the proportion of rewritten data. The authors first fix the FLOPs budget and include the model inference budget incurred by rewriting. Then, under two settings: (1) natural data are abundant and the rewriting model is stronger than the training model, and (2) natural data are limited and the rewriting model is of the same scale as the training model. They conduct extensive scaling-law fitting experiments and successfully perform extrapolation.

Critical strengths

The most important strength of this paper is that it fills the gap in existing scaling law research regarding the mixing ratio of rewritten data. Rewriting is an important means of pretraining data augmentation, yet it currently lacks systematic study. Through extensive experiments, the authors fit an extrapolatable scaling law for the proportion of rewritten data, which is an important contribution to LLM pretraining.

Critical weaknesses

When conducting extrapolation experiments, for example, with a 3B model and W=24, the formula gives R=0.47, while the actually observed optimum is R=0.71, and this ratio gap is relatively large.

Questions for authors

Regarding the phenomenon in the extrapolation experiments where the ratio prediction gap is relatively large for the 3B model, whereas predictions are relatively accurate for smaller models, is there a risk that the fitted formula may be inaccurate when extrapolated to larger models? Alternatively, when model size reaches 3B or thereabouts, could the model's capability performance undergo a significant change (specifically with respect to the rewritten data ratio), and would the fitted formula need to be treated piecewise?

1 comment

Thank you for the positive assessment of our work and for bringing up these points!

On the prediction gap. The gap between the selected R^∗=0.47\hat{R}^* = 0.47 and the observed optimum R=0.71R = 0.71 is largely a discretization and flat-basin effect. We test R∈0,0.13,0.29,0.47,0.71,1R \in \\{0, 0.13, 0.29, 0.47, 0.71, 1\\}, so 0.470.47 and 0.710.71 are adjacent values in the sweep, and the observed optimum can only snap to one of them, while the true continuous optimum need not coincide with either. Moreover, the loss curve is quite flat across this region, so the gap barely affects performance: selecting R=0.47R = 0.47 instead of 0.710.71 costs 1.141.14\\% in Paloma BPB at W=24W = 24 and 0.50.5\\% at W=21W = 21 (Figure 4, top row).

On larger models and a piecewise formulation. This is a question we were interested in as well. The held-out loss predictions provide no evidence of a qualitative change around 3B: the same per-size fit predicts the 3B loss at the best observed RR within 0.740.74\\% at both held-out budgets, and within 0.270.27\\% mean error across all RR (Table 6). What the broad loss basin at high epoch budgets does make harder is identifying the precise location of the minimizer, which is where the gap you note appears. We note these held-out tests extrapolate in epoch budget at fixed size; our compute-limited experiments provide the complementary evidence across model sizes (to 22B), under a different data and rewriter regime (Table 8). We have added a sentence to the paper noting the limited identifiability of R∗R^* within the basin.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

We thank the reviewers for highlighting the importance of the problem we study (jmX3 and QBNX) and the extensiveness of our experiments (ZaHQ). Your constructive feedback has improved our draft significantly, and we have revised the PDF based on your comments, along with minor wording and typo fixes.

To summarize, the main changes to the manuscript were:

  1. Base model comparison in the main text (Reviewer ZaHQ): the SmolLM2-1.7B base column now appears alongside Instruct in Table 1, with the comparison against both stated in Section 3.2.1 and the closing claim. At 1/111/11th the training compute, our 102210^{22}-FLOP run is within 4.24.2\\% of the base model's macro BPB (0.9830.983 vs. 0.9430.943), while slightly exceeding both the base and Instruct models on QA accuracy.
  2. Corpus dependence highlighted more prominently (Reviewer ZaHQ): the abstract now conditions the headline rewrite fractions on the seed corpus, and the DCLM ablation (R^∞∗=0.078\hat{R}^*_\infty = 0.078 vs. 0.3280.328, Appendix N.7) is referenced from the introduction and Figure 2. It is also summarized in the main text.
  3. Clarifications in Section 3.3.1: a note that coarse grid resolution and a broad loss basin at high epoch budgets make R∗R^* less sharply identified even as loss forecasts remain accurate (Reviewer jmX3).

For Reviewer QBNX, whose questions concern predictivity and robustness to the rewriter, we address them in the individual response using existing held-out extrapolation results (Figure 3, Tables 6 and 8, Appendix N.1.1), the derivation of E(R)E(R), and prior evidence on rewriter scaling. Please see the individual responses for our replies to all other points. Thank you all again for your helpful comments!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.