- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a novel rewriting scaling law that integrates synthetic data mixture ratios into the standard Chinchilla-style framework by modeling rewrite-dependent irreducible loss and data repetition penalties. This formulation allows for the joint optimization of model size, training tokens, and compute-adjusted rewrite fractions, transforming synthetic data allocation from a heuristic task into a predictable, loss-minimizing objective.
Critical strengths
- The authors extend scaling by explicitly modeling generation costs as a compute constraint. This provides a rigorous, actionable framework to optimize the trade-off between model scaling and synthetic pre-training overhead. This may shift synthetic data allocation from heuristics to predictable optimization for training stage.
Critical weaknesses
- The paper frames the proposed rewriting framework as a "Scaling Law," but the law’s predictive power appears heavily reliant on empirical curve-fitting constants () that are specific to a given rewriter-dataset pair. Since the irreducible loss term is a function of synthetic data quality, which is a highly stochastic and difficult-to-control variable, the current framework lacks a theoretical mechanism to account for variations in rewriter capability or output distribution. Consequently, the framework serves more as an a posteriori descriptive tool for a specific experiment rather than a predictive a priori scaling law. The paper does not adequately address whether the functional form of remains stable under significant distribution shifts or when using rewriters that introduce novel, higher-order reasoning capabilities not captured by the current four FinePhrase styles.
Questions for authors
- The rewriting scaling law is empirically fitted using a specific rewriter (SmolLM2-1.7B in the compute-limited setting). Since the irreducible loss term relies on parameters that are contingent on the rewriter's output quality, how robust is this scaling law to changes in the rewriter's capability? Specifically, if the rewriter is replaced with a significantly more powerful model (e.g., a frontier-scale LLM), would the current scaling law merely require a re-fitting of these parameters, or would the functional form of likely need to be revised to account for the qualitative shift in synthetic data density?
2 comments
As the discussion period closes on Nov 18, we wanted to check whether our response addresses your question about refitting versus revising the functional form. We would be happy to clarify anything further, and we would appreciate it if you would consider updating your assessment if your concerns have been resolved. Thank you!
Thank you for the review! We address the predictivity concern and your question below.
On fitted constants and predictivity. We agree that the fitted constants are setup-specific. This is common to scaling laws: the Chinchilla coefficients [1] are likewise specific to their own corpus and architecture. What transfers is the functional form of the scaling law, calibrated at small scale and validated on held-out larger scales. Our own setup illustrates this in that our compute-optimal allocation is roughly 4 tokens-per-parameter even on natural data alone, while Chinchilla's is roughly 20.
A main contribution of our paper is that the rewrite-dependent component of loss is modelable in a scale-invariant way, and we test this directly: 82 IsoFLOP-optimal observations across a 3,333× compute range collapse onto a single curve with RMSE 0.0044 BPB (Appendix N.1.1, Figure 17). We then validate its predictive consequences out of sample: fitting only through 3×1020 FLOPs predicts the held-out 1022-FLOP hero run (33× extrapolation) prior to training within 1.1 in loss, and all four 1022 runs within 1.4 (Figure 3, Table 8); data-limited fits through W=18 predict held-out W=21 and 24 within 0.74 at the best observed R (Table 6).
On robustness to the rewriter. In every setting we have tested, changing the rewriter or the seed corpus required refitting constants, not revising the functional form. The same form fits an external rewriter trained at more than 1023 FLOPs and four of our own rewriters from 550M–3B parameters, and DCLM as a second seed corpus (Appendix N.7). The derivation of E(R) (Appendices L and N.1.2) uses nothing specific to rewriting: it describes how the loss floor changes when training on a convex combination of two text distributions, with constants λ and ρ set by how much of the evaluation distribution each component distribution covers. A different rewriter or corpus changes those distributions, and hence the constants, but not the functional form. The modeling assumption that rewrites affect only this loss floor is supported by the scale-invariant collapse in Figure 17. Finally, in the scenario of a much stronger rewriter, the FinePhrase study underlying our rewrites [2] sweeps generator capacity and finds downstream quality saturates near 1B parameters, with "negligible gains" beyond it for most rephrasing prompts, suggesting limited sensitivity to further rewriter scaling.
[1] https://arxiv.org/abs/2203.15556
[2] https://arxiv.org/abs/2604.13977