acceptodds
Under review as a conference paper at ICLR 2027

Why Does Rewriting Pretraining Data Improve Generalization?

Abstract

Language models are increasingly pretrained on synthetic rewrites of human text, which can improve downstream generalization. This gain is puzzling, as rewriting models can benefit substantially larger learners even on capabilities they do not themselves possess. We present an explanation through a rewriting-pretraining gap. We show that regularities in natural language allow rewriting models to learn simple transformations that generalize to unseen inputs. Applying these transformations changes the prediction problems encountered during pretraining, increasing the computationally bounded information accessible to the learner and inducing new representations useful for downstream tasks. We formalize this mechanism using algorithmic information theory and validate its predictions in controlled experiments. Our account further predicts that rewriting need not preserve factual content. Consistent with this prediction, we show that fictional rewrites improve compositional generalization on a challenging two-hop knowledge task.

Primary area:
foundation or frontier models, including LLMs
Keywords:
synthetic data, pretraining, generalization, minimum description length

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_9pQuOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
3(+2)
Presentation
2(+1)
Soundness
3(+2)
Read review
Paper summary

The paper analyzes the phenomenon of generalization as a result of fine-tuning models with data rewritten by a smaller model. They propose three types of rewriting --reordering, deletions and fictional-- and investigate their impact on a model using task specific measures (accuracy) and an information theoretic measure (epiplexity). The authors investigate three specific research questions, that contribute information about the level of generalization achieved through the added training, and its impact on downstream tasks.

Critical strengths

Understanding the impact of synthetic data, particularly data generated by a smaller model, is a worthwhile endeavour, and can bring interesting insights into the capabilities and behaviour of LLMs.

Critical weaknesses

The paper is difficult to follow, it introduces concepts that are then not used, it overloads terminology and notations. The experimental set-up is not very clear, and as a result of that, the results presented are difficult to interpret. There are also working assumptions that are not self-evident, and require justification.

The paper makes a big claim, but the testing on very specific tasks and forms that fit the fact/relation type data that is rewritten do not justify the broad conclusion.

Working assumptions: lines 160-161 and 179-181: "We can therefore view these transformations as in-context recall coupled with restructuring..." -- Since the purpose of the study is insight into a model's behaviour, making such assumptions about the model is not reasonable.

Dropped concepts:

  • Section 3, MDL principle: it is first defined in terms of the Kolmogorov complexity, which is then not mentioned anymore, and not relevant for the following discussion. Starting the section from line 113 instead would omit nothing relevant for the rest of the paper.
  • the seed z is mentioned, but then is often dropped in a formula. It should be kept wherever relevant (e.g. left hand side of formula 10, formula 12)

Overloaded terms and notation:

  • "p" is used for multiple concepts -- probablity distribution, probablity of the data, models. Because of this, it is difficult to differentiate where p(x) or p*(x) refers to a probability, or to the result of applying a model to input data x.
  • "P" is used for the set of models/model class, and for prompts
  • keys, values, queries, "k", "v", "q" -- these are notations that are strongly associated with the attention mechanism, which is also mentioned in the paper, but are used differently in describing the rewriting process. It would be useful to have some alternate terms, e.g. "attributes" instead of "keys" maybe (and one such change would break the strong priming of attention mechanism concepts)
  • "A" denotes a training algorithm but also the domain of the values

Unclear experimental process:

  • the input that is rewritten is presented as tuples in the paper. Is that how it is presented as input for rewriting or is the result of rewriting actual text?
  • the description of the experimental set-up (lines 335-341) is not very clear. It makes it seem like rewritten models are trained on the rewritten D and D_ID, while the others are trained only on D (which includes D_ID). Is it possible that the difference in performance comes from the fact that the additional data for the model pretrained with rewrites reinforces the patterns in that particular data?

Why are the epiplexity values for the Extraction task so much greater than for the reversal and composition?

Questions for authors

The paper requires substantial rewrites to be clear, no specific question could address that. But any reply to the issues raised in the Critical weaknesses section would be appreciated, particularly if some of the issues signaled are misunderstandings.

6 comments

We thank the reviewer for the careful and constructive review, and for acknowledging the importance of the research problem. We address each raised weakness below.

W1. The paper is difficult to follow.
W4. Dropped concepts.
W5. Overloaded terms and notation.

We thank the reviewer for constructive feedback on the presentation side. We have made changes to improve the representation and clarity. In summary, they consist of the following:

  • W1: Improved the experimental set up at the beginning of section 6.
  • W4:
    • Shorten the MDL principle in Preliminaries, which is not necessary to explain our core idea.
    • Clarified the connection between the seed z and the document x_z in Secinon 5.
  • W5:
    • Fix p as the probabilistic and P as the model class. We denote p(x) as the likelihood of data x computed by the model p.
    • Revise our terms key-value pairs for the minimal model of rewriting to field-value pairs.
    • Define A as the domain of values and Adv as the algorithm.

The revisions in the updated manuscript are highlighted in blue.

W2. Big claim, specific tasks.

We would like to kindly remark to the reviewer that the knowledge tasks we consider are broadly used to study learning and generalization of language models in a controlled manner [1-4]. We have adopted these settings to reduce confounding effects from noise and variation in natural text [5], allowing us to examine rewriting more directly.

To provide evidence beyond these settings, we additionally report a C4 pretraining experiment using “high-quality” (HQ) rewrites following [6]. We train a GPT-2-small model for 15B tokens per condition, comparing original C4 against C4 with 30% of the data replaced by HQ rewrites. We estimate epiplexity during pretraining and evaluate downstream perplexity on The Pile, an out-of-domain dataset [6], without further fine-tuning. The results are shown below.

Pretraining dataPerplexity on The Pile (↓)MNLI accuracy after fine-tuning (↑)Estimated epiplexity (Mbits)
Original C456.4873.3415275.86
C4 with HQ rewrites (30%)47.8773.9715397.03

HQ replacement reduces Pile perplexity by 15.2% and increases test accuracy by 0.63 on MNLI, accompanied by an increase of 122 mega bits in estimated epiplexity. These results are consistent with the trend observed in our controlled experiments that also appears in natural-text pretraining. This suggests that our account may be relevant to rewriting beyond the controlled fact/relation settings examined in our main experiments.


[1] Allen-Zhu & Li, Physics of Language Models: Part 3.1, Knowledge Storage and Extraction, 2024.

[2] Ma et al., Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge, 2026.

[3] Lin et al., Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge, 2026.

[4] Wang et al., Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization, 2024.

[5] Allen-Zhu, Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers, 2025.

[6] Maini et al., Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling, 2024.

W3. Evidence for model assumptions?

We agree and have added a new empirical test of whether rewriting can be viewed as in-context recall coupled with restructuring. More precisely, we have formalized this view in the algorithm in Listing 2, which makes three predictions:

  • In-context recall The output value is copied from the source, from the field indicated in the instruction, so replacing the value in the source should be reflected in the output.
  • Associative value recall The value is retrieved through the field-value pairs given in the context [7].
  • Restructuring The output template is determined by the requested field and the style. We test all three in TinyLlama-1.1B-Chat, the rewriting model used in the paper.

Each source document describes a synthetic person. A request denotes one field to be indicated in output, and the style is either bio or question:

Source: Name: Isabelle Brody Ainsworth / Birth city: Akron, OH / ...
Request: Birth city
Style: bio or question
Output:
Bio: Isabelle Brody Ainsworth was born in Akron, OH.
or
Question: Where was Isabelle Brody Ainsworth born? Answer: Akron, OH.

1. In-context recall. We request each of 6 attributes for 1,000 people, and replace one value in the source at a time.

SourceExpected outputRate
OriginalRequested value99.1%
Requested value replacedNew value99.3%
Another value replacedRequested value preserved99.7%

This results shows the model follows the given prompt well, which can be shown as in-context recall.

2. Associative value recall. We rank the 704 attention heads by activation patching: we copy a head’s output from the clean run into a run with the requested value replaced. Then measure the fraction of the logit difference that this restores at the position where the value is predicted [8, 9]. The top head, L18H6, places 0.59 to 0.87 of its attention on the value of the requested field and at most 0.14 on any other field. We then ablate heads by replacing each output with its mean over 1,000 people.

Heads ablatedRewriting accuracy
None99.4%
Top 20 heads51.4%
20 random heads (5 seeds)≥ 95%

These results suggest that the model may perform associative recall for rewriting.

3. Restructuring. For 1,000 people with a randomly chosen style, we change only the style or only the requested field.

PredictionExpected outputResult
Template depends on the styleTemplate of the new style, same value100%
Template depends on the requested fieldTemplate of the new field, same style90.0%

Overall, above results support our explanation of the minimal rewriting model. We also add these results with details in App D.


[7] Arora et al., Zoology: Measuring and Improving Recall in Efficient Language Models, 2023.

[8] Zhang & Nanda, Towards Best Practices of Activation Patching in Language Models: Metrics and Methods, ICLR 2024.

[9] Wang et al., Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, ICLR 2023.

Reviewer_neWEOpenReview ↗
Rating
3(+2)
Confidence
3(−1)
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper provides theory for why rewrites is effective for pre-training, even when the rewriting model is small or lacks the target downstream capabilities. They formalize a "rewriting-pretraining gap," defining model rewriting as a low-complexity in-context lookup-and-template-filling operation (via RASP-L). They argue that applying such transformations changes the causal prediction problem during pretraining, increasing the computationally bounded structural information (epiplexity, following Finzi et al., 2026) that is being learned. To test this hypothesis, they evaluate on three synthetic knowledge task: biography attribute extraction, knowledge reversal, and two-hop composition.

Critical strengths

Parts of the conceptual framing are insightful. Specifically, the intuition in Section 5, that removing or reordering accessible in-context signal (such as a prefix seed zz) prevents a causal learner from relying on in-context shortcuts and forces it to internalize structural regularities into its parameters is an intuitive way to think about how data formatting shapes autoregressive representation learning.

(They are not the first to discuss this mechanism, however, as https://arxiv.org/pdf/2608.12218 already formalizes how informative training contexts and model weights act as competing carriers of task-relevant information, such that abundant in-context signal reduces parametric knowledge storage. )

Critical weaknesses

(1) False Premise

First, the paper motivates the "paradox" of rewrites by citing large-scale web-rewriting pipelines (e.g., Maini et al., 2024, 2025; Su et al., 2025; Nguyen et al., 2025; Kang et al., 2025; Niklaus et al., 2026) as evidence that rewriting human data with a small LM paradoxically outperforms raw human data. This premise misrepresents what those papers actually show. For some, rewriting is just a proxy for data cleaning since much of internet data is filled with messy text. In other cases, the rewriting does not replace the original text but augments it, which is also rather intuitive why that would help.

While the authors seem to implicitly concede that this is some form of data augmentation, they claim that existing theories of data augmentation cannot explain rewrites because "in pretraining, rewriting operates on unlabeled data with no notion of a label that must remain invariant". This is a strawman of both self-supervised learning theory and natural language structure. First, while pre-training text does not have a single class label yy, documents contain underlying latent labels, semantic variables and relational tuples for instance, that remain invariant across rephrasings. Second, classical invariance and data augmentation theories do offer a plausible explanation to why expressing the same underlying fact across diverse surface templates helps: while it does not add new information, surface variation prevents overfitting to specific phrasings.

(2) Unoriginal Experiments

Nearly the entire empirical section is a reproduction of existing experiments from prior work. Knowledge Extraction (permuting and paraphrasing synthetic biographies) is taken directly from Allen-Zhu & Li (2023a); Knowledge Reversal (injecting identity bridges like "e has a self named e") is taken directly from Ma et al. (2026); and Two-Hop Composition (along with the cross-layer bridge injection) is taken directly from Lin et al. (2026) and Wang et al. (2024).

Questions for authors

N/A

7 comments

We thank the reviewer for the detailed and thoughtful review, and for acknowledging the usefulness of our framing. We address each raised weakness below.

S1. Relation to concurrent work [1]

We would like to kindly remind the reviewer that, as per official ICLR policy [2], papers published after July 17, 2026 are considered contemporaneous and the existence of such cannot be a basis for rejection (see the last item of FAQ for Reviewers). [1] appeared in arXiv on August 12, 2026.

That said, while [1] proposes a related idea, we believe that it is complementary to our work. The paper studies how long-document pretraining affects the information encoded in model parameters, by training on the same data with varying context lengths, and finds that the minimum task information that the weights must hold can decrease as the context grows (Prop. 4.2 therein). Therefore, the approach is specialized to when the context length changes, whereas our question and the approach targets why data rewriting can help in a more broad context, including but not limited to withholding information by shortening context length (see our Props. 5.1-5.3). Importantly, the use of computationally bounded information (Eq. (5)) is important for this generality, as transformations that do not drop information, such as reordering and fictional rewrites, would not increase Shannon information, whereas [1] uses Shannon information for the theory.

We will cite [1] and discuss it as related work in the next revision.


[1] Uzunoglu et al., Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge, 2026.

W1-1. False Premise: data cleaning and augmentation instead of replacement?

As the reviewer pointed out, our motivation makes a new interpretation of existing empirical results in [3-8]. Our premise is not that existing explanations therein are wrong, but that experiments suggest they are not enough (also see our responses to W1-2 and W1-3). As such, we focus on the observation that adding rewrites produced by a small language model to human data can improve downstream generalization, which is grounded in unexplained results in [3-8] and we have verified in our experiments. We have revised the introduction to better acknowledge existing explanations and communicate our position. Below, we address two specific points raised by the reviewer.

Proxy of data cleaning. We remark that gains of rewriting are reported even when the source is already of high quality (see Table 10 of [5] and Table 2, 4 of [8]), and mixtures of rewrites and original text work better than rewrites alone (see Table 7 of [8]). We argue that explanations purely based on data cleaning, such as that web rephrasing is “akin to an aggressive data filtering or quality enhancement step” [7], or that rephrasings “effectively reduce noise and errors in low-quality data” [5], cannot fully account for this empirical evidence.

Augmentation instead of replacement. While augmentation is an intuitive source of gain, this idea alone is insufficient to explain why some kind of augmentation helps more than the other. For instance, at matched training tokens including augmentation, mixing in language model rewriting matches or exceeds raw text (see Table 1 of [3] and Table 1 of [6]), while other augmentations such as synonym replacement and random deletion help much less (see Fig. 6 of [3]), and repetition or naive continuation gives no gain (see Fig. 4 of [4]). In addition, our approach based on epiplexity does not contradict with the observation that mixing rewritten and raw text can be better, as epiplexity (Eq. (4)) has been used to optimize the mixture ratio of several data domains [9]. As such, we view this axis (mixture of rewriting and raw data) as complementary.

Together, these observations suggest that the form into which documents are rewritten matters in addition to cleanliness and quantity, and that a small model suffices to produce it. Our account addresses these two aspects: a rewriting can be simple for a small model to perform, and can still change the learner’s prediction problem so that it acquires more structure.


[2] ICLR 2027 Reviewer Guide, https://iclr.cc/Conferences/2027/ReviewerGuidelines.

[3] Maini et al., Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling, 2024.

[4] Maini et al., BeyondWeb Lessons from Scaling Synthetic Data for Trillion-scale Pretraining, 2025.

[5] Su et al., Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset, 2025.

[6] Nguyen et al., Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models, 2025.

[7] Kang et al., Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls, 2025.

[8] Niklaus et al., How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data, 2026.

[9] Finzi et al., From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence, 2026.

W1-2. False premise: classical invariance, data augmentation, and SSL theories?

We acknowledge that our sentence about data augmentation was too strong, and have toned it down in the introduction. Nevertheless, we argue that theory on invariance and data augmentation, as well as self-supervised learning theory and natural language structure, are insufficient to explain certain aspects of rewriting that motivates our work. We elaborate below.

Classical invariance and data augmentation theories. While rephrasing often preserves latent semantic variables and factual relations, we note that existing theories on data augmentation require more than the existence of such invariants [10-13]. Most importantly, they require the task to be fully compatible with the augmentation group, in the sense that training and testing shares the same invariance under augmentations [10-13]. This is assumption is essential, as these theories derive their guarantees from reducing the hypothesis space through group symmetries. However, in the context of language model pretraining, we apply rewrites to unstructured text and expect generalization in different tasks, leaving a gap from the theory of data augmentation. For example, in our experiments on biography attribute extraction, rewriting is applied on biographies, whereas the downstream task only receives a person’s name. Furthermore, we note that our Section 6.3 introduces fictional rewrites that alters the original factual relations, which improves generalization as well and is therefore unexplained by invariance.

Self-supervised learning theories. We note that most existing theories on self-supervised learning with data augmentation [14-18] do not fundamentally remove the aforementioned assumption. Instead of fixing one labeled task, these theories let the augmentations define a whole family of admissible downstream tasks, then proves a guarantee for every task in that family [14-18]. Importantly, this does not remove the requirement that the task input has to be compatible with the augmentation group. As such, the same limitation as classical invariance and data augmentation theories applies.

Surface variations. We agree that surface variation can reduce overfitting to specific phrasings. However, our empirical results on knowledge reversal suggests that this explanation is insufficient. In the setting, the original dataset already contains both the forward and the reverse statement of every in-distribution fact as in Table 2 of our paper, which adds surface variety without adding information as the reviewer suggested. Yet, we remark that training on these surface variations does not generalize to reverse questions about held-out facts. In the experiment, we have shown that what helps is adding identity statements, which are not surface variations, and actually removes information in the classical sense. We further remark that this finding is consistent with our theory, as the amount of structural information defined and measured in epiplexity increases (see Table 3).

Natural language structure. Lastly, we provide related evidence that the benefit of rewriting does not purely stem from the invariances intrinsic in the natural language structure. If it did, we would expect relatively stable benefit across different model architectures, as invariance is usually considered a property of data distribution and the task itself, largely agnostic to the choice of the learner (e.g., see [12] and especially [19]). However, [20] reports that, in the attribute retrieval setting similar to our work, sentence permutation helps autoregressive transformers considerably more than bidirectional ones in terms of downstream task generalization. A dependence on the learner of this kind is natural in our account, which is stated for a computationally bounded learner.


[10] Chen et al., A Group-Theoretic Framework for Data Augmentation, 2020.

[11] Tahmasebi et al., Generalization Bounds for Canonicalization: A Comparative Study with Group Averaging, 2025.

[12] Lyle et al., On the Benefits of Invariance in Neural Networks, 2020.

[13] Dao et al., A Kernel Theory of Modern Data Augmentation, 2019.

[14] Balestriero & LeCun, Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods, 2022.

[15] Johnson et al., Contrastive Learning Can Find an Optimal Basis for Approximately View-invariant Functions, 2023.

[16] HaoChen et al., Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss, 2022.

[17] Arora et al., A Theoretical Analysis of Contrastive Unsupervised Representation Learning, 2019.

[18] Dubois et al., Improving Self-Supervised Learning by Characterizing Idealized Representations, 2022.

[19] Kim et al., Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance, 2024.

[20] Allen-Zhu & Li, Physics of Language Models: Part 3.1, Knowledge Storage and Extraction, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.