- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 3(+2)
- Presentation
- 2(+1)
- Soundness
- 3(+2)
Read review
Paper summary
The paper analyzes the phenomenon of generalization as a result of fine-tuning models with data rewritten by a smaller model. They propose three types of rewriting --reordering, deletions and fictional-- and investigate their impact on a model using task specific measures (accuracy) and an information theoretic measure (epiplexity). The authors investigate three specific research questions, that contribute information about the level of generalization achieved through the added training, and its impact on downstream tasks.
Critical strengths
Understanding the impact of synthetic data, particularly data generated by a smaller model, is a worthwhile endeavour, and can bring interesting insights into the capabilities and behaviour of LLMs.
Critical weaknesses
The paper is difficult to follow, it introduces concepts that are then not used, it overloads terminology and notations. The experimental set-up is not very clear, and as a result of that, the results presented are difficult to interpret. There are also working assumptions that are not self-evident, and require justification.
The paper makes a big claim, but the testing on very specific tasks and forms that fit the fact/relation type data that is rewritten do not justify the broad conclusion.
Working assumptions: lines 160-161 and 179-181: "We can therefore view these transformations as in-context recall coupled with restructuring..." -- Since the purpose of the study is insight into a model's behaviour, making such assumptions about the model is not reasonable.
Dropped concepts:
- Section 3, MDL principle: it is first defined in terms of the Kolmogorov complexity, which is then not mentioned anymore, and not relevant for the following discussion. Starting the section from line 113 instead would omit nothing relevant for the rest of the paper.
- the seed z is mentioned, but then is often dropped in a formula. It should be kept wherever relevant (e.g. left hand side of formula 10, formula 12)
Overloaded terms and notation:
- "p" is used for multiple concepts -- probablity distribution, probablity of the data, models. Because of this, it is difficult to differentiate where p(x) or p*(x) refers to a probability, or to the result of applying a model to input data x.
- "P" is used for the set of models/model class, and for prompts
- keys, values, queries, "k", "v", "q" -- these are notations that are strongly associated with the attention mechanism, which is also mentioned in the paper, but are used differently in describing the rewriting process. It would be useful to have some alternate terms, e.g. "attributes" instead of "keys" maybe (and one such change would break the strong priming of attention mechanism concepts)
- "A" denotes a training algorithm but also the domain of the values
Unclear experimental process:
- the input that is rewritten is presented as tuples in the paper. Is that how it is presented as input for rewriting or is the result of rewriting actual text?
- the description of the experimental set-up (lines 335-341) is not very clear. It makes it seem like rewritten models are trained on the rewritten D and D_ID, while the others are trained only on D (which includes D_ID). Is it possible that the difference in performance comes from the fact that the additional data for the model pretrained with rewrites reinforces the patterns in that particular data?
Why are the epiplexity values for the Extraction task so much greater than for the reversal and composition?
Questions for authors
The paper requires substantial rewrites to be clear, no specific question could address that. But any reply to the issues raised in the Critical weaknesses section would be appreciated, particularly if some of the issues signaled are misunderstandings.
6 comments
W2. Big claim, specific tasks.
We would like to kindly remark to the reviewer that the knowledge tasks we consider are broadly used to study learning and generalization of language models in a controlled manner [1-4]. We have adopted these settings to reduce confounding effects from noise and variation in natural text [5], allowing us to examine rewriting more directly.
To provide evidence beyond these settings, we additionally report a C4 pretraining experiment using “high-quality” (HQ) rewrites following [6]. We train a GPT-2-small model for 15B tokens per condition, comparing original C4 against C4 with 30% of the data replaced by HQ rewrites. We estimate epiplexity during pretraining and evaluate downstream perplexity on The Pile, an out-of-domain dataset [6], without further fine-tuning. The results are shown below.
| Pretraining data | Perplexity on The Pile (↓) | MNLI accuracy after fine-tuning (↑) | Estimated epiplexity (Mbits) |
|---|---|---|---|
| Original C4 | 56.48 | 73.34 | 15275.86 |
| C4 with HQ rewrites (30%) | 47.87 | 73.97 | 15397.03 |
HQ replacement reduces Pile perplexity by 15.2% and increases test accuracy by 0.63 on MNLI, accompanied by an increase of 122 mega bits in estimated epiplexity. These results are consistent with the trend observed in our controlled experiments that also appears in natural-text pretraining. This suggests that our account may be relevant to rewriting beyond the controlled fact/relation settings examined in our main experiments.
[1] Allen-Zhu & Li, Physics of Language Models: Part 3.1, Knowledge Storage and Extraction, 2024.
[2] Ma et al., Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge, 2026.
[3] Lin et al., Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge, 2026.
[4] Wang et al., Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization, 2024.
[5] Allen-Zhu, Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers, 2025.
[6] Maini et al., Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling, 2024.
W3. Evidence for model assumptions?
We agree and have added a new empirical test of whether rewriting can be viewed as in-context recall coupled with restructuring. More precisely, we have formalized this view in the algorithm in Listing 2, which makes three predictions:
- In-context recall The output value is copied from the source, from the field indicated in the instruction, so replacing the value in the source should be reflected in the output.
- Associative value recall The value is retrieved through the field-value pairs given in the context [7].
- Restructuring The output template is determined by the requested field and the style. We test all three in TinyLlama-1.1B-Chat, the rewriting model used in the paper.
Each source document describes a synthetic person. A request denotes one field to be indicated in output, and the style is either bio or question:
Source: Name: Isabelle Brody Ainsworth / Birth city: Akron, OH / ...
Request: Birth city
Style: bio or question
Output:
Bio: Isabelle Brody Ainsworth was born in Akron, OH.
or
Question: Where was Isabelle Brody Ainsworth born? Answer: Akron, OH.
1. In-context recall. We request each of 6 attributes for 1,000 people, and replace one value in the source at a time.
| Source | Expected output | Rate |
|---|---|---|
| Original | Requested value | 99.1% |
| Requested value replaced | New value | 99.3% |
| Another value replaced | Requested value preserved | 99.7% |
This results shows the model follows the given prompt well, which can be shown as in-context recall.
2. Associative value recall. We rank the 704 attention heads by activation patching: we copy a head’s output from the clean run into a run with the requested value replaced. Then measure the fraction of the logit difference that this restores at the position where the value is predicted [8, 9]. The top head, L18H6, places 0.59 to 0.87 of its attention on the value of the requested field and at most 0.14 on any other field. We then ablate heads by replacing each output with its mean over 1,000 people.
| Heads ablated | Rewriting accuracy |
|---|---|
| None | 99.4% |
| Top 20 heads | 51.4% |
| 20 random heads (5 seeds) | ≥ 95% |
These results suggest that the model may perform associative recall for rewriting.
3. Restructuring. For 1,000 people with a randomly chosen style, we change only the style or only the requested field.
| Prediction | Expected output | Result |
|---|---|---|
| Template depends on the style | Template of the new style, same value | 100% |
| Template depends on the requested field | Template of the new field, same style | 90.0% |
Overall, above results support our explanation of the minimal rewriting model. We also add these results with details in App D.
[7] Arora et al., Zoology: Measuring and Improving Recall in Efficient Language Models, 2023.
[8] Zhang & Nanda, Towards Best Practices of Activation Patching in Language Models: Metrics and Methods, ICLR 2024.
[9] Wang et al., Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, ICLR 2023.
We thank the reviewer for the careful and constructive review, and for acknowledging the importance of the research problem. We address each raised weakness below.
We thank the reviewer for constructive feedback on the presentation side. We have made changes to improve the representation and clarity. In summary, they consist of the following:
The revisions in the updated manuscript are highlighted in blue.