- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors make the observation that in teacher forcing training of autoregressive models, each data token is used both as input to the model (context) and as label for the loss (target). Then, throughout the paper, the authors investigate how does each role, context and target, affects the model. They do this with controlled corruptions schemes (they set the corruption rate to 10%), and the corrupted tokens are sampled either from uniform distribution over the vocabulary or from a pre-trained LLM (GPT2).
The paper seems to include only analysis of experiments with no clear results. Notably, the presented experiments and discussions does not seem to yield any suggestions to change/improve training or inference algorithms of LLMs, as well as no new techniques for data curation.
Critical strengths
None
Critical weaknesses
Below I present the most concerning weaknesses of the paper.
Figures:
-
Regarding Figure 2b, it is unclear whether all target tokens are corrupted or only some. It is unclear what does "Own-token loss" stands for, is it the loss averaged only on the corrupted labels? or is it averaged over all tokens? (i.e., Cross Entropy (CE) loss with some corrupted labels). Most worryingly, in lines 214-215 it is stated that "Predictable replacement selects a highly ranked whole-word candidate from a frozen language model given the same clean prefix.", then in light the results in Figure 2, in line 232 the author conclude that "Predictable replacements" (i.e., tokens replaced by sampling from a pre-trained LLM) are harder to predict than the clean token they replaced. However, in Appendix A.4, the author elaborate on how the predictable replacement where sampled. Specifically, they state that the 10 most probable token where excluded since they often contain the original clean token. Thus the confusion above is not a results (as presented in the main paper) but rather this is the expected result by design of the experiment. I find this highly misleading.
-
Regarding Figure 3, the y-axis label "Validation LLM" does no make it clear that this is CE loss on held outset with both context and target clean, and that the labels refer only to how the model was trained. This is also not explained in the caption, and the reader needs look for it in the main text.
-
Regarding Figure 5a, I could not find any place where is explained what is " held-out window". Additionally, the x-axis label "Context-only – Target-only cross entropy" is very unclear and uncommon in scientific writing. Probably the words "difference" or "minus" should have been used.
-
Regarding Figure 5b, in equation 4, own-loss shift is defined per token and only with clean context and corrupted target (following the notation defined in equations 2 and 3). However, in the figure, it is shown only for the label "context-only" which mean corrupted context and clean target. Additionally, it is not defined on which tokens is own-loss shift averaged in the figure.
-
Regarding Figure 6a, it seems show two figure, it is still not clear to me.
-
Regarding Figure 7b, I had a hard understanding why is it interesting to measure the "Damage recovered".
Lack of clear message: Maybe the strongest weakness of the paper is that it lack a clear message. It includes a very long analysis of multiple experiments. However, I could not find a clear statement which the authors try to prove or convince to reader. The analysis does not yield any new method or improvements for curating the data. Additionally, no changes or improvements to training or inference algorithms are proposed. It is also notable that there no comparison to findings from previous works.
Are the roles decoupled?: The authors state that for understanding the two roles each token plays, i.e., context and a target, it is required to decouple the two. However, I would like suggest an explanation which appears consistent across the results presented in figures 1,3,4,5, and 6: the lower the token's loss (equivalently, the more predictable or fit to prefix), the more likely the model will "trust" it as a context. This can explains why low fit-to-prefix and random token cause less damage to the context-only model and it less affected by the fit of the continuation. For low fit-to-prefix token the model gives it less weight in the context or produce a wider distribution for the next token. On the other hand, for likely tokens the model gives high weight in the context or it is being more confident about its next token prediction and produce a picked distribution for the next token. Then in case the continuation fit, it yields a lower loss but if the continuation does not fit it causes a jump in loss. To conclude, it is not clear to me that indeed the data presented by the authors support the argument that the roles are decoupled.
Questions for authors
I would appreciate if the authors could address the two concerns in the Weakness section.
4 comments
Response to Reviewer 4XG5 (Part2/2)
W1: Figures
Figure 2b. You point out that Predictable replacements being harder to predict than the original tokens follows from the design and is not a finding. This is correct, and the paper does not present it as a finding. The main text introduces the two replacements with: "To compare tokens of very different difficulty at the same positions, we construct two corruptions that differ in how the replacement is chosen (Appendix A.4)". Figure 2b measures how large a difference in difficulty this construction produces. The conclusion of lines 232–233 is that Predictable replacements are much easier to predict than Random ones; "though still harder than the tokens they replace" only notes that they are still wrong tokens.
On what is measured, own loss is defined where it first appears (lines 52–53): "the loss at the noisy token's own position". Both corruptions target 10% of the training positions (lines 234–237), not all targets. Figure 2b considers 128 matched positions that are replaced in both corpora. At each position, the same fixed clean GPT-2-large scores the original token, the Random replacement and the Predictable replacement under the same clean prefix. Each of the three distributions has 128 values, and the dashed lines are their means.
How the candidates are chosen and what this choice does to training are two different questions. Appendix A.4 gives the candidate range of ranks 10–50 and the reason: the top-ranked candidates are often the original token or a close variant, and replacing with them would introduce no noise. A Predictable replacement, however easy to predict, has to be a wrong token, so its mean own loss falling between the original and Random is expected, and we do not present this order as surprising. The question we raise is whether Predictable replacement, being easier to predict than Random, is therefore less harmful to train on. Figure 3 answers that it depends on the role in which the replacement occurs. The rank rule does not determine this reversal; it is measured by the training experiments.
Figure 5b. The own-loss shift in the lower row is a property of the four replacement sets, measured under a fixed clean model. The current legend makes it look as if it belonged to the context-only condition, and we will correct this.
Figure 7b. Damage recovered puts the two systems on a common scale: 0 means the same as no intervention, and 100 means the level of clean training.
Figure 6a. Each point is one kind of replacement: the horizontal axis is how easy the replaced tokens themselves are to predict, and the vertical axis is how much harder the first human token after them becomes.
Figures 3 and 5a. We agree with the remaining comments on labels and will revise them. All held-out cross entropies are measured with clean context and clean targets, and a held-out window in Figure 5a is a contiguous stretch of 1,024 tokens of the held-out set.
Thank you again for the effort you put into this review. We hope these clarifications help your re-evaluation.
Reviewer response
I see that my "lack of clear messages" point is strongly aligned with the other two reviewers. Both other reviewers have flagged serious presentation concerns and validity of the problem (at least as is currently presented).
Considering this, I keep my score.
Dear Reviewer 4XG5,
Thank you for following up; we truly appreciate the time and effort you have put into reviewing our paper.
We would like to gently clarify the record on two points:
-
Your initial rating of 1 in Soundness rested on two question. In our response, we clarified the design role of Figure 2b and ruling out an inference-time jump and confirming that the reversal forms during training. Could you clarify whether these results address your original technical concerns, or whether any specific soundness issues remain?
-
On "lack of a clear message": We addressed each of your original concern in W2, pointing to the relevant sections and concrete results from paper. We fully acknowledge that the presentation needs improvement, but we believe this is distinct from whether the paper presents a substantive finding or a proposed method. Both other reviewers accurately identified our central result in their summaries. Reviewer n9hN also described the scope as "clear and well-bounded", the results as "robust", and the empirical study as "quite strong", while recognizing the proposed approach in Section 4.2.
We would appreciate it if these distinctions and our specific responses to the original concerns could be taken into account. We have committed to revising the text and figure labels to address the presentation issues, and we hope the resolution of your technical questions can be reflected in the Soundness and overall evaluation.
Response to Reviewer 4XG5 (Part1/2)
Dear Reviewer 4XG5:
Thank you for the detailed comments, and especially for the explanation you propose at the end, which is the question we most want to discuss.
W3: Are the roles decoupled?
Your explanation is that the better a token fits its prefix, the more the model trusts it as context and the more confident its next prediction is; when the continuation does not fit, the loss jumps. We tested this directly. We take a GPT-2-large trained only on clean text and place the original token, a Random replacement and a Predictable replacement at the same 16,384 positions under the same prefix, and measure its prediction of the next token.
Half of your explanation holds: after a random token the model widens its prediction, and after a token that fits the prefix it is almost as confident as after the original token.
The consequence it predicts does not hold for the immediate prediction: the NLL of the true next token is lower after a Predictable token than after a Random one. The larger damage of Predictable replacement under context-only training therefore cannot be explained by a jump in loss at inference; it forms during training. Our understanding is that context-only training teaches the model a dependency from the wrong token to the real continuation, and a dependency on a token that is far from plausible is the less harmful one.
On whether the roles are "really decoupled", we think the wording caused a misunderstanding. We do not claim that the two roles are unrelated; your explanation itself shows that the same property connects them. "Decoupling" refers to the experimental operation. Ordinary training always applies both effects together, and the differences between the two replacements offset each other: injected in the ordinary way, Predictable and Random replacement cause almost the same damage (0.17 vs. 0.16). Only when the same noisy tokens are routed into one role at a time does Predictable replacement turn out to be less damaging as a target (0.07 vs. 0.12) and more damaging as context (0.12 vs. 0.05).
W2: Lack of a clear message
We respect the time you spent on the paper, but we respectfully disagree with four statements in this paragraph, and address them one by one.
"I could not find a clear statement which the authors try to prove." The claim of the paper is that a token's own loss measures only its role as a prediction target, while its effect on training as context can go in the opposite direction, so understanding and controlling what a model learns from a token requires decoupling the two roles. It is stated in the abstract, the Introduction, the contributions, the end of Section 3.3 and the Conclusion.
"The analysis does not yield any new method or improvements for curating the data." The analysis identifies two problems in how pretraining data are currently selected. First, token-level cleaning masks or down-weights the loss at a suspicious position, on the assumption that once the bad token is located, removing its loss removes its harm. Section 4.2 gives loss masking the exact oracle positions, and it recovers −13% (GPT-2) and −0.3% (SmolLM2) of the damage. Second, when synthetic pretraining data are generated, greedier decoding makes the text easier to predict, but a generated stretch of 32 tokens disturbs the independent continuation after it more than random tokens do (Figure 6a).
"No changes or improvements to training or inference algorithms are proposed." Section 4.2 proposes and evaluates a change to training, the token-level reliability flag. At the same positions where loss masking recovers none of the damage, it recovers 39% and 41%.
"There no comparison to findings from previous works." Section 2 places our findings against prior work, for example why selection methods that score a token at its own position capture only its target role, and how our fixed human continuation differs from exposure-bias and model-collapse studies in which generated text writes its own continuation. Experimentally, the oracle loss mask is the best case of methods built on loss masking, in which every noisy token is found and its loss removed, and it recovers none of the damage.