acceptodds
Under review as a conference paper at ICLR 2027

Decoupling Token Roles in Autoregressive Pretraining

Abstract

Autoregressive pretraining increasingly draws on heterogeneous data, making it important to understand how a model learns from an individual token. The next-token prediction objective naturally identifies a token's contribution with its own loss. However, each token is not only a prediction target but also context for what follows. Using controlled corruption, we decouple these two roles and find a reversal: making a noisy token easier to predict reduces its damage as a target but increases it as context. The same decoupling helps explain text generated by language models: generation selects each token by its fit to the prefix, while its role as context is never tested against an independently determined continuation, because that continuation is generated to fit it. At known corrupted positions, acting through the context can reduce damage that removing the token's own loss does not. Understanding and controlling what a model learns from a token therefore requires decoupling its roles.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Autoregressive Pretraining, Next-Token Prediction, Data Section, Language Model, Noisy Tokens, Synthetic Data

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4XG5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors make the observation that in teacher forcing training of autoregressive models, each data token is used both as input to the model (context) and as label for the loss (target). Then, throughout the paper, the authors investigate how does each role, context and target, affects the model. They do this with controlled corruptions schemes (they set the corruption rate to 10%), and the corrupted tokens are sampled either from uniform distribution over the vocabulary or from a pre-trained LLM (GPT2).

The paper seems to include only analysis of experiments with no clear results. Notably, the presented experiments and discussions does not seem to yield any suggestions to change/improve training or inference algorithms of LLMs, as well as no new techniques for data curation.

Critical strengths

None

Critical weaknesses

Below I present the most concerning weaknesses of the paper.

Figures:

  1. Regarding Figure 2b, it is unclear whether all target tokens are corrupted or only some. It is unclear what does "Own-token loss" stands for, is it the loss averaged only on the corrupted labels? or is it averaged over all tokens? (i.e., Cross Entropy (CE) loss with some corrupted labels). Most worryingly, in lines 214-215 it is stated that "Predictable replacement selects a highly ranked whole-word candidate from a frozen language model given the same clean prefix.", then in light the results in Figure 2, in line 232 the author conclude that "Predictable replacements" (i.e., tokens replaced by sampling from a pre-trained LLM) are harder to predict than the clean token they replaced. However, in Appendix A.4, the author elaborate on how the predictable replacement where sampled. Specifically, they state that the 10 most probable token where excluded since they often contain the original clean token. Thus the confusion above is not a results (as presented in the main paper) but rather this is the expected result by design of the experiment. I find this highly misleading.

  2. Regarding Figure 3, the y-axis label "Validation LLM" does no make it clear that this is CE loss on held outset with both context and target clean, and that the labels refer only to how the model was trained. This is also not explained in the caption, and the reader needs look for it in the main text.

  3. Regarding Figure 5a, I could not find any place where is explained what is " held-out window". Additionally, the x-axis label "Context-only – Target-only cross entropy" is very unclear and uncommon in scientific writing. Probably the words "difference" or "minus" should have been used.

  4. Regarding Figure 5b, in equation 4, own-loss shift is defined per token and only with clean context and corrupted target (following the notation defined in equations 2 and 3). However, in the figure, it is shown only for the label "context-only" which mean corrupted context and clean target. Additionally, it is not defined on which tokens is own-loss shift averaged in the figure.

  5. Regarding Figure 6a, it seems show two figure, it is still not clear to me.

  6. Regarding Figure 7b, I had a hard understanding why is it interesting to measure the "Damage recovered".

Lack of clear message: Maybe the strongest weakness of the paper is that it lack a clear message. It includes a very long analysis of multiple experiments. However, I could not find a clear statement which the authors try to prove or convince to reader. The analysis does not yield any new method or improvements for curating the data. Additionally, no changes or improvements to training or inference algorithms are proposed. It is also notable that there no comparison to findings from previous works.

Are the roles decoupled?: The authors state that for understanding the two roles each token plays, i.e., context and a target, it is required to decouple the two. However, I would like suggest an explanation which appears consistent across the results presented in figures 1,3,4,5, and 6: the lower the token's loss (equivalently, the more predictable or fit to prefix), the more likely the model will "trust" it as a context. This can explains why low fit-to-prefix and random token cause less damage to the context-only model and it less affected by the fit of the continuation. For low fit-to-prefix token the model gives it less weight in the context or produce a wider distribution for the next token. On the other hand, for likely tokens the model gives high weight in the context or it is being more confident about its next token prediction and produce a picked distribution for the next token. Then in case the continuation fit, it yields a lower loss but if the continuation does not fit it causes a jump in loss. To conclude, it is not clear to me that indeed the data presented by the authors support the argument that the roles are decoupled.

Questions for authors

I would appreciate if the authors could address the two concerns in the Weakness section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4XG5 (Part1/2)

Dear Reviewer 4XG5:

Thank you for the detailed comments, and especially for the explanation you propose at the end, which is the question we most want to discuss.

W3: Are the roles decoupled?

Your explanation is that the better a token fits its prefix, the more the model trusts it as context and the more confident its next prediction is; when the continuation does not fit, the loss jumps. We tested this directly. We take a GPT-2-large trained only on clean text and place the original token, a Random replacement and a Predictable replacement at the same 16,384 positions under the same prefix, and measure its prediction of the next token.

Fixed clean modelOriginalRandomPredictable
NLL of the token itself4.0817.616.71
Entropy of the next-token distribution2.474.052.78
NLL of the true next token2.477.506.38

Half of your explanation holds: after a random token the model widens its prediction, and after a token that fits the prefix it is almost as confident as after the original token.

The consequence it predicts does not hold for the immediate prediction: the NLL of the true next token is lower after a Predictable token than after a Random one. The larger damage of Predictable replacement under context-only training therefore cannot be explained by a jump in loss at inference; it forms during training. Our understanding is that context-only training teaches the model a dependency from the wrong token to the real continuation, and a dependency on a token that is far from plausible is the less harmful one.

On whether the roles are "really decoupled", we think the wording caused a misunderstanding. We do not claim that the two roles are unrelated; your explanation itself shows that the same property connects them. "Decoupling" refers to the experimental operation. Ordinary training always applies both effects together, and the differences between the two replacements offset each other: injected in the ordinary way, Predictable and Random replacement cause almost the same damage (0.17 vs. 0.16). Only when the same noisy tokens are routed into one role at a time does Predictable replacement turn out to be less damaging as a target (0.07 vs. 0.12) and more damaging as context (0.12 vs. 0.05).

W2: Lack of a clear message

We respect the time you spent on the paper, but we respectfully disagree with four statements in this paragraph, and address them one by one.

"I could not find a clear statement which the authors try to prove." The claim of the paper is that a token's own loss measures only its role as a prediction target, while its effect on training as context can go in the opposite direction, so understanding and controlling what a model learns from a token requires decoupling the two roles. It is stated in the abstract, the Introduction, the contributions, the end of Section 3.3 and the Conclusion.

"The analysis does not yield any new method or improvements for curating the data." The analysis identifies two problems in how pretraining data are currently selected. First, token-level cleaning masks or down-weights the loss at a suspicious position, on the assumption that once the bad token is located, removing its loss removes its harm. Section 4.2 gives loss masking the exact oracle positions, and it recovers −13% (GPT-2) and −0.3% (SmolLM2) of the damage. Second, when synthetic pretraining data are generated, greedier decoding makes the text easier to predict, but a generated stretch of 32 tokens disturbs the independent continuation after it more than random tokens do (Figure 6a).

"No changes or improvements to training or inference algorithms are proposed." Section 4.2 proposes and evaluates a change to training, the token-level reliability flag. At the same positions where loss masking recovers none of the damage, it recovers 39% and 41%.

"There no comparison to findings from previous works." Section 2 places our findings against prior work, for example why selection methods that score a token at its own position capture only its target role, and how our fixed human continuation differs from exposure-bias and model-collapse studies in which generated text writes its own continuation. Experimentally, the oracle loss mask is the best case of methods built on loss masking, in which every noisy token is found and its loss removed, and it recovers none of the damage.

AuthorsOpenReview ↗

Response to Reviewer 4XG5 (Part2/2)

W1: Figures

Figure 2b. You point out that Predictable replacements being harder to predict than the original tokens follows from the design and is not a finding. This is correct, and the paper does not present it as a finding. The main text introduces the two replacements with: "To compare tokens of very different difficulty at the same positions, we construct two corruptions that differ in how the replacement is chosen (Appendix A.4)". Figure 2b measures how large a difference in difficulty this construction produces. The conclusion of lines 232–233 is that Predictable replacements are much easier to predict than Random ones; "though still harder than the tokens they replace" only notes that they are still wrong tokens.

On what is measured, own loss is defined where it first appears (lines 52–53): "the loss at the noisy token's own position". Both corruptions target 10% of the training positions (lines 234–237), not all targets. Figure 2b considers 128 matched positions that are replaced in both corpora. At each position, the same fixed clean GPT-2-large scores the original token, the Random replacement and the Predictable replacement under the same clean prefix. Each of the three distributions has 128 values, and the dashed lines are their means.

How the candidates are chosen and what this choice does to training are two different questions. Appendix A.4 gives the candidate range of ranks 10–50 and the reason: the top-ranked candidates are often the original token or a close variant, and replacing with them would introduce no noise. A Predictable replacement, however easy to predict, has to be a wrong token, so its mean own loss falling between the original and Random is expected, and we do not present this order as surprising. The question we raise is whether Predictable replacement, being easier to predict than Random, is therefore less harmful to train on. Figure 3 answers that it depends on the role in which the replacement occurs. The rank rule does not determine this reversal; it is measured by the training experiments.

Figure 5b. The own-loss shift in the lower row is a property of the four replacement sets, measured under a fixed clean model. The current legend makes it look as if it belonged to the context-only condition, and we will correct this.

Figure 7b. Damage recovered puts the two systems on a common scale: 0 means the same as no intervention, and 100 means the level of clean training.

Figure 6a. Each point is one kind of replacement: the horizontal axis is how easy the replaced tokens themselves are to predict, and the vertical axis is how much harder the first human token after them becomes.

Figures 3 and 5a. We agree with the remaining comments on labels and will revise them. All held-out cross entropies are measured with clean context and clean targets, and a held-out window in Figure 5a is a contiguous stretch of 1,024 tokens of the held-out set.


Thank you again for the effort you put into this review. We hope these clarifications help your re-evaluation.

Reviewer_4XG5OpenReview ↗

Reviewer response

I see that my "lack of clear messages" point is strongly aligned with the other two reviewers. Both other reviewers have flagged serious presentation concerns and validity of the problem (at least as is currently presented).

Considering this, I keep my score.

Dear Reviewer 4XG5,

Thank you for following up; we truly appreciate the time and effort you have put into reviewing our paper.

We would like to gently clarify the record on two points:

  1. Your initial rating of 1 in Soundness rested on two question. In our response, we clarified the design role of Figure 2b and ruling out an inference-time jump and confirming that the reversal forms during training. Could you clarify whether these results address your original technical concerns, or whether any specific soundness issues remain?

  2. On "lack of a clear message": We addressed each of your original concern in W2, pointing to the relevant sections and concrete results from paper. We fully acknowledge that the presentation needs improvement, but we believe this is distinct from whether the paper presents a substantive finding or a proposed method. Both other reviewers accurately identified our central result in their summaries. Reviewer n9hN also described the scope as "clear and well-bounded", the results as "robust", and the empirical study as "quite strong", while recognizing the proposed approach in Section 4.2.

We would appreciate it if these distinctions and our specific responses to the original concerns could be taken into account. We have committed to revising the text and figure labels to address the presentation issues, and we hope the resolution of your technical questions can be reflected in the Soundness and overall evaluation.

Reviewer_2jYcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to understand what a natural language model learns from each token during pretraining. To achieve that, they claim to decouple the two roles of tokens: as targets to be predicted and as context to be referred to. Their remark is that it is insufficient to evaluate the current token xix_i only by the loss at the current position ("own loss"), and that it is necessary to consider how well it is predicted and how well it is used as context to predict later tokens. They perform empirical studies to conclude that tokens that are easy to predict can sometimes be more damaging in context, and that each loss depends on whether xix_i fits to the prefix or the continuation.

Critical strengths
  1. Unique approach. The paper suggests decoupling tokens' roles (prediction target or context), which were previously combined in LM pretraining, to understand what LMs learn from tokens.
  2. Rich experimental results. The paper conducts a rich empirical study showing that easily-predictable tokens can sometimes be harmful in context, and vice versa.
Critical weaknesses
  1. Lack of discussion regarding the validity of the problem. The paper never provides evidence that clearly shows what problems arise from the fact that general pre-training methods for autoregressive language models do not distinguish tokens' roles as "prediction targets" and "context".
  2. Unclear experimental setup. The paper claims that the context-only model is worse than the target-only model under predictable replacement, and better under random replacement. However, the context-only model might learn only to generate a likely-correct token from a corrupted context, thereby underperforming on normal predictions. Therefore, the claim is not novel.
  3. Concerns of reproducibility. The paper never provides any algorithmic explanations to show how to measure each observable in its experiments.
  4. Strong assumption to the practical problem. In Section 4.2, the paper applies the separated roles of tokens to the practical problem of intervening on individual tokens during training, assuming the exact position of the corruption, but this assumption is too strong in real settings.
Questions for authors
  1. Why is it important to understand what kind of training data a language model uses and what the model learns from each token? There is no section that discusses the specific challenges that arise from not knowing what a language model learns from tokens during pretraining.
  2. Line 44 states, “The effect of the currently generated token being used as context appears in the loss of subsequent tokens.” If that is the case, then it seems the token’s role is already taken into account within the entire sequence, so wouldn’t it be unnecessary to decouple the roles?
  3. In Section 4, why are you comparing human text and LM tokens? Wasn't this paper supposed to be about understanding the roles of tokens and their dependency on prefix fit and context fit?
  4. What do your empirical results indicate about the LM's performance?
  5. In Figure 1, aren’t the results of “predictable” totally dependent on the performance of the small model? Is the small model that is used to choose a noisy token trained to predict a clear token suitable for a given prefix? If so, it is not surprising that the target-only model shows a lower loss, while the context-only model shows the opposite.
  6. In Figure 3, isn’t the fact that the loss of the context-only model becomes large under the “predictable” case due to (minor) contextual breakdowns caused by the small model generating alternative tokens at each token position?
  7. In the fourth paragraph in the Introduction, how do you define “damage”?
  8. In Section 3.3, isn’t the fact that the reversal between the two roles due to the fact that the context-only model just earns the ability to generate a likely-correct token from a corrupted context?

Update after rebuttal. I acknowledge that this paper offers some useful insights, but I think this content is better suited to workshops. Note that this is not negative feedback, and I encourage the authors to move forward with this research and present it at a future conference. Accordingly, I changed the "informativeness" score from 1 to 2, but I keep my rating.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 2jYc (Part1/2)

Dear Reviewer 2jYc:



Thank you for your comments. Several of your questions point to places the paper did not make clear.

W1, Q1, Q2 and Q4: Why the problem matters

You ask why it is important to understand what a model learns from each token (W1, Q1), and why the roles need to be decoupled if a token's effect as context already appears in later losses (Q2). These are the same question, and we answer them together.

For the objective over the whole sequence you are right: it already includes the effect of the context. But a token-level decision, such as selecting or masking position kk, sees only ℓk\ell_k. For a wrong token that fits its prefix, ℓk\ell_k is low, and the damage it causes is carried by the positions after kk.

This is more than a conceptual distinction. The common way to handle a suspicious token is to mask or down-weight the loss at its own position. The assumption behind it is that once the right positions are found, removing their losses removes their harm. Section 4.2 tests this assumption by giving the training algorithm the exactly correct positions: masking their losses recovers none of the damage (−13% and −0.3%), while marking the same positions as noisy in the input, so that later predictions read the mark, recovers 39% and 41%.

For model performance (Q4), this means that the damage caused by the same amount of noise depends on the role through which it enters training, and noise with a lower own loss is not safer. Judging or handling a token by its own loss treats a wrong token that fits its prefix as safe, which is the problem W1 asks for.

W2, Q5, Q6 and Q8: Is the result only a breakdown of the context?

These points share one concern: the context-only result may only reflect that the tokens chosen by the small model break the context, or that the model only learns to predict from a corrupted context. This can be tested directly. We take a GPT-2-large trained only on clean text, place a Random or a Predictable replacement at the same 16,384 positions, and measure its prediction of the true next token. The NLL is 2.47 with the original token, 6.38 after a Predictable token and 7.50 after a Random token. A Random token therefore breaks the context more at the moment it is read. If the damage in training were only the accumulation of this breakdown, Random replacement would be worse under context-only noising, and the result is the opposite.

Target-only and context-only noising also use exactly the same positions and exactly the same replacement tokens (Q5). The same tokens are better than random ones in one role and worse in the other, which the quality of the small model's choices cannot explain.

You say the context-only model learns to predict from an unreliable context. We agree, and this is also how we understand the result. We do not agree that the claim is therefore not novel. A result that can be explained afterwards was not thereby known beforehand: injected in the ordinary way, the two replacements cause almost the same damage (0.17 vs. 0.16), and judged by own loss, Predictable replacement should be the safer one. Only when the same noisy tokens are routed into one role at a time does it turn out to be more damaging as context (0.12 vs. 0.05).

AuthorsOpenReview ↗

Response to Reviewer 2jYc (Part2/2)

W4: Known positions

You point out that Section 4.2 assumes known positions, which does not hold in real settings. This is the purpose of the experiment: we hold fixed how the corrupted positions are found, in order to study how to intervene on them. The assumption is very strong, and the finding is that even under it, simply masking the losses recovers none of the damage. This bears directly on practice.

Q3: Why Section 4 compares human and generated text

As the beginning of Section 4.1 states, generated text is Predictable replacement at scale: every generated token is chosen by its fit to the prefix. This is the property that, in Section 3, makes a token safer as a target and possibly more harmful as context.

W3 and Q7: Definitions

Damage is the increase in clean held-out cross entropy over matched clean training. It is defined at the start of that paragraph (line 65) and again in Section 3.2; we will make the definition in the Introduction more prominent. The computation of each measured quantity is given in the appendix: the own-loss shift in Appendix A.6, the two fits of generated text in Appendix B.2, damage recovered in Appendix C, and the construction of the replacements in Appendices A.4 and A.5.


Thank you again for the effort you put into this review. We hope these clarifications help your re-evaluation.

I thank the authors for a careful reply.

[SUMMARY] The authors carefully clarify my concerns. However, the unclear problem setting and the lack of pseudocode for experiments raise concerns regarding contributions to the LLM community and reproducibility. I therefore keep my rating at 1. Please see below for details.

Q1,4: Practical problem setting

  • The authors argue that Q1 and Q2 are the same, but in my mind, that’s wrong. Q1 focuses on “What they intend as a practical problem,” while Q2 focuses on “How did the authors come up with the proposed approach?”
  • This paper remains a preliminary study. It does not propose any novel training approach.
  • I understand that relying solely on loss masking could potentially lead to an increase in cross-entropy during training. However, the evaluation provided in the submitted and revised versions is limited to a simple NLL (negative log-likelihood) measurement; it does not include assessments of accuracy or perplexity (PPL) on standard benchmarks. Consequently, there is insufficient evidence to determine whether the improvements to the properties of loss masking, which the authors highlight, are truly significant or beneficial to the LLM community.
  • Although many experiments have been conducted, the lack of pseudocode for them raises concerns regarding reproducibility.

Detailed comments.

  • The answer to Q1 is more informative than Section 1, and I understood this paper addresses one of the LLM training techniques, “loss masking,” that removes the loss of tokens deemed inappropriate (or corrupted) from the objective function (however, the tokens themselves remain in the input sequence). In Figure 7 (Section 4.2), the authors confirm that even when they can precisely identify the location of a corrupted token, simply removing the loss at that position is insufficient to prevent a performance drop. However, if they can indicate (“flag”) to subsequent predictions that the token at that position is unreliable, they can partially recover from the performance drop.
  • I sincerely suggest that the authors describe the primary problem of loss masking in Section 1, and provide Figure 7 as a preliminary experiment in Section 1 or soon after.

Q2: The background the authors came up with the approach

  • The authors argue that Q1 and Q2 are the same, but in my mind, it’s wrong. Q1 focuses on “What they intend as a practical problem,” while Q2 focuses on “How did the authors come up with the proposed approach?”
  • The answer to Q2 is missing, and I still cannot understand why the authors introduce the roles “prediction target” and “contextual reference” for tokens.
  • If the authors simply want to compare the effects of “loss masking” and “reliability flags,” I don’t think it’s absolutely necessary to separate the roles of the tokens.

Q3: Why Section 4 compares human and generated text

  • This paper focuses on causal LLMs, not diffusion models. Therefore, I'm still wondering why LLM training needs to consider consistency with subsequent tokens.

Q7: Definitions of metrics

  • Please define all metrics introduced for the first time in the paper using mathematical formulas.
Reviewer_n9hNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper looks at how noisy tokens during pretraining damage the NLL of a trained model. Rather than just flipping a token in general, it looks at two cases: Flip the token only as prediction target, but then use the true token as context for the next tokens, or train on the true token, but replace it in the context for the upcoming tokens. They then pretrain small language models on this adapted data. They make three findings:

  1. Both the noise as prediction target and the noise in context hurt individually
  2. When noise is strong (= random replaced token), being a prediction target hurts more
  3. When the noise is little (plausible token inserted by a small model, but != the real token), more damage comes from misleading the context.
Critical strengths
  1. The scope is clear and well-bounded
  2. Results are robust across models, train regimes (pretraining, continued training), and after trying to reduce confounders (Sec 3.3)
  3. As a toy-experiment / isolated study, it is quite strong since exhausts all experiments around this effect that come to mind.
Critical weaknesses
  1. The scope is more of a workshop paper, since it is a single effect studied on relatively small scale, and without really making improvements to the state of the art (injecting noise and breaking a model is the easy part, dealing with unknown noise and improving a model is the hard part). It does go into a direction of an actually useful method in the last three paragraphs of Sec 4.2, but is very preliminary/pilot, and without comparison to the SOTA or any of the cited related works in the field.
  2. Throughout the text, there are instances that read AI-written, introducing undefined new terms, being vague, weirdly dense, odd word choices, and the likes. I often had to read sentences two or three times to piece together what is happening, because they only make sense once you read a bit of what comes after. It is not as bad as in some other submissions in my batch, but readability suffers (and in turn, reproducibility).
  3. The analysis in 3.3 is good to make everything more precise (because some inserted tokens might be good), but remains somewhat vague. I recommend rewriting the text, plus reporting raw data in Fig 5b, rather than splitting into “low” and “high” (without defining the boundary)
  4. I’m not sure whether Section 4’s framing is really precise. It says it’s about what happens in practice, but from what I can tell, it’s really just about extending from individual replaced tokens to spans of 32 replaced tokens.
  5. Since everything is more of an isolated study / toy-experiment, there are no comparisons to related works / SOTA, although there are already works that build up on this effect to propose methods.

Notes that don’t influence my score:

  1. I wouldn’t call it “context-only training”, but rather “context-only noising” or so.
Questions for authors

Less of a question than a suggestion: If this paper ends up getting rejected, I would encourage you to take all existing effects as a pretext / motivation, and then write a paper that uses this to propose a method that actually improves pretraining. Similar to what you did at the last three paragraphs in the paper, but then of course in an actual benchmark setup, with comparisons to SOTA methods, without injected noise but by mitigating existing noise, etc.

4 comments
AuthorsOpenReview ↗

Response to Reviewer n9hN (Part1/2)

Dear Reviewer n9hN:

We sincerely thank you for reading the paper carefully and for the candid suggestion at the end.

W1 and W5: Contribution

Our view is that showing the limits of a common understanding is itself a contribution, and does not have to come with a new method. Understanding deep learning requires rethinking generalization (ICLR 2017) is an example, and our use of token-level noise follows it. That paper proposed no method, and randomizing the labels is about the easiest experiment one can think of; its value lay in what the experiment showed.

You say that injecting noise and breaking a model is the easy part, and we agree. But when the noise is injected in the ordinary way, Predictable and Random replacement damage the model almost equally (0.17 vs. 0.16). Only when the same noisy tokens are routed into one role at a time does the order reverse: at the same positions, Predictable replacement is less damaging than Random as a target (0.07 vs. 0.12) and more damaging as context (0.12 vs. 0.05). We consider this the main contribution of the paper.

You say that dealing with unknown noise and improving a model is the hard part. We fully agree, and a complete solution is beyond the scope of this paper. What the paper does show is why it is hard, and why the current view of the problem is flawed. The common way to handle a suspicious token is to mask or down-weight the loss at its own position. The assumption behind it is that once the right positions are found, removing their losses removes their harm. Section 4.2 tests this assumption by giving the training algorithm the exactly correct positions: masking their losses recovers none of the damage (−13% and −0.3%), while marking the same positions in the input, so that later predictions read the mark, recovers 39% and 41%. The practical consequence is direct: when unreliable text is still read by later predictions, masking its own loss does not deal with it.

On W5, could you point us to the works you have in mind that build on this effect? If they are methods that first locate suspicious tokens and then mask or down-weight their losses, the oracle setting of Section 4.2 already gives the upper bound of the locating step: under Predictable replacement, masking the loss does not help even when every position is located correctly. A comparison with any particular detector would therefore not change this conclusion.

W3: "Low" and "high" in Figure 5b

"Low" and "high" are four constructed replacement sets, with no threshold on a continuous variable. At each position, the high-fit group takes candidates from the top of the teacher's ranking and the low-fit group draws uniformly from the candidates that pass the filter; within each group we take the candidates with the lowest and the highest continuation loss. The held-out NLL of the four sets is below, with arrows going from low to high fit to the prefix.

ContinuationTarget-onlyContext-only
Fits3.356 → 3.2383.267 → 3.252
Does not fit3.355 → 3.2433.295 → 3.339

These are four training endpoints. We will revise the text to state the construction explicitly and report these values.

AuthorsOpenReview ↗

Response to Reviewer n9hN (Part2/2)

W4: Framing of Section 4

You are right about the title. Section 4.1 is still a controlled experiment, in which 32-token generated stretches are placed into human text, and "in practice" says too much. We will change it to describe settings closer to practice.

W2 and W6: Writing

We accept the points on writing. A paper that only describes observations, with an observation tool of our own, turned out to be much harder to write than a method paper in an existing setting. We will rewrite the passages where a term is used before it is defined. "Context-only noising" is more accurate than our name, and we will adopt it.

Q1: A method that improves pretraining without injected noise

This is indeed our next step, and we have some first results in a setting without injected noise. Each token is predicted causally as usual; after the model has read a short stretch of the real continuation, the state that this token leaves for later predictions is revised. On FineWeb-Edu with GPT-2 XS, this reaches a validation NLL of 3.31 with 5B tokens; interpolating the baseline learning curve, a standard Transformer needs about 6.24B tokens to reach the same level, so at this point about 20% fewer training tokens are needed. We also note that the wall-time advantage of the current algorithm shrinks as the training data grows, and we do not yet have the compute to test it at a larger scale.


Thank you again for the effort you put into this review. We do will take your suggestion seriously.

Thanks for the response!

W1, W5

I agree that the contribution is not zero. I do like scientific understanding papers. However, and I think in this point the other reviewers agree, the contribution here is more that of a workshop paper than of a full-conference paper, since the experiments remain toy scale (if I compare to e.g. the other papers in my batch). I would really cheer for this as a workshop paper, to gather insights from the community and then build a full methods paper out of it.

W2, W3, W4, W6

Thanks for adapting this!

Q1

I think this is the right direction to dig into next. A full conference paper will require this analysis to be built out, compared to SOTA, etc, so I do not feel comfortable passing it as is right now, but I encourage to pursue this further.


I'll maintain my score of 2, because I think the scope of this paper is that of an ACL Findings or ICLR workshop paper, but not fully at the stage of an ICLR main paper yet. However, I want to note to the AC that I disagree with the score 1 reviewers. I do not think this paper deserves 1-1-2. I have clear AI slop papers in my batch that have a higher score (also clearly better papers with the same low score though, the scores this year are closer to random than I have ever experienced it). I do not agree that this falls into the category of clear reject / AI slop papers, and I disagree (though respectfully) with 2jYc's critique that not having pseudocode would be a ground for rejection. The paper is not too complicated to reproduce. (I agree with their other points though). Overall, I think a 2-2-2 or 2-2-1 would be a fairer rating of this paper, but I don't want to prescribe the other reviewers scores.

Dear Reviewer n9hN,

Thank you for the follow-up, and for engaging with the paper so carefully throughout the discussion.

We respect your assessment of the scope, and we will take your suggestion: building the analysis into a method.

We are also grateful for your note to the AC and for your views. It is encouraging that the paper was read as a careful study with "human intelligence" beyond the rating itself, and your comments throughout have helped us see how to make it clearer.

Thank you again for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.