acceptodds
Under review as a conference paper at ICLR 2027

Self-Generated Feedback Destabilizes Test-Time Training: A Causal Decomposition of Long-Horizon Adaptation

Abstract

Test-time training (TTT) gives a model writable state: information can persist in its weights after the source text leaves the attention window. The same mechanism creates feedback when the model learns from its own output: each update changes the model that generates the next training example. We study this feedback over 128K-token streams. Retaining generated-text writes worsens prediction of independent human-written text at three TTT-E2E scales (125M, 760M, and 3B), and the same failure occurs when Adam updates Qwen3-4B’s existing weights. Three matched comparisons provide a causal decomposition of the result. First, Fixed Generation, in which a frozen model generates every training chunk, removes 97.5% of the measured damage even though the adapting model continues to update. Second, Recorded Replay of the same degraded text shows that reading it through attention causes one loss, while retaining its updates stores an additional loss in the weights. Third, branching from an identical state before one update reveals the local conflict: the update predicts its source chunk better but predicts new real text worse. This cost grows after closed-loop adaptation, and a few source trajectories account for most large failures. Settlement evaluates the complete candidate state on independent real text before commitment, reducing the endpoint gap to at most .07 nats at 125M and 760M while retaining real-text adaptation. Thus, optimizing the current self-generated example can make later, independent inputs harder to predict.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-time training, long-context language modeling, online adaptation, fast weights, self-generated feedback, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XeCQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper explores test-time training (TTT) when the model learns from its own generated text. It mainly compares Closed Loop: uses updated model to generate next round training chunk and Writes Off: puts training chunk into context without any model update. This paper found the root cause of degradation in TTT comes from the feedback loop in which updated weights determine the text used for later updates. In the end, the authors proposed Settlement that each update is held pending and committed only if it does not increase loss on independent real text.

Critical strengths

This paper conduct a complete and well-defined experiment control. By comparing different model scale, books, seeds, generator update, and schedule, this paper achieves state restoration and generator isolation carefully. This carefully defined experiment rules out Fixed Generation and real-text learning that update model weights with less harm.

Critical weaknesses

W1: First, this paper restrict the scope on simple self-writing TTT that apply next-token updates base on the model's own unconditioned, generation without any reward, verifier, or interleaved external input. Under this setting, the Closed Loop collapses into a repetition attractor (repeated-4 = .99) and it kind of push the model weight into a state showing abnormal behaviours. This is a well known problem of TTT and this paper just designed a set of experiment to reveal this already known problem without any insight about why using updated weights determine the text used for later updates will decrease performance.

More fundamentally, the influence of TTT self-writing is in two side: it can help or harm for the following generation. This paper only captures the harm of self-writing: all language-model outcomes are Negative Log Likelihood on independent human-written text, which generated chunks cannot help by construction. This makes the Write-Off works as performance celling because it do not update model weight in a negative direction.

W2. Limited novelty of Settlement. Settlement commits an update only if it does not increase loss on held-out real text. This is a standard validation-based accept/reject rule. It should work as baseline instead of proposed solution. This Settlement also shows less practical as it requires identifiable real text for validation and needs the real deployment of the model update to assess its quality.

Questions for authors
  1. In Page 9, line 316, the authors claimed that "The write effect is confirmed at both 125M and 3B." However, the table 4 shows a significant 12x less NLL than 125M scale. The same effect also appears in table 1 as 3B scale with significant less H difference. Could you please justify why?

  2. Could the authors provide some qualitative inspection of the closed-loop generated training chunks that also degrade the performance of replay recorded so significantly.

  3. The proposed Settlement is defined at very late section 6 and used before it in section 5. The authors would be better to change the order of presentation.

  4. The author would be better to evaluate at least one setting where self-writing could help, such as recall of self-generated content beyond the window

4 comments
AuthorsOpenReview ↗

Response to Reviewer XeCQ: W1

Thank you for your time and your constructive suggestions. We appreciate your recognition of our “complete and well-defined experiment control” and that our paper “achieves state restoration and generator isolation carefully.”


W1.1: "paper restrict the scope on .... TTT that ....base on the model's own unconditioned, generation without a..... interleaved external input."

Thanks for your comments. We suppose that the reviewer may have misunderstood our scope. Generation starts from an 8K real-text prefix (Sec. 3). We also test interleaved real text (Sec. 4; Appendix B.3), real-text validation through Settlement (Sec. 6.2), and task-reward validation in WebShop (Appendix G).

In Table 12, real text makes up 31% of each stream, either spread evenly or delivered in blocks. The model always learns from real text. Learning from generated text, compared with simply reading it, adds 0.0694 nats of NLL with evenly spaced real text and 0.7155 nats with blocks. Real-text training therefore does not necessarily eliminate the harm, and its timing matters. Unfiltered Closed Loop is a baseline, not our entire study.

W1.2. "paper just designed a set of experiment to reveal this already known problem"

Thanks for the feedback. We respectfully disagree. Earlier studies examined collapse from repeated training on synthetic data [1–3]. Recent work has developed test-time learning in other settings [4–9]. Our contribution is not another observation of collapse. We show how retained updates harm prediction and how feedback amplifies that harm.

First, generated text alone does not explain the degradation. In Fixed Generation, the learner still trains on generated text, but its updates cannot affect future generations. Excess NLL falls from 2.8308 to 0.0707 nats, a 97.5% reduction (Table 8). Removing this feedback eliminates nearly all the degradation in this setting, although the learner still trains on generated text.

Second, an update can fit its training text better while making other predictions worse. In Recorded Replay, learning from the same recorded text adds 2.268 nats of excess NLL compared with reading it without retaining updates (Table 4). Single-update experiments show this tradeoff directly: an update can lower loss on generated text while raising loss on independent real text (Table 5). Greater conflict between the two gradients is associated with greater harm (Table 6).

Third, repetition is a warning sign, but it does not tell us which updates to retain. aTTT downweights repeated tokens [10]. We replay recorded passages on fixed receiver models and measure how much learning from each passage changes real-text NLL. Mean write cost rises from 0.0439 to 0.7512 nats, while the median falls from 0.0422 to 0.0064 (Figure 4; Table 19). Large write costs are concentrated in a few passages. Q2 shows that similarly repetitive passages can have very different write costs.

These findings motivate Settlement: it checks an update’s effect on independent real-text loss before retaining it. PWUP likewise recognizes “a more specific explanation of the failure than parameter drift or poor generated text alone,” while 9HjW finds our interventions “more persuasive than correlations between drift and performance.”

W1.3. "generated chunks cannot help by construction".

You have raised an important point. However, our new experiments show that selected generated updates can improve prediction on held-out real text. At 3B, Settlement and No-Generated have access to the same real-text passages and run the same number of model calls. Settlement selects generated updates using same-book validation; No-Generated discards every generated update. Settlement lowers NLL by 0.001337 nats/token across 16 PG19 books (Appendix H.1). On 32 additional books, the reduction is 0.001098 nats/token.

These results show that self-writing can improve independent real-text prediction. Q4 also tests recall of self-generated answers after they are removed from context.

[1] Shumailov et al. AI Models Collapse When Trained on Recursively Generated Data. Nature, 2024.

[2] Alemohammad et al. Self-Consuming Generative Models Go MAD. ICLR, 2024.

[3] Gerstgrasser et al. Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data. COLM, 2024.

[4] Sun et al. Learning to (Learn at Test Time): RNNs with Expressive Hidden States. ICML, 2025.

[5] Akyürek et al. The Surprising Effectiveness of Test-Time Training for Few-Shot Learning. ICML, 2025.

[6] Hu et al. Test-Time Learning for Large Language Models. ICML, 2025.

[7] Tandon et al. End-to-End Test-Time Training for Long Context. arXiv, 2025.

[8] Bansal et al. Let's (not) just put things in Context: Test-time Training for Long-context LLMs. ICLR, 2026.

[9] Zhu et al. Self-Guided Test-Time Training for Long-Context LLMs. arXiv, 2026.

AuthorsOpenReview ↗

Response to Reviewer XeCQ: W2

W2. Settlement: Motivation, Design, and Practical Use

Per-update validation is not common in continuous test-time training for language models. Earlier methods address unstable adaptation through sample selection (EATA [11]), optimisation and recovery (SAR [12]), or weight restoration (CoTTA [13]). These methods do not use independent real-text prediction to decide whether to retain each proposed update. Concurrent work also explores selective commitment: VANE [14] uses future visual observations, while CGPI [15] uses navigation-specific interaction evidence. These methods operate in different settings and do not study persistent feedback from self-generated language.

Settlement goes beyond a standard accept/reject rule by validating the accumulated model state before committing an update. Settlement is motivated by a specific failure mechanism established in our experiments. An update can improve the model's fit to generated text while hurting its prediction on real text. More importantly, two updates can each pass validation but become harmful when combined.

Table 26 tests this problem directly. Joint Commit checks two updates separately. It tests W+δ1W+\delta_1 and W+δ2W+\delta_2, but then commits W+δ1+δ2W+\delta_1+\delta_2 without testing the combined model. In our additional analysis, we examined 2,226 pairs of updates that passed validation individually. In 238 pairs (10.7%), adding the second update increased validation loss relative to keeping the first alone. This shows that an update that looks safe on its own may no longer help after another update has been accepted.

Settlement avoids this problem by checking the model after combining the proposed update with all previously accepted updates. If the first update is kept, the second is tested on top of it. The second update is accepted only if the resulting model does not increase validation loss relative to the current model. We also compared this approach with a standard fixed-reference validation baseline. Both methods apply updates sequentially and use the same scheduled validation passages and evaluation budget. Fixed-Reference accepts an update as long as the resulting model remains better than the original model. Sequential Validation instead requires it to be no worse than the model currently in use.

PolicyReal-text B ↑
Joint Commit−1.093
Scheduled Fixed-Reference−0.027
Sequential Validation+0.019
Settlement+0.038

B measures improvement in real-text NLL change relative to No Updates. Joint Commit performs poorly despite validating individual updates. Under the same scheduled validation protocol, Sequential improves B over Fixed-Reference by 0.046 nats, with positive differences across all three offset groups. This matters because an update can still outperform the original model while making the current model worse.

The claim that Settlement requires deploying an update before evaluating it is incorrect. Algorithm 1 evaluates the accepted state and the candidate state on the same validation text, starting from the same restored context. Both evaluations are read-only. The candidate enters the live model only after passing validation, so evaluating it does not require exposing subsequent live generation to an unvalidated update.

Settlement is not restricted to settings with identifiable real text. Real-text loss provides the validation signal in our language-model experiments; in agent workflows, candidate updates can be evaluated using task reward or success. Our submission includes WebShop, where Settlement selects updates using validation reward and improves exact task success over Writes Off. We also added ALFWorld results, where Settlement uses task-success validation.

ExperimentWrites OffSettlementDifference
WebShop exact success16.00%18.53%+2.53 pp
ALFWorld unseen-task success, rank (8)86.8%91.0%+4.2 pp

In WebShop, Settlement retains 10 of 180 generated updates. Its exact-success improvement over Writes Off is 2.53 percentage points. The ALFWorld results are means over three paired seeds; the archived records do not establish separation between validation and outcome evaluation, so we treat them as supporting observations.

[10] Wang, et al. "No Time Like the Present: Agentic Test-Time Training for LLM Agents." arXiv, 2026.

[11] Niu et al. Efficient Test-Time Model Adaptation without Forgetting. ICML, 2022.

[12] Niu et al. Towards Stable Test-Time Adaptation in Dynamic Wild World. ICLR, 2023.

[13] Wang et al. Continual Test-Time Domain Adaptation. CVPR, 2022.

[14] Ji et al. VANE: Reliable Test-Time Training for Vision-Language-Action Models via Future Visual Representation Prediction. arXiv, 2026.

[15] Li et al. Credit-Guided Policy Improvement for Test-time Adaptive Vision-Language Navigation. arXiv, 2026.

AuthorsOpenReview ↗

Response to Reviewer XeCQ: Q1 and Q2

Q1. Effect smaller at 3B

The write effect is consistently positive, but its magnitude differs substantially across models. Our claim concerns the presence of the effect, not a uniform effect size across scales.

In Table 4, retaining updates adds 2.268 nats of excess NLL at 125M, compared with 0.1927 nats [0.1043, 0.3272] at 3B. Both effects are positive, although the latter is approximately twelve times smaller.

Importantly, the results do not follow a monotonic scaling trend. Table 1 reports excess NLL increases of 3.01, 6.00 and 0.40 nats at 125M, 760M and 3B, respectively. The smaller models are trained reproductions, whereas 3B uses a released checkpoint with a different training lineage. Model size and training history are therefore not independently controlled, so these experiments cannot establish why the effect is smaller at 3B.

Our conclusion is consequently narrower and well supported: retaining self-generated updates can harm independent prediction at multiple scales, but the severity is model-dependent. We will revise the statement to read: “Retaining generated-text updates adds a positive replay cost at both 125M and 3B, although the effect is substantially smaller at 3B.”

Q2. Qualitative inspection of the generated training text

Thank you for this suggestion. We inspected the actual Closed Loop generations and compared them with Fixed Generation, where the learner continues to update but the generator remains frozen. The following excerpts come from the stored 125M outputs (Book 3, seed 1) at corresponding stream positions.

ChunkClosed LoopDistinct-2 / Repeated-4Fixed GenerationDistinct-2 / Repeated-4
1"I never once heard anybody call it any other of his Perrieries..."0.93 / 0.00"an ingenuity which could understand so much of that genial little intelligence..."0.89 / 0.00
12"the gentleman--the author, the colonel--the colonel!"0.64 / 0.07"Harder--harder than when I was under your cares. And, you say..."0.85 / 0.01
30"I could not write--I could not write--I could not write..."0.01 / 0.99"Mr. Giles was going home to London, and even could stand the sight of her fancy-paper..."0.89 / 0.01
105"I could not write--I could not write--I could not write..."0.01 / 0.99"an extended plunge,--a retreat,--a slip step,--jumping crocodiles with a flute..."0.91 / 0.00

Closed Loop progressively degenerates into repeated short phrases, while Fixed Generation maintains lexical diversity. By chunk 30, the Closed Loop text has almost entirely collapsed. Across all 30 inspected Closed Loop streams (six books, five seeds), 29 end in exact cycles of at most 25 tokens. This agrees with Table 8, where final repeated-4 reaches 0.9902 for Closed Loop, compared with 0.0082 for Fixed Generation.

We also examined the actual recordings used in the fixed-receiver Replay experiment and matched the text to its measured write cost. Two passages from the same source trajectory provide a particularly clear contrast:

Recorded sourceGenerated textDistinct-2 / Repeated-4Replay write cost (nats)
Book 2, seed 7, stage 81"to have anything to have done with you about it..."0.02 / 0.98+0.02
Book 2, seed 7, stage 97"ononononononon..."0.00 / 1.00+5.38 to +6.97

These examples come from a separate recording used for Table 19. Diversity scores describe the displayed chunks; write costs are measured over the corresponding 8,192-token passages, with ranges across three receiver configurations.

Despite similarly extreme repetition, the two passages have markedly different write costs. Table 19 confirms that damage is concentrated rather than uniform: the mean passage-level write cost rises from 0.0439 to 0.7512 nats, while the median falls from 0.0422 to 0.0064. At the final measured position, only 12 of 72 comparisons exceed 0.5 nats, all originating from just four of the 24 recorded source sequences. A small subset of source trajectories therefore accounts for the large passage-level losses.

Why does replaying fixed text still cause harm? Table 4 isolates the effect of retaining updates from identical recorded text. Reading alone increases real-text NLL by 1.606 nats relative to the receiver control; reading and writing increases it by 3.874 nats. The additional 2.268 nats comes from persistent weight updates. Table 5 provides complementary evidence of a transfer conflict: after Closed Loop adaptation, a single update lowers source-chunk NLL by 0.0354 nats on average while increasing independent real-text NLL by 0.0381 nats.

The key finding is that repetition alone does not identify harmful updates. The qualitative examples show how Closed Loop changes its training text. Replay and single-update experiments show that learning from this text can improve source fit while harming independent prediction.

AuthorsOpenReview ↗

Response to Reviewer XeCQ: Q3 and Q4

Q3. Presentation order

Thank you for this helpful suggestion. Sections 4 and 5 establish that self-generated feedback degrades prediction and explain why an update can fit its source while harming new text. These findings motivate the simple validation rule introduced as Settlement in Section 6: retain an update only after checking its effect on independent text.

The WebShop comparison in Section 5 was the only place in Sections 4–5 that reported a Settlement result before its definition. We have moved this result to Section 6.2, page 8, lines 423–426, after the method definition and real-text results, and before the comparison with Source Masking. The revised passage compares Settlement with Closed Loop and Fixed Generation and explains how selective retention supports task performance. These changes are highlighted in blue. We thank the reviewer again for helping us improve the clarity of the presentation.

Q4. Can self-writing provide a benefit?

Yes. Our new experiments show that self-writing can help a model remember its own generated answers beyond the available context, as well as improve language modelling and downstream task performance.

1. Recalling self-generated answers.

We tested the reviewer's suggestion directly using 64 questions from 2WikiMultihopQA, two generation seeds, and a frozen Qwen3.8-27B model [16] with trainable rank-16 LoRA adapters. The model first generated answers with the supporting passages in context. We then trained the adapters on these answers, conditioned on the corresponding questions and record IDs. The training schedule was chosen on separate calibration examples and fixed before testing (16 Adam steps per four answers, learning rate (10−4))(10^{-4})).

All conditions then processed the same 32K tokens of real text and eight real-text updates. For evaluation, we cleared the KV cache and asked the original questions again, providing neither the supporting passages nor the generated answers.

ConditionQA F1 ↑Exact recall ↑
No generated-answer update17.3718.75%
Write-All61.5693.75%
Shuffled-answer update16.29—

Self-writing raises QA F1 by 44.19 points (95% paired bootstrap CI [29.40, 57.89]), while exact recall of the model's own answers increases from 18.75% to 93.75%. When we shuffle the same generated answers across questions, keeping the training budget unchanged, F1 drops to 16.29. The benefit therefore comes from learning the question–answer associations, not simply from additional exposure to generated text.

2. What happens with a single update? We also examined a single-update setting with the 3B TTT-E2E model, using one update on a generated person–location record followed by 64K tokens of real text, well beyond its 8K attention window.

ConditionAnswer NLL ↓
No update3.9241
Correct record update3.7082
Permuted record update3.7070

Writing the correct record reduces answer NLL by 0.2159 nats (95% CI [0.1809, 0.2534]). However, permuting the person–location associations produces almost the same improvement. This suggests that a single update can improve answer likelihood without reliably preserving the underlying associations. The Qwen experiment offers stronger evidence of recall, particularly through its shuffled-answer control.

3. Benefits beyond recall.

We further find that self-generated updates can improve independent language modeling. In our new 3B PG19 experiment, covering 16 books and three seeds per book, Settlement retains 154/384 generated updates and improves real-text NLL over Gen-Off by 0.001337 nats/token (95% CI [0.000255, 0.002466]). This improvement remains positive after correction for three primary comparisons.

We see similar benefits in agent tasks:

SettingWrites OffSettlementRetained updates
WebShop exact success ↑16.00%18.53%10/180
WebShop mean reward ↑0.33090.382310/180
ALFWorld, LoRA rank 8 ↑86.8%91.0%12/36
ALFWorld, LoRA rank 16 ↑86.8%90.0%7/36

The agent results are encouraging, although the WebShop comparison uses five seeds and ALFWorld uses three paired seeds, so the precision of these gains remains limited.

The generated-answer experiment directly addresses the suggested setting: the model remembers its generated answers better after training on them, even after processing 32K tokens of other text and clearing its context. TShuffling the answers across questions removes the benefit, indicating that the model learns which answer belongs to which question. In a separate 3B experiment, retaining selected generated updates also lowers NLL on held-out real text compared with rejecting all generated updates.

[16] Qwen Team. Qwen3.8-27B. Hugging Face model card, 2026. https://huggingface.co/Qwen/Qwen3.8-27B

Reviewer_9HjWOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies persistent test-time updates from a model's own generated text over long streams. It reports deterioration on independent real-text prediction across TTT-E2E scales and an ordinary Adam adaptation setting. A frozen generator, fixed-text replay, and paired single-update branches distinguish the role of feedback, retained writes, and local source-fit versus transfer conflict. A validation-based commitment rule, Settlement, tests proposed states on separate real text before retaining them. Further experiments examine real-text cadence, sampling, source variability, corrupted source labels, and a WebShop extension.

Critical strengths
  1. The empirical controls are well chosen. Fixed Generation substantially reduces damage while the learner continues updating on generated text; recorded replay holds the text fixed while varying retained writes; single-update branches compare source-fit and transfer from the same starting state. Evaluation state restoration, replicated effect directions and clustered uncertainty make these more persuasive than correlations between drift and performance.
  2. The boundary experiments are informative and appropriately qualified. The paper tests full-support decoding, real-text cadence, reset and update-dose alternatives, an ordinary Adam adaptation, and heterogeneous source effects. It does not claim monotonic scaling with model size. Settlement's all-real experiment also measures retained adaptation benefit, addressing the possibility that the policy universally rejects learning.
Critical weaknesses
  1. Settlement's operational evidence is not sufficiently specified. Algorithm 1 provides a clear sequential commitment rule and separates admission text from outcome evaluation, but the validation bank's initialization, size, selection and reuse remain unclear. This is especially consequential in Appendix F: observed labels are used to locate real validation text, yet at 100% label error the method admits 120/120 truly real candidates and 0/219 generated candidates. The precise source of validation evidence must be specified to interpret robustness to corrupted labels. This is a protocol question, not evidence of leakage. In the main suites, only 22/936 and 18/312 generated candidates are admitted, so the results chiefly establish damage prevention, not useful generated-text learning beyond Writes Off or reliable source masking.
  2. The agent extension supports a metric-specific conclusion. In Table 28, Closed Loop has the lowest exact success but the highest average reward: 0.4356 versus 0.3724 for Fixed Generation and 0.3823 for Settlement. The text correctly emphasizes exact success, but the opposing ordering matters because Settlement itself validates reward. The result should be interpreted as a tradeoff rather than broad preservation of agent capability. Paired uncertainty for the policy differences and a clearer account of the validation-goal protocol would help assess this extension.
Questions for authors
  1. For Tables 23-27, what initializes the validation bank, how is text selected/reused, and what actual validation tokens are available at 100% source-label error? Does an independently trusted prefix or bank remain intact?
  2. Under matched external-text and compute budgets, when does Settlement improve over Writes Off or source masking by admitting useful generated updates? How sensitive is it to validation-bank reuse or distribution shift?
  3. Why do WebShop reward and exact success rank the policies differently? Please define the reported uncertainty and give paired effects for both metrics, along with validation-goal reuse details.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 9HjW: W1 and W2

Thank you for your constructive comments and suggestions. We appreciate your recognition that our “empirical controls are well chosen” and our “boundary experiments are informative and appropriately qualified,” as well as your positive comments that our evaluation is “more persuasive than correlations between drift and performance.”


W1.1. Additional details on Settlement's validation protocol

Thank you for your comment.

Settlement does not require accurate source labels to decide which updates to retain. It evaluates pending updates using trusted real-text validation evidence, while observed labels determine when these checks occur. In Tables 23–24, Settlement checks pending updates on newly arriving real text before learning from it. When no new real text is available, it cycles through eight real-text chunks saved from warm-up (Appendix E.6, p. 23, lines 1198–1207).

For Table 27, we reserve eight validation passages from each receiver’s PG19 book before corrupting the candidate labels. Each passage has 1,024 prediction targets per sequence. The observed labels determine when Settlement checks pending updates. Each check uses the next reserved passage in the schedule, which is available in advance and can be reused across checks until the stream reaches it. These passages stay fixed across error rates. We measure final NLL on seven other passages from the same book and keep both sets out of training. At 100% label error, every candidate received a decision. Counting by true source, Settlement accepted 120/120 real-text updates and 0/219 generated updates (Appendix F.1, p. 24, lines 1245–1254). Q1 gives the validation and reuse protocols for Tables 23–27.

W1.2. When Settlement helps beyond damage prevention

Thank you for pointing this out.

In the original experiments, Settlement reduced Closed Loop degradation while retaining 22 of 936 generated updates at 125M and 18 of 312 at 760M (Table 23, p. 21, lines 1097–1107).

Our additional experiments show that selectively retaining generated updates can improve held-out real-text prediction, rather than merely preventing degradation.

At 3B, Settlement lowered held-out NLL relative to No-Generated and gated source replay using same-book validation. We repeated the comparison on 32 additional PG19 books and again found lower NLL than No-Generated. The 3B study generated and validated updates differently from the original Closed Loop experiments. Q2 reports the matched controls and validation sensitivity.

In a separate online ALFWorld experiment [1], Settlement retained 12 of 36 candidate update blocks across three paired seeds at a LoRA rank of 8. Mean success on unseen tasks was 91.0% with Settlement and 86.8% with Writes Off.

In the original 125M mixed-stream experiment, Settlement’s point estimate for final NLL was lower at 21.2% label error (3.8286 vs. 3.8377 for Source Masking). The paired interval for the difference first excluded zero at 38.9% label error (3.8215 vs. 3.8584) and continued to exclude zero at every higher rate tested (Table 27, p. 23, lines 1230–1240).

In the separate 125M fixed-bank experiment, Source Masking had lower NLL with correct labels (3.803 vs. 3.902). When every candidate label was flipped and the validation bank stayed fixed, Settlement had lower NLL (3.902 vs. 4.268) (Appendix F.1, p. 24, lines 1256–1275). Q2 reports the matched-budget comparisons and the effects of validation source, bank size, and reuse.

[1] Shridhar et al. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. ICLR, 2021.

W2. Additional analysis of reward and exact success in the agent extension

WebShop reward credits partial task completion, while exact success requires all requested specifications to be satisfied. In four of five Closed Loop runs, the agent bought an item in 99–100% of episodes and fully satisfied the request in only 12–14.7%. Those partial matches help explain why Closed Loop had the highest mean reward (0.4356) and the lowest exact success (10.53%). Settlement selects updates based on reward improvements on ten reused validation goals, which does not guarantee higher reward on the separate test goals.

Across the five paired runs, Settlement’s exact success rate was 8.00 percentage points higher than Closed Loop’s and 2.53 points higher than Writes Off’s. The corresponding mean reward differences were −0.0533 and +0.0514, with intervals spanning zero. Q3 reports the paired intervals for both metrics.

Settlement reused ten validation goals across all 36 checks in each run. Removing the one overlapping evaluation goal left the estimates nearly unchanged. With five additional goals matched to training tasks removed, the estimated exact-success gain over Writes Off was +1.39 percentage points (95% CI [−1.05, 3.83]). Appendix G.1 reports the paired comparisons, validation schedule, and overlap analysis (p. 25, lines 1296–1319).

AuthorsOpenReview ↗

Response to Reviewer 9HjW: Q1

Q1. Validation protocols and the 100% candidate-label error result

Tables 23–24. When new real text arrives, Settlement checks pending updates on it before learning from it. Otherwise, it cycles through the first eight real-text chunks from warm-up, with 1,024 prediction targets per chunk per sequence. This bank stays fixed. We measure final NLL on separate text (Appendix E.6, p. 23, lines 1198–1207).

Table 25. Every candidate update comes from real text. Settlement checks each update on the next reserved passage from the same PG19 book. The passage is available in advance and is reused at each check until the stream reaches its scheduled position. We keep the validation passages out of training and measure adaptation on separate passages (Appendix E.6, p. 23, lines 1209–1210).

Table 26. Joint Commit and Sequential use the eight validation passages at chunks 15, 31, …, 127. We reserve the seven passages at chunks 23, 39, …, 119 for evaluation. Settlement checks pending updates on the next real-text chunk outside those two sets, before learning from it. Ordinary writing applies each update immediately (Appendix E.5, p. 22, lines 1144–1155).

Table 27. We reserve the validation and evaluation passages from each receiver’s PG19 book before corrupting any candidate labels. Both sets stay fixed across error rates and are kept out of training. An update labeled external triggers a check if a validation passage is still ahead in the schedule. Settlement scores all pending updates, including the one that triggered the check, on the next reserved passage. The experiment provides that passage in advance, and several checks can use it before the stream reaches it. Checks stop when the schedule runs out of passages. Any updates still queued at the end receive no admission decision.

Flipping a label can change when an update is checked and which passage it is scored on. Even at 100% label error, all eight reserved real-text validation passages (8,192 distinct prediction targets per sequence) remain available independently of the corrupted candidate labels. At this error rate, all 339 candidates received an accept or reject decision. Counting by true source, Settlement accepted 120/120 real-text updates and 0/219 generated updates (Appendix F.1, p. 24, lines 1245–1254).

PurposeChunk positionsPrediction targets per sequence
Validation15, 31, …, 1278 × 1,024
Final NLL23, 39, …, 1197 × 1,024

For the separate fixed-bank experiment, we use one passage from each of four PG19 books outside the receiver and evaluation pool. We reuse the same 4,096 prediction targets at all 20 checks and keep the check times fixed across error rates (Appendix F.1, p. 24, lines 1256–1275).

AuthorsOpenReview ↗

Response to Reviewer 9HjW: Q2

Q2. Useful generated updates, matched budgets, and validation sensitivity

Thank you for your comment.

At 3B, Settlement used validation passages from the same book and achieved lower held-out NLL than No-Generated and gated source replay under matched model calls. In the 16-book PG19 experiment, Settlement kept 154 of 384 proposals, with three generation seeds per book. In a separate replication on 32 additional books, Settlement again had lower NLL than No-Generated, by 0.001098 nat/token (95% CI [0.000257, 0.001974]). In each study, the interval for the gain over No-Generated stayed above zero after adjustment for the three primary comparisons.

We matched access to external text and the number and types of forward and backward passes. All policies started from the same state after learning an 8K real-text prefix. They shared eight generated notes and two validation passages from the same book, each with 1,024 prediction targets. Each policy ran eight candidate backward passes, 16 validation forward passes, 16 audit forward passes, and 32 forward chunks for the two readouts. No-Generated discarded every note update. Gated source replay used prefix text for the updates and applied the same validation rule. Direct bank training used the eight update slots to learn from the validation passages. We measured final NLL on a separate continuation with a fresh attention cache. The table reports paired 95% intervals for the 16-book experiment, resampling books after averaging seeds within each book (Appendix H.1, p. 25, lines 1324–1349).

ControlControl NLL − Settlement NLL (nat/token)95% CI
No-Generated+0.001337[0.000255, 0.002466]
Gated source replay+0.001903[0.000859, 0.003073]
All-Generated+0.001605[0.000651, 0.002595]
Direct bank training−0.003946[−0.005401, −0.002727]

On the original 16 books, the adjusted 95% interval for the gain over No-Generated was [0.000033, 0.002725]. Direct bank training achieved lower NLL than Settlement on both sets of books.

Source Masking and Writes Off. In fixed replay, Settlement lowered final NLL relative to Writes Off at 125M by 0.11190 nat/token (95% CI [0.05778, 0.18013]). The 3B interval included zero. With every label flipped and the validation text intact, Settlement had lower final NLL than Source Masking at both scales. Source Masking led at both scales with correct labels. At a 40% label-flip probability, it still led at 3B, and the 125M comparison was inconclusive.

All policies in fixed replay received the same text and made the same model calls. Settlement and Source Masking committed at the same scheduled checks, with fresh PG19 validation text available at each check. In the original mixed-stream experiment (Table 27), the observed labels determined when Settlement checked pending updates against reserved PG19 passages. Against No-Generated in fixed replay, the 125M gain was small at the endpoint and negative when averaged over the stream. At 3B, No-Generated had lower final NLL (Appendix H.4, Table 35, p. 29, lines 1512–1543).

Validation reuse and distribution. In the replication on 32 books, we compared two orders for the checks, using each validation passage four times in both schedules. Grouping the checks instead of alternating the passages changed the gain over No-Generated by +0.000014 nat/token on average across banks (adjusted interval [−0.000110, 0.000129]). Replacing same-book passages with Wikipedia reduced the gain by 0.001762 nat/token on average across schedules (adjusted interval [0.000978, 0.002643]). With another PG19 book or Wikipedia as the bank, the intervals for the gain over No-Generated included zero. Appendix H.2 reports the corresponding comparisons for the original 16 books (p. 26, lines 1350–1371).

At 125M, increasing the fixed PG19 bank from 2K to 32K targets per check narrowed the gap to No-Generated by 0.000181 nat/token. Refreshing the 8K PG19 bank narrowed it by 0.000138 nat/token. No-Generated still had lower estimated NLL in all nine conditions. With the fixed 8K PG19 bank, Settlement had lower NLL than the real-text control, which used 92.6% of the matrix-operation budget for generation, candidate updates, and validation to train on the bank (Appendix H.3, p. 26, lines 1373–1403).

With same-book validation, Settlement improved on No-Generated at 3B. Direct bank training achieved lower NLL, and correct source labels favored Source Masking in fixed replay. At four uses per passage, the effect of check order was uncertain. Replacing same-book validation with Wikipedia reduced the gain.

Reviewer_PWUPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies test-time training when a model learns from its own outputs and those updates affect subsequent generations. Through frozen-generation controls, recorded replay, and matched single-update interventions, it separates feedback effects from attention contamination and persistent weight updates. It also proposes Settlement, which accepts an update only when the resulting state passes an independent validation check. The main contribution is an empirical explanation of how improved fit to generated text can coexist with worse prediction of independent real text.

Critical strengths

S1: The intervention design provides useful evidence beyond observing performance degradation. Fixed Generation interrupts feedback while retaining adaptation, and recorded replay holds the text fixed when comparing reading with writing. Together, these comparisons support a more specific explanation of the failure than parameter drift or poor generated text alone.

S2: The distinction between source fit and transfer is informative. The single-update experiments and concentrated failure trajectories show why optimizing the current example is an unreliable criterion for retaining an update. Results across TTT-E2E scales and conventional weight updates make this finding relevant beyond one architecture. This diagnostic contribution is the main reason I lean toward acceptance.

Critical weaknesses

W1: Settlement's practical advantage over source masking is less established than the diagnostic contribution. Figure 5 shows that it retains very few generated-text updates and performs similarly to source masking when provenance labels are correct. Its advantage emerges under corrupted labels, while it still requires identifiable real text for validation. This limits the evidence that it enables useful learning from generated outputs, rather than primarily preventing harmful writes.

Questions for authors

Q1: Under what deployment conditions would reliable real-text validation be available while source provenance is sufficiently uncertain to favor Settlement over source masking?

Q2: Within the reported experiments, is there a setting where accepting generated-text updates provides a clear benefit over retaining only real-text updates?

2 comments
AuthorsOpenReview ↗

Response to Reviewer PWUP: W1 and Q1

Thank you for your thorough review, positive comments, and insightful suggestions. We appreciate your recognition that the "intervention design provides useful evidence beyond observing performance degradation,” that “the distinction between source fit and transfer is informative,” and that our results provide a “more specific explanation of the failure than parameter drift or poor generated text alone.”


Regarding W1. Practical advantage over source masking

Thank you for your comments. The key difference is that source masking relies on where an update comes from, while Settlement evaluates whether keeping it actually helps. Our new experiments show why this distinction matters.

First, source labels can be unreliable, where training trajectories combine external observations and model-generated content. In our new 125M experiment, Source Masking's NLL rises from 3.803 to 4.268 as label errors reach 100%, while Settlement remains at 3.902 using a fixed, trusted validation bank.

Second, knowing the source does not tell us whether an update is useful. In our new experiment on ALFWorld [1], all candidate updates are generated, yet some are worth retaining. Settlement selects 12 of 36 updates and achieves 91.0% unseen-task success, compared with 86.8% when all updates are rejected (LoRA rank 8). This shows that selected generated updates can improve performance, rather than simply being a source of harm.

These results support our central argument: an update should be judged by how well it transfers to independent data, not simply by its source. Settlement requires reliable validation evidence and additional compute, but offers a useful alternative when source labels are unreliable or insufficient to guide update selection. We provide further details in Q1 and Q2.

[1] Shridhar et al. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. ICLR, 2021.


Regarding Q1. Trusted validation with uncertain provenance

Settlement is intended for settings where an operator can maintain a small, trusted reference set but cannot reliably identify the origin of every incoming update. For example, a curated reference bank may be set aside before adaptation, while subsequent documents are edited, combined or relabelled. Trust in the reference text does not require trust in the provenance labels attached to candidate updates.

We evaluated this distinction in a new 125M experiment. We set aside four real-text passages with 4,096 prediction targets before corrupting the candidate-source labels. We reuse the same bank at all 20 validation checks and keep it out of training and final scoring. Appendix F.1 reports results at all eight error rates across five paired seeds and four receiver books per seed (p. 24, lines 1245–1275).

Candidate-label errorSource Masking NLL ↓Settlement NLL ↓
0%3.8033.902
61.9%3.8893.902
87.6%4.0803.902
100%4.2683.902

With correct labels, Source Masking performs better. As labels become unreliable, its performance deteriorates, whereas Settlement remains stable because the validation evidence is unchanged. At 100% label error, Settlement achieves 0.366 nats lower NLL. In this experiment, Settlement's validation data and admission schedule are fixed independently of source labels, so corrupting those labels does not affect its decisions.

ALFWorld offers a complementary setting in which provenance is uninformative: generated action updates share the same source category, so source identity alone cannot determine which updates are useful. Using independent task-success validation, Settlement reaches 91.0% and 90.0% unseen-task success at LoRA ranks 8 and 16, compared with 86.8% for Writes Off. Across three runs per LoRA rank, Settlement retained 12 of 36 candidate update blocks at rank 8 and 7 of 36 at rank 16 (Appendix I, Table 36).

The relevant deployment condition is therefore specific: independent, task-relevant validation must remain trustworthy even when candidate provenance is unreliable or uninformative. A trusted validation set gives Settlement a way to choose which updates to keep when source labels are unreliable or all candidates share the same source. In ALFWorld, task success on the validation games lets Settlement distinguish between blocks of generated updates that a source label would treat alike.

AuthorsOpenReview ↗

Response to Reviewer PWUP: Q2

Q2. Do generated-text admissions improve over real-only learning?

Yes. Our new 3B experiment provides direct evidence that selectively retaining generated updates can improve real-text prediction beyond real-only learning. The improvement is statistically supported, showing that Settlement can do more than prevent harmful updates.

1. New evidence from language-model experiments.

We evaluated 16 unseen PG19 books, with three generation seeds per book (48 runs). All methods start from the same checkpoint after learning an identical 8K real-text prefix, and face the same eight proposed updates. We compare Settlement against three alternatives: rejecting all generated updates (Gen-Off), replaying the original real source text under the same validation rule, and accepting every generated update (All-Generated).

Settlement uses two independent 1,024-token real-text passages from each book for validation, each reused four times. The protocol includes eight candidate backward passes, 16 validation forward passes, and 16 additional forward passes for independent evaluation. Final NLL is measured on separate real text with a fresh attention state. Confidence intervals are paired across books, averaging the three seeds for each book.

Settlement retains 154/384 generated updates and achieves lower real-text NLL than all three alternatives:

ControlSettlement’s NLL gain (nat/token)95% CI
Gen-Off (discard all generated updates)+0.001337[0.000255, 0.002466]
Gated real-source replay+0.001903[0.000859, 0.003073]
All-Generated (keep all generated updates)+0.001605[0.000651, 0.002595]

Crucially, the improvement over Gen-Off remains positive after correcting for all three primary comparisons (adjusted 95% CI [0.000033, 0.002725]). The table reports unadjusted intervals.

2. Supporting evidence from agent tasks.

Our WebShop and new ALFWorld experiments show similar benefits in task performance.

ExperimentWrites OffSettlement
WebShop exact success16.00%18.53%
WebShop mean reward0.33090.3823
ALFWorld unseen success (LoRA rank 8)86.8%91.0%
ALFWorld unseen success (LoRA rank 16)86.8%90.0%

In WebShop, Settlement retains 10/180 generated updates and improves exact success by 2.53 percentage points (paired 95% t interval [0.02, 5.04]). However, the exact sign-flip test gives p = 0.0625, and the reward improvement remains statistically uncertain. ALFWorld shows consistent gains across two LoRA ranks, although these are averages over three paired seeds without confidence intervals.

3. Why selective updates matter.

Table 25 further shows that even real-text updates are not always beneficial. When all candidates come from real text, Settlement retains 89/113 updates and improves adaptation benefit over accepting all updates by 0.0083 nats [0.0020, 0.0161]. This supports our central finding: the value of an update depends on its effect on future prediction, not simply whether its source is real or generated.

Taken together, these results establish a measurable benefit from selectively retaining generated updates. The 3B experiment demonstrates improved real-text prediction over two tested real-only policies under the specified update and validation budget. WebShop and ALFWorld provide complementary evidence from downstream tasks.

Comments on the paper

2 comments

Dear Reviewers, kindly read the authors' response and engage in the discussion to resolve any remaining issues prior to the Nov 18 AoE deadline. Thanks!

Dear reviewers, a quick reminder that the author–reviewer discussion closes on November 18 (AoE).

PWUP, you noted Settlement’s limited advantage over source masking when labels are accurate and questioned whether reliable real-text validation is available when provenance is uncertain. You also asked whether retaining generated-text updates helps beyond retaining only real-text updates.

Do the authors’ responses adequately address these concerns? If any remain, please let us know before the discussion deadline.

9HjW, thank you for following up. Please feel free to share any additional comments before the discussion deadline.

XeCQ, thank you for following up. Please feel free to share any additional comments before the discussion deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.