acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Self Improvement by Scaling Unsupervised World Internalization

Abstract

Agents accumulate experience as they interact with an environment, but current systems typically reuse this experience through context or external memory while keeping the underlying model fixed. While flexible, this constrains the form of learning models can do especially under novel and complex environments. We ask whether weight updates can instead improve the model's ability to solve tasks by learning directly from its own experience, without feedback on whther its attempts succeed. We study world internalization, a training approach that converts information acquired through unsupervised exploration into parameter updates. World internalization trains on insights and summaries extracted from exploratory trajectories, together with a small fraction of trajectories that teach the model to integrate newly acquired knowledge with existing cognitive priors for reasoning and action. This enables the model to internalize environmental knowledge through continued training analogous to pretraining and midtraining while preserving its ability to use that knowledge for downstream tasks. We evaluate world internalization across three domains with distinct forms of environmental knowledge and interaction: Harvey's Legal Agent Benchmark, StudyBench, and Equational Theories. We observe that internalization reliably improves downstream task success despite receiving no supervision on improving task solving. Moreover, performance scales nearly log-linearly as internalization data increases from 10M to 100M tokens. In all, our results show that scaling the internalization of autonomous experience can enable self improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-time training, continual learning, LLM, Mid-training

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vE6VOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies "world internalization": converting an agent's exploration experience in an environment into weight updates without downstream tasks, rewards, or success feedback. The proposed U-WIN uses the frozen base model to generate two types of data: Type I notes that summarize what was learned during exploration, used to acquire environmental knowledge, and Type II solution trajectories generated with relevant notes in context and trained with the notes removed, used to learn how to apply that knowledge. Across legal-document, codebase, and mathematical environments, downstream and closed-book performance generally improve as internalization data scales from 1M to 100M tokens, and Type II data helps preserve the model's agentic behavior.

Critical strengths
  • The setting is fully unsupervised (no downstream questions, no rewards, no trajectory filtering), and the paper clearly distinguishes it from test-time training, reinforcement learning, and synthetic continued pretraining.
  • All training data is generated by the frozen base model without a stronger teacher, which rules out distillation from a stronger model as an explanation and makes the self-improvement claim more credible.
  • The paper shows a scaling trend from 1M to 100M tokens and candidly acknowledges that it does not perform genuine continual learning.
Critical weaknesses
  • The tasks emphasize memory reinforcement rather than generalization, and the method may induce hallucination. The closed-book and recall tests probe content the model has already seen in its notes, and the Equational Theories test items may be derivable from known implications; the gains in the Fig. 5 examples come from directly recalling specific matter IDs and file paths. In addition, removing the notes when training on Type II trajectories (In context distillation) may teach the model to "recall" facts without grounding and cause hallucination, yet the paper does not test.
  • Limited scope and an overstated title. Only the Qwen3.5 family is used, forgetting of general capabilities outside the target environment is not measured, and no continual or test-time self-improvement experiments of the kind implied by the title are conducted.
  • Missing comparisons. The paper compares neither against letting the base model retrieve the same notes at test time (e.g., RAG) nor against methods such as synthetic continued pretraining, so it cannot show that updating weights is better than keeping the experience in context.
  • Writing and rigor issues:
    • Typos and grammar: "whther" (Abstract); "their goal is distill" and "in a the environment" (Sec. 2); "faor C&H" (Sec. 5); "Equational Theorie" (Fig. 2 caption).
    • Inconsistencies: the StudyBench model is "Qwen3.8-27B" in Sec. 5 but "Qwen3.5-27B" in Sec. 1 and Fig. 4; the Equational Theories training scales stop at 30M in Sec. 5, whereas Tables 2 and 4 and Figs. 2–3 include 100M; the DSPy codebase size is given as ~0.33M, ~650K, and 0.46M (Fig. 4) tokens.
    • Sructure: The caption of Table 4 is incomplete; Sec. 4.2 refers to Type I as "Stage I"; Sec. 6 has a single subsection (6.1, titled for Type I data) that also contains the Type II analyses and the conclusion.
    • Eleven leftover :chatgpt-content-referenceindex="N" markers remain in Appendix A.1–A.3.
Questions for authors

See weakness.

Reviewer_d4W6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how an agent can adapt to an unfamiliar environment before downstream evaluation questions are revealed. The proposed method, U-WIN, uses a frozen post-trained model to generate two types of training data from autonomous exploration: Type I consists of consolidated notes summarizing exploratory trajectories, and Type II consists of synthetic problem-solving trajectories generated with access to these notes (which are stripped out during student training). The model is fine-tuned on a mixture of both data types in a one-time adaptation step. Across the Harvey Legal Benchmark, StudyBench (DSPy and OpenClaw), and Equational Theories, U-WIN improves both closed-book knowledge retention and downstream agentic performance. Ablations show that training on notes outperforms training on raw documents or observations, and that including Type II trajectories mitigates the behavioral degradation caused by training on notes alone.

Critical strengths
  • Raw exploratory trajectories are often noisy and poorly suited for direct next-token prediction. U-WIN sensibly uses the model's existing in-context abilities to consolidate raw observations into structured notes (Type I). The token-matched comparison on C&H provides clear empirical backing for this design.

  • The study of how data mixture affects knowledge retention and agentic execution is interesting and practical. The training on the mixture of Type I and Type II behavioral trajectories internalizes environmental knowledge without ruining post-trained agentic behavior, while training on notes alone breaks the agent's behavior protocol.

Critical weaknesses
  • Figure 4 shows that Type I notes alone improve knowledge acquisition, and Figure 6 shows that joint Type I+II training improves over notes-only training. However, a Type-II-only baseline at a matched supervised-token budget is missing, leaving it unclear what direct note training adds once note-conditioned trajectory distillation is available. This distinction matters in relation to Online Experiential Learning [1], which already consolidates extracted experiential knowledge into model parameters through knowledge-conditioned context distillation. ExpeL [2] also provides an earlier precedent for extracting reusable insights into external memory, while concurrent work RSIAgent [3] studies related autonomous exploration and memory construction. A Type-II-only comparison would help establish the distinctive empirical value of combining direct note training with trajectory distillation in U-WIN’s setting of studying an environment before downstream questions are known.

  • Section 4 presents forward-KL regularization, base-trajectory replay, and model merging as part of the method, but Appendix A.1 states that the main runs use none of them. These mechanisms directly target catastrophic forgetting and protocol adherence, which are central to the paper's claim that U-WIN preserves agentic behavior.

[1] Ye, Tianzhu, et al. "Online experiential learning for language models." arXiv preprint arXiv:2603.16856 (2026).

[2] Zhao, Andrew, et al. "Expel: Llm agents are experiential learners." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.

[3] Zhu, Sibo, et al. "RSIAgent: Autonomous exploration for recursive self-improvement in new environments." arXiv preprint arXiv:2609.15364 (2026).

Questions for authors
  • How do you differentiate U-WIN from parametric distillation methods like OEL? Do you have a token-matched Type-II-only ablation showing the marginal gain of training directly on Type I notes?

  • Comparing Type I to Type I + II adds both note-conditioned reasoning and general format rehearsal. Would Type II's gains simply come from rehearsing valid tool-use trajectories? How does Type I + II perform against Type I + unconditioned synthetic trajectories?

  • Were forward KL, replay, and model merging actually used in the main runs?

Reviewer_PJMVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes U-WIN, a method for improving LLM agents through unsupervised exploration of an environment. The method constructs two types of training data: (1) notes summarizing knowledge acquired during exploration and (2) knowledge-conditioned trajectories demonstrating how to use the acquired knowledge. The authors report improvements in knowledge retention and downstream task performance across several environments.

Critical strengths

The paper addresses an interesting problem of learning from autonomous experience without explicit task-level supervision, which is an important topic in agentic and continual learning. The experiments show encouraging improvements across different environments, with some useful ablation studies.

Critical weaknesses

Despite the positive empirical results, I have several concerns about the scientific contribution and experimental validation.

  1. (Unclear methodological contribution) The proposed method appears to combine existing techniques, particularly synthetic continued pretraining and context distillation from the literature. While the combination may be practically useful and can absolutely be a novel contribution too, I find it difficult to identify a distinctive methodological insight or generalizable principle established by this work.

  2. (Insufficient experimental validation) The experiments do not sufficiently isolate the benefits of the proposed design choices. In particular, it is unclear whether autonomous exploration provides meaningful advantages over simpler synthetic data generation directly from the environment. Similarly, the improvements from Type II data may largely reflect the preservation of existing agentic capabilities rather than learning to integrate internalized knowledge with reasoning and action. Stronger controlled comparisons would be necessary to distinguish these explanations.

  3. (Limited generalizability) Last but not least, although the authors evaluate several environments, the effectiveness of the method may depend heavily on the specific exploration, note-generation, and training procedures. The paper provides limited insight into when or why the approach should work more generally. Furthermore, the experiments involve a single adaptation procedure rather than repeated test-time self-improvement, despite the broader claim in the title.


Overall, I appreciate the motivation and empirical effort, but I am not convinced that the paper establishes a sufficiently clear or generalizable scientific contribution. The results primarily demonstrate that a particular combination of existing techniques improves performance in the evaluated environments, without adequately establishing the benefits of its distinguishing components. I therefore lean toward rejection.

Questions for authors
  1. (Methodological contribution) What is the key methodological insight established by U-WIN beyond combining synthetic continued pretraining and context distillation? In particular, Figure 4 compares against training on raw documents and observations, but not against simpler methods of generating synthetic training data directly from the same environment. What evidence supports the importance of autonomous exploration in the proposed method?

  2. (Experimental validation) To what extent do the experiments establish the effectiveness of the proposed knowledge internalization mechanism, rather than the benefits of additional domain-specific training data? For example, Figure 6 suggests that Type II data help preserve existing agentic capabilities. How can we distinguish this effect from the claimed improvement in using internalized knowledge for reasoning and action?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.