acceptodds
Under review as a conference paper at ICLR 2027

Training LLM Agents for Spontaneous, Reward-Free Self- Evolution via World Knowledge Exploration

Abstract

Most agents today `self-evolve' by following rewards and rules defined by humans. However, this process remains fundamentally dependent on external supervision; without human guidance, the evolution stops. In this work, we train agents to possess an intrinsic meta-evolution capability to spontaneously learn about unseen environments prior to task execution. To instill this ability, we design an outcome-based reward mechanism that measures how much an agent's self-generated world knowledge improves its success rate on downstream tasks. This reward signal is used exclusively during the training phase to teach the model how to explore and summarize effectively. At inference time, the agent requires no external rewards or human instructions. It spontaneously performs native self-evolution to adapt to unknown environments using its internal parameters. When applied to Qwen3-30B and Seed-OSS-36B, this shift to native evolution yields a 20% performance increase on WebVoyager and WebWalker. Most strikingly, the generated world knowledge even enables a compact 14B Qwen3 model to outperform the unassisted Gemini-2.5-Flash, establishing a new paradigm for truly evolving agents. Code is available at https://anonymous.4open.science/status/self_eolving_wk-036D.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Evolving, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x3nzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training framework that enables LLM agents to proactively explore unfamiliar environments and construct reusable world knowledge without access to downstream tasks at inference time. The method employs an outcome-based reward measuring the downstream utility of generated knowledge, followed by supervised fine-tuning (SFT) on teacher-generated trajectories and iterative rejection-sampling fine-tuning (RFT). Experiments on WebWalker and WebVoyager demonstrate substantial improvements in task performance, with additional results suggesting that the generated knowledge can transfer across different backbone models.

Critical strengths

S1. Effective training framework for task-independent exploration.

The paper introduces an outcome-based reward that optimizes environment exploration according to the downstream utility of generated knowledge. Combining teacher-guided SFT with iterative RFT provides a practical approach to learning long-horizon exploration strategies. The separation between task-dependent training rewards and task-independent inference is particularly interesting.

S2. Strong empirical performance and cross-model transferability.

The method achieves substantial improvements on WebWalker, increasing Qwen3-30B-A3B accuracy from 20.71% to 45.34%. Experiments across different backbone models and knowledge-transfer settings further demonstrate its practical potential. These results suggest that the generated world knowledge captures useful information beyond model-specific representations.

Critical weaknesses

W1. Insufficient justification of the claimed self-evolution capability.

The proposed method primarily performs environment exploration and external knowledge construction, without modifying the agent's policy during inference. Moreover, it relies on manually designed website preprocessing and exploration prompts, which weakens the claim of workflow-free autonomy. The authors should better distinguish their approach from existing exploration and memory-based adaptation methods.

W2. Lack of sufficiently controlled comparisons and end-to-end efficiency analysis.

The main baseline does not receive the additional environment exploration and knowledge available to the proposed method, making it difficult to isolate the contribution of the learned exploration policy. Although prompt-only baselines are included, comparisons with conventional exploration strategies under matched computational and information budgets remain insufficient. Furthermore, the reported efficiency improvements do not fully account for the cost of generating world knowledge.

Questions for authors

Q1. What distinguishes the proposed method from conventional exploration and external-memory construction?

Since inference-time adaptation occurs through generated world knowledge rather than policy updates, could the authors clarify the operational definition of self-evolution? In particular, what capabilities distinguish the proposed method from existing knowledge-based agent adaptation approaches?

Q2. How much improvement remains under matched exploration budgets?

Could the authors compare their method against conventional exploration or summarization strategies with identical exploration steps, token budgets, and knowledge-length constraints? This would help isolate the contribution of the learned exploration policy.

Q3. How important is the graph-based website preprocessing?

Could the authors provide an ablation removing the graph-based preprocessing G(U)G(U)? Since this component explicitly structures the exploration process, understanding its contribution is important for evaluating the claimed autonomy.

Q4. Does the method improve end-to-end computational efficiency?

The reported reduction in downstream execution steps does not account for the additional cost of generating world knowledge. Could the authors report total token consumption, browser interactions, and wall-clock time, including the exploration phase?

Reviewer_Hzn3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes a “native evolution” paradigm. The agent explores a new environment in many cases a webite and compresses what it sees into a structured Markdown document, before receiving any specific tasks. The document is tne loaded into context for downstream tasks. The paper repeatedly claims that its agents evolve in a workflow-free and task-independent way, without human guidance, and contrasts this with prior self-evolving agents that it calls an “illusion” for depending on human-defined workflows.

Critical strengths

There is some merit in the overal ambition. Having an agent explore an unfamiliar environment before any task arrives, and distill that into reusable knowledge, is an appealing direction. It differs from the usual reactive, task-conditioned agent setup.Also, the level of transparency is appreciated, Code is released, prompts are included and the training and evaluation websites are listed.

Critical weaknesses

“workflow-free,” “spontaneous,” and “task-independent” claims need to be toned down. Too many hand wavy statements that are sometimes too strong for what is supported. The input is still processed by a hand-designed pipeline, the inference prompt still perscribes a detaild workflow, it seems that downstream use is also scripted, so there are very strong claims in this work that are not supported. Generally, the paper is not well written for example even the captions are just brief one liners there is an overuse of terms for example in the conclusion suddenly the notion of intrinsic meta evolution appears which has not be introduced before.

Questions for authors

Which experiments support statements such as current self-evolution being “an illusion,” results that “prove” innate adaptive ability, and the work “paving the way toward AGI”? Do the reported results use the scripted decision prompt in Appendix D.5? How do results change with a standard task prompt that simply provides the knowledge document? How does the method perform when started from only a homepage URL, with a minimal prompt, no link-graph preprocessing, and no helper functions? Without this ablation, it is unclear whether the trained model or the scaffolding drives the gains.

Reviewer_E6YJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for training LLM agents to proactively explore unseen environments and construct reusable world knowledge before receiving specific tasks. Experiments on WebWalker and WebVoyager demonstrate improved downstream performance and cross-model knowledge transfer.

Critical strengths
  1. The paper addresses an interesting problem of task-independent environment exploration for LLM agents.
  2. The proposed reward provides a practical way to evaluate the utility of generated world knowledge. The experiments show improvements across different backbone models and benchmarks.
Critical weaknesses
  1. Overstated self-evolution claims. The method primarily generates external world knowledge without updating model parameters at inference time.
  2. Insufficient baseline comparisons. The paper lacks comparisons with existing agent memory methods and external knowledge augmentation approaches.
  3. Incomplete cost analysis. The paper reports improvements in downstream execution efficiency but does not account for the overall cost of initial environment exploration and world knowledge generation.
Questions for authors
  1. Could the authors clarify how the proposed approach relates to self-evolving agents?
  2. Could the authors include more baselines involving agent memory and external knowledge augmentation approaches?
  3. Could the authors provide details on the cost of initial environment exploration and world knowledge generation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.