- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper addresses a key limitation of current LLM agents operating in unfamiliar environments: they often start executing tasks immediately based on prior knowledge, without first developing an adequate understanding of the environment’s states, objects, constraints, or available actions. The authors characterize this behavior as “premature exploitation” and argue that it stems from a missing capability: autonomous exploration. Rather than treating exploration as something that emerges incidentally during task solving, the paper proposes modeling it as a distinct behavior that can be explicitly trained.
To encourage this capability, the authors introduce Exploration Checkpoint Coverage (ECC), a verifiable metric that measures how broadly an agent explores an environment. ECC is incorporated into an interleaved reinforcement-learning framework that alternates between exploration-oriented and task-oriented training. At inference time, the proposed Explore-then-Act framework allows the agent to first explore the environment without being given a specific task, summarize the knowledge it acquires, and then use that grounded understanding to solve the subsequent task. Experiments across several interactive benchmarks suggest that this approach can encourage agents to engage in more deliberate and strategically useful information-seeking behavior.
Critical strengths
(Intuitive idea, easy to follow)
The paper is easy to read and follow. I find the idea presented to be intuitive, although lacking generalizability where “exploration coverage” might be hard to measure (see weaknesses below). In general, I don’t have doubts that this paper’s method should work given it is properly implemented, and kudos to the authors for writing things clearly.
(Generalization)
The authors evaluate the GRPO (interleaved) checkpoint trained on ALFWorld in SciWorld (or ALFWorld-Variant), and report performance improvements. In some ways that is the true measure of learning autonomous exploration, and experiments demonstrating the efficacy of this is a significant strength of this paper in my opinion.
Critical weaknesses
(Definition of Exploration Checkpoint Coverage)
The definition of exploration checkpoint coverage involves the fraction of exploration checkpoints visited/covered vs the total number of such checkpoints. While this is a fine “exploration” metric, it has the major disadvantage of having to know the total number of exploration checkpoints apriori. This would require either annotation by a human or by a larger teacher model, which hinders scalability of this paper’s approach. Could the authors propose a mechanism that would not require such ground truth information and provide experiments to demonstrate how capable their algorithm is in such scenarios?
(Missing important baselines)
There are some key prior works that should be cited/discussed and compared against. For example:
-
The idea of using exploration for a fixed budget, and then using the exploration summary as a hint/memory for the “act” step is very similar to EMPO$^2$ [2], which was also evaluated on similar benchmarks. Could the authors add a comparison to this method and discuss the differences? In my opinion, EMPO$^2$ is a superior method since it does not require annotation about the exploration checkpoints, and the model itself generates the hints to try out in the next steps.
-
Another similar method for teaching models information-seeking capabilities is -Belief-RL [3], but no comparison to it is provided in the paper.
-
Finally, why is the “Explore-then-Act” framework necessary? Can’t one essentially jointly optimize both in a single rollout (after all, that is how modern agents like Claude Code/Codex works, where they perform exploration and then act in the environment, typically in the same rollout)? It seems one need to (1) define/instruct the agents to explore the environment first, before acting, in the system prompt, (2) define ECC or some rubric based reward to incentivize exploration before exploitation, (3) jointly optimize exploration and exploitation in a single trajectory, either via GRPO or its modern variants like GDPO [5]. Could the authors give an explanation for why this particular approach is superior?
(Lack of necessary related works)
The paper’s related works/cited works need more work. I could not find a citation to any relevant prior work on policy gradients. Despite using GRPO for training their policies, the original GRPO paper [4] is not cited, and neither are any of its predecessors. The explore-then-act framework is similar to earlier works in memory generation/memory augmented task execution, but I could not find a proper discussion and connection to prior works in this paper.
Questions for authors
(Reconciliation with earlier work)
I am curious how the authors address earlier work that addresses autonomous exploration in LLMs. The example work of Paprika [1] comes to mind. I think the authors should cite Paprika and its predecessors in the related work and discuss the similarities/differences. I am especially curious about the reconciliation of the two papers’ findings — Paprika claims that training on a diverse range of environments with success only outcomes can teach the models to autonomously explore in unseen environments, specially when exploration/strategic information is essential to succeed in the tasks. The key ingredient in their work is that the environments are designed to be partial information, so as to require exploration to succeed and the outcome only success reward already captures this. A cool comparison would be training the models for exploration in similar environments, and observing how much benefit the ECC rewards actually provide.
(Question about LLM Backbones)
The paper mentions that the LLM backbones used in this work are Llama-3.1-8B, Qwen3-4B, or Qwen2.5-7B. However, some of these models don’t have any instruction following capabilities, and I am not sure if they can even perform decently/follow instructions in a multi-turn interaction setting. Did the authors mean Llama-3.1-8B-Instruct or Qwen2.5-7B-Instruct, for example? A clarification on this would be highly helpful.
References
[1] Training a Generally Curious Agent, https://arxiv.org/abs/2502.17543
[2] Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization, https://openreview.net/forum?id=UOzxviKVFO
[3] Intrinsic Credit Assignment for Long Horizon Interaction, https://arxiv.org/pdf/2602.12342
[4] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, https://arxiv.org/abs/2402.03300
[5] GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization, https://arxiv.org/abs/2601.05242