acceptodds
Withdrawn from ICLR 2027

Look Before You Leap: Autonomous Exploration for LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model based agents often fail in unfamiliar environments due to premature exploitation: a tendency to act on prior knowledge before acquiring sufficient environment-specific information. We identify autonomous exploration as a critical yet underexplored capability for building adaptive agents. To formalize and quantify this capability, we introduce Exploration Checkpoint Coverage, a verifiable metric that measures how broadly an agent discovers key states, objects, and affordances. Our systematic evaluation reveals that agents trained with standard task-oriented reinforcement learning consistently exhibit narrow and repetitive behaviors that impede downstream performance. To address this limitation, we develop a training strategy that interleaves task-execution rollouts and exploration rollouts, with each type of rollout optimized by its corresponding verifiable reward. Building on this training strategy, we propose the Explore-then-Act paradigm, which decouples information-gathering from task execution: agents first utilize an interaction budget to acquire grounded environmental knowledge, then leverage it for task resolution. Our results demonstrate that learning to systematically explore is imperative for building generalizable and real-world-ready agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Reinforcement Learning, Exploration

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_rYjKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses a key limitation of current LLM agents operating in unfamiliar environments: they often start executing tasks immediately based on prior knowledge, without first developing an adequate understanding of the environment’s states, objects, constraints, or available actions. The authors characterize this behavior as “premature exploitation” and argue that it stems from a missing capability: autonomous exploration. Rather than treating exploration as something that emerges incidentally during task solving, the paper proposes modeling it as a distinct behavior that can be explicitly trained.

To encourage this capability, the authors introduce Exploration Checkpoint Coverage (ECC), a verifiable metric that measures how broadly an agent explores an environment. ECC is incorporated into an interleaved reinforcement-learning framework that alternates between exploration-oriented and task-oriented training. At inference time, the proposed Explore-then-Act framework allows the agent to first explore the environment without being given a specific task, summarize the knowledge it acquires, and then use that grounded understanding to solve the subsequent task. Experiments across several interactive benchmarks suggest that this approach can encourage agents to engage in more deliberate and strategically useful information-seeking behavior.

Critical strengths

(Intuitive idea, easy to follow)

The paper is easy to read and follow. I find the idea presented to be intuitive, although lacking generalizability where “exploration coverage” might be hard to measure (see weaknesses below). In general, I don’t have doubts that this paper’s method should work given it is properly implemented, and kudos to the authors for writing things clearly.

(Generalization)

The authors evaluate the GRPO (interleaved) checkpoint trained on ALFWorld in SciWorld (or ALFWorld-Variant), and report performance improvements. In some ways that is the true measure of learning autonomous exploration, and experiments demonstrating the efficacy of this is a significant strength of this paper in my opinion.

Critical weaknesses

(Definition of Exploration Checkpoint Coverage)

The definition of exploration checkpoint coverage involves the fraction of exploration checkpoints visited/covered vs the total number of such checkpoints. While this is a fine “exploration” metric, it has the major disadvantage of having to know the total number of exploration checkpoints apriori. This would require either annotation by a human or by a larger teacher model, which hinders scalability of this paper’s approach. Could the authors propose a mechanism that would not require such ground truth information and provide experiments to demonstrate how capable their algorithm is in such scenarios?

(Missing important baselines)

There are some key prior works that should be cited/discussed and compared against. For example:

  1. The idea of using exploration for a fixed budget, and then using the exploration summary as a hint/memory for the “act” step is very similar to EMPO$^2$ [2], which was also evaluated on similar benchmarks. Could the authors add a comparison to this method and discuss the differences? In my opinion, EMPO$^2$ is a superior method since it does not require annotation about the exploration checkpoints, and the model itself generates the hints to try out in the next steps.

  2. Another similar method for teaching models information-seeking capabilities is Δ\Delta-Belief-RL [3], but no comparison to it is provided in the paper.

  3. Finally, why is the “Explore-then-Act” framework necessary? Can’t one essentially jointly optimize both in a single rollout (after all, that is how modern agents like Claude Code/Codex works, where they perform exploration and then act in the environment, typically in the same rollout)? It seems one need to (1) define/instruct the agents to explore the environment first, before acting, in the system prompt, (2) define ECC or some rubric based reward to incentivize exploration before exploitation, (3) jointly optimize exploration and exploitation in a single trajectory, either via GRPO or its modern variants like GDPO [5]. Could the authors give an explanation for why this particular approach is superior?

(Lack of necessary related works)

The paper’s related works/cited works need more work. I could not find a citation to any relevant prior work on policy gradients. Despite using GRPO for training their policies, the original GRPO paper [4] is not cited, and neither are any of its predecessors. The explore-then-act framework is similar to earlier works in memory generation/memory augmented task execution, but I could not find a proper discussion and connection to prior works in this paper.

Questions for authors

(Reconciliation with earlier work)

I am curious how the authors address earlier work that addresses autonomous exploration in LLMs. The example work of Paprika [1] comes to mind. I think the authors should cite Paprika and its predecessors in the related work and discuss the similarities/differences. I am especially curious about the reconciliation of the two papers’ findings — Paprika claims that training on a diverse range of environments with success only outcomes can teach the models to autonomously explore in unseen environments, specially when exploration/strategic information is essential to succeed in the tasks. The key ingredient in their work is that the environments are designed to be partial information, so as to require exploration to succeed and the outcome only success reward already captures this. A cool comparison would be training the models for exploration in similar environments, and observing how much benefit the ECC rewards actually provide.

(Question about LLM Backbones)

The paper mentions that the LLM backbones used in this work are Llama-3.1-8B, Qwen3-4B, or Qwen2.5-7B. However, some of these models don’t have any instruction following capabilities, and I am not sure if they can even perform decently/follow instructions in a multi-turn interaction setting. Did the authors mean Llama-3.1-8B-Instruct or Qwen2.5-7B-Instruct, for example? A clarification on this would be highly helpful.

References

[1] Training a Generally Curious Agent, https://arxiv.org/abs/2502.17543

[2] Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization, https://openreview.net/forum?id=UOzxviKVFO

[3] Intrinsic Credit Assignment for Long Horizon Interaction, https://arxiv.org/pdf/2602.12342

[4] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, https://arxiv.org/abs/2402.03300

[5] GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization, https://arxiv.org/abs/2601.05242

Reviewer_TJeVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to improve Large Language Model (LLM)-based agents operating in unfamiliar environments. The authors note that agents trained using standard task-oriented reinforcement learning often exhibit limited and repetitive behavioral patterns, which hinder performance in subsequent tasks. To address this, they propose an "Explore-then-Act" strategy, the core idea of ​​which is to explore the environment before executing tasks. Experimental results demonstrate that learning to understand the new environment facilitates the execution of subsequent tasks.

Critical strengths

This paper is well written. The core idea of Explore-then-Act is presented clearly and easy to follow.

Critical weaknesses

The concept of acquiring prior knowledge about the environment through exploration—and leveraging that knowledge to improve subsequent task execution—is not new. In fact, long before the rise of large language models (LLMs), exploration was already a long-standing research topic in the field of AI agents. The observations, insights, and conclusions presented in this paper have been discussed repeatedly in existing literature.

Questions for authors

For many complex real-world problems, it is difficult to pre-define a series of exploration checkpoints. How can your method be extended to handle such scenarios? Within your "Explore-then-Act" paradigm, how should the trade-off between exploration and exploitation be balanced?

Reviewer_fQNPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies premature exploitation as a problem in LLM agents: the tendency to act on prior knowledge before sufficiently exploring an unfamiliar environment. It introduces Exploration Checkpoint Coverage (ECC) to evaluate goal-free exploration and reports low coverage and premature termination in task-trained models.

The proposed method interleaves GRPO updates on exploration rollouts rewarded by ECC and task rollouts rewarded by task success. Experiments on ALFWorld, ScienceWorld, TextCraft, and an ALFWorld variant show improvements under both Direct execution and Explore-then-Act, which uses a summary of prior exploration to support task execution. Validation also includes comparisons with matched interaction budgets and a fixed executor, alongside transfer to environments excluded from training.

Critical strengths
  1. The training procedure is simple and requires little change to standard GRPO.   Exploration rollouts receive ECC rewards and are interleaved with task rollouts. Environment-specific ECC extractors are needed, but the training method itself is straightforward. The reward makes sense in environments where exploration is important, such as when agents must locate objects or verify action effects. Parts of the method could be useful in other agent research.

 

  1. Exploration training improves downstream task performance, including under Direct execution.   In Table 2, Qwen3-4B’s ALFWorld success rate increases from 84.6% for Task-Only Direct to 90.5% for Interleaved Direct, and reaches 92.7% with Interleaved E-t-A. The improvement under Direct is meaningful, as it shows a benefit even without a separate exploration phase at test time. The analysis of repeated actions and premature termination also helps connect these gains to changes in agent behavior.
Critical weaknesses
  1. ECC’s design needs clearer justification, given its central role in both evaluation and training.   The choice of locations, objects, and affordances seems heuristic and closely tied to the evaluated environments, leaving its broader applicability unclear. The main text provides the coverage formula and checkpoint examples, while the construction and verification details are deferred to Appendix E. I would also like to see more analysis of whether higher coverage reflects the acquisition of useful information and whether agents actually use this information during task execution.

 

  1. I am not fully convinced that the low ECC scores in Section 4.2 alone demonstrate a limitation in exploration ability that affects task performance.   Task-oriented training optimizes task success, so a model may acquire the information needed for a task while having little incentive to explore the environment broadly. The argument would be more convincing with concrete examples or analysis showing what task-relevant information these models fail to acquire and how this leads to problems during task execution.
Questions for authors
  1. ECC components and checkpoint design in other environments.  Have you evaluated downstream task performance when each ECC category (locations, objects, or affordances) is removed from the exploration reward one at a time during training? I would like to understand whether all three contribute to the gains or whether the results depend heavily on one category. Could you also explain the rationale for these categories and how checkpoints would be defined, selected, and verified in web environments or more complex real-world tasks? Which design principles would carry over, and which aspects would require environment-specific changes?

 

  1. Training and evaluation prompts.   The Exploration Strategy and Important Notes in Appendix D.5 may substantially affect exploration behavior. Did you also include these sections in the task prompts used for Task-Only training and Direct evaluation? For reproducibility, could you provide the full prompts for these settings and for both exploration and task rollouts during Interleaved training? In particular, was the task prompt used in Interleaved training identical to the one used for Task-Only training?
Reviewer_fNwJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses premature exploitation in LLM-based agents by formalizing autonomous exploration as a distinct, trainable capability and introducing Exploration Checkpoint Coverage (ECC), a verifiable coverage metric over environment-defined checkpoints (locations, objects, affordances). The authors propose an interleaved GRPO training scheme that alternates between task-execution rollouts (task-completion reward) and exploration rollouts (ECC reward), and evaluate an Explore-then-Act (E‑t‑A) paradigm where agents first explore to collect grounded knowledge before acting on a task. Across ALFWorld, ScienceWorld, TextCraft, and an ALFWorld variant, they find that task-only RL suppresses exploration, while exploration-aware training improves ECC, reduces looping, and yields consistent downstream gains, even under fixed interaction budgets and some cross-environment transfer.

Critical strengths

Paper introduces ECC as a practical, verifiable, goal-free exploration metric that directly rewards discovery of environment structure, objects, and affordances. They further proposes an interleaved GRPO training strategy that cleanly decouples exploration and task-execution objectives while keeping total rollout budgets fixed. Empirical study covers evaluation of approach across multiple interactive text environments. Although authors show that exploration-aware training can consistently translate an exploration budget into downstream gains, including under distributional shifts and partial transfer, but I feel that thorough baseline testing is missing such as similar inference only study [1]. While reliance on hand-crafted checkpoints limits immediate scalability, and comparisons to some closely related exploration/mapping baselines could be strengthened, the conceptual clarity and practical effectiveness are compelling.

[1] Verma, Nikhil, and Manasa Bharadwaj. "LEAP & LEAN: Look-ahead Planning and Agile Navigation for LLM Agents." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track). 2025.

Critical weaknesses

ECC depends on a hand-specified checkpoint set per environment; scalability and objectivity of checkpoint construction for open-ended or real-world domains (e.g., web, GUI) are uncertain. What if they are missing? Also binary coverage may be insensitive to redundancy, action costs, or temporal sequencing; agents could optimize for breadth over depth without capturing precondition structure critical to tasks.

Novelty is missing. Limited direct comparisons to recent exploration (like one shared above) or map-then-act [1] baselines beyond the provided ablations. Stronger baselines would clarify ECC’s advantage beyond the specific instantiations tested. Details of the knowledge summarization process and its reliability are under-specified; failure analysis indicates some high-coverage runs fail due to summarization/usage, but there is no controlled ablation of the summarizer quality.

[1] Liu, Yuxin, et al. "MAP: A Map-then-Act Paradigm for Long-Horizon Interactive Agent Reasoning." arXiv preprint arXiv:2605.13037 (2026).

Questions for authors
  • How are checkpoints constructed and validated to avoid leakage and ensure they measure general exploration rather than task-specific shortcuts? Can you quantify annotator effort and inter-rater reliability, and provide examples per environment category (location/object/affordance)?
  • How robust is ECC to reward hacking (e.g., repeatedly touching easy checkpoints or partial observations counting as coverage)? Do you enforce novelty or per-checkpoint credit once to prevent exploitation?
  • What is the exact summarization pipeline for K (prompt templates, extraction vs free-form notes, token limits)? Can you ablate its quality (e.g., compress/noisy summaries) to measure sensitivity of E‑t‑A to summarization fidelity?
  • In the fixed-budget setting, can you report failure categories where high ECC did not translate to gains (e.g., summarization loss vs execution misapplication). Whether adding lightweight planning/memory modules closes the gap?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.