acceptodds
Under review as a conference paper at ICLR 2027

ExpVoyager: Direct Experience Navigation for Dynamic Agent Skill Synthesis

32 views

Abstract

Learning from experience in LLM agents has become a key paradigm for developing self-evolving agents that continuously learn and expand their capabilities. Within this paradigm, synthesizing the agent skill has emerged as a promising solution for transforming accumulated experience into reusable procedural knowledge, serving as an important layer for the harness system that supplies agents at runtime. Despite its potential, existing approaches largely abstract past experience into fixed procedural knowledge before downstream demands are known, which risks discarding knowledge that later becomes critical while retaining instance-specific details irrelevant to future tasks. In this paper, we reframe agent skill synthesis as a dynamic navigation problem over past experience, where agents actively explore accumulated trajectories on demand for the current task with targeted and fine-grained access to experience knowledge. To this end, we propose ExpVoyager, a novel framework in which a skill curator navigates raw experience across different views and resolutions, continually identifying reusable procedural knowledge from what it observes while tracking remaining knowledge needs that guide where to navigate next. Extensive experiments demonstrate both the effectiveness and versatility of ExpVoyager, showing consistent improvements in downstream task performance, continual gains as the experience space scales, and practical compatibility with existing skills under efficient experience access.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agents, experience, self-evolving agents, skill, memory, navigation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gWH7OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ExpVoyager proposes dynamic skill synthesis through active navigation of accumulated agent trajectories. Instead of constructing a fixed skill library or retrieving only top-ranked experiences, a curator navigates trajectory-level and step-level views, maintains structured knowledge and unresolved questions, and generates a task-conditioned skill for a frozen executor. The approach is evaluated on ALFWorld, WebShop, and ScienceWorld.

Critical strengths
  1. The paper addresses an important information-loss problem in agent memory systems: fixed procedural abstractions can discard details needed by future tasks. The distinction between retrieving summaries and actively inspecting original experiences is meaningful.

  2. The experiments demonstrate consistent gains across several environments and executor backbones. On ALFWorld with Qwen3.5-9B, ExpVoyager improves success to 63.7%, compared with 51.1% for SkillTTA. Component ablations and matched-budget agentic retrieval comparisons provide additional evidence for the proposed navigation mechanism.

Critical weaknesses

W1. Substantial test-time overhead relative to the marginal performance gains.

The default configuration performs 20 navigation rounds and consumes approximately 385K end-to-end tokens per ALFWorld task, compared with approximately 52K for the base agent. More importantly, the five-round configuration already achieves 60.4% success using approximately 159K tokens, while the 20-round configuration reaches 63.7%. Thus, more than doubling the token cost produces only a 3.3-percentage-point improvement. The practical value of the default navigation budget is therefore questionable, particularly for latency-sensitive or frequently repeated tasks.

W2. The distinction from general-purpose agentic retrieval remains insufficiently established.

The key mechanisms—iterative tool-based retrieval, maintaining accumulated knowledge, tracking unresolved questions, and adapting subsequent search queries—overlap considerably with established agentic search and structured reasoning workflows. The paper includes comparisons against iterative retrieval baselines under matched experience-access budgets, which is valuable. However, these baselines do not appear to receive the same fine-grained trajectory interface and structured navigation state simultaneously. Consequently, the relative contribution of the proposed navigation formulation remains incompletely isolated.

W3. Limited adaptation to new evidence encountered during task execution.

ExpVoyager synthesizes a task-conditioned skill before the executor begins interacting with the target environment. Once execution starts, the generated skill is effectively fixed. This creates a mismatch between experience navigation and execution feedback: newly observed failures or environmental conditions cannot directly trigger further evidence gathering and skill revision. The method may therefore be less effective in environments where critical uncertainties only become apparent during execution.

W4. The framework lacks explicit validation of synthesized procedural knowledge.

The curator derives procedural guidance from prior trajectories and maintains knowledge items and unresolved questions through language-model-based updates. However, identifying a pattern in previous experience does not guarantee that the pattern is causally valid or applicable to the current task. The paper does not systematically test robustness to contradictory, misleading, or low-quality trajectories, nor does it provide an explicit verification mechanism for newly synthesized guidance. This leaves the reliability of experience transfer uncertain.

W5. Cold-start behavior reveals a potentially important negative-transfer problem.

The online self-evolution experiments show that ExpVoyager can provide little benefit or even degrade performance when the available experience pool is small. This indicates that the method is not uniformly beneficial throughout continual deployment. Although its performance improves as experience accumulates, the paper does not establish a principled criterion for determining when the available evidence is sufficient to justify navigation. An adaptive fallback to the base agent or a confidence-aware navigation policy could be important for practical deployment.

W6. Scalability claims are stronger than the demonstrated retrieval setting.

The experience-scaling experiments use nested subsets of a pool containing up to approximately 1,000 trajectories. The navigation interface relies on field-specific regular-expression search and full-trajectory inspection. It is unclear how efficiently this design would scale to much larger, heterogeneous experience repositories. In particular, retrieval latency, indexing cost, and navigation quality under a substantially expanded corpus are not systematically evaluated.

W7. Dependence on rich recorded trajectories limits applicability.

The experience interface assumes access to detailed records containing observations, actions, outcomes, and often the executor's recorded reasoning. In realistic deployments, such internal reasoning traces may be unavailable, incomplete, inconsistent, or generated by different models. The method does not sufficiently evaluate whether its advantages persist with action-observation-only logs, highly compressed records, or trajectories collected by heterogeneous agents.

Questions for authors
  1. Can the authors report a complete performance–cost Pareto frontier, including latency, curator calls, and token consumption? Is there an adaptive stopping rule that approaches the 20-round performance with substantially fewer navigation operations?

  2. How much of the gain is preserved when a strong agentic retrieval baseline receives the same trajectory/step-level interface and maintains an explicit knowledge-and-open-questions state?

2 comments
AuthorsOpenReview ↗

Follow-up on our responses

Dear reviewer gWH7,

Thank you again for your detailed review and valuable feedback. Your comments have helped us strengthen our evaluation and clarify the scope and limitations of our approach. We hope our responses and additional analyses help address your concerns. If any concerns remain, we would appreciate the opportunity to discuss them further.

Best regards,

Authors of Paper #37719

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, particularly the stronger retrieval baselines and additional evaluations on scalability and heterogeneous experience. However, I still have substantial concerns regarding the novelty and practical efficiency of the proposed method. The distinction from existing agentic retrieval approaches remains insufficiently clear, and the considerable test-time computational overhead is not fully justified by the performance gains. Furthermore, the lack of execution-time adaptation and the remaining issues with knowledge reliability limit the method's practical applicability. Overall, while the additional experiments strengthen the empirical evidence, they do not fully resolve my concerns about the methodological contribution and generality. Nevertheless, considering the authors' substantial efforts and the additional evidence provided, I am willing to raise my rating from 1 (Clear Reject) to 2 (Weak Reject).

Reviewer_NxW3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how LLM agents can reuse past execution experience to generate task-specific skill guidance without updating the executor's parameters. The authors argue that pre-constructed skills may discard information useful for future tasks, while similarity-based retrieval may overlook important procedural details. To address this, ExpVoyager introduces an LLM-based skill curator that actively navigates historical experience through trajectory-level and step-level views. It maintains curated knowledge and open questions to guide further exploration and ultimately generates a task-specific skill for a frozen executor. Experiments on ALFWorld, WebShop, and ScienceWorld demonstrate improvements in task success across different model configurations, experience scales, and evaluation settings.

Critical strengths
  1. The problem has practical value, and the main performance gains are clear. Reusing historical experience to guide new tasks is an important challenge for agents operating over extended periods. In Table 1, with Qwen3.5-9B serving as both curator and executor, ExpVoyager achieves success rates (SR) of 63.7%, 28.7%, and 37.4% on ALFWorld, WebShop, and ScienceWorld, respectively, exceeding the best baseline reported for each benchmark by 12.6, 4.5, and 4.8 percentage points. Performance gains also persist when Gemma4-31B is used as the executor.

  2. The experiments are fairly comprehensive and provide intermediate traces that facilitate analysis. Figures 4–7 cover online experience accumulation, experience-pool growth, integration with existing skills, and budget-matched comparisons. Table 3 and Figure 8 examine the two core components. Appendix B.3 and Tables 7–13 provide prompts and interface specifications, while Appendix C.2 presents successful and failed cases. Together, these results support the method’s practical potential and help clarify its behavior and limitations.

Critical weaknesses

Insufficient characterization of end-to-end efficiency

Although ExpVoyager improves task success, its practical efficiency remains unclear. The default setting allows up to 20 navigation rounds, introducing multiple sequential LLM calls for experience access and state updates. According to Table 5, end-to-end token usage increases by approximately 7.43× on ALFWorld and 8.92× on ScienceWorld compared with the no-skill ReAct baseline. While the paper provides budget-matched comparisons, fewer execution steps do not necessarily imply lower overall latency. The lack of wall-clock latency measurements, detailed token and LLM-call breakdowns, and a clear accounting of preprocessing and task-time costs makes it difficult to fully assess the trade-off between performance improvements and computational overhead.

Questions for authors
  1. What is the actual latency? Please report the end-to-end runtime of the proposed method and the main baselines under the same hardware and inference settings, separating pre-execution experience navigation from task execution. This would clarify whether fewer execution steps offset the latency introduced by additional sequential LLM calls.

  2. Where does the additional overhead come from? Please report input/output token usage and LLM call counts separately for action selection, state updates, skill generation, and task execution, together with the actual number of navigation rounds used. Please also clarify the trade-off between success rate and actual runtime under lower navigation budgets.

  3. How are costs fairly accounted for over long-term use? Does the cost comparison consistently include experience-index construction, offline skill construction, and online updates? Please clarify which costs can be reused and amortized across tasks, and report the proposed method’s cumulative cost and success rate relative to the main baselines over a sequence of tasks.

1 comment
AuthorsOpenReview ↗

Follow-up on our responses

Dear reviewer NxW3,

Thank you again for your thoughtful review and constructive feedback. Your comments have helped us better characterize the computational costs and performance–cost trade-offs of our approach. We hope our responses and additional analyses address your concerns. Please let us know if any points remain unaddressed. We would be happy to address them.

Best regards,

Authors of Paper #37719

Reviewer_Qu5YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that turning past agent experience into fixed skills ahead of time throws away information that later tasks may need. As an alternative, ExpVoyager builds a skill for each task at test time: a curator model searches the raw trajectories and writes a skill.md for a frozen executor. In the main experiments, the method beats four skill and memory baselines on ALFWorld, WebShop and ScienceWorld, in some settings by a wide margin. Further experiments cover an online setting, scaling with more experience, combining with existing skills, and a comparison against iterative retrieval at a matched token budget.

Critical strengths
  1. The motivation is clear and well supported. Before presenting the method, the authors build verified oracle skills on ALFWorld. Using an LLM judge, they show that pre-built skills keep under half of the useful knowledge in the source trajectories, and that top-k retrieval reaches only a small part of it.
  2. The gains are consistent, and the authors give comprehensive ablation. Improvements hold across three benchmarks, two executors and three curators. Even a 9B curator helps a 31B executor. Fig. 7 / Table 5 give the baselines an iterative retrieval loop with roughly the same pre-execution token budget. The baselines stay flat or get worse, while ExpVoyager keeps improving up to about 20 rounds.
Critical weaknesses
  1. I am not fully convinced the baselines are as strong as they should be, or that the main comparison is fair on cost. Several baselines do worse than using no skill at all. On WebShop with Qwen3.5-9B, SkillTTA scores 34.3 and ReasoningBank 41.6, while plain ReAct gets 50.3. The budget-matched check covers only ALFWorld and ScienceWorld. It leaves out WebShop, where the baselines look weakest. It also uses only simple iterative top-k as the comparison, not a stronger agentic baseline.
  2. The contribution might be narrower than the "reframing" language suggests. In practice, the method is direct corpus search (as in GrepSeek) applied to trajectories, plus structured working memory. And the "w/o Navigable Interface" ablation (plain grep over trajectory files, with the state kept) still reaches 55.3 SR. That is above every baseline, including SkillTTA at 51.1.
Questions for authors
  1. Why do SkillTTA and RBank fall far below the base agent on WebShop? Did you tune retrieval depth and prompt format for the 9B backbone?
  2. The case study (Fig. 12) shows that the curator conditions on the target's initial observation (e.g., there is no coffee table). Do all baselines, including AWM and RBank retrieval and SkillTTA synthesis, receive the same target context?
2 comments
AuthorsOpenReview ↗

Follow-up on our responses

Dear reviewer Qu5Y,

Thank you again for carefully reviewing our work and providing valuable feedback. Your comments have helped us strengthen our comparisons and clarify the scope of our contribution. We hope our responses and additional experiments address your concerns. If any points remain insufficiently addressed, we would welcome the opportunity to discuss them further.

Best regards,

Authors of Paper #37719

Thank you to the authors for the thoughtful and detailed rebuttal. The responses resolve my concerns. I maintain my positive recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.