- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes APEX, a self-evolving LLM agent framework designed to mitigate exploration collapse, where agents repeatedly exploit familiar strategies and fail to discover better alternatives. APEX maintains an explicit DAG-based strategy map and introduces Fork Discovery to identify unexplored opportunities, Policy Selection to balance exploration and exploitation, and graph-based reward propagation to guide future decisions. Experiments on nine Jericho games and WebArena demonstrate consistent improvements over existing memory-based and self-evolving agent baselines.
Critical strengths
-
The paper identifies exploration collapse as a practical limitation of self-evolving LLM agents, highlighting that accumulating successful experiences does not necessarily lead to better exploration.
-
The combination of strategy maps, Fork Discovery, and uncertainty-aware Policy Selection provides a coherent mechanism for maintaining and expanding the exploration space.
Critical weaknesses
-
The core components largely build upon established ideas, including graph-based planning, frontier exploration, Thompson Sampling, and reward propagation. The main contribution appears to be their integration rather than a fundamentally new algorithm.
-
Prior methods such as Voyager, Go-Explore, and recent graph-based skill planning frameworks share similar motivations or mechanisms. The paper does not sufficiently clarify its unique technical contributions relative to these approaches.
-
Most experiments emphasize final task performance rather than systematically measuring strategy diversity, exploration coverage, or the discovery rate of novel high-reward strategies over extended interactions.
-
The framework relies on LLM-generated milestones and prerequisite dependencies, but provides limited evaluation of incorrect dependencies, redundant milestones, and error propagation as the strategy map grows.
Questions for authors
Can the authors provide direct quantitative evidence that APEX mitigates exploration collapse, such as strategy diversity, exploration coverage, or novel strategy discovery rates over longer interaction horizons?
Official Comment by Authors
Dear Reviewer D6dN,
Thank you again for your thoughtful review and constructive feedback. In our responses, we clarified how APEX constructs and refines dependencies, verifies milestones, and transfers knowledge across tasks. We also provided analyses of dependency correction and milestone outcomes, step-level comparisons between LLM-assigned and environment rewards, and three-seed results to further address your concerns about reliability and robustness.
With around one day remaining in the discussion period, we would greatly appreciate it if you could revisit our responses and let us know whether these clarifications and supplementary results address your concerns. If any questions remain, we would be happy to discuss them before the discussion closes.
Thank you again for your time and consideration.