- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SPMR for continual tool retrieval, with the goal of preserving historical tool retrieval performance as new tools are introduced incrementally. The authors break forgetting down into Storage, Access, and Competition, and develop a framework that combines box-indexed structured prompt memory, geometric routing, and adaptive multi-path recovery.
Critical strengths
Meaningful problem: Continual tool retrieval is a relevant problem for evolving LLM agents, since newly introduced tools may interfere with the retrieval of existing ones.
Interesting perspective: The Storage–Access–Competition decomposition offers an intuitive perspective on retrieval forgetting that goes beyond conventional parameter preservation.
Useful ablations: Experiments on ToolBench and ToolRet show that box-based routing and multi-path aggregation improve retrieval performance relative to their simplified variants.
Critical weaknesses
Limited empirical competitiveness: SPMR is substantially outperformed by CODA-Prompt on both retrieval accuracy and forgetting metrics across both benchmarks. Additional results in Appendix B further show that SPMR often ranks behind several baselines on Recall@5 and NDCG@5, which makes its overall empirical advantages unconvincing.
Insufficient validation of the core motivation: Although the paper decomposes forgetting into Storage, Access, and Competition, it lacks controlled experiments that quantify each component's individual contribution to retrieval failures. In addition, the proposed multi-path recovery resembles conventional top-k routing and prediction ensembling, which makes its distinct methodological contribution unclear.
Incomplete comparison with recent CL methods: The evaluated baselines mainly cover prompt-based and traditional continual learning approaches, overlooking recent LoRA-based methods such as ProgLoRA and JANUS-LoRA. Discussing these methods and evaluating appropriately adapted parameter-efficient baselines would strengthen the experimental comparison.
Unclear practical advantages: Despite its substantially lower accuracy than CODA-Prompt, SPMR offers relatively modest persistent-memory savings while requiring multiple backbone forward passes during inference. The paper lacks sufficient latency and computational-cost analysis to justify this trade-off, as well as comparisons with non-parametric retrievers that can incorporate new tools through index updates.
Questions for authors
Performance–Memory trade-off: Table 1 shows that CODA-Prompt substantially outperforms SPMR while requiring only slightly more persistent memory. Could the authors compare the two methods under a matched memory budget to determine whether SPMR offers a meaningful trade-off?
Forgetting mechanism: Figure 3 provides examples of multi-path recovery, but how frequently do Storage, Access, and Competition failures occur across the evaluation sets? What proportion of recovered queries can be specifically attributed to correcting routing failures?
Comparison with recent CL methods: Could the authors clarify how SPMR differs from recent parameter-efficient continual learning methods such as ProgLoRA, and whether its structured prompt memory offers advantages over an appropriately adapted LoRA-based approach?