- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes MM-RouterEvolve, a self-evolving multimodal skill routing framework that leverages historical routing feedback to improve skill retrieval. The key idea is a dual-timescale memory mechanism, where short-term query memory retrieves relevant past experiences to enhance query representations, while long-term skill memory accumulates positive and negative feedback to refine skill representations. The authors also introduce MM-SkillBench-SE, a multimodal skill routing benchmark containing approximately 35K skills and corresponding synthetic queries and experiences. Experiments on MM-SkillBench-SE and the text-only SkillRouter benchmark show improvements over static retrieval baselines, while ablation studies demonstrate the benefits of the proposed memory components.
Critical strengths
This paper attempts to improve skill routing through short-term and long-term router updates, addressing a practical need.
Critical weaknesses
-
Prior works such as ReflecTool[1], R3-Skill[2], and DRAFT[3] have explored experience-based memory for tool selection, negative-supervision-based skill routing, and iterative refinement of tool documentation using execution feedback, respectively. The proposed Query-side and Skill-side Memory mechanisms overlap with these existing approaches to some extent. However, the experiments mainly compare against static retrievers and a simple kNN Memory baseline, without fair comparisons with relevant experience-based or self-evolving methods under the same historical data and feedback budget. Therefore, it remains unclear whether the improvements come from the proposed dual-timescale mechanism or simply from additional supervision and accumulated experiences.
-
The paper claims to achieve self-evolution by leveraging feedback from successful and failed skill executions. However, the current description seems to rely mainly on determining retrieval correctness based on the ground-truth skill, rather than actually executing skills and verifying their outputs. Moreover, the separation between training data, historical experiences, and test queries is not clearly explained, nor is it clear whether test samples and their feedback labels are reused across multiple evolution rounds. This raises concerns about whether the reported self-evolution genuinely reflects generalization improvements from execution experiences, rather than additional label supervision or memorization of test samples.
[1] Yusheng Liao, Shuyang Jiang, Yanfeng Wang, and Yu Wang. 2025. ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13507–13531, Vienna, Austria. Association for Computational Linguistics.
[2] Wang, Zifei, et al. "Skill Is Not Document: A Query-Conditional Benchmark and Two-Stage Retriever for LLM Agent Skill Routing." arXiv preprint arXiv:2606.03565 (2026).
[3] Qu, Changle, et al. "From exploration to mastery: Enabling llms to master tools via self-driven interactions." International Conference on Learning Representations. Vol. 2025. 2025.
Questions for authors
-
The paper lacks sufficient details on the sources, collection and filtering of the MM-SkillBench-SE’s 35K skills, as well as synthetic query generation, quality control, and data splitting. Since the benchmark is one of the main contributions and supports almost all experiments, could the authors provide a complete and reproducible construction protocol?
-
Several tables (tables 2, 3, 4, and 5) report MRR@10 higher than Recall@10, which contradicts the metric definitions in the paper. Could the authors clarify the metric calculations and relevance labels, and check for possible implementation or reporting errors?
-
The ablations mainly compare query-side and skill-side Memory, without isolating the effects of Hard Negative Loss or specific query/skill augmentation strategies. Comparisons with simpler methods using the same historical data and feedback budget are also missing. Could the authors provide more fine-grained and fair ablations?
-
The paper mainly demonstrates self-evolution through performance improvements over evolution rounds, without analyzing how historical experiences affect retrieval decisions. Moreover, Figure 3 does not specify the dataset or the evaluation protocol. Could the authors clarify these settings and provide case studies or quantitative analyses of retrieval changes?
-
The multimodal experiments rely on a synthetic benchmark, while external validation is limited to the text-only SkillRouter dataset. Could the authors provide experiments on real-world multimodal skill retrieval datasets or discuss this limitation?
-
Accumulating long-term memory, MLLM-based experience filtering, and dynamically updating skill embeddings introduce additional overhead, but their costs are not reported. Could the authors provide efficiency and scalability analyses, including latency, computation, and memory usage?
-
Offline Retriever Fine-tuning focuses on conventional query–skill matching, while self-evolution augments the inputs with historical experiences, potentially introducing a training-inference distribution mismatch. Could the authors evaluate self-evolution directly on a pretrained embedding model without task-specific fine-tuning, and clarify the relationship between these two stages and the necessity of fine-tuning?