acceptodds
Withdrawn from ICLR 2027

Self-Evolving Multimodal Skill Routing with Dual-Timescale Experience Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

External skill libraries expand the capabilities of multimodal agents, but routing queries to the accurate multimodal skills becomes increasingly challenging as these libraries scale. Existing static routers leave post-execution feedback untapped, preventing routing policies from improving with experience. In this work, we formalize self-evolving multimodal skill retrieval, in which a router continually adapts multimodal query and skill representations through execution feedback. Our key insight is that these representations should evolve at different timescales: queries benefit from rapid adaptation leveraging past interactions of similar requests, whereas skill profiles demand stable semantic consolidation across diverse task contexts. Motivated by this, we propose MM-RouterEvolve, a new self-evolving framework that couples query-side and skill-side adaptation through a dual-timescale experience memory. It organizes multimodal queries, retrieved skills, and feedback verifying their compatibility into unified experience units. A bounded short-term memory uses recent routing outcomes to contextualize incoming queries, while a persistent long-term memory consolidates execution feedback to refine skill profiles and distinguish functionally similar skills. This design connects immediate routing adaptation with cumulative refinement of the skill library. Evaluated on two benchmarks, i.e., MM-SkillBench-SE (multimodal) and SkillRouter (text-only), MM-RouterEvolve substantially outperforms static baselines and unilateral query/skill adaptation paradigms, showing that experience-driven self-evolution of both query and skill representations yields more robust routing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Agent, Self-Improving Routing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hvRROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MM-RouterEvolve, a self-evolving multimodal skill routing framework that leverages historical routing feedback to improve skill retrieval. The key idea is a dual-timescale memory mechanism, where short-term query memory retrieves relevant past experiences to enhance query representations, while long-term skill memory accumulates positive and negative feedback to refine skill representations. The authors also introduce MM-SkillBench-SE, a multimodal skill routing benchmark containing approximately 35K skills and corresponding synthetic queries and experiences. Experiments on MM-SkillBench-SE and the text-only SkillRouter benchmark show improvements over static retrieval baselines, while ablation studies demonstrate the benefits of the proposed memory components.

Critical strengths

This paper attempts to improve skill routing through short-term and long-term router updates, addressing a practical need.

Critical weaknesses
  1. Prior works such as ReflecTool[1], R3-Skill[2], and DRAFT[3] have explored experience-based memory for tool selection, negative-supervision-based skill routing, and iterative refinement of tool documentation using execution feedback, respectively. The proposed Query-side and Skill-side Memory mechanisms overlap with these existing approaches to some extent. However, the experiments mainly compare against static retrievers and a simple kNN Memory baseline, without fair comparisons with relevant experience-based or self-evolving methods under the same historical data and feedback budget. Therefore, it remains unclear whether the improvements come from the proposed dual-timescale mechanism or simply from additional supervision and accumulated experiences.

  2. The paper claims to achieve self-evolution by leveraging feedback from successful and failed skill executions. However, the current description seems to rely mainly on determining retrieval correctness based on the ground-truth skill, rather than actually executing skills and verifying their outputs. Moreover, the separation between training data, historical experiences, and test queries is not clearly explained, nor is it clear whether test samples and their feedback labels are reused across multiple evolution rounds. This raises concerns about whether the reported self-evolution genuinely reflects generalization improvements from execution experiences, rather than additional label supervision or memorization of test samples.

[1] Yusheng Liao, Shuyang Jiang, Yanfeng Wang, and Yu Wang. 2025. ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13507–13531, Vienna, Austria. Association for Computational Linguistics.

[2] Wang, Zifei, et al. "Skill Is Not Document: A Query-Conditional Benchmark and Two-Stage Retriever for LLM Agent Skill Routing." arXiv preprint arXiv:2606.03565 (2026).

[3] Qu, Changle, et al. "From exploration to mastery: Enabling llms to master tools via self-driven interactions." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors
  1. The paper lacks sufficient details on the sources, collection and filtering of the MM-SkillBench-SE’s 35K skills, as well as synthetic query generation, quality control, and data splitting. Since the benchmark is one of the main contributions and supports almost all experiments, could the authors provide a complete and reproducible construction protocol?

  2. Several tables (tables 2, 3, 4, and 5) report MRR@10 higher than Recall@10, which contradicts the metric definitions in the paper. Could the authors clarify the metric calculations and relevance labels, and check for possible implementation or reporting errors?

  3. The ablations mainly compare query-side and skill-side Memory, without isolating the effects of Hard Negative Loss or specific query/skill augmentation strategies. Comparisons with simpler methods using the same historical data and feedback budget are also missing. Could the authors provide more fine-grained and fair ablations?

  4. The paper mainly demonstrates self-evolution through performance improvements over evolution rounds, without analyzing how historical experiences affect retrieval decisions. Moreover, Figure 3 does not specify the dataset or the evaluation protocol. Could the authors clarify these settings and provide case studies or quantitative analyses of retrieval changes?

  5. The multimodal experiments rely on a synthetic benchmark, while external validation is limited to the text-only SkillRouter dataset. Could the authors provide experiments on real-world multimodal skill retrieval datasets or discuss this limitation?

  6. Accumulating long-term memory, MLLM-based experience filtering, and dynamically updating skill embeddings introduce additional overhead, but their costs are not reported. Could the authors provide efficiency and scalability analyses, including latency, computation, and memory usage?

  7. Offline Retriever Fine-tuning focuses on conventional query–skill matching, while self-evolution augments the inputs with historical experiences, potentially introducing a training-inference distribution mismatch. Could the authors evaluate self-evolution directly on a pretrained embedding model without task-specific fine-tuning, and clarify the relationship between these two stages and the necessity of fine-tuning?

Reviewer_mJagOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MM-RouterEvolve, a framework that uses execution feedback to improve multimodal skill retrieval over time. The main idea is to update both query and skill representations through a dual-timescale memory mechanism. Short-term memory uses recent interactions to better interpret incoming queries, while long-term memory accumulates feedback to refine skill descriptions. The authors also introduce MM-SkillBench-SE, a benchmark containing approximately 35K skills with multimodal queries and execution feedback. Experiments on multimodal and text-only benchmarks show improvements over static retrieval methods and single-sided adaptation approaches.

Critical strengths

The paper addresses a practical limitation of existing skill retrieval methods, which typically do not use execution feedback to improve future routing decisions.

The proposed MM-SkillBench-SE benchmark provides a useful setting for evaluating multimodal skill retrieval with execution feedback, particularly as the number of available skills increases.

Critical weaknesses

Table 3 shows that combining query-side and skill-side adaptation improves performance. However, this does not fully justify the need for two separate memory mechanisms with different update timescales. A comparison with a unified memory that updates both representations would help establish the benefit of this design.

The main improvements are reported on the newly constructed MM-SkillBench-SE benchmark, while the gains on the existing SkillRouter benchmark are relatively small.

The proposed framework requires additional memory retrieval, MLLM-based experience verification, and skill representation updates. However, the experiments do not report inference latency, memory usage, or the cost of maintaining the system as the skill library grows. These factors are important for practical deployment.

Questions for authors
  1. Why must query and skill representations use different timescales? The paper assumes queries require rapid adaptation and skills require stable refinement, but this seems like a design choice rather than an established requirement.

  2. Why not use a single experience memory? A unified memory could retrieve relevant past query–skill–feedback examples and use them to improve both representations. The authors need to demonstrate why separating the memories is necessary.

  3. What exactly is evolving? If the system only retrieves previous experiences and rewrites query/skill descriptions using an LLM, the claimed self-evolution may be closer to retrieval-augmented contextualization and profile updating than genuine learning.

Reviewer_PyqTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a self-evolving multimodal skill-routing framework called MM-RouterEvolve, designed to address the challenge of achieving precise matching when multimodal agents access large-scale external skill libraries. Existing static routers often overlook feedback generated after task execution, causing systems to repeat errors when faced with similar queries, that is, their strategies fail to improve with experience. This research transforms execution feedback into proactive evolutionary signals, noting that the representations of queries and skills require updates on different timescales.

Critical strengths
  1. Utilizing a mechanism with limited capacity, this component leverages routing verification results from recent, similar requests to provide immediate context for new queries, helping to resolve ambiguities in a user’s vague intent.

  2. Employing persistent storage, this component preserves the original semantics of skills while integrating execution feedback—evaluated by a Multimodal Large Language Model (MLLM) across various task environments—to iteratively refine skill profiles, thereby distinguishing between candidate skills with similar functions.

  3. The framework encapsulates multimodal queries, retrieved skills, and the feedback (success or failure) from the verification process into structured "experience units." These units collectively drive the evolution of both query and skill representations, all without requiring online model fine-tuning.

  4. The authors have constructed a large-scale benchmark dataset containing approximately 35,000 skills. Experiments demonstrate that MM-RouterEvolve significantly outperforms existing static multimodal retrieval baselines (such as VLM2Vec, Qwen3VL-Emb, etc.) as well as routing methods that rely solely on one-sided adaptation, across both multimodal benchmarks (MM-SkillBench-SE) and text-only benchmarks (SkillRouter). Within this framework, positive feedback (from successful outcomes) and negative feedback (from unsuccessful ones) play complementary roles in the self-evolution process, facilitating both error correction and performance reinforcement.

Critical weaknesses

*1.The framework's reliance on a Multimodal Large Language Model (MLLM) to act as a gatekeeping controller for integrating feedback into the long-term skill memory introduces substantial computational overhead and inference latency. This heavy dependency potentially compromises the system's scalability and real-time responsiveness in high-throughput, large-scale agent deployments.

  1. The empirical evaluation of the self-evolution mechanism is constrained to a relatively limited number of discrete evolutionary rounds (e.g., 19 rounds). The study fails to investigate the system's robustness over highly extended, continuous interactive lifecycles, leaving critical questions unanswered regarding potential noise accumulation, experience saturation, and concept drift over time.

  2. While the proposed MM-RouterEvolve demonstrates clear superiority over static retrievers and a naive kNN-based memory baseline, the evaluation lacks comparisons against other contemporary,SOTA dynamic agent self-improvement and workflow optimization frameworks.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.