acceptodds
Under review as a conference paper at ICLR 2027

ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning

Abstract

Multimodal continual instruction tuning (MCIT) aims to enable multimodal large language models to acquire new capabilities from sequential tasks while preserving previously learned knowledge. Existing methods primarily mitigate catastrophic forgetting by constraining parameter updates or separating task-specific adaptations. However, continual adaptation can also benefit from external knowledge that provides domain-specific information and reusable reasoning patterns for solving diverse instructions. For example, to answer "How many red cubes are to the left of the sphere?", domain knowledge can provide relevant concepts about objects and spatial relations, while reasoning knowledge can specify ordered operations such as object recognition, spatial filtering, and counting. Despite this potential, how to leverage external knowledge for continual adaptation remains largely unexplored in existing MCIT methods. To this end, we propose ReCAP, a retrieval-guided framework that leverages external knowledge to guide capability reuse during continual adaptation. At each continual stage, ReCAP uses external search and an LLM to incrementally build a knowledge base of domain, reasoning, and format knowledge based on the current-stage training data. For each instruction, retrieved domain knowledge guides generation, while retrieved reasoning knowledge selects and orders capability modules to form an instance-specific capability path. As these capability modules are reused across stages, subsequent adaptation can overwrite previously learned parameters. To enable stable cross-stage reuse, ReCAP introduces adaptive subspace recycling, which parameterizes reusable capability modules with shared bases and stage-specific cores, protects historically important directions while recycling residual capacity. Extensive experiments on MCIT benchmarks show that ReCAP achieves SOTA performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
multimodal large language model, computer vision, continual learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MmLuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article studies multimodal continuous instruction fine-tuning (MCIT) and argues that existing methods only prevent forgetting in the parameter space and do not utilize external knowledge. The author proposes ReCAP: at each stage, external search and LLM are used to construct domain, inference, and format knowledge cards from the current training data; When reasoning, domain cards serve as prompts and provide ordered capability paths for concatenating LoRA style modules in the shared capability pool. In order to reuse the same capability in multiple stages without overlapping each other, the author proposes adaptive subspace recycling: using a shared base and a stage specific core to parameterize the module, protecting the main direction of historical energy and recycling low-energy directions, and then selecting the stage core based on retrieval clues during inference. On LLaVA-1.5-7B, ReCAP is 3.5 and 2.3 points higher than the strongest baseline SAME on UCIT and CoIN, respectively, and still leads when both are connected to the same knowledge base for vanilla RAG comparison.

Critical strengths
  • The idea of transforming external knowledge from "additional context" to "interface for selecting and combining ability modules" is innovative, and the baseline of vanilla RAG, which is also integrated into the knowledge base, has not improved significantly. In most cases, it has slightly decreased, indicating that the benefits mainly come from the way knowledge is used rather than the additional information itself. This is the main reason why I tend to accept it.
  • The adaptive subspace recycling design is clear, with strict reparameterization guarantees that do not change historical parameterization. The analysis of the protection ratio also shows that historical energy is concentrated in a few directions, providing a specific and interpretable mechanism for balancing "protection and plasticity".
Critical weaknesses
  • After removing the stage core routing, the performance dropped from 70.65 to 31.50, which is much greater than the impact of other components. This indicates that the method relies heavily on identifying "which stage the sample comes from" during inference, while routing uses strong task clues such as format cards and visual similarity. This raises doubts about how much of the benefit comes from implicit task identity recognition, rather than the ability reuse emphasized in the paper.
  • The construction of the knowledge base used external search and GPT-4o mini, which are additional resources and calculations that the baseline method does not have. The paper did not report the construction cost, nor did it verify the effectiveness under the condition of only using training data and no external search, which affected the fairness of the comparison with the baseline.
Questions for authors
  • What performance can be achieved if the stage core routing is replaced with a simple task identity predictor (such as visual prototype based nearest neighbor)? This can help determine whether the benefits of routing mainly come from task identification.
  • What is the result of ReCAP when LLM generates cards using only training data without external search? How many API calls and time are required for building a knowledge base?
  • How is the set of abilities S determined and standardized? How are the newly emerged capability names mapped to existing capabilities at different stages?
  • Is the main result the average of multiple random seeds? How much is the improvement of 3.5 and 2.3 points relative to the seed variance?
1 comment

Thank you for the author's response. I have increased my score from 2 to 3.

Reviewer_bkmCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReCAP, a retrieval-guided framework for multimodal continual instruction tuning. At each stage, it constructs domain, reasoning, and format knowledge cards from the current training instructions, external search results, and an LLM. Retrieved domain knowledge provides generation guidance, while reasoning cards specify an ordered sequence of capability modules. These modules use shared bases and stage-specific cores, with adaptive subspace recycling to protect historically important directions and retain capacity for subsequent learning. At inference time, the model selects a stored core for each activated capability. Experiments on UCIT and CoIN evaluate the overall framework, its components, and its computational costs.

Critical strengths
  1. The framework makes an interesting connection between retrieval and continual adaptation. The three types of knowledge cards have distinct roles, and reasoning cards influence which modules are used and how they are composed. This gives retrieval a more substantive role than simply supplying additional context. The shared-basis, stage-specific-core design also offers a concrete way to balance reuse with preservation.

  2. The results are competitive, and the knowledge-access controls are useful. ReCAP achieves final average scores of 70.65 on UCIT and 69.10 on CoIN, compared with 67.12 and 66.82 for SAME. Importantly, the paper also evaluates LoRA-FT and SAME with access to the same knowledge base during both training and evaluation. These comparisons help establish that the gains are not explained solely by access to external information. The training and inference measurements further clarify the practical trade-offs.

Critical weaknesses
  1. The contribution of semantic capability composition is not yet sufficiently distinguished from that of core routing. In Table 3, removing domain guidance reduces performance from 70.65 to 70.02, and replacing ordered composition with parallel composition reduces it to 70.08. Removing stage-specific core routing produces a much larger drop, to 31.50. These results establish the importance of choosing appropriate historical parameters, but they provide less direct evidence that the semantic ordering supplied by reasoning cards is responsible for the improvement. The sequential-versus-parallel comparison changes the computational structure as well as the use of ordering.

Similarly, the routing accuracy in Appendix C.1 measures agreement with a reference realization associated with the task and capability. This is a useful measure of core selection, but it does not directly establish that modules have learned reusable operations corresponding to their semantic descriptions. A simpler instruction-cluster routing baseline, together with a controlled test of path ordering, would help clarify the distinctive contribution of capability reuse.

  1. The role of low-energy recycling in preserving historical behavior needs stronger evidence. Appendix E establishes exact preservation during reparameterization and rank expansion, as well as a bound on historical energy outside the protected subspaces. These properties motivate the construction, but they do not characterize how historical behavior changes during subsequent training. Even with a frozen historical core CτC_\tau, updating the shared bases changes the historical operator UCτVUC_\tau V. Low initial energy in the recyclable directions alone does not bound the effects of later updates.

The retention results are encouraging, but the current ablation removes adaptive subspace recycling as a whole. Separating protection, recycling, and expansion would give a clearer account of why this mechanism works.

Questions for authors
  1. With the activated modules and path length held fixed, how does the retrieved ordering compare with random or reversed orderings?
  2. How does ReCAP compare with a simpler instruction-cluster routing approach that retains the same stage-specific cores but does not use semantically defined capability paths?
  3. Which components remain in the “w/o Adaptive Subspace Recycling” variant? Could the authors also show how historical module outputs change during recycling?
Reviewer_M5JjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReCAP for multimodal continual instruction tuning. The method retrieves knowledge cards to provide domain guidance and to select and order reusable capability modules. It then combines a shared basis, stage-specific cores, and adaptive subspace recycling to preserve previously acquired capabilities while learning new tasks. The paper validates the overall performance on UCIT and CoIN benchmarks, and further provides component ablations, sensitivity analyses, and efficiency comparisons.

Critical strengths
  1. Integrating retrieval with parameter adaptation is of research value. Using retrieved reasoning steps to organize capability modules lets external knowledge participate in structuring the model's computation. Moreover, the shared basis and stage-specific cores are a well-motivated design for sharing capabilities and retaining previously acquired ones.
  2. The overall experimental results are reasonably convincing. On UCIT and CoIN, the final average performance exceeds that of SAME by 3.53 and 2.28 points, respectively, and the final average forgetting on UCIT is 0.70. The RAG baseline built on the same knowledge base and the efficiency comparisons in the appendix further support the overall effectiveness of the method.
Critical weaknesses
  1. The current experiments do not establish that the gains come from the emphasized capability reuse and reasoning order. In Table 3, removing domain guidance or switching to parallel composition reduces performance by only 0.63 and 0.57 points, respectively, whereas the impact of stage-core routing is far greater. The routing accuracy in Appendix C.1 measures whether inputs are routed back to the core of their corresponding task, which does not directly show that cross-task capability composition is effective. The paper currently lacks (i) a control with identical routing and a comparable parameter budget but no capability sharing, and (ii) a control that keeps the module set fixed and only shuffles the execution order. Consequently, attributing the performance gains to the paper's key contributions remains insufficiently supported.
  2. The knowledge base is central to the method, yet the paper gives too little information to reproduce it. Although the appendix provides the generation prompts, it does not specify the clustering algorithm and its key thresholds, or the number of cards and the size of the capability pool at each stage. Nor does it provide verifiable retrieval sources or a snapshot of the knowledge base. Since these choices directly determine capability partitioning, retrieval, and routing, it is difficult to assess how much the results depend on how the knowledge base was constructed.
  3. The authors conduct experiments only on LLaVA-v1.5-7B (released in 2023) and do not evaluate additional architectures or more recent state-of-the-art models. Consequently, the current results cannot demonstrate the generalizability of the proposed method.
Questions for authors
  1. Could the authors add the controlled comparisons described above, namely the "no capability sharing" and "shuffled module order" settings, and report the variability across repeated runs? These results would directly bear on the assessment of how much capability reuse and ordered composition contribute.
  2. Could the authors provide the complete knowledge-base construction configuration and the number of cards and capabilities at each stage, and release a snapshot of the knowledge base with provenance records? In addition, do the costs reported in Tables 9–10 include knowledge-base construction, as well as the encoding and retrieval overhead of Qwen3-Embedding-4B?
  3. Could the authors add more SOTA LMMs on their method implementation?
1 comment

Thanks for the authors' response. It has addressed most of my major concerns. Whether the proposed method can generalize well to larger models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.