- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article studies multimodal continuous instruction fine-tuning (MCIT) and argues that existing methods only prevent forgetting in the parameter space and do not utilize external knowledge. The author proposes ReCAP: at each stage, external search and LLM are used to construct domain, inference, and format knowledge cards from the current training data; When reasoning, domain cards serve as prompts and provide ordered capability paths for concatenating LoRA style modules in the shared capability pool. In order to reuse the same capability in multiple stages without overlapping each other, the author proposes adaptive subspace recycling: using a shared base and a stage specific core to parameterize the module, protecting the main direction of historical energy and recycling low-energy directions, and then selecting the stage core based on retrieval clues during inference. On LLaVA-1.5-7B, ReCAP is 3.5 and 2.3 points higher than the strongest baseline SAME on UCIT and CoIN, respectively, and still leads when both are connected to the same knowledge base for vanilla RAG comparison.
Critical strengths
- The idea of transforming external knowledge from "additional context" to "interface for selecting and combining ability modules" is innovative, and the baseline of vanilla RAG, which is also integrated into the knowledge base, has not improved significantly. In most cases, it has slightly decreased, indicating that the benefits mainly come from the way knowledge is used rather than the additional information itself. This is the main reason why I tend to accept it.
- The adaptive subspace recycling design is clear, with strict reparameterization guarantees that do not change historical parameterization. The analysis of the protection ratio also shows that historical energy is concentrated in a few directions, providing a specific and interpretable mechanism for balancing "protection and plasticity".
Critical weaknesses
- After removing the stage core routing, the performance dropped from 70.65 to 31.50, which is much greater than the impact of other components. This indicates that the method relies heavily on identifying "which stage the sample comes from" during inference, while routing uses strong task clues such as format cards and visual similarity. This raises doubts about how much of the benefit comes from implicit task identity recognition, rather than the ability reuse emphasized in the paper.
- The construction of the knowledge base used external search and GPT-4o mini, which are additional resources and calculations that the baseline method does not have. The paper did not report the construction cost, nor did it verify the effectiveness under the condition of only using training data and no external search, which affected the fairness of the comparison with the baseline.
Questions for authors
- What performance can be achieved if the stage core routing is replaced with a simple task identity predictor (such as visual prototype based nearest neighbor)? This can help determine whether the benefits of routing mainly come from task identification.
- What is the result of ReCAP when LLM generates cards using only training data without external search? How many API calls and time are required for building a knowledge base?
- How is the set of abilities S determined and standardized? How are the newly emerged capability names mapped to existing capabilities at different stages?
- Is the main result the average of multiple random seeds? How much is the improvement of 3.5 and 2.3 points relative to the seed variance?
Thank you for the author's response. I have increased my score from 2 to 3.