- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the limitations of existing continual multimodal retrieval evaluations in terms of domain diversity, training protocols, and evaluation settings. The authors introduce SCOPE, an evaluation framework spanning seven heterogeneous visual domains, three continual learning training protocols, and both task-specific and global-gallery evaluation. Using SCOPE, the authors find that the effectiveness of continual learning methods depends strongly on the training and evaluation settings, that simple fine-tuning remains competitive, and that preserving local neighbourhood structure is insufficient to maintain fine-grained retrieval rankings. Additionally, the authors propose Adapter Routing and Consolidation (ARC), which limits cross-task interference through intra-task expert construction, prototype-based routing, and selective adapter consolidation. Experiments show that ARC achieves the highest average retrieval performance across the evaluated training protocols, gallery settings, and CLIP backbones.
Critical strengths
1.The paper demonstrates that methods performing well under task-specific evaluation do not necessarily retain their advantage under global-gallery evaluation. The changes in method rankings reported in Table 3 provide direct empirical evidence that evaluation settings can substantially affect conclusions about continual learning methods. This finding introduces a new experimental standard for research on continuous multimodal retrieval, offering significant research value. 2.ARC combines task-specific low-rank adapters, prototype-based routing, and selective consolidation into a relatively clear framework. Across the experiments involving three training protocols, two gallery settings, and different CLIP backbones, ARC demonstrates consistently strong average retrieval performance. In particular, its performance under global-gallery evaluation suggests that limiting cross-task interference through parameter isolation is a promising direction for further investigation.
Critical weaknesses
1.Insufficient validation of annotation accuracy and semantic uniqueness in the retrieval datasets. Semantic Specificity is one of the key design principles of SCOPE, but the paper lacks a corresponding annotation review or analysis of semantic ambiguity. Particularly under global-gallery evaluation, multiple semantically valid matches may exist across different datasets, while the single-positive Recall@K would count valid matches as incorrect. This could affect the evaluation results. Therefore, the reliability of retrieval results and method rankings under global-gallery evaluation needs further validation. 2.Lack of direct evaluation of catastrophic forgetting. The paper mainly uses Average Final Recall@1 to measure final retrieval performance, but does not report Average Forgetting or complete task-wise retrieval performance changes throughout training. Therefore, the current results do not clearly distinguish whether ARC's improvements come from better adaptation to new tasks or more effective preservation of previously acquired knowledge. They also provide insufficient direct evidence to support the proposed explanation that ARC reduces cross-task interference.
Questions for authors
1.Have the authors conducted any annotation accuracy checks or semantic uniqueness analysis for the image-text pairs in SCOPE? In particular, are there multiple semantically valid matches under global-gallery evaluation? Could these potential annotation ambiguities affect Recall@K and the relative rankings of the evaluated methods? 2.Could the authors provide the changes in retrieval performance for each task throughout continual training and report metrics such as Average Forgetting or Backward Transfer? Would these results further demonstrate that ARC is more effective than the compared methods at reducing forgetting of previously learned tasks, rather than achieving better final retrieval performance?
2 comments
Raise my rating from 2 to 3
The new relaxed evaluation in Appendix G addresses my concern about semantic ambiguity, as the method rankings remain unchanged when potential alternative valid matches are considered. The added Learning Accuracy, Average Forgetting, and per-task retrieval trajectories also address my concern about catastrophic forgetting. These results provide stronger evidence that ARC's improvements mainly come from better knowledge retention rather than better adaptation to individual tasks. However, the semantic ambiguity analysis relies on CLIP similarity, which may not identify all semantically valid matches. A small-scale human annotation check would help further validate the results. In addition, the forgetting analysis is limited to the Separate Protocol and does not cover the Overlapping Protocol. I believe these are areas that could still be improved. I appreciate the substantial improvements made by the authors during the rebuttal. The additional experiments have largely addressed my main concerns. Therefore, I raise my rating from 2 to 3.
Thank you for the thoughtful review and for recognizing that our evaluation findings set a new experimental standard with significant research value and that ARC is a clear framework with consistently strong performance. We address the remaining concerns below.
All SCOPE datasets use their original public annotations, which were chosen for their precise, image-specific captions. To test semantic uniqueness under global-gallery evaluation, we re-scored the retrieval process, also accepting the top-1 items from other datasets that are near-duplicates of the ground truth or paraphrases of the query. These were judged using frozen CLIP, making them independent of the evaluated method.
While such alternative matches do exist, they are rare: even with this lenient criterion, R@1 increases by no more than 2.7 points, and the ranking of all methods remains unchanged. ARC remains in first place even when all cross-dataset errors are counted as correct, and it has the lowest cross-dataset confusion rate of all the methods. We have added this analysis to the revised Appendix G Table 12.
We now report learning accuracy (LA, R@1 on each task right after training on it), average forgetting (F, average drop in R@1 on earlier tasks by the end of the sequence) and final R@1 for all methods under the separate protocol. Under this definition, Backward Transfer equals −F.
ARC learns each task nearly as well as fine-tuning, with an LA of 46.6 compared with 48.3, but forgets almost nothing, with an F of 0.08 compared with 20.95. Its higher final performance therefore comes from retaining earlier tasks, not from learning each task better. Shared-parameter methods that learn well, such as FT, EWC and DKR, forget substantially. MoE and TA also exhibit low forgetting, but achieve significantly lower learning accuracy, 38.8 and 37.0 respectively. The per-task trajectories on an early, an intermediate and a late task show the same pattern. We added the forgetting metrics to Section 4.4, Table 6, and the trajectories to Appendix H, Tables 13–15.
The Seq-LoRA baseline and the cross-dataset error analysis further support this explanation: a single shared LoRA forgets most of each task, and ARC has the lowest share of global-gallery errors that retrieve an item from another dataset, 9.9% compared with 17.7% for fine-tuning. These results are in Appendix E, Table 10 and Appendix G, Table 12 of the revised manuscript.
Summary
We thank the reviewer for acknowledging our findings as setting a new standard for experiments and for recognising ARC's strong performance. We believe the new analyses address both concerns. If our responses have resolved these concerns, we would be grateful if the reviewer would consider raising their score.