acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Continual Learning for Multimodal Retrieval

Abstract

Retrieval is a core capability of vision-language models (VLMs), yet continual adaptation for retrieval remains underexplored, with existing evaluations largely restricted to narrow domains and a single training protocol. We introduce SCOPE, a reproducible framework for continual multimodal retrieval spanning seven heterogeneous visual domains, three training protocols, and both task-specific and global-gallery evaluation. Using SCOPE, we find that conclusions about continual learning (CL) methods depend strongly on the evaluation setting: method rankings change across training protocols, task-specific galleries conceal cross-task retrieval failures, and simple fine-tuning remains competitive with dedicated CL methods. Our analysis further shows that preserving local neighbourhood structure is insufficient to preserve fine-grained retrieval rankings. As a reference method for SCOPE, we introduce Adapter Routing and Consolidation (ARC), which limits cross-task interference through low-rank adapters and selective consolidation. ARC consistently outperforms the evaluated methods across SCOPE settings. Together, these results establish SCOPE as a comprehensive evaluation framework and highlight the importance of diverse training protocols and gallery settings for reliably assessing progress in continual multimodal retrieval.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, retrieval, multimodality, clip, evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_tP1ROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limitations of existing continual multimodal retrieval evaluations in terms of domain diversity, training protocols, and evaluation settings. The authors introduce SCOPE, an evaluation framework spanning seven heterogeneous visual domains, three continual learning training protocols, and both task-specific and global-gallery evaluation. Using SCOPE, the authors find that the effectiveness of continual learning methods depends strongly on the training and evaluation settings, that simple fine-tuning remains competitive, and that preserving local neighbourhood structure is insufficient to maintain fine-grained retrieval rankings. Additionally, the authors propose Adapter Routing and Consolidation (ARC), which limits cross-task interference through intra-task expert construction, prototype-based routing, and selective adapter consolidation. Experiments show that ARC achieves the highest average retrieval performance across the evaluated training protocols, gallery settings, and CLIP backbones.

Critical strengths

1.The paper demonstrates that methods performing well under task-specific evaluation do not necessarily retain their advantage under global-gallery evaluation. The changes in method rankings reported in Table 3 provide direct empirical evidence that evaluation settings can substantially affect conclusions about continual learning methods. This finding introduces a new experimental standard for research on continuous multimodal retrieval, offering significant research value. 2.ARC combines task-specific low-rank adapters, prototype-based routing, and selective consolidation into a relatively clear framework. Across the experiments involving three training protocols, two gallery settings, and different CLIP backbones, ARC demonstrates consistently strong average retrieval performance. In particular, its performance under global-gallery evaluation suggests that limiting cross-task interference through parameter isolation is a promising direction for further investigation.

Critical weaknesses

1.Insufficient validation of annotation accuracy and semantic uniqueness in the retrieval datasets. Semantic Specificity is one of the key design principles of SCOPE, but the paper lacks a corresponding annotation review or analysis of semantic ambiguity. Particularly under global-gallery evaluation, multiple semantically valid matches may exist across different datasets, while the single-positive Recall@K would count valid matches as incorrect. This could affect the evaluation results. Therefore, the reliability of retrieval results and method rankings under global-gallery evaluation needs further validation. 2.Lack of direct evaluation of catastrophic forgetting. The paper mainly uses Average Final Recall@1 to measure final retrieval performance, but does not report Average Forgetting or complete task-wise retrieval performance changes throughout training. Therefore, the current results do not clearly distinguish whether ARC's improvements come from better adaptation to new tasks or more effective preservation of previously acquired knowledge. They also provide insufficient direct evidence to support the proposed explanation that ARC reduces cross-task interference.

Questions for authors

1.Have the authors conducted any annotation accuracy checks or semantic uniqueness analysis for the image-text pairs in SCOPE? In particular, are there multiple semantically valid matches under global-gallery evaluation? Could these potential annotation ambiguities affect Recall@K and the relative rankings of the evaluated methods? 2.Could the authors provide the changes in retrieval performance for each task throughout continual training and report metrics such as Average Forgetting or Backward Transfer? Would these results further demonstrate that ARC is more effective than the compared methods at reducing forgetting of previously learned tasks, rather than achieving better final retrieval performance?

2 comments

Thank you for the thoughtful review and for recognizing that our evaluation findings set a new experimental standard with significant research value and that ARC is a clear framework with consistently strong performance. We address the remaining concerns below.

(W1 & Q1) Have the authors conducted any annotation accuracy checks or semantic uniqueness analysis for the image-text pairs in SCOPE? In particular, are there multiple semantically valid matches under global-gallery evaluation? Could these potential annotation ambiguities affect Recall@K and the relative rankings of the evaluated methods?

All SCOPE datasets use their original public annotations, which were chosen for their precise, image-specific captions. To test semantic uniqueness under global-gallery evaluation, we re-scored the retrieval process, also accepting the top-1 items from other datasets that are near-duplicates of the ground truth or paraphrases of the query. These were judged using frozen CLIP, making them independent of the evaluated method.

MethodStrict R@1Relaxed: Near-dup.Relaxed: ParaphraseCross-dataset err. (%) ↓Upper bound
Zero-shot24.925.426.711.936.8
FT26.627.028.517.744.3
MoE33.834.536.113.447.2
ARC43.043.545.09.952.9

While such alternative matches do exist, they are rare: even with this lenient criterion, R@1 increases by no more than 2.7 points, and the ranking of all methods remains unchanged. ARC remains in first place even when all cross-dataset errors are counted as correct, and it has the lowest cross-dataset confusion rate of all the methods. We have added this analysis to the revised Appendix G Table 12.

(W2 & Q2) Lack of direct evaluation of catastrophic forgetting. Insufficient direct evidence to support the proposed explanation that ARC reduces cross-task interference.

We now report learning accuracy (LA, R@1 on each task right after training on it), average forgetting (F, average drop in R@1 on earlier tasks by the end of the sequence) and final R@1 for all methods under the separate protocol. Under this definition, Backward Transfer equals −F.

MethodLA (↑)F (↓)Final (↑)
FT48.2620.9530.30
TA37.044.5233.17
EWC47.4517.0932.81
Mod-X46.7013.7334.93
C-CLIP41.007.1734.88
L2P34.2410.2625.45
DKR48.1221.2629.89
MoE38.752.0137.03
ARC46.640.0846.58

ARC learns each task nearly as well as fine-tuning, with an LA of 46.6 compared with 48.3, but forgets almost nothing, with an F of 0.08 compared with 20.95. Its higher final performance therefore comes from retaining earlier tasks, not from learning each task better. Shared-parameter methods that learn well, such as FT, EWC and DKR, forget substantially. MoE and TA also exhibit low forgetting, but achieve significantly lower learning accuracy, 38.8 and 37.0 respectively. The per-task trajectories on an early, an intermediate and a late task show the same pattern. We added the forgetting metrics to Section 4.4, Table 6, and the trajectories to Appendix H, Tables 13–15.

The Seq-LoRA baseline and the cross-dataset error analysis further support this explanation: a single shared LoRA forgets most of each task, and ARC has the lowest share of global-gallery errors that retrieve an item from another dataset, 9.9% compared with 17.7% for fine-tuning. These results are in Appendix E, Table 10 and Appendix G, Table 12 of the revised manuscript.

Summary

We thank the reviewer for acknowledging our findings as setting a new standard for experiments and for recognising ARC's strong performance. We believe the new analyses address both concerns. If our responses have resolved these concerns, we would be grateful if the reviewer would consider raising their score.

Reviewer_tP1ROpenReview ↗

Raise my rating from 2 to 3

The new relaxed evaluation in Appendix G addresses my concern about semantic ambiguity, as the method rankings remain unchanged when potential alternative valid matches are considered. The added Learning Accuracy, Average Forgetting, and per-task retrieval trajectories also address my concern about catastrophic forgetting. These results provide stronger evidence that ARC's improvements mainly come from better knowledge retention rather than better adaptation to individual tasks. However, the semantic ambiguity analysis relies on CLIP similarity, which may not identify all semantically valid matches. A small-scale human annotation check would help further validate the results. In addition, the forgetting analysis is limited to the Separate Protocol and does not cover the Overlapping Protocol. I believe these are areas that could still be improved. I appreciate the substantial improvements made by the authors during the rebuttal. The additional experiments have largely addressed my main concerns. Therefore, I raise my rating from 2 to 3.

Reviewer_WGduOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCOPE, an evaluation framework for continual image-text retrieval with CLIP. Seven datasets from heterogeneous visual domains are ordered by zero-shot difficulty and arranged in three training protocols. Either as one dataset per task, tasks mixing two domains, or a long sequence that repeats every domain. The final average Recall@1 is measured either against the task-specific gallery or against a global gallery encompassing all tasks. Held-out COCO retrieval and zero-shot classification are measured along the sequence of tasks. Nine methods are compared on CLIP ViT-B/16 and a subset of them on two other backbones. The authors also propose Adapter Routing and Consolidation (ARC), which clusters each task in the frozen CLIP space and trains a LoRA expert per cluster. An expert is selected for each query-candidate pair by its prototype similarity, and the similar experts are merged via CoreSpace. ARC achieves the highest Recall@1 in all evaluated settings.

Critical strengths

S1) The comparison between task-specific and global-gallery evaluation in Table 3 is a valuable contribution. Merging the galleries changes the ranking of the methods considerably, such that MoE drops from the second place to the lower half, which reveals cross-task retrieval failures hidden by the commonly used task-specific evaluation.

S2) ARC does not require the knowledge of the task at inference and achieves the best final R@1 across all protocols, gallery settings and backbones, with a noticeable margin over the best baseline in Table 2.

Critical weaknesses

W1) The benchmark is not sufficiently specified. The dataset sizes, split construction, number of test queries, gallery sizes and caption selection are not reported. In particular, it is unclear how the multiple captions per image in Flickr30K are handled under the single-positive formulation of Section 3.1. For a paper where main contribution is a reproducible evaluation framework, these are essential details.

W2) All experiments use a single seed, so the finer comparisons are unreliable. Several of the reported rank changes rest on differences below 0.5 of R@1. It is also unclear how the hyperparameters were selected. Appendix E mentions k and tau from an ablation on the overlapping protocol without stating the split, and the tuning of the baselines is not documented at all, so the fairness of the comparison is hard to assess.

W3) The source of ARC gains is unclear. The final R@1 depends on how well each task was learned and how much was forgotten afterwards. Since only the final number is reported, it is hard to tell whether ARC learns better or forgets less. A sequential-LoRA baseline would be good to see. Furthermore, the domain order is never varied.

W4) The cost of the pair-dependent routing is not quantified. Eq. 3 selects the expert per query-candidate pair, so the scoring does not fit standard single-index retrieval with one embedding per gallery item. Appendix B.1 acknowledges multiple query encodings and one gallery copy per expert, but no latency or memory is measured and no scalable implementation (e.g., expert-specific indexes or ARC as a re-ranker) is evaluated.

Minor weaknesses:

MW1) In Figure 3, FT has a higher mean R@1 than MoE with ViT-B/32 but is ranked lower. In Table 4, the definitions in Appendix F imply MRR >= R@1 + (R@5 - R@1)/5, but the reported values of FT and Mod-X violate this.

Questions for authors

Q1) Please provide the dataset and split statistics, the query and gallery sizes, and the handling of multiple captions per image.

Q2) Please report means and standard deviations over several seeds. How were the hyperparameters of all methods selected, and how was MoE adapted to retrieval?

Q3) Can you provide per-task retrieval trajectories, and a sequential-LoRA baseline. And to see how robust are the results to a different domain order.

Q4) What are the query latency, memory and gallery storage of ARC relative to fine-tuning.

5 comments

Thank you for the constructive review and for recognizing that our task-specific vs. global-gallery comparison is a valuable contribution. We address the remaining concerns below.

(W2 & Q2) Please report means and standard deviations over several seeds.

Multiple random seeds. We re-ran all methods with three seeds under the separate protocol and now report the mean and standard deviation for every method. ARC’s lead over the next-best method is 9.9 R@1 for T->I and 8.5 for I->T, more than 11 times the larger standard deviation of the two methods, so its first place does not depend on the seed. A few adjacent baselines, such as Mod-X and C-CLIP (0.03 apart on average), differ by less than their seed-to-seed variation, which is consistent with our finding that rankings among existing methods are fragile. All methods now use the same hyperparameter-tuning and training budgets, and we have updated all results in the paper accordingly. We will run more seeds for other protocols in any final version of this work. The results of the multi-seed runs have now been added to Table 16 in Appendix I.

MethodT→II→T
FT30.38 (0.10)30.50 (0.23)
TA32.70 (0.03)33.70 (0.07)
EWC32.57 (0.15)33.17 (0.05)
Mod-X34.54 (0.09)35.41 (0.08)
C-CLIP34.03 (0.09)35.86 (0.10)
L2P25.27 (0.36)23.86 (1.60)
DKR30.43 (0.41)29.95 (0.29)
MoE35.84 (0.21)38.20 (0.12)
ARC45.77 (0.52)46.73 (0.74)

(W2 & Q2) How were the hyperparameters of all methods selected, and how was MoE adapted to retrieval?

MoE implementation. For MoE, we follow the original paper (Yu et al., “Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters”) but replace the classification loss with the symmetric CLIP contrastive loss and use frozen CLIP image embeddings in the DDAS task selector. Each image is routed independently and captions are encoded with the frozen text encoder. We have now added these details to Appendix A.

Hyperparameters selection. For each method, we tuned the learning rate, weight decay and each method's own regularisation or architecture parameters. Following the reviewer's suggestion we provide more detailed hyperparameter setup in Appendix A.

(W3 & Q3) Can you provide per-task retrieval trajectories, and a sequential-LoRA baseline. And to see how robust are the results to a different domain order.

Trajectories. We added per-task learning trajectories for an early, an intermediate and a late task. The table below shows R@1 on Flickr30K, the first task, after training on each subsequent task under the separate protocol. R@1 is averaged over I2T and T2I, and Drop is the difference between R@1 right after training on Flickr30K and at the end of the sequence.

MethodT1T2T3T4T5T6T7Drop (↓)
FT86.986.482.681.479.577.070.816.1
TA86.283.381.880.580.079.778.97.3
EWC86.786.383.783.281.478.475.611.1
Mod-X86.385.782.882.881.580.377.09.3
C-CLIP83.283.183.483.182.782.079.93.3
L2P79.077.477.477.974.273.973.55.5
DKR85.985.483.182.779.977.969.916.0
MoE83.783.483.483.583.483.483.50.2
ARC86.886.886.786.786.886.886.80.0

Methods that update shared parameters steadily forget Flickr30K as more recent tasks are introduced. MoE and C-CLIP forget less, but they perform worse at the initial task. ARC learns the task as well as fine-tuning, maintaining that level of performance, and KreaM and Sketch demonstrate a similar trend. These trajectories have been added to Appendix H, Tables 13–15.

Sequential LoRA baseline. To separate the effect of low-rank adaptation from that of expert isolation and routing, we added Seq-LoRA. It trains a single LoRA module with the same configuration as one ARC expert sequentially on all tasks, without isolation, routing or consolidation.

Task-specific R@1Global R@1Flickr30K (task 1)KreaM (task 4)Sketch (task 6)
MethodT→II→TT→II→TLearnedFinalLearnedFinalLearnedFinal
ZS26.027.624.126.0––––––
FT30.330.326.327.186.970.844.424.415.37.2
Seq-LoRA11.511.77.78.286.832.842.36.414.61.7
ARC46.246.940.244.086.886.843.343.312.812.8

Seq-LoRA learns each new task about as well as ARC, but forgets most of this knowledge as later tasks arrive. Its final average R@1 falls far below FT and even below the zero-shot model, on both task-specific and global galleries. A low-rank update shared across all tasks is therefore even more susceptible to interference than full fine-tuning. These results support that ARC's gains come from isolating adaptations and routing between them, not from low-rank adaptation itself. We added these results to Appendix E, Table 10.

Robustness to task order. We added a Reverse Separate protocol, which uses the same seven tasks in reverse order.

SeparateReverse Separate
MethodT→II→TRankT→II→TRank
FT30.330.3739.138.35
TA32.733.6532.633.68
EWC32.433.2640.240.42
Mod-X34.535.3339.339.34
C-CLIP33.935.8437.037.56
L2P25.225.7929.829.09
DKR30.129.7840.340.13
MoE35.738.3235.538.77
ARC46.246.9144.845.61

Consistently with our previous findings about the other protocol changes, reversing the task order substantially changes the ranking of several baselines. ARC remains first under both orderings, about 4.5 R@1 above the next-best method in the reversed order. This supports our broader finding that conclusions drawn from a single continual setup can be fragile: task order, like the training protocol and gallery setting, can change which baselines appear effective, while ARC's advantage holds across all of these settings. We added these results to Appendix I, Table 17.

(W4 & Q4) What are the query latency, memory and gallery storage of ARC relative to fine-tuning.

Inference costs.

By default, ARC routes each query–candidate pair to an expert, so a query must be encoded under several experts and the gallery stored once per expert. To address this, we added a per-item variant in which each query and each gallery item is routed once, independently, using its frozen-backbone embedding, and encoded with its selected expert. Every gallery item then has a single fixed embedding, so per-item routing is compatible with standard single-index retrieval.

We measure cost on the global gallery (41k images, 45k captions), the largest gallery in our evaluation, on one NVIDIA B200, under the separate protocol (14 experts). Latency is per single query and covers query encoding and routing; scoring the gallery adds ≈0.3 ms for every method.

MethodExtra params (M)Gallery storage (MB)Latency T→I (ms)Latency I→T (ms)
FT / single-model methods016914.114.6
L2P0.0416914.029.4
MoE11.416914.7113.8
ARC, per-item20.616939.740.4
ARC, pair-level20.62539182.3178.6

Pair-level routing needs 15 times the gallery storage of a standard dual encoder and is about 13 times slower. Per-item routing keeps gallery storage identical to FT and is 4.5 times faster than pair-level routing. Its query latency is 2.8 times that of a single dual encoder, close to L2P and well below MoE for image queries, at 40 ms versus 114 ms. It loses at most 1.0 R@1 on task-specific galleries and is better on the global gallery. We therefore recommend per-item routing when inference cost matters and state this in the limitations. We added this analysis to Appendix F, Table 11, and the routing comparison to Section 4.4, Table 7.

(MW1) In Figure 3, FT has a higher mean R@1 than MoE with ViT-B/32 but is ranked lower. In Table 4, the definitions in Appendix F imply MRR >= R@1 + (R@5 - R@1)/5, but the reported values of FT and Mod-X violate this.

We have corrected the ranks in the backbone comparison and the MRR values in what is now Table 5, which now satisfy this bound for all methods. The metric definitions are now in Appendix K.

(W1 & Q1) Please provide the dataset and split statistics, the query and gallery sizes, and the handling of multiple captions per image.

We report splits and gallery sizes for all the dataset evaluated below.

DatasetDomainSplitTrain imagesTrain captionsEval imagesEval captionsCaptions/img
Flickr30KNaturalKarpathy30,014150,0701,0005,0005
Lexica-SDAI-generated80/209,4709,4702,3682,3681
WikiArtArtwork80/2065,08465,08416,27116,2711
KreaMAI-generated50/507,4527,4527,4527,4521
FlintstonesCartoonOfficial20,13220,1322,3092,3091
SketchSketch80/207,9997,9992,0002,0001
ROCOv2MedicalOfficial59,96259,9629,9279,9271
Global gallery (7 training datasets)41,32745,327
COCO (held-out)NaturalKarpathy113,287566,4355,00025,0005

For Flickr30K and COCO, which provide five captions per image, we sample one caption per image at each training step and keep all five for evaluation. In text-to-image retrieval, each caption is a query with exactly one correct image. In image-to-text retrieval, an image query counts as correct if any of its five captions appears in the top K. All other datasets have one caption per image. These details are now in Appendix B, Table 9.

Summary

We would like to thank the reviewer for recognising the value of our evaluation framework and ARC's strong performance. We believe that the additional experiments and clarifications address the concerns raised. If these concerns have been resolved, we would be grateful if the reviewer would consider raising their score.

I would like to thank the authors for the extensive answer with clarifications and additional results. Although ARC exhibits one of the largest standard deviations over multiple seeds out of all the compared methods, it is still relatively small and the noise wouldn't affect the performance ranking. The dataset statistics and the description of the MoE adaptation resolve my questions. The additional section about hyperparameters in the appendix is useful for future reproduction. Tuning the parameters wrt to test scores is not ideal but at least it is consistent for all methods and the authors are honest about it.

I see the new results with the per-task learning trajectory as especially insightful, along with the reversed order and the Seq-LoRA baseline, and I believe they would make the paper (appendix) stronger. These results verify ARC is stable and outperforms the rest in all settings. The reversed order also shows that the shared-parameter baselines are very sensitive to the task order. I am not sure how to explain the ARC dropping by three times the std for the reversed order as it shouldn't in any way depend on the order.

Lastly, the cost analysis along with a more efficient per-item strategy that preserves most of the performance is valuable and makes the method more useful in practice.

I have noticed that the separate-protocol results of several baselines differ a lot from the original submission (FT drops from 37.5 to 30.3). Please state what has changed in the re-run, and why, and whether the other protocols use the same settings.

Overall, I see the answers and new results as positive, making the paper stronger. I am willing to update my score to weak accept.

Comments on the paper

4 comments

Thank you for the thoughtful questions and for recognizing that SCOPE exposes evaluation choices that materially affect conclusions, supported by a broad and well-organized empirical study. We address the remaining concerns below.

(W1 & Q1) ARC does not define a coherent global retrieval function. For each query-candidate pair, ARC chooses the expert that maximizes the pair's prototype compatibility. Consequently, two candidates for the same query can be scored in different adapted spaces, and ranking comparability across the global gallery is no longer guaranteed. This undercuts the paper's stated goal of preserving a shared embedding space and may inflate the global-gallery result relative to methods that use one encoder per query. Can the authors replace pair-level routing with a query-level or globally fixed expert choice and report task-specific and global-gallery retrieval under that restriction?

We added a per-item routing variant (Section 4.4, Table 7; Appendix F, Table 11). Each query and each gallery item is routed once, independently, and embedded with its selected expert. Every item therefore has a single fixed embedding, all candidates for a query are scored against the same query embedding, and the gallery can be stored in a standard single-index retrieval system.

RoutingTask-specific T→ITask-specific I→TGlobal T→IGlobal I→T
ARC, pair-level46.246.940.244.0
ARC, per-item45.245.941.744.4
ARC, task-ID (oracle)46.047.0––
MoE (best baseline)35.738.331.536.1

Per-item routing is within 1.0 R@1 of pair-level routing on task-specific galleries and better on the global gallery, where the reviewer's concern applies. It remains about 10 points above the best baseline in all settings. ARC's global-gallery advantage therefore does not depend on pair-level scoring. Pair-level routing also matches oracle task-ID routing to within 0.2 points, so the routing itself is accurate.

(W2 & Q2) How many experts remain after consolidation, and what are the per-query latency, memory, and gallery-storage costs for task-specific and global retrieval?

Retained experts (Appendix C). ARC merges a new expert only when its prototypes match an existing one (β = 0.9), so the pool grows with the number of distinct domains rather than the number of tasks:

ProtocolExperts createdExperts retained
Separate (7 disjoint domains)1414
Overlapping147
Long sequence (domains recur)2814

In the long sequence, every expert from the second pass is merged into its first-pass counterpart. Each expert adds 1.47M parameters (about 1% of CLIP ViT-B/16).

Inference costs. We measure cost on the global gallery (41k images, 45k captions), the largest gallery in our evaluation, on one NVIDIA B200, under the separate protocol (14 experts). Latency is per single query and covers query encoding and routing; scoring the gallery adds ≈0.3 ms for every method.

MethodExtra params (M)Gallery storage (MB)Latency T→I (ms)Latency I→T (ms)
FT / single-model methods016914.114.6
L2P0.0416914.029.4
MoE11.416914.7113.8
ARC, per-item20.616939.740.4
ARC, pair-level20.62539182.3178.6

ARC with per-item routing remains in the same latency range as established input-dependent methods: it is faster than MoE on image queries (40 vs. 114 ms) and slower on text queries (40 vs. 14 ms), while improving R@1 by 8–10 points over the strongest baseline across all settings. It therefore offers a good practical trade-off. Pair-level routing gives a small additional gain under task-specific evaluation (up to 1.0 R@1) and can be used when computation is not a constraint, for example in controlled benchmarking of CL methods as in our work. We added the cost analysis to Appendix F, Table 11, and the routing comparison to Section 4.4, Table 7 of the revised manuscript.

(W2 & Q3) Can the main ARC comparisons be repeated over at least three seeds with the same hyperparameter-selection budget for all methods?

We re-ran all methods with three seeds (42, 100, 1000) under the separate protocol. All methods share the same training budget and the same hyperparameter-tuning budget. Mean (std) of the average final R@1:

MethodT→II→T
FT30.38 (0.10)30.50 (0.23)
TA32.70 (0.03)33.70 (0.07)
EWC32.57 (0.15)33.17 (0.05)
Mod-X34.54 (0.09)35.41 (0.08)
C-CLIP34.03 (0.09)35.86 (0.10)
L2P25.27 (0.36)23.86 (1.60)
DKR30.43 (0.41)29.95 (0.29)
MoE35.84 (0.21)38.20 (0.12)
ARC45.77 (0.52)46.73 (0.74)

We re-ran all methods with three seeds under the separate protocol and now report the mean and standard deviation for every method. ARC’s lead over the next-best method is 9.9 R@1 for T->I and 8.5 for I->T, more than 11 times the larger standard deviation of the two methods, so its first place does not depend on the seed. A few adjacent baselines, such as Mod-X and C-CLIP (0.03 apart on average), differ by less than their seed-to-seed variation, which is consistent with our finding that rankings among existing methods are fragile. All methods now use the same hyperparameter-tuning and training budgets, and we have updated all results in the paper accordingly. We will run more seeds for other protocols in any final version of this work. The results of the multi-seed runs have now been added to Table 16 in Appendix I.

Summary

We thank the reviewer for highlighting the value of our evaluation findings and the breadth of our empirical study. We believe that the new analyses of routing, robustness and cost address the concerns raised. If our responses have resolved these concerns, we would be grateful if the reviewer would consider raising their score.

Dear Reviewers,

As the discussion phase comes to an end, we'd appreciate it if you could take a moment to review the authors' responses. If you have any remaining concerns or questions, please raise them with the authors before the discussion closes.

Best,

Your AC

AuthorsOpenReview ↗

Discussion summary

We thank all the reviewers for their constructive feedback and for the time they dedicated to our work. We especially appreciate the feedback from Reviewers WGdu and tP1R during the discussion phase, which helped us strengthen the paper. Both reviewers acknowledged that our additional experiments addressed their main concerns and raised their scores.

Reviewer ZvSs stated that their three questions were limited to items that could materially change the assessment. Although we did not receive further feedback during the discussion period, we believe our additional experiments and clarifications thoroughly addressed each of these questions:

  • Coherent global retrieval (Q1): A new per-item routing variant gives every item a single fixed embedding. It stays within 1.0 R@1 of pair-level routing on task-specific galleries, improves on the global gallery, and remains 8–10 points above the best baseline (Table 7).
  • Experts and inference cost (Q2): 14 experts are retained, and per-item routing keeps gallery storage identical to fine-tuning, with query latency comparable to existing input-dependent methods (Appendix C, Table 11).
  • Seeds and tuning budget (Q3): We ran all methods over three seeds with equal tuning budgets under Separate Protocol. ARC’s lead over the next-best method exceeds 11× its standard deviation, so its first place does not depend on the seed (Table 16).

We hope this summary is helpful to the AC in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.