acceptodds
Withdrawn from ICLR 2027

Training-Free Continual Learning For Vision Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual learning (CL) for multimodal large language models is currently benchmarked with parameter-heavy mixture-of-experts and LoRA routing methods. A large part of the reported gains on popular CL suites comes from tasks being trivially separable, so a router only needs to identify the task, instead of transferring knowledge across tasks. To address this, we build a continual multimodal multiple-choice benchmark by splitting two standard evaluation suites: MMMU (6 course categories / 30 fine categories) and MMStar (6 course categories / 18 fine categories) into disjoint task streams. Following previous work, we quantify their separability with a nearest-prototype routing probe and demonstrate that our benchmark is not trivially separable. We also introduce COMEM (Content- based Mem), a training-free, content-based k-nearest neighbor memory for a frozen Vision-Language Model (VLM). COMEM stores, per input example, the model’s last-layer question representation as a key and question-conditioned embedding of the correct answer’s content as a value. At test time, it interpolates the model’s own answer choice distribution with a distribution obtained by matching each option to a similarity-weighted retrieved answer meaning. Because the memory is append-only and avoids modifying model weights, COMEM has essentially zero catastrophic forgetting by construction. Across all four evaluation streams, COMEM consistently improves over a frozen model near-zero backward transfer, as well as matches or outperforms the mixture-of-LoRA-experts current state-of-the-art method, all without updating any model weights.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Incremental Learning, Class-Incremental Learning, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KU1GOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes continual learning benchmarks derived from coarse and fine category splits of MMMU and MMStar, asserting that existing benchmarks suffer from trivial task separability. It introduces CoMem, a training-free retrieval method for vision-language models. The method does not update model weights and does not require task identity at inference. On the proposed benchmarks, it obtains accuracy comparable to or slightly better than CL-MoE

Critical strengths

The paper identifies an important limitation of existing continual learning benchmarks. High task separability may help expert routing without requiring meaningful knowledge transfer. The proposed intra-dataset task streams offer a alternative benchmark.

Critical weaknesses
  • I am unclear about the contribution and positioning of the paper. The proposed benchmark is constructed by splitting only two existing datasets using predefined subject/category labels. There is no new dataset or guidelines defines. Also, task order, task difficulty, semantic similarity between tasks, and cross-task transfer are not studied.
  • The motivation is high task separability of existing benchmarks. But, reducing nearest-prototype task-identification accuracy from approximately 100% to 60–77% does not establish that the resulting benchmark measures meaningful knowledge transfer.
  • Although the paper cites recent multimodal continual learning approaches, only CL-MoE is evaluated. And the proposed retrieval-based method is not compared against simple kNN memory, retrieval-based in-context learning, or other training-free alternatives.
  • All experiments use only one split and a fixed task order.
  • Major numerical inconsistencies across all experimental tables. several results are inconsistent across tables 1, 3, 4 and 5
  • Paper needs major revision. Figure1, and 2 have same captions and there are many typos, Table numbers references are wron
Questions for authors
  • How does the proposed benchmark measure continual knowledge transfer rather than sequential adaptation to different subject categories? What is the proposes formulation or guideline?
  • How does CoMEM compare against simple nearest-neighbor answer retrieval or retrieval-based in-context learning under the same memory budget?
  • How sensitive are the results to task ordering, task difficulty, and semantic overlap between categories?
  • What evidence shows that CoMEM acquires transferable knowledge rather than primarily retrieving memorized answer information?
  • Tables 1, 3, 4, and 5 report different CoMEM accuracies under apparently identical settings? Which values are correct?
Reviewer_W8vkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoMEM, a training-free continual learning framework for vision-language models based on an external nearest-neighbor memory. The method stores question representations as retrieval keys and embeddings of correct answer content as values. At inference time, it aggregates retrieved answer representations, matches them to the current question's options, and interpolates the resulting distribution with the frozen model's predictions. The paper also constructs four continual learning streams from MMMU and MMStar, motivated by the high task separability of existing cross-dataset benchmarks.

Critical strengths

Relevant Benchmark Motivation. Investigating whether existing benchmarks reward task identification rather than effective adaptation is meaningful. Within-dataset task streams provide a useful complementary evaluation setting, and the separability analysis supports the claim that these streams are harder to distinguish using the evaluated probe.

Simple and Practical Method. CoMEM requires no gradient updates and uses the frozen model's representations to incorporate new labeled examples. Storing answer content rather than option letters addresses a meaningful issue in multiple-choice tasks, where the same letter has different meanings across questions.

Useful Initial Evaluation. The experiments cover coarse- and fine-grained streams on two datasets, compare against CL-MoE, and examine memory capacity, the number of retrieved neighbors, and prediction gating. Reporting backward transfer alongside final accuracy is also appropriate for the continual learning setting.

Critical weaknesses

Inconsistent Experimental Reporting. Section 5.1 specifies unbounded memory, k = 16, and no gating for the main experiments, and Section 6 claims the ablations use this configuration, yet the tables disagree: Table 1 reports 37.0% on MMMU-fine versus 37.6% at k = 16 in Table 5, and 39.0% on MMStar-fine versus 40.6% in the no-gating row of Table 4. Moreover, after accounting for column order, the unbounded-memory row of Table 3 coincides exactly with the confidence-gating row of Table 4, suggesting undisclosed configuration differences. These discrepancies are comparable in size to the reported gains, so the central comparison cannot be evaluated reliably until they are resolved.

Insufficiently Isolated Contribution. The framework combines familiar elements — frozen representations, kNN retrieval, and interpolation with the model's predictions — and the experiments do not establish the contribution of the specific design choices (answer-content values, question-conditioned channel) beyond adding a standard retrieval memory. All comparisons are against CL-MoE or fine-tuning; at least one well-matched simple retrieval baseline (e.g., 1-NN answer-content matching, retrieval-only prediction without interpolation, or the isolated channel alone) is needed to determine whether the proposed construction improves over straightforward retrieval.

Benchmark Interpretation and Stability. Routing accuracies of 60%–77% show that substantial task information remains available; lower separability alone does not establish that the benchmark measures cross-task transfer. An analysis of the contribution of memories from other tasks would substantiate this interpretation. In addition, results use a single 80/20 split and task order, and the 0.9–2.9 point improvements over CL-MoE (with MMStar test sets of only ~300 examples) lack uncertainty estimates across seeds, splits, or task orders.

Overstated Forgetting Claim. Freezing the backbone prevents parameter interference, but the prediction system also depends on the evolving memory: new entries change nearest-neighbor sets and aggregated answer meanings for earlier tasks, and reservoir sampling evicts entries under bounded budgets. The near-zero BWT is empirical evidence of limited forgetting, not a construction-level guarantee; the claim should be narrowed accordingly.

Questions for authors

Q1. Please provide the exact configuration for each table, explain the discrepancies, and identify the results obtained under the stated default configuration.

Q2. Can you report at least one well-matched simple retrieval baseline under the default configuration — e.g., matching the current options against the single nearest stored answer content, retrieval-only prediction without interpolation with the model, or the isolated (non-question-conditioned) value channel alone?

Q3. The improvements over CL-MoE (0.9–2.9 points) are reported on a single 80/20 split and fixed task order, with MMStar test sets of only ~300 examples total. Can you provide uncertainty estimates across seeds, splits, or task orders?

Q4. Could you clarify the scope of the construction-level forgetting claim, and ideally report the fraction of retrieved memories originating from other tasks, as a direct diagnostic of cross-task transfer?

Reviewer_ZNcrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual learning for multimodal large language models and introduces four task streams constructed from coarse- and fine-grained categories in MMMU and MMStar. A nearest-prototype probe suggests that these streams have lower task separability than the cross-domain benchmark examined in the paper. The proposed method, CoMem, augments a frozen LLaVA-v1.5-7B model with an external memory. It retrieves examples using multimodal question representations, stores both isolated and question-conditioned representations of the correct answer text, and matches the retrieved answer content against the current answer options. The resulting retrieval distribution is interpolated with the frozen model's predictions. The paper reports improvements over the frozen model on all four streams, performance comparable to or better than CL-MoE, and small backward-transfer changes, without gradient-based adaptation.

Critical strengths
  1. The benchmark construction addresses a useful evaluation concern.

Forming task streams within MMMU and MMStar reduces the pronounced dataset-level differences that can make task identification straightforward in heterogeneous streams. The accompanying separability probe provides evidence for this motivation under the representations and classifier examined. This offers a useful setting for studying continual multimodal learning when task boundaries are less readily distinguishable from input features.

  1. The method provides a practical way to reuse labeled experience without updating model parameters.

CoMem stores answer content rather than option letters, allowing retrieved examples to inform questions with different candidate answers. Its forward-only adaptation is simple and avoids gradient-based training. The reported improvements over the frozen model and near-zero backward transfer are encouraging evidence for the overall approach, although the reporting issues below need to be resolved before the precise gains can be assessed confidently.

Critical weaknesses
  1. Unexplained inconsistencies between main and ablation results.

Section 5.1 specifies unbounded memory, no gating, and k = 16; Section 6 states that its results follow this configuration. Yet corresponding final accuracies differ:

  • MMStar fine: Table 1, 39.0%; Table 4, no gating, 40.6%.
  • MMStar coarse: Table 1, 41.4%; Table 3, unbounded memory, 39.4%.
  • MMMU fine: Table 1, 37.0%; Table 5, k = 16, 37.6%.

These unexplained differences make it difficult to assess the main results and effects attributed to individual design choices. What accounts for the discrepancies: configuration changes, different runs, or reporting errors? Please specify any differences in configuration or evaluation between Table 1 and Tables 3-5, and provide consistent results for the default setting.

  1. The benefit of the dual-channel answer representation is insufficiently validated. Section 4 combines independently encoded and question-conditioned answer representations, but Section 6 does not compare this design with either channel used alone. It therefore remains unclear whether combining the two channels improves performance over a simpler single-channel representation. Please compare isolated-only, question-conditioned-only, and hybrid variants under otherwise identical settings. The paper also claims that question conditioning benefits quantitative questions whose answers, such as single numbers, have little meaning in isolation. A performance breakdown on these questions would help substantiate this claim.

  2. The choice of CL-MoE baseline settings is insufficiently explained. Section 5.1 uses four experts for CL-MoE, while the original paper uses eight in its main experiments. Although the original paper also evaluates a four-expert variant, the manuscript does not explain why this configuration was selected for the new benchmark. It is also unclear whether the stated LoRA learning rate of 5×10−55\times10^{-5} applies to CL-MoE, whose original setup uses 2×10−42\times10^{-4}. Please report the exact CL-MoE settings and explain how they were selected, including any validation used to guide this choice. This information would help assess whether the baseline was appropriately configured for the comparison.

Questions for authors

See the Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.