acceptodds
Under review as a conference paper at ICLR 2027

R-MLLM: Dynamic Model Editing for Robust Multimodal Large Language Model under Limited Data

Abstract

In real-world scenarios, unknown visual corruptions may continue to emerge and degrade model performance. Existing robust Multimodal Large Language Models (MLLMs) are typically optimized for predefined corruptions and thus generalize poorly to unknown corruptions. Although adapting these models to new corruptions can improve their performance on newly emerging corruptions, it often leads to catastrophic forgetting of previously acquired robustness knowledge. To address these challenges, we formulate the visual robustness task as a continual learning problem and propose a model editing framework for data-efficient continual adaptation to visual corruptions. Specifically, we fine-tune the visual encoder with an additional robustness loss to obtain corrupted key-value pairs. We then propose a dynamic key-value pair construction strategy that adaptively weights samples according to their quality, enabling effective model updates with limited data. To preserve pretrained knowledge, we further introduce a principal component constraint that regularizes parameter updates during adaptation and mitigates catastrophic forgetting. Finally, we introduce VCBench, a benchmark covering vision questions under 17 common real-world visual corruptions, to evaluate the robustness and continual adaptation ability of MLLMs. Extensive experiments across multiple benchmarks show that our method improves robustness over strong baselines, achieving up to a 7.6% average accuracy gain on VCBench.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal Large Language Models, Model Editing, Visual Robustness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7mh8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes R-MLLM, a model editing framework designed to improve the robustness of multimodal large language models against visual corruptions under limited-data settings. The method fine-tunes the visual encoder using a robustness loss, introduces a dynamic key-value weighting strategy to prioritize high-quality samples, and employs a principal component constraint to mitigate catastrophic forgetting. The authors also introduce VCBench, a visual robustness benchmark covering 17 corruption types, with a filtering strategy intended to reduce reliance on language priors. Experiments on multiple benchmarks demonstrate improvements in visual robustness while largely preserving the model's original capabilities.

Critical strengths

Strengths

  1. Clear method design. The paper combines visual feature alignment, dynamic key-value weighting, and principal component constraints to address feature shifts caused by visual corruptions, the influence of low-quality samples on model updates, and catastrophic forgetting, respectively. The overall framework is clearly structured, with reasonable motivations for each component.

  2. Relatively comprehensive experimental evaluation. The authors evaluate their method on VCBench, R-Bench, and other visual robustness benchmarks, along with ablation studies to examine the contributions of individual components. The results demonstrate improvements in visual robustness under limited-data settings while largely preserving the model's original capabilities.

Critical weaknesses
  1. Insufficient justification for the continual learning formulation.

The paper formulates visual robustness as a continual learning problem, where new corruption types arrive sequentially. However, the practical motivation for this formulation remains insufficiently justified. The evaluated corruptions are all predefined and widely studied in existing robustness benchmarks, dating back to ImageNet-C (2019), rather than representing genuinely novel or emerging degradation patterns. The authors should clarify which real-world deployment scenarios require sequential adaptation to different corruption types, rather than joint robustness training or test-time adaptation. Moreover, the current experiments do not sufficiently demonstrate why this continual learning formulation is necessary or preferable to these alternatives.

  1. Questionable validity of the VCBench filtering strategy.

The construction of VCBench raises a fundamental concern. The authors claim that selecting samples answered correctly on clean images but incorrectly on corrupted images eliminates questions solvable by language priors. However, this criterion only identifies corruption-sensitive responses and does not establish that the questions require visual information. A question may still be answerable using language priors even if a model produces an incorrect response when given a corrupted image. Furthermore, the benchmark is constructed using the intersection of samples filtered by two reference models (Qwen and LLaVA), which may introduce model-dependent selection bias, particularly given that Qwen is also the primary model family evaluated in this work. The authors should provide text-only evaluations and investigate how the choice of reference models affects the resulting benchmark.

  1. Limited novelty of VCBench compared with existing benchmarks.

The novelty of VCBench appears limited. The 17 corruption types are already covered by existing corruption benchmarks, including the standard and extended ImageNet-C corruptions. Moreover, benchmarks such as R-Bench have explicitly addressed the evaluation of MLLMs under real-world visual corruptions. Therefore, the primary distinction of VCBench seems to lie in its sample-filtering strategy rather than in new corruption types or evaluation settings. Given the concerns regarding this filtering strategy, the authors should more clearly articulate the unique contribution of VCBench and provide a systematic comparison with existing benchmarks to demonstrate its additional value.

  1. Insufficient evaluation of generalizability across MLLM architectures.

The generalizability of the proposed framework remains insufficiently demonstrated. Although the experiments include Qwen2.5-VL-3B and Qwen2.5-VL-7B, both models belong to the same architecture family. Since the method relies on specific FFN projection layers in the visual encoder, it is unclear whether the proposed editing strategy can be effectively applied to other MLLM architectures. Evaluating the method on additional model families, such as InternVL or LLaVA, would help substantiate its claimed general applicability.

Questions for authors
  1. Continual learning setting. Could the authors clarify the practical scenarios that motivate sequential adaptation to different corruption types? In particular, why is continual model editing preferable to joint robustness training or test-time adaptation in these scenarios?

  2. Validity of VCBench filtering. How do the authors justify the claim that the proposed filtering criterion eliminates questions answerable through language priors? Could the authors provide text-only evaluation results to support this claim? Additionally, how sensitive is VCBench to the choice of reference models used for filtering, particularly since Qwen is also the primary model family evaluated?

  3. Novelty of VCBench. Given that the 17 corruption types are already covered by existing benchmarks, what specific evaluation limitations does VCBench address beyond the proposed sample-filtering strategy? Could the authors provide evidence that VCBench offers additional evaluation value compared with existing benchmarks?

  4. Generalizability. Can the authors provide evidence that the proposed framework generalizes beyond the Qwen2.5-VL family? Since the method relies on specific FFN projection layers, results on at least one MLLM with a different visual encoder architecture would help establish its general applicability.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7mh8 (1/2)

Thank you for your diligent efforts in reviewing our paper and for recognizing the contributions of our work.

Q1: Practical scenarios and method advantages.

Continual learning setting. Could the authors clarify the practical scenarios that motivate sequential adaptation to different corruption types? In particular, why is continual model editing preferable to joint robustness training or test-time adaptation in these scenarios?

Thank you for raising this question. To address your question, we clarify the practical scenario considered in our experiments and add comparisons with joint training and test-time adaptation (TTA) methods to further evaluate the effectiveness of our approach. In real-world applications, such as long-term autonomous driving, models may encounter different image corruptions sequentially as vehicles operate across different regions, seasons, and weather conditions. For example, a vehicle may first encounter rainy conditions, then foggy or snowy environments, and later return to previously encountered conditions. Joint training requires retraining on all previous and new corruption scenarios whenever a new corruption appears, resulting in high training costs. TTA methods primarily adapt to the current test distribution and may suffer from forgetting previously encountered corruptions. We therefore study how to incrementally update an existing model using a small number of clean and corrupted image pairs for each new corruption. Our goal is to adapt to new corruptions while retaining performance on previous corruptions.

To simulate a model encountering a new corruption, we first train a model on seven corruption types: brightness, elastic_transform, gaussian_noise, jpeg_compression, saturate, spatter, and contrast. We call this model "base model". Fog is not used to train this model. Then, we introduce fog data for adaptation to this new corruption. Our method and the two TTA methods, DeYO and SAR, start from the same base model. Joint-training is trained jointly on data from all eight corruption types. Ours is obtained by further editing the base model to adapt to fog. As shown in the table below, Our method improves accuracy on fog from 35.45% to 39.10%, a gain of 3.65 percentage points. Its average accuracy on the seven previous corruptions changes from 36.38% to 36.10%, a decrease of 0.28 percentage points. Across all eight corruptions, our method achieves 36.47%, compared with 33.76% for joint training, 35.72% for DeYO, and 35.64% for SAR. These results show that our method adapts to fog while largely retaining its performance on the previous corruptions.

CorruptionBase model (reference)Joint-trainingDeYOSAROurs
brightness45.2641.2344.7944.9245.15
elastic_transform40.7534.3740.3640.4440.36
gaussian_noise23.0221.8921.8522.2422.76
jpeg_compression41.5337.8039.3638.6241.53
saturate41.7939.7140.9639.5641.40
spatter37.1032.6736.1935.8436.36
contrast25.2026.6724.1124.9325.11
fog35.4535.7238.1238.5339.10
Avg (first seven)36.3833.4835.3735.2236.10
Avg (all eight)36.2633.7635.7235.6436.47
AuthorsOpenReview ↗

Response to Reviewer 7mh8 (2/2)

Q2&Q3: Construction and validation of VCBench.

Validity of VCBench filtering. How do the authors justify the claim that the proposed filtering criterion eliminates questions answerable through language priors? Could the authors provide text-only evaluation results to support this claim? Additionally, how sensitive is VCBench to the choice of reference models used for filtering, particularly since Qwen is also the primary model family evaluated?

Novelty of VCBench. Given that the 17 corruption types are already covered by existing benchmarks, what specific evaluation limitations does VCBench address beyond the proposed sample-filtering strategy? Could the authors provide evidence that VCBench offers additional evaluation value compared with existing benchmarks?

Thank you for pointing this out. We add experiments evaluating multiple MLLMs on VCBench to further demonstrate the validity and fairness of the benchmark. Our goal is to select questions that become difficult under visual corruptions. We keep questions that the filtering models answer correctly on clean images but incorrectly on corrupted images. This criterion identifies questions on which corruptions affect the models' answers. It does not show that these questions cannot be answered from text alone. We will revise the description of VCBench to clarify this point.

When corruptions are added to existing VQA benchmarks, many questions show little change in accuracy. VCBench focuses on questions that are sensitive to corruptions. To check whether the performance drop also appears in models that were not used for filtering, we evaluate LLaVA-v1.6-vicuna-13B, InternVL3.5-8B, and GLM-4.1V-9B-Thinking. We compare their results on clean images and images with Gaussian noise in both VCBench and RealWorldQA. As shown below, these models also show large performance drops on VCBench.

ModelVCBench (clean)VCBench (noise)RealWorldQA (clean)RealWorldQA (noise)
LLaVA-NeXT-13B75.07%31.54%(↓43.53)58.17%55.16%(↓3.01)
GLM-4.1V-9B-Thinking73.24%31.62%(↓41.62)69.80%64.05%(↓5.75)
InternVL3.5-8B71.76%28.15%(↓43.61)67.19%63.01%(↓4.18)
Qwen2.5-VL-7B(filtering model)70.13%23.64%(↓46.49)67.97%61.65%(↓6.32)

Q4: Generalizability across different MLLM architectures.

Generalizability. Can the authors provide evidence that the proposed framework generalizes beyond the Qwen2.5-VL family? Since the method relies on specific FFN projection layers, results on at least one MLLM with a different visual encoder architecture would help establish its general applicability.

Thank you for highlighting the importance of evaluating generalizability across different MLLM architectures. We add experiments on multiple MLLMs, including LLaVA-v1.6-vicuna-13B, InternVL3.5-8B, and GLM-4.1V-9B-Thinking, to further evaluate the generalizability of our method. As shown in the table below, our method consistently improves the robustness of all three models.

ModelNoiseBlurWeatherDigitalAvg
InternVL3.5-8B25.2622.7027.5433.3028.18
InternVL3.5-8B +Ours33.15 (↑7.89)37.21 (↑14.51)37.75 (↑10.21)43.82 (↑10.52)38.71 (↑10.53)
InternVL3.5-8B (Oracle)33.7438.4240.0345.3640.15
LLaVA-v1.6-vicuna-13B26.9327.3934.9036.3732.22
LLaVA-v1.6-vicuna-13B + Ours37.40 (↑10.47)36.89 (↑9.50)42.35 (↑7.45)44.51 (↑8.14)40.98 (↑8.76)
LLaVA-v1.6-vicuna-13B (Oracle)38.6437.9545.0945.9442.61
GLM-4.1V-9B15.8519.3125.2329.0723.33
GLM-4.1V-9B + Ours24.13 (↑8.28)25.83 (↑6.52)31.28 (↑6.05)33.43 (↑4.36)29.39 (↑6.06)
GLM-4.1V-9B (Oracle)25.6526.9734.1234.2830.92
AuthorsOpenReview ↗

Gentlel Request to Consider Our Rebuttal

Dear Reviewer 7mh8,

Thank you again for your thoughtful feedback and suggestions. As the discussion period ends soon, we wanted to check whether our responses address your questions. We have provided additional explanations and experimental results addressing your concerns, including comparisons with joint training and TTA, further evaluations of VCBench using multiple MLLMs, and cross-architecture generalizability experiments on InternVL3.5, LLaVA-v1.6, and GLM-4.1V. We have also clarified the scope and motivation of our benchmark filtering strategy. We would be glad to address any remaining questions before the deadline. If our responses and additional results have addressed your concerns, we would be grateful if you would update your score.

Reviewer_4f57OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper adapts Qwen2.5-VL to synthetic image corruptions using about 100 clean/corrupted pairs per corruption type, and frames this as continual learning. Stage 1 fine-tunes the FFN projections of the vision encoder with the next-token loss plus a cosine loss that aligns corrupted features with clean ones. Stage 2 collects key-value pairs from the fine-tuned encoder and computes new weights in closed form from a ridge-type objective. That objective has two additions: per-sample weights based on feature quality ("dynamic KV") and an extra penalty along the principal components of the keys. The paper also introduces VCBench, which keeps only items that two base MLLMs answer correctly on clean images and incorrectly under corruption.

Critical strengths
  • Editing the vision encoder instead of the LLM is a sensible choice for corruption robustness. The comparison of fine-tuning strategies (Table 7) is a useful data point for the low-data regime.
  • The closed-form stage only needs accumulated second-order statistics. Adding a corruption is cheap, and the result does not depend on corruption order.
Critical weaknesses
  • The evaluation favors the method. VCBench is filtered so that the base models fail on every item, which depresses base performance by construction. The appendix itself attributes the large VCBench gains to this filtering. Training and test use the same corruption functions, and no corruption is held out, even though the paper is motivated by unknown corruptions. On R-Bench, the improvement over the base model comes almost entirely from captioning; several VQA/MCQ splits are below the base model. Clean-benchmark accuracy drops by about 2.8 points on average.
  • The continual-learning framing does not match the protocol. Algorithm 1 pools statistics from all corruptions and solves once, and the paper does not say how stage 1 is run across tasks. The method is closer to closed-form model merging than to continual learning. Comparing it against sequentially trained CL methods is therefore not like-for-like, and merging baselines would be more relevant.
Questions for authors
  • Are keys and values taken from the same fine-tuned projection? If so, how can the sample weights reduce the influence of low-quality pairs?
  • How does the stage-1 model do on the clean benchmarks? How do WiSE-FT or a clean-key preservation term compare at matched robustness?
  • Did the baselines use the alignment loss or paired clean images? How was λ selected for each row of Table 4?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 4f57 (1/4)

We appreciate the time and effort you have devoted to reviewing our manuscript and offering feedback.

Q1: About the fairness of VCBench.

VCBench is filtered so that the base models fail on every item, which depresses base performance by construction. The appendix itself attributes the large VCBench gains to this filtering.

Thank you for pointing this out. We add experiments evaluating multiple MLLMs on VCBench to further demonstrate the validity and fairness of the benchmark. We designed VCBench to select challenging questions for evaluating visual robustness. Simply adding corruptions to existing VQA benchmarks does not necessarily provide a meaningful robustness evaluation, as many questions can still be answered correctly after corruption. This is because the corruptions may not affect the key visual information needed to answer these questions.

To address this issue, we use filtering models to select questions that it answers correctly on clean images but incorrectly on the corresponding corrupted images. In this way, VCBench focuses on questions whose answers are more sensitive to visual corruptions. Also, to check whether the performance drop also appears in models that were not used for filtering, we evaluate LLaVA-NeXT-13B, GLM-4.1V-9B-Thinking, InternVL3.5-8B, Qwen2.5-VL-7B(filtering model). We test them on clean images and images with Gaussian noise in both VCBench and RealWorldQA. As shown below, all models show large performance drops on VCBench. These results demonstrate that VCBench selects questions that are sensitive to visual corruptions, enabling effective evaluation of models’ visual robustness. Existing benchmarks may still contain many easy questions because the corruptions do not affect the visual information needed to answer them. Such questions make it harder to reveal weaknesses in visual robustness.

ModelVCBench (clean)VCBench (noise)RealWorldQA (clean)RealWorldQA (noise)
LLaVA-NeXT-13B75.07%31.54%(↓43.53)58.17%55.16%(↓3.01)
GLM-4.1V-9B-Thinking73.24%31.62%(↓41.62)69.80%64.05%(↓5.75)
InternVL3.5-8B71.76%28.15%(↓43.61)67.19%63.01%(↓4.18)
Qwen2.5-VL-7B(filtering model)70.13%23.64%(↓46.49)67.97%61.65%(↓6.32)

Q2: About held-out corruptions.

Training and test use the same corruption functions, and no corruption is held out, even though the paper is motivated by unknown corruptions.

Thank you for raising this question. We study a few-shot adaptation setting rather than a zero-shot setting. When a new corruption is encountered, we assume that a small number of samples are available for adaptation. Our goal is to use these limited samples to adapt the model to the new corruption while retaining its performance on previous corruptions.

AuthorsOpenReview ↗

Response to Reviewer 4f57 (2/4)

Q3: Performance on R-Bench.

On R-Bench, the improvement over the base model comes almost entirely from captioning; several VQA/MCQ splits are below the base model.

Thank you for this careful observation. Here, the Oracle refers to a task-specific reference model obtained by independently fine-tuning the base model on the corresponding task. The relatively weaker performance on MCQ and VQA is largely attributable to the limited capabilities of the corresponding fine-tuned Oracle models on these tasks. Our editing method is designed to preserve the capabilities of the Oracle model rather than further enhance its task-specific abilities. Consequently, the performance of the edited model is largely influenced by the capabilities of the Oracle. We expect that using an Oracle with stronger MCQ and VQA capabilities would also lead to better editing performance on these tasks. We further investigate this by using different task-specific Oracle models. The best results are highlighted in bold, excluding the Oracle references.

MethodDataParams.MCQ (Low)MCQ (Mid)MCQ (High)VQA (Low)VQA (Mid)VQA (High)CAP (Low)CAP (Mid)CAP (High)
Qwen2.5-VL-3B——0.64110.60220.57320.48720.48540.49040.37780.37040.3330
Robust-R1 (SFT and RL)10K3090M0.65290.63910.60970.49140.49090.49800.40680.37810.3484
Ours1.7K280M0.64710.60870.56100.45340.47210.49160.45990.44620.3703
CAP Oracle1.7K280M0.65880.62730.56710.46260.48120.49940.46420.45120.3745
Ours1.7K280M0.65880.63350.60980.49880.49450.50720.33210.33510.3294
MCQ/VQA Oracle1.7K280M0.67060.65220.61590.50370.49820.51140.34630.33930.3330

Q4: Performance drop on clean benchmarks.

Clean-benchmark accuracy drops by about 2.8 points on average.

Thank you for this careful observation. After adaptation to corruptions, model will show a decline in performance on clean benchmarks. Our method achieves the smallest average decline among the compared model editing and continual learning methods, As shown in the table below.

MethodSEEDBenchA-OKVQARealWorldQAMMBench
Qwen2.5-VL-7B77.0888.7367.9786.23
AlphaEdit75.3085.5062.8881.17
MEMIT74.7285.3360.9281.67
ROME62.1875.3451.0774.20
NSE74.1383.6159.4182.03
ZSCL62.4964.9554.9261.37
EWC54.3159.4246.2958.26
SD-LoRA72.3683.6361.6282.01
Ours75.1386.1164.1883.41
AuthorsOpenReview ↗

Response to Reviewer 4f57 (3/4)

Q5: Details of the algorithm and experimental setup.

The continual-learning framing does not match the protocol.

Thank you for asking us to clarify the procedure. We will clarify our experimental settings to address this concern. Algorithm 1 illustrates the adaptation procedure for the first corruption. We have revised the description of Algorithm 1 to further clarify the adaptation procedure and avoid potential ambiguity.

In a continual learning setting, a model learns tasks one at a time. It needs to learn each new task while retaining its performance on previous tasks. Performance is evaluated throughout the task sequence to measure both learning and forgetting.

Our setting follows the same principles. Each corruption is treated as a task, and the model adapts to corruptions sequentially using only the current corruption data. Knowledge from previous corruptions is retained in the KV aggregation matrices. After each stage, we evaluate the model on both the current and previously learned corruptions. Thus, our setting evaluates both adaptation to new corruptions and retention of previous knowledge. We add experiment to evaluate the model's continual learning performance using four continual learning metrics: MFT, MFN, MAA, and BWT,which are well-defined in MLLM-CL. Our method achieves the best performance across all reported continual learning metrics among the compared methods. This demonstrates its ability to adapt to new corruptions while retaining performance on previous corruptions. The comparative results are presented in the table below.

MethodMFT ↑MFN ↑MAA ↑BWT ↑
MEMIT27.3025.9528.95-1.47
AlphaEdit28.3126.3029.38-2.19
ZSCL25.2724.0026.67-1.39
SD-LoRA28.1027.7131.38-0.42
Ours29.4629.1633.12-0.33

Q6: Compare with model-merging baselines.

Comparing it against CL methods is not like-for-like, merging baselines would be more relevant.

Thank you for suggesting this comparison. Model Merging combines parameters or task-specific updates from independently trained models through strategies such as parameter averaging [1], weighted summation [2], and conflict-aware merging [3,4]. While effective for a limited number of tasks, it incurs increasing storage costs as more task-specific models are retained. Moreover, accumulating task-specific updates may cause parameter interference, hindering adaptation to new tasks and retention of previous knowledge. We evaluate the knowledge retention rate as the number of tasks increases. The retention rate compares the current average accuracy with the best performance previously achieved on each corruption. A higher rate indicates less forgetting. As shown in Table below, Model Merging exhibits a declining knowledge retention rate as the number of tasks increases.

New taskOursUniform SoupTask Arithmetic
brightness100.00100.00100.00
elastic_transform99.6398.6397.28
gaussian_noise99.3597.5196.65
jpeg_compression98.6897.0296.34
saturate98.2496.4995.68
spatter98.1995.8594.26
contrast98.1094.1493.17
fog98.4394.1393.26
motion_blur98.1393.0592.15
shot_noise97.5592.2091.50
speckle_noise97.1791.7690.76
defocus_blur96.8891.2589.27

In contrast, Model Editing incorporates newly acquired knowledge into a unified target model rather than merging multiple task-specific models. In our method, previous knowledge is accumulated in two fixed-size K⊤KK^\top K and K⊤VK^\top V matrices, keeping the additional storage constant as the number of tasks increases. Model Editing provides a more suitable framework for continual adaptation. We further compare our method with existing model-merging baselines, as shown in the table below. Our method achieves better performance across all corruption categories.

MethodNoiseBlurWeatherDigitalAvg
Uniform Soup17.5321.8430.9731.6726.44
Task Arithmetic16.2318.5919.6618.8518.38
TIES-Merging16.8522.4731.8631.5826.58
DARE16.8520.9425.8725.0122.57
Ours18.9224.2735.3933.3628.91

[1] Wortsman, Mitchell, et al. “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.” ICML, 2022.

[2] Ilharco, Gabriel, et al. “Editing Models with Task Arithmetic.” ICLR, 2023.

[3] Yadav, Prateek, et al. “TIES-Merging: Resolving Interference When Merging Models.” NeurIPS, 2023.

[4] Yu, Le, et al. “Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch.” ICML, 2024.

Reviewer_3YVjOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how multimodal large language models (MLLMs) can continually adapt to visual corruptions with limited training data, focusing on robustness to new corruption types and the forgetting of existing knowledge. It proposes R-MLLM, which frames visual robustness adaptation as continual learning and updates the FFN projection layers in the visual encoder through model editing. The paper also introduces VCBench, a benchmark covering 17 common real-world visual corruptions. The method achieves good experimental results.

Critical strengths
  1. The motivation is clear, and the problem is practically relevant.
  2. The method’s three components are interpretable.
  3. The paper introduces a new benchmark and provides extensive experiments.
Critical weaknesses
  1. Although the paper aims to address catastrophic forgetting, it provides insufficient experimental evidence of forgetting during training. At a minimum, it should report stage-by-stage forgetting across the full task sequence.
  2. The results in Table 4 are inconsistent: all values in the Avg. column are incorrect. They should be 21.84, 28.57, 29.45, and 29.84, respectively. Table 1 has similar inconsistencies. These errors seriously undermine confidence in the experimental results.
  3. The paper lacks continual learning metrics to support the claim that R-MLLM mitigates catastrophic forgetting.
  4. The task sequence is unclear. Are the first four tasks trained jointly, followed by joint training on four additional tasks, or are tasks learned one at a time? Is knowledge from previous tasks stored during training, and are any parameters reset between stages?
Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3YVj (1/2)

Thank you for your efforts for reviewing our paper and offering valuable feedback.

Q0: Our model editing experiment setting.

Thank you for your valuable feedback. We briefly clarify our method and experimental setting.

In our experiment, the model adapts to one corruption at a time. At stage NN, we fine-tune a model initialized from the original pretrained parameters using only the current corruption data and extract the corresponding KV pairs. We combine the KV pairs from the current corruption with those from previous corruptions to edit a model. These steps correspond to lines 5–8 of Algorithm 1 in the revised manuscript.

Our method is insensitive to corruption order and does not require more storage as the number of corruptions increases. We add together the KV statistics from different corruptions to edit the model. Changing the order of addition does not change the result, so our method is insensitive to corruption order. We only retain the sums of K⊤KK^\top K and K⊤VK^\top V from previous corruptions, rather than the individual KV pairs for each corruption, so storage does not grow with the number of corruptions.

Q1&Q3: Evaluation protocol for continual learning and catastrophic forgetting.

Although the paper aims to address catastrophic forgetting, it provides insufficient experimental evidence of forgetting during training. At a minimum, it should report stage-by-stage forgetting across the full task sequence. The paper lacks continual learning metrics to support the claim that R-MLLM mitigates catastrophic forgetting.

Thank you for your valuable suggestions. Our method is to improve visual robustness using continual learning scope, rather than to develop a continual learning method. The original experiments focused on robustness and did not include a detailed continual learning evaluation. We add a sequential adaptation experiment and continual learning metrics to evaluate forgetting at each stage of the full task sequence. The model adapts to corruptions in the following order: brightness, elastic_transform, gaussian_noise, jpeg_compression, saturate, spatter, contrast, fog, motion_blur, shot_noise, speckle_noise, defocus_blur. This experimental setup simulates practical scenarios in which a model continuously encounters previously unseen corruptions and adapts to them over time.

We evaluate the model's continual learning performance using four continual learning metrics: MFT, MFN, MAA, and BWT,which are well-defined in MLLM-CL [1]. Our method achieves the best performance across all reported continual learning metrics among the compared methods. This demonstrates its ability to adapt to new corruptions while retaining performance on previous corruptions. The comparative results are presented in the table below.

MethodMFT ↑MFN ↑MAA ↑BWT ↑
MEMIT27.3025.9528.95-1.47
AlphaEdit28.3126.3029.38-2.19
ZSCL25.2724.0026.67-1.39
SD-LoRA28.1027.7131.38-0.42
Ours29.4629.1633.12-0.33

To verify that our model effectively mitigates catastrophic forgetting, We also report how well the model preserves previous corruption performance after learning each new corruption. The retention rate compares its current average accuracy with the best performance on each corruption. A higher rate means less forgetting. Our method achieves the highest retention 96.88% in the final stage The results are shown below.

New taskOursSD-LoRAZSCLAlphaEditMEMIT
brightness100.00100.00100.00100.00100.00
elastic_transform99.6398.3089.7498.6497.82
gaussian_noise99.3598.1687.7598.5197.15
jpeg_compression98.6897.9586.9997.9397.36
saturate98.2497.6985.9497.3396.90
spatter98.1997.4684.5896.4596.44
contrast98.1097.2385.7396.0795.42
fog98.4396.6785.4695.5394.30
motion_blur98.1396.1985.1095.3793.75
shot_noise97.5595.7084.7194.1993.74
speckle_noise97.1795.3284.4193.4792.28
defocus_blur96.8894.3484.1192.7592.26

[1] Zhao, Hongbo, et al. MLLM-CL: Continual learning for multimodal large language models. arXiv preprint arXiv:2506.05453 (2025).

AuthorsOpenReview ↗

Response to Reviewer 3YVj (2/2)

Q2: Calculation of average performance across corruption types.

The results in Table 4 are inconsistent: all values in the Avg. column are incorrect. They should be 21.84, 28.57, 29.45, and 29.84, respectively. Table 1 has similar inconsistencies. These errors seriously undermine confidence in the experimental results.

Thank you for pointing this out. We would like to clarify that the average values reported in the paper are not calculated as the simple arithmetic mean of the four corruption categories. Instead, they are calculated by averaging the results across all 17 individual corruption types. Taking Qwen2.5-VL-3B as an example, the Noise, Blur, Weather, and Digital categories contain 4, 3, 4, and 6 corruption types, respectively. Therefore, the overall average is calculated by weighting the average accuracy of each category according to the number of corruption types it contains:

18.9×4+24.3×3+35.4×4+33.4×617=28.9\frac{18.9\times4+24.3\times3+35.4\times4+33.4\times6}{17}=28.9 instead of 18.9+24.3+35.4+33.44=28.0\frac{18.9+24.3+35.4+33.4}{4}=28.0 We will clarify the calculation procedure for the reported averages in the revised rebuttal to avoid any potential ambiguity.

Q4: Definition of sequential training tasks.

The task sequence is unclear. Are the first four tasks trained jointly, followed by joint training on four additional tasks, or are tasks learned one at a time? Is knowledge from previous tasks stored during training, and are any parameters reset between stages?

Thank you for this comment.

  1. Are the first four tasks trained jointly, followed by joint training on four additional tasks, or are tasks learned one at a time?

In our setting, tasks are learned one at a time.

  1. Is knowledge from previous tasks stored during training?

Yes. We store knowledge from previous corruptions in two KV aggregation matrices and update them as new corruptions are learned.

  1. Are any parameters reset between stages?

Yes. At each stage, we reset the model used for fine-tuning to the original pretrained parameters.

Thank you for the authors’ response. Most of my concerns have been addressed, and I am willing to maintain my score and increase my confidence.

AuthorsOpenReview ↗

Thank You for Your Constructive Feedback

Dear Reviewer 3YVj,

We sincerely thank you for your encouraging assessment in the initial review and for your positive follow-up. We are glad that our responses and revisions have addressed most of your concerns.

We greatly appreciate your constructive suggestions, which have helped us strengthen the experimental evidence, clarify our contributions, and improve the overall quality of our work.

Thank you again for your time, careful consideration, and valuable feedback throughout the ICLR review process.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Author General Response

We sincerely thank all reviewers for their time and valuable comments!


We have revised the main paper and supplementary material in response to the reviewers' comments. The main revisions are:

  • Pages 4–5: Clarified the continual adaptation protocol, KV aggregation matrices, and experimental settings. We also updated Algorithm 1 to better explain the adaptation procedure at each stage.
  • Appendix A.9 (Page 18): Added continual learning metrics and stage-by-stage retention results.
  • Appendix A.10 (Pages 18–19): Added corruption sensitivity tests on additional MLLMs.
  • Appendix A.11 (Page 19): Added comparisons with model-merging baselines.
  • Appendix A.12 (Page 20): Added experiments on preserving clean performance.
  • Appendix A.13 (Page 20): Added a hyperparameter ablation study.
  • Appendix A.14 (Pages 20–21): Added comparisons with joint training and test-time adaptation (TTA).
  • Appendix A.15 (Page 21): Added evaluations on additional MLLMs.
  • Appendix A.16 (Pages 21–22): Added R-Bench results using different Oracle models.

We are pleased that reviewers recognize the practical relevance of our problem, the clarity of our method design, and the comprehensiveness of our experimental evaluation.

1. Practical relevance and clear method design:

  • "The motivation is clear, and the problem is practically relevant." (Reviewer 3YVj)
  • "The method’s three components are interpretable." (Reviewer 3YVj)
  • "Editing the vision encoder instead of the LLM is a sensible choice for corruption robustness." (Reviewer 4f57)
  • "The closed-form stage only needs accumulated second-order statistics." (Reviewer 4f57)
  • "The overall framework is clearly structured, with reasonable motivations for each component." (Reviewer 7mh8)

2. Comprehensive experimental evaluation and robustness improvements:

  • "The paper introduces a new benchmark and provides extensive experiments." (Reviewer 3YVj)
  • "The comparison of fine-tuning strategies (Table 7) is a useful data point for the low-data regime." (Reviewer 4f57)
  • "Relatively comprehensive experimental evaluation." (Reviewer 7mh8)
  • "The results demonstrate improvements in visual robustness under limited-data settings while largely preserving the model's original capabilities." (Reviewer 7mh8)

3. Clear presentation:

  • "Clear and easy to follow." (Reviewer 3YVj, Presentation assessment)
  • "Clear and easy to follow." (Reviewer 7mh8, Presentation assessment)
AuthorsOpenReview ↗

Summary for the Area Chair

Dear Area Chair,

We sincerely thank you and the reviewers for their constructive feedback. We have addressed the concerns through additional experiments, methodological clarifications, and manuscript revisions. Below, we summarize our contributions, key rebuttal improvements, and the current review status.

Contributions.

  • Continual adaptation under limited data. We formulate MLLM visual robustness adaptation as a continual learning problem, where new visual corruptions are introduced sequentially using limited paired clean and corrupted images.
  • Dynamic model editing framework. We propose R-MLLM, which directly edits the visual encoder rather than the language model. It combines a robustness loss for feature alignment, dynamic key-value (KV) weighting to prioritize reliable samples, and a principal component constraint to mitigate catastrophic forgetting. Its optimization admits a closed-form solution using only two fixed-size KV aggregation matrices. Additive KV aggregation makes the method insensitive to task order, with constant additional memory regardless of the number of tasks.
  • VCBench benchmark. We introduce VCBench, covering 17 visual corruptions across four categories, with a filtering strategy to identify questions sensitive to visual corruptions.
  • Consistent empirical improvements. R-MLLM improves average VCBench accuracy by 7.56 and 7.03 percentage points on Qwen2.5-VL-7B and Qwen2.5-VL-3B, respectively, while largely preserving pretrained capabilities.

Main additions and improvements from the rebuttal.

  • Continual learning and forgetting analysis (3YVj, 4f57). We added a 12-corruption evaluation using four continual learning metrics (MFT, MFN, MAA, and BWT). R-MLLM achieves the best results across all four metrics, with a final knowledge retention rate of 96.88%, outperforming existing continual learning and model editing baselines.

  • Clarification of experimental protocol (3YVj). We clarified that corruption tasks are learned one at a time rather than jointly or in groups. At each stage, the Stage-1 model is reinitialized from the original pretrained parameters and trained only on the current corruption, while previous knowledge is preserved through accumulated KV statistics.

  • VCBench validation and claim clarification (4f57, 7mh8). We evaluated additional MLLMs not used in benchmark construction. Under Gaussian noise, these models show accuracy drops of 41.62–43.61 percentage points on VCBench, compared with only 3.01–5.75 points on RealWorldQA. We also clarified that our filtering criterion aims to identify questions more sensitive to visual corruptions, refining the scope and purpose of VCBench.

  • Comparisons with joint training and TTA (7mh8). We added comparisons with joint training and test-time adaptation methods (DeYO and SAR). In a controlled fog adaptation experiment, R-MLLM improves fog accuracy from 35.45% to 39.10%, while reducing average accuracy on seven previously encountered corruptions by only 0.28 percentage points. Its average accuracy across all eight corruptions reaches 36.47%, outperforming joint training (33.76%), DeYO (35.72%), and SAR (35.64%).

  • Comparisons with model-merging methods (4f57). Following the reviewer's suggestion, we compared R-MLLM with Uniform Soup, Task Arithmetic, TIES-Merging, and DARE. R-MLLM achieves the best results across all four corruption categories, with 28.84% average accuracy versus 26.57% for TIES-Merging and 26.44% for Uniform Soup. Stage-by-stage retention results further demonstrate its advantages as the number of tasks increases.

  • Cross-architecture generalizability (7mh8). We extended evaluations to InternVL3.5-8B, LLaVA-v1.6-vicuna-13B, and GLM-4.1V-9B, achieving average accuracy improvements of 10.53, 8.76, and 6.06 percentage points, respectively, demonstrating applicability across different MLLM architectures.

  • Clean performance and implementation analysis (4f57). We reported an average clean-benchmark accuracy decrease of 2.80 percentage points relative to the original model. We also compared Stage-1 fine-tuning, WiSE-FT, and clean-key preservation, and added hyperparameter sensitivity analysis, clarification of dynamic KV weighting, and evaluations using different task-specific Oracle models on R-Bench.

Revisions (in blue). Pages 4–5 clarify the protocol and method. Appendices A.9–A.16 add supporting experiments and analyses.

Status of the Reviews.

All three reviewers now recommend Weak Accept.

  • Reviewer 3YVj acknowledged that most concerns were addressed, maintained the original score, and increased confidence in the paper.
  • Reviewer 4f57 recognized that the additional experiments and clarifications addressed most concerns and raised the recommendation to Weak Accept.
  • Reviewer 7mh8 acknowledged our substantial efforts and additional experiments and raised the rating to Weak Accept.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.