- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes R-MLLM, a model editing framework designed to improve the robustness of multimodal large language models against visual corruptions under limited-data settings. The method fine-tunes the visual encoder using a robustness loss, introduces a dynamic key-value weighting strategy to prioritize high-quality samples, and employs a principal component constraint to mitigate catastrophic forgetting. The authors also introduce VCBench, a visual robustness benchmark covering 17 corruption types, with a filtering strategy intended to reduce reliance on language priors. Experiments on multiple benchmarks demonstrate improvements in visual robustness while largely preserving the model's original capabilities.
Critical strengths
Strengths
-
Clear method design. The paper combines visual feature alignment, dynamic key-value weighting, and principal component constraints to address feature shifts caused by visual corruptions, the influence of low-quality samples on model updates, and catastrophic forgetting, respectively. The overall framework is clearly structured, with reasonable motivations for each component.
-
Relatively comprehensive experimental evaluation. The authors evaluate their method on VCBench, R-Bench, and other visual robustness benchmarks, along with ablation studies to examine the contributions of individual components. The results demonstrate improvements in visual robustness under limited-data settings while largely preserving the model's original capabilities.
Critical weaknesses
- Insufficient justification for the continual learning formulation.
The paper formulates visual robustness as a continual learning problem, where new corruption types arrive sequentially. However, the practical motivation for this formulation remains insufficiently justified. The evaluated corruptions are all predefined and widely studied in existing robustness benchmarks, dating back to ImageNet-C (2019), rather than representing genuinely novel or emerging degradation patterns. The authors should clarify which real-world deployment scenarios require sequential adaptation to different corruption types, rather than joint robustness training or test-time adaptation. Moreover, the current experiments do not sufficiently demonstrate why this continual learning formulation is necessary or preferable to these alternatives.
- Questionable validity of the VCBench filtering strategy.
The construction of VCBench raises a fundamental concern. The authors claim that selecting samples answered correctly on clean images but incorrectly on corrupted images eliminates questions solvable by language priors. However, this criterion only identifies corruption-sensitive responses and does not establish that the questions require visual information. A question may still be answerable using language priors even if a model produces an incorrect response when given a corrupted image. Furthermore, the benchmark is constructed using the intersection of samples filtered by two reference models (Qwen and LLaVA), which may introduce model-dependent selection bias, particularly given that Qwen is also the primary model family evaluated in this work. The authors should provide text-only evaluations and investigate how the choice of reference models affects the resulting benchmark.
- Limited novelty of VCBench compared with existing benchmarks.
The novelty of VCBench appears limited. The 17 corruption types are already covered by existing corruption benchmarks, including the standard and extended ImageNet-C corruptions. Moreover, benchmarks such as R-Bench have explicitly addressed the evaluation of MLLMs under real-world visual corruptions. Therefore, the primary distinction of VCBench seems to lie in its sample-filtering strategy rather than in new corruption types or evaluation settings. Given the concerns regarding this filtering strategy, the authors should more clearly articulate the unique contribution of VCBench and provide a systematic comparison with existing benchmarks to demonstrate its additional value.
- Insufficient evaluation of generalizability across MLLM architectures.
The generalizability of the proposed framework remains insufficiently demonstrated. Although the experiments include Qwen2.5-VL-3B and Qwen2.5-VL-7B, both models belong to the same architecture family. Since the method relies on specific FFN projection layers in the visual encoder, it is unclear whether the proposed editing strategy can be effectively applied to other MLLM architectures. Evaluating the method on additional model families, such as InternVL or LLaVA, would help substantiate its claimed general applicability.
Questions for authors
-
Continual learning setting. Could the authors clarify the practical scenarios that motivate sequential adaptation to different corruption types? In particular, why is continual model editing preferable to joint robustness training or test-time adaptation in these scenarios?
-
Validity of VCBench filtering. How do the authors justify the claim that the proposed filtering criterion eliminates questions answerable through language priors? Could the authors provide text-only evaluation results to support this claim? Additionally, how sensitive is VCBench to the choice of reference models used for filtering, particularly since Qwen is also the primary model family evaluated?
-
Novelty of VCBench. Given that the 17 corruption types are already covered by existing benchmarks, what specific evaluation limitations does VCBench address beyond the proposed sample-filtering strategy? Could the authors provide evidence that VCBench offers additional evaluation value compared with existing benchmarks?
-
Generalizability. Can the authors provide evidence that the proposed framework generalizes beyond the Qwen2.5-VL family? Since the method relies on specific FFN projection layers, results on at least one MLLM with a different visual encoder architecture would help establish its general applicability.
6 comments
Response to Reviewer 7mh8 (2/2)
Q2&Q3: Construction and validation of VCBench.
Validity of VCBench filtering. How do the authors justify the claim that the proposed filtering criterion eliminates questions answerable through language priors? Could the authors provide text-only evaluation results to support this claim? Additionally, how sensitive is VCBench to the choice of reference models used for filtering, particularly since Qwen is also the primary model family evaluated?
Novelty of VCBench. Given that the 17 corruption types are already covered by existing benchmarks, what specific evaluation limitations does VCBench address beyond the proposed sample-filtering strategy? Could the authors provide evidence that VCBench offers additional evaluation value compared with existing benchmarks?
Thank you for pointing this out. We add experiments evaluating multiple MLLMs on VCBench to further demonstrate the validity and fairness of the benchmark. Our goal is to select questions that become difficult under visual corruptions. We keep questions that the filtering models answer correctly on clean images but incorrectly on corrupted images. This criterion identifies questions on which corruptions affect the models' answers. It does not show that these questions cannot be answered from text alone. We will revise the description of VCBench to clarify this point.
When corruptions are added to existing VQA benchmarks, many questions show little change in accuracy. VCBench focuses on questions that are sensitive to corruptions. To check whether the performance drop also appears in models that were not used for filtering, we evaluate LLaVA-v1.6-vicuna-13B, InternVL3.5-8B, and GLM-4.1V-9B-Thinking. We compare their results on clean images and images with Gaussian noise in both VCBench and RealWorldQA. As shown below, these models also show large performance drops on VCBench.
| Model | VCBench (clean) | VCBench (noise) | RealWorldQA (clean) | RealWorldQA (noise) |
|---|---|---|---|---|
| LLaVA-NeXT-13B | 75.07% | 31.54%(↓43.53) | 58.17% | 55.16%(↓3.01) |
| GLM-4.1V-9B-Thinking | 73.24% | 31.62%(↓41.62) | 69.80% | 64.05%(↓5.75) |
| InternVL3.5-8B | 71.76% | 28.15%(↓43.61) | 67.19% | 63.01%(↓4.18) |
| Qwen2.5-VL-7B(filtering model) | 70.13% | 23.64%(↓46.49) | 67.97% | 61.65%(↓6.32) |
Q4: Generalizability across different MLLM architectures.
Generalizability. Can the authors provide evidence that the proposed framework generalizes beyond the Qwen2.5-VL family? Since the method relies on specific FFN projection layers, results on at least one MLLM with a different visual encoder architecture would help establish its general applicability.
Thank you for highlighting the importance of evaluating generalizability across different MLLM architectures. We add experiments on multiple MLLMs, including LLaVA-v1.6-vicuna-13B, InternVL3.5-8B, and GLM-4.1V-9B-Thinking, to further evaluate the generalizability of our method. As shown in the table below, our method consistently improves the robustness of all three models.
| Model | Noise | Blur | Weather | Digital | Avg |
|---|---|---|---|---|---|
| InternVL3.5-8B | 25.26 | 22.70 | 27.54 | 33.30 | 28.18 |
| InternVL3.5-8B +Ours | 33.15 (↑7.89) | 37.21 (↑14.51) | 37.75 (↑10.21) | 43.82 (↑10.52) | 38.71 (↑10.53) |
| InternVL3.5-8B (Oracle) | 33.74 | 38.42 | 40.03 | 45.36 | 40.15 |
| LLaVA-v1.6-vicuna-13B | 26.93 | 27.39 | 34.90 | 36.37 | 32.22 |
| LLaVA-v1.6-vicuna-13B + Ours | 37.40 (↑10.47) | 36.89 (↑9.50) | 42.35 (↑7.45) | 44.51 (↑8.14) | 40.98 (↑8.76) |
| LLaVA-v1.6-vicuna-13B (Oracle) | 38.64 | 37.95 | 45.09 | 45.94 | 42.61 |
| GLM-4.1V-9B | 15.85 | 19.31 | 25.23 | 29.07 | 23.33 |
| GLM-4.1V-9B + Ours | 24.13 (↑8.28) | 25.83 (↑6.52) | 31.28 (↑6.05) | 33.43 (↑4.36) | 29.39 (↑6.06) |
| GLM-4.1V-9B (Oracle) | 25.65 | 26.97 | 34.12 | 34.28 | 30.92 |
Gentlel Request to Consider Our Rebuttal
Dear Reviewer 7mh8,
Thank you again for your thoughtful feedback and suggestions. As the discussion period ends soon, we wanted to check whether our responses address your questions. We have provided additional explanations and experimental results addressing your concerns, including comparisons with joint training and TTA, further evaluations of VCBench using multiple MLLMs, and cross-architecture generalizability experiments on InternVL3.5, LLaVA-v1.6, and GLM-4.1V. We have also clarified the scope and motivation of our benchmark filtering strategy. We would be glad to address any remaining questions before the deadline. If our responses and additional results have addressed your concerns, we would be grateful if you would update your score.
Response to Reviewer 7mh8 (1/2)
Thank you for your diligent efforts in reviewing our paper and for recognizing the contributions of our work.
Q1: Practical scenarios and method advantages.
Thank you for raising this question. To address your question, we clarify the practical scenario considered in our experiments and add comparisons with joint training and test-time adaptation (TTA) methods to further evaluate the effectiveness of our approach. In real-world applications, such as long-term autonomous driving, models may encounter different image corruptions sequentially as vehicles operate across different regions, seasons, and weather conditions. For example, a vehicle may first encounter rainy conditions, then foggy or snowy environments, and later return to previously encountered conditions. Joint training requires retraining on all previous and new corruption scenarios whenever a new corruption appears, resulting in high training costs. TTA methods primarily adapt to the current test distribution and may suffer from forgetting previously encountered corruptions. We therefore study how to incrementally update an existing model using a small number of clean and corrupted image pairs for each new corruption. Our goal is to adapt to new corruptions while retaining performance on previous corruptions.
To simulate a model encountering a new corruption, we first train a model on seven corruption types: brightness, elastic_transform, gaussian_noise, jpeg_compression, saturate, spatter, and contrast. We call this model "base model". Fog is not used to train this model. Then, we introduce fog data for adaptation to this new corruption. Our method and the two TTA methods, DeYO and SAR, start from the same base model. Joint-training is trained jointly on data from all eight corruption types. Ours is obtained by further editing the base model to adapt to fog. As shown in the table below, Our method improves accuracy on fog from 35.45% to 39.10%, a gain of 3.65 percentage points. Its average accuracy on the seven previous corruptions changes from 36.38% to 36.10%, a decrease of 0.28 percentage points. Across all eight corruptions, our method achieves 36.47%, compared with 33.76% for joint training, 35.72% for DeYO, and 35.64% for SAR. These results show that our method adapts to fog while largely retaining its performance on the previous corruptions.