- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MaLoRA to reduce text-centric bias in multimodal language models. The method combines modality-specific low-rank adapters in the decoder key projection, learned gating, distribution alignment using multi-kernel MMD, and a structure-preserving regularizer. The authors report improvements across several backbones and benchmarks, including hallucination and irrelevant-text evaluations.
Critical strengths
The paper addresses an important problem: models following textual cues instead of relevant visual evidence.
The evaluation covers multiple backbones and a broad range of tasks. The HallusionBench improvement on LLaVA is particularly encouraging.
The parameter-matched LoRA comparison and component ablations provide useful evidence beyond comparisons with the original model.
Critical weaknesses
Not clear to me why visual and textual representations should have matching distributions. Different modalities may naturally need different representations.
Results show improvements, but less clear whether the proposed explanation is correct. Could the method help for reasons other than reducing the modality gap?
Wonder whether better visual encoder/connector training, or more training examples requiring visual evidence, could address the same problem. Relationship to these alternatives not sufficiently discussed.
Unclear how much the alignment loss adds. Gate supervision alone already gives much of the LLaVA improvement in Table 6.
Some experimental details hard to follow: data splits, data used for hyperparameter selection, meaning of “±,” and different LoRA ranks reported.
Questions for authors
I wonder if this problem can be solved with more data that combines text and visual, i.e., if this can be solved by scaling law?
4 comments
Rebuttal by Authors
2. Could the performance gains come from causes other than reducing the modality gap? How much additional benefit does the alignment loss provide, given that gate supervision already explains much of the LLaVA improvement? (Weakness 2 & Weakness 4)
MaLoRA jointly introduces modality-specific adaptation, dynamic gating, and multiple auxiliary training objectives. The performance gains therefore cannot be attributed solely to reducing the modality gap.
This is supported by the WeMath ablation results. On LLaVA-1.5-7B, adding gate supervision alone increases accuracy from 35.00% to 36.78%, close to the 36.84% achieved by full MaLoRA. Because correctness on WeMath depends jointly on the image, the question text, mathematical symbols, and language reasoning, this shows that the gating mechanism itself can provide a substantial benefit. The performance improvement is therefore not entirely due to reducing the modality gap.
To further test whether this observation also holds on a hallucination task with stronger visual dependence, we followed the training and evaluation settings in Table 4 and added a full component ablation on the AMBER Discriminative subset:
| LLaVA-1.5-7B | Qwen3-VL-8B | |||
|---|---|---|---|---|
| 77.60% | 81.51% | |||
| ✓ | 79.33% | 81.94% | ||
| ✓ | 78.71% | 81.80% | ||
| ✓ | 78.78% | 81.63% | ||
| ✓ | ✓ | 78.84% | 81.63% | |
| ✓ | ✓ | 79.62% | 81.91% | |
| ✓ | ✓ | 80.35% | 82.08% | |
| ✓ | ✓ | ✓ | 80.49% | 82.60% |
On AMBER Discriminative, the ablation results show that the performance improvement does not come solely from reducing the modality gap. Compared with Gram + Gate (row 5), adding MMD (row 8) improves Qwen3-VL-8B and LLaVA-1.5-7B by 0.97% and 1.65%, respectively. Removing gate supervision reduces the scores by 0.52 and 0.14 points, respectively. Thus, although gate supervision contributes to the improvement, it cannot by itself explain the gain from full MaLoRA. MMD provides an additional and important performance gain.
We also added an MMD-weight sensitivity analysis for Qwen3-VL-8B on MMStar and WeMath. For each fixed , we averaged the results over the tested combinations of Gram and Gate weights. The results are as follows:
| MMD Weight | Average Accuracy on MMStar | Average Accuracy on WeMath |
|---|---|---|
| 0.05 | 72.67% | 69.10% |
| 0.15 | 73.04% | 69.21% |
| 0.3 | 73.30% | 70.37% |
Within the tested range , after averaging over the different Gram and Gate weight combinations, MMStar accuracy increases from 72.67% to 73.04% and 73.30%. Neither task shows performance degradation, and the overall trend is upward, indicating that MMD is reasonably robust over the tested range.
Overall, the core role of MaLoRA is to align visual and textual representations in the attention key space, thereby reducing cross-modal geometric mismatch and making visual evidence more accessible to textual queries. Dynamic gating, modality-specific adaptation, and structure-preserving regularization provide additional expressive capacity and structural constraints for key-space alignment. The performance gains of MaLoRA therefore arise from the coordinated effects of key-space alignment and its individual components, rather than simply from reducing the modality gap.
3. Can improved visual-encoder/connector training, more high-quality image-text data, or scaling laws solve the same problem? (Weakness 3 & Question)
This question can be summarized as whether updating the model architecture, using higher-quality training data, and increasing model scale can eliminate the representation discrepancy. To examine this, we added a key-space analysis on Qwen3.5-397B-A17B. The following table reports the arithmetic means of the MMD/JS measurements for layers 57, 58, and 59, with layer indices starting from 0:
| Dataset | Image-text MMD | Image-text JS | Image-image MMD | Image-image JS | Text-text MMD | Text-text JS |
|---|---|---|---|---|---|---|
| MMBench-EN | 0.436 | 0.476 | 0.03 | 0.054 | 0.016 | 0.071 |
| MMMU | 0.442 | 0.414 | 0.007 | 0.031 | 0.011 | 0.07 |
Across the shallow, middle, and deep layers measured on both datasets, the image-text MMD and JS are higher than the corresponding within-modality controls. The table further shows that this discrepancy remains in the deeper layers. This indicates that cross-modal key-space discrepancy is not limited to earlier or smaller models; it remains observable in a new-generation large-scale model such as Qwen3.5-397B-A17B. Therefore, improving visual-encoder/connector training and adding high-quality image-text training data do not resolve the image-text discrepancy in the key space. We will add these experimental results to the main text.
Rebuttal by Authors
4. Can adding training examples that require visual evidence solve the same problem? (Weakness 3)
Table 4 compares different fine-tuning methods using the same Word-or-Vision training data. On LLaVA-1.5-7B, MaLoRA achieves a HallusionBench score of 59.17, compared with 49.96 for standard LoRA and 47.56 for the parameter-matched LoRA-r72 baseline. This indicates that the same visual-related training signal cannot fully replace a specialized adaptation method and training objectives.
5. Data splits, hyperparameter selection, the meaning of “±,” and LoRA rank settings (Weakness 5)
Thank you for pointing out these issues affecting reproducibility. We clarify them as follows:
Data split. For datasets with official splits, we follow the corresponding official splits. For datasets without official train/test splits, we use an 8:2 split with a fixed random seed. All PEFT methods being compared are trained for one full epoch and evaluated using the checkpoint at the end of that epoch. Methods within the same experiment use the same data split and training protocol.
Hyperparameter selection. We use MMStar and WeMath for hyperparameter comparison because they are representative and relatively small, making them suitable for grid search. Based on these results, we select , , and and use this setting for the other benchmarks.
Meaning of “±”. The “±” in the tables denotes the standard deviation over three experimental runs.
LoRA rank settings. The standard LoRA configuration uses . LoRA-r72 uses rank is included as an additional baseline with a trainable-parameter budget close to that of MaLoRA. The entry in Table 8 is a typo, and we will correct it in the paper.
Rebuttal by Authors
We sincerely appreciate your insightful comments and kindly invite you to let us know whether our detailed responses have addressed your concerns or if any further clarification would be helpful.
Rebuttal by Authors
Thank you for recognizing the importance of the text-centric bias problem, the broad evaluation setup, and the parameter-matched and component ablation experiments. In response to your questions about the motivation for alignment, the contributions of individual components, alternative solutions, and experimental details, we further clarified the methodological assumptions and added the AMBER ablation and MMD-weight sensitivity analyses. Our detailed responses are provided below.
1. Why should visual and textual representations be distributionally aligned? Should different modalities retain different representations? (Weakness 1)
We do not claim that visual and textual representations should be identical. Section 2 (page 3) of the original paper explicitly states that visual and textual tokens “originate from distinct semantic spaces” and therefore require different adaptation directions. Section 2 (page 4) further emphasizes that key-space alignment should “reduce cross-modal discrepancy while preserving modality-specific geometry.” Thus, our claim is that alignment should improve compatibility in the attention space, rather than eliminate modality differences.
Mechanistically, Equation (1) in Section 2 shows that the attention score is determined by dq⊤k. It also explains that visual and textual keys may exhibit distributional differences even when they share the same projection, so a query may be better matched to textual keys. Sections 3.2–3.4 therefore treat MMD as a soft constraint and combine it with modality-specific LoRA branches and Gram-reference regularization. The full objective still includes the task loss.
Empirically, Section 4.5 and Figure 4 show that Text-to-Visual attention increases after alignment, indicating that visual evidence becomes more accessible to textual queries. Therefore, our claim is that visual information can be used more effectively while modality-specific structure is preserved, rather than that the two modalities should be forced to completely overlap.