acceptodds
Under review as a conference paper at ICLR 2027

MaLoRA: Mitigating Text-Centric Bias in Multimodal Large Language Models through Key-Space Alignment

Abstract

Multimodal large language models (MLLMs) often exhibit text-centric bias under joint image–text inputs, over-relying on textual signals while under-utilizing visual evidence. We analyze decoder self-attention and observe persistent cross-modal misalignment in the attention key space, where visual and textual keys form separated distributions consistent with preferential attention to text tokens. To mitigate this bias, we introduce MaLoRA, a text-centric bias mitigation training framework that directly intervenes in the self-attention key space to rebalance visual and textual evidence. MaLoRA combines modality-aware gated key adaptation, multi-kernel maximum mean discrepancy (MMD) alignment between visual and textual key distributions, and Gram-reference regularization to preserve within-modality key geometry during alignment. We compare MaLoRA with training-free decoding methods and trained adaptation baselines across multiple MLLM backbones. Evaluations cover general multimodal benchmarks and robustness tests targeting text-dominant failures, including image–text conflict and irrelevant-context settings, as well as hallucination benchmarks such as POPE, AMBER, and HallusionBench. The results show that MaLoRA reduces key-space divergence and improves robustness to text-dominant failures, with its clearest gains on hallucination-sensitive benchmarks and competitive performance in conflict-sensitive settings.

Primary area:
generative models
Keywords:
Multimodal & Large Multimodal Models, Language, Vision & Multi-modal

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tGBFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MaLoRA to reduce text-centric bias in multimodal language models. The method combines modality-specific low-rank adapters in the decoder key projection, learned gating, distribution alignment using multi-kernel MMD, and a structure-preserving regularizer. The authors report improvements across several backbones and benchmarks, including hallucination and irrelevant-text evaluations.

Critical strengths

The paper addresses an important problem: models following textual cues instead of relevant visual evidence.

The evaluation covers multiple backbones and a broad range of tasks. The HallusionBench improvement on LLaVA is particularly encouraging.

The parameter-matched LoRA comparison and component ablations provide useful evidence beyond comparisons with the original model.

Critical weaknesses

Not clear to me why visual and textual representations should have matching distributions. Different modalities may naturally need different representations.

Results show improvements, but less clear whether the proposed explanation is correct. Could the method help for reasons other than reducing the modality gap?

Wonder whether better visual encoder/connector training, or more training examples requiring visual evidence, could address the same problem. Relationship to these alternatives not sufficiently discussed.

Unclear how much the alignment loss adds. Gate supervision alone already gives much of the LLaVA improvement in Table 6.

Some experimental details hard to follow: data splits, data used for hyperparameter selection, meaning of “±,” and different LoRA ranks reported.

Questions for authors

I wonder if this problem can be solved with more data that combines text and visual, i.e., if this can be solved by scaling law?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for recognizing the importance of the text-centric bias problem, the broad evaluation setup, and the parameter-matched and component ablation experiments. In response to your questions about the motivation for alignment, the contributions of individual components, alternative solutions, and experimental details, we further clarified the methodological assumptions and added the AMBER ablation and MMD-weight sensitivity analyses. Our detailed responses are provided below.

1. Why should visual and textual representations be distributionally aligned? Should different modalities retain different representations? (Weakness 1)

We do not claim that visual and textual representations should be identical. Section 2 (page 3) of the original paper explicitly states that visual and textual tokens “originate from distinct semantic spaces” and therefore require different adaptation directions. Section 2 (page 4) further emphasizes that key-space alignment should “reduce cross-modal discrepancy while preserving modality-specific geometry.” Thus, our claim is that alignment should improve compatibility in the attention space, rather than eliminate modality differences.

Mechanistically, Equation (1) in Section 2 shows that the attention score is determined by q⊤kd\frac{q^\top k}{\sqrt{d}}. It also explains that visual and textual keys may exhibit distributional differences even when they share the same projection, so a query may be better matched to textual keys. Sections 3.2–3.4 therefore treat MMD as a soft constraint and combine it with modality-specific LoRA branches and Gram-reference regularization. The full objective still includes the task loss.

Empirically, Section 4.5 and Figure 4 show that Text-to-Visual attention increases after alignment, indicating that visual evidence becomes more accessible to textual queries. Therefore, our claim is that visual information can be used more effectively while modality-specific structure is preserved, rather than that the two modalities should be forced to completely overlap.

AuthorsOpenReview ↗

Rebuttal by Authors

2. Could the performance gains come from causes other than reducing the modality gap? How much additional benefit does the alignment loss provide, given that gate supervision already explains much of the LLaVA improvement? (Weakness 2 & Weakness 4)

MaLoRA jointly introduces modality-specific adaptation, dynamic gating, and multiple auxiliary training objectives. The performance gains therefore cannot be attributed solely to reducing the modality gap.

This is supported by the WeMath ablation results. On LLaVA-1.5-7B, adding gate supervision alone increases accuracy from 35.00% to 36.78%, close to the 36.84% achieved by full MaLoRA. Because correctness on WeMath depends jointly on the image, the question text, mathematical symbols, and language reasoning, this shows that the gating mechanism itself can provide a substantial benefit. The performance improvement is therefore not entirely due to reducing the modality gap.

To further test whether this observation also holds on a hallucination task with stronger visual dependence, we followed the training and evaluation settings in Table 4 and added a full component ablation on the AMBER Discriminative subset:

Lmmd\mathcal{L}_{\mathrm{mmd}}Lgram\mathcal{L}_{\mathrm{gram}}Lgate\mathcal{L}_{\mathrm{gate}}LLaVA-1.5-7BQwen3-VL-8B
77.60%81.51%
✓79.33%81.94%
✓78.71%81.80%
✓78.78%81.63%
✓✓78.84%81.63%
✓✓79.62%81.91%
✓✓80.35%82.08%
✓✓✓80.49%82.60%

On AMBER Discriminative, the ablation results show that the performance improvement does not come solely from reducing the modality gap. Compared with Gram + Gate (row 5), adding MMD (row 8) improves Qwen3-VL-8B and LLaVA-1.5-7B by 0.97% and 1.65%, respectively. Removing gate supervision reduces the scores by 0.52 and 0.14 points, respectively. Thus, although gate supervision contributes to the improvement, it cannot by itself explain the gain from full MaLoRA. MMD provides an additional and important performance gain.

We also added an MMD-weight sensitivity analysis for Qwen3-VL-8B on MMStar and WeMath. For each fixed λmmd\lambda_{\mathrm{mmd}}, we averaged the results over the tested combinations of Gram and Gate weights. The results are as follows:

MMD WeightAverage Accuracy on MMStarAverage Accuracy on WeMath
0.0572.67%69.10%
0.1573.04%69.21%
0.373.30%70.37%

Within the tested range λmmd∈{0.05,0.15,0.3}\lambda_{\mathrm{mmd}} \in \{0.05, 0.15, 0.3\}, after averaging over the different Gram and Gate weight combinations, MMStar accuracy increases from 72.67% to 73.04% and 73.30%. Neither task shows performance degradation, and the overall trend is upward, indicating that MMD is reasonably robust over the tested range.

Overall, the core role of MaLoRA is to align visual and textual representations in the attention key space, thereby reducing cross-modal geometric mismatch and making visual evidence more accessible to textual queries. Dynamic gating, modality-specific adaptation, and structure-preserving regularization provide additional expressive capacity and structural constraints for key-space alignment. The performance gains of MaLoRA therefore arise from the coordinated effects of key-space alignment and its individual components, rather than simply from reducing the modality gap.

3. Can improved visual-encoder/connector training, more high-quality image-text data, or scaling laws solve the same problem? (Weakness 3 & Question)

This question can be summarized as whether updating the model architecture, using higher-quality training data, and increasing model scale can eliminate the representation discrepancy. To examine this, we added a key-space analysis on Qwen3.5-397B-A17B. The following table reports the arithmetic means of the MMD/JS measurements for layers 57, 58, and 59, with layer indices starting from 0:

DatasetImage-text MMDImage-text JSImage-image MMDImage-image JSText-text MMDText-text JS
MMBench-EN0.4360.4760.030.0540.0160.071
MMMU0.4420.4140.0070.0310.0110.07

Across the shallow, middle, and deep layers measured on both datasets, the image-text MMD and JS are higher than the corresponding within-modality controls. The table further shows that this discrepancy remains in the deeper layers. This indicates that cross-modal key-space discrepancy is not limited to earlier or smaller models; it remains observable in a new-generation large-scale model such as Qwen3.5-397B-A17B. Therefore, improving visual-encoder/connector training and adding high-quality image-text training data do not resolve the image-text discrepancy in the key space. We will add these experimental results to the main text.

AuthorsOpenReview ↗

Rebuttal by Authors

4. Can adding training examples that require visual evidence solve the same problem? (Weakness 3)

Table 4 compares different fine-tuning methods using the same Word-or-Vision training data. On LLaVA-1.5-7B, MaLoRA achieves a HallusionBench score of 59.17, compared with 49.96 for standard LoRA and 47.56 for the parameter-matched LoRA-r72 baseline. This indicates that the same visual-related training signal cannot fully replace a specialized adaptation method and training objectives.

5. Data splits, hyperparameter selection, the meaning of “±,” and LoRA rank settings (Weakness 5)

Thank you for pointing out these issues affecting reproducibility. We clarify them as follows:

Data split. For datasets with official splits, we follow the corresponding official splits. For datasets without official train/test splits, we use an 8:2 split with a fixed random seed. All PEFT methods being compared are trained for one full epoch and evaluated using the checkpoint at the end of that epoch. Methods within the same experiment use the same data split and training protocol.

Hyperparameter selection. We use MMStar and WeMath for hyperparameter comparison because they are representative and relatively small, making them suitable for grid search. Based on these results, we select λmmd=0.30\lambda_{\mathrm{mmd}}=0.30, λgram=0.10\lambda_{\mathrm{gram}}=0.10, and λgate=0.15\lambda_{\mathrm{gate}}=0.15 and use this setting for the other benchmarks.

Meaning of “±”. The “±” in the tables denotes the standard deviation over three experimental runs.

LoRA rank settings. The standard LoRA configuration uses r=64r=64. LoRA-r72 uses rank r=72r=72 is included as an additional baseline with a trainable-parameter budget close to that of MaLoRA. The entry r=16r=16 in Table 8 is a typo, and we will correct it in the paper.

AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely appreciate your insightful comments and kindly invite you to let us know whether our detailed responses have addressed your concerns or if any further clarification would be helpful.

Reviewer_BDHkOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MaLoRA to mitigate text-centric bias in MLLMs through attention key-space alignment. The method combines modality-aware gated LoRA, MMD-based distribution alignment, and Gram regularization. Experiments across multiple MLLMs demonstrate improvements in multimodal understanding, hallucination mitigation, and cross-domain transfer.

Critical strengths
  1. The paper provides an interesting perspective on text-centric bias by investigating cross-modal discrepancies in the attention key space.
  2. The proposed method is well-motivated, and the combination of modality-aware adaptation and distribution alignment is reasonable.
  3. Extensive experiments across different backbones, benchmarks, and model scales demonstrate consistent improvements. The ablation studies and representation analyses further support the effectiveness of the method.
Critical weaknesses
  1. Although the paper demonstrates key-space discrepancies and increased Text-to-Visual attention after alignment, it remains unclear whether the discrepancies themselves directly cause text-centric bias. Could you provide the MMD or JS on big models like Qwen3.5-397B-A17B? Big model may still have big MMD/JS on Image vs. Text, but they still have good MLLM abilities.
  2. Table 6 reports the downstream performance of different loss combinations, but does not show how each combination affects MMD and JS divergence. Given the motivation in Table 1, such an analysis would strengthen the paper.
  3. Table 7 provides encouraging cross-domain results in image benchmarks, It would be valuable to examine whether video understanding capabilities are also preserved.
  4. Non-negligible inference overhead. Appendix Table 14 reports a 30.75% inference latency increase, which may limit practical applicability.
Questions for authors

Most questions are shown in the weakness section. Some more questions:

  1. In Figure 1 (a), the t-SNE visualization on LLaVA-1.5-7B is shown. However, LLaVA-1.5 is a out-of-time model, could you provide the t-SNE statistics on Qwen3-VL?
  2. I think the case in Figure 2 is too simple for modern MLLMs.
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for recognizing the key-space analysis perspective, the method design, and the experiments across backbones and benchmarks. In response to your suggestions concerning representations in large models, video understanding, inference overhead, and t-SNE visualization, we added the corresponding analyses and experiments. Our detailed responses are provided below:

1. Relationship between key-space discrepancies and text-centric bias, and MMD/JS analysis on larger models (Weakness 1) You mentioned: “Although the paper demonstrates key-space discrepancies and increased Text-to-Visual attention after alignment, it remains unclear whether the discrepancies themselves directly cause text-centric bias.” Model capability is jointly determined by multiple factors. In Section 2 of the original paper, we framed this as a mechanistic hypothesis: although visual and textual keys share the same projection, they may still exhibit distributional differences; when a query is better matched to textual keys, it can produce higher similarity and text-biased attention. We further described key-space misalignment as an “important factor” in text-centric bias.

The original Figure 4 shows that MaLoRA increases Text-to-Visual attention. To provide a more direct illustration of the role of key-space alignment, we added an ablation study on the AMBER Discriminative hallucination benchmark. The main results are shown below:

ModelMMD removedFull MaLoRAGain from adding MMD
Qwen3-VL-8B81.63%81.63\%82.60%82.60\%+0.97%+0.97\%
LLaVA-1.5-7B78.84%78.84\%80.49%80.49\%+1.65%+1.65\%

Setting λmmd=0\lambda_{\mathrm{mmd}}=0 reduces the accuracy of LLaVA-1.5-7B and Qwen3-VL-8B from 80.49% and 82.60% to 78.84% and 81.63%, respectively. This shows that, even when Gram regularization and gate supervision are already present, adding MMD still yields a substantial additional benefit. These results support the contribution of key-space alignment to improving multimodal behavior.

In response to your request, we also added MMD and JS measurements for Qwen3.5-397B-A17B. For efficiency, we sampled 100 examples from the official validation sets of MMBench-EN and MMMU, analyzed the model's key space, and computed the image-text discrepancy together with image-image and text-text within-modality controls. The table below reports the arithmetic means over the specified layers:

DatasetLayer indicesImage--Text MMDImage--Text JSImage--Image MMDImage--Image JSText--Text MMDText--Text JS
MMBench-EN0, 1, 20.8130.8130.8860.8860.040.040.0610.0610.0080.0080.1050.105
MMBench-EN29, 30, 310.4560.4560.5150.5150.0310.0310.0530.0530.0170.0170.0710.071
MMBench-EN57, 58, 590.4360.4360.4760.4760.030.030.0540.0540.0160.0160.0710.071
MMMU0, 1, 20.7590.7590.8460.8460.0080.0080.0370.0370.0090.0090.1060.106
MMMU29, 30, 310.4690.4690.4950.4950.0080.0080.0330.0330.0030.0030.070.07
MMMU57, 58, 590.4420.4420.4140.4140.0070.0070.0310.0310.0110.0110.070.07

Across all layer groups on both datasets, the image-text MMD and JS are higher than the corresponding within-modality controls, and this difference remains present in the deeper layers. At layers 57–59, the image-text MMD and JS on MMBench-EN are approximately 19.14 and 7.60 times the corresponding within-modality means, respectively; on MMMU, the ratios are 49.00 and 8.17. These results indicate that strong multimodal capability does not necessarily mean that visual and textual keys are already well aligned.

Our supplementary experiment further shows that, even in a large model such as Qwen3.5-397B-A17B, the key-space distributions of images and text still exhibit a clear discrepancy; increasing model scale does not eliminate this phenomenon. This experiment shows that the key-space discrepancy studied in this paper is stable across model scales and is not specific to small or early-generation multimodal models.

From a theoretical perspective, when a query is better matched to textual keys, it produces higher similarity and text-biased attention. The controlled ablations show that imposing an alignment constraint on this discrepancy can improve downstream performance. Together, these results support the necessity of studying key-space alignment. The MMD/JS analysis on the large model further shows that existing training procedures based on the current optimization objectives, training data, and model architecture do not fully align cross-modal key discrepancies; the cross-modal key-space discrepancy persists at larger scales.

AuthorsOpenReview ↗

Rebuttal by Authors

2. A gap between image and text keys does not by itself imply that query vectors prefer text (Weakness 2 & Question 2)

The downstream results in Table 6 need to be accompanied by per-configuration measurements of representation distributions to analyze the role of each training objective more directly. We therefore added distributional ablations for Qwen3-VL-8B and LLaVA-1.5-7B on WeMath. The results are as follows:

LLaVA-1.5-7B:

Lmmd\mathcal{L}_{\mathrm{mmd}}Lgram\mathcal{L}_{\mathrm{gram}}Lgate\mathcal{L}_{\mathrm{gate}}WeMath (%)MMDJS
35.00±0.1135.00 \pm 0.110.3580.3580.2250.225
✓36.78±0.03(↑1.78)36.78 \pm 0.03 (\uparrow 1.78)0.3600.3600.2300.230
✓36.15±0.15(↑1.15)36.15 \pm 0.15 (\uparrow 1.15)0.3580.3580.2260.226
✓36.67±0.13(↑1.67)36.67 \pm 0.13 (\uparrow 1.67)0.2440.2440.1470.147
✓✓36.67±0.11(↑1.67)36.67 \pm 0.11 (\uparrow 1.67)0.3600.3600.2310.231
✓✓36.61±0.15(↑1.61)36.61 \pm 0.15 (\uparrow 1.61)0.2360.2360.1420.142
✓✓36.32±0.07(↑1.32)36.32 \pm 0.07 (\uparrow 1.32)0.2430.2430.1460.146
✓✓✓36.84±0.17(↑1.84)36.84 \pm 0.17 (\uparrow 1.84)0.2360.2360.1420.142

Qwen3-VL-8B:

Lmmd\mathcal{L}_{\mathrm{mmd}}Lgram\mathcal{L}_{\mathrm{gram}}Lgate\mathcal{L}_{\mathrm{gate}}WeMath (%)MMDJS
68.10±0.1368.10 \pm 0.130.3450.3450.1630.163
✓69.88±0.17(↑1.78)69.88 \pm 0.17 (\uparrow 1.78)0.3550.3550.1700.170
✓68.74±0.11(↑0.64)68.74 \pm 0.11 (\uparrow 0.64)0.3490.3490.1650.165
✓68.16±0.17(↑0.06)68.16 \pm 0.17 (\uparrow 0.06)0.2680.2680.1180.118
✓✓69.66±0.06(↑1.56)69.66 \pm 0.06 (\uparrow 1.56)0.3470.3470.1640.164
✓✓70.00±0.13(↑1.90)70.00 \pm 0.13 (\uparrow 1.90)0.2680.2680.1180.118
✓✓68.62±0.06(↑0.52)68.62 \pm 0.06 (\uparrow 0.52)0.2920.2920.1290.129
✓✓✓70.75±0.14(↑2.65)70.75 \pm 0.14 (\uparrow 2.65)0.2930.2930.1290.129

For both models, all four configurations with MMD enabled have lower MMD and JS values than the four configurations without MMD. Compared with standard LoRA, using only Gram regularization, gate supervision, or their combination does not reduce either metric, indicating that these components cannot replace an explicit distribution-alignment objective. A further comparison between Gram + Gate and full MaLoRA shows that adding MMD substantially reduces the key-space discrepancy for both models.

3. Whether video understanding capabilities are preserved (Weakness 3)

Thank you for suggesting that we add cross-domain performance experiments on a video-understanding benchmark. Using Qwen2.5-VL-7B as the backbone, we followed the fine-tuning and evaluation settings used in our earlier cross-benchmark experiments and added results on Video-MME-v2:

ModelVideo-MME-v2 accuracyVideo-MME-v2 official grouped score
Base23.38%8.91
LoRA23.91%9.48
QLoRA24.00%9.37
MaLoRA24.16%9.50

On Video-MME-v2, MaLoRA improves accuracy over Base by 0.78%. It also improves the official grouped score by 0.59 points and achieves the highest score on both metrics. This metric divides the 3,200 questions into 800 related four-question groups, averages the score within each group, and jointly reflects answer accuracy, consistency, and reasoning coherence.

Therefore, on the video benchmark evaluated, MaLoRA outperforms both LoRA and QLoRA on the two reported metrics. This is consistent with the design of MaLoRA: while improving cross-modal alignment, it uses a within-modality structure-preservation constraint to retain as much of the pretrained model's general multimodal capability as possible. These results provide additional evidence that video understanding capability is preserved. We will add the video-understanding benchmark results to the paper.

AuthorsOpenReview ↗

Rebuttal by Authors

4. Inference overhead and practical applicability (Weakness 4)

The 30.75% increase in inference latency reported in Table 14 of the original paper is a real practical cost of MaLoRA. This overhead mainly comes from token-level dynamic gating, which cannot be merged. MMD, Gram regularization, and gate supervision are computed only during training. To present this trade-off more fully, we added inference time and peak memory measurements for the PEFT methods, as well as measurements for training-free hallucination-mitigation methods.

MethodPeak GPU Memory During Inference (GB)Time to Generate 20 Tokens (s)Inference Overhead
Base16.90.8150.00%
LoRA16.90.8080.00%
LoRA-r7216.90.8130.00%
QLoRA (A)6.92.1157.55%
DoRA16.910.8281.56%
AdaLoRA16.90.8030.00%
VCD16.931.59797.40%
ICD16.931.60298.00%
PAI16.91.52388.25%
OPERA17.984.977515.23%
MaLoRA16.940.99230.75%

Under the same settings, the inference overheads of the training-free methods VCD, ICD, PAI, and OPERA are 97.40%, 98.00%, 88.25%, and 515.23%, respectively, all substantially higher than MaLoRA's 30.75%. Thus, MaLoRA exchanges a moderate gating inference overhead for parameter-efficient adaptation and offers a more favorable effectiveness-efficiency trade-off under our experimental settings.

5. t-SNE statistics on Qwen3-VL (Question 1)

Appendix C.4 and Figure 6 (page 22) of the original manuscript already include a Base-versus-MaLoRA comparison for Qwen3-VL-8B on MMMU and MMBench-EN. We will strengthen this citation and present the analysis on the newer backbone more explicitly in the main paper.

We also analyzed the results shown in the figure. The silhouette scores annotated in Figure 6 are:

ModelDatasetBaseMaLoRA
Qwen3-VL-8BMMMU0.4520.372
Qwen3-VL-8BMMBench-EN0.5470.388

The silhouette scores decrease by 17.70% and 29.07% on the two benchmarks, respectively, indicating that the separation between visual and textual keys is substantially reduced. This is consistent with the reduced separation between visual and textual representations in the visualization. We will add the figure citation and these values to the main text.

6. The case in Figure 2 is too simple (Question 2)

Figure 2 is a schematic illustration of the framework. The simple example is intended to provide an intuitive explanation of how MaLoRA mitigates image-text conflicts. We will update the example in the framework illustration in the final version.

The actual model-based examples in the original manuscript are shown in Figures 3 and 5. They compare the outputs of Qwen3-VL-8B, a LoRA-fine-tuned model, and a MaLoRA-fine-tuned model under irrelevant-text interference. For example, Figure 3 shows interference in visual text/number recognition, while Figure 5 shows changes in responses to traffic-sign scenarios.

Reviewer_GQZjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MaLoRA introduces modality-specific, gated LoRA branches into the key projections of the attention mechanism. By incorporating a distribution-alignment loss, a Gram regularization term, and gate supervision, the method aims to reduce the model's over-reliance on text. The paper presents experimental results based on three main backbone models, covering general VQA tasks, hallucination benchmarks, and scenarios involving irrelevant text. It also provides an analysis of the key distributions and the resulting shifts in the attention mechanism.

Critical strengths
  1. Intervening at the key projection stage is a sensible design choice. Preserving intra-modality structure also addresses a concern with simply forcing image and text representations together.
  2. Table 4 offers the most compelling evidence. Its fine-tuned baselines use the same training data and include parameter-matched LoRA. On HallusionBench with LLaVA, MaLoRA scores 59.17, compared with 48.01 for Base and 49.96 for standard LoRA. The irrelevant-text tests and transfer results also provide useful checks beyond standard task-specific fine-tuning.
Critical weaknesses
  1. The separation between model selection and final evaluation is unclear. Appendix A describes an 8:2 train/test split when official splits are unavailable, followed by 27 MMStar runs to choose the loss weights. It does not specify an independent validation set, and the score of 75.00 in Table 2 matches the best grid-search result. This leaves unclear whether the reported test set was excluded from hyperparameter selection. The training data for the models in Table 12 and their separation from the irrelevant-text test subsets also need clarification.
  2. A gap between image and text keys does not by itself imply that queries prefer text. Lower MMD and higher visual attention after joint fine-tuning support the explanation, but do not isolate alignment from the other changes. Table 6 does show a contribution from MMD: adding it to Gram and gate supervision raises Qwen3's WeMath score from 69.66 to 70.75. My concern is that these loss ablations are only on WeMath. A corresponding comparison on direct text-bias or hallucination tests would better establish what alignment contributes.
Questions for authors
  1. Are the MMStar samples used for hyperparameter selection disjoint from those used in Table 2? Please also identify the training data used for Table 12 and whether its test images or variants of the same image-question pair overlap with training.
  2. On the irrelevant-text or hallucination tests, how does full MaLoRA compare with removing only MMD, keeping the gated architecture, other losses, parameter budget, training data and optimization settings fixed? This would help isolate the contribution of alignment.
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for recognizing our intervention at the attention key projection and our design choice of preserving within-modality structure during alignment. In response to your questions about data splits, hyperparameter selection, and the contribution of the alignment loss, we further clarified the experimental protocol and added component ablation results on AMBER Discriminative. Our detailed responses are provided below.

1. The separation between model selection and final evaluation is unclear (Weakness 1 & Question 1)

We used a consistent training and evaluation protocol for all PEFT comparison methods: each method was trained for one full epoch and then evaluated using the final checkpoint. For MMStar specifically, to ensure a fair data split, we used an 8:2 split with a fixed random seed, yielding separate training and test sets. This protocol was used consistently for the three models in Table 2 and for the hyperparameter-search experiments in Table 10.

We performed hyperparameter searches on MMStar and WeMath because they are representative of the problem we study and are relatively small, allowing us to obtain reliable conclusions quickly. After the experiments, we found that λmmd=0.30\lambda_{\mathrm{mmd}}=0.30, λgram=0.10\lambda_{\mathrm{gram}}=0.10, and λgate=0.15\lambda_{\mathrm{gate}}=0.15 produced relatively strong results on both datasets and were reasonably robust to the hyperparameter settings. We therefore used this set of hyperparameters for the experiments on the benchmarks in Table 2. We will clarify this protocol in the paper.

The training data used by the models in Table 12, and whether these data are independent of the irrelevant-text test subsets, also require further clarification. Table 12 uses a cross-dataset transfer setting. We fine-tuned the models on the official OCR-VQA training set, and none of the images, questions, or variants of the image-question pairs in the training set appears in the VQAv2-irrelevant, DocVQA-irrelevant, or OpenPhish-irrelevant test subsets.

2. A gap between image and text keys does not by itself imply that query vectors prefer text (Weakness 2 & Question 2)

Thank you for recognizing the effectiveness of our method in increasing Text-to-Visual attention. You mentioned that "a gap between visual and textual keys by itself does not imply that query vectors prefer text." In the mechanistic analysis in Section 2 and Equation (1) of the original paper, we show that the attention score is determined by q⊤kd\frac{q^\top k}{\sqrt{d}}. Because decoder queries inherit the language-oriented geometric structure of the language model, they may be more compatible with textual keys, so text tokens receive higher attention while visual keys become relatively less accessible. To further examine the role of key-space alignment, following your suggestion and using the same protocol as Table 4 in the original paper, we added component ablation experiments for Qwen3-VL-8B and LLaVA-1.5-7B on the Discriminative subset of the AMBER benchmark. The results are as follows:

Lmmd\mathcal{L}_{\mathrm{mmd}}Lgram\mathcal{L}_{\mathrm{gram}}Lgate\mathcal{L}_{\mathrm{gate}}LLaVA-1.5-7BQwen3-VL-8B
77.60%81.51%
✓79.33%81.94%
✓78.71%81.80%
✓78.78%81.63%
✓✓78.84%81.63%
✓✓79.62%81.91%
✓✓80.35%82.08%
✓✓✓80.49%82.60%

Comparing the full model (row 8) with the model that only removes the MMD loss (row 5), adding MMD increases accuracy by 0.97% and 1.65% for the two models, respectively. Thus, when dynamic gating and structure-preserving regularization are already present, MMD still provides an additional task-performance gain, supporting an independent contribution of key-space alignment to reliance on visual evidence.

Thank you for the detailed response and ablations. The new results substantially address my concern about the contribution of MMD, and the clarification of the Table 12 training data is helpful.

One point remains unclear regarding mmstar: were the scores in table 10 computed on the same 20% test split used for the final results in table 2, or was a separate validation set used for hyperparameter selection? A brief clarification would help me assess the independence of the final evaluation.

AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your follow-up questions. The MMStar results in Tables 10 and 2 were evaluated on the same 20% subset of the dataset, without a separate validation set. For Table 10, we performed hyperparameter search using a fixed training random seed. For Table 2, we fixed the selected hyperparameters, repeated training with different random seeds, and aggregated the results.

We further clarify the scope of our hyperparameter search as follows. Given the relatively small size of MMStar, systematically searching over the three loss weights incurs a low computational cost. We therefore selected MMStar for the grid search reported in Table 10. After selecting the final loss weights, we fixed them and applied them uniformly to all datasets in Table 2 other than MMStar, without further dataset-specific tuning of these weights.

Regarding hyperparameter sensitivity, the MMStar scores across all 27 loss-weight configurations in Table 10 range from 71.67% to 75.00%, consistently exceeding the standard LoRA score reported in Table 2 (70.33%). This indicates that the observed performance gains are not confined to a single optimal weight configuration, supporting the view that our framework is relatively insensitive to the specific loss-weight values and exhibits a degree of hyperparameter robustness.

Furthermore, with the final selected loss weights fixed, the model continued to achieve consistent performance improvements on benchmarks other than MMStar. This provides further support for the cross-task applicability of the selected configuration, indicating that performance gains can be achieved without tuning these loss weights separately for each dataset. We will explicitly clarify the data usage, the scope of hyperparameter search, and the setup for repeated runs in the captions of Tables 2 and 10 and in Appendix A.2–A.3.

Comments on the paper

1 comment

Dear Reviewers,

Please read the authors' response. Please evaluate whether your concerns are addressed or if anything needs further attention. Please engage with the authors while we have time.

Thanks, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.