acceptodds
Under review as a conference paper at ICLR 2027

Reducing Hallucinations in Multimodal LLMs with Modality-Aware Sparse Autoencoders

Abstract

Multimodal large language models (MLLMs) often hallucinate visual content absent from the input image. Preference optimization has become a common post-training approach for mitigating hallucinations, but existing preference signals often rely on LLM-as-Judge supervision that can be sensitive to linguistic style rather than visual grounding. We thus introduce Modality-Aware Sparse Autoencoders (MA-SAE), an SAE architecture that separates vision-specific, text-specific, and cross-modal features to provide a visual grounding signal for preference labeling. By isolating modality-specific patterns, such as hedging and stylistic scaffolding, MA-SAE learns interpretable cross-modal features from which we derive a grounding score measuring how well a response is supported by the image. Across three MLLM backbones, MA-SAE learns more coherent and better aligned cross-modal features than existing SAE baselines, achieving the best performance across five feature quality metrics. Using its grounding score for preference labeling, MA-SAE substantially reduces hallucinations under Direct Preference Optimization (DPO), consistently outperforming LLM-as-Judge and other preference-labeling methods across three benchmarks. The grounding signal further transfers to mDPO and on-policy preference tuning, where MA-SAE achieves strong overall performance. These results show that interpretable cross-modal features can provide a direct signal of visual grounding, offering an alternative to LLM-as-Judge supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Vision-Language Models, Hallucination Mitigation, Sparse Autoencoders

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dygVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Modality-Aware Sparse Autoencoders for reducing hallucinations in MLLM. The method mainly divided the SAE features into vision-specific, text-specific and cross-modal banks. Then, a Barlow Twin is used to align corresponding vision-text features. Experimental results on several benchmarks verified the effectiveness of the proposed methods over different preference-labeling baselines.

Critical strengths
  1. The paper addresses an important problem in MLMMs and propsoed a princiled approach to reducing hallucinations.

  2. The method is evaluated across several multiple PO settings, and the experimental improvements suggest that modality-aware features can provide useful grounding signals for reducing hallucination.

  3. The paper is well-orgnized and easy to follow.

Critical weaknesses
  1. The novelty of the proposed methods is incrementatl. Modality-specific feature partition and cross-modal alignment largely combine existing SAE techniques

  2. The paper's presentation could be significantly improved. The core motivation and key insights related to linguistics bias should be introduced in the main text rather than in the appendix, as they are essential for understanding the core contribution of the proposed method.

  3. In the proposed methods, the Barlow Twin is used to perform the cross-modal alignment. While increasing diagonal correlations encourages alignment between identically indexed visual and textual features, driving off-diagonal correlations toward zero implicitly assumes that features with different indices should be uncorrelated across modalities. However, different features may capture semantically related or complementary concepts, and their cross-modal correlations are not necessarily undesirable. The paper should provide a clearer justification for this constraint.

  4. Following the previous concern, it's not clear of why not using MSE or KL loss to perform the alignment?

  5. MA-SAE produces substantially shorter responses, which may affect hallucination metrics. The authors should conduct length-controlled comparisons to verify that the reported improvements reflect better visual grounding rather than reduced response verbosity.

Questions for authors

Please refer to the weakness.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank Reviewer dygV for the thoughtful comments. We have revised the manuscript to address the reviewers' comments, with changes highlighted in blue. We address your specific concerns as follows.


W1. The novelty of the proposed methods is incremental. Modality-specific feature partition and cross-modal alignment largely combine existing SAE techniques.

A1. We respectfully note that our contribution extends beyond the SAE architecture itself. To our knowledge, MA-SAE is the first work to use modality-disentangled SAE features as a preference-labeling signal for multimodal preference pairs, without relying on conventional preference-labeling methods such as LLM-as-judge, and to demonstrate that such supervision effectively reduces hallucinations. Prior multimodal SAE methods primarily focus on interpretability (e.g., VL-SAE and GSAE), while SAE-V uses vanilla SAE features to filter existing preference data for alignment, rather than directly assigning preference labels based on visual grounding.

Although modality-specific feature partitioning may appear straightforward, it is a principled design motivated by the linguistic bias we identify in LLM judges for multimodal preference labeling. In terms of interpretability, MA-SAE outperforms existing SAEs designed primarily for interpretability across all five feature-quality metrics. When applied to preference labeling, we further demonstrate that disentangling modality-specific context from cross-modal grounding features helps mitigate the linguistic bias observed in LLM-as-judge supervision. This enables MA-SAE to generate preference labels that lead to substantially greater reductions in hallucinations than LLM-as-judge or alternative labeling baselines. These findings establish a novel connection between modality-disentangled SAE representations and preference alignment, demonstrating how modality-aware feature separation can serve as a more effective grounding signal for multimodal preference supervision.


W2. The core motivation and key insights related to linguistics bias should be introduced in the main text rather than in the appendix.

A2. We thank the reviewer for this valuable suggestion. We have revised the manuscript to incorporate the key findings on linguistic bias in LLM judges into the main text (Section 4.2).


W3. Does the Barlow Twins off-diagonal regularization suppress meaningful correlations between semantically related or complementary cross-modal features? The paper should provide a clearer justification for this regularization.

A3. We thank the reviewer for this insightful observation. We clarify that the off-diagonal term in Barlow Twins serves as a soft regularizer rather than a hard constraint. In our implementation, it is assigned a small weight of λoff=0.005\lambda_{\mathrm{off}}=0.005 and averaged over all pcross(pcross−1)p_{\mathrm{cross}}(p_{\mathrm{cross}}-1) off-diagonal feature pairs. Thus, it mildly discourages redundant features without explicitly forcing semantically related features to be uncorrelated.

To further investigate this concern, we conduct an ablation varying λoff\lambda_{\mathrm{off}} over four orders of magnitude. We find that a small, nonzero regularization weight encourages feature diversity (i.e., lower inter-feature similarity) while preserving intra-feature similarity. Moreover, feature pairs that frequently co-occur (e.g., co-activate on the same inputs within each modality) retain a mean off-diagonal cross-modal correlation of 0.60, compared with approximately zero for non-co-occurring pairs. This indicates that the off-diagonal regularization does not suppress meaningful semantic co-occurrence.

Consistent with the reviewer's concern, stronger off-diagonal regularization degrades feature quality. Increasing λoff\lambda_{\mathrm{off}} by four orders of magnitude reduces MMS v–t from 0.24 to 0.06. These findings motivate our choice of a small, nonzero λoff\lambda_{\mathrm{off}}, which balances feature diversity and semantic coherence. We have revised Section 3.2 to clarify this design choice.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

W4. Following the previous concern, it's not clear of why not using MSE or KL loss to perform the alignment?

A4. Both MSE and KL alignment losses admit trivial solutions in which the cross-modal bank becomes inactive or learns constant features, rather than meaningful cross-modal representations.

  • MSE alignment loss: MSE can trivially reach its minimum of zero when the cross-modal features for both image and text are zero. The cross-modal bank can therefore become inactive while the modality-specific banks handle reconstruction. Empirically, when training MA-SAE with MSE alignment, we observe that the cross-modal bank nearly collapses, averaging only 0.04 active features per token.

  • KL alignment loss: KL normalizes the features into a probability distribution, avoiding the all-zero solution. However, it still admits a constant solution in which the same features activate across all image and text tokens, regardless of the input. Empirically, KL alignment leads to this collapse, with the cross-modal bank activating a nearly fixed set of features across all samples and only 57 of 8,192 cross-modal features activating on both modalities.

In contrast, inactive or constant features have no variation across samples and therefore cannot satisfy the diagonal correlation objective of Barlow Twins, resulting in a high alignment loss. We have revised Section 3.2 to discuss these degenerate solutions of MSE and KL alignment and to clarify why we adopt the Barlow Twins objective.

As requested, we further compare the feature quality of these three alignment objectives on LLaVA-1.5-7B. As shown below, Barlow Twins consistently outperforms MSE and KL across all metrics, demonstrating its effectiveness in learning meaningful cross-modal representations.

Alignment LossMMS v-tMMS v-vMMS t-t
MSE0.09010.28340.6363
KL0.00580.03440.0991
Barlow Twins (MA-SAE)0.23860.65090.6960
AuthorsOpenReview ↗

Official Comment by Authors (3/3)

W5. The authors should conduct length-controlled comparisons to verify that the reported improvements reflect better visual grounding rather than reduced response verbosity.

A5. We thank the reviewer for this important suggestion. We first highlight that on-policy tuning provides a more controlled setting, where preferred and rejected responses have more comparable lengths, mitigating response-length confounding during DPO training. In this setting, MA-SAE achieves both higher object coverage and lower hallucination rates than LLM-as-Judge, demonstrating that its hallucination reductions are not solely attributable to reduced coverage. To further investigate this concern, we conduct two additional experiments:

1. Matched-coverage comparison. For the off-policy LLaVA models, we compare MA-SAE and LLM-as-Judge on AMBER images for which both models identify exactly the same set of ground-truth objects (442 images for 7B and 479 for 13B). At matched coverage, MA-SAE consistently achieves lower CHAIR, HalRate, and Cog, as shown below. The reductions are statistically significant (p<0.001p < 0.001) and comparable to those observed on the full benchmark. These results demonstrate that MA-SAE's hallucination improvements cannot be explained solely by reduced coverage.

ModelMethodCHAIR ↓Cover ↑HalRate ↓Cog ↓
LLaVA-1.5-7BLLM-as-Judge5.654.731.72.4
MA-SAE3.854.715.81.5
LLaVA-1.5-13BLLM-as-Judge6.156.132.42.5
MA-SAE3.956.117.51.4

2. Length-controlled preference-labeling experiment. To directly evaluate whether MA-SAE's grounding score captures factual correctness independently of response length, we conduct a controlled experiment using over 2,000 hallucination-free captions from MSCOCO and Object HalBench, verified against ground-truth object annotations. For each caption, we construct a hallucinated counterpart by replacing all mentions of one grounded object with an object absent from the image, using two strategies:

  • Random swap: The replacement object is uniformly sampled from objects absent from the image.

  • Plausible swap: The replacement object is selected from absent objects that frequently co-occur with the scene's objects in MSCOCO (e.g., fork → knife in a dining scene), creating more realistic and challenging hallucinations.

This controlled setup ensures that each response pair differs only by the substitution of a grounded object with a hallucinated one, thereby preserving linguistic style and response length. We evaluate pairwise preference-labeling accuracy, measured as the proportion of pairs in which the hallucination-free response is preferred. MA-SAE consistently outperforms the LLM-as-Judge baseline across both settings and maintains high accuracy even on plausible swaps, where hallucinated objects are plausible given the scene context.

MethodRandom swap Accuracy (%)Plausible swap Accuracy (%)
Length50.050.0
LLM-as-Judge81.679.3
MA-SAE (LLaVA-7B)94.388.4
MA-SAE (LLaVA-13B)96.690.5
MA-SAE (InternVL3-8B)95.688.7

Together, these analyses provide complementary evidence that MA-SAE's hallucination reductions reflect better visual grounding rather than merely reduced response length or object coverage.

We have added a new Appendix J to analyze coverage differences between off-policy and on-policy tuning and to present these additional experiments. We have also summarized the findings in Section 4.4 of the revised manuscript.


We hope these clarifications are helpful. We would be happy to address any further questions and would appreciate the reviewer’s consideration of the new analyses and results above.

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We would like to kindly follow up on our response to your comments and hope that our additional experiments and clarifications have addressed your concerns.

We would be grateful to hear whether our response has addressed your concerns, so that we have an opportunity to clarify any remaining questions before the discussion period ends.

Thank you again for your valuable feedback and thoughtful review!

Reviewer_VxuDOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work aims to tackle the hallucinations in multimodal LLMs and the authors proposed Modality-Aware Sparse Autoencoders which separates vision-specific, text-specific and cross-modal feature to provide a visual grounding score for preference labeling. Experiments across three MLLM backbones demonstrate the effectiveness of the proposed method in reducing hallucinations.

Critical strengths
  1. The proposed method identifies the linguistic biases, which is reasonable
  2. The authors decomposed the feature space into vision-specific, text-specific and cross-modal features and they used cross-modal features for label preference pairs.
  3. The grounding-score can provide a direct visual grounding signal for preference supervision without relying on LLM judges.
Critical weaknesses
  1. The proposed method computes a grounding score with cross-modal bank to measure how well a response is supported by the image, which aims to filter out text-specific patterns and vision-specific patterns. However, if two sentences contain the same objects with the image while represent different semantics, how to tackle this issue.
  2. What is the pairwise ranking accuracy of the grounding score.
  3. The authors stated that in the case of an exact score tie, the preferred response is selected at random. I would like to check how frequently such ties occur and how reliable the resulting preference labels are.
  4. The authors argued that the LLM-as-Judge can be sensitive to linguistic style rather than visual grounding, which is identified as linguistic biases. I t remains unclear whether the grounding score is robust to linguistic style variation. For example, I would like to know whether the proposed method is sensitive with the changes in hedging, preamble or stylistic filler.
Questions for authors

See weakness above

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank reviewer VxuD for the insightful comments. We have revised the manuscript to address the reviewers' comments, with changes highlighted in blue. We address your specific concerns as follows.


W1. How does MA-SAE handle sentences that mention the same objects but convey different semantics?

A1. We thank the reviewer for this insightful question. As discussed in our limitations section, this work focuses on object-level hallucination, the most widely studied form of multimodal hallucination [1], while leaving attribute- and relation-level hallucinations for future work. Here, we outline a future direction that could tackle this case.

MA-SAE features are extracted from the MLLM's contextualized hidden states rather than individual words, so two sentences mentioning the same objects but conveying different semantics (e.g., "a man feeding a horse" vs. "a horse feeding a man") can produce different feature activations and potentially different grounding scores. However, because our current formulation aggregates these features over the entire response, such compositional differences may not be reliably captured. Future work could extend the grounding score to explicitly capture these semantic differences, for example, through phrase-level grounding that verifies whether attributes and actions are supported by the image regions corresponding to the objects they describe. We have revised Section 5 of the manuscript to discuss this limitation and a potential direction for future work.

[1] Bai et al. "Hallucination of Multimodal Large Language Models: A Survey."


W2. What is the pairwise ranking accuracy of the grounding score.

A2. We thank the reviewer for this question. To directly evaluate the pairwise ranking accuracy of the proposed grounding score, we construct controlled response pairs that differ in factual correctness while minimizing confounding factors such as linguistic style and response length, as also suggested by Reviewer fG6z.

Experimental setup. We start with over 2,000 hallucination-free captions from MSCOCO and Object HalBench images, verified against ground-truth object annotations. For each caption, we construct a hallucinated counterpart by replacing all mentions of one grounded object with an object absent from the image, using two strategies.

  • Random swap: The replacement object is uniformly sampled from objects absent from the image.

  • Plausible swap: The replacement object is selected from absent objects that frequently co-occur with the scene's objects in MSCOCO (e.g., fork → knife in a dining scene), creating more realistic and challenging hallucinations.

This controlled setup ensures that each response pair differs only by the substitution of a grounded object with a hallucinated one, thereby preserving linguistic style and response length. We evaluate pairwise preference-labeling accuracy, measured as the proportion of pairs in which the hallucination-free response is preferred. MA-SAE consistently outperforms the LLM-as-Judge baseline across both settings and maintains high accuracy even on plausible swaps, where hallucinated objects are plausible given the scene context.

MethodRandom swap Accuracy (%)Plausible swap Accuracy (%)
LLM-as-Judge81.679.3
MA-SAE (LLaVA-7B)94.388.4
MA-SAE (LLaVA-13B)96.690.5
MA-SAE (InternVL3-8B)95.688.7

We have included this pairwise preference-labeling accuracy experiment in the new Appendix J.2 and summarized its key findings in Section 4.4 of the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W3. I would like to check how frequently such ties occur and how reliable the resulting preference labels are.

A3. Exact score ties between distinct responses are extremely rare, occurring in only 3–5 out of 10,000 pairs (≤0.05%) across the three backbones in the off-policy setting and at most 11 out of 10,000 pairs (≤0.11%) in the on-policy setting. Therefore, random tie-breaking affects only a negligible fraction of preference labels and has minimal impact on the overall reliability of the preference signal.

BackboneOff-policy tiesOn-policy ties
LLaVA-1.5-7B5 / 10,00011 / 10,000
LLaVA-1.5-13B5 / 10,00011 / 10,000
InternVL3-8B3 / 10,0002 / 10,000

W4. It remains unclear whether the grounding score is robust to linguistic style variation. For example, I would like to know whether the proposed method is sensitive with the changes in hedging, preamble or stylistic filler.

A4. We thank the reviewer for this question. We apply the linguistic hedging rewrite to the same 96 MMHal-Bench responses and evaluate their grounding scores using MA-SAE across all three backbones. We observe an average decrease of only 0.56% in the grounding score, with no statistically significant change (p>0.05p > 0.05), suggesting that MA-SAE is relatively insensitive to linguistic variations.

We further examine this behavior at scale by analyzing hedging in preferred and rejected responses from VLFeedback labeled by MA-SAE. As shown in Appendix A.4, preferred and rejected responses contain nearly identical numbers of hedging expressions (3,032 vs. 2,977 occurrences; 1.02×). In contrast, LLM-as-Judge-based preference labels favor hedged responses by 1.48× (2,664 vs. 1,794 occurrences; Appendix A.2). These findings suggest that MA-SAE preference labeling is substantially less sensitive to hedged wording than LLM-as-Judge supervision.


We hope these clarifications are helpful. We would be happy to address any further questions or concerns the reviewer may have.

Reviewer_VxuDOpenReview ↗

Rating after rebuttal

Thank you for the detailed rebuttal. The authors added more experiments to strengthen the paper. However, I carefully read other reviewers' command and I agree that the contribution is a litte limited which seems like a combination of exsiting studies, and grounding beyond the object presence remains unverified. Hence, I will reduce my rate to 2.

We thank the reviewer for taking the time to carefully consider our rebuttal and the additional experiments. We appreciate your acknowledgment that these experiments strengthen the paper.

Regarding the remaining concerns, we would like to emphasize that our primary contribution is not merely combining existing SAE techniques, but introducing modality-disentangled SAE features as a direct visual grounding signal for preference labeling, without relying on LLM-as-Judge supervision. Our new controlled preference-labeling experiments further demonstrate the effectiveness of this approach.

Although modality-specific feature partitioning may appear straightforward, it is a principled design motivated by the linguistic biases we identify in LLM-as-Judge preference labeling. MA-SAE not only outperforms existing multimodal SAE methods across all five feature-quality metrics but also establishes a novel connection between modality-disentangled representations and preference alignment.

We sincerely appreciate your feedback and hope you will reconsider your assessment in light of these clarifications and the additional experimental evidence.

Reviewer_fG6zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Modality-Aware Sparse Autoencoders (MA-SAE) to mitigate visual hallucinations in multimodal large language models. Motivated by the linguistic-style biases of LLM-based preference judgments, the method separates vision-specific, text-specific, and cross-modal features through modality-aware sparse autoencoding. A cross-modal alignment objective is introduced, and the overlap of shared features between images and responses is used to construct grounding-based preference labels.

Experiments across three MLLM backbones demonstrate substantial hallucination reductions under DPO, with additional results on mDPO and on-policy preference tuning.

Critical strengths
  1. Meaningful empirical improvements across multiple models and training settings.

MA-SAE achieves substantial reductions in hallucination metrics across three MLLM backbones. The improvements extend beyond standard DPO to mDPO and on-policy preference tuning. Importantly, the authors use the same candidate-response pools and training configurations for the main preference-labeling comparisons, helping isolate the effects of different preference signals.

  1. A well-motivated representation design with supporting analyses.

The separation of modality-specific and shared features provides an intuitive approach to constructing visual grounding signals. The paper includes feature-quality comparisons, matched-budget evaluations, component ablations, and qualitative analyses. Appendix J also investigates response length and grounded-object preservation, addressing an important alternative explanation for hallucination reduction.

Critical weaknesses
  1. Substantial coverage degradation and an insufficiently explained hallucination–informativeness trade-off.

Although MA-SAE significantly reduces hallucination, Table 3 shows substantial reductions in AMBER Coverage for both LLaVA backbones. For example, Coverage decreases from 54.3 to 48.0 on LLaVA-7B and from 54.8 to 49.6 on LLaVA-13B compared with LLM-as-Judge. These reductions raise concerns about whether the method improves visual grounding while preserving useful visual information.

Appendix J partially addresses this concern by showing that shorter responses retain similar average numbers of grounded objects. However, this analysis does not directly explain the substantial reduction in AMBER Coverage, and the relationship between the evaluation samples and aggregation procedures is insufficiently specified. The on-policy results show more favorable coverage behavior, but the contrasting outcomes across training settings remain unexplained.

A comparison at matched coverage levels would help determine whether the hallucination improvements persist independently of reduced visual content coverage.

  1. Insufficient direct validation of the proposed grounding-based preference signal.

The proposed grounding score measures the overlap of shared cross-modal features between images and responses, rather than directly verifying the factual correctness of generated statements. Although the feature-quality evaluations, object-level grounding analysis, and downstream results provide supporting evidence, they do not fully establish whether the proposed score produces more reliable preference judgments than existing visual alignment signals.

In particular, the paper lacks a sufficiently controlled evaluation of preference-labeling accuracy that separates factual correctness from response length and linguistic style. Such an evaluation would help clarify whether the improvements arise from more accurate visual grounding judgments or from a preference for more conservative responses.

Furthermore, the LLM-as-Judge baseline relies on GPT-4V ratings from VLFeedback, but the manuscript does not clearly specify which rating dimensions are used. The authors should clarify whether the judge baseline already uses only the Visual Faithfulness scores. If not, such a comparison would help isolate the added value of the proposed preference signal.

These concerns limit the evidence for the added value of the proposed grounding-based preference labeling mechanism.

Questions for authors
  1. Table 3 reports substantial AMBER Coverage reductions for both LLaVA backbones, whereas Appendix J reports nearly unchanged average numbers of grounded objects. Are these results computed on identical evaluation samples using the same object-matching criteria? Could the authors clarify their relationship and explain the observed coverage degradation?

  2. Does MA-SAE maintain its hallucination reduction advantage when evaluated at matched AMBER Coverage levels? Additionally, what explains the different coverage behavior between off-policy and on-policy preference tuning?

  3. Which VLFeedback rating dimensions are used to construct the LLM-as-Judge baseline? Does it already use only the Visual Faithfulness scores? If not, how would such a baseline compare with MA-SAE?

  4. Can the authors directly evaluate the preference-labeling accuracy of the proposed grounding score using controlled response pairs that vary factual correctness while keeping linguistic style and response length comparable?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank Reviewer fG6z for the insightful comments. We have revised the manuscript to address the reviewers' comments, with changes highlighted in blue. We address your specific concerns as follows.


Q1. Why do Table 3 and Appendix J report different coverage behavior? Are the evaluation samples and object-matching criteria consistent?

A1. We thank the reviewer for this careful observation. Both analyses use the same 1,004 AMBER images and the same object-matching criteria. They differ in the preference-tuning setting. Appendix J analyzes the on-policy setting (corresponding to Table 5), whereas Table 3 reports the off-policy setting. We focus on the on-policy setting in Appendix J because its candidate responses exhibit more comparable lengths, reducing the confounding effect of response-length differences within preference pairs during DPO training. We have revised the corresponding appendix section to clarify this distinction. We further clarify the differences in candidate response lengths and explain their impact on the observed coverage degradation in the off-policy setting in our response to the following question (Q2/W1).


Q2/W1. Does MA-SAE maintain its hallucination reduction advantage when evaluated at matched AMBER Coverage levels? Additionally, what explains the different coverage behavior between off-policy and on-policy preference tuning?

A2. We first investigate the difference in coverage between off-policy and on-policy preference tuning and find that it is largely explained by differences in their candidate response pools rather than the grounding score itself.

In the off-policy setting, candidates are generated by 12 different MLLMs with substantial variation in verbosity and descriptive breadth (52% median relative length gap). Responses containing fewer unsupported objects often come from less verbose models that produce shorter responses and mention fewer objects overall. Consequently, DPO trained on these preference pairs also learns to generate shorter responses that mention fewer objects, leading to reduced object coverage.

In contrast, on-policy candidates are sampled from the same model, resulting in more comparable response lengths (14% median relative length gap). This reduces the confounding effect of response-length differences within preference pairs during DPO training and allows preference tuning to focus more directly on visual grounding. We summarize these preference-pair statistics for LLaVA-1.5-7B below.

MA-SAE preference pairsOff-policyOn-policy
Median relative length gap within a pair52%14%
Preferred response is shorter76%59%

These statistics show that on-policy preference tuning provides a more controlled setting, where preferred and rejected responses have more comparable lengths, mitigating the confounding effect of response-length differences during DPO training. In this setting, MA-SAE achieves both higher coverage and lower hallucination rates than LLM-as-Judge, demonstrating that its hallucination reduction is not solely attributable to reduced coverage.

To further verify that MA-SAE's hallucination reductions are not merely a by-product of reduced coverage or response length, we conduct two additional experiments suggested by the reviewer.

  • Matched-coverage comparison: For the off-policy LLaVA models, we compare MA-SAE and LLM-as-Judge on AMBER images for which both models identify exactly the same set of ground-truth objects (442 images for 7B and 479 for 13B). At matched coverage, MA-SAE consistently achieves lower CHAIR, HalRate, and Cog, as shown below. The reductions are statistically significant (p<0.001p < 0.001) and comparable to those observed on the full benchmark. These results demonstrate that MA-SAE's hallucination improvements cannot be explained solely by reduced coverage.
ModelMethodCHAIR ↓Cover ↑HalRate ↓Cog ↓
LLaVA-1.5-7BLLM-as-Judge5.654.731.72.4
MA-SAE3.854.715.81.5
LLaVA-1.5-13BLLM-as-Judge6.156.132.42.5
MA-SAE3.956.117.51.4
  • Length-controlled comparison: We further conduct a length-controlled preference-labeling experiment using response pairs that differ in factual correctness while preserving linguistic style and response length. MA-SAE consistently outperforms LLM-as-Judge. We provide the experimental details and results in our response to Q4/W2.

We have added a new Appendix J containing these analyses and summarized their key findings in Section 4.4 of the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q3. Which VLFeedback rating dimensions are used for LLM-as-Judge preference labeling, and how does a Visual-Faithfulness-only baseline compare with MA-SAE?

A3. Our LLM-as-Judge baseline follows the default VLFeedback protocol, which evaluates responses along three dimensions: Helpfulness, Visual Faithfulness, and Ethical Considerations. Preference labels are determined by averaging the three scores. As requested, we additionally construct a Visual-Faithfulness-only baseline using the same candidate pairs.

We find that the two labeling strategies are nearly equivalent, agreeing on 98.6% of preference pairs. We further investigate the reasons for this high agreement. First, Ethical Considerations provides limited discriminative information, as 88% of responses receive the maximum score. Second, Helpfulness and Visual Faithfulness are strongly correlated in their pairwise score differences (r = 0.80). Consequently, averaging all three dimensions yields nearly identical preference rankings to using Visual Faithfulness alone.

As requested, we further train DPO and mDPO using preference pairs labeled by LLM-as-judge based solely on Visual Faithfulness across all three backbones. As shown below, performance remains nearly unchanged compared with the original LLM-as-judge baseline using all three rating dimensions.

DPO Results

BackboneLabelingAMBER CHAIRCoverHalRateCogOH CHAIR_sOH CHAIR_iMSCOCO CHAIR_sMSCOCO CHAIR_i
LLaVA-1.5-7BH+VF+E5.854.332.72.443.6710.7252.0015.09
VF only6.154.232.52.346.3311.5653.0015.54
LLaVA-1.5-13BH+VF+E6.154.833.62.741.3310.6849.6015.16
VF only5.655.131.02.343.3311.3049.6015.58
InternVL3-8BH+VF+E7.971.263.25.730.337.6338.0011.66
VF only7.770.562.45.633.337.8040.8012.48

mDPO Results

BackboneLabelingAMBER CHAIRCoverHalRateCogOH CHAIR_sOH CHAIR_iMSCOCO CHAIR_sMSCOCO CHAIR_i
LLaVA-1.5-7BH+VF+E4.252.423.72.037.3310.8145.2014.26
VF only4.152.323.11.738.3310.6044.0013.92
LLaVA-1.5-13BH+VF+E3.752.120.21.535.339.4942.2013.58
VF only3.551.919.21.534.009.6142.4013.79
InternVL3-8BH+VF+E7.972.063.46.333.007.8140.0011.99
VF only7.672.061.06.131.007.4738.4011.19

Q4/W2. Can the authors directly evaluate the preference-labeling accuracy of the proposed grounding score using controlled response pairs that vary factual correctness while keeping linguistic style and response length comparable?

A4. We thank the reviewer for this thoughtful suggestion and have conducted the requested experiment.

Experimental setup. We start with over 2,000 hallucination-free captions from MSCOCO and Object HalBench images, verified against ground-truth object annotations. For each caption, we construct a hallucinated counterpart by replacing all mentions of one grounded object with an object absent from the image, using two strategies.

  • Random swap: The replacement object is uniformly sampled from objects absent from the image.

  • Plausible swap: The replacement object is selected from absent objects that frequently co-occur with the scene's objects in MSCOCO (e.g., fork → knife in a dining scene), creating more realistic and challenging hallucinations.

This controlled setup ensures that each response pair differs only by the substitution of a grounded object with a hallucinated one, thereby preserving linguistic style and response length. We evaluate pairwise preference-labeling accuracy, measured as the proportion of pairs in which the hallucination-free response is preferred. MA-SAE consistently outperforms the LLM-as-Judge baseline across both settings and maintains high accuracy even on plausible swaps, where hallucinated objects are plausible given the scene context.

MethodRandom swap Accuracy (%)Plausible swap Accuracy (%)
Length50.050.0
LLM-as-Judge81.679.3
MA-SAE (LLaVA-7B)94.388.4
MA-SAE (LLaVA-13B)96.690.5
MA-SAE (InternVL3-8B)95.688.7

We have included this controlled preference-labeling experiment in the new Appendix J.2 and summarized its key findings in Section 4.4 of the revised manuscript.


We hope these clarifications are helpful. We would be happy to address any further questions and would appreciate the reviewer’s consideration of the new analyses and results above.

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We would like to kindly follow up on our response to your comments and hope that our additional experiments and clarifications have addressed your concerns.

We would be grateful to hear whether our response has addressed your concerns, so that we have an opportunity to clarify any remaining questions before the discussion period ends.

Thank you again for your valuable feedback and thoughtful review!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and Area Chair

We sincerely thank all three reviewers for their thoughtful and constructive feedback. We have carefully addressed their concerns through additional analyses, experiments, and manuscript revisions, with changes highlighted in blue. Below, we summarize the major updates.

  • Novelty and methodological contribution (dygV, VxuD). We emphasize that MA-SAE goes beyond combining existing SAE techniques. Motivated by the linguistic biases we identify in LLM-as-Judge supervision, MA-SAE disentangles modality-specific patterns from cross-modal grounding features, outperforming existing multimodal SAEs across all five feature-quality metrics. More importantly, our work establishes a novel connection between modality-disentangled SAE representations and preference alignment, demonstrating that interpretable cross-modal features can serve as a direct visual grounding signal for preference labeling and substantially reduce hallucinations without relying on LLM-as-Judge supervision.

  • Controlled evaluation of visual grounding (fG6z, dygV). We conducted length-controlled preference-labeling experiments on over 2,000 caption pairs, where MA-SAE achieves 88.4–96.6% accuracy, outperforming LLM-as-Judge (79.3–81.6%). Matched-coverage evaluations under off-policy DPO further confirm significantly lower hallucination rates (p < 0.001), demonstrating that the improvements are not solely due to shorter responses or reduced object coverage. These additional analyses are included in the new Appendix J, with key findings summarized in Section 4.4 of the revised manuscript.

  • Additional baselines and robustness analyses (fG6z, VxuD). We evaluated a Visual-Faithfulness-only LLM-as-Judge baseline under DPO and mDPO, finding nearly identical performance to the original baseline. We also verified MA-SAE's robustness to linguistic hedging and quantified the negligible frequency of preference-label ties.

  • Alignment objective ablations (dygV). We compared the Barlow Twins loss used in MA-SAE with the MSE and KL alignment losses suggested by the reviewer. Through ablation experiments, we found that MSE and KL lead to degenerate representations, learning inactive or constant cross-modal features rather than meaningful cross-modal representations. We have clarified these findings and the motivation for using Barlow Twins in Section 3.2 of the revised manuscript.

  • Improved presentation (dygV, VxuD). We moved the key findings on linguistic bias from the appendix to the main text (Section 4.2) and expanded the discussion of limitations and future directions (Section 5).

We thank the reviewers again for their valuable suggestions and welcome any further questions or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.