- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a Modality-Aware Sparse Autoencoders for reducing hallucinations in MLLM. The method mainly divided the SAE features into vision-specific, text-specific and cross-modal banks. Then, a Barlow Twin is used to align corresponding vision-text features. Experimental results on several benchmarks verified the effectiveness of the proposed methods over different preference-labeling baselines.
Critical strengths
-
The paper addresses an important problem in MLMMs and propsoed a princiled approach to reducing hallucinations.
-
The method is evaluated across several multiple PO settings, and the experimental improvements suggest that modality-aware features can provide useful grounding signals for reducing hallucination.
-
The paper is well-orgnized and easy to follow.
Critical weaknesses
-
The novelty of the proposed methods is incrementatl. Modality-specific feature partition and cross-modal alignment largely combine existing SAE techniques
-
The paper's presentation could be significantly improved. The core motivation and key insights related to linguistics bias should be introduced in the main text rather than in the appendix, as they are essential for understanding the core contribution of the proposed method.
-
In the proposed methods, the Barlow Twin is used to perform the cross-modal alignment. While increasing diagonal correlations encourages alignment between identically indexed visual and textual features, driving off-diagonal correlations toward zero implicitly assumes that features with different indices should be uncorrelated across modalities. However, different features may capture semantically related or complementary concepts, and their cross-modal correlations are not necessarily undesirable. The paper should provide a clearer justification for this constraint.
-
Following the previous concern, it's not clear of why not using MSE or KL loss to perform the alignment?
-
MA-SAE produces substantially shorter responses, which may affect hallucination metrics. The authors should conduct length-controlled comparisons to verify that the reported improvements reflect better visual grounding rather than reduced response verbosity.
Questions for authors
Please refer to the weakness.
4 comments
Official Comment by Authors (2/3)
W4. Following the previous concern, it's not clear of why not using MSE or KL loss to perform the alignment?
A4. Both MSE and KL alignment losses admit trivial solutions in which the cross-modal bank becomes inactive or learns constant features, rather than meaningful cross-modal representations.
-
MSE alignment loss: MSE can trivially reach its minimum of zero when the cross-modal features for both image and text are zero. The cross-modal bank can therefore become inactive while the modality-specific banks handle reconstruction. Empirically, when training MA-SAE with MSE alignment, we observe that the cross-modal bank nearly collapses, averaging only 0.04 active features per token.
-
KL alignment loss: KL normalizes the features into a probability distribution, avoiding the all-zero solution. However, it still admits a constant solution in which the same features activate across all image and text tokens, regardless of the input. Empirically, KL alignment leads to this collapse, with the cross-modal bank activating a nearly fixed set of features across all samples and only 57 of 8,192 cross-modal features activating on both modalities.
In contrast, inactive or constant features have no variation across samples and therefore cannot satisfy the diagonal correlation objective of Barlow Twins, resulting in a high alignment loss. We have revised Section 3.2 to discuss these degenerate solutions of MSE and KL alignment and to clarify why we adopt the Barlow Twins objective.
As requested, we further compare the feature quality of these three alignment objectives on LLaVA-1.5-7B. As shown below, Barlow Twins consistently outperforms MSE and KL across all metrics, demonstrating its effectiveness in learning meaningful cross-modal representations.
| Alignment Loss | MMS v-t | MMS v-v | MMS t-t |
|---|---|---|---|
| MSE | 0.0901 | 0.2834 | 0.6363 |
| KL | 0.0058 | 0.0344 | 0.0991 |
| Barlow Twins (MA-SAE) | 0.2386 | 0.6509 | 0.6960 |
Official Comment by Authors (3/3)
W5. The authors should conduct length-controlled comparisons to verify that the reported improvements reflect better visual grounding rather than reduced response verbosity.
A5. We thank the reviewer for this important suggestion. We first highlight that on-policy tuning provides a more controlled setting, where preferred and rejected responses have more comparable lengths, mitigating response-length confounding during DPO training. In this setting, MA-SAE achieves both higher object coverage and lower hallucination rates than LLM-as-Judge, demonstrating that its hallucination reductions are not solely attributable to reduced coverage. To further investigate this concern, we conduct two additional experiments:
1. Matched-coverage comparison. For the off-policy LLaVA models, we compare MA-SAE and LLM-as-Judge on AMBER images for which both models identify exactly the same set of ground-truth objects (442 images for 7B and 479 for 13B). At matched coverage, MA-SAE consistently achieves lower CHAIR, HalRate, and Cog, as shown below. The reductions are statistically significant () and comparable to those observed on the full benchmark. These results demonstrate that MA-SAE's hallucination improvements cannot be explained solely by reduced coverage.
| Model | Method | CHAIR ↓ | Cover ↑ | HalRate ↓ | Cog ↓ |
|---|---|---|---|---|---|
| LLaVA-1.5-7B | LLM-as-Judge | 5.6 | 54.7 | 31.7 | 2.4 |
| MA-SAE | 3.8 | 54.7 | 15.8 | 1.5 | |
| LLaVA-1.5-13B | LLM-as-Judge | 6.1 | 56.1 | 32.4 | 2.5 |
| MA-SAE | 3.9 | 56.1 | 17.5 | 1.4 |
2. Length-controlled preference-labeling experiment. To directly evaluate whether MA-SAE's grounding score captures factual correctness independently of response length, we conduct a controlled experiment using over 2,000 hallucination-free captions from MSCOCO and Object HalBench, verified against ground-truth object annotations. For each caption, we construct a hallucinated counterpart by replacing all mentions of one grounded object with an object absent from the image, using two strategies:
-
Random swap: The replacement object is uniformly sampled from objects absent from the image.
-
Plausible swap: The replacement object is selected from absent objects that frequently co-occur with the scene's objects in MSCOCO (e.g., fork → knife in a dining scene), creating more realistic and challenging hallucinations.
This controlled setup ensures that each response pair differs only by the substitution of a grounded object with a hallucinated one, thereby preserving linguistic style and response length. We evaluate pairwise preference-labeling accuracy, measured as the proportion of pairs in which the hallucination-free response is preferred. MA-SAE consistently outperforms the LLM-as-Judge baseline across both settings and maintains high accuracy even on plausible swaps, where hallucinated objects are plausible given the scene context.
| Method | Random swap Accuracy (%) | Plausible swap Accuracy (%) |
|---|---|---|
| Length | 50.0 | 50.0 |
| LLM-as-Judge | 81.6 | 79.3 |
| MA-SAE (LLaVA-7B) | 94.3 | 88.4 |
| MA-SAE (LLaVA-13B) | 96.6 | 90.5 |
| MA-SAE (InternVL3-8B) | 95.6 | 88.7 |
Together, these analyses provide complementary evidence that MA-SAE's hallucination reductions reflect better visual grounding rather than merely reduced response length or object coverage.
We have added a new Appendix J to analyze coverage differences between off-policy and on-policy tuning and to present these additional experiments. We have also summarized the findings in Section 4.4 of the revised manuscript.
We hope these clarifications are helpful. We would be happy to address any further questions and would appreciate the reviewer’s consideration of the new analyses and results above.
Dear Reviewer,
We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We would like to kindly follow up on our response to your comments and hope that our additional experiments and clarifications have addressed your concerns.
We would be grateful to hear whether our response has addressed your concerns, so that we have an opportunity to clarify any remaining questions before the discussion period ends.
Thank you again for your valuable feedback and thoughtful review!
Official Comment by Authors (1/3)
We thank Reviewer dygV for the thoughtful comments. We have revised the manuscript to address the reviewers' comments, with changes highlighted in blue. We address your specific concerns as follows.
W1. The novelty of the proposed methods is incremental. Modality-specific feature partition and cross-modal alignment largely combine existing SAE techniques.
A1. We respectfully note that our contribution extends beyond the SAE architecture itself. To our knowledge, MA-SAE is the first work to use modality-disentangled SAE features as a preference-labeling signal for multimodal preference pairs, without relying on conventional preference-labeling methods such as LLM-as-judge, and to demonstrate that such supervision effectively reduces hallucinations. Prior multimodal SAE methods primarily focus on interpretability (e.g., VL-SAE and GSAE), while SAE-V uses vanilla SAE features to filter existing preference data for alignment, rather than directly assigning preference labels based on visual grounding.
Although modality-specific feature partitioning may appear straightforward, it is a principled design motivated by the linguistic bias we identify in LLM judges for multimodal preference labeling. In terms of interpretability, MA-SAE outperforms existing SAEs designed primarily for interpretability across all five feature-quality metrics. When applied to preference labeling, we further demonstrate that disentangling modality-specific context from cross-modal grounding features helps mitigate the linguistic bias observed in LLM-as-judge supervision. This enables MA-SAE to generate preference labels that lead to substantially greater reductions in hallucinations than LLM-as-judge or alternative labeling baselines. These findings establish a novel connection between modality-disentangled SAE representations and preference alignment, demonstrating how modality-aware feature separation can serve as a more effective grounding signal for multimodal preference supervision.
W2. The core motivation and key insights related to linguistics bias should be introduced in the main text rather than in the appendix.
A2. We thank the reviewer for this valuable suggestion. We have revised the manuscript to incorporate the key findings on linguistic bias in LLM judges into the main text (Section 4.2).
W3. Does the Barlow Twins off-diagonal regularization suppress meaningful correlations between semantically related or complementary cross-modal features? The paper should provide a clearer justification for this regularization.
A3. We thank the reviewer for this insightful observation. We clarify that the off-diagonal term in Barlow Twins serves as a soft regularizer rather than a hard constraint. In our implementation, it is assigned a small weight of λoff=0.005 and averaged over all pcross(pcross−1) off-diagonal feature pairs. Thus, it mildly discourages redundant features without explicitly forcing semantically related features to be uncorrelated.
To further investigate this concern, we conduct an ablation varying λoff over four orders of magnitude. We find that a small, nonzero regularization weight encourages feature diversity (i.e., lower inter-feature similarity) while preserving intra-feature similarity. Moreover, feature pairs that frequently co-occur (e.g., co-activate on the same inputs within each modality) retain a mean off-diagonal cross-modal correlation of 0.60, compared with approximately zero for non-co-occurring pairs. This indicates that the off-diagonal regularization does not suppress meaningful semantic co-occurrence.
Consistent with the reviewer's concern, stronger off-diagonal regularization degrades feature quality. Increasing λoff by four orders of magnitude reduces MMS v–t from 0.24 to 0.06. These findings motivate our choice of a small, nonzero λoff, which balances feature diversity and semantic coherence. We have revised Section 3.2 to clarify this design choice.