- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors aim to reduce hallucinations in multimodal reasoning models by addressing cases where attention heads remain focused on a dominant internal pattern instead of adapting to new information. They analyze changes in attention head outputs over time and propose a training-free method that detects persistent concentration and dynamically adjusts head contributions during generation. Experiments on five models and eight benchmarks show that the method reduces hallucinations and improves visual understanding and reasoning performance.
Critical strengths
- The paper is clearly written, with a well-motivated problem and a clear explanation of the proposed method.
- The paper proposes a training-free method that dynamically adjusts attention head contributions while preserving efficient inference, supported by extensive experiments across multiple models and benchmarks.
Critical weaknesses
- Motivation for spectral analysis. The paper identifies interesting spectral patterns in attention-head contributions. However, it remains unclear why spectral analysis is particularly suitable for characterizing hallucination-related dynamics, and how it differs from simpler head-level statistics or coordination measures.
- Insufficient evidence for the proposed mechanism. The authors associate persistent spectral concentration with hallucinations, but the experiments mainly establish correlations rather than causality. Moreover, changes in head outputs do not necessarily reflect new visual evidence, making it unclear whether the proposed signal truly captures failures to incorporate visual information.
Questions for authors
Regarding Weakness 1: Could the authors further clarify the novelty of this work compared with existing studies on attention dynamics, spectral analysis, and dynamic head adjustment (e.g., RAL, FLASH, and AHEAD)? In particular, what new insights or methodological advances does SPLIT offer beyond these approaches?
Regarding Weaknesses 2: Can the authors provide stronger evidence that persistent spectral concentration directly contributes to hallucinations rather than merely correlating with them? In particular, can the improvements be distinguished from those achieved by simpler head reweighting strategies?
4 comments
Thank you for prompting a closer look at the spectral intervention
Thank you again for asking how the effect of spectral carrier assignment can be separated from that of simpler reweighting. Tables 12–13 report the edit-norm-matched shuffle comparison and the paired target-removal test. If you have any remaining questions about these controls or their interpretation, we would be happy to discuss them.
The authors have substantially strengthened the paper through additional analyses and controlled interventions. The revised experiments provide convincing evidence for the existence of population-level spectral coordination, its distinction from fixed-head effects and unequal head energy, and the effectiveness of the proposed carrier assignment strategy. I therefore consider most of my original soundness concerns adequately addressed.
Nevertheless, my remaining concern is primarily about the degree of conceptual novelty. While failed spectral release provides an interesting characterization of head-population dynamics associated with hallucination, the proposed method ultimately implements a spectral-guided dynamic head reweighting strategy. The spectral analysis offers a more structured criterion for identifying and adjusting head contributions, but its role as a fundamentally new mechanistic insight remains less compelling. In particular, the empirical identification of a new failure signature and the effectiveness of an intervention based on that signature do not necessarily establish a substantially new understanding of multimodal hallucination.
Overall, I appreciate the authors' substantial efforts during the rebuttal and acknowledge that the paper is technically stronger. However, I remain somewhat unconvinced that the conceptual and methodological advances are sufficiently substantial. I have increased my score from 2 to 3 to acknowledge the substantial improvements made during the rebuttal, although my assessment remains closer to the midpoint (2.5), given the remaining concerns about the paper's novelty.
Thank you for the follow-up on SPLIT’s conceptual contribution
Thank you for revisiting the paper and for clarifying your remaining concern about novelty. RAL rebalances heads according to their perception or reasoning roles, FLASH uses a spectral score to identify vision heads for attention modulation, and AHEAD adjusts head contributions based on visual grounding and language prior potentials. SPLIT studies how a shared direction in the residual stream evolves across heads during decoding.
Figure 2 shows that this direction can persist even as the heads carrying it change. Figure 4 shows that when internal innovation rises, matched factual events tend to disperse spectral concentration more strongly than hallucinated events. Together, these observations characterize hallucination-related drift in terms of how the head population responds to internal change. In SPLIT, this temporal pattern determines when to intervene, and the current mode determines which heads to reweight. The edit-norm-matched shuffle in Table 12 tests the latter choice and shows that using the current carrier assignment improves final answers at a comparable edit size.
We would welcome any further thoughts on whether this distinction addresses your remaining concern about the paper’s conceptual contribution.
Response to Reviewer 6ic9 on Spectral Analysis and Intervention Controls
Thank you for asking what the spectrum contributes and whether simpler head reweighting can explain the gains. We added a concrete example and tested the gain assignment with the starting prefix held fixed.
Q1. What the spectrum adds. The leading share measures how much normalized innovation energy one residual direction explains. For orthonormal e₁ and e₂, the row sets (e₁, e₁, −e₁, −e₁) and (e₁, −e₁, e₂, −e₂) have identical row energies and mean signed pairwise cosine. Their leading shares are 1 and 1/2, respectively. The trigger combines retained concentration with high innovation, and the leading eigenvector identifies which heads carry that mode (Sections 3.2–3.3).
RAL rescales contributions of perception- and reasoning-oriented heads across layers. FLASH selects vision heads using a Spectral Vortex Score and applies spectral modulation. AHEAD adjusts head contributions using visual-grounding and language-prior potentials. SPLIT derives both the trigger and the carrier assignment from the evolving joint geometry of head innovations in shared residual coordinates. The gain acts after native SDPA under a reference-energy constraint. Section 2 includes the comparison.
Q2. Does the carrier assignment affect the answer? At an eligible gate, all branches start from the same prefix and KV state. The edited branches apply one lagged edit and continue to the final answer. Questions without an eligible gate use the no-edit continuation in all branches. The two shuffle controls are calibrated to reference gain RMS or the realized pre-W_O edit norm. Revised Table 12 reports correct final answers out of 120 balanced POPEv2 questions per model.
SPLIT improves accuracy by 12.50 and 9.17 points over the shuffle calibrated to the current edit norm. In full generation, it also improves both CHAIR hallucination rates and both POPEv2 metrics over token-varying Shuffled, Random, and Energy assignments on both backbones. We added the full-generation results to the revised PDF as Table 14.
Does the answer respond to the image? We use the same 120 original/target-removed POPEv2 image pairs for both backbones. Within a pair we hold the question fixed and replay the same generated prefix where available, with separate image-conditioned KV states. A correct flip means Yes on the original and No after removal. We added the paired results to the revised PDF as Table 13.
In the subsets with realized edit norms matched in both views, SPLIT yields 7 and 8 more correct flips than the corresponding shuffle. The fixed-prefix comparison tests how gain assignment affects final answers at matched edit size. Changing the image tests whether those answers respond to the target's presence. Section 3.3 calls the head-output signal internal innovation, and Figure 4 reports its observational relationship with spectral concentration and hallucination.