- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes R²-VLM, a training-free VLM framework for hallucination detection and correction. The detection component employs 13 perturbation operators to actively perturb visual, linguistic, and joint cross-modal representations at four levels of severity, and then constructs a hallucination risk score through energy preservation, directional alignment, and output-distribution shift. For samples classified as medium-to-high risk, RSG is further employed to inject visual statistics and spectral information into the decoder’s hidden states in the form of residuals. The authors demonstrate the algorithm’s effectiveness through a series of experiments.
Critical strengths
-
The training-free hallucination mitigation mechanism does not require retraining the model, making it cost-effective.
-
Unlike approaches that train probes solely from unperturbed hidden states, this paper actively applies structured interventions to the representations and then observes their stability. In particular, it examines visual, language, and joint cross-modal subspaces separately, a design that offers some diagnostic value.
Critical weaknesses
-
Some theories and motivations require further clarification.
-
This appears to be a rather unpolished manuscript. The mechanical writing style and Figure 1 significantly detract from the manuscript’s quality.
Questions for authors
-
In Section 5, Theorem 5.2, the authors claim that as long as , spectral injection guarantees improved energy stability. However, the proof of Theorem 5.2 in Appendix A actually expands to: $ ||\Pi_k^\perp H_v||_F^2 + 2\delta \langle \Pi_k^\perp H_v, \Pi_k^\perp\hat\phi_v \rangle + \delta^2 ||\Pi_k^\perp\hat\phi_v||_F^2$. Subsequently, because the last term is positive, the author concludes that for any , the energy will strictly increase. If the cross term $ \langle \Pi_k^\perp H_v, \Pi_k^\perp\hat\phi_v \rangle < 0$, how does this conclusion hold? Furthermore, Theorem 5.2 discusses the energy ratio , but the proof analyzes only the perturbed numerator energy and does not explicitly address the change in the denominator resulting from the addition of . Therefore, I hope the author will provide a complete derivation of the ratio in the rebuttal, rather than merely demonstrating an increase in the numerator.
-
In the Motivation section of Sec. 5, the paper explicitly describes the theoretical motivation for the correction as high-frequency visual signal attenuation. However, in Definition 5.1, is actually selected from the top-K frequency modes with the largest magnitudes, rather than high-frequency modes. More importantly, in Sec. 7 / Figure 6, the authors themselves found that the low-order frequency window performed best on approximately half of the tasks. Therefore, I would like the authors to clarify: Is RSG actually restoring high-frequency information, or is it enhancing dominant spectral components? Furthermore, uses , which is a magnitude-only spectrum; the phase is discarded. If the phase and spatial arrangement are not preserved, how exactly does the paper’s claim of restoring fine-grained spatial visual evidence hold true?
-
I would like the authors to explain, with reference to Sec. 3, Assumption 3.1, why representational fragility is a sufficient and necessary condition for hallucination. Intuitively, a sample with a correct answer may still have a fragile representation, while a hallucinated answer may appear very stable due to a strong language prior. Figure 2 itself already shows significant overlap between the correct and hallucinated distributions in several settings, and Appendix H more directly illustrates situations where the risk score is virtually ineffective.