acceptodds
Withdrawn from ICLR 2027

-VLM: Detecting and Correcting Vision-Language Hallucination via Active Representational Perturbation and Residual Spectral Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language Models (VLMs) can confidently generate text that contradicts the visual input, a failure commonly referred to as . Existing mitigation methods predominantly analyze unperturbed forward passes, limiting their ability to assess the stability of internal representations under perturbations. We propose -VLM, a training-free, single-pass framework for hallucination detection and correction based on active representational perturbation. Our detection module applies structured latent perturbations to the hidden states of a frozen VLM across visual, linguistic, and cross-modal subspaces. These perturbations yield a calibrated hallucination risk score and reveal that joint cross-modal reliability provides a stronger indicator of hallucination than unimodal signals alone. To correct detected failures, we introduce , which restores visual evidence by additively injecting a per-sample spectral decomposition of the visual representation into the decoder hidden states. We further establish theoretical guarantees showing that the risk score is bounded within , the adaptive threshold converges, and RSG strictly improves representational energy stability. Empirically, across Qwen2.5-VL (3B/7B) and LLaVA-1.5 (7B) on ColorBench, POPE, and CableDetection, our risk score exhibits a strong correlation with ground-truth correctness, achieving a point-biserial correlation of up to and Hedges' effect size of up to (). Furthermore, RSG nearly doubles cross-modal representational alignment from to and achieves relative average accuracy improvements of on color-attribute tasks and on CableDetection, while avoiding the performance degradation observed with competing subtractive correction baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models (VLMs), Multimodal Large Language Models (MLLMs), perception hallucination, hallucination detection, hallucination mitigation, hallucination correction, R²-VLM, Residual Spectral Grounding (RSG), active representational perturbation, representational resilience, hidden-state analysis, latent-space perturbation, structured latent perturbation, visual subspace, linguistic subspace, cross-modal subspace, cross-modal reliability, visual grounding, cross-modal grounding, representational fragility, structural stability, hallucination risk score, adaptive risk scoring, adaptive flagging, dynamic thresholds, sliding-window calibration, LOW/MEDIUM/HIGH risk tiers, single-pass inference, training-free inference, test-time mitigation, decoder hidden states, visual encoder representations, language decoder representations, spectral decomposition, spectral grounding, high-frequency visual information, Fourier transform, discrete Fourier transform (DFT), frequency-domain analysis, content centroid, complexity signal, spectral grounding token, residual injection, additive correction, residual correction, skip connections, representation alignment, Linear CKA, cross-modal alignment, Frobenius energy ratio, directional centroid alignment, KL divergence, output divergence, energy stability, subspace erasure, SVD, singular vectors, rank collapse, token permutation, token ablation, semantic drift, anisotropic spectral scaling, adversarial robustness, joint manifold projection, energy equalisation, perturbation severity, fine-grained attribute grounding, visual-language grounding, uncertainty, confidence margin, calibration, statistical significance, point-biserial correlation, Spearman correlation, Hedges’ g, ColorBench, POPE, CableDetection, Qwen2.5-VL, Qwen-VL, LLaVA-1.5, inference efficiency, quantized inference, safety-critical VLMs, medical imaging, industrial inspection, autonomous driving, hallucination robustness, representation engineering

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1rRHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes R²-VLM, a training-free VLM framework for hallucination detection and correction. The detection component employs 13 perturbation operators to actively perturb visual, linguistic, and joint cross-modal representations at four levels of severity, and then constructs a hallucination risk score through energy preservation, directional alignment, and output-distribution shift. For samples classified as medium-to-high risk, RSG is further employed to inject visual statistics and spectral information into the decoder’s hidden states in the form of residuals. The authors demonstrate the algorithm’s effectiveness through a series of experiments.

Critical strengths
  1. The training-free hallucination mitigation mechanism does not require retraining the model, making it cost-effective.

  2. Unlike approaches that train probes solely from unperturbed hidden states, this paper actively applies structured interventions to the representations and then observes their stability. In particular, it examines visual, language, and joint cross-modal subspaces separately, a design that offers some diagnostic value.

Critical weaknesses
  1. Some theories and motivations require further clarification.

  2. This appears to be a rather unpolished manuscript. The mechanical writing style and Figure 1 significantly detract from the manuscript’s quality.

Questions for authors
  1. In Section 5, Theorem 5.2, the authors claim that as long as ⟨ϕ^v,Πk⊥ϕ^v⟩>0\langle \hat\phi_v,\Pi_k^\perp\hat\phi_v \rangle > 0, spectral injection guarantees improved energy stability. However, the proof of Theorem 5.2 in Appendix A actually expands to: $ ||\Pi_k^\perp H_v||_F^2 + 2\delta \langle \Pi_k^\perp H_v, \Pi_k^\perp\hat\phi_v \rangle + \delta^2 ||\Pi_k^\perp\hat\phi_v||_F^2$. Subsequently, because the last term is positive, the author concludes that for any δ>0\delta > 0, the energy will strictly increase. If the cross term $ \langle \Pi_k^\perp H_v, \Pi_k^\perp\hat\phi_v \rangle < 0$, how does this conclusion hold? Furthermore, Theorem 5.2 discusses the energy ratio γE\gamma_E, but the proof analyzes only the perturbed numerator energy and does not explicitly address the change in the denominator resulting from the addition of δϕ^v\delta\hat\phi_v. Therefore, I hope the author will provide a complete derivation of the ratio in the rebuttal, rather than merely demonstrating an increase in the numerator.

  2. In the Motivation section of Sec. 5, the paper explicitly describes the theoretical motivation for the correction as high-frequency visual signal attenuation. However, in Definition 5.1, ϕv\phi_v is actually selected from the top-K frequency modes with the largest magnitudes, rather than high-frequency modes. More importantly, in Sec. 7 / Figure 6, the authors themselves found that the low-order frequency window performed best on approximately half of the tasks. Therefore, I would like the authors to clarify: Is RSG actually restoring high-frequency information, or is it enhancing dominant spectral components? Furthermore, ϕv\phi_v uses ∣H^v(f)∣|\hat H_v(f)|, which is a magnitude-only spectrum; the phase is discarded. If the phase and spatial arrangement are not preserved, how exactly does the paper’s claim of restoring fine-grained spatial visual evidence hold true?

  3. I would like the authors to explain, with reference to Sec. 3, Assumption 3.1, why representational fragility is a sufficient and necessary condition for hallucination. Intuitively, a sample with a correct answer may still have a fragile representation, while a hallucinated answer may appear very stable due to a strong language prior. Figure 2 itself already shows significant overlap between the correct and hallucinated distributions in several settings, and Appendix H more directly illustrates situations where the risk score is virtually ineffective.

Reviewer_kmbdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes R²-VLM, a training-free framework for detecting and correcting hallucinations in vision-language models. The detection module applies structured perturbations to visual, linguistic, and cross-modal hidden representations to estimate hallucination risk. For high-risk predictions, the Residual Spectral Grounding (RSG) module injects visual statistical and spectral information into decoder hidden states. The authors evaluate the framework on ColorBench, POPE, and CableDetection, reporting improvements in hallucination detection and visual grounding.

Critical strengths

None.

Critical weaknesses
  • Serious numerical inconsistencies in the reported experimental results. Figure 3 reports mean CKA values of 0.3468 and 0.6595, whereas averaging the 16 displayed heatmap values yields approximately 0.5450 and 0.5633, respectively. This directly contradicts the claimed near-doubling of cross-modal alignment unless an alternative aggregation procedure was used but not disclosed. Such a substantial discrepancy in a central experimental result raises serious concerns about the reliability of the reported findings.

  • Contradictory definitions of the RSG injection mechanism. Equation (2) defines the injection starting layer as ℓ=⌊ρL⌋\ell^=\lfloor\rho L\rfloor, whereas Appendix I defines it as ℓ=max⁡(1,L−⌊ρL⌋)\ell^=\max(1,L-\lfloor\rho L\rfloor). These definitions are incompatible and lead to substantially different intervention ranges. More critically, under the reported CableDetection setting of ρ=1\rho=1, Equation (2) produces a zero denominator L−ℓ∗L-\ell^*, making the update undefined. This fundamental inconsistency raises serious concerns about the correctness and reproducibility of the proposed method.

Questions for authors
  • Inconsistencies in CKA Results. How were the mean CKA values in Figure 3 calculated? Why do they differ substantially from the averages of the displayed heatmap values? Could the authors provide the original CKA matrices and the exact aggregation procedure?

  • Contradictory RSG Implementation. Which definition of the injection starting layer was actually used in the experiments, Equation (2) or Appendix I? How was the division-by-zero issue avoided when ρ=1\rho=1? Please clarify the implemented update rule and reconcile the conflicting definitions.

Reviewer_e44POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes R²-VLM, a training-free, single-pass detect-and-correct framework for vision-language hallucination that evaluates structural stability under active representational perturbation. Its detection module applies structured latent perturbations across visual, linguistic, and cross-modal subspaces to produce a calibrated hallucination risk score, while its Residual Spectral Grounding (RSG) module restores lost visual evidence by additively injecting a per-sample spectral decomposition of the visual representation into the decoder’s hidden states. The authors report that the risk score strongly correlates with ground-truth correctness and that RSG achieves average accuracy improvements of 2.4% on colour-attribute tasks and 1.84% on CableDetection.

Critical strengths
  1. The paper investigates a useful diagnostic question and reports heterogeneous outcomes. Joint versus unimodal perturbation sensitivity could provide actionable information about VLM failure modes. Table 2 reports meaningful risk–error associations, including point-biserial correlation 0.536 on Qwen-7B/ColorBench, while Appendix H explicitly documents a poorly separable Qwen-3B industrial setting. Reporting that failure case is useful and prevents the empirical story from appearing uniformly positive.

  2. The additive correction has positive empirical results. Table 3 reports average ColorBench accuracy increasing from 0.4921 to 0.5159 for Qwen-3B, with particularly larger gains on recognition and extraction. This motivates further investigation of sample-dependent additive signals, although it does not establish the proposed spectral explanation or validate the end-to-end detector.

Critical weaknesses
  1. The detector’s vision-to-answer pathway is unclear. Under the final-layer procedure described in Sections 3–4, perturbing only visual-token rows leaves the last textual state and its LM-head logits unchanged, implying zero decision KL. Energy and centroid shifts alone therefore do not establish answer sensitivity to visual perturbations. Appendix C lists 52 LM-head projections without describing downstream attention recomputation. Any earlier-layer hooks or additional computation should be specified and included in the cost. Moreover, the full risk includes a substantial confidence-margin term without a margin-only detection comparison, leaving the added value of active probing unresolved.

  2. The central stability and calibration guarantees are not established as written. Theorem 5.2 claims an improvement in an energy ratio from a nonzero residual component of the injected feature. Appendix A, p. 12, instead argues from an expansion of post-erasure energy: it does not control the potentially negative cross term, and does not account for the changed denominator of the ratio. A positive quadratic term alone cannot justify the claimed inequality. The proof also acts on visual states, whereas the implemented RSG correction acts on language-decoder states, requiring an additional connection.

Questions for authors
  1. At exactly what layer and point in the forward computation is each perturbation applied, and what is recomputed before measuring decision KL? For a vision-only perturbation of final-layer visual rows, what changes the last-token logits? Please provide the precise forward graph and separate baseline, detection-only, and full-pipeline latency, including the conditional second backbone pass in Algorithm 1.

  2. Can you provide a corrected statement and proof of Theorem 5.2 that handles the cross term, ratio denominator, projection convention, and dimensions, and explains its relevance to language-state injection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.