acceptodds
Desk rejected at ICLR 2027

Schrödinger's Prior: Quantum-inspired Representation Intervention for Hallucination Mitigation in LVLMs

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Vision-Language Models (LVLMs) achieve strong multimodal performance but remain vulnerable to hallucinations, where language priors can override visually grounded evidence. Existing mitigation methods therefore tend to suppress prior-related information, implicitly treating priors as intrinsically harmful. We challenge this assumption and reveal a context-conditioned prior duality: the same prior can provide useful knowledge or induce hallucination depending on the current image, prompt, and decoding state. We further show that, when beneficial and harmful prior effects coexist, no fixed context-independent attenuation can improve both. Based on this finding, we propose SchröHallu, a training-free, quantum-inspired inference framework that models unresolved prior utility as a superposition of hallucination-inducing and knowledge-beneficial states. SchröHallu strengthens visual evidence, decomposes hidden representations into evidence, prior, and residual components, and measures whether contextual evidence sufficiently resolves the prior toward a harmful state. Only then is a minimum-norm correction applied along the decision-relevant direction of the prior subspace, preserving grounded evidence and residual capabilities. Extensive experiments across three representative LVLMs consistently reduce hallucinations while largely preserving perceptual and reasoning performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Schrödinger's Prior, Hallucinations, Prior Intervention, Large Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bqR8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SchroHallu, a training-free method for hallucination mitigation in LVLMs. It first strengthens visual evidence and decomposes hidden representations into visual-evidence, prior, and residual components. The method then determines whether the prior is beneficial or hallucination-inducing under the current multimodal context, and applies a minimum-norm correction only when needed along the decision-relevant prior direction. Experiments across multiple models and datasets show consistent hallucination reduction while largely preserving perceptual and reasoning abilities. Ablations, case studies, efficiency analysis, and oracle-based evaluations further support the method.

Critical strengths
  1. The context-dependent treatment of model priors is a meaningful strength. Instead of assuming that language priors are uniformly harmful, SchroHallu evaluates their role under the current multimodal context and selectively intervenes only when the prior is likely to induce hallucination.

  2. The empirical validation is comprehensive and well structured. The method is evaluated across multiple LVLMs and benchmarks, with additional ablations, case studies, efficiency analysis, and oracle-based evaluations that consistently support both its effectiveness and the proposed context-conditioned intervention strategy.

Critical weaknesses
  1. The methodological novelty over HulluEdit is not sufficiently clear. SchroHallu’s evidence/prior/residual decomposition and online subspace construction substantially build on HulluEdit. The main additions are context-conditioned prior-utility estimation and decision-axis minimum-norm correction, and the paper should more clearly establish whether these extensions constitute a sufficiently distinct methodological contribution.

  2. The technical necessity of the quantum-inspired formulation is limited. The authors explicitly state in Appendix E: Discussion that removing the quantum terminology while keeping the same harmful/beneficial roles, deterministic measurement, and intervention rule leaves the algorithm, outputs, and performance unchanged. This makes the quantum-inspired formulation appear largely conceptual, while its actual technical value remains unclear.

  3. The evaluation of generation quality is insufficient. Aggressive hallucination suppression may lead to more conservative, repetitive, or templated outputs, but the paper mainly reports hallucination-oriented metrics such as CHAIR. Additional generation-quality metrics, such as BLEU-5, recall, and repetition/diversity measures, would help verify that hallucination reduction does not come at the cost of text quality.

Questions for authors
  1. The methodological novelty of SchroHallu over HulluEdit requires further clarification. The two methods share a similar evidence/prior/residual decomposition and online subspace construction framework, so the independent value of context-conditioned prior-utility estimation and decision-axis correction should be more clearly established.

  2. It remains necessary to verify whether hallucination reduction comes at the cost of text generation quality. Additional metrics such as BLEU-5, recall, and repetition/diversity measures would help determine whether the method produces more conservative, repetitive, or templated outputs.

  3. The statistical reliability of the reported gains is not yet sufficiently clear. Several improvements over strong baselines are relatively small, so confidence intervals or significance tests would help establish whether these gains are robust.

  4. The MME evaluation is incomplete. The paper reports only selected subsets, including Existence, Count, Position, and Color. Reporting the overall MME score would provide a more complete assessment of the method’s effect on multimodal capabilities.

6 comments

Response to Reviewer bqR8 (Part 1/2): Novelty and Generation Quality

Thank you for the constructive review. We clarify our novelty and quantum-inspired motivation, then present additional evidence on generation quality, statistical reliability, and full MME evaluation. The revised manuscript has been uploaded to the submission system, with all changes highlighted in green.

[W1 & Q1] Methodological Novelty Beyond HulluEdit

SchröHallu decides whether and where to intervene, rather than only tuning suppression strength. We build on HulluEdit's evidence/prior/residual decomposition, but differ from its VCR/PCR-guided shrinkage through contextual utility assessment: qtH=DtVtCtPStq_t^H=D_tV_tC_t^PS_t combines prior–visual disagreement, visibility, decision relevance, and visual counter-evidence. An edit is considered only if qtHBt>mtTq_t^HB_t>m_t^T and is restricted to the prior-projected decision axis dtPd_t^P, leaving visual and residual components unchanged. Removing H/K polarization or the prior-axis certificate raises CHAIR CSC_S from 21.8% to 32.6% or 38.2%, respectively. The novelty is the utility-aware decision principle and localized correction, not the borrowed decomposition or minimum-norm solver.

The distinction from HulluEdit's prior shrinkage is made explicit in Introduction (page 2), Section 2.2 (page 3), and Section 4.2 (page 6). The utility-aware contextual measurement and sufficient decision-axis correction are specified in Sections 4.3–4.4 (pages 7–8), and the contribution of the H/K and prior-axis components is quantified in Table 5 (page 30).

[W2] Why the Quantum-Inspired Perspective Matters

The quantum-inspired view motivates two linked advances: context-conditioned prior utility and intervention only after sufficient harmful evidence. Treating H/K utility as unresolved until interaction with the image, prompt, and decoding state leads to deterministic contextual measurement and sufficiency-gated, decision-local correction. We do not invoke quantum computation or physical quantum states. Relabeling ∣H⟩|H\rangle and ∣K⟩|K\rangle would leave the equations unchanged, but not erase their conceptual motivation: this perspective explains why identifying a prior alone does not justify suppressing it. Its value is in guiding the new decision principle, not adding quantum-specific operations.

The conceptual role of the Schrödinger-inspired formulation is clarified in Section 3.3 (page 5). Its computation-preserving relabeling interpretation is explained in Appendix E (page 30), while the operational contextual measurement and associated module ablation appear in Section 4.3 (page 6-7) and Table 4 (page 28), with component terminology clarified in Appendix G (page 32).

[W3 & Q2] Hallucination Reduction Without Degrading Generation Quality

The expanded caption-quality evaluation indicates that hallucination reduction does not rely on conservative or repetitive output. On LLaVA-1.5-7B, CHAIR CSC_S/CIC_I decreases from 48.0/14.3 to 21.8/6.5, while BLEU-5 and Recall rise from 2.1 to 3.4 and 74.5 to 76.6. Across all three LVLMs (Table R1), SchröHallu achieves highest Recall and Distinct-2, lowest Repeat-2, and highest or tied-highest BLEU-5 among the compared methods. These results support improved grounding while retaining descriptive content and lexical diversity.

Table R1: CHAIR and generation-quality comparisons (%).

ModelMethodCSC_S↓CIC_I↓BLEU-5↑Recall↑Repeat-2↓Distinct-2↑
LLaVA-1.5-7B
Vanilla48.014.32.174.511.123.3
OPERA44.612.82.275.610.324.1
VTI41.411.43.167.54.227.2
Nullu50.213.81.974.08.130.2
SVAR25.06.72.969.530.918.6
HulluEdit51.412.92.272.511.324.4
SHIELD36.610.31.874.18.430.1
LVLMs-Saliency35.68.21.571.142.125.5
SchröHallu21.86.53.476.63.835.4
LLaVA-1.5-13B
Vanilla48.813.12.173.211.123.5
OPERA42.613.22.368.811.726.4
VTI32.89.23.463.75.627.1
Nullu49.213.32.073.110.724.6
SVAR28.88.82.368.621.924.1
HulluEdit50.613.42.372.811.323.8
SHIELD50.015.31.269.27.135.7
LVLMs-Saliency39.88.81.669.733.428.5
SchröHallu27.87.93.574.03.038.5
DeepSeek-VL-Chat-7B
Vanilla25.07.11.163.210.633.4
OPERA22.66.41.163.49.534.3
VTI24.08.41.065.335.021.2
Nullu25.26.91.164.610.932.0
SVAR21.46.72.157.624.831.2
HulluEdit30.88.71.061.628.128.2
SHIELD29.88.50.962.08.532.4
LVLMs-Saliency25.87.31.163.310.333.5
SchröHallu20.66.22.166.28.136.5

The caption-quality findings are summarized in Section 5.2 (page 9-10). Full BLEU-5, Recall, Repeat-2, and Distinct-2 comparisons across the three LVLMs are presented in Appendix F.4 (Table 15, page 33).

Response to Reviewer bqR8 (Part 2/2): Statistical Reliability and MME

[Q3] Statistical Reliability of the Improvements

The supplied uncertainty estimates indicate broad nominal statistical support, with explicit exceptions. Of 24 POPE/CHAIR comparisons, 20 have unadjusted p<0.05p<0.05, including all 18 POPE entries. On LLaVA-1.5-7B, Random F1 improves by 1.1 pp (95% CI [0.37, 1.83], p=0.003p=0.003) and CHAIR CSC_S by 3.2 pp (95% CI [0.81, 5.59], p=0.009p=0.009).

Table R2: Differences against the strongest baseline (%; positive gains favor SchröHallu).

Model / MetricBest baselineOursGain (pp)95% CI of gainpp
LLaVA-1.5-7B
POPE-Ran. Acc. ↑SHIELD (90.7)91.2+0.5[0.12, 0.88]0.010
POPE-Ran. F1 ↑HulluEdit (90.5)91.6+1.1[0.37, 1.83]0.003
POPE-Pop. Acc. ↑HulluEdit (87.5)87.8+0.3[0.05, 0.55]0.019
POPE-Pop. F1 ↑HulluEdit (87.6)88.2+0.6[0.17, 1.03]0.006
POPE-Adv. Acc. ↑LVLMs-Saliency (83.3)83.9+0.6[0.13, 1.07]0.012
POPE-Adv. F1 ↑HulluEdit (83.4)83.8+0.4[0.06, 0.74]0.021
CHAIR CSC_S ↓SVAR (25.0)21.8+3.2[0.81, 5.59]0.009
CHAIR CIC_I ↓SVAR (6.7)6.5+0.2[-0.58, 0.98]0.615
LLaVA-1.5-13B
POPE-Ran. Acc. ↑HulluEdit (90.6)91.4+0.8[0.26, 1.34]0.004
POPE-Ran. F1 ↑HulluEdit (90.8)91.2+0.4[0.08, 0.72]0.014
POPE-Pop. Acc. ↑HulluEdit (88.0)88.7+0.7[0.23, 1.17]0.004
POPE-Pop. F1 ↑HulluEdit (88.3)88.9+0.6[0.16, 1.04]0.008
POPE-Adv. Acc. ↑LVLMs-Saliency (83.4)85.8+2.4[1.55, 3.25]<0.001
POPE-Adv. F1 ↑LVLMs-Saliency (84.1)85.0+0.9[0.25, 1.55]0.007
CHAIR CSC_S ↓SVAR (28.8)27.8+1.0[-0.05, 2.05]0.063
CHAIR CIC_I ↓SVAR (8.8)7.9+0.9[0.16, 1.64]0.017
DeepSeek-VL-Chat-7B
POPE-Ran. Acc. ↑VTI (90.4)90.8+0.4[0.07, 0.73]0.017
POPE-Ran. F1 ↑VTI (90.3)90.6+0.3[0.04, 0.56]0.025
POPE-Pop. Acc. ↑Nullu (88.6)89.3+0.7[0.23, 1.17]0.004
POPE-Pop. F1 ↑Nullu (88.4)88.7+0.3[0.04, 0.56]0.024
POPE-Adv. Acc. ↑Nullu (85.5)86.0+0.5[0.13, 0.87]0.008
POPE-Adv. F1 ↑Nullu (85.6)86.2+0.6[0.15, 1.05]0.009
CHAIR CSC_S ↓SVAR (21.4)20.6+0.8[-0.22, 1.82]0.125
CHAIR CIC_I ↓OPERA (6.4)6.2+0.2[-0.43, 0.83]0.530

The statistical-reliability findings are summarized in Section 5.2 (page 9-10), and all 24 metric-wise confidence intervals and significance tests are reported in Appendix F.5 (Table 16, page 34).

Reviewer_HbyYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SchröHallu, a training-free, quantum-inspired framework for mitigating hallucinations in large vision-language models. Its central premise is that model priors can be beneficial or harmful depending on the current visual-linguistic context, motivating selective rather than uniform suppression. The method strengthens visual attention, decomposes hidden representations into visual-evidence, prior, and residual components, and measures contextual prior harmfulness using prior–visual disagreement, decision relevance, and visual support. It then applies a gated minimum-norm correction within the prior subspace, subject to a full-vocabulary consistency check. Evaluation covers LLaVA-1.5-7B/13B and DeepSeek-VL-Chat-7B on POPE, CHAIR, AMBER, and MME, supplemented by component ablations, prior-utility diagnostics, parameter analyses, and efficiency measurements. The reported results show reduced hallucinations and improvements on selected visual-perception tasks, with additional inference overhead.

Critical strengths
  • Meaningful research objective. The paper addresses the tension between reducing hallucinations and retaining useful prior knowledge. Distinguishing the presence of prior information from its harmfulness in a particular context motivates selective intervention rather than indiscriminate suppression. This is a practically relevant objective for improving visual faithfulness without unnecessarily degrading model capabilities.

  • Promising reported results. Table 1 reports improvements on both discriminative and generative hallucination benchmarks across two model families. For LLaVA-1.5-7B, CHAIR CS/CIC_S/C_I decreases from 48.0/14.348.0/14.3 to 21.8/6.521.8/6.5, alongside improvements on all three POPE splits. These substantial reported gains constitute the strongest empirical argument in favor of the paper.

Critical weaknesses
  • Inconsistent editing mechanism. Eq. (12) defines ytV=arg⁡max⁡yWy⊤h~ty_t^V=\arg\max_y W_y^\top\tilde h_t. Appendix B.3 accepts an edit only when the resulting top-1 prediction remains ytVy_t^V; otherwise, it restores h~t\tilde h_t. Thus, both branches output exactly the same token as Visual-only under Algorithm 1. This cannot explain the reported improvement over Visual-only in Table 4 (CS:30.2→21.8C_S:30.2\rightarrow21.8). The author needs to explain the relationship between the described algorithm and its actual implementation, the reasons for any discrepancies, and the results of the ablation experiments.

  • Inconsistent gating mechanism. Eq. (22) requires a strictly positive intervention threshold mtT>0m_t^T>0, but Eq. (106) defines mT,t=[wt⊤h~t]+m_{T,t}=\left[w_t^\top\tilde{h}t\right]+, where wt=WytP−WytVw_t=W_{y_t^P}-W_{y_t^V}. Since ytV=arg⁡max⁡yWy⊤h~ty_t^V=\arg\max_y W_y^\top\tilde{h}t, we necessarily have wt⊤h~t≤0w_t^\top\tilde{h}t\leq 0, implying mT,t=0m_{T,t}=0. Furthermore, qtH>0q_t^H>0 implies Bt>0B_t>0 under the paper's definitions, so the sufficiency condition qtHBt>mT,tq_t^H B_t>m_{T,t} becomes equivalent to qtH>0q_t^H>0, imposing no additional filtering. This directly contradicts Table 9, where the proportion of surviving steps decreases from 14.7% to 7.0% after the sufficiency gate for LLaVA-1.5-7B.

  • Limited methodological novelty and heuristic design. Prior work, including LanP, already recognizes that language priors can be beneficial. Visual-attention reinforcement and evidence-preserving decomposition follow Jiang et al. and Lin et al.. As clearly stated in Appendix E, quantum analogies can aid in explanation, but they do not in and of themselves constitute an additional technical contribution. The technical contribution therefore rests primarily on contextual gating and decision-direction editing.

  • Capability trade-offs and high overhead. The main MME comparison covers only four perceptual subtasks, providing limited evidence for preserving general reasoning capabilities. AMBER Cover decreases from 51.8 to 47.2, while Figure 5 shows that increasing the visual reinforcement coefficient from α=0.5\alpha=0.5 to α=0.6\alpha=0.6 improves CHAIR CSC_S but drops F1 from 75.7 to 59.1, revealing a parameter-sensitive quality trade-off. Throughput also falls from 50.0 to 27.2 tokens/s, a 45.6% reduction, imposing a substantial practical cost.

  • Generalization to modern backbones. Experiments cover only two older model families: LLaVA-1.5 (7B/13B) and DeepSeek-VL-Chat (7B). Evaluation on more recent backbones, such as Qwen3-VL, is needed to establish the method's applicability beyond these architectures.

Questions for authors
  1. Editing and Gating mechanism. How can prior editing change the generated output under greedy decoding when both acceptance and rejection preserve the same top-1 token ytVy_t^V? What exact intervention threshold is used in the implementation?

  2. Technical contribution. What justifies the multiplicative score qtH=DtVtCtPStq_t^H=D_t V_t C_t^P S_t over simpler margin- or confidence-based gates? What is the difference between this and related work?

Other questions pls refer to weakness. The author's response in the rebuttal will affect my final score.

5 comments

Response to Reviewer HbyY (Part 1/2): Algorithm and Novelty

Thank you for the detailed assessment. We clarify the decoder and present new evaluations. The implemented edit is not equivalent to Visual-only decoding, and its sufficiency gate is non-degenerate. The revised manuscript has been uploaded to the submission system, with all changes highlighted in red.

[W1 & Q1] Why Editing Can Change the Output

The equivalence argument overlooks the actual visual-candidate selection. Let ztN,ztAz_t^N,z_t^A denote native and visually reinforced logits, and ytN=arg⁡max⁡ztNy_t^N=\arg\max z_t^N. If reinforcement changes the top-1 token, Full retains that visual decision. Otherwise, it selects the highest-ranked eligible visual competitor ytV≠ytNy_t^V\ne y_t^N, not necessarily Visual-only's top-1, and tests a minimum-norm prior edit. The active logits become ztout=ztT+Wδht∗z_t^{\mathrm{out}}=z_t^T+W\delta h_t^*. An edit is accepted only if the corrected full-vocabulary top-1 is ytVy_t^V; otherwise ztTz_t^T is retained. Acceptance and rejection can therefore produce different tokens. This explains the CHAIR gain over Visual-only (CSC_S: 30.2% → 21.8%, F1: 75.6% → 75.7%). The LM-head edit affects later steps through the generated prefix.

The revised candidate-selection and target-path rules are described in Section 4.3 (page 6), and the corrected-logit update that can change the emitted token is specified in Section 4.4 (page 8). The full-vocabulary guard and common-prefix/cache semantics are detailed in Appendix B.3 (page 25) and Algorithm 1 (page 26); the Visual-only versus Full ablation is shown in Table 4 (page 28).

[W2] Why the Sufficiency Gate Is Not Degenerate

The qtH>0q_t^H>0 reduction follows from Eq. (106) as currently written, not from the executable gate. Its target margin is

mtT=[ztT(ytN)−ztT(ytV)]+,ztT={ztA,open-ended, ztN,closed Yes/No.m_t^T=[z_t^T(y_t^N)-z_t^T(y_t^V)]_+,\quad z_t^T=\begin{cases}z_t^A,&\text{open-ended},\\\ z_t^N,&\text{closed Yes/No}.\end{cases}

For an actionable conflict, mtT>0m_t^T>0. Editing requires Bt>mtTB_t>m_t^T and qtHBt>mtTq_t^HB_t>m_t^T, followed by full-vocabulary verification; thus positive harmfulness alone is insufficient. The margin is computed online. We have corrected Eq. (106) and specify the target logits, candidate construction, and gating order.

The online target margin and non-degenerate two-stage sufficiency inequalities are specified in Section 4.4 (Eqs. 21–22, page 8) and implemented in Algorithm 1 (page 26). Their numerical settings and the online margin definition are documented in Appendix I (Eqs. 102–103, page 36), with the stepwise decision-funnel analysis in Appendix E (page 29-30) and Table 9 (page 30).

[W3] Context-Conditioned Prior Duality and Its Decision Principle

Our contribution has two connected parts. (1) Insight: LanP examines beneficial language priors, especially under insufficient visual evidence. We ask a different question: should a prior be preserved or corrected for the current decoding decision? Nullu, HulluEdit, and SHIELD mitigate prior-/bias-related effects without explicitly resolving their utility at each active decision. On long-form CHAIR, Nullu/HulluEdit score CS=50.2%/51.4%C_S=50.2\%/51.4\%, worse than Vanilla's 48.0%. Fig. 1 shows opposite POPE effects from the same prior removal and MME losses under excessive suppression. Prior utility is context-conditioned, not an intrinsic label of a representation.

(2) Decision principle: Jiang et al. motivate visual reinforcement, and Lin et al. evidence-preserving decomposition/editing. Beyond these foundations, SchröHallu separates prior localization, contextual harmfulness measurement, and intervention sufficiency: identifying a prior need not imply harm, nor does detected harm automatically justify editing. Only well-supported conflicts receive a local minimum-norm correction. Removing evidence consensus, prior-axis certification, or contextual polarization raises CHAIR CSC_S from 21.8% to 33.2%, 38.2%, or 32.6%. Proposition 3.2.2 formalizes why fixed attenuation fails across opposing prior utilities. The novelty is the insight-to-decision link, not the quantum analogy or an isolated component.

The insight-to-decision contribution and its relation to LanP, Nullu, and HulluEdit are clarified in and Section 2.2 (page 3), while the fixed-attenuation result is positioned in Section 3.2 (page 4). The U–visual consensus and multiplicative reliability design are detailed in Section 4.3 (page 6-7), with measurement ablations in Table 5 (page 28) and matched-edit comparisons in Appendix F.3 (Table 14, page 33).

Response to Reviewer HbyY (Part 2/2): Additional Evaluations

[W4] Capability Preservation and Efficiency

(1) Full MME. We expand the four perceptual tasks to all 10 perception and four cognition subtasks (Tables R1–R2). SchröHallu leads in perception (1477.3 vs. 1254.3 Vanilla / 1379.42 HulluEdit) and cognition (385.7 vs. 288.7 Vanilla / 351.3 Nullu), improving all four cognition tasks over Vanilla. This supports MME-measured capabilities.

Table R1: MME Perception — total and ten subtasks (LLaVA-1.5-7B).

MethodTotalExistenceCountPositionColorPostersCelebritySceneLandmarkArtworkOCR
Vanilla1254.3175.6124.6114.0151.0117.2111.7144.8130.7108.975.8
VIT1264.4190.0138.3131.7145.0101.067.4146.0124.5110.5110.0
Nullu1330.9190.0121.1105.6156.7127.6115.6147.9131.7113.0121.7
DeCo1341.5175.0128.398.3125.0125.3152.3152.5154.5110.3120.0
HulluEdit1379.4195.0105.0126.7160.0123.5126.2150.0153.0120.0120.0
SchröHallu1477.3195.0158.3135.0165.0129.6132.6154.8161.0121.0125.0

Table R2: MME Cognition — total and four subtasks.

MethodCognition TotalCommon SenseCalculationTranslationCode
Vanilla288.7111.250.050.077.5
VTI292.1112.140.085.055.0
Nullu351.3112.165.081.792.5
DeCo298.9116.457.562.562.5
HulluEdit286.8114.350.065.057.5
SchröHallu385.7120.765.097.50102.50

(2) AMBER Cover. Despite a Cover decrease (51.8 → 47.2 vs. Vanilla), SchröHallu achieves CHAIR 3.4 and Hal 17.0, with higher Cover than Nullu (42.2) and HulluEdit (42.5). (3) Sensitivity. At the selected α=0.5\alpha=0.5, CHAIR CSC_S improves from 48.0% to 21.8% with F1 maintained (75.7% vs. 75.6% Vanilla); α=0.6\alpha=0.6 further lowers CSC_S but drops F1 to 59.1%, so it is not adopted. (4) Overhead. Throughput falls by 45.6% versus Vanilla (50.0 → 27.2 tokens/s), but our leading hallucination performance comes with better speed/memory than SHIELD (27.2/15.5 vs. 23.3/20.2 tokens/s/GB) and LVLMs-Saliency (27.2/15.5 vs. 1.9/32.4), offering a competitive practical trade-off.

The expanded MME results and AMBER coverage trade-off are summarized in Section 5.2 (page 9-10). All ten Perception and four Cognition MME subtasks are provided in Appendix F.1 (Tables 10–11, pages 31). The speed and memory comparisons are reported in Section 5.4 (Table 3, page 10), and the visual-reinforcement sensitivity analysis in Appendix H (Figure 5, page 34).

[W5] Generalization to Modern LVLMs

We additionally evaluate Qwen3-VL-8B-Instruct, extending the evidence to four models across three families. SchröHallu leads on all six POPE metrics; on CHAIR, CS/CIC_S/C_I fall to 34.2/5.9 from 52.0/9.5 (Vanilla). On AMBER it achieves CHAIR/Hal/Cog 4.5/53.0/2.7, retaining Cover (74.4 vs. 74.2); MME perception/cognition reach 1752.7/657.9, with the highest total 2410.6 (vs. 2380.6 Vanilla / 2402.9 DeCo). This verifies effectiveness beyond LLaVA and DeepSeek.

Table R3: Qwen3-VL POPE (Accuracy/F1, %) and CHAIR (%).

MethodRandomPopularAdversarialCHAIR CS/CIC_S/C_I ↓
Vanilla92.5/92.089.1/88.887.0/87.052.0/9.5
OPERA92.6/92.289.3/89.186.8/86.948.2/9.6
Nullu91.3/91.186.8/87.184.2/84.937.0/13.1
SVAR92.8/92.589.5/89.486.9/87.150.4/10.3
HulluEdit92.0/91.588.5/88.286.7/86.646.4/6.8
SHIELD90.6/90.086.9/86.685.1/84.953.2/10.5
LVLMs-Saliency90.7/90.887.4/88.686.6/86.145.3/8.7
SchröHallu93.1/92.989.9/89.787.4/87.334.2/5.9

Table R4: Qwen3-VL AMBER (%) and MME.

MethodAMBER CHAIR ↓Cover ↑Hal ↓Cog ↓MME Percep. ↑Cogn. ↑Total ↑
Vanilla7.974.261.23.61742.7637.92380.6
Nullu12.258.563.73.71728.5631.82360.3
DeCo7.674.158.33.81747.9655.02402.9
SHIELD7.673.167.34.41647.8630.72278.5
HulluEdit4.668.255.23.11740.5627.92368.4
SchröHallu4.574.453.02.71752.7657.92410.6

The expanded evaluation scope is stated in Section 5.1 (pages 8). The Qwen3-VL cross-family analysis is presented in Appendix F.2 (pages 31–32), with its POPE/CHAIR results in Table 12 (page 32) and AMBER/MME results in Table 13 (page 32).

[Q2] Why Multiplicative Measurement Rather Than Simple Gates?

(1) Justification and evidence. Margin/confidence measures prediction strength, not context-specific prior harmfulness. Our multiplicative score combines disagreement, candidate visibility, prior-axis relevance, and independent visual support as a conservative soft conjunction: insufficient evidence in one factor cannot be offset by another. At default thresholds, Margin-only/Confidence-only accept 304/318 edits (42.16%/43.98%), versus 59 (8.09%) for SchröHallu, yet yield worse CHAIR CSC_S (30.4/31.6 vs. 21.8). With 59 accepted edits in each method, SchröHallu still leads (CSC_S: 21.8 vs. 24.3/26.1; CIC_I: 6.5 vs. 6.7/6.9). Thus frequency alone cannot explain the gain.

Table R5: Comparison of simple gates and contextual measurement on CHAIR.

GateSettingAcceptedAccepted/EligibleCSC_S ↓CIC_I ↓
Margin-onlyDefault30442.16%30.46.9
Confidence-onlyDefault31843.98%31.67.4
SchröHalluDefault598.09%21.86.5
Margin-onlyMatched598.09%24.36.7
Confidence-onlyMatched598.09%26.16.9
SchröHalluMatched598.09%21.86.5

(2) Difference from existing gates. FuzzyCD [1] uses confidence for contrastive weights; DEGAP [2] and RUDDER [3] regulate visual perturbation/injection; Causal Route Gating [4] selects text-dominant routes for suppression. Our score measures evidence reliability for harmful priors in the current competing decision. It does not itself trigger editing: the supported correction must additionally exceed the target margin. The distinction is the measurement objective and separate sufficiency test—not multiplication alone.

The multiplicative reliability factors and U–visual directional agreement are explained in Section 4.3 (page 6-7), and the separate target-margin sufficiency criterion is given in Section 4.4 (Eq. 22, page 8). Comparisons against margin-only and confidence-only gates, including the 59-edit matched setting, are provided in Appendix F.3 (Table 14, page 33).

We hope these results and clarifications warrant a more favorable assessment.

References: [1] Kim et al., FuzzyCD, ICCV 2025. [2] Seo et al., DEGAP, ICML 2026. [3] Zou et al., Adaptive Residual-Update Steering, ICML 2026. [4] Cheng et al., Causal Route Gating, ICML 2026.

Reviewer_nViuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that language-derived priors in LVLMs can be beneficial or harmful depending on the current multimodal context. It formalizes this dependence through signed contributions to a competing-token margin and shows that fixed attenuation cannot improve every context when these contributions have opposite signs.

The proposed training-free method, SchröHallu, first reinforces visual attention, decomposes the representation into evidence, prior, and residual subspaces, and constructs a context-dependent harmfulness score. It then applies a minimum-norm update within the prior subspace, subject to a full-vocabulary consistency check. Experiments report hallucination reductions on POPE, CHAIR, and AMBER, together with selected MME perception results, across three LVLM checkpoints.

Critical strengths
  1. The motivation identifies a meaningful decision problem. Treating all prior-related information as something to remove can discard useful knowledge; separating where a component resides from whether it helps a particular decision is a sensible basis for designing conservative interventions. This is a useful modeling distinction even if context-dependent prior utility is not, by itself, a new phenomenon.
  2. The minimum-norm update is a concrete consequence of the goal of limiting collateral changes. Projecting the competing-token direction into the prior subspace gives a precise optimization problem, rather than leaving “selective intervention” as a purely verbal proposal. Its orthogonality guarantee is clear, although its semantic interpretation requires further justification.
  3. The paper attempts to examine the proposed mechanism as well as benchmark performance. Oracle-utility diagnostics, controlled context pairs, and risky-conflict analyses address whether the harmfulness score tracks the intended functional distinction. These analyses are relevant to the motivation and more informative than aggregate hallucination scores alone.
Critical weaknesses
  1. The method does not yet connect the proposed insight to a change in decoding behavior. Equation (12) defines y_V as the top-1 token of the visually strengthened state. Algorithm 1 accepts an edit only if its top-1 token remains y_V and otherwise retains the strengthened state; line 26 then uses greedy decoding. Both branches therefore emit the same token. Unless the edit changes future model computation through an additional state update, the generated sequence should match visual strengthening alone, in tension with Table 4. This concerns the operational validity of the central contribution, rather than merely an omitted experiment.
  2. The novelty of the motivating insight is overstated relative to what is established. Section 2 already acknowledges prior work on beneficial language priors and the damage from excessive suppression. Proposition 3.2.2 shows that attenuating a component helps or hurts according to its signed contribution to the correct-token margin; this follows directly from the linear decomposition and does not establish that existing context-dependent methods assume priors are intrinsically harmful. The quantum terminology adds no computation, as Appendix E acknowledges. The paper needs to identify the new decision principle beyond recognizing mixed prior utility and applying a confidence-gated edit.
  3. The operational harmfulness score is not fully justified by the conceptual formulation. The theory defines utility using an oracle-preferred alternative, whereas inference uses disagreement with the strengthened path, projected visual support, and a product of reliability factors. The same model supplies these signals, so agreement can reflect correlated error rather than independent evidence that a prior is harmful. Appendix E provides relevant diagnostics, but projection preservation alone does not establish semantic evidence preservation. The unresolved step is why these particular proxies reliably implement the proposed distinction when visual evidence is ambiguous or incorrect.
Questions for authors
  1. What operation lets the accepted Stage IV edit change output relative to Stage I alone? Please reconcile Equation (12), Algorithm 1 lines 20 and 26, and Table 4, specifying any write-back into future model states or caches.
  2. What is the new conceptual or decision-theoretic principle beyond the beneficial-prior observations already acknowledged in Section 2 and ordinary context-dependent selective editing? Which assumption or prediction distinguishes the proposed framework from those alternatives?
  3. Under what conditions should the proposed score distinguish a harmful prior from an incorrect visual preference? Please explain the rationale for its factors and the scope of the existing oracle-utility/context-pair diagnostics, especially when the strengthened visual path is unreliable.
2 comments

Response to Reviewer nViu (Part 1/2): Decoding Validity and Methodological Novelty

Thank you for your careful and constructive review. We clarify that Stage IV can change the emitted token in the implemented decoder, and that our contribution comprises context-conditioned prior duality and the resulting decision-specific intervention principle. The revised manuscript has been uploaded to the submission system, with all changes highlighted in blue.

[W1 & Q1] Stage IV Changes the Actual Decoding Decision

The claimed equivalence between Stage I and the full method holds for the simplified candidate definition, but not for our implemented algorithm. The code distinguishes the native winner ytNy_t^N from an eligible visual competitor ytCy_t^C. If visual strengthening has already changed the winner, Stage IV avoids a redundant edit; otherwise, it can select the visual-path runner-up, which need not be the strengthened path's pre-edit top-1.

For the logits used in token selection, the pre-edit margin is mtT=[zttarget(ytN)−zttarget(ytC)]+m_t^T=[z_t^{\mathrm{target}}(y_t^N)-z_t^{\mathrm{target}}(y_t^C)]_+. An edit is considered only if the context-supported budget satisfies Δt=qtHBt>mtT\Delta_t=q_t^H B_t>m_t^T. The minimum-norm prior-axis update then changes the pairwise margin to mtT−Δt<0m_t^T-\Delta_t<0. The full-vocabulary guard accepts the edit only if ytCy_t^C becomes the new top-1. The corrected logits are used immediately for token selection, and the selected token enters both cached decoding branches at the next step; no manual KV-cache write-back is necessary.

The empirical gain beyond Stage I is visible in Table 4: adding Stage IV reduces CHAIR CSC_S from 30.2% to 21.8% and CIC_I from 7.7% to 6.5%. We have aligned Eq. (12), the intervention-margin definition, and Algorithm 1 with the actual competitor-selection and logit-editing rules.

The corrected native-winner and eligible visual-competitor selection, including the task-specific target path, is specified in Section 4.3 (Eq. 12, page 6). The target-margin sufficiency test and output-changing logit correction are detailed in Section 4.4 (page 8), with the full-vocabulary guard and shared-prefix/cache behavior in Appendix B.3 (page 25) and Algorithm 1 (page 26). The improvement over visual strengthening alone is reported in Table 4 (page 28).

[W2 & Q2] Two Contributions: Context-Conditioned Prior Duality and a Decision Principle

Our novelty has two connected parts: an empirical insight and the decision rule it motivates. First, prior-oriented approaches such as Nullu, HulluEdit, and SHIELD identify and suppress or correct particular prior/bias signals, but do not explicitly resolve the utility of each identified contribution for the current decision. Such interventions can inadvertently remove useful information: in Table 1, under max_new_tokens=512, Nullu and HulluEdit worsen LLaVA-1.5-7B CHAIR CSC_S relative to Vanilla (50.2%/51.4% vs. 48.0%). More directly, Fig. 1 shows that the same whole-prior removal has opposite effects across POPE contexts, while strong suppression harms MME perception and cognition. Our insight is therefore not simply that beneficial priors exist, but that a prior mechanism's utility is determined by the particular image, prompt, and decoding decision.

Second, prior localization ≠\neq harmfulness ≠\neq intervention necessity. SchröHallu checks the prior's contribution to the competing decision, resolves contextual harmful evidence, and edits only when that evidence is sufficient to overcome the current decision margin. The edit is further restricted to the prior-projected decision direction by minimum-norm optimization. This jointly specifies whether to intervene and how much to change, rather than uniformly weakening an identified prior. Proposition 3.2.2 formalizes the limitation of fixed attenuation; the novelty lies in converting this insight into an operational, decision-local rule.

Table 1: CHAIR comparison (%)

MethodCSC_SCIC_I
Vanilla48.014.3
Nullu50.213.8
HulluEdit51.412.9
SchröHallu21.86.5

The distinction between prior localization, contextual harmfulness, and intervention necessity is clarified in Section 2.2 (page 3); the fixed-attenuation argument and the separation of decomposition from utility assessment appear in Section 3.2 (page 4) and Section 4.2 (page 6), respectively. The long-form CHAIR comparison and component ablations are reported in Table 1 (page 9) and Table 5 (page 28).

Response to Reviewer nViu (Part 2/2): Contextual Measurement and Reliability

[W3 & Q3] Rationale, Operating Conditions, and Diagnostic Scope

Oracle-signed utility defines the theoretical target; our inference-time score is a conservative harmful-intervention certificate, not an oracle label or calibrated probability. We address the three specific concerns as follows.

(1) Distinguishing a harmful prior from a mistaken visual preference. Intervention requires all three directional conditions: the prior-projected margin supports the native winner, visual strengthening increases the competitor's relative support, and the protected visual subspace UtU_t also favors that competitor. A visual preference unsupported by UtU_t is rejected. The remaining candidates must additionally satisfy qtHBt>mtTq_t^HB_t>m_t^T and the full-vocabulary guard. Disagreement or visual confidence alone cannot trigger an edit.

(2) Why these factors? Pairwise fidelity/disagreement measures the conflict; two-path visibility downweights weak or imbalanced preferences; prior-axis excess above its rank-dependent isotropic baseline tests decision relevance; U--visual consensus checks directional agreement; and attention-head support discounts unreliable enhancement. Their multiplicative combination reduces intervention support when any required cue is weak. These are complementary within-model checks, not statistically independent observations. In Table 2, removing U--visual consensus raises CHAIR CSC_S from 21.8% to 33.2%, and removing prior-axis certification raises it to 38.2%, supporting their practical necessity.

(3) What do existing diagnostics establish? The opposite oracle utilities in Fig. 3 (+0.282+0.282 and −0.341-0.341) and context-varying prior-removal effects in Fig. 1 support prior duality; Appendix E ablations show that the measurement constraints matter. When visual enhancement is mistaken but UtU_t disagrees, the method abstains; correlated errors in both paths can still lead to false edits. We will clarify this scope and evaluate oracle-utility agreement and accepted-edit correctness stratified by visual-candidate reliability. Geometric preservation is an exact subspace property, not a claim of guaranteed semantic correctness.

Table 2: Ablation study (%)

MethodCSC_SCIC_I
Vanilla48.014.3
w/o U+A evidence consensus33.27.9
w/o prior-axis concentration certificate38.210.7
SchröHallu21.86.5

The disagreement, rank-dependent prior-axis certificate, U–visual consensus, and normalized contextual polarization are specified in Section 4.3 (pages 6–7), followed by the operational sufficiency checks in Section 4.4 (page 8). Their component ablations are reported in Table 5 (page 28), while the oracle-utility, paired-context, and incorrect-visual-candidate diagnostics are provided in Appendix E (Tables 6–8, pages 28–29).

We hope these implementation clarifications, empirical results, and precise scope statements address the reviewer's concerns and support a more favorable assessment.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their constructive feedback and continued engagement. We would like to emphasize three key points.

1. Two complementary contributions. SchröHallu's novelty lies in an empirically grounded and theoretically characterized discovery of context-conditioned prior utility, and a decision-level intervention principle derived from this finding. Unlike methods that primarily regulate prior suppression, SchröHallu distinguishes prior localization, contextual harmfulness, and intervention necessity. We believe both the underlying insight and its methodological realization are essential to assessing the paper's overall contribution.

2. Mathematical and methodological clarification. We have clarified the actual decoding procedure, including candidate selection, intervention sufficiency, and full-vocabulary verification. Controlled experiments with identical intervention counts further demonstrate the effectiveness of contextual measurement (CHAIR CSC_S: 21.8% vs. 24.3%/26.1% for margin-/confidence-only gating).

3. Strengthened empirical validation. We expanded evaluations to four LVLMs across three model families, supplemented complete MME results and generation-quality analyses, and clarified that the quantum-inspired formulation serves as conceptual motivation rather than a claim of quantum computational advantage.

All rebuttal-related revisions have been incorporated into the revised manuscript and highlighted in red/blue/green. We have also provided detailed responses to each reviewer.

We respectfully hope that these clarifications and the combined empirical, theoretical, and methodological contributions will receive comprehensive consideration in the final assessment. We sincerely appreciate everyone's time and thoughtful evaluation.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.