acceptodds
Under review as a conference paper at ICLR 2027

Visual-Evidence-Guided Latent Correction for Hallucination Mitigation in LVLMs

Abstract

Large Vision-Language Models (LVLMs) remain prone to generating hallucinated content inconsistent with the input image during autoregressive generation. Recent online self-correction paradigms integrate verification and correction into the generation process, enabling models to roll back and regenerate upon detecting hallucinations. This addresses the limitations of conventional generation adjustment, which struggles to revise existing errors, and post-hoc verification, which relies on post-generation processing. However, existing online correction mechanisms mainly rely on sampling and text-conditioned adjustments after rollback, while lacking an explicit mechanism to incorporate visual evidence into the correction process after rollback. To address this issue, we propose Visual-Evidence-Guided Latent Correction (VE-LC). After hallucination detection and rollback, but before text regeneration, VE-LC performs Latent Visual Correction (LVC), which conducts reasoning in continuous latent space to re-integrate visual information before resuming standard autoregressive generation. We further introduce Local Hallucination Correction Supervision (LHCS), which strengthens local hallucination correction through edited-token reweighting and a visual candidate contrastive constraint. Experiments on six visual hallucination and general multimodal benchmarks and three LVLM architectures show that VE-LC can effectively reduce visual hallucinations while maintaining content coverage and general multimodal capabilities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Vision-Language Models, Hallucination, Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_A8yeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops Visual Evidence-guided Latent Correction (VE-LC), aiming to reduce hallucinations in large vision-language models (LVLMs). VE-LC is based on a monitoring and adaptation framework and incorporates the Latent Visual Correction (LVC) technique, which guides the subsequent reasoning process to better align with visual evidence. Furthermore, the method includes Local Hallucination Correction Supervision (LHCS), a mechanism that combines weighted correction signals with a contrastive learning objective. Experimental results across three LVLM architectures and six benchmarks demonstrate that the method significantly reduces hallucinations.

Critical strengths
  1. This paper examines the critical issue of visual hallucinations in LVLMs. This problem is important and undermines the reliability and credibility of the responses generated by the models, particularly in real-world application scenarios.

  2. The method is well motivated. The theoretical foundation is robust, and the explanation is clear. After identifying significant limitations in existing automatic self-calibration methods, the authors present a rational and effective solution, i.e., the use of latent correction based on visual evidence.

  3. The structure of this paper is clear, and the presentation is appropriate. The logical organization is rigorous and the writing is accessible, enabling readers to easily understand the proposed method and experimental results.

Critical weaknesses
  1. The novelty is limited. VE-LC seems to be a simple combination of existing works, i.e., OPERA (CVPR 24), VTI (ICLR 25), and MemVR (ICML 25).

  2. The experimental comparisons may be somewhat unfair. VE-LC is training-based, while most baselines in Table 1 and Table 6 are training-free. The authors should divide the baselines into training-based/free categories. For comparisons with training-free approaches, the authors are encouraged to use the no-training variant of VE-LC.

  3. Table 6 shows that VE-LC incurs 4.38× inference overhead compared to the base model. This additional overhead may undermine the utility of VE-LC in real-world applications. Moreover, since VE-LC involves the training phase, the computational complexity of this phase should also be computed and demonstrated.

  4. Some recent works should be more clearly compared or discussed [1,2,3,4,5].

  5. Minor: Line 247, modify EXPERIMENTS 4,500 to EXPERIMENTS.

References:

[1] Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering, ICLR 25.

[2] Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models, ICML 25.

[3] Hallucination-aware Intermediate Representation Edit in Large Vision-Language Models, ICLR 26.

[4] Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding, CVPR 26.

[5] Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models, CVPR 26.

Questions for authors

See weaknesses. I would like to raise my scores if the authors can effectively address my concerns.

3 comments
AuthorsOpenReview ↗

Response to Reviewer A8ye: Novelty, Fair Comparisons, and Computational Cost

We thank the reviewer for the constructive feedback and positive assessment of our motivation and framework.

W1. Novelty and correction mechanism. Our contribution is how visual evidence is used after rollback, not rollback or visual reuse alone. OPERA uses attention-based penalties and retrospection-allocation; VTI [1] applies precomputed visual/textual steering directions; MemVR [2] reinjects visual information through FFNs when uncertainty is high. VE-LC instead computes a correction signal at the current post-rollback prefix.

Specifically, LVC recursively updates continuous hidden states over the retained image-and-text KV cache and converts the normalized before/after logit difference into a residual for local regeneration (Eqs. 2-3). Visual alignment supervision encourages these states to retain correction-relevant evidence (Eq. 1). LHCS further emphasizes edited tokens and contrasts visually supported/unsupported candidates (Eqs. 4-8). Thus, the distinction lies in the construction and application of the correction signal and its local supervision.

A new controlled diagnostic supports this motivation: given a confirmed error location, we roll back and suppress the hallucinated object and its synonyms. Among 149 LLaVA-v1.5-7B cases, only 27 (18.12%) achieve visually consistent recovery; 121 (81.21%) still contain hallucinations, and one (0.67%) degrades. This shows that suppressing the identified concept alone does not reliably restore visually grounded continuations; it is not a direct comparison against VE-LC.

W2. Comparison fairness. We agree that training requirements must be explicit. We add AMP and POVID as training-based baselines, and MemVR and VTI as training-free baselines. Selected results on LLaVA-v1.5-7B are below. "Training" means additional method-specific training.

MethodTrainingAMBER CHAIR↓Cover↑HalRate↓Cog↓CHAIRs↓CHAIRi↓
MemVRNo7.551.834.94.351.614.6
VTINo4.848.621.42.234.610.5
AMPYes12.752.856.35.338.012.0
POVIDYes9.652.450.97.050.412.5
REVERSEYes9.049.540.53.834.09.9
VE-LCYes4.147.216.81.029.48.4

VE-LC improves both AMBER CHAIR and HalRate over all four added baselines, but its lower Cover reflects a hallucination-coverage trade-off. We do not claim uniform superiority: MemVR scores 73.75 on MMBench versus 73.73 for VE-LC. The reported VE-LC uses trained adapters, despite a frozen backbone. A no-additional-training VE-LC result is not included here; the cross-category comparisons therefore do not isolate the effects of training.

W3. Inference and training cost. We agree that 4.38× latency versus the base model is substantial and limits latency-sensitive deployment. In our timing experiments, 57% of responses did not trigger correction; this avoids rollback and latent correction, but not verification overhead. All online verification, rollback, correction, and regeneration are included in latency.

Under the evaluated configurations, VE-LC reduces mean latency from REVERSE's 58.90 s to 16.62 s (71.8%) and achieves 3.61× its output throughput, with similar output lengths (115.24 vs. 113.05 tokens). "Relatively low overhead" refers only to these comparisons, not to the base model. Appendix D specifies single-A40 inference timing with batch size 1.

For training, the base model is frozen and approximately 159.91M LoRA parameters are optimized. The recorded schedule comprises 2,500 initial, 375 contrastive, and 500 replay-training steps. The final training loop took approximately 6 h. This loop-level measurement should not be confused with a complete stage-wise accounting of training cost.

W4. Additional related work. Beyond the comparisons with [1,2] above, HIRE [3] selectively edits intermediate representations using an Editor and Router. We have not included HIRE results because the required pretrained Editor/Router checkpoints were not available to us. LEAD [4] uses entropy-triggered continuous embeddings and visual anchors; [5] identifies divergent-thinking steps and intervenes in MCoT decoding. These are relevant representation/decoding interventions, whereas VE-LC focuses on local regeneration after detected errors and rollback, using a recursively computed logit residual and correction-specific supervision. Reference numbers follow the review.

W5. Typographical error. Thank you for identifying this error. The heading at Line 247 should read "EXPERIMENTS"; the extraneous "4,500" will be removed.

Thanks for the detailed responses. After carefully reading the rebuttal, I still think the novelty of this paper is somewhat limited. Despite this, the authors' additional comparisons (W2), experiments (W3) and discussions (W4) effectively addressed my concerns. Please add these results in the revised PDF, and I'm adjusting my rating to 3.

Thank you for your careful consideration of our responses and your constructive feedback. We will incorporate the additional comparisons, experimental results, and related discussions into the revised manuscript.

Reviewer_BPRwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VE-LC, an online hallucination correction framework for large vision-language models. After a visual consistency verifier detects an unsupported segment, the system rolls back the generated text and its KV cache. Latent Visual Correction then performs continuous hidden-state updates and uses the resulting logit difference to modify subsequent token generation. Training combines visual feature alignment with Local Hallucination Correction Supervision, consisting of edited-token weighting and a preference loss between supported and unsupported continuations. The method is trained using a correction/preservation dataset derived from VisualCoT. Experiments cover six benchmarks, with primary comparisons on LLaVA-v1.5-7B and additional evaluations on Qwen2.5-VL-7B and InternVL3-1B.

Critical strengths
  1. The reported improvements on generative hallucination benchmarks are effective. These results make the proposed correction strategy worth investigating, even though the associated tradeoffs require closer examination.

  2. The LVC-only ablation provides a meaningful positive signal. This is stronger evidence than an improvement obtained solely by producing less informative descriptions.

Critical weaknesses
  1. The complete correction system is insufficiently specified, particularly the verifier and the interface between latent computation and decoding. Figure 2 introduces a frozen “Visual Consistency Verifier,” and Section 3.1 assumes that it identifies the hallucination onset. However, the manuscript does not identify its architecture or checkpoint, explain its detection and localization procedure, or specify its triggering threshold, invocation frequency, and correction-attempt budget.

  2. Several parts of LVC are also unresolved. Equation (2) leaves the normalization operator unspecified, and the correction coefficient α is not reported. The mapping G from a fixed number of latent states to a potentially variable number of target visual patches is not defined. The text also does not sufficiently explain positional handling and temporary cache updates during latent recursion, or how long residual-guided decoding remains active after a trigger.

  3. The training objectives explicitly condition on the latent sequence, whereas inference discards temporary latent states and uses a logit residual. The relationship between these two computations needs clarification.

  4. The experiments do not adequately isolate the proposed visual mechanism, and some conclusions exceed the demonstrated tradeoffs. Table 5 compares the base model, LVC, and the complete method, but does not include a standard correction fine-tuning baseline using the same data and LoRA budget, an LHCS-only variant, or a comparison that holds the verifier and correction budget fixed while changing only the regeneration mechanism.

  5. Preservation of coverage is not uniform. On InternVL3-1B, coverage decreases from 71.9 to 62.6 in Table 3. In Table 5, adding LHCS reduces coverage from 51.4 to 47.2. These tradeoffs are acknowledged locally, but the abstract and broader conclusions are stronger than the evidence supports.

  6. VE-LC is substantially faster than the reported REVERSE configuration, which is valuable, but still incurs 4.38× the base-model latency. Its average latency is similar to VCD while producing approximately 115 versus 256 tokens. Thus, this comparison does not establish comparable generation throughput.

Questions for authors
  1. What exactly is the frozen verifier, and how does it identify the rollback position? Please provide its checkpoint, trigger rule, invocation frequency, stopping conditions, and maximum correction budget. Please also specify G, the normalization operator, α, and the latent-state/cache update procedure. Do Equations (5)–(6) optimize the residual-corrected distribution in Equation (3), or a different distribution conditioned on retained latent states?

  2. With training data, LoRA budget, verifier, and decoding settings held fixed, how does VE-LC compare with standard correction fine-tuning followed by ordinary rollback/regeneration? Can the authors additionally isolate the effects of visual alignment and LHCS?

  3. How do VE-LC and REVERSE compare at similar coverage and inference budgets? Please report the verifier thresholds and correction-attempt limits used for both methods, together with correction frequencies. Does the advantage persist when accounting for the substantial coverage reduction on InternVL3?

4 comments
AuthorsOpenReview ↗

Response to Reviewer BPRw (1/2): Implementation and Training-Inference Interface

We thank the reviewer for identifying details that require clarification. This reply addresses W1-W3 and Q1; Part 2 presents the additional comparisons and trade-offs.

W1 / Q1. Frozen verifier and rollback. For LLaVA-v1.5-7B, the verifier is an independently trained, frozen L1-regularized logistic-regression classifier (C=0.03, balanced class weights) with feature standardization. Its 2,051-dimensional input comprises per-layer/per-head image-attention and BOS/sink-attention masses, predictive entropy, the top-two probability margin, and generated-token log probability. Its checkpoint contains the fitted classifier and standardizer, separately from the LoRA correction adapter.

Candidate object mentions are identified by longest-match recognition over the CHAIR/MSCOCO synonym vocabulary. Each newly completed mention is scored once. Correction is triggered when P(ungrounded)>0.62155888P(\mathrm{ungrounded})>0.62155888. This threshold is calibrated on an image-disjoint split to retain approximately 90% of grounded mentions. The classifier supplies the correction decision; the first subtoken of the triggered mention supplies the rollback position. Generated tokens, KV caches, and the attention mask are truncated to the preceding multimodal prefix. Thus, this implementation uses an object-mention-based trigger.

W2 / Q1. Normalization and visual alignment. In Eqs. (2)-(3), we use

r_t=\sigma(z_t^{\mathrm{base}})\frac{d_t-\mu(d_t)}{\sigma(d_t)+10^{-6}},\qquad \widetilde z_t=z_t^{\mathrm{base}}+\alpha r_t,$$ where $\mu$ and $\sigma$ are the mean and population standard deviation over the vocabulary dimension, with $K=8$ and $\alpha=0.1$. Here $z_t^{\mathrm{base}}$ comes from the frozen base model; $z_t^{(0)}$ and $z_t^{(K)}$ come from the trained correction model before and after latent updates. For Eq. (1), our LLaVA implementation resamples the projected ROI patch sequence, or full-image patch sequence when ROI annotations are unavailable, at evenly spaced indices to obtain $K$ visual targets. Thus $T_r=K$, and $G$ is the identity mapping in the shared LVLM hidden space. The variable patch count is handled by target resampling, rather than by expanding the latent sequence. These targets provide training supervision only; inference requires no ROI annotations. **W1-W2 / Q1. Positions, caches, and stopping conditions.** After rollback, the rejected mention and its synonyms are suppressed. For a retained prefix of length $L$, the $K$ latent updates occupy positions $L,\ldots,L+K-1$ in a temporary branch. Its KV cache grows at each update, allowing later latent states to attend to earlier ones while the retained prefix remains unchanged. After computing the residual, this branch is discarded. Only the selected real token is committed at position $L$ to the base- and correction-model prefix caches; no latent slots remain in the committed history. The residual is recomputed at each decoding position within a window of at most $W=8$ committed real tokens. Thus $K=8$ counts updates per position, whereas $W=8$ limits the number of real-token positions receiving correction. Ordinary autoregressive decoding resumes after this window. A new rollback restarts it. Each response allows at most 40 rollbacks, after which generation continues without further rollbacks. Generation terminates at EOS or the evaluation-specific token limit. **W3 / Q1. Training versus inference distributions.** Eq. (5) and Eqs. (6)-(7) supervise the latent-conditioned correction distribution $p_\theta(y_t\mid V,x,H_{\mathrm{Lat}},y_{<t})$ through edited-token weighting and visually supported/unsupported candidate ranking. Eq. (1) supplies visual alignment supervision, and Eq. (8) combines these objectives. They train the correction model, rather than directly optimizing the residual-adjusted distribution in Eq. (3). At inference, the trained correction model still performs latent computation. Its normalized before/after logit difference guides base-model token selection. Discarding the temporary states therefore does not discard their influence: that influence has already entered the token probabilities through the residual. The selected token is retained, and the residual is recomputed at subsequent positions within the window. This explains the connection between training and decoding without claiming that their two distributions are identical.
AuthorsOpenReview ↗

Title Response to Reviewer BPRw (2/2): Additional Controls, Coverage, and Efficiency Comment

W4 / Q2. Same-checkpoint local-correction control. On the same 200-sample Qwen local-correction set, we compare one Conditional LoRA checkpoint with K=0K=0 and K=8K=8, keeping adapter weights and other decoding settings fixed. Both branches decode directly from correction-model logits; the K=8K=8 branch uses latent-conditioned decoding. Neither uses the residual fusion in Eq. (3). This is a local-correction diagnostic, not an end-to-end ablation of residual-guided decoding.

ConfigurationSem. Acc.↑Sem./Part.↑Token F1↑Length MAE↓
Base Qwen0.0200.1000.07014.62
Same LoRA, K=00.1700.5700.33410.71
Same LoRA, K=80.2450.6150.4322.24

With identical weights, latent recurrence improves semantic accuracy by 7.5 percentage points and Token F1 by 0.098. This supports an inference-time benefit of latent computation under this trained checkpoint. It does not replace an independently trained standard correction baseline with matched data and LoRA budget. Table 5 evaluates LHCS added to LVC, not LHCS alone; neither comparison separately isolates visual alignment and LHCS.

W4-W5 / Q2-Q3. Shared-verifier comparison and coverage. We provide an additional AMBER comparison on LLaVA-v1.5-7B for this rebuttal:

MethodCHAIR↓Cover↑HalRate↓Cog↓
LLaVA-v1.5-7B11.249.947.94.7
REVERSE9.049.540.53.8
REVERSE + our verifier8.851.639.93.2
VE-LC (supplementary)5.451.826.41.8

The supplementary VE-LC row is distinct from Table 1's 4.1/47.2/16.8/1.0 and does not replace it. Compared with REVERSE using our verifier, VE-LC reduces CHAIR by 3.4 points and HalRate by 13.5 points at nearly equal coverage (51.8 versus 51.6). Replacing REVERSE's verifier does not close the observed gap. This is shared-verifier evidence at comparable coverage, not a fully fixed-budget comparison isolating regeneration.

For budget transparency, the previously recorded native-verifier REVERSE configuration used a threshold of 0.003 and up to 50 generation attempts. VE-LC uses the threshold in Part 1 and up to 40 rollbacks. On the supplementary VE-LC AMBER run, 345/1,004 responses (34.36%) triggered correction, averaging 0.87 rollbacks per response over all responses. Generation attempts and rollbacks count different operations, so these caps do not establish equal compute. The native-verifier REVERSE trigger statistics cannot be assigned to the new verifier-swapped row, for which matching intervention and compute accounting is not reported here.

For InternVL3-1B, Table 3 shows Cover decreasing from 71.9 to 62.6 while HalRate decreases from 56.6 to 28.9. On Qwen2.5-VL-7B, Cover changes from 64.3 to 64.1. In Table 5, adding LHCS changes Cover from 51.4 to 47.2 and HalRate from 29.0 to 16.8. These are architecture- and configuration-dependent hallucination-coverage trade-offs, not uniform coverage preservation. The supplementary LLaVA comparison does not establish an advantage at matched coverage on InternVL; that control is not reported here.

W6. Inference efficiency. We agree that 4.38× base-model latency is substantial. Similar latency to VCD does not imply comparable throughput: Table 6 reports 15.36 tokens/s for VCD versus 6.93 for VE-LC. We do not claim comparable raw-token throughput.

In saved outputs for 100 images, 97 VCD responses contained at least ten consecutive repetitions of the same whitespace-delimited token. This concerns our local reproduction, not an intrinsic defect in VCD, and does not overturn its reported raw-token throughput. It limits interpreting the longer outputs as more useful descriptive content.

Against the evaluated REVERSE configuration, VE-LC reduces mean latency from 58.90 s to 16.62 s (71.8%) and achieves 3.61× the reported output-token throughput, with similar output lengths (115.24 versus 113.05 tokens). Timing includes online verification, rollback, correction, and regeneration. In the timing analysis, 57% of responses required no correction, although verification overhead remained. These efficiency results apply to the Table 6 configurations, not to the new verifier-swapped comparison or a matched-compute experiment.

Reviewer_BPRwOpenReview ↗

Thank you for your responses.

Thank you for the detailed responses. The description of the LLaVA verifier, the rollback rule, and the temporary latent cache address my implementation concerns. My main remaining concern is how much the proposed correction mechanism contributes to the end-to-end result. The same-checkpoint Qwen experiment shows that latent recurrence helps on the local correction task. Both branches decode directly from the correction model, however, so this experiment leaves open how much the mechanism contributes in the online system that uses Eq. (3). I therefore maintain my rating of 2.

Thank you for clarifying your remaining concern. We agree that the earlier same-checkpoint Qwen experiment evaluates local correction through direct decoding from the correction model, rather than the online system using Eq. (3). To address the online-system question directly, we report a residual-strength ablation on LLaVA-v1.5-7B, evaluating complete generated responses.

1. Online ablation with fixed trained weights.

Across the VE-LC variants below, we keep the base model, trained correction checkpoint, evaluation samples, verifier and threshold, rollback and suppression policies, correction limits, and other decoding settings unchanged. We vary only α\alpha in

z~t=ztbase+αrt.\widetilde z_t=z_t^{\mathrm{base}}+\alpha r_t.

At α=0\alpha=0, the residual does not affect token selection, while verification, rollback, and suppression remain enabled. This is therefore not the unmodified base model or removal of visual-alignment training. At positive α\alpha, the latent-derived residual adjusts base-model logits within the online correction windows. The policies are fixed, although realized correction locations and frequencies can change as the generated responses change.

ConfigurationCHAIRs↓CHAIRi↓Object Recall↑Avg. length
LLaVA-v1.5-7B49.8015.5076.87100.70
VE-LC, α=0\alpha=049.6013.8478.51100.90
VE-LC, α=0.05\alpha=0.0537.2010.3775.43101.43
VE-LC, α=0.1\alpha=0.129.408.3972.15101.88
VE-LC, α=0.2\alpha=0.228.007.7770.83101.75
VE-LC, α=0.3\alpha=0.327.407.5669.82102.00

2. Contribution to final-response hallucination metrics.

Relative to α=0\alpha=0, our default α=0.1\alpha=0.1 reduces CHAIRs from 49.60 to 29.40 and CHAIRi from 13.84 to 8.39, by 20.20 and 5.45 percentage points, respectively. This provides end-to-end evidence for the contribution of enabling the latent-derived residual in Eq. (3) under the same trained weights and online policies, rather than comparing differently fine-tuned models or directly decoding from the correction model.

Average length does not decrease (100.90 to 101.88), so the lower hallucination scores are not accompanied by shorter responses on average. However, object Recall decreases from 78.51 to 72.15. We acknowledge this hallucination-recall trade-off: similar length does not imply preserved object coverage.

3. Sensitivity around the default α=0.1\alpha=0.1.

The hallucination metrics improve sharply up to α=0.1\alpha=0.1, with smaller additional gains thereafter. Increasing α\alpha from 0.1 to 0.3 reduces CHAIRs/CHAIRi by only a further 2.00/0.83 points, while object Recall drops by another 2.33 points. This pattern is consistent with α=0.1\alpha=0.1 providing a practical trade-off among the evaluated settings, rather than optimizing every metric.

This ablation complements the earlier Qwen diagnostic by testing the residual's contribution to final outputs in the online system. Its conclusion is an end-to-end reduction in hallucination metrics under fixed trained weights; it does not establish superiority at matched recall or separately isolate the contributions of visual-alignment and LHCS training.

Reviewer_ZjtEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VE-LC extends online self-correction in the style of REVERSE [1]. When a visual consistency verifier flags a hallucination at position ii, decoding rolls back to the prefix y<iy_{< i} and truncates the KV cache. Latent Visual Correction (LVC) then performs K=8K = 8 recursive latent updates, h(k)=Fθ(h(k−1);Kc)h^{(k)} = F_\theta(h^{(k-1)}; K_c), and adds a normalized logit residual to the base logits, z~t=ztbase+α,N(zt(K)−zt(0))\tilde{z}_t = z^{\mathrm{base}}_t + \alpha \\, \mathcal{N}(z^{(K)}_t - z^{(0)}_t). During training, the latent states are aligned with ROI or full-image visual features through an MSE loss. Local Hallucination Correction Supervision (LHCS) adds a cross-entropy term that up-weights edited tokens and a margin ranking loss between visually supported and unsupported candidates. LoRA adapters are trained on 3,900 samples derived from Visual CoT. The paper reports improvements on AMBER, CHAIR, POPE, MME, MMBench and MMHal-Bench with LLaVA-v1.5-7B, together with results on Qwen2.5-VL-7B and InternVL3-1B.

Critical strengths
  1. Relevant question. The paper asks whether the correction step after rollback can be explicitly grounded in the image rather than relying only on resampling, which is a sensible direction for online self-correction.
  2. Broad evaluation. The experiments cover generative, discriminative, general-capability and open-ended benchmarks, plus two additional backbones.
  3. Lower cost than REVERSE. Because correction is triggered only on demand, average latency (16.6 s) is well below that of REVERSE (58.9 s).
Critical weaknesses

W1. LVC adds no new visual information at inference.

The latent updates attend to the same multimodal cache KcK_c, which contains the same visual tokens that ordinary decoding already attends to. No ROI, re-encoding or additional visual input is used at inference. Mechanically, LVC therefore amounts to KK additional forward passes of computation, a form of test-time compute similar to continuous thoughts [2] or pause tokens [3], rather than a re-integration of visual evidence. The visual alignment loss only acts during training. Supporting the claim that LVC injects visual evidence requires causal controls: the same KK latent steps with λLat=0\lambda_{\mathrm{Lat}} = 0, latent steps driven by constant or random inputs (a pause-token-style control), and K=0K = 0 with the same LoRA adapters.

W2. The latent mechanism and its loss closely follow LVR.

Eq. 1 is the objective of Latent Visual Reasoning (LVR) [4]: an MSE between latent hidden states and ROI visual embeddings, built from the same Visual CoT ROI annotations, with ROI used only during training and a fixed latent budget (LVR reports 4, 8 or 16 steps; VE-LC uses 8). LVR is cited only in the related work, and the overlap is not discussed. The new element is invoking this latent reasoning after a rollback and converting it into a logit residual.

W3. The logit residual is an under-specified contrastive extrapolation.

If zbase=z(0)z^{\mathrm{base}} = z^{(0)}, then z~=z(0)+α,N(z(K)−z(0))\tilde{z} = z^{(0)} + \alpha \\, \mathcal{N}(z^{(K)} - z^{(0)}) amplifies the change between the predictions before and after the latent steps. This is a contrastive-decoding operation in the spirit of DoLa [5], not visual grounding as such. The normalization N(⋅)\mathcal{N}(\cdot) is not defined, it is not stated whether zbasez^{\mathrm{base}} comes from the frozen model or the LoRA-adapted one, and it is unclear how a residual computed once at the rollback position becomes the per-step residual rtr_t applied to subsequent tokens.

W4. The verifier is not specified.

The visual consistency verifier determines when rollback occurs and where the onset position ii lies, yet the paper never states what it is (e.g., REVERSE's trained confidence tokens [1] or an external model), how it is trained, how accurate it is, or how often it fires. If it is REVERSE's model, VE-LC is built on REVERSE's fine-tuned LLaVA, and comparing VE-LC against the plain LLaVA-v1.5 conflates two changes. The key controlled comparison is also missing: the same verifier with REVERSE's resampling versus the same verifier with LVC.

W5. The effect of fine-tuning is confounded with LVC.

VE-LC trains LoRA adapters on 3,900 correction and preservation samples, while most baselines are training-free. Missing ablations include LoRA fine-tuning on the same data with LWCE\mathcal{L}_{\mathrm{WCE}} only (with rollback but without latent steps), sweeps over KK and α\alpha, and LVC with λLat=0\lambda_{\mathrm{Lat}} = 0. Table 5 does not state how the "+LVC" variant was trained. Table 4 compares a model trained for local correction against an untrained "Base Qwen" (a different backbone from the main experiments); the base model's Length MAE of 14.6 indicates that it does not even follow the task format, so the table cannot isolate the contribution of latent reasoning.

W6. The cross-architecture results have no baselines.

Table 3 compares VE-LC only with the base models. REVERSE provides a Qwen2.5-VL variant [1] and could have been included, and general-capability benchmarks are not reported for these two backbones. Coverage also drops noticeably on InternVL3-1B (AMBER Cover 71.9 to 62.6).

W7. The efficiency claim is overstated.

Mean latency is 4.38 times that of the base model, P95 latency is about 7.5 times higher, and throughput falls from 30.0 to 6.9 tokens per second. Rollback frequency and the number of latent invocations per response are not reported, so the "low overhead" claim cannot be assessed.

Questions for authors
  1. Verifier (W4). What exactly is the visual consistency verifier? How is it trained, how accurate is it, how is the onset position ii determined, and how often does it trigger per response? Is it REVERSE's model, and if so, is VE-LC fine-tuned on top of REVERSE's weights?
  2. Controlled comparison with REVERSE (W4). How does VE-LC compare with REVERSE when both use the same verifier and the same fine-tuned base model, differing only in the correction step?
  3. Visual evidence at inference (W1). What are the results with λLat=0\lambda_{\mathrm{Lat}} = 0, with pause-style latent inputs, and for K∈0,2,4,8,16K \in \\{0, 2, 4, 8, 16\\}?
  4. Latent update details (W1, W2). Which hidden state, from which layer, is h(0)h^{(0)}? Is h(k−1)h^{(k-1)} fed back as an input embedding, as in Coconut [2]? How does LVC differ from LVR [4] beyond being invoked after rollback?
  5. Residual details (W3). What is N(⋅)\mathcal{N}(\cdot), is zbasez^{\mathrm{base}} produced by the frozen or the LoRA-adapted model, and over how many tokens after rollback is the residual applied?
  6. Visual alignment head. What is the architecture of GG, how are the K=8K = 8 latent states matched to TrT_r target tokens when Tr≠KT_r \neq K, and in which feature space (before or after the projector) are the targets defined?
  7. Fine-tuning confound (W5). What do LoRA fine-tuning on the same data with LWCE\mathcal{L}_{\mathrm{WCE}} only, and rollback without latent steps, achieve? How exactly was the "+LVC" variant in Table 5 trained, and why does Table 4 use a Qwen backbone?
  8. Cross-architecture baselines (W6). How does VE-LC compare with REVERSE's Qwen2.5-VL variant [1], and how do the Qwen2.5-VL-7B and InternVL3-1B versions perform on the general-capability benchmarks (MME, MMBench)?
  9. Cost (W7). What are the rollback frequency and the number of latent invocations per response?
  10. Minor. The "Rec." column in Table 2 appears to report F1 (80.10 for LLaVA-v1.5 matches the F1 average in Table 7); please correct the header. The section heading "4 Experiments 4,500" contains a stray number.

References

[1] T.-H. Wu et al. Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling. NeurIPS, 2025.

[2] S. Goyal et al. Think Before You Speak: Training Language Models with Pause Tokens. ICLR, 2024.

[3] B. Li et al. Latent Visual Reasoning. ICLR, 2026.

[4] Y.-S. Chuang et al. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. ICLR, 2024.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZjtE (1/2): Mechanism, Controls, and Implementation

We thank the reviewer for the detailed assessment. We distinguish inherited components, implementation details, and the evidence provided by additional controls.

W1-W2 / Q3-Q4. Existing visual evidence and relation to LVR. LVC does not acquire new images, ROI crops, or re-encoded visual features at inference. It further processes the retained multimodal context. We adopt LVR's ROI-supervised alignment objective and do not claim latent recurrence or MSE alignment as independent innovations. VE-LC instead uses temporary latent computation after rollback to construct a logit residual for local regeneration, together with LHCS supervision. The residual has a contrastive form; that operation alone does not establish visual grounding.

The fixed-input Repeated h0h_0 control gives the following AMBER results:

ConfigurationCHAIR↓Cover↑HalRate↓Cog↓
LLaVA-v1.5-7B11.249.947.94.7
Repeated h0h_07.651.634.64.1
VE-LC4.147.216.81.0

VE-LC reduces hallucinations more, but also has lower coverage. This comparison does not fully isolate visual supervision from additional computation. A λLat=0\lambda_{\mathrm{Lat}}=0 training control and the full K∈{0,2,4,8,16}K\in\{0,2,4,8,16\} sweep are not reported here; we do not claim these controls establish the standalone effect of alignment or the optimality of K=8K=8.

W5 / Q7. Fine-tuning control and Tables 4-5. On the same 200-sample Qwen local-correction set, we compare one Conditional LoRA checkpoint with K=0K=0 and K=8K=8. Both branches decode directly from correction-model logits; the latter uses latent-conditioned decoding. Neither uses Eq. (3)'s residual fusion.

ConfigurationSem. Acc.↑Sem./Part.↑Token F1↑Length MAE↓
Base Qwen0.0200.1000.07014.62
Same LoRA, K=00.1700.5700.33410.71
Same LoRA, K=80.2450.6150.4322.24

With identical weights, recurrence improves semantic accuracy by 7.5 points and Token F1 by 0.098. This local diagnostic on Qwen is distinct from the LLaVA online evaluation, and is not a separately trained, same-data CE-only rollback/regeneration baseline. Length MAE measures absolute token-length deviation from the reference, not format compliance itself.

The LVC-only variant uses uniform token weights and LCE+λLatLLatL_{\mathrm{CE}}+\lambda_{\mathrm{Lat}}L_{\mathrm{Lat}}. Full VE-LC uses LWCE+λLatLLat+λconLconL_{\mathrm{WCE}}+\lambda_{\mathrm{Lat}}L_{\mathrm{Lat}}+\lambda_{\mathrm{con}}L_{\mathrm{con}}, with edited-token weight 3, λLat=0.05\lambda_{\mathrm{Lat}}=0.05, λcon=0.25\lambda_{\mathrm{con}}=0.25, and margin 0.5. Thus Table 5 is an incremental component comparison, not a complete isolation of all training effects.

W3 / Q4-Q6. Residual, visual targets, and temporary states. In our LLaVA implementation, ztbasez_t^{\mathrm{base}} comes from the frozen base model; zt(0)z_t^{(0)} and zt(K)z_t^{(K)} come from the same LoRA correction model before/after latent updates. They need not equal the base logits. The normalization is

d_t&=z_t^{(K)}-z_t^{(0)},\\ r_t&=\sigma(z_t^{\mathrm{base}})\frac{d_t-\mu(d_t)}{\sigma(d_t)+10^{-6}},\\ \widetilde z_t&=z_t^{\mathrm{base}}+\alpha r_t. \end{aligned}$$ Here $\mu$ and $\sigma$ denote the mean and population standard deviation over the vocabulary dimension. Thus $\sigma(z_t^{\mathrm{base}})$ is a scalar rescaling factor. We use $K=8$ and $\alpha=0.1$. The residual is recomputed at each decoding position, not reused unchanged across tokens. Our supplementary $\alpha:0\to0.1$ test changes CHAIRs/CHAIRi from 49.60/13.84 to 29.40/8.39. This tests residual strength, not removal of visual-alignment training. For alignment, projected ROI patches, or full-image patches without ROI annotations, are resampled at evenly spaced indices into $K$ targets. Thus $T_r=K$ and $G=I$ in the shared hidden space; targets are after the multimodal projector and used only during training. Starting from the current-position hidden state after rollback, the recurrence feeds continuous hidden states back rather than sampled tokens. For prefix length $L$, latent positions $L,\ldots,L+K-1$ and their growing cache are temporary. After computing the residual, this branch is discarded; only the selected real token is committed at position $L$ to both prefix caches. Correction lasts at most $W=8$ committed real-token positions, each using $K=8$ updates. A new rollback restarts the window. Training supervises latent-conditioned prediction; inference transfers its before/after prediction change through the residual rather than retaining latent states in the committed history.
AuthorsOpenReview ↗

Response to Reviewer ZjtE (2/2): Verification, Baselines, and Efficiency

W4 / Q1-Q2. Verifier and shared-verifier comparison. Our LLaVA verifier is independently trained and frozen, not REVERSE's confidence-token mechanism; our correction adapter is not initialized from REVERSE's weights. It is an L1 logistic-regression classifier (C=0.03, balanced classes) with feature standardization. Its 2,051 inputs comprise per-layer/head image-attention and BOS/sink-attention masses, predictive entropy, top-two probability margin, and generated-token log probability.

Completed object mentions are identified by longest matching against CHAIR/MSCOCO synonyms and scored once. Rollback occurs when P(ungrounded)>0.62155888P(\mathrm{ungrounded})>0.62155888, calibrated on an image-disjoint split to retain approximately 90% of grounded mentions. This is a calibration target, not an overall detection-accuracy estimate. Rollback starts at the mention's first subtoken; tokens and caches are truncated to the preceding prefix, and the rejected mention and synonyms are suppressed. At most 40 rollbacks are allowed; generation then continues without further rollback and ends at EOS or the token limit.

Additional AMBER results on LLaVA-v1.5-7B are:

MethodCHAIR↓Cover↑HalRate↓Cog↓
Base11.249.947.94.7
REVERSE9.049.540.53.8
REVERSE + our verifier8.851.639.93.2
VE-LC (supplementary)5.451.826.41.8

Against REVERSE using our verifier, VE-LC reduces CHAIR by 3.4 points and HalRate by 13.5 points at nearly equal coverage. This controls the verifier choice, not the fine-tuned weights or all correction budgets; it is not a comparison differing only in regeneration. The supplementary VE-LC row does not replace Table 1's 4.1/47.2/16.8/1.0.

W5. Broader baseline coverage. We add AMP and POVID (training-based), and MemVR and VTI (training-free). Selected results are below. MME-4 sums Existence, Count, Position, and Color, not the full MME suite.

MethodTrainingAMBER CHAIR↓Cover↑HalRate↓Cog↓CHAIRs↓CHAIRi↓POPE F1↑MME-4↑MMB↑
MemVRNo7.551.834.94.351.614.685.81587.2273.75
VTINo4.848.621.42.234.610.581.17580.7471.01
AMPYes12.752.856.35.338.012.084.66518.3366.42
POVIDYes9.652.450.97.050.412.585.50588.3364.90
VE-LCYes4.147.216.81.029.48.485.87652.4073.73

VE-LC improves AMBER CHAIR/HalRate over these four baselines with lower Cover; MemVR's MMBench score is slightly higher. These comparisons broaden coverage but do not replace matched-training controls.

W6 / Q8. Cross-architecture scope and general capabilities. Table 3 tests applicability relative to each base model, not superiority over competing correction methods. The released REVERSE Qwen checkpoint uses Qwen2.5-VL-3B-Instruct trained on a 100K subset, whereas our Qwen evaluation uses 7B with LoRA. This mismatch does not preclude a matched retraining study; such a comparison is not reported here.

Additional general-capability results are:

ModelExistence↑Count↑Position↑Color↑MME-4↑MMB↑
Qwen2.5-VL-7B195.55164.15150.60178.15688.4584.46
+ VE-LC200.00153.85156.50195.10705.4587.16
InternVL3-1B190.00133.33105.00140.00568.3376.99
+ VE-LC190.00136.67118.33151.67596.6779.94

The sums and MMBench scores improve, but not every subtask. InternVL's AMBER Cover falls from 71.9 to 62.6 while HalRate falls from 56.6 to 28.9; Qwen Cover changes from 64.3 to 64.1. We therefore acknowledge architecture-dependent trade-offs rather than uniform coverage preservation.

W7 / Q9. Cost and invocation frequency. VE-LC incurs 4.38× mean latency and approximately 7.47× P95 latency versus the base model, with throughput falling from 30.04 to 6.93 tokens/s. This is substantial overhead. Compared only with the evaluated REVERSE configuration, mean latency decreases from 58.90 to 16.62 s (71.8%) and P95 from 173.63 to 34.57 s (80.1%); output-token throughput is 3.61× as high, with similar lengths (115.24 vs. 113.05 tokens).

Across three timing runs on 100 images, 57% of responses trigger no correction. Averages over all responses are 1.81 rollbacks and 10.55 latent invocations; each invocation has eight updates, giving 84.40 updates per response. All verification, rollback, correction, and regeneration costs are included; no correction does not mean no verification cost.

Q10. Minor issues. The POPE column labeled "Rec." in Table 2 should be "F1", consistent with Table 7. The heading at Line 247 should read "EXPERIMENTS" without "4,500".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.