acceptodds
Withdrawn from ICLR 2027

Markov Proposals: Iterating Visual Attention for Vision–Language Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite rapid progress, Large Vision-Language Models (LVLMs) still generate descriptions unsupported by the visual input. One possible contributor is insufficient use of visual evidence during generation, involving both the amount of attention allocated to image tokens and its distribution across image regions. We propose a training-free method that guides the language decoder's attention using a spatial prior derived from the vision encoder. We interpret the encoder's self-attention as a weighted directed graph over image patches as nodes and propagate attention through this graph as a discrete-time Markov chain. The resulting Markov proposal incorporates indirect relationships among nodes to expose spatial structure beyond direct attention connections. We blend this image-dependent prior into the decoder's image-token attention at selected layers and normalize the full attention row. This intervention increases total image attention while guiding its spatial allocation through a combination of the shared proposal and the decoder's token-dependent attention. Our empirical analysis shows that a frozen LVLM can benefit from increased attention to annotated object regions. Controlled comparisons further show that Markov steering improves supported-token margins beyond proportional image-attention scaling, supporting a benefit from proposal-based redistribution of existing visual information. The method introduces no additional learned parameters and requires no auxiliary data or training. Experiments on LLaVA-1.5-7B and MiniGPT-4 demonstrate improvements across CHAIR, SPICE, BLEU, and POPE.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Large Vision-Language Models, Hallucination Mitigation, Markov Chains, Visual Grounding, Training-Free Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7gbsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free method that constructs an image-dependent spatial prior through Markov propagation over visual encoder attention and blends this prior into LLM attention. The reported results demonstrate reduced object hallucination on LLaVA-1.5 and MiniGPT-4.

Critical strengths

None.

This method has some conceptual novelty and a plausible motivation (like interprete attention as a graph). However, I do not consider these merits sufficient to support acceptance.

Critical weaknesses
  1. I consider the primary value of a training-free method to be its ability to improve strong existing models at low additional cost. However, the evaluation is restricted to the outdated MLLMs (LLaVA-1.5 and MiniGPT-4). It therefore remains unclear whether the method provides meaningful improvements on more recent, stronger MLLMs

  2. This paper shows insufficient engagement with recent work. I find that the references include only papers published in 2025 or earlier. For an ICLR 2027 submission, I believe the lack of coverage of more recent literature is a significant weakness.

Questions for authors
  1. How does the proposed method perform on more recent and stronger MLLMs, such as Qwen3-VL, InternVL3.5, or GLM-4.5V? Evaluation on at least one SOTA MLLM would provide stronger evidence for the effectiveness and generalizability of the proposed method.
Reviewer_SqLdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propagates vision-encoder attention to construct a spatial prior, blends it into decoder attention over image tokens, and renormalizes the full row. This changes both image-attention mass and spatial distribution. Experiments on LLaVA-1.5 and MiniGPT-4 evaluate caption hallucination and visual grounding.

Critical strengths
  1. Large reductions in hallucination. On LLaVA/COCO, CHAIRS falls from 51.8 to 20.2 and CHAIRI from 13.3 to 5.3. Table 2 reports the best result in seven of eight settings, with AD-HH among the baselines. Nocaps and MiniGPT provide transfer evidence. The encoder-to-decoder connection is simple and requires no learned auxiliary module.

  2. Relevant controls and quality checks. Markov exceeds proportional mass-only steering by 0.1773 in the fixed-prefix supported-word margin, with a positive paired-bootstrap interval. The mass-only control matches the local update rule, although multilayer internal states can diverge. Figure 4c already compares complete captions under the same blend: one to three bounces improves CHAIRS from 22.4 to 20.2 and CHAIRI from 5.8 to 5.3, while SPICE changes from 21.99 to 21.88. This supports a modest propagation benefit. SPICE, BLEU, POPE, category scores and blending sweeps provide further checks. Steering annotated regions also outperforms equal-area outside regions, supporting the usefulness of object-associated representations. The blending sweep exposes deterioration at excessive strength, and the authors explicitly discuss informative omissions.

Critical weaknesses
  1. The hyperparameter selection protocol may favor the reported COCO results. The main COCO results and the ablation study appear to use essentially the same evaluation setup. The final configuration, including the steering-layer range, blending coefficient α\alpha, and number of Markov propagation steps nn, is selected based on the observed trade-off between CHAIR and SPICE on COCO. In contrast, the competing baselines are run using hyperparameters reported in their original papers rather than being tuned on the same evaluation subset. This creates a potential fairness issue: part of the reported advantage may come from benchmark-specific tuning rather than from the method itself. The paper should clearly separate validation and test data for hyperparameter selection, and ideally provide either a comparable tuning budget for the baselines or a configuration selected independently of the final evaluation set.

  2. The current experiments do not sufficiently establish that the Markov iteration itself is the main source of improvement. The central algorithmic novelty is the iterative propagation of encoder attention as a Markov chain. However, the ablation shows that increasing the number of propagation steps from n=1n=1 to n=3n=3 only provides a relatively modest improvement compared with the much larger overall gain over the original model. This raises the possibility that most of the benefit comes from injecting an encoder-derived spatial prior into the decoder, rather than from the proposed high-order Markov propagation specifically. The paper therefore needs stronger structural baselines, such as no propagation (n=0n=0), direct use of the raw encoder attention, one-step propagation, attention rollout/composition, simple spatial smoothing, or the stationary distribution of the same transition matrix. Without these comparisons, it is difficult to determine whether the Markov formulation contributes something fundamentally different from generic attention redistribution.

  3. The diagnostic analysis does not use exactly the same proposal initialization as the main algorithm. In the main method, the Markov chain is initialized from a one-hot seed corresponding to the image patch with the largest total incoming attention. In the diagnostic analysis, however, the proposal is initialized from a normalized distribution over the incoming attention of all patches. These are substantially different initialization strategies: one concentrates all probability mass on a single location, while the other starts from a soft global distribution. This inconsistency is important because the diagnostic experiment is used to support the main mechanistic claim that Markov steering improves supported-token preference beyond merely increasing image-attention mass. If the diagnostic uses a different proposal construction, it does not directly validate the mechanism actually used in the main experiments. The authors should repeat the controlled analysis using exactly the same initialization as the final method, or clearly justify why the two constructions are equivalent for the claimed conclusion.

  4. The query-independent proposal is not fully consistent with the claimed vision–language alignment mechanism. The Markov proposal is computed only from the image and remains fixed across all decoding steps. It is not conditioned on the question, the current generated token, or the semantic content being discussed. However, the visually relevant region generally depends strongly on the language context. For example, different questions about the same image may require attending to entirely different objects or regions. This makes the method closer to a fixed image-saliency prior than to a truly language-conditioned alignment mechanism. The issue is particularly important for small, less salient, or secondary objects, where the globally dominant visual region may not correspond to the queried concept. The paper should evaluate multiple questions referring to different regions of the same image and compare against a query-conditioned proposal or token-conditioned attention prior.

  5. The large reduction in CHAIR may be partially confounded by shorter and more conservative captions. The paper reports very large improvements in object hallucination metrics, but it also explicitly observes that the proposed steering reduces caption length and trades descriptive coverage for precision. This is a major confounding factor for CHAIR: if the model mentions fewer objects, it naturally has fewer opportunities to hallucinate unsupported objects. Therefore, lower CHAIR does not by itself demonstrate that the model has learned substantially better visual grounding. The additional semantic metrics partially address this issue, but they do not fully separate grounding improvement from reduced coverage, especially because some semantic categories such as cardinality and size actually decrease. A stronger evaluation should control for caption length or the number of object mentions, and report metrics such as matched-length CHAIR, object precision/recall/F1, or hallucination rate under approximately equal descriptive coverage.

  6. Several important components of the method remain heuristic and are not sufficiently ablated. The final performance depends on multiple design choices, including the encoder layer used to construct the proposal, the seed-selection strategy, head aggregation, propagation depth, decoder steering range, and blending coefficient. However, only the latter three receive systematic ablation. In particular, the encoder layer is selected from a preliminary scan over only a small number of images, while uniform head averaging and the maximum-incoming-attention seed are adopted without strong empirical justification. These choices directly determine the spatial prior, and the paper itself acknowledges that an unfavorable seed can propagate attention toward irrelevant regions. Therefore, the method should include ablations over encoder layers, different head aggregation schemes, and alternative initializations such as random seeds, top-kk seeds, soft incoming-attention initialization, and uniform initialization. Such experiments are necessary to show that the method captures a robust property of the model rather than depending on a favorable combination of heuristic choices.

*Minor Issues

  1. Erel et al. already use multi-bounce attention and guide diffusion generation; revise the segmentation-only contrast. Discuss PAI/VAR as attention-intervention precedents while identifying the distinct encoder prior.

  2. Correct VTI's visual direction to perturbed-versus-original image features; paired captions construct its textual direction. Explain its omission from Table 2.

  3. Specify CLS handling, normalization, layer numbering, and development splits. Report relative latency and peak memory, including the additional encoder pass. Release the diagnostic candidate pairs, prefixes, and image IDs, plus Nocaps-adapted CHAIR rules; absolute 2.5-3 s/image does not isolate the extra encoder cost.

Questions for authors

Please refer to Weaknesses.

Reviewer_pw79OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free method to mitigate hallucinations in large vision-language models (LVLMs). The method interprets the vision encoder's self-attention as a Markov transition matrix and performs iterative propagation to construct a spatial attention prior. This prior is then blended into the language decoder's attention to image tokens during inference. Experiments on LLaVA-1.5-7B and MiniGPT-4 demonstrate substantial reductions in object hallucination on CHAIR, along with improvements in SPICE and BLEU.

Critical strengths
  • Training-free hallucination mitigation. The method requires no additional training or learned parameters and achieves substantial improvements on CHAIR across two LVLMs.

  • Controlled attention analysis. The authors compare Markov steering with proportional image-attention scaling, providing evidence that redistributing attention can offer benefits beyond simply increasing total visual attention.

Critical weaknesses
  • W1. Limited novelty and insufficient comparison with closely related work. The central idea of mitigating LVLM hallucinations through inference-time attention intervention has been extensively explored. In particular, PAI (ECCV 2024) enhances attention to image tokens, DAMRO (EMNLP 2024) uses vision-encoder attention to identify problematic visual tokens and intervene during decoding, and EVAS (EMNLP 2025) redistributes visual attention maps across attention heads. These highly relevant methods are neither discussed nor evaluated. Furthermore, interpreting self-attention as a Markov chain and using iterative propagation to capture indirect spatial relationships was already introduced by Erel et al. (NeurIPS 2025), which the paper cites. The proposed method therefore appears to be a relatively straightforward combination of existing attention propagation and intervention techniques. Its distinctive contribution and advantages over simpler alternatives remain insufficiently established.

  • W2. Insufficient validation of the proposed mechanism and potential evaluation confounds. The mass-only control demonstrates the benefit of spatial redistribution but does not isolate the contribution of Markov propagation. Figure 4(c) shows only modest differences across propagation depths, while comparisons against simpler encoder-derived attention priors are missing. Moreover, the method reduces caption length, which can lower CHAIR scores without necessarily improving visual grounding or descriptive coverage. The evaluation is also limited to relatively old LVLM architectures, and inference overhead is not adequately quantified.

Questions for authors

Q1. Novelty and missing comparisons with prior attention-intervention methods. How does the proposed approach differ fundamentally from PAI (ECCV 2024), DAMRO (EMNLP 2024), and EVAS (EMNLP 2025)? These methods already manipulate visual attention during inference to mitigate hallucinations. Could the authors compare against them under matched evaluation and decoding settings? Such comparisons are important for establishing the contribution beyond existing attention-steering techniques.

Q2. Is iterative Markov propagation actually necessary? Figure 4(c) shows relatively small improvements from one to three propagation steps (CHAIR-S: 22.4 to 20.2). Although the mass-only control is useful, it does not establish whether Markov propagation is superior to simpler spatial priors. Could the authors compare against unpropagated CLIP attention, normalized attention column sums, and other simple prior distributions, while matching the total image-attention mass?

Q3. Does reduced caption length explain part of the hallucination reduction? The paper acknowledges that steering shortens generated captions. Since CHAIR-S is sensitive to the number of mentioned objects, could the authors report average caption lengths, object coverage, and hallucination rates under length-matched generation? This is particularly important given the large CHAIR improvements but relatively modest POPE gains.

Q4. Generalization and inference efficiency. The proposal is image-dependent but query-independent, potentially limiting its usefulness for questions involving small or less salient objects. Could the authors evaluate more recent LVLMs and open-ended VQA tasks beyond object-presence questions? Additionally, Appendix 7.3 acknowledges an extra vision-encoder pass in the LLaVA implementation. What are the end-to-end latency and throughput overheads relative to the original model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.