acceptodds
Under review as a conference paper at ICLR 2027

Can Topological Decoding Make VLMs as Reliable as PageRank Made Web Retrieval?

Abstract

Vision-Language Models (VLMs) frequently suffer from generative semantic drift, where a heavy reliance on frozen parametric weights overrides discrete visual evidence, causing compounding hallucinations. Existing decoding interventions attempt to mitigate this but often fracture linguistic fluency or incur prohibitive latency. To resolve this factuality-fluency-latency trade-off, we introduce Topological Decoding, an inference-time architecture that fuses discrete Visual Grounding Graphs (VGG) with the VLM's logits via a PageRank-inspired algorithm. To accurately localize the VGG and achieve high spatial precision with minimal latency, we employ Group Relative Policy Optimization (GRPO) for VLM object recognition. During inference, a Controllable Faithfulness Dial () is introduced to provide users with dynamic flexibility, interpolating between factual grounding and linguistic fluency. Adding only 0.3 seconds of latency, Extensive evaluation on 1.2 million Visual Genome queries demonstrates our approach consistently outperforms state-of-the-art interventions in spatial faithfulness, achieving 72.8% compared to VCD (52.1%), OPERA (56.4%), and Gradient-based decoding (59.2%). Furthermore, our RL-adapted policy achieves 94.1% [email protected] on RefCOCO while fully preserving native multimodal capabilities on MT-Bench and MMMU.

Primary area:
generative models
Keywords:
Vision-Language Models (VLMs), Object Hallucination, Inference-Time Decoding, Visual Grounding, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_QKu9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper combines SFT and IoU-rewarded GRPO for bounding-box generation with inference-time interpolation between a VLM's next-token distribution and tokens supported by predefined Visual Grounding Graphs. It reports improved entity matching with limited decoding overhead. The problem is relevant, but the submission does not sufficiently establish novelty or hallucination reduction. Unspecified VGG provenance and inconsistencies between the equations and reported quantities substantially limit confidence in the conclusions.

Critical strengths
  1. Object-specific grounding addresses a relevant source of errors in visual descriptions.
  2. The interpolation parameter is interpretable, and the evaluation considers localization, fluency, latency, and capability preservation.
Critical weaknesses
  1. The contribution of graph topology is unclear. Equations (3), (8), and (9) do not use graph edges; a box-to-attribute lookup table appears equivalent. Interpolation and IoU-rewarded GRPO have precedents in kNN-LM (Khandelwal et al., 2020) and Visual-RFT (Liu et al., 2025). Page 2 replaces PageRank with IoU localization, yet page 9 invokes undefined PageRank convergence and self-attention modulation. The incremental contribution needs clarification.

  2. VGG provenance and the deployment setting are unspecified. The source of inference-time boxes and attributes is unclear. Evaluation annotations imply an oracle setting; automatic prediction requires construction accuracy, error propagation, and total latency. Baselines need equivalent information. Appendix B checks training-text insertion only, leaving graph reliability and deployment without annotations unestablished.

  3. Reported quantities conflict with the mathematical definitions; implementation details and logs are needed.

    • Mean advantage. Equation (6) defines Am=(Rm−μR)/σRA_m=(R_m-\mu_R)/\sigma_R, giving M−1∑mAm=0M^{-1}\sum_m A_m=0 for complete, equally weighted groups with nonzero variance. Section 4.1 and Table 1 instead report E[Am]\mathbb{E}[A_m] rising from 0.10 to 0.78.

    • GRPO loss. Equation (7)'s clipped term is bounded above by Am+ϵ∣Am∣A_m+\epsilon|A_m|. Since M−1∑m∣Am∣≤1M^{-1}\sum_m|A_m|\le1, complete-group averaging and nonnegative KL imply LGRPO≥−ϵ=−0.2L_{\mathrm{GRPO}}\ge-\epsilon=-0.2. Table 1 reports -0.260 at step 400 and -0.755 at step 2,000. Different scaling or aggregation must be specified.

    • Probability normalization. Figure 1(b)'s PfinalP_{\mathrm{final}} values (0.680, 0.640, 0.140, 0.120) sum to 1.58. Yellow and green imply α=0.4\alpha=0.4; if red and ripe are the two supported tokens, normalized fusion gives 0.38 and 0.34. The displayed values match an unnormalized gate.

    • Factuality guarantees. Unsupported tokens can retain positive Pfinal(x)=αPbase(x)P_{\mathrm{final}}(x)=\alpha P_{\mathrm{base}}(x) for α>0\alpha>0. Allocating 60% of mass across supported tokens does not guarantee they win greedy selection. Section 3.3 overstates the guarantees.

  4. Hallucination evaluation and model coverage are insufficient. Equation (11) measures coverage without explicitly penalizing hallucinations, while Eq. (10) supplies target region descriptions. POPE comparisons are limited and do not clearly evaluate the complete method. CHAIR, AMBER, MMHal-Bench, and HallusionBench results are absent, and only Qwen2-VL-7B-Instruct is tested.

  5. Additional experimental details require clarification. Fluency intervals (±0.02 to ±0.04, Tables 5–6) exceed the normal-approximation 95% half-width bound of 0.0035 if all 1.26M queries are averaged; clarify the bootstrap procedure. The RefCOCO testA baseline (15.8%) versus Wang et al. (2024)'s 93.6% requires protocol and parsing clarification. Training datasets differ between Section 3.1 and Appendix B.2; specify the RefCOCO/+/g splits and deduplication.

Khandelwal et al. (2020). Generalization through Memorization: Nearest Neighbor Language Models. ICLR.

Liu et al. (2025). Visual-RFT: Visual Reinforcement Fine-Tuning. arXiv:2503.01785.

Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. arXiv:2409.12191.

Questions for authors
  1. Can the VGG construction procedure, annotation access, and information available to each baseline be specified?
  2. Can the metric/loss calculation code and training logs reconcile the advantage, loss, and probability-normalization discrepancies above?
  3. Can the complete decoder be evaluated with direct hallucination metrics, matched-information ablations, and an additional backbone?
Reviewer_xhZoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Topological Decoding, a method that combines information from object-specific Visual Grounding Graphs (VGGs) with a VLM’s next-token probabilities to improve the balance among faithfulness, linguistic fluency, and inference speed. The framework uses SFT and GRPO to train the model to output object bounding boxes, then uses their overlap with boxes stored in the VGGs to select information about the object currently being described. A parameter α controls the relative contributions of the model’s original generation probabilities and the VGG-based distribution. The paper evaluates object localization accuracy on RefCOCO and compares faithfulness, fluency, and inference latency across decoding methods on Visual Genome. It also examines how varying α affects the trade-off between faithfulness and fluency.

Critical strengths
  1. The parameter α provides explicit control over the balance between faithfulness and fluency. Experiments show how this balance changes as the weight of VGG information is adjusted.

  2. On Visual Genome, the method achieves a higher target entity match ratio than the base model with only a small increase in generation time.

Critical weaknesses
  1. The paper proposes directly incorporating factual information from VGGs into generation probabilities, but does not compare against a simple approach that provides the same list of facts in the prompt. The current results therefore make it difficult to determine whether the performance gains are specific to the proposed probability combination scheme or could also be obtained by providing the same information in another way. This makes it difficult to support the claim that the proposed method is more effective than approaches that provide external information in the input context.

  2. The primary faithfulness metric, Entity Match Ratio, measures the extent to which target entities are included in a response. However, it is unclear how additional incorrect objects or attributes in the response are penalized. This makes it difficult to distinguish between mentioning more target information and generating less incorrect content. Consequently, the current evaluation does not sufficiently support the claim that the proposed method reduces the generation of incorrect content beyond increasing the inclusion of target facts.

Questions for authors

In the Visual Genome evaluation, input questions are constructed by prefixing the dataset's image region descriptions with "Describe". To what extent do the objects and attributes already included in these descriptions overlap with the target entities used in the faithfulness evaluation? Please explain how the evaluation distinguishes between responses that simply repeat information provided in the question and those that incorporate additional facts from the image or VGG that are not present in the question.

Reviewer_vg4TOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Topological Decoding, an inference-time method for reducing attribute hallucination in VLMs. It has two parts.

  1. Box emission: Qwen2-VL-7B-Instruct is LoRA-adapted to emit a <box> for the object it is describing. This uses SFT on base-model responses with boxes inserted by Gemini, followed by GRPO with an IoU reward.
  2. Gated mixing. During decoding, the emitted box is matched by IoU (threshold τ) against a predefined set of Visual Grounding Graphs (VGGs), each an object box plus an attribute set. Tokens in the matched attribute set get a uniform share of probability mass (1−α), mixed with α·softmax of the model’s logits.

The paper motivates this as an analogue of combining semantic relevance with PageRank authority. It evaluates on ~1.26M Visual Genome region queries from images it says are disjoint from COCO train2014, and reports:

  • 72.8% entity-match faithfulness at α = 0.65, versus 52.1% (VCD), 56.4% (OPERA) and 59.2% (a gradient-based method)
  • Gemini-judged fluency of 4.35/5 and latency of 4.5 s, versus 4.2 s for the base model (Table 5)
  • an α sweep (Table 6)
  • 94.1% [email protected] on RefCOCO testA
  • unchanged MT-Bench, MMMU
Critical strengths
  1. The decoding rule is simple and cheap. Once a box and a matching graph are available, it is a single convex mixture per step. It adds about 0.3 s per query over the base model (Table 5), and the trade-off can be changed at inference without any retraining.
  2. The decoding rule is simple and cheap. Once a box and a matching graph are available, it is a single convex mixture per step. It adds about 0.3 s per query over the base model (Table 5), and the trade-off can be changed at inference without any retraining.
  3. The α sweep (Table 6, Fig. 3b) reports faithfulness and fluency together across the full range of the dial, with latency alongside. This is the right way to present a controllable decoding parameter, and it lets a user choose an operating point for their own application.
Critical weaknesses
  1. The method needs “the predefined set of Visual Grounding Graphs”. The paper never says where these graphs come from at inference. For the visual genome experiments, the only source i can see is vg's own object/attribute notations. Please address this in the paper explicitly.
  2. It is not stated what happens when several VGGs exceed τ. τ itself is only mentioned in passing. Please elucidate.
  3. Figure 1(b) does not follow Eqs. 3 and 9. The unboosted tokens (yellow 0.35 -> 0.14, green 0.3 -> 0.12) imply α = 0.4. Under Eq. 9 the two active tokens should then become 0.4 * 0.2 + 0.60.5 = 0.38 (red) and 0.40.1 + 0.6*0.5 = 0.34 (ripe). The figure instead shows 0.68 and 0.64, which is the full 0.6 added to each token without normalization, and the four displayed values sum to 1.58. The unnormalized rule is much stronger, so the authors should state which rule produced Tables 5–6 and ablate the two.
  4. Several parts of the decoding rule are left unspecified. When no attribute matches, Eq. 3 leaves total mass α < 1 with no stated renormalization. It is unclear whether the gate stays active at every step after the box is emitted; if it does, it would keep boosting attributes already generated, which could explain the “fragmented, repetitive” output at α = 0.2.
  5. Eq. 8 tests membership of a single vocabulary token, so multi-token attributes such as “bi-color” or “unripe” (Fig. 1a) are not handled, and ties among several VGGs above τ are not resolved. The uniform split gives each attribute only (1−α)/|O_i*|, so the claim in 3.3 that verified entities are guaranteed to dominate fails for objects with many attributes. Finally, the SFT data insert a box only after the first mention (3.1, step 4), so the model learns to emit one box per response. Section 1 (point 2), however, motivates the method by the model moving across objects during generation. Pseudocode for the full decoding loop would resolve most of this.
  6. 4.1 states that Qwen2-VL cannot natively output coordinates. But Qwen2-VL is trained for grounding. The training data are also unclear. Section 3.1 says RefCOCO only, while Appendix B.2 reports ~142,000 instances across RefCOCO, RefCOCO+ and RefCOCOg. RefCOCO by itself contains 142,209 expressions across all splits, including val and test, and RefCOCO+/g training images overlap RefCOCO evaluation images. The authors should evaluate the base model with its native grounding prompt and list the exact training splits.
  7. Faithfulness and fluency are both judged by Gemini-3.1-pro-preview, the same model that generated the SFT data, with no judge prompts and no human agreement check.
  8. Lee et al. (2025) is credited for the SFT+GRPO grounding methodology and for GRPO itself. GRPO was introduced by Shao et al. (DeepseekMath)
  9. Chen & Zhao (2025) is cited in 2.4 for SFT-based coordinate generation in VLMs, but the reference lists “Equivariant geometric bordisms and universal complexes”, a topology paper.
  10. Wang et al. (2025a), cited in 2.4 as a LoRA grounding hybrid, is listed as “Segment any leaf 3D,” a leaf segmentation paper.
  11. Jin et al. (2024) and Xie et al. (2024): same DOI and page range. Both entries (pp. 11 and 13) end with “pp. 16867–16878 and doi 10.63317/4fisde58hr4n
Questions for authors
  1. Where do the Visual Grounding Graphs come from at inference time? For the Visual Genome experiments, are they built from VG's own object and attribute annotations? If so, how would the method work on images that have no such annotations?
  2. Which update rule produced Tables 5–6? Was it the normalized mixture in Eq. 9 or the unnormalized boost shown in Fig. 1(b)? Can you ablate the two?
  3. Can you give pseudocode for the full decoding loop? It should cover:
    • how ties are broken when several VGGs exceed τ, and what value τ takes;
    • how probability mass is renormalized when no attribute matches;
    • whether the gate stays active after an attribute has been generated;
    • how multi-token attributes are handled;
    • how the model handles responses that describe more than one object.
  4. What are the exact training splits for RefCOCO, RefCOCO+ and RefCOCOg? Do any of them overlap the RefCOCO testA evaluation images? Also, how does base Qwen2-VL score with its native grounding prompt?
  5. Can you release the Gemini judge prompts and report agreement with human raters? Gemini also generated the SFT data, so judging with it may be biased.
1 comment

The authors did not respond during the discussion period, so my concerns about how the VGGs are built at inference time, the mismatch between Eq. 9 and Fig. 1(b), and the faulty references (including the one cited for the core SFT and GRPO method) remain, and I am lowering my rating to 1 (clear rejection)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.