acceptodds
Withdrawn from ICLR 2027

H-GRPO: PERMUTATION-INVARIANT REINFORCE- MENT LEARNING FOR IMPROVING SPATIAL GROUNDED REASONING

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) have achieved strong performance on visual reasoning tasks, yet a correct final answer does not necessarily imply that the model followed a correct or visually grounded reasoning process. We study how VLMs can be trained to produce explicit intermediate reasoning steps that are both semantically meaningful and grounded in visual evidence. We represent each reasoning step as a structured triplet consisting of a sub-question, sub-answer, and supporting bounding box, enabling the reasoning process to be evaluated along both semantic and spatial dimensions. To optimize this representation, we introduce Hungarian-GRPO (H-GRPO), a reinforcement-learning framework that aligns predicted and reference reasoning steps using permutation-invariant Hungarian matching, avoiding the assumption that valid reasoning steps must follow a fixed ordering. Experiments show that optimizing grounded reasoning improves both intermediate reasoning quality and answer accuracy, with H-GRPO variant achieving the strongest overall grounding and decomposition performance. Our analyses show that grounding quality and answer correctness are related but distinct, while counterfactual evidence masking confirms that the localized regions contain information important to the model's predictions. These findings motivate evaluating VLMs beyond answer accuracy and demonstrate the value of explicit, spatially grounded reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, grounded spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Po71OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces H-GRPO, a reinforcement learning framework for improving spatially grounded reasoning in vision-language models. Instead of optimizing only final-answer correctness, the method represents intermediate reasoning as structured triplets consisting of a sub-question, sub-answer, and supporting bounding box, allowing both semantic reasoning quality and visual grounding to be evaluated. Its main technical contribution is a permutation-invariant Hungarian matching objective that aligns predicted reasoning steps with reference steps without assuming a fixed step order; the paper also introduces H-GRPO+, which first uses fixed-order supervision and then applies permutation-invariant optimization.

The authors construct a 10,000-example synthetic grounded reasoning dataset for SFT and RL training, and evaluate the approach on RoboSpatial, CV-Bench, and EmbSpatial-Bench. Results show consistent improvements in intermediate grounding and decomposition quality, with competitive or improved answer accuracy on most benchmarks. The paper further analyzes the relationship between grounding quality and answer correctness and performs evidence-masking interventions, showing substantial accuracy drops when predicted evidence regions are removed.

Critical strengths
  • Well-motivated structured reasoning formulation. The paper goes beyond final-answer supervision by explicitly representing each intermediate step with a sub-question, sub-answer, and supporting bounding box. This makes the reasoning process more inspectable and allows semantic and spatial grounding quality to be evaluated directly.

  • Useful permutation-invariant optimization and thorough process-level evaluation. H-GRPO uses Hungarian matching to avoid penalizing valid reasoning decompositions solely because their step order differs from the reference. The experiments also evaluate grounding quality and evidence dependence in addition to answer accuracy, including counterfactual masking of predicted evidence regions.

Critical weaknesses
  • The current objective establishes reference alignment, but does not sufficiently establish faithful perception-to-reasoning integration. H-GRPO rewards semantic and spatial agreement between generated steps and reference triplets, but a high matching score does not necessarily demonstrate that the localized visual evidence is faithfully internalized and actually used in subsequent reasoning. This distinction has been explicitly addressed in recent work. Guo et al. [1] explicitly decouple visual interpretation from downstream reasoning; CogFlow [2] introduces a dedicated knowledge-internalization stage and visual-gated policy optimization; EVICT [3] further distinguishes cited evidence from decision-relevant evidence through counterfactual evidence dropout; and MathFlow [4] similarly separates perceptual information extraction from subsequent reasoning to make the role of visual evidence more explicit. Although this paper includes an evidence-masking experiment, it remains an aggregate post-hoc intervention and does not verify whether the evidence associated with each intermediate step is causally used by subsequent reasoning.

  • The reliability of the synthetic process supervision is insufficiently established. GVRS reasoning traces are generated conditionally on the ground-truth final answer, making them potentially post-hoc rationales rather than independently derived reasoning trajectories. Moreover, only 100 examples out of the 10,000 synthetic samples are manually inspected. This concern is especially relevant given that prior work such as Bootstrapping Grounded Chain-of-Thought [5] explicitly addresses factual errors in synthetic CoT supervision through grounding-based verification. Since H-GRPO directly optimizes similarity to these synthetic references, errors or stylistic artifacts in the references may be reinforced during training.

  • The permutation-invariance assumption may be too strong for genuinely sequential reasoning. Hungarian matching treats alternative step orders as equivalent, but some reasoning steps have explicit dependencies and should not be freely reordered. This is also somewhat inconsistent with the paper’s own generation prompt, which requires the sub-questions to form a logical and sequential progression. A controlled step-order perturbation experiment would help clarify when permutation invariance is beneficial.

  • The semantic reward may be insensitive to critical spatial errors. The method uses Sentence-BERT cosine similarity for sub-question and sub-answer matching. However, spatially opposite statements such as “left” vs. “right” or “above” vs. “below” can remain highly similar in embedding space. The paper does not provide hard-negative analysis to show that the reward reliably distinguishes such relation reversals.

[1] GUO Z, LIU M, WANG Q, et al. Integrating Visual Interpretation and Linguistic Reasoning for Geometric Problem Solving[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025: 3988-3998.

[2] CHEN S, XU Y, XIE J, et al. CogFlow: Bridging Perception and Reasoning through Knowledge Internalization for Visual Mathematical Problem Solving[C]//The Fourteenth International Conference on Learning Representations. 2026.

[3] KOTTE V. EVICT: Evidence-Sufficiency Verification via Counterfactual Dropout for Visually-Grounded Selective Question Answering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2026: 11251-11259.

[4] CHEN S, YUAN H, XU Y, et al. MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics. 2026: 967-992.

[5] XIA J, TONG B, ZANG Y, et al. Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025: 208-217.

Questions for authors
  • How were the reference reasoning triplets for RoboSpatial, CV-Bench, and EmbSpatial-Bench constructed and validated?

  • Can the authors report the human evaluation results and inter-annotator agreement for the 100 manually checked GVRS samples?

  • How robust is the Sentence-BERT reward to critical spatial relation reversals such as left/right and above/below? For example, do pairs such as “left of” vs. “right of” or “above” vs. “below” receive sufficiently different similarity scores? A small hard-negative analysis would help determine whether the semantic reward can reliably distinguish the errors that matter most for these benchmarks.

  • Can permutation-invariant matching assign high reward to reasoning steps that are individually correct but ordered incoherently?

  • Can the authors add random or area-matched masking controls to verify that the evidence-masking performance drop is specifically caused by removing decision-relevant regions?

Reviewer_u1aKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose H-GRPO, a reinforcement learning post-training framework for spatially grounded reasoning in Vision-Language Models (VLMs). To address ungrounded intermediate reasoning caused by optimizing solely for final answers, each step is formulated as a structured triplet of sub-question, sub-answer, and evidence bounding box. H-GRPO employs bipartite Hungarian matching to align predicted and reference steps for permutation-invariant process rewards, alongside a two-stage variant (H-GRPO+) with fixed-order warm-up. Experiments across out-of-distribution spatial benchmarks validate its effectiveness.

Critical strengths
  1. Formulating visual spatial reasoning into explicit "sub-question, sub-answer, bounding box" triplets and applying bipartite Hungarian matching to compute process rewards directly addresses ungrounded intermediate steps. This design relaxes the strict sequential assumptions of traditional supervision, avoiding false penalties on valid permutations of equivalent reasoning steps.

  2. The evaluation extends beyond final answer accuracy by establishing a multi-dimensional protocol covering spatial grounding (IoU), semantic similarity, sample-level correlation, and counterfactual occlusion. These empirical analyses convincingly reveal that intermediate visual grounding quality correlates with, yet is not equivalent to, final answer correctness.

  3. The authors introduce a scalable synthetic data pipeline (GVRS) that combines multi-model prompt selection with SAM3 bounding box refinement to curate 10,000 spatial reasoning trajectories. This dataset provides a practical supervision resource and evaluation testbed for fine-grained multimodal process supervision in future research.

Critical weaknesses

Major weakness:

  1. The proposed method lacks sufficient novelty. Compared to topical visually grounded reasoning baselines (e.g., VGR, TreeVGR), it introduces no substantive methodological innovation, essentially applying a similar paradigm to solve an analogous problem.

  2. In Eq. (4), multiplying R_answer directly by the truncated R_HS = max(0, S_HS - gamma) without a standalone R_answer term lacks theoretical variance or convergence guarantees. Valid reasoning paths with different granularity than the synthetic reference (S_HS <= gamma) receive zero task reward, likely causing H-GRPO+ (67.82%) to underperform H-GRPO-NG (72.23%) and GRPO-QA (69.38%) on RoboSpatial.

  3. Treating reasoning steps as an unordered set in Eqs. (2)-(3) contradicts the causal dependencies of multi-step reasoning, especially since Appendix I explicitly prompts for a sequential coarse-to-fine progression. Ignoring directed acyclic graph (DAG) constraints, omitting semantic supervision on the final reasoning text r, and collapsing scores into a single scalar prevent fine-grained credit assignment.

  4. The construction and verification of ground-truth reference triplets D* for calculating test IoU and semantic similarity in Tables 1-4 on QA-only benchmarks (RoboSpatial, CV-Bench, EmbSpatial-Bench) are undocumented. Moreover, counterfactual occlusion lacks random-area masking and SFT/GRPO-QA baselines, while Appendix B.1/D omits the actual scores and inter-annotator agreement from the 3-expert human evaluation.

Minor weakness:

  1. Experimental configurations and statistical evaluations are incomplete: the gradient accumulation steps for the GRPO stage are left blank in Appendix Table 6, the exact training step split between the fixed-order warm-up and Hungarian matching stages in H-GRPO+ is unspecified.

  2. Visual presentation need improvement: the pie chart in Appendix Fig. 2 is too low-resolution to read numbers on dark sectors, and high-saturation red blocks in Fig. 7 obscure key failure text.

Questions for authors
  1. Why use a purely multiplicative form (R_answer * R_HS) in Eq. (4) instead of retaining a base correctness term like R_answer * (1 + beta * R_HS)? Please provide ablations verifying if zeroing rewards when S_HS <= 0.5 causes H-GRPO+ to trail H-GRPO-NG and GRPO-QA on RoboSpatial.

  2. How were the test-set reference triplets D* in Tables 1-4 constructed and verified, and could the gains merely reflect fitting to the synthetic pipeline? Also, what are the accuracy drops under equal-area random masking and predicted-box masking on GRPO-QA?

  3. Please provide the missing gradient accumulation steps in Table 6, the exact step ratio and training time comparison between the two stages of H-GRPO+, and the quantitative statistics (mean, std, and inter-annotator agreement) for the 100-sample human evaluation in Appendix D.

Reviewer_Cob2OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces H-GRPO, a reinforcement learning framework for spatially grounded visual reasoning that represents intermediate steps as sub-question, sub-answer, and evidence-box triplets. Its main contribution is a permutation-invariant process reward that uses Hungarian matching to align predicted and reference steps based on semantic similarity and spatial overlap, combined with final-answer correctness. The authors construct a synthetic grounded reasoning dataset and evaluate on RoboSpatial, CV-Bench, and EmbSpatial-Bench, reporting improved intermediate grounding and generally better answer accuracy over answer-only GRPO. Ablations and evidence-masking experiments further examine the roles of step matching, grounding supervision, and the relevance of predicted evidence regions.

Critical strengths
  • The idea is reasonable.
  • The writing is well and this paper is easy to follow.
Critical weaknesses
  • The benefit of Hungarian matching is not convincingly isolated. All RL variants start from SFT on traces that explicitly encourage sequential reasoning, potentially reducing the ordering variation that motivates the method. The paper does not quantify how often Hungarian matching differs from identity matching or demonstrate that these differences correspond to valid alternative orderings. Its accuracy gains over fixed matching are small and inconsistent (+0.53, −0.04, and +0.66 percentage points across the three benchmarks), without multi-seed uncertainty estimates.

  • Permutation invariance does not fully address alternative reasoning paths. Hungarian matching accommodates reordered steps, but valid decompositions may also differ in granularity, intermediate conclusions, or supporting evidence, which one-to-one reference matching can still penalize. Conversely, some reasoning steps have genuine dependencies that should not be order-invariant. The paper should clarify the scope of its formulation and distinguish matching an unordered evidence set from evaluating a logically coherent reasoning chain.

  • The process metrics provide limited independent evidence of improved reasoning. Evaluation largely reuses the semantic-similarity and spatial-overlap criteria optimized during training. Higher scores may therefore reflect closer reference alignment rather than more correct or logically valid reasoning. In particular, Sentence-BERT similarity is not validated for distinguishing contradictory spatial statements, and the reward does not explicitly verify consistency between the intermediate triplets, synthesized reasoning path, and final answer.

  • Improvements in grounding do not consistently translate into better task performance. On RoboSpatial, H-GRPO+ achieves 67.82% accuracy, below both answer-only GRPO (69.38%) and fixed matching (70.28%), while removing the grounding reward yields the highest accuracy (72.23%). Although the paper acknowledges this trade-off, it remains unclear what substantive reasoning benefit the improved process metrics capture when answer accuracy deteriorates.

  • The evaluation provides limited coverage of the central methodological claim and comparison with prior work. The three benchmarks primarily concern static, object-centric spatial reasoning, without targeted tests of valid step reorderings or alternative decompositions. Moreover, comparisons with existing grounded reasoning and RL methods are largely restricted to reported RoboSpatial results rather than controlled experiments under a shared protocol, making the contribution beyond prior grounded RL approaches difficult to assess.

  • TreeVGR [R1] actually supervises intermediate reasoning paths using IoU rewards. It at least should be added to the discussion.

  • The citation format seems wrong.

References

[R1] Traceable evidence enhanced visual grounded reasoning: Evaluation and method. ICLR 2026.

Questions for authors

Please refer to weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.