acceptodds
Withdrawn from ICLR 2027

Think in Graphs: Disentangled Graph-Centric Multimodal Reasoning via Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning has significantly improved reasoning capabilities in large language models, yet extending these advances to vision-language models remains challenging due to insufficient modeling of spatial and logical relations among visual entities. Existing methods rely on isolated visual entities and final-answer supervision, which causes reasoning to drift toward language priors rather than visual relational structures. We propose GRATS, a two-stage multimodal reasoning framework that consists of constructing relational graphs from images and performing graph-centric chain-of-thought reasoning over the generated graphs. We further introduce a novel RL method with stage-specific credit assignment to optimize the model's graph construction and graph reasoning capabilities simultaneously. Experiments on seven multimodal reasoning benchmarks show that GRATS consistently improves performance on both Physical Perception and Diagram & Quantitative Reasoning tasks. Code is available at: https://anonymous.4open.science/r/GRATS.

Primary area:
reinforcement learning
Keywords:
Structured Reasoning, Multimodal Reasoning, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DYC2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GRATS, a two-stage reinforcement learning framework for multimodal reasoning that first constructs an explicit relational graph from image-question inputs and then performs graph-centric chain-of-thought reasoning over the generated structure. Its main contribution is a stage-specific credit assignment scheme that separately computes graph-level and reasoning-level rewards and applies the corresponding advantages to graph-construction and reasoning tokens, with the goal of improving structured visual perception and reducing reasoning drift toward language priors. The method is evaluated with Qwen3-VL-4B/8B on seven multimodal reasoning benchmarks covering spatial perception, counting, chart understanding, mathematical reasoning, and comprehensive multimodal evaluation, where it consistently outperforms GRPO/DAPO-based baselines; the paper further provides stage-wise ablations, attention analyses, sensitivity studies, qualitative cases, and a computational cost breakdown to support the effectiveness of the proposed graph-centric training framework.

Critical strengths
  1. Clear and potentially useful decomposition of multimodal reasoning. GRATS explicitly separates relational graph construction from graph-centric reasoning and introduces stage-specific credit assignment for the two phases. This is a meaningful design choice because it targets a real limitation of outcome-only RL in multimodal settings: errors in visual structure extraction and errors in subsequent reasoning otherwise receive the same coarse supervision. The proposed formulation is conceptually simple and could be broadly useful beyond the specific benchmarks considered here.

  2. Strong and relatively broad empirical gains. The method is evaluated on seven benchmarks spanning spatial perception, counting, chart reasoning, mathematical reasoning, and general multimodal evaluation, and consistently outperforms GRPO/DAPO baselines under both 4B and 8B settings. The improvements are particularly large on several tasks, suggesting that the graph-centric representation and two-stage optimization provide substantial practical benefit rather than only marginal gains.

Critical weaknesses
  1. The paper does not directly validate whether the proposed joint training actually improves or preserves graph-construction quality. A central claim of GRATS is that stage-specific credit assignment improves both perception and reasoning. However, the current evidence for the perception stage is indirect: Figure 3 reports a combined training reward, Appendix E evaluates downstream QA accuracy, and Figure 4 analyzes attention patterns, but none directly measures the correctness of the generated relational graphs. This is particularly important because both stages share the same policy, so reasoning-stage optimization may also alter graph-generation behavior. Moreover, the graph-level reward itself depends substantially on downstream reasoning correctness and perplexity rather than direct graph supervision, with the leak-controlled reasoning score receiving 80% weight in the Phase-1 reward. Therefore, the current results establish that the generated graphs are useful for downstream answering, but do not establish that graph perception itself is improved or even preserved under joint two-stage optimization.

  2. The practical training efficiency of the two-stage framework is insufficiently characterized. GRATS samples multiple graphs per input and multiple reasoning trajectories per graph, while the reasoning reward uses a GPT-4o-based correctness score. The computational analysis emphasizes that the additional phase-wise advantage computation is negligible, but the reported runtime shows that Phase-1 and Phase-2 rollouts themselves dominate training time (596.69s and 543.81s per iteration, respectively). The paper does not report the latency or monetary cost of GPT-4o judging, nor whether this cost is included in Figure 5. As a result, the current analysis does not clearly quantify the end-to-end training overhead of GRATS relative to standard GRPO.

Questions for authors
  1. Clarification: Can the authors provide a direct evaluation of graph-construction quality, particularly comparing Stage-1-only training with the full joint Stage-1/Stage-2 training? Since the two stages share the same policy, this would clarify whether reasoning-stage optimization preserves, improves, or degrades the quality of the generated relational graphs.

  2. Clarification: Since the graph-level reward relies heavily on downstream reasoning correctness and perplexity rather than direct graph supervision, how do the authors ensure that a graph with high reward is actually visually and relationally correct, rather than merely sufficient for producing the correct final answer?

  3. Clarification: How is GPT-4o-based reward computation implemented during training? In particular, how many judge evaluations are required, are requests batched/cached/asynchronous, and is GPT-4o latency included in the runtime reported in Figure 5? A total end-to-end training cost comparison with standard GRPO would materially clarify the practical overhead of the method.

  4. Clarification: Training data scale and train/evaluation separation. The training set contains only approximately 8K unique examples, sampled from the same seven benchmark families used for evaluation, while GRATS achieves substantial gains on several datasets. Could the authors clarify the exact training/evaluation split for each benchmark, especially VSR, which is described as being used for both training and evaluation, and confirm whether the training and evaluation instances are strictly disjoint at both the sample and image levels? In addition, have the authors examined performance as a function of training-set size (e.g., 25%, 50%, 75%, and 100% of the training data)? Such a scaling analysis would help determine whether the reported gains consistently arise from the proposed training framework or are sensitive to the particular small, in-domain training set.

  5. Clarification: Could the authors clarify the apparent inconsistency between Table 1 and Appendix Table 2? Appendix Table 2 is labeled as Qwen3-VL-4B, but its full-GRATS results on TallyQA/ChartQA/MathVista (75.0/77.9/58.8) match the 8B results in Table 1 rather than the reported 4B results (68.8/76.7/58.0).

Reviewer_J5nsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GRATS, a multimodal reasoning framework that constructs a question-conditioned relational graph and then generates graph-based reasoning trajectories. It separately computes group-relative advantages for graph construction and reasoning, assigning each signal to the corresponding generation stage. Experiments across seven benchmarks and two model sizes show improvements over the reported baselines.

Critical strengths
  1. The paper focues on a relevant limitation of multimodal reasoning: models may rely on language priors rather than visual relations. Explicit relational representations provide a reasonable basis for addressing this issue.

  2. The nested sampling and stage-specific advantage estimation are promising. Comparing reasoning trajectories under a fixed graph reduces variation due to graph quality, while aggregating downstream performance provides a task-oriented learning signal for graph construction.

  3. The evaluation covers multiple task categories and model sizes. The stage-wise ablations support the usefulness of both components within the proposed framework.

Critical weaknesses
  1. The benefit of graph structure over structured CoT remains unclear. From my perspetive, the claimed question-conditioned graphs just simply encode an answer-oriented reasoning trace. Also, for each VQA, there are diversed graph representation to get the final answer.

  2. The reward design is insufficiently specified and validated (Eq. (4)–(7)). The perplexity ablation does not rule out language-prior bias, while its reward transformation, normalization, and leakage detection rules remain unclear.

  3. The novelty relative to Scene Graph Thinking (SaGe) [1] is not adequately discussed. A controlled comparison is needed to establish the contribution of stage-specific credit assignment (Eq (8)–(12)) beyond existing graph-based visual reasoning.

  4. The evaluation relies mainly on relatively old benchmarks and final-answer accuracy. Newer benchmarks (e.g., MMStar for comprehensive capbility and V*/HR Bench for fine-grained perception) and targeted tests of graph fidelity and traversal would better support the claims, while matched-budget ablations would separate structural benefits from additional inference scaling.

Reference: [1] Yang et al., Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models, ICML 2026

Questions for authors
  1. It would be helpful to clarify the benefit of the graph representation beyond structured CoT. Matched-budget comparisons with relational text or structured CoT, together with tests of alternative-path exploration and graph reuse, would strengthen this claim.

  2. Please consider providing the complete reward definitions, including the perplexity transformation, normalization, and leakage detection rules. Further analysis of language-prior bias and the leakage detector’s reliability would help support the reward design.

  3. A discussion of the differences from Scene Graph Thinking (SaGe) would help clarify the contribution. A controlled comparison with a shared-advantage objective would also provide stronger evidence for the benefit of stage-specific credit assignment.

  4. Please consider extending the evaluation to newer benchmarks, such as MMStar and V*/HRBench, and adding direct assessments of graph fidelity and traversal.

Reviewer_1vSMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GRATS, a two-stage framework for graph-centric multimodal reasoning. In the first stage, the model constructs a relational graph from the input image. In the second stage, it performs step-by-step reasoning over the generated graph using predefined graph operations. The authors further introduce a reinforcement learning strategy that samples K candidate graphs and M reasoning trajectories per graph, computes separate advantages for graph construction and reasoning, and assigns them to the corresponding tokens during policy optimization.

Experiments on seven multimodal reasoning benchmarks using Qwen3-VL-4B and 8B show improvements over several baselines, including GRPO and DAPO.

Critical strengths

1. A reasonable approach to stage-specific credit assignment.

I find the hierarchical sampling and advantage estimation strategy to be the most interesting part of the paper. Computing graph-level advantages across candidate graphs and reasoning-level advantages across trajectories conditioned on the same graph is a reasonable way to provide more targeted learning signals than standard outcome-based RL. Although this does not fully disentangle perception and reasoning errors, the idea is intuitive and potentially useful.

2. Consistent improvements over the evaluated baselines.

The paper reports improvements across multiple benchmarks and model scales. The authors also include stage-wise ablations and some analysis of training dynamics. These results are encouraging, although I have concerns about the evaluation settings, as discussed below.

Critical weaknesses

1. Limited novelty and insufficient differentiation from existing work.

My main concern is the limited novelty of the proposed method. The task itself is not new, and most of the components, including scene graph construction, graph-centric chain-of-thought reasoning, and reinforcement learning for structured visual reasoning, have been explored in previous work.

In particular, recent works such as SaGe (Scene Graph Thinking), SpatialThinker, and R1-SGG have investigated related directions involving scene graphs and reinforcement learning. MIRL also studies the separation of perception and reasoning during RL optimization. I do not think the paper sufficiently discusses how its approach differs from these existing methods, especially SaGe and MIRL.

The most distinctive contribution seems to be the hierarchical grouping strategy and stage-specific advantage assignment. While I consider this a reasonable design, the current experiments do not clearly demonstrate how much improvement comes specifically from this component, rather than from the use of structured graph representations or the additional reasoning process.

I would like to see stronger comparisons or ablations that isolate this contribution. Overall, I find the methodological novelty somewhat limited for ICLR.

2. Insufficient evaluation of graph correctness and the claimed disentanglement, together with concerns about experimental reliability.

The paper aims to disentangle graph construction from graph reasoning, but I do not think the current reward design and evaluation fully support this claim.

First, the graph-level reward includes structural and formatting checks, but it does not explicitly verify whether the predicted relations between nodes are correct with respect to the image. Even if the nodes are correctly identified, the edges may still represent incorrect spatial or logical relationships. Moreover, the graph-level reward relies heavily on downstream answer correctness, which does not necessarily reflect the correctness of the constructed graph.

Second, while the authors provide stage-wise ablations, they do not directly evaluate graph construction accuracy and reasoning correctness separately. Most of the evidence is based on final-answer accuracy. As a result, it is difficult to determine whether improvements come from better visual relation modeling, better reasoning over the graph, or other factors. The claimed disentanglement and error attribution are therefore not sufficiently demonstrated.

I think graph intervention experiments would be particularly useful here. For example, modifying or removing relevant edges in the generated graph and measuring the effect on the final answer could help determine whether the model actually follows the graph during reasoning.

There are also some concerns about the experimental setup that require clarification:

Training/test splits: Appendix A.2 states that training samples are drawn from the seven evaluated benchmarks, including MathVista and MathVision. However, the exact splits and the separation between training and evaluation samples are not sufficiently clear. The authors should clarify this to rule out potential data overlap.

Baseline performance: The reported performance of the original Qwen3-VL models is substantially lower than the official results on benchmarks such as MathVista and MathVision. It is unclear whether this is due to different evaluation subsets, prompting strategies, generation settings, or other factors. These differences need to be explained before the reported improvements can be properly assessed.

These issues make it difficult to fully evaluate the effectiveness of the proposed method.

Questions for authors

1. Novelty and related work: Could the authors clarify the main technical differences between GRATS and recent methods such as SaGe and MIRL? In particular, what is the advantage of the proposed hierarchical grouping and stage-specific advantage assignment over existing structured reasoning and decoupled RL approaches?

2. Relation correctness: How does the graph-level reward ensure that the generated edges accurately reflect visual relations in the image? Could the authors provide a direct evaluation of graph correctness, particularly relation accuracy, rather than relying mainly on formatting checks and downstream answer correctness?

3. Disentanglement and graph intervention: Could the authors evaluate graph construction and graph reasoning more independently? I would particularly like to see graph intervention experiments, such as removing, modifying, or replacing relations, to test whether the final reasoning actually depends on the generated graph. An ablation comparing the proposed stage-specific advantages with standard GRPO under the same graph-generation format and sampling budget would also help isolate the contribution.

4. Training and evaluation splits: Could the authors specify the exact training and evaluation splits for each benchmark and confirm that there is no overlap? This is especially important for MathVista, MathVision, and MME.

5. Baseline results and reporting: Could the authors explain why the reported Qwen3-VL baseline scores on MathVista and MathVision are substantially lower than the official results? Please clarify the evaluation subsets, prompts, generation settings, and scoring protocols. Also, Table 2 is labeled as a Qwen3-VL-4B ablation, but its full-model results appear to match the 8B results in Table 1. Could the authors verify this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.