acceptodds
Under review as a conference paper at ICLR 2027

Considering the Forest and Every Leaf: Various Interleaved Visual Inputs for Abstractive Analysis

Abstract

Abstractive visual reasoning (AVR) requires multi-modal large language models (MLLMs) to interpret human-defined visual representations, such as mind maps and relational diagrams. Existing AVR benchmarks primarily study individual abstract images, leaving open how models connect concrete visual evidence with abstract relational structures distributed across multiple images. We introduce Viviana, a benchmark for reasoning over sequences composed of entity images, visualized knowledge triples, and subgraph diagrams. Viviana contains 22,568 instances across 14 tasks, ranging from visual grounding to cross-image relational and graph reasoning. We also construct local-to-global chain-of-thought supervision and propose Gated Knowledge-informed GRPO (GKGRPO), which combines answer rewards with local and global knowledge rewards through an answer-conditioned, two-level gate. On Qwen3-VL models, GKGRPO achieves the highest overall score among the evaluated post-training methods. The resulting models also improve on established multi-image understanding and abstract visual reasoning benchmarks, indicating that structured cross-image supervision can transfer beyond the visual knowledge graph setting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-modal Knowledge Graphs, Multi-image Reasoning, Knowledge-informed Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LLywOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-image visual reasoning that combines concrete entity images with abstract relational diagrams. The authors introduce VIVIANA, which covers entity recognition, cross-image relational reasoning, and subgraph reasoning. The dataset is constructed through sampling and visualization of multimodal knowledge graphs, with a teacher model generating local-to-global CoT supervision. The authors further propose GKGRPO, which switches between two fixed coefficients for the combined local and global keyword reward depending on whether the final answer is fully correct.

Critical strengths
  1. The task design connects concrete visual recognition with abstract relational reasoning. VIVIANA organizes entity images, visualized triples, and subgraphs into tasks at different semantic granularities. This provides a platform for studying how local perception supports relational composition.
  2. The experiments cover multiple training methods and external benchmarks. The authors compare supervised fine-tuning, preference optimization, and several reinforcement learning methods at two model scales, and apply the reward design to different optimization algorithms. Results on external benchmarks show that training gains can transfer beyond knowledge graph images, demonstrating the practical value of the dataset and training approach.
Critical weaknesses
  1. The dataset’s value as a reliable evaluation and training resource is not sufficiently validated. VIVIANA is a central contribution, but the current presentation does not establish whether its tasks genuinely require cross-image visual reasoning (rather than textual dependency) or whether the synthesized intermediate reasoning is reliable. Complete examples, tests of dependence on textual inputs, and a clear assessment of CoT quality are missing.
  2. The independent value and robustness of the algorithmic contribution remain unclear. GKGRPO achieves small gains over the main RL baselines, but the paper does not report variability across repeated runs. The current evidence is therefore insufficient to establish a consistent advantage of the proposed gating mechanism over existing methods.
  3. The contributions of the reward components are not sufficiently isolated. The experiments do not separately evaluate the local and global knowledge rewards. They also lack a fixed-weight control and sensitivity analysis of the gating coefficients, making it difficult to distinguish the benefits of answer-conditioned gating from those of adjusting the knowledge reward weight.
Questions for authors
  1. Please provide representative examples containing the entity images, visualized triples or subgraphs, question, reference answer, and CoT. Please also clarify whether graph nodes contain entity names, images, or both.
  2. Some tasks may allow textual shortcuts. In Appendix A.2, the T4.2 prompt already provides the relational path in text, potentially allowing the model to answer using text and prior knowledge without integrating multiple images. Please clarify whether the actual inputs include this textual path and report performance without visual inputs.
  3. Section 4.2 defines the local and global reference keyword sets but does not explain how they are constructed. Please specify their sources and the construction rules for each task.
  4. Does the knowledge reward assess only keyword occurrence, without checking whether the resulting statements are correct? Statements with opposite meanings, such as “A is located in B” and “A is not located in B,” may contain the same keywords. Answer-conditioned gating cannot distinguish these reasoning traces when their final answers are identical. Please clarify how this limitation affects the validity of the reward as process supervision.
  5. Please explain the choice of (\beta_1=0.2) and (\beta_2=0.05) and provide a sensitivity analysis. Does answer-conditioned gating still provide consistent gains over a tuned fixed-weight setting where (\beta_1=\beta_2)? This comparison would help separate the contribution of gating from the effect of adjusting the knowledge reward weight.
Reviewer_uudfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce VIVIANA, a benchmark of 22,568 instances across 14 tasks combining entity photographs, visualized facts, and subgraphs. Further, they propose GKGRPO, which gates keyword-based knowledge rewards by answer correctness. Experiments on two Qwen3-VL model variants show improvements on VIVIANA and positive transfer to external benchmarks.

Critical strengths
  1. The paper addresses a relevant setting involving cross-image alignment between concrete entities and abstract relational structures.
  2. The task suite covers capabilities ranging from entity grounding to graph reasoning.
  3. Experiments include multiple post-training baselines, two model sizes, and external evaluations.
Critical weaknesses
  1. The paper provides task templates and a pipeline diagram, but NO examples showing the images, synthesized reasoning, and gold answer. This makes it difficult to evaluate cross-image evidence use and annotation quality.

  2. The knowledge reward is underspecified. Keyword construction is not explained, and substring recall cannot establish reasoning correctness. Intuitively, different valid reasoning paths may involve different entities, making a fixed reference keyword set potentially restrictive.

  3. Some templates expose the intended visual evidence in text. T4.2 explicitly provides the entity–relation path, while T2.2 supplies entity identities. Text-only and image-ablation baselines are needed to establish whether these tasks require visual reasoning.

  4. The contribution of gating is not isolated. GKGRPO changes both the conditioning mechanism and reward weights. Tuned constant-weight baselines and repeated-run statistics are needed, especially given the modest gains over KGRPO.

  5. Set-valued scoring may reward overprediction. The stated rules for T2.2 and T6.1 measure reference-triple recall without specifying penalties for false positives. My concern is that this could compromise evaluation and the gate’s interpretation of “fully correct.”

Questions for authors

Please see the weakness section above.

Reviewer_zLpGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on Abstractive Visual Reasoning (AVR) in Multimodal Large Language Models (MLLMs). The authors introduce VIVIANA, a comprehensive benchmark comprising 14 diverse tasks ranging from grounding to multi-hop graph reasoning. To train models on this structured multimodal data, the authors design a local-to-global Chain-of-Thought (CoT) supervision pipeline and propose GKGRPO (Gated Knowledge-informed GRPO). GKGRPO augments standard output-accuracy rewards with local and global knowledge-recall rewards. GKGRPO outperforms standard SFT and surpasses eight preference-optimization and RL baselines, while also showing positive transfer to external multi-image benchmarks (e.g., MMIU, MANTIS, and BLINK).

Critical strengths
  1. The paper expands the frontier of AVR benchmarks by moving beyond single-image charts to multi-image sequences. VIVIANA features 14 diverse tasks and over 89k image occurrences, providing a challenging and well-structured testbed for evaluating joint relational and perceptual reasoning in MLLMs.

  2. Incorporating intermediate reasoning rewards into RL without inducing reward hacking is a difficult challenge. The proposed answer-conditioned hard gate provides an effective way to enable stable RL optimization.

Critical weaknesses
  1. The novelty of the proposed GKGRPO method appears to lie mainly in introducing new rewards for specific tasks. Although this is valuable from an engineering perspective, the method largely lacks generalizable academic insight. Moreover, the evaluation of open-ended generation tasks relies exclusively on Qwen3.6 as an automated judge, posing a notable risk of evaluation bias.

  2. In Section 3.1, the authors acknowledge that the data are partitioned at the instance level (8:1:1), noting that "entities, source images, or triples may occur in more than one split." Consequently, the test set measures recombined reasoning over familiar entities and graph components rather than true zero-shot abstraction over unseen concepts or topologies. This design substantially undermines the benchmark's claim to measure AVR generalization.

Questions for authors
  1. Can the authors provide results using other models as judges?

  2. When evaluated under a strict zero-shot entity split (where entities and triples in the test set are completely disjoint from those in the training set), how much do GKGRPO and the baselines degrade? Does GKGRPO still retain its competitive margin?

Comments on the paper

1 comment

Dear reviewers,

The rebuttal period is ending soon. Please read the authors rebuttal and check if they addressed your concerns. Please acknowledge the rebuttal and adjust your scores accordingly.

Thanks.

-AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.