- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Summary
The paper introduces VBVR-Pro, a large-scale dataset and benchmark designed to improve native visual reasoning through image, video, and interleaved text-image generation, alongside verifiable reward scorers and RL experiments. While the scale of the dataset and the reported numbers are impressive, I have significant concerns regarding the paper's novelty, framing, and experimental validity. In particular, the paper does not sufficiently distinguish native visual reasoning from existing approaches such as Thinking with Images and tool-assisted VLM reasoning, despite making broad claims about visual reasoning. Much of the strongest experimental evidence is also concentrated on video generation models, with limited validation of generalization to other model types. Furthermore, the paper lacks controlled comparisons to establish whether the improvements arise from increased task diversity, better data quality, or simply additional training data. Several central claims, particularly regarding the importance of intermediate visual states over language-based reasoning, are also insufficiently supported by the experiments. Overall, while I acknowledge the substantial engineering effort and promising results, I am not convinced that the paper adequately establishes its novelty over existing work or provides sufficient experimental evidence to justify its broader claims.
Confusing Framing and Contributions
The first issue that comes to mind is that the paper does not actually establish VBVR-Pro as a novel or better method for improving visual reasoning. The entire paper frames itself as a novel dataset for enhancing native visual reasoning, but does not mention improving native visual reasoning on what kind of models in the abstract, and discusses weaknesses of Large Language Models and how they fail to be effective in the physical world. The main problem is that the paper tries to make broad and general claims about things like "our method improves visual reasoning" and "intermediate visual states matter more than language-based reasoning", but throughout most of the paper it tackles models that were designed for video and image generation / editing, or a few specially trained unified models (e.g. ThinkMorph). It seems to mix together a couple of domains and topics (e.g. video reasoning, unified model reasoning) and market it as visual reasoning. Combining them does make the paper look like it has lots of contributions but the real issue is whether they are genuine contributions. E.g. modest contributions in the VLM space + modest contributions in the Video Generation space does not equate to an overall impressive contribution. I will try my best to evaluate it under a VLM context, and under a Video Reasoning Context.
VLMs and Native Visual Reasoning
As of the moment, I don't think it's appropriate to frame it as "a scalable and verifiable suite for native visual reasoning". First of all, native visual reasoning is very ill-defined. What does it mean by "native"? The abstract states "Native visual reasoning treats visual states (i.e., images and videos) not as merely inputs to be understood or outputs to be rendered, but the intermediate steps through which a problem is solved." If the main point of native visual reasoning is that there are intermediate steps, most of the new VLMs can interleave images, but it's not established at all that it's necessary or beneficial for them to do so. Most of the new VLMs are also pretrained with multimodal data, so they aren't less "multimodal" than any other generation models. So I find it unjustified to leave out normal VLMs like Gemma4 or Qwen3.5. The authors may rebut by saying that native visual reasoning requires models with image/video output capabilities, so normal VLMs do not qualify. But the problem is that in order to make this point, you must first prove that this restriction / qualification is necessary. The main problem is whether native visual reasoning under the VLM setting is a useful setting at all, and if you can demonstrate that it outperforms "normal" visual reasoning in the same environment (e.g. same compute budget, same base models). Improvements gained from interleaving visual information in VLM reasoning is proven and 2-3 years old, unless the authors can demonstrate that native visual reasoning is not a rebrand of the same method, then it's not appropriate to frame native visual reasoning as a new paradigm. The datasets themselves aren't novel for VLMs either, there are dozens of papers on VLM reasoning in mazes (Figure 4) and VLM reasoning with geometry and 3d shapes (Figure 5). I've seen tasks like this 2 years ago. Throughout the paper I don't see any tasks that are "novel" or have not been introduced before as a VLM training task / benchmark. It also doesn't help that there isn't a catalog of all 300 tasks anywhere for me to inspect novelty in the task design, but that there is a 249MB prep video in the supplementary with things mentioned in the paper. I do not think there is any novelty to claim here, under VLM and general Visual Reasoning context.
Video Reasoning
As established, there lacks novelty and contribution in the VLM setting, which in fact is also a small portion of the paper. Alot of the main evaluations occur for video reasoning and video generation models. If the authors were to refer to Table 4, all of the other datasets and benchmarks they compared against explicitly frame it as "video reasoning", which is significantly different from visual reasoning. I recommend the authors make significant revisions to make sure that their contribution is explicitly tailored towards video reasoning, because if the paper were to be evaluated as a visual reasoning paper, it has little novelty.
Conclusion
The paper attempts to bundle together a couple of very different tasks that usually do not share one paper in order to make it appear to target a very general contribution. Fundamentally, VBVR-Pro attempts to propose "native visual reasoning" without clearly defining it or show why it isn't a rebrand of Think with Images and does not extent to general VLMs. It is more akin to a video reasoning dataset / benchmark that just happens to be usable for VLMs, Image-Generation Models, and Unified Models, but does not demonstrate useful novelty in either. Finally, many of the concepts such as verifiable rewards and weaknesses in VLM-as-a-judge were already present in previous papers the authors cited (VBVR - icml), and claims such as gains resulting from training on the dataset transfers to other domains are mere minimal requirements for a dataset.
Critical strengths
- There are 300 tasks and the dataset is very big.
- The authors propose an accurate verifier instead of a VLM judge.
Critical weaknesses
-
Native Visual Reasoning is not well defined, and it's not established that this is a genuinely different paradigm than thinking with images, visual chain-of-thought, and tool-assisted VLM reasoning. This is crucial if the main contribution of the paper is a dataset / benchmark that is tailored for native visual reasoning. The paper does not convincingly explain what fundamentally differentiates its approach from these methods or why generating intermediate visual states directly should be preferred over alternative approaches.
-
Insufficient evidence that intermediate visual states are responsible It's shown in Appendix D.2 reports that 209 of the 300 tasks (69.7%) use the Last-Frame image-output regime. Only 91 tasks use Key-Frame or Multi-Frame outputs. This further weakens the native visual reasoning claim.
Questions for authors
-
Can you explicitly define what Native Visual Reasoning is and why it only works for Unified Models and not VLMs? How does generated visual states from a unified model stored in the context differ from a ground-truth visual state stored in the context from VLM tool-use? (e.g. Thinking with Images). VLM with tool use and ground-truth intermediate visual states is more efficient than Unified Models and likely better. Unless authors can prove otherwise, I believe it's not appropriate to market this benchmark as something tailored for "native visual reasoning". A benchmark and dataset for an unjustified setting defeats its purpose. I am willing to raise the score if this issue can be resolved in either manner.
-
Can you provide the full catalog of tasks?
-
I would like to see a controlled comparison between VBVR and VBVR-Pro training for Wan2.2 (Table4), the current comparison is unfair if you are training it with different amounts of data.
-
Can you address the other weaknesses in the Critical Weakness section?
11 comments
Response Part 2
2. Native Visual Reasoning is an emerging but already established research direction
This research setting is distinguished by its computational and output representations, rather than by an arbitrary distinction between model families. Importantly, Native Visual Reasoning is an emerging but already established research direction, not an unjustified paradigm introduced by this paper. This direction can be traced to Google DeepMind's Video Models Are Zero-Shot Learners and Reasoners (Wiedemer et al., September 2025), which demonstrated that video generation models such as Veo 3 exhibit emergent zero-shot capabilities on visual reasoning tasks, including maze solving and symmetry transformations. Since then, a growing literature has investigated different dimensions of this research direction:
Benchmarks: MME-CoF (Guo et al., 2025), V-ReasonBench (Luo et al., 2025), RULER-Bench (He et al., 2025), VideoThinkBench (Tong et al., CVPR 2026), and MME-CoF-Pro (Qi et al., 2026) systematically evaluate visual generative reasoning across spatial, logical, physical, temporal, and planning tasks.
Training data and scalable learning: VBVR (Wang et al., ICML 2026) established large-scale procedural training for video generative reasoning, while OpenCoF (Chen et al., 2026) further investigated learning Chain-of-Frames reasoning from dedicated visual trajectory supervision.
Generative reasoning methods: DiffThinker (He et al., ICML 2026), and Hierarchical Denoising for Multi-Step Visual Reasoning (Qian et al., 2026) investigate approaches to improving visual generative reasoning through visual-state planning, progressive visual transformations, or structured denoising.
Theoretical and mechanistic understanding: Recent studies provide compelling evidence that video generators exhibit reasoning mechanisms fundamentally distinct from language models. A few works (Demystifying Video Reasoning (Wang et al., 2026), Video Models Reason Early (Newman et al., 2026), DiffThinker (He et al., 2026)) reveal that video reasoning emerges through a Chain-of-Steps mechanism, where models explore and progressively refine candidate solutions during diffusion denoising, rather than through sequential linguistic reasoning.
- Wiedemer et al. (2025). Video Models Are Zero-Shot Learners and Reasoners. arXiv:2509.20328.
- Guo et al. (2026). Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark. CVPR Findings 2026.
- Luo et al. (2025). V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models. arXiv:2511.16668.
- He et al. (2025). RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligence. arXiv:2512.02622.
- Tong et al. (2026). Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm. CVPR 2026.
- Qi et al. (2026). MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hints. ECCV 2026.
- Wang et al. (2026). A Very Big Video Reasoning Suite. ICML 2026.
- Chen et al. (2026). OpenCoF: Learning to Reason Through Video Generation. NeurIPS 2026.
- He et al. (2026). DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models. ICML 2026.
- Qian et al. (2026). Hierarchical Denoising for Multi-Step Visual Reasoning. NeurIPS 2026.
- Wang et al. (2026). Demystifying Video Reasoning. ECCV 2026.
- Newman et al. (2026). Video Models Reason Early: Exploiting Plan Commitment for Maze Solving. arXiv:2603.30043.
3. Distinction from VLM-based "video reasoning": reasoning on video (where video is the input) versus reasoning by generating video (where video is the reasoning medium, output, and the subject of evaluation).
In VLM-based video reasoning, video serves primarily as the input modality. The model processes the input video to extract relevant spatial and temporal cues, but performs reasoning and produces its final answer in the language domain.
By contrast, in native visual reasoning through video generation, it directly constructs a solution by generating task-consistent visual states (i.e., images and videos), without relying on textual reasoning to derive or express the answer. For example, in the “Key–Door Navigation” task (this link), the model must generate a video showing an agent navigating through a maze, collecting the required keys, and reaching the destination. Success is evaluated directly from the generated video, based on whether the agent collects the required keys, reaches the destination, and follows a valid, continuous path without crossing walls. Therefore, the generated video is the solution itself, with its correctness directly evaluated based on the generated visual states and their transitions.
Response Part 3
4. Distinction from VLM-based "Thinking with Images": the roles of text and visual states are fundamentally different.
We respectfully disagree with equating native visual reasoning with Thinking with Images and tool-assisted VLM reasoning. However, this comparison overlooks a critical distinction concerning the roles of text and visual information in the problem-solving process. In commonly studied tool-assisted Thinking with Images approaches (Hu et al., 2024; Wu and Xie, 2024; Shen et al., 2025; Qi et al., 2025), the language remains the primary reasoning medium. Operations such as zooming, cropping, drawing bounding boxes, or adding auxiliary lines provide additional visual observations or clues that help the language model perform its subsequent reasoning. The model decides which visual operation to perform, receives the resulting image, continues its reasoning through text tokens, and eventually produces a textual answer.
In native visual reasoning through generation, these roles are substantially reversed. Text may specify the task, provide high-level guidance, or assist in organizing visual transformations. However, the task solution is constructed in visual space, and correctness is evaluated directly from the generated image or video.
We acknowledge that some prior visual chain-of-thought methods also generate intermediate visual states. We do not claim that visual generation has never been used in multimodal reasoning. Rather, the key distinction lies in whether visual generation serves as an auxiliary tool for language-based reasoning or as the primary mechanism for constructing the solution. In native visual reasoning setting, video/image reasoning can operate entirely without intermediate textual reasoning. In interleaved text-image generation, even if the language is generated, our experiment (Paper Table 3, replacing meaningful text with placeholders barely affects performance) shows that language is less critical to success than visual states.
- Hu et al. (2024). Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models. NeurIPS 2024.
- Wu and Xie (2024). V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs. CVPR 2024.
- Shen et al. (2025). ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration. EMNLP 2025.
- Qi et al. (2025). CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning. ICLR 2025.
5. Why do we include ThinkMorph and interleaved models? To investigate the relative importance of text and visual states, not to combine unrelated research areas.
We respectfully disagree with your interpretation of our inclusion of image, video, and interleaved models as combining different research topics. This interpretation overlooks the actual experimental motivation.
ThinkMorph was originally trained on tasks such as Jigsaw Assembly and Spatial Navigation, where performance is evaluated based on generated textual answers. In contrast, we retrain ThinkMorph to directly construct solutions in the visual domain, with correctness evaluated from the generated visual states. Thus, we include ThinkMorph not to evaluate its original research objectives, but as a representative interleaved generator under our unified visual reasoning evaluation protocol. This enables us to investigate whether language is truly necessary or helpful for solving visual tasks in native visual reasoning.
In Table 3, replacing meaningful intermediate text with uninformative placeholders leaves ThinkMorph’s score essentially unchanged (0.373 to 0.374) and only slightly reduces SenseNova-U1’s score (0.638 to 0.629). In contrast, reducing the visual trajectory to a single image lowers their scores to 0.349 and 0.527, respectively. These results suggest that visual states play a greater role than intermediate text in the reasoning process. In summary, native visual reasoning offers a new way to explore visual intelligence through visual generation. This is an exploration beyond the common intelligence paradigm of language model intelligence through text. Models construct solutions through visual states, making images and videos both the medium of reasoning and the basis for evaluating task success.
Response Part 1
Thank you for the detailed comments. However, we believe that several of the review's central concerns arise from a fundamental misunderstanding of the research setting, the definition of native visual reasoning, and the experimental design of VBVR-Pro. In particular, we would like to clarify the important differences between native visual reasoning through generation, conventional VLM-based visual understanding, and tool-assisted reasoning (e.g., thinking with images). These paradigms differ fundamentally in their objectives and underlying reasoning mechanisms, and we believe that recognizing these distinctions is essential for evaluating our contributions within the intended scope of the paper.
We would first like to clarify this conceptual distinction, as it directly addresses your concerns about novelty, scope, experimental validity, and the role of intermediate visual states.
1. Native visual reasoning is explicitly defined as reasoning through visual generation, not merely reasoning with visual inputs or producing intermediate images.
By “native visual reasoning'', we mean "native reasoning through visual generation" ( line 140, line 480, line 866, line 867, line 886), where the generated visual states (i.e., images and videos) are themselves the medium of reasoning. In language models, visual inputs are encoded into representations that condition a language model, while explicit reasoning is typically carried out through text tokens. The model may describe spatial transformations, predict coordinates, or express a solution only in linguistics. By contrast, native visual reasoning through generation directly models the required visual transformations, with the solution constructed and expressed in image or video space. As such, the present study explores an approach to visual reasoning fundamentally different from that of language models.
For example, a text-based solution to a maze may describe a route as a sequence of directions or coordinates (e.g., <think>From S at [452, 59], the only available move is West to [359, 59] …… </think> \boxed{[([(0,4), (0,3), (1,3), (2,3), (3,3), (3,2), (3,1), (2,1), (2,2), (1,2), (1,1), (0,1), (0,0), (1,0), (2,0), (3,0), (4,0), (4,1), (4,2), (5,2)]} by Du et al. (CVPR 2026).). A visual generator instead constructs the solution directly as a drawn path on an image or an agent’s movement through the maze in a video. In this setting, visual generation is not merely a rendering step added after reasoning, but a reasoner with the task-consistent visual state itself as the problem-solving objective. Our experiments cover image, video, and interleaved text-image generators, not only unified models (Table 2).