acceptodds
Under review as a conference paper at ICLR 2027

VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning

11 views

Abstract

Native visual reasoning treats visual states (i.e., images and videos) not as merely inputs to be understood or outputs to be rendered, but the intermediate steps through which a problem is solved. We introduce **VBVR-Pro**, a closed-loop testbed that makes native visual reasoning through generation trainable, verifiable, optimizable, and experimentally controllable. **1) Task scaling.** VBVR-Pro curates a task space of *300* procedurally generated tasks. Models trained on VBVR-Pro show strong transfer beyond the proposed suite across seven external video reasoning benchmarks. Analysis suggests that these gains reflect visual reasoning rather than instruction-pattern fitting. **2) Verifiable rewards.** VBVR-Pro provides verifiable reward scorers for task-grounded evaluation. While the prevalent VLM-as-a-judge paradigm shows recurring failures, deterministic scorers align more closely with human judgments, and can further serve as reliable reward signals for large-scale multi-task reinforcement learning. **3) Modality study.** VBVR-Pro enables comparisons across leading image, video, and interleaved generators under a shared task distribution and scoring protocol. Our analysis shows that video generation remains strongest for modeling spatiotemporal persistency, while interleaved generation provides a compute-efficient alternative where visual states are crucial. We release all data, models, scorers, and code to facilitate future research.

Primary area:
datasets and benchmarks
Keywords:
Visual Reasoning, Image Generation, Video Generation, Interleaved Text-Image Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XAJnOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Summary

The paper introduces VBVR-Pro, a large-scale dataset and benchmark designed to improve native visual reasoning through image, video, and interleaved text-image generation, alongside verifiable reward scorers and RL experiments. While the scale of the dataset and the reported numbers are impressive, I have significant concerns regarding the paper's novelty, framing, and experimental validity. In particular, the paper does not sufficiently distinguish native visual reasoning from existing approaches such as Thinking with Images and tool-assisted VLM reasoning, despite making broad claims about visual reasoning. Much of the strongest experimental evidence is also concentrated on video generation models, with limited validation of generalization to other model types. Furthermore, the paper lacks controlled comparisons to establish whether the improvements arise from increased task diversity, better data quality, or simply additional training data. Several central claims, particularly regarding the importance of intermediate visual states over language-based reasoning, are also insufficiently supported by the experiments. Overall, while I acknowledge the substantial engineering effort and promising results, I am not convinced that the paper adequately establishes its novelty over existing work or provides sufficient experimental evidence to justify its broader claims.

Confusing Framing and Contributions

The first issue that comes to mind is that the paper does not actually establish VBVR-Pro as a novel or better method for improving visual reasoning. The entire paper frames itself as a novel dataset for enhancing native visual reasoning, but does not mention improving native visual reasoning on what kind of models in the abstract, and discusses weaknesses of Large Language Models and how they fail to be effective in the physical world. The main problem is that the paper tries to make broad and general claims about things like "our method improves visual reasoning" and "intermediate visual states matter more than language-based reasoning", but throughout most of the paper it tackles models that were designed for video and image generation / editing, or a few specially trained unified models (e.g. ThinkMorph). It seems to mix together a couple of domains and topics (e.g. video reasoning, unified model reasoning) and market it as visual reasoning. Combining them does make the paper look like it has lots of contributions but the real issue is whether they are genuine contributions. E.g. modest contributions in the VLM space + modest contributions in the Video Generation space does not equate to an overall impressive contribution. I will try my best to evaluate it under a VLM context, and under a Video Reasoning Context.

VLMs and Native Visual Reasoning

As of the moment, I don't think it's appropriate to frame it as "a scalable and verifiable suite for native visual reasoning". First of all, native visual reasoning is very ill-defined. What does it mean by "native"? The abstract states "Native visual reasoning treats visual states (i.e., images and videos) not as merely inputs to be understood or outputs to be rendered, but the intermediate steps through which a problem is solved." If the main point of native visual reasoning is that there are intermediate steps, most of the new VLMs can interleave images, but it's not established at all that it's necessary or beneficial for them to do so. Most of the new VLMs are also pretrained with multimodal data, so they aren't less "multimodal" than any other generation models. So I find it unjustified to leave out normal VLMs like Gemma4 or Qwen3.5. The authors may rebut by saying that native visual reasoning requires models with image/video output capabilities, so normal VLMs do not qualify. But the problem is that in order to make this point, you must first prove that this restriction / qualification is necessary. The main problem is whether native visual reasoning under the VLM setting is a useful setting at all, and if you can demonstrate that it outperforms "normal" visual reasoning in the same environment (e.g. same compute budget, same base models). Improvements gained from interleaving visual information in VLM reasoning is proven and 2-3 years old, unless the authors can demonstrate that native visual reasoning is not a rebrand of the same method, then it's not appropriate to frame native visual reasoning as a new paradigm. The datasets themselves aren't novel for VLMs either, there are dozens of papers on VLM reasoning in mazes (Figure 4) and VLM reasoning with geometry and 3d shapes (Figure 5). I've seen tasks like this 2 years ago. Throughout the paper I don't see any tasks that are "novel" or have not been introduced before as a VLM training task / benchmark. It also doesn't help that there isn't a catalog of all 300 tasks anywhere for me to inspect novelty in the task design, but that there is a 249MB prep video in the supplementary with things mentioned in the paper. I do not think there is any novelty to claim here, under VLM and general Visual Reasoning context.

Video Reasoning

As established, there lacks novelty and contribution in the VLM setting, which in fact is also a small portion of the paper. Alot of the main evaluations occur for video reasoning and video generation models. If the authors were to refer to Table 4, all of the other datasets and benchmarks they compared against explicitly frame it as "video reasoning", which is significantly different from visual reasoning. I recommend the authors make significant revisions to make sure that their contribution is explicitly tailored towards video reasoning, because if the paper were to be evaluated as a visual reasoning paper, it has little novelty.

Conclusion

The paper attempts to bundle together a couple of very different tasks that usually do not share one paper in order to make it appear to target a very general contribution. Fundamentally, VBVR-Pro attempts to propose "native visual reasoning" without clearly defining it or show why it isn't a rebrand of Think with Images and does not extent to general VLMs. It is more akin to a video reasoning dataset / benchmark that just happens to be usable for VLMs, Image-Generation Models, and Unified Models, but does not demonstrate useful novelty in either. Finally, many of the concepts such as verifiable rewards and weaknesses in VLM-as-a-judge were already present in previous papers the authors cited (VBVR - icml), and claims such as gains resulting from training on the dataset transfers to other domains are mere minimal requirements for a dataset.

Critical strengths
  • There are 300 tasks and the dataset is very big.
  • The authors propose an accurate verifier instead of a VLM judge.
Critical weaknesses
  • Native Visual Reasoning is not well defined, and it's not established that this is a genuinely different paradigm than thinking with images, visual chain-of-thought, and tool-assisted VLM reasoning. This is crucial if the main contribution of the paper is a dataset / benchmark that is tailored for native visual reasoning. The paper does not convincingly explain what fundamentally differentiates its approach from these methods or why generating intermediate visual states directly should be preferred over alternative approaches.

  • Insufficient evidence that intermediate visual states are responsible It's shown in Appendix D.2 reports that 209 of the 300 tasks (69.7%) use the Last-Frame image-output regime. Only 91 tasks use Key-Frame or Multi-Frame outputs. This further weakens the native visual reasoning claim.

Questions for authors
  • Can you explicitly define what Native Visual Reasoning is and why it only works for Unified Models and not VLMs? How does generated visual states from a unified model stored in the context differ from a ground-truth visual state stored in the context from VLM tool-use? (e.g. Thinking with Images). VLM with tool use and ground-truth intermediate visual states is more efficient than Unified Models and likely better. Unless authors can prove otherwise, I believe it's not appropriate to market this benchmark as something tailored for "native visual reasoning". A benchmark and dataset for an unjustified setting defeats its purpose. I am willing to raise the score if this issue can be resolved in either manner.

  • Can you provide the full catalog of tasks?

  • I would like to see a controlled comparison between VBVR and VBVR-Pro training for Wan2.2 (Table4), the current comparison is unfair if you are training it with different amounts of data.

  • Can you address the other weaknesses in the Critical Weakness section?

11 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for the detailed comments. However, we believe that several of the review's central concerns arise from a fundamental misunderstanding of the research setting, the definition of native visual reasoning, and the experimental design of VBVR-Pro. In particular, we would like to clarify the important differences between native visual reasoning through generation, conventional VLM-based visual understanding, and tool-assisted reasoning (e.g., thinking with images). These paradigms differ fundamentally in their objectives and underlying reasoning mechanisms, and we believe that recognizing these distinctions is essential for evaluating our contributions within the intended scope of the paper.

We would first like to clarify this conceptual distinction, as it directly addresses your concerns about novelty, scope, experimental validity, and the role of intermediate visual states.


Weakness 1 & Question 1: Definition, Research Scope, and Novelty of Native Visual Reasoning


1. Native visual reasoning is explicitly defined as reasoning through visual generation, not merely reasoning with visual inputs or producing intermediate images.

By “native visual reasoning'', we mean "native reasoning through visual generation" ( line 140, line 480, line 866, line 867, line 886), where the generated visual states (i.e., images and videos) are themselves the medium of reasoning. In language models, visual inputs are encoded into representations that condition a language model, while explicit reasoning is typically carried out through text tokens. The model may describe spatial transformations, predict coordinates, or express a solution only in linguistics. By contrast, native visual reasoning through generation directly models the required visual transformations, with the solution constructed and expressed in image or video space. As such, the present study explores an approach to visual reasoning fundamentally different from that of language models.

For example, a text-based solution to a maze may describe a route as a sequence of directions or coordinates (e.g., <think>From S at [452, 59], the only available move is West to [359, 59] …… </think> \boxed{[([(0,4), (0,3), (1,3), (2,3), (3,3), (3,2), (3,1), (2,1), (2,2), (1,2), (1,1), (0,1), (0,0), (1,0), (2,0), (3,0), (4,0), (4,1), (4,2), (5,2)]} by Du et al. (CVPR 2026).). A visual generator instead constructs the solution directly as a drawn path on an image or an agent’s movement through the maze in a video. In this setting, visual generation is not merely a rendering step added after reasoning, but a reasoner with the task-consistent visual state itself as the problem-solving objective. Our experiments cover image, video, and interleaved text-image generators, not only unified models (Table 2).

  • Du et al. (2026). Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalization. CVPR 2026.
AuthorsOpenReview ↗

Response Part 2

2. Native Visual Reasoning is an emerging but already established research direction

This research setting is distinguished by its computational and output representations, rather than by an arbitrary distinction between model families. Importantly, Native Visual Reasoning is an emerging but already established research direction, not an unjustified paradigm introduced by this paper. This direction can be traced to Google DeepMind's Video Models Are Zero-Shot Learners and Reasoners (Wiedemer et al., September 2025), which demonstrated that video generation models such as Veo 3 exhibit emergent zero-shot capabilities on visual reasoning tasks, including maze solving and symmetry transformations. Since then, a growing literature has investigated different dimensions of this research direction:

Benchmarks: MME-CoF (Guo et al., 2025), V-ReasonBench (Luo et al., 2025), RULER-Bench (He et al., 2025), VideoThinkBench (Tong et al., CVPR 2026), and MME-CoF-Pro (Qi et al., 2026) systematically evaluate visual generative reasoning across spatial, logical, physical, temporal, and planning tasks.

Training data and scalable learning: VBVR (Wang et al., ICML 2026) established large-scale procedural training for video generative reasoning, while OpenCoF (Chen et al., 2026) further investigated learning Chain-of-Frames reasoning from dedicated visual trajectory supervision.

Generative reasoning methods: DiffThinker (He et al., ICML 2026), and Hierarchical Denoising for Multi-Step Visual Reasoning (Qian et al., 2026) investigate approaches to improving visual generative reasoning through visual-state planning, progressive visual transformations, or structured denoising.

Theoretical and mechanistic understanding: Recent studies provide compelling evidence that video generators exhibit reasoning mechanisms fundamentally distinct from language models. A few works (Demystifying Video Reasoning (Wang et al., 2026), Video Models Reason Early (Newman et al., 2026), DiffThinker (He et al., 2026)) reveal that video reasoning emerges through a Chain-of-Steps mechanism, where models explore and progressively refine candidate solutions during diffusion denoising, rather than through sequential linguistic reasoning.

  • Wiedemer et al. (2025). Video Models Are Zero-Shot Learners and Reasoners. arXiv:2509.20328.
  • Guo et al. (2026). Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark. CVPR Findings 2026.
  • Luo et al. (2025). V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models. arXiv:2511.16668.
  • He et al. (2025). RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligence. arXiv:2512.02622.
  • Tong et al. (2026). Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm. CVPR 2026.
  • Qi et al. (2026). MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hints. ECCV 2026.
  • Wang et al. (2026). A Very Big Video Reasoning Suite. ICML 2026.
  • Chen et al. (2026). OpenCoF: Learning to Reason Through Video Generation. NeurIPS 2026.
  • He et al. (2026). DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models. ICML 2026.
  • Qian et al. (2026). Hierarchical Denoising for Multi-Step Visual Reasoning. NeurIPS 2026.
  • Wang et al. (2026). Demystifying Video Reasoning. ECCV 2026.
  • Newman et al. (2026). Video Models Reason Early: Exploiting Plan Commitment for Maze Solving. arXiv:2603.30043.

3. Distinction from VLM-based "video reasoning": reasoning on video (where video is the input) versus reasoning by generating video (where video is the reasoning medium, output, and the subject of evaluation).

In VLM-based video reasoning, video serves primarily as the input modality. The model processes the input video to extract relevant spatial and temporal cues, but performs reasoning and produces its final answer in the language domain.

By contrast, in native visual reasoning through video generation, it directly constructs a solution by generating task-consistent visual states (i.e., images and videos), without relying on textual reasoning to derive or express the answer. For example, in the “Key–Door Navigation” task (this link), the model must generate a video showing an agent navigating through a maze, collecting the required keys, and reaching the destination. Success is evaluated directly from the generated video, based on whether the agent collects the required keys, reaches the destination, and follows a valid, continuous path without crossing walls. Therefore, the generated video is the solution itself, with its correctness directly evaluated based on the generated visual states and their transitions.

AuthorsOpenReview ↗

Response Part 3

4. Distinction from VLM-based "Thinking with Images": the roles of text and visual states are fundamentally different.

We respectfully disagree with equating native visual reasoning with Thinking with Images and tool-assisted VLM reasoning. However, this comparison overlooks a critical distinction concerning the roles of text and visual information in the problem-solving process. In commonly studied tool-assisted Thinking with Images approaches (Hu et al., 2024; Wu and Xie, 2024; Shen et al., 2025; Qi et al., 2025), the language remains the primary reasoning medium. Operations such as zooming, cropping, drawing bounding boxes, or adding auxiliary lines provide additional visual observations or clues that help the language model perform its subsequent reasoning. The model decides which visual operation to perform, receives the resulting image, continues its reasoning through text tokens, and eventually produces a textual answer.

In native visual reasoning through generation, these roles are substantially reversed. Text may specify the task, provide high-level guidance, or assist in organizing visual transformations. However, the task solution is constructed in visual space, and correctness is evaluated directly from the generated image or video.

We acknowledge that some prior visual chain-of-thought methods also generate intermediate visual states. We do not claim that visual generation has never been used in multimodal reasoning. Rather, the key distinction lies in whether visual generation serves as an auxiliary tool for language-based reasoning or as the primary mechanism for constructing the solution. In native visual reasoning setting, video/image reasoning can operate entirely without intermediate textual reasoning. In interleaved text-image generation, even if the language is generated, our experiment (Paper Table 3, replacing meaningful text with placeholders barely affects performance) shows that language is less critical to success than visual states.

  • Hu et al. (2024). Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models. NeurIPS 2024.
  • Wu and Xie (2024). V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs. CVPR 2024.
  • Shen et al. (2025). ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration. EMNLP 2025.
  • Qi et al. (2025). CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning. ICLR 2025.

5. Why do we include ThinkMorph and interleaved models? To investigate the relative importance of text and visual states, not to combine unrelated research areas.

We respectfully disagree with your interpretation of our inclusion of image, video, and interleaved models as combining different research topics. This interpretation overlooks the actual experimental motivation.

ThinkMorph was originally trained on tasks such as Jigsaw Assembly and Spatial Navigation, where performance is evaluated based on generated textual answers. In contrast, we retrain ThinkMorph to directly construct solutions in the visual domain, with correctness evaluated from the generated visual states. Thus, we include ThinkMorph not to evaluate its original research objectives, but as a representative interleaved generator under our unified visual reasoning evaluation protocol. This enables us to investigate whether language is truly necessary or helpful for solving visual tasks in native visual reasoning.

In Table 3, replacing meaningful intermediate text with uninformative placeholders leaves ThinkMorph’s score essentially unchanged (0.373 to 0.374) and only slightly reduces SenseNova-U1’s score (0.638 to 0.629). In contrast, reducing the visual trajectory to a single image lowers their scores to 0.349 and 0.527, respectively. These results suggest that visual states play a greater role than intermediate text in the reasoning process. In summary, native visual reasoning offers a new way to explore visual intelligence through visual generation. This is an exploration beyond the common intelligence paradigm of language model intelligence through text. Models construct solutions through visual states, making images and videos both the medium of reasoning and the basis for evaluating task success.

Reviewer_1m8NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims at addressing the challenges of scalable training, reliable evaluation, and effective optimization in native visual reasoning. To tackle these challenges, the authors try to introduce VBVR-Pro, a unified framework comprising 300 procedurally generated tasks, verifiable reward scorers, and a benchmark supporting image, interleaved text-image, and video generation. Extensive experiments demonstrate strong generalization across seven external benchmarks, improved alignment with human judgments, and effective reinforcement learning with verifiable rewards, highlighting the importance of intermediate visual states and the potential of generation-based visual reasoning.

Critical strengths
  1. VBVR-Pro provides 300 procedurally generated tasks covering five cognitive capabilities and supports image, interleaved text-image, and video generation. The unified task distribution and evaluation protocol facilitate systematic comparisons across different visual reasoning paradigms.
  2. The proposed verifiable reward scorers achieve stronger agreement with human preferences than VLM-based judges while offering lower evaluation costs and deterministic results. These scorers also provide task-grounded reward signals for reinforcement learning.
  3. The proposed RLVR approach outperforms both VLM-based reward learning and MSE-based reward optimization. These results demonstrate the potential of task-specific verifiable rewards for optimizing native visual reasoning.
Critical weaknesses
  1. The previous VBVR paper already introduces procedurally generated tasks and verifiable scoring mechanisms. VVBVR-Pro builds upon these components by expanding task coverage, supporting additional generation formats, and introducing large-scale multi-task RL experiments. A clearer distinction between inherited components and new technical contributions would help highlight the novelty of this work.
  2. Although VBVR-Pro demonstrates improved generalization, the newly introduced tasks are both more diverse and more challenging than those in the original VBVR. It remains unclear to what extent the observed improvements can be attributed to task diversity rather than increased task difficulty. A controlled ablation separating these factors would strengthen the conclusions.
Questions for authors
  1. Figure 1 does not clearly illustrate the overall workflow of VBVR-Pro. Could the authors provide a more informative overview that better explains how the different components, including task generation, model training, verifiable scoring, and reinforcement learning, are connected?
  2. Compared with the previously published VBVR framework, what are the main technical innovations of VBVR-Pro beyond expanding the number of tasks and supporting additional generation formats?
4 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for the careful review and helpful suggestions. We clarify the contributions beyond VBVR, provide additional experimental results, and explain the overall workflow below.


Weakness 1 & Question 2: “[...] A clearer distinction between inherited components and new technical contributions would help highlight the novelty of this work.”


Thank you for your suggestion to clarify the contributions beyond VBVR. VBVR-Pro builds on VBVR’s procedural task generation and verifiable scoring framework, while introducing four main contributions.

  1. Extend video to image and interleaved text-image generation. VBVR focuses exclusively on video modality. We extend the scope to aligned video, image and interleaved text-image formats and train and evaluate models across all three modalities. This makes the suite accessible to a broader research community, including researchers working on image and interleaved generation. It also enables a more comprehensive analysis of visual reasoning across modalities, examining which tasks benefit from each modality and how intermediate visual states contribute to reasoning (Sections 2 and 4).
  2. The first large-scale multi-task RL training recipe. While VBVR is limited to SFT and prior visual reasoning RL studies (e.g., Zhu et al., 2026) only explore a few isolated tasks, we introduce the first large-scale multi-task RL recipe, jointly optimizing video models with verifiable rewards across 50 diverse tasks. We find that semantically meaningful rollout diversity is important for effective video reasoning RL: rollouts should explore different task-solving decisions, rather than merely vary at the pixel level. We therefore adopt Coefficients-Preserving Sampling (CPS) to balance task-level semantic exploration and visual quality under high rollout noise. Evaluated via both scorers and human preferences, our recipe consistently outperforms SFT and VLM-reward RL with strong generalization to 50 unseen out-of-domain tasks (Section 5; Appendix C). These results in turn validate the effectiveness of both our curated task data and our revised scorers (described below).
  3. Scorer reimplementation and validation. We substantially revise the scoring mechanisms to better assess task outcomes, intermediate processes, and constraint satisfaction. Specifically, we replace pixel-level comparisons in affected tasks with task-relevant semantic checks for greater robustness to rendering variations. For example, a circle marking the answer may vary in size or position while still indicating the same answer. Our scorers can handle these pixel-level differences between semantically equivalent outputs. We also expand the evaluation criteria to cover previously overlooked requirements and adjust the weights to better reflect their importance. To assess the revised scorers’ accuracy, we conduct instance-level human-preference validation and show improved model-level agreement with human judgments compared with VBVR (Section 3 and Figure 4(c)).
  4. Task redesign and expansion. We reimplement and revise the original 150 tasks to improve visual clarity and consistency with the intended task rules. For example, we refine the geometry and layouts of domino tasks to make the objects and their interactions easier to interpret, and correct inconsistencies in wall collisions and rebound behavior in ball-bouncing tasks. Furthermore, we introduce 150 new tasks with greater visual complexity and reasoning depth (Section 2; Appendices D.5–D.6).

Overall, VBVR-Pro goes beyond extending the VBVR task set: it substantially improves task design and evaluation, introduces multimodal training and evaluation, and demonstrates the utility of verifiable rewards through multi-task RL experiments.

  • Zhu et al. (2026). Video Models Can Reason with Verifiable Rewards. NeurIPS 2026.
AuthorsOpenReview ↗

Response Part 2


Weakness 2: “[...] It remains unclear to what extent the observed improvements can be attributed to task diversity rather than increased task difficulty. [...]”


Thank you for suggesting this ablation. We conduct additional comparisons with Wan2.2-TI2V-5B, keeping the total number of training samples fixed across three settings.

  • Exp. 1: Training on the original 100 VBVR tasks.
  • Exp. 2: Training on all 250 VBVR-Pro training tasks.
  • Exp. 3: Training on a 100-task mixture comprising 50 randomly selected original VBVR tasks and 50 randomly selected newly introduced tasks.

First, introducing harder tasks alone does not yield a clear overall improvement (Exp. 1 vs. Exp. 3). Replacing 50 original tasks with more challenging newly introduced tasks leaves the overall score nearly unchanged (38.05 versus 37.86). This suggests that increasing task difficulty alone brings limited benefit to overall performance.

Second, increasing task diversity improves performance (Exp. 3 vs. Exp. 2). Expanding task coverage from the 100-task mixture to the full 250-task training set improves the overall score from 37.86 to 46.71. This supports the positive contribution of broader task diversity to model performance.

Together, these comparisons support the contribution of broader task diversity and suggest that the improvements cannot be attributed simply to increased task difficulty.

\begin{array}{cl|c|ccccccc|ccccccc} \hline & & \rlap{\hspace{2.774em}\smash{\rule[-0.480em]{3.141em}{0.06em}}} & \rlap{\hspace{4.392em}\textbf{In-Domain by Category}}\rlap{\hspace{2.618em}\smash{\rule[-0.480em]{19.208em}{0.06em}}} & & & & & & \rlap{\hspace{2.563em}\smash{\rule[-0.480em]{3.141em}{0.06em}}} & \rlap{\hspace{3.180em}\textbf{Out-of-Domain by Category}}\rlap{\hspace{2.618em}\smash{\rule[-0.480em]{19.067em}{0.06em}}} & & & & & & \\\\ \textbf{Exp.} & \textbf{Training tasks} & \textbf{Overall} & \textbf{Avg.} & \smash{\rule[-4.784em]{0.07em}{5.733em}} & \textbf{Abst.} & \textbf{Know.} & \textbf{Perc.} & \textbf{Spat.} & \textbf{Trans.} & \textbf{Avg.} & \smash{\rule[-4.784em]{0.07em}{5.733em}} & \textbf{Abst.} & \textbf{Know.} & \textbf{Perc.} & \textbf{Spat.} & \textbf{Trans.} \\\\ \hline 1 & \text{VBVR (100 tasks)} & 38.05 & 44.48 & & 59.49 & 32.86 & 35.65 & 45.40 & 52.19 & 31.61 & & 42.25 & 25.41 & 25.17 & 41.29 & 36.77 \\\\ 2 & \text{VBVR-Pro (250 tasks)} & 46.71 & 59.92 & & 63.51 & 54.43 & 46.42 & 62.50 & 75.03 & 33.51 & & 45.89 & 18.70 & 24.89 & 45.51 & 48.84 \\\\ 3 & \text{50 original VBVR tasks + 50 new tasks} & 37.86 & 45.48 & & 58.30 & 33.00 & 35.30 & 50.14 & 54.07 & 30.24 & & 45.58 & 28.83 & 22.70 & 34.72 & 34.30 \\\\ \hline \end{array}$$ --- >**Question 1**: “Figure 1 does not clearly illustrate the overall workflow of VBVR-Pro. Could the authors provide a more informative overview that better explains how the different components, including task generation, model training, verifiable scoring, and reinforcement learning, are connected?” --- Thank you for your suggestion. We will revise the caption of Figure 1 to better explain how the components are connected. We clarify the workflow as follows: 1. **Task Generation (left conveyor belt)**: Our **300 task generators** produce aligned training data for video, image, and interleaved text-image generation (Section 2). 2. **Model Training (three central branches)**: The generated data are used to train models across the three modalities, represented by the three parallel conveyor belts labeled "Video," "Image," and "Interleaved" (Section 4). 3. **Verifiable Scoring (upper-center panel)**: The ball-shaped robot above the conveyor belt illustrates the evaluation process by scanning the generated outputs, while the icons in the “Verifiable Reward Scorers” panel represent task-specific scoring tools. These scorers evaluate outputs against explicitly defined task requirements, providing both evaluation metrics for benchmarking and continuous reward signals for reinforcement learning (Section 3). 4. **Reinforcement Learning (right-side module and bottom feedback loop)**: The right-side "RL Rollouts" module illustrates video generation rollouts evaluated using these scorers, while the bottom “Policy Optimization” loop illustrates how reinforcement learning uses these reward signals to iteratively improve the video generation model (Section 5).

Thank you for the rebuttal.

The response clarifies the contributions of VBVR-Pro beyond the original VBVR. Although the additional comparisons provide useful evidence, they do not fully disentangle the effects of task diversity and task difficulty, and the improvements in out-of-domain generalization remain relatively limited. Overall, I maintain my positive assessment and original rating.

AuthorsOpenReview ↗

Response Round 2

Thank you for your follow-up comment, and we appreciate your positive assessment. We would like to provide a more detailed explanation of the rationale behind our additional experimental design, which was specifically intended to distinguish the contribution of task diversity from that of task difficulty.

First, Exp. 1 vs. Exp. 3 controls the number of training tasks (100) and the total number of training samples, while replacing 50 original VBVR tasks with 50 newly introduced, generally more difficult VBVR-Pro tasks(randomly sampled). This comparison examines whether introducing harder tasks, without increasing the number of distinct training tasks, is sufficient to improve performance. The nearly identical overall scores (38.05 vs. 37.86) suggest that simply changing to more difficult tasks does not lead to a clear performance gain.

Second, Exp. 3 vs. Exp. 2 keeps the total number of training samples fixed but expands task coverage from 100 to 250 tasks. The substantial improvement in overall performance (37.86 vs. 46.71) suggests that broader task diversity provides benefits beyond those achieved by merely substituting harder tasks into a fixed-size task set.

Task difficulty is inherently challenging to define objectively across heterogeneous visual reasoning tasks, as different tasks involve distinct reasoning skills and visual operations. To better address your concern, we would sincerely appreciate your guidance on what additional experimental settings or criteria you would consider sufficient\textcolor{blue}{\text{sincerely appreciate your guidance on what additional experimental settings or criteria you would consider sufficient}} to fully disentangle the effects of task diversity and task difficulty. We would be more than happy to do alternative experiments\textcolor{blue}{\text{more than happy to do alternative experiments}} that you believe would more directly address this concern.


the improvements in out-of-domain generalization remain relatively limited


We sincerely thank you for your careful examination of our experimental results. Although when comparing Exp. 2 with Exp. 3, OOD improvement (+3.27) is smaller than ID improvement (+8.85). At the same time, we would like to note that the OOD subset remains highly challenging. Even Veo 3.1, a very large-scale proprietary model, achieves an OOD score of only 30.5 on our benchmark, which is lower than that of Exp. 2.

Reviewer_L9qZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a scalable and verifiable generation and evaluation framework for native visual reasoning named VBVR-Pro. Specifically, VBVR-Pro consists of three core components: the procedural task generation framework, the verifiable reward scoring, and the generative model training. The procedural task generation framework generates aligned visual solutions and metadata-grounded supervision through task-specific algorithmic solvers and multimodal trajectory rendering. The verifiable reward scoring broadens the coverage of reproducible visual reasoning evaluation via structured semantic extraction and task-specific rule verification. Finally, the generative model training framework optimizes existing generative models, balancing task-level reasoning performance and visual fidelity.

The experimental evaluation in this paper assesses the effectiveness of VBVR-Pro across eight benchmark datasets. Results demonstrate that VBVR-Pro achieves better overall visual reasoning performance while improving generalization to unseen tasks, outperforming several mainstream generative baselines.

Critical strengths
  • The idea of investigating whether models can reason by generating and updating visual states in images or videos is considered novel.
  • The proposed VBVR-Pro Dataset is beneficial for the community.
Critical weaknesses
  • The authors aim to provide reliable rewards through explicit rules, but the actual scorers typically encode multiple requirements as weighted terms and soft gates. The resulting scores can reflect the degree of task completion without necessarily indicating that the key constraints have been satisfied.
  • The verifiable rewards do not clearly distinguish complete task success from high partial scores. I understand that the authors intend to use continuous rewards to distinguish complete failure, partial completion, and near-success, providing richer learning signals for RL than binary rewards. However, the paper does not seem to sufficiently distinguish a model receiving a high partial reward from fully satisfying the task requirements. If a key requirement is assigned a small weight, the model can increase its overall score by improving other components that are easier to optimize. This introduces a risk that higher rewards may primarily reflect improvements in partial task completion, without corresponding gains in complete task success rates.
  • Furthermore, the scorer appears to entangle reasoning ability, generation fidelity, and the ease of parsing outputs. The final score depends on whether the model solves the task correctly, whether the parser can read the outputs easily, and whether objects and events are correctly recovered. Consequently, the overall score does not clearly indicate which capability has actually improved. As discussed in Appendix C.5, visual fidelity can also affect measured performance. Does this suggest that the overall score itself may not be straightforward to interpret?
Questions for authors
  • This paper relies almost entirely on synthetic data. Does this limit the framework’s applicability or its adaptability to other settings?
  • As the paper points out, expanding task coverage is more important than simply increasing the number of instances for a small set of tasks. However, generating instances for existing task types is much easier than introducing new task types. Does this mean that adding a new task type requires substantial manual intervention and validation?
4 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for the careful review and thoughtful questions about the scoring framework and its applicability. We provide clarifications and additional evidence in our responses below.


Weakness 1: “The resulting scores can reflect the degree of task completion without necessarily indicating that the key constraints have been satisfied.”


Thank you for raising this concern. We acknowledge that a weighted reward is not equivalent to a binary guarantee of satisfying every constraint. However, our scorers are explicitly designed to penalize violations of core requirements**: for each task, we explicitly define evaluation criteria corresponding to its core requirements and carefully design the weights and penalties to ensure that violations of key constraints substantially reduce the final reward, ensuring that apparent last frame success alone does not yield a high reward. For example, in the “Key–Door Navigation” task (this link) in Fig. 4(a), although the agent nearly reaches the target gate, it fails to collect the required key and crosses walls multiple times, resulting in a low reward. This is because the scorer evaluates not only whether the agent reaches the destination, but also whether it collects the correct keys and follows a valid, continuous path through the maze without crossing walls. Similarly, the scorers also evaluate intermediate processes in other tasks, such as path following and forward motion in “Rolling Ball” task (this link) and the order of eating events and corresponding ball growth in “Greedy Ball Eating” task (this link) (Appendix E.3). Please click on the links to see the details in videos.

For transparency, we also provide the scorer code (this link) and an interactive website (this link) detailing each task’s key criteria and showing how individual requirements are checked and combined into the final reward.


Weakness 2: “[...] However, the paper does not seem to sufficiently distinguish a model receiving a high partial reward from fully satisfying the task requirements. [...]”


We appreciate your questions and concerns about our scorers. We provide further clarification and supporting evidence in the following three points.

First, our scoring design prioritizes core task requirements through higher weights. For example, the “Rolling Ball” scorer (this link) evaluates the ball’s final position together with whether it follows the path and moves in the correct forward direction (Appendix E.3). Even if the ball reaches the correct final position, a zero score for path following or forward motion caps the final reward at 0.2. Scene and background preservation are also evaluated, but these secondary criteria can only reduce the task-completion score and cannot compensate for failures in the core requirements. Our website (this link) further illustrates how these criteria contribute to the final reward.

Second, we validate the resulting scores against human judgments and observe strong agreement with human preferences. Our scorers achieve per-vote agreement above 0.60, compared with 0.54 for GPT-5.5 and 0.52 for Gemini-3.1-Pro. Scorer-based model rankings also closely match human rankings (Section 3 and Appendix E.2).

Third, human evaluation supports improved task performance after RLVR. Human annotators assess generated videos from the six models in Table 5 according to how well they satisfy the task requirements, and we use these ratings to compute each model’s win rate in pairwise comparisons with the other models. Under this evaluation, the win rate increases from 0.473 for the SFT-trained baseline to 0.541 after RLVR using rewards provided by our scorers. This provides independent evidence that optimizing our rewards improves human-assessed task performance.

AuthorsOpenReview ↗

Response Part 2


Weakness 3: “Furthermore, the scorer appears to entangle reasoning ability, generation fidelity, and the ease of parsing outputs. [...] Does this suggest that the overall score itself may not be straightforward to interpret?”


Thank you for your comments on the robustness of our output parsing. We would like to clarify the following three points.

First, our scorers are designed to tolerate reasonable rendering variations. We use task-specific extraction methods to identify shapes, colors, text, and object movements, then evaluate whether the task requirements are met. For example, in the “Greedy Ball Eating” task (this link), we detect the balls using color segmentation and contour analysis, then track them across frames to assess the eating order and ball growth (Appendix E.3). Our scorers thus operate on task-relevant semantic information rather than pixel-level similarity, allowing minor rendering differences without requiring an exact visual match.

Second, explicit requirements for output format and answer parsing are common evaluation practices. In LLM evaluation, IFEval (Zhou et al., 2023) includes instructions requiring JSON-formatted responses. Beyond format requirements, evaluators also use specific markers to extract answers. For example, the original GSM8K (Cobbe et al., 2021) evaluator extracts numerical answers following “####”. Similarly, the MATH evaluator in OpenAI’s simple-evals extracts final answers following “Answer:”. This practice also extends to VLM evaluation, where MMMU-Pro (Yue et al., 2025) uses parsing rules to identify answers presented in forms such as “Answer: B” or “(B)”. Following the same principle, we design our tasks and visual layouts so that correct outputs are clearly identifiable. At the same time, our scorers accommodate reasonable rendering variations and do not require an exact pixel-level match. For example, slight differences in a circle’s size or position can be tolerated when they do not violate the task requirements.

Third, we would like to point out that the fidelity penalties in Appendix C.5 reflect task-requirement violations, not parsing failures. The model receives credit for completing the foreground task but is penalized for changing the background, because the task prompt explicitly states: “Only modify what the task requires; keep the background and any other element unchanged.” Such penalties also help RL improve video quality while respecting the task requirements. Importantly, fidelity is a secondary criterion that can only reduce the task-completion score rather than independently earning reward. Furthermore, in our experience, we also observe that stronger video generation models in our evaluation generally produce sufficiently clear outputs, making severe blur an uncommon obstacle to parsing. Consequently, these models are unlikely to be incorrectly evaluated or penalized due to parsing failures.

  • Zhou et al. (2023). Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911.
  • Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168.
  • Yue et al. (2025). MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. ACL 2025.

Question 1: “This paper relies almost entirely on synthetic data. Does this limit the framework’s applicability or its adaptability to other settings?”


Thank you for your question about generalization. Although our training data are synthetic, the learned capabilities generalize beyond the training setting. Table 4 shows improvements across all seven external video reasoning benchmarks, which involve various real-world visual scenarios. For example, V-ReasonBench includes tasks such as "Shape Fitting" and "Block Sliding", while MME-CoF-Pro includes "Embodied Reasoning" and "4D Dynamics Reasoning", all featuring real-world scenes. Training Wan2.2-I2V-A14B on VBVR-Pro improves its overall score on V-ReasonBench from 10.21 to 38.22 and on MME-CoF-Pro from 24.76 to 48.39. Appendix B.5 and Figure 8 further illustrate transfer of this model beyond the training distribution, such as ordering blocks and generating a robotic pick-and-place sequence. The nearest-neighbor analysis shows that these examples are visually distinct from the retrieved training samples. These findings support the framework’s applicability beyond synthetic tasks.

AuthorsOpenReview ↗

Response Part 3


Question 2: “[...] Does this mean that adding a new task type requires substantial manual intervention and validation?”


We appreciate your concern about the effort required to expand task coverage. We agree that introducing new task types currently requires more manual effort than generating additional instances of existing tasks. As discussed in the section 2, we argue that expanding task diversity is more important than simply increasing the number of instances within existing task types.

To further support this claim, we conduct an additional controlled experiment comparing the two scaling strategies using the Wan2.2-TI2V-5B model. Exp. 1 trains on the original 100 VBVR tasks. Exp. 2 expands the training set to 250 VBVR-Pro tasks while keeping the total number of training samples equal to Exp. 1. Exp. 3 uses the same 250 tasks with all available training samples of VBVR-Pro. Comparing Exp. 1 and Exp. 2, expanding task coverage improves the overall score from 38.05 to 46.71. In contrast, increasing the number of samples within the same 250 tasks yields only a marginal overall improvement, from 46.71 to 47.00 (Exp. 2 vs. Exp. 3). These results suggest that, when sufficient training data are available, increasing task diversity is more beneficial than adding more instances of existing tasks.

This argument also motivated our careful task design process when working on VBVR-Pro. Specifically, our tasks are guided by a cognitive-science-inspired framework covering five cognitive faculties and were developed through a collaborative effort involving more than 50 researchers and engineers (Section 2). Following implementation, we manually review each task generator by inspecting rendered samples and verifying its solver implementation (Appendix D.1). Although developing new task types currently requires substantial effort, this rigorous process is essential for ensuring meaningful task diversity, correctness, and reliable evaluation. Looking forward, as AI agents become more capable, we envision increasingly automating the design, implementation, and validation of new task types, making it easier to expand task coverage at scale.

\begin{array}{clc|c|ccccccc|ccccccc} \hline & & & \rlap{\hspace{2.774em}\smash{\rule[-0.480em]{3.141em}{0.06em}}} & \rlap{\hspace{4.392em}\textbf{In-Domain by Category}}\rlap{\hspace{2.618em}\smash{\rule[-0.480em]{19.208em}{0.06em}}} & & & & & & \rlap{\hspace{2.563em}\smash{\rule[-0.480em]{3.141em}{0.06em}}} & \rlap{\hspace{3.180em}\textbf{Out-of-Domain by Category}}\rlap{\hspace{2.618em}\smash{\rule[-0.480em]{19.067em}{0.06em}}} & & & & & & \\\\ \textbf{Exp.} & \mathbf{\\#}\textbf{Tasks} & \mathbf{\\#}\textbf{Samples} & \textbf{Overall} & \textbf{Avg.} & \smash{\rule[-4.784em]{0.07em}{5.733em}} & \textbf{Abst.} & \textbf{Know.} & \textbf{Perc.} & \textbf{Spat.} & \textbf{Trans.} & \textbf{Avg.} & \smash{\rule[-4.784em]{0.07em}{5.733em}} & \textbf{Abst.} & \textbf{Know.} & \textbf{Perc.} & \textbf{Spat.} & \textbf{Trans.} \\\\ \hline 1 & \text{VBVR (100 tasks)} & N & 38.05 & 44.48 & & 59.49 & 32.86 & 35.65 & 45.40 & 52.19 & 31.61 & & 42.25 & 25.41 & 25.17 & 41.29 & 36.77 \\\\ 2 & \text{VBVR-Pro (250 tasks)} & N & 46.71 & 59.92 & & 63.51 & 54.43 & 46.42 & 62.50 & 75.03 & 33.51 & & 45.89 & 18.70 & 24.89 & 45.51 & 48.84 \\\\ 3 & \text{VBVR-Pro (250 tasks)} & N_{\text{full}} & 47.00 & 64.10 & & 68.70 & 62.30 & 49.80 & 62.10 & 78.60 & 30.00 & & 45.30 & 15.80 & 18.10 & 50.50 & 45.20 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Follow-up on Our Responses and Revised Manuscript

Thank you again for your thoughtful review and valuable feedback. We hope our responses have been helpful in addressing your concerns, and we would be very happy to discuss any remaining questions or thoughts you may have.

Please also kindly note that the section and appendix numbers in our responses now refer to the revised PDF. We sincerely appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.