acceptodds
Under review as a conference paper at ICLR 2027

WOVEN: Weaving Visual World Modeling into Multimodal LLMs

11 views

Abstract

Multimodal large language models (MLLMs) exhibit scattered failures across spatial, embodied, physical, and temporal reasoning. We hypothesize that these failures reflect a shared deficit in action-conditioned visual transition reasoning over triplets, and investigate its internal organization to derive a systematic training recipe. To enable controlled study, we introduce WOVEN, a benchmark and training source that factorizes transition supervision along independently controllable semantic dimensions using diverse, realistic rollouts from video-pretrained generative models. WOVEN reveals a substantial gap to human performance, motivating a direct study of trainability and transfer. Across controlled training subsets of only approximately 2,000 examples, post-training improves 22 of 26 external benchmarks, with gains up to 27.3 percentage points. Positive transfer persists across scales, and individual downstream tasks also benefit from distinct supervision sources; WOVEN examples can further replace part of task-specific training data while retaining comparable accuracy, establishing transition reasoning as a shared, trainable primitive. More importantly, controlled comparisons yield a training recipe organized by the query operator over : curricula sharing an operator exhibit consistently similar downstream gain profiles, whereas shared actions, scenes, or domains do not explain this grouping. Additional findings connect broader state interventions to robustness and delineate complementary supervision needs and transfer boundaries. Our work establishes visual transition reasoning as a reusable foundation for improving diverse downstream capabilities, paving the way for systematic visual world-model training guided by the structure of transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
World Model, Multimodality, Spatial Intelligence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RuHZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WOVEN, a synthetic dataset for visual transition reasoning, organized by scene, action, and reasoning operation. Through controlled training studies, the authors examine how this supervision transfers to external tasks. They find that reasoning operations better predict transfer than action similarity and propose data-selection guidelines, with prospective validation on held-out benchmarks.

Critical strengths

WOVEN provides a useful framework for studying transferable visual reasoning. Its structured taxonomy supports comparisons across supervision sources and helps connect spatial, physical, temporal, and embodied tasks.

The empirical study is extensive and includes meaningful controls. The authors examine data quality, answer-position bias, training-content controls, and subset-selection effects. Prospective evaluation provides additional evidence that the selection guidelines can help beyond the benchmarks used to derive them.

Critical weaknesses

The contribution of reasoning type is not fully isolated. Subsets share similar sizes and training protocols, but may differ in difficulty and input complexity. Comparisons with alternative spatial or video-reasoning supervision under matched budgets are also limited, leaving unclear which properties make WOVEN particularly effective.

Broad transfer does not imply a broadly beneficial training configuration. The 22-of-26 result selects the best configuration separately for each benchmark, while full-dataset training substantially degrades some capabilities. Appendix O supports targeted selection, but does not establish how to balance gains and losses when several capabilities matter simultaneously.

Questions for authors

Could the authors compare WOVEN with a representative spatial or video-reasoning training source under matched budgets, and assess whether subset difficulty or input complexity explains part of the observed transfer patterns?

What explains the negative transfer from full-dataset training, and how should users choose a mixture when a target application requires several reasoning capabilities? What gains and regressions should they expect from one fixed configuration?

7 comments
AuthorsOpenReview ↗

Response to Reviewer RuHZ

Dear Reviewer RuHZ,

Thank you for the thoughtful review, and for recognizing WOVEN as a useful framework for studying transferable visual reasoning, with an extensive study, meaningful controls, and prospective evaluation.

In response to your comments, we have made the following revisions to the paper (marked in blue):

  • Difficulty and input complexity (W1a, Q1): Added three controls for subset difficulty and question length (Appendix G.13; summary in §5.1).
  • Matched-budget comparison (W1b, Q1): Added a comparison with OpenSpatial on Qwen2.5-VL-3B and Qwen3-VL-8B (Appendix Q, Table 121).
  • Several capabilities (W2, Q2): Added what one full-set model gains and loses (Appendix G.13) and how to use the recipe when several capabilities matter (Appendix O).
  • Additional validation: Tested the training recipe on two more target domains (Appendix O, Table 119).

Yes, following your suggestion, we respectfully ran both analyses you mentioned, and both support the paper's conclusions: subset difficulty and input complexity do not explain the transfer pattern, and at a matched budget WOVEN shows favorable results over a representative spatial training source on two backbones. We also report what one fixed configuration gains and loses.

  • Difficulty and input complexity (W1a, Q1): with both as covariates, and under two further controls, the reasoning-family effect keeps its size and being the strongest perdictor (same-family coefficient 0.53 → 0.50, all p < 0.001).
  • Matched-budget comparison (W1b, Q1): WOVEN shows favorable results over JoyAI-Image's OpenSpatial data on Qwen2.5-VL-3B (49.6 vs. 45.8, better on 13 of 19 benchmarks) and on Qwen3-VL-8B (56.1 vs. 55.1, better on 12 of 19 benchmarks).
  • One configuration, several capabilities (W2, Q2): one fixed full-set model improves 14 of 19 benchmarks across different domains, and mixtures chosen by reasoning operation win on two additional target domains (+2.0 points, p = 0.009; +6.6 points, p = 0.002).
    • For the rest of the 26 benchmarks (i.e seven long-video benchmarks), they are too costly to run within the rebuttal period, and the 19 benchmarks here already span spatial, embodied, physical and temporal tasks. They will still be added in the camera-ready version.

Our detailed responses are provided below.

AuthorsOpenReview ↗

W1a / Q1. Subset difficulty and input complexity do not explain the transfer pattern.

Accordingly, we added this analysis to Appendix G.13 and summarized it in §5.1.

To test this directly, we measured each subset's difficulty (base-model accuracy on its own WOVEN test questions) and input length (words per question, including options), and each benchmark's difficulty (base accuracy). We then re-tested the family pattern of §5.1 (same family +0.32, same action −0.22, neither −0.23) under three controls, over all 55 subset pairs, with p-values from permuting subset labels:

ControlResultp
Pair regression with difficulty and length differences as covariatessame-family coefficient 0.53 → 0.50; same-action ≈ 00.0004
Pairs matched on difficulty (at most 5 points apart)same family +0.36 (9 pairs) vs. other −0.24 (16 pairs)0.0001
Benchmark difficulty regressed out of the profilessame family +0.32, same action −0.26, neither −0.23< 0.001

Under every control the reasoning-family effect keeps its size, so it reflects what the subsets teach rather than how hard or long their questions are.

AuthorsOpenReview ↗

W1b / Q1. At a matched budget, WOVEN shows favorable results over OpenSpatial on both backbones.

Accordingly, we added this comparison as Appendix Q (Table 121), with a pointer in §4.2. OpenSpatial is the spatial training data of JoyAI-Image (3D scans with camera and object transformation supervision). We trained Qwen2.5-VL-3B and Qwen3-VL-8B with the same recipe for both sources and evaluated them with one harness on 19 benchmarks:

Average over 19 benchmarks (%)Qwen2.5-VL-3BQwen3-VL-8B
Base46.755.0
+ OpenSpatial45.855.1
+ WOVEN49.656.1
Benchmarks where WOVEN is better13 of 1912 of 19

The comparison also shows which property makes WOVEN effective: its leads are on benchmarks about change (camera motion, manipulation, embodied action, physical dynamics), while OpenSpatial's smaller leads are on static spatial relations. This agrees with App. H: when the questions and format stay the same but the action–outcome rules are wrong, the gain disappears. The correct transition content is therefore what produces the effect.

Reviewer_uzC3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies the failure of current multimodal LLMs on spatial, embodied, physical, and temporal reasoning task.

It first constructs a benchmark for visual transition reasoning and comprehensively evaluate over 38 frontier MLLMs.

It then constructs training data with video generation model to complete the missing part in this field. It also provides training recipe for enhancing current models performance using the proposed data. Full-set training improves 12 of 26 external benchmarks.

Critical strengths

1, The dataset provides a useful way to study different queries over related visual transitions.

2, Human evaluation provides supporting quality check for the collected training data.

3, The learning and transfer gains are substantial.

4, The experiments comprehensive cover various foundation models and it provides a detailed training recipe.

Critical weaknesses

1, The results are across many diagnostics, benchmarks, and training choices, making the main insight hard to notice. It is unclear whether the main contribution is the dataset, one shared ability, or a general rule for choosing training data. A clearer link between the central claim and the experiments that establish it would make the contribution easier to assess.

2, The gain 22 out of 26 is achieved by selecting different best subsets and training stages for each benchmark. But the overall training only achieved enhancing 12 out of 26 benchmarks.

3, The data contribution needs a comparison with JoyAI-Image, which already uses about 3.3M OpenSpatial examples from 3D scans/videos and camera/object transformation supervision. JoyAI-Image also reports gains of 5.3 points in spatial average and 4.9 points on embodied QA (ERQA) for Qwen3-VL-8B. A comparison using the same model, training budget, and shared benchmarks would make the claimed benefit clearer.

Questions for authors

1, When generating different actions from the same initial image (Section 3.2), how do you keep unrelated scene details consistent? How do you check that the other answer options are wrong, rather than other plausible outcomes of the stated action (Section 3.3)?

2, For "what if the action never happened?", why is the first frame the correct answer (Table 1)? Could the scene still change over time even if that particular action does not happen?

3, Section 4.3 reports similar patterns of benchmark gains and tests replacing some benchmark training data with WOVEN. Why do these results show one shared transition ability rather than several skills improving together?

4, What will be the performance difference with the OpenSpatial data proposed by JoyAI-Image? It enhances spatial/embodied tasks for Qwen3-VL. Can you compare their training benefits using the same model?

10 comments
AuthorsOpenReview ↗

Response to Reviewer uzC3

Dear Reviewer uzC3,

Thank you for the detailed comments, we truly appreciate the time and effort you put into carefully reviewing our paper. We also thank you for recognizing our dataset as a useful way to study queries over related visual transitions, the human quality checks, the substantial transfer gains, and the breadth of models and training recipes.

In response to your comments, we have made the following revisions to the paper (marked in blue):

  • Main contribution (W1): Clarified the roles of the dataset, the shared ability and the training recipe (end of §1), and organized §4 by question (§4.1–4.3).
  • 12 vs. 22 (W2): Added an analysis of which benchmarks the full-set model does not improve and why (Appendix G.13).
  • Comparison with OpenSpatial (W3, Q4): Added a matched-budget comparison on Qwen2.5-VL-3B and Qwen3-VL-8B (Appendix Q, Table 121).
  • Data quality (Q1): Stated how unrelated details are kept consistent and why distractors are wrong by construction (§3.2, §3.3).
  • Counterfactual removal (Q2): Explained where removal questions are asked (Appendix B.7).
  • One shared ability (Q3): Reorganized §4.3 into two parts and added what separate effects would predict.
  • Additional validation: Tested the training recipe on two more target domains (Appendix O, Table 119).

Following your suggestion (W3, Q4), we compared WOVEN with JoyAI-Image's OpenSpatial data under the same model, training budget, and benchmarks. WOVEN shows favorable results on both models: 49.6 vs. 45.8 on Qwen2.5-VL-3B (better on 13 of 19 benchmarks), and 56.1 vs. 55.1 on Qwen3-VL-8B, the backbone JoyAI-Image reports on (better on 12 of 19). The full table is in our response to W3 / Q4 below.

We also tested the paper's proposed training recipe on two more target domains as additional prospective validation, and it holds in both: reasoning-matched supervision beats action-matched supervision by 2.0 points on real driving questions (p = 0.009) and by 6.6 points on real manipulation-ordering questions (p = 0.002), at equal budgets with 8 to 16 seeds per mixture.

Below, we respond to each weakness and question in turn.

AuthorsOpenReview ↗

W1. The dataset is our research tool, the shared ability is our central finding, and the training recipe builds on that finding.

Accordingly, we revised the paper (changes in blue): the end of §1 now states these three roles, §4 opens with the section that answers each question, and §4.2 and §4.3 are organized by claim.

  1. Which of the three is the main contribution? The shared ability. Our central question is whether reasoning about visual changes can be learned in one setting and transferred to others. WOVEN provides a controlled way to study this question: how visual transition reasoning can be learned, transferred, and systematically improved. We find that visual transition reasoning is one shared ability reused among multiple domains and is cheaply teachable. The training recipe builds on this finding: choose training data by reasoning operation, not by action, scene or domain.

  2. Which experiments establish each part?

    (a) Current models lack this ability: the best of 38 MLLMs reaches 65.8%, and humans reach 92.3% (§4.1; Table 10; App. C.2, F).

    (b) It is cheap to teach and is reused across tasks (§4.2). About 7 hours of generated video raise WOVEN accuracy from 26.4% to 89.3%, and to 88.2% on held-out scenes (App. G.2, G.16). One full-set model improves 12 of 26 external benchmarks (Table 2), and subsets of about 2,000 items improve 22 (Table 3). The gains pass our checks on selection and seeds (App. M), circular scoring (App. I), and other scales and model families (App. G.15, J), and as expected they are absent on the four benchmarks unrelated to transitions (§5.4).

    (c) It is one shared ability (§4.3; see Q3). A model trained on one type improves on the types it does not train on (43 of 44 comparisons; App. G.17, G.8, L), and not through a shortcut (Table 86; App. K). All 11 subsets improve mostly the same benchmarks (Fig. 4), and a subset that improves WOVEN accuracy more also transfers more (r = 0.86; App. G.13). Wrong transition rules give no gain (App. H), and WOVEN can replace a large share of a task’s own training data (Fig. 3; App. G.14).

    (d) The training recipe holds (§5). Subsets that teach the same reasoning operation transfer alike, and subsets that only share an action type do not (Fig. 4; §5.1; App. G.13). It also holds on held-out questions (App. O) and on the two new target domains above.

In the camera-ready version, we will give each question its own section: whether the ability is learnable and transferable, whether it is one shared ability, and the training recipe. In this revision we keep the section numbers unchanged so that they match your review.

AuthorsOpenReview ↗

W2. 12 and 22: which benchmarks one full-set model does not improve, and why.

Accordingly, we added this analysis to Appendix G.13, with a pointer in §4.2.

  1. How is the 22 counted, and can the selection add to it? A benchmark counts as improved when its best subset gains more than 2 points over the base. In Appendix M, we measure how much this selection can add by luck alone: on ten representative benchmarks, two from each domain, the actual gain is larger than what such luck can give, after Holm correction and in each of three GRPO seeds. The four benchmarks unrelated to transitions went through the same selection, and none gained more than 2 points.

  2. Then why does one full-set model not improve the other 14?

    These 14 benchmarks fall into two groups.

    (a) Four are outside transition reasoning (3DSRBench, CoreCognition, PerceptionTest, EgoTaskQA). No subset improves them either (at most +1.4 points), so transition data is not expected to help them (recipe item 4).

    (b) Ten are related to transition. Under the full-set model their change stays under our 2-point threshold, so we do not count them as improved. The reason is that the subsets do not help them equally. For example, the six causal and counterfactual subsets raise these ten by 2.4 points on average, but the four temporal subsets give them almost no gain (0.0 on average; above 2 points in only 4 of 40 cases). For the 12 improved benchmarks, both groups help (4.3 and 2.6 points). The full-set model mixes all 11 subsets, so for these ten the helpful data is mixed with data that does not help them, and the gain falls under the threshold. This also agrees with recipe item (3): what is learned from temporal supervision does not carry over to the other reasoning families. In addition, the gains are smallest on TVBench and TempCompass (2.8 and 2.4 points with the best subset), which test fine-grained temporal perception, such as speed and direction, and our current data has room to improve there.

In short, one full-set model improves the benchmarks that most kinds of transition data help, and the others need the right subsets. This is also why we need the recipe. With the best subset each of them gains 2.4 to 8.9 points, so the ability does transfer to them, and the recipe in §5 tells which subsets to choose.

Reviewer_F9d6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

WOVEN investigates visual transition reasoning as a shared training target for improving spatial, embodied, physical, and temporal reasoning in multimodal LLMs. The authors introduce a 36,076-example dataset and benchmark built from generated videos, organized by scene, action type, and reasoning operation. Post-training on controlled subsets of approximately 2,000 examples collectively improves 22 of 26 external benchmarks, with gains of up to 27.3 percentage points. Controlled transfer analyses and prospective experiments suggest that reasoning family provides a useful basis for selecting supervision across different actions. The benefits concentrate on transition-related tasks, while static perception and some reasoning capabilities require separate supervision.

Critical strengths

The controlled transfer experiments provide persuasive evidence that reasoning family is a useful organizing principle for selecting transferable supervision. Rather than reporting aggregate downstream gains alone, the authors train on 11 controlled subsets defined by action type and reasoning family, with approximately 2,000 examples each, and compare their transfer profiles across external benchmarks (Sections 4–5). Subsets sharing a reasoning family exhibit more similar patterns of downstream benefit than subsets sharing only an action type. After per-benchmark standardization, their mean pairwise correlations are +0.32, −0.22, and −0.23 for pairs sharing a reasoning family, an action type, or neither, respectively; the ordering persists under three alternative normalizations (Figure 4; Appendix G.13, Table 43). The prospective validation strengthens this finding by fixing supervision mixtures before training and evaluating them on benchmarks not used to derive the recipe, with the same 1,500-item budget and eight seeds per mixture (Appendix O). On camera-motion questions, selecting relevant reasoning supervision across different actions outperforms emphasizing camera-related actions (58.4% versus 57.8%); on agent-action questions, the recipe also outperforms uniform allocation (37.9% versus 36.4%). Together, the controlled comparisons, normalization checks, and prospective validation make a convincing practical contribution: they turn an observed transfer pattern into an actionable principle for allocating training data according to the reasoning a target task requires.

Critical weaknesses

The generality of the supervision-selection rule across visual domains is less firmly established than the transferability of the learned capability. The improvement on real-image SPAR-Bench questions from 26.4% to 54.9% provides convincing evidence that transition supervision from generated videos can benefit reasoning on real images (Appendix H). However, this comparison alone does not establish that selecting supervision by reasoning family is consistently more effective than selecting it by action type in new visual domains; that conclusion requires matched comparisons between training mixtures. The prospective experiments provide such evidence on held-out benchmarks and are a valuable validation (Appendix O). Nevertheless, the most direct reasoning-versus-action selection comparison focuses on camera-motion questions, where Recipe exceeds Camera-heavy by 0.58 percentage points (95% CI [0.02, 1.14], p = 0.04; Table 116). This supports the rule in the tested setting, while leaving its consistency across other action types and visual domains less clear. Repeating equal-budget comparisons of reasoning-matched and action-matched supervision across several distinct target domains would clarify how broadly the selection principle applies, beyond demonstrating that WOVEN-trained capabilities transfer.

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer F9d6

Dear Reviewer F9d6,

Thank you for the thoughtful review, and for recognizing that our controlled transfer experiments and prospective validation turn the transfer pattern into an actionable rule for allocating training data.

In response to your comments, we have made the following revision to the paper (marked in blue):

  • More target domains: Added the two new target domains to Appendix O (Table 119) and summarized the result on three target domains in §5.4.

Following your suggestion, we respectfully repeated the equal-budget comparison of reasoning-matched and action-matched supervision on two more target domains, and our results show that the selection rule holds in both: reasoning-matched supervision wins by 2.0 points on real driving questions (p = 0.009) and by 6.6 points on real manipulation-ordering questions (p = 0.002). Together with camera motion in App. O, the rule is now confirmed on three distinct target domains.

Our detailed response is provided below.

We followed the App. O protocol exactly (Qwen2.5-VL-3B, equal budgets, mixtures fixed before training) and trained over 70 additional models. Both target sets consist of real photographs and didn’t participate in deriving the rule. Accuracy (%), mean over seeds; base = untrained Qwen2.5-VL-3B:

Target domainBaseReasoning-matchedAction-matchedΔp
Driving, 1,500 items (16 vs. 16 seeds)30.144.042.1+1.90.08
Driving, 3,000 items (15 vs. 16 seeds)30.144.542.4+2.10.06
Driving, both budgets+2.00.009
Manipulation ordering, 1,500 items (8 vs. 8 seeds)14.625.919.2+6.60.002

The manipulation-ordering result is the sharpest test of the rule: the reasoning-matched mixture contains no manipulation data at all, yet it beats the mixture built from manipulation data by 6.6 points.

How the two target domains and the mixtures are defined:

  • Driving (forward prediction): 1,355 PhysBench questions showing a street scene with a drawn path, asking which view the camera will see after driving along it. Reasoning-matched: the forward causal-dynamics subsets of all four agent action types (82% of the budget). Action-matched (Navigation-heavy): the navigation causal and temporal subsets (543 items each), which even holds more navigation forward-prediction data (543 vs. 307 items). The p-value over both budgets is computed by shuffling seed labels within each budget.
  • Manipulation ordering (temporal ordering): 431 PhysBench questions asking for the correct order of 3 to 4 photos of a real robot or tabletop manipulation. Reasoning-matched (Temporal-heavy): 79% of the budget on the four temporal subsets, none of which shows manipulation. Action-matched (Manipulation-heavy): the manipulative causal and counterfactual subsets (543 items each).
  • Agent actions without the right operation: the App. O Recipe mixture (causal and counterfactual subsets of agent actions) scores 18.8 on manipulation ordering, 7.1 points below Temporal-heavy (p < 0.001). Showing the right actions does not substitute for teaching the right reasoning operation.

These results match what our recipe predicts: what transfers to a new domain is the reasoning operation, not the action, and this holds from generated video to real photographs.

In summary, we appreciate your thoughtful feedback, and following your suggestion, we have tested the selection rule on two more target domains, and it holds in both. We genuinely hope that these additional results help address your concern, and we would appreciate it if they could be reflected in your final score.

Thanks again for your time and effort put into reviewing our work.

AuthorsOpenReview ↗

A gentle follow-up on our response

Dear Reviewer F9d6,

We hope this message finds you well. We know this is a busy time, and we truly appreciate the time you have already spent on our paper. Your suggestion to test more target domains made the paper stronger, and we are grateful for it.

Since the discussion period ends on November 18, we would just like to kindly check whether our response and the revised paper have addressed your concerns. If anything is still unclear, please let us know, and we will be happy to clarify it right away. If your concerns have been addressed, we would be very grateful if you could consider updating your score.

Thank you again for your time and kind help.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.