- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces WOVEN, a synthetic dataset for visual transition reasoning, organized by scene, action, and reasoning operation. Through controlled training studies, the authors examine how this supervision transfers to external tasks. They find that reasoning operations better predict transfer than action similarity and propose data-selection guidelines, with prospective validation on held-out benchmarks.
Critical strengths
WOVEN provides a useful framework for studying transferable visual reasoning. Its structured taxonomy supports comparisons across supervision sources and helps connect spatial, physical, temporal, and embodied tasks.
The empirical study is extensive and includes meaningful controls. The authors examine data quality, answer-position bias, training-content controls, and subset-selection effects. Prospective evaluation provides additional evidence that the selection guidelines can help beyond the benchmarks used to derive them.
Critical weaknesses
The contribution of reasoning type is not fully isolated. Subsets share similar sizes and training protocols, but may differ in difficulty and input complexity. Comparisons with alternative spatial or video-reasoning supervision under matched budgets are also limited, leaving unclear which properties make WOVEN particularly effective.
Broad transfer does not imply a broadly beneficial training configuration. The 22-of-26 result selects the best configuration separately for each benchmark, while full-dataset training substantially degrades some capabilities. Appendix O supports targeted selection, but does not establish how to balance gains and losses when several capabilities matter simultaneously.
Questions for authors
Could the authors compare WOVEN with a representative spatial or video-reasoning training source under matched budgets, and assess whether subset difficulty or input complexity explains part of the observed transfer patterns?
What explains the negative transfer from full-dataset training, and how should users choose a mixture when a target application requires several reasoning capabilities? What gains and regressions should they expect from one fixed configuration?
7 comments
W1a / Q1. Subset difficulty and input complexity do not explain the transfer pattern.
Accordingly, we added this analysis to Appendix G.13 and summarized it in §5.1.
To test this directly, we measured each subset's difficulty (base-model accuracy on its own WOVEN test questions) and input length (words per question, including options), and each benchmark's difficulty (base accuracy). We then re-tested the family pattern of §5.1 (same family +0.32, same action −0.22, neither −0.23) under three controls, over all 55 subset pairs, with p-values from permuting subset labels:
| Control | Result | p |
|---|---|---|
| Pair regression with difficulty and length differences as covariates | same-family coefficient 0.53 → 0.50; same-action ≈ 0 | 0.0004 |
| Pairs matched on difficulty (at most 5 points apart) | same family +0.36 (9 pairs) vs. other −0.24 (16 pairs) | 0.0001 |
| Benchmark difficulty regressed out of the profiles | same family +0.32, same action −0.26, neither −0.23 | < 0.001 |
Under every control the reasoning-family effect keeps its size, so it reflects what the subsets teach rather than how hard or long their questions are.
W1b / Q1. At a matched budget, WOVEN shows favorable results over OpenSpatial on both backbones.
Accordingly, we added this comparison as Appendix Q (Table 121), with a pointer in §4.2. OpenSpatial is the spatial training data of JoyAI-Image (3D scans with camera and object transformation supervision). We trained Qwen2.5-VL-3B and Qwen3-VL-8B with the same recipe for both sources and evaluated them with one harness on 19 benchmarks:
| Average over 19 benchmarks (%) | Qwen2.5-VL-3B | Qwen3-VL-8B |
|---|---|---|
| Base | 46.7 | 55.0 |
| + OpenSpatial | 45.8 | 55.1 |
| + WOVEN | 49.6 | 56.1 |
| Benchmarks where WOVEN is better | 13 of 19 | 12 of 19 |
The comparison also shows which property makes WOVEN effective: its leads are on benchmarks about change (camera motion, manipulation, embodied action, physical dynamics), while OpenSpatial's smaller leads are on static spatial relations. This agrees with App. H: when the questions and format stay the same but the action–outcome rules are wrong, the gain disappears. The correct transition content is therefore what produces the effect.
Response to Reviewer RuHZ
Dear Reviewer RuHZ,
Thank you for the thoughtful review, and for recognizing WOVEN as a useful framework for studying transferable visual reasoning, with an extensive study, meaningful controls, and prospective evaluation.
In response to your comments, we have made the following revisions to the paper (marked in blue):
Yes, following your suggestion, we respectfully ran both analyses you mentioned, and both support the paper's conclusions: subset difficulty and input complexity do not explain the transfer pattern, and at a matched budget WOVEN shows favorable results over a representative spatial training source on two backbones. We also report what one fixed configuration gains and loses.
Our detailed responses are provided below.