- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether video diffusion models can learn to reason about physical interactions of 2D gear systems. E.g., once a gear starts turning, the rotation direction of every other connected gears follow its chain of interactions. By fine-tuning Wan2.1 and probing its internal features, the authors discovered two main reasoning strategies: a breadth-first-search-like mechanism layer-by-layer for short chains, and a divide-and-conquer-like mechanism for longer chains where the model figures out local gear rotations first and then merges them for global consistency. Ultimately, the results show that while video models can learn structured physical reasoning, their ability to generalize is strongly tied to the complexity of their training data.
Critical strengths
-
Very interesting topic. Instead of just evaluating the visual quality, the paper investigates capability to learn the structured physical reasoning itself, which gives a useful perspective on the capabilities and limitations of generative models.
-
Well-scope formulation. Gear systems provide explicit yet complex physical constraints. The authors can systematically vary gear count, height, etc., making it easier to study under ablation setting.
-
Interesting generalization findings. The observation that a model trained on systems with up to 10 gears can handle systems containing 100 gears when the kinematic height remains within the training distribution is notable.
-
I also appreciate the PCA and MLP probing analysis which gives deeper insight on how kinematic information is represented across transformer layers.
Critical weaknesses
-
Evidences from PCA and MLP may not be conclusive because even if the kinematic information is presented in intermediate features, it is not necessarily that the model actually uses the proposed algorithms to generate the final outputs.
-
The scope of 2D gear system is narrow. There is big gap from understanding gears to generalize to real-world video understanding tasks. I understand that one paper could not do everything, but perhaps this should be in a discussion section.
-
The study relies on a single model configuration. The experiments primarily investigate Wan2.1 with 1.3B parameters. Perhaps many findings are not true on different models trained with different technics/data.
Questions for authors
-
How much of the change in reasoning behavior is attributable to kinematic height, as opposed to differences in gear count and topology?
-
On temporal variation (sec. E3), the paper notes that individual frames do not encode identical parity information and that temporal pooling hurts performance. I think this is counter-intuitive (although I believe in this paper's empirical results), so could the authors clarify how the temporal dimension interacts with the spatial reasoning steps that leads to this worse result from temporal pooling?
-
Could you test more on different generation models?
3 comments
Thank you, the rebuttal is reasonable, so I kept my weak accept rating.
Author Response to Reviewer rLFZ (2/2)
We thank the reviewer for their response. For completeness, we address the remaining questions raised in the original review below.
Q1: How much of the change in reasoning behavior is attributable to kinematic height, as opposed to differences in gear count and topology?
We believe that the kinematic height in the training distribution is the primary factor influencing the model's reasoning behavior, rather than gear count or graph topology:
- Topology (Branching Factor): In Table 2, we train our model solely on linear graphs yet demonstrate that it still learns parallel-BFS-like reasoning and generalizes to non-linear graphs (arbitrary tree topologies). This indicates that graph topology (specifically, the branching factor) is not the primary driver of the reasoning strategy.
- Gear Count vs. Kinematic Height: Table 1 shows that when the model is trained up to a specific kinematic height and gear count, it exhibits reasonable generalization to unseen gear counts within the in-distribution kinematic height, whereas performance drops sharply at out-of-distribution kinematic heights. This indicates that the model's reasoning capability is governed primarily by kinematic height rather than total gear count.
Q2: Temporal variation
Thank you for this insightful question. We believe this temporal variation arises because not all frames are equally responsible for inferring rotational parity:
- The first frame does not require inferring rotational parity, as the initial state of the system is already provided as input.
- Even in subsequent frames, the full-attention mechanism allows the model to synchronize rotational parity information across frames at any time rather than independently computing it at every frame.
Because each frame is not enforced to execute the same reasoning operation, concatenation of all the feature vectors outperforms average temporal pooling in our MLP probing experiments.
Q3: testing on different generation models
Please refer to our original response to W3. Additionally, we have updated our results on CogVideoX-5B (https://cog-dit.github.io/#rebuttal-only). When trained on systems with up to 20 gears, CogVideoX-5B exhibits both parallel-BFS-like reasoning and a divide-and-conquer-like strategy, with parallel-BFS-like reasoning occurring up to a kinematic height of approximately 10. As shown in the evaluation table (https://cog-dit.github.io/#rebuttal-only), this leads to strong generalization to unseen gear counts up to a kinematic height of 10, and weaker generalization beyond that depth. This behavior is consistent with our experiments on Wan2.1-1.3B (Table 15, Figure 18, and Figure 25 in Appendix), further suggesting that our findings are not specific to Wan2.1-1.3B.
Author Response to Reviewer rLFZ (1/2)
We thank the reviewer for their thoughtful feedback and constructive comments. We are glad the reviewer found the topic interesting and well-scoped, and that our generalization findings and feature probing analyses provide useful insights. Below, we address each of the points in detail.
W1: Evidences from PCA and MLP may not be conclusive
We agree with the reviewer that feature probing does not indicate the model is executing exact algorithms as formal code. To reflect this, we intentionally use the suffix "-like" (i.e., parallel-BFS-like and divide-and-conquer-like) to describe high-level computational patterns.
For example, our interpretation of parallel-BFS-like reasoning is grounded in Figure 3, which shows that rotational parity is resolved depth-by-depth across transformer layers, starting from the driving gear. This order closely resembles a parallel BFS traversal. While the transformer may not run the exact algorithm, the evolution of its internal representations mirrors how these classical algorithms solve the task, hence we refer to this as parallel-BFS-like reasoning .
Furthermore, we have strong indications that these discovered reasoning mechanisms contribute to the final output. For instance, the model's generalization to unseen gear counts when kinematic height remains in-distribution (Table 1) is explained by the hypothesis that the model executes a depth-bounded parallel-BFS-like mechanism internally, which is consistent with our feature probing analysis visualized in Figures 3c and 4.
W2: The scope of 2D gear system is narrow
Our goal is to probe the upper-bound capabilities of video transformers to perform kinematic-chain reasoning with minimal confounding factors: if a model struggles with kinematic reasoning in this simplified setting, it will likely struggle even more in complex physical environments. Furthermore, narrowing the domain allows us to investigate internal reasoning mechanisms in depth rather than superficially covering broader settings. Specifically, the gear setting is particularly well-suited for our initial analysis for two reasons: (1) the gears' spatial positions do not change, allowing us to focus on the model's ability to infer rotational parity while easily monitoring its internal representations, and (2) all gears move simultaneously, enabling the study of long-chain kinematic reasoning without requiring long frame sequences.
Nevertheless, we agree that extending our analysis to other physical phenomena is a valuable next step, and we have revised the Discussion and Conclusion section of the paper to discuss this direction.
W3: Analysis with a different video backbone
To verify that our findings are not specific to Wan2.1-1.3B (a 30-layer DiT with cross-attention), we conducted additional experiments on CogVideoX-5B (a 42-layer MMDiT). Due to time and compute constraints during the rebuttal period, we partially fine-tuned CogVideoX-5B on systems with gear counts and kinematic heights up to 20. Despite limited training, MLP probing (visualized at https://cog-dit.github.io/#rebuttal-only) already reveals the same linear trend observed in Wan2.1-1.3B (Figures 3.d and 4 in the paper): parity prediction accuracy increases approximately linearly across transformer layers, which is characteristic of parallel BFS-like reasoning. Furthermore, as shown in the evaluation table on the rebuttal webpage (https://cog-dit.github.io/#rebuttal-only), when evaluated on unseen gear counts (e.g., 70, 90, 100 gears), the model generalizes to in-distribution kinematic heights (e.g., depths 8, 9, 10), while exhibiting low performance at out-of-distribution depths (e.g., depths 30, 40, etc.).
Please note that because training has not yet fully converged under the limited rebuttal budget, we currently focus our analysis on kinematic heights up to approximately 10, where the model performs reliably. We will continue updating the supplemental webpage with further results as training progresses. Nevertheless, these preliminary results already confirm that our core findings are not specific to Wan2.1-1.3B.