acceptodds
Withdrawn from ICLR 2027

Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video generation models produce strikingly realistic sequences and are increasingly proposed as world models, yet recent benchmarks reveal pronounced deficits in their physical reasoning. This raises the question of whether these models internalize physical principles or merely reproduce familiar motion patterns. We address this by probing internal representations of video DiTs for simulator-derived ground-truth physical quantities spanning kinematic motion and rigid-body dynamics under gravity and contact. We find that these quantities are linearly decodable with high accuracy early in the denoising process, substantially outperforming a baseline decoded directly from the model's own noised latents—indicating that the relevant physical information is actively constructed during denoising, rather than already present in the input. Additionally, we show that activations at on-object tokens carry the relevant information and that quantities defined over multiple frames are readable from single latent-frames. Hence, information is sharply localized within the token sequence and is computed globally but stored locally. The probes further show partial extrapolation, transferring unchanged to scene variations and object configurations they were never fit on, so what they read is not simply a correlate of the training scenes. When fitted directly in the raw activation space, the probing directions can serve as steering vectors to change the model's output.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, video diffusion models, linear probing, activation steering, diffusion transformers, physical reasoning, representation analysis, causal intervention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rTyzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores whether physical quantities can be identified in the internal activations of large video diffusion models. Using simulator-generated videos with ground-truth physical quantities, the authors train linear probes to decode properties such as speed, direction, and gravity. The paper further analyzes where and when these signals appear in the model. It also evaluates transfer across changes in confounding factors. The results show strong linear decodability of physical information in large video diffusion models.

Critical strengths
  1. The experiments cover several large video diffusion models. This work analyzes three different backbones and evaluates the signals across model compoents, which provides a relatively comprehensive empirical characterization.

  2. The paper makes a meaningful effort to control possible confounding factors. In addition to matched latent baselines, the transfer experiments independently vary across confounding factors. These controls improve the rigor of the empirical analysis and help test whether the decoded signals are tied only to particular scene configurations.

Critical weaknesses
  1. The main conclusion relies heavily on linear probing, but the paper does not sufficiently justify why linear decodability should be interpreted as evidence of an internal physical representation. Without nonlinear comparisons or further analysis, the experiments establish linear accessibility, but not the rigorous representation form of the underlying physical information.

  2. The connection between the controlled variables and the claimed physical concepts has not been rigorously identified. A probe predicting gravity, speed, or momentum could still rely on correlated visual or dynamical features. This means that the decoded directions cannot be uniquely attributed to the claimed physical quantities.

  3. The practical value of these linearly decodable signals remains unclear. Given that prior work has already probed physical information in video-model representations, the paper does not demonstrate a concrete application showing how these representations can be used to improve video generation (in terms of physical correctness) or the diagnosis of generated videos.

Questions for authors
  1. To what extent do the controlled variations isolate the intended physical concept? While the confounder tests seem useful, it remains unclear whether or why they are sufficient to rule out correlations and uniquely attribute the decoded direction to the claimed physical quantity

  2. Beyond model diagnosis, what practical uses do the authors see for these representations? How can they be used to improve physical generation or control in a concrete way? Or even to diagnose a model or the generated video?

Reviewer_EA7xOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work investigates whether physical quantities can be predicted from the activation features of image-to-video diffusion transformers. Three video DiTs are evaluated: Wan 2.1 14B, CogVideoX-1.5 5B, and Open-Sora 2.0-11B. Linear probing is performed using simulation data to assess the predictability of physical quantities through $R^2$ scores. Additional steering experiments using simulation data demonstrate the feasibility of steering feature vectors.

Critical strengths

Using a well-constructed simulation dataset, this work analyzes rigid-body mechanics in image-to-video generation and demonstrates that basic physical properties such as direction can be predicted even at high noise levels. As the noise level decreases, more precise predictions of physical quantities become possible.

Critical weaknesses

Rationale for feature selection

  • Activations. The rationale for selecting layer activations is not sufficiently provided (line 83). An analysis of the characteristics of these features and whether they are more informative than alternative features is important. Furthermore, this work uses these activation features for spatio-temporal localization (Sec. 3.3). The effectiveness of this approach should be evaluated against existing methods [1, 2].
  • Layers. Physical quantity analysis (Sec. 3.2) uses the mean over blocks. Spatio-temporal localization (Sec. 3.3) uses a fixed mid-depth layer (21). Steering (Sec. 3.5) uses manually selected model-specific layers (Wan 2.1: 21, CogVideoX-1.5: 3, Open-Sora 2.0: 16). Generalizing experimental findings from these manually selected layers to the overall properties of Video DiTs is not sufficiently justified.

Limited research scope

  • Rigid-body mechanics. Although this work focuses exclusively on rigid-body mechanics, the feature selection, and layer selection strategies as well as the steering strategy (SELRP) appear somewhat heuristic. This raises concerns about interpreting the findings as general properties of video DiTs.
  • Image-to-video models. This work considers only image-to-video models. Consequently, it remains unclear whether the observed properties also hold across other video DiT architectures such as text-to-video models, world models, and world-action-models.
  • Steering. The use of simulation data for fine-grained analysis is reasonable (Sec. 3.1-3.4). However, the practical applicability of the resulting understanding of physical quantities in video DiTs to video controllability should be validated on real-world videos and compared with existing studies [3, 4]. Additional concerns arise from the reliance on heuristically selected model-specific layers in the simulation setting.

[1] Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformers, CVPR 2026.

[2] ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features, ICML 2026.

[3] Learning Plug-and-play Memory for Guiding Video Diffusion Models, arXiv:2511.19229.

[4] Video Diffusion Models are Training-free Motion Interpreter and Controller, NeurIPS 2024.

Questions for authors
  • What motivated the selection of these models and the I2V task? If the goal is to identify general properties of video generation or Video DiTs, evaluation across diverse architectures such as T2V and I2V as well as WM and WAM is important.
  • What motivated the choice of SLERP as the interpolation strategy for steering? Is spherical geometry guaranteed in this setting or does SLERP demonstrate superior empirical performance?
  • Can the layer activation approach proposed in this work enable more fine-grained applications than existing localization methods [1, 2] and steering methods [3, 4]?
Reviewer_AXfjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the intermediate activations of video diffusion Transformers contain physical quantities that can be read out linearly. The authors use videos of rolling, collisions, and projectile motion with precise physical labels, training linear probes on three frozen video generation models to analyze how readable parameters like speed, direction, momentum, and gravity are across different layers, noise levels, and spatial locations. The paper also tests calibrated cross-scene transfer and uses full-token probe directional interventions in the generation process, changing the motion direction of the ball in some samples.

Critical strengths

It has a relatively systematic controlled experimental design. The paper uses precise simulator labels, analyzes three video generation models, and provides controls like input latent variables, positions near objects versus background, as well as video-level cross-validation. This makes experiments under different conditions more comparable and also boosts the credibility of the empirical results.

Another good point is that the paper extends the representational analysis to actual generative interventions. Experiments with directional interventions on fixed random seeds show that certain internal interventions can indeed change the direction of motion in the generated videos. Compared to just reporting probe predictions, this offers additional and valuable evidence, even though the conclusions are currently limited to motion direction.

Critical weaknesses

The new contributions relative to the closest previous work haven't been clearly distinguished yet. While this paper discusses some previous works, it lacks targeted experiments in comparable settings to show exactly what new findings go beyond existing conclusions. Bigger models, more targets, and expanded scenes have some empirical value, but it's still unclear if these expansions bring enough really new insights. This is a major factor affecting how I judge the strength of the contributions.

Existing experiments haven't really distinguished between learning physical knowledge and learning motion patterns. Video generation models can naturally produce things like falling or colliding objects, but these behaviors could just come from learning motion patterns in the training data and don't necessarily mean the model understands the physical relationships behind them. So even reading out physical quantities like speed or gravity from intermediate features might just be decoding the motion patterns the model has already learned. For example, being able to estimate gravity from parabolic motion features doesn't prove that the model uses the concept of gravity when generating that motion. Likewise, changing the direction of generated motion could just be controlling the motion pattern. The paper discusses whether the model internalizes physics or just reproduces familiar motion patterns, but current evidence doesn't really distinguish between these two explanations. This makes me still question the scope of the paper's physics interpretations and how significant these readouts are compared to general motion representation analysis.

Questions for authors

Please address the questions raised in Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.