- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Chain-of-Time, a zero-shot multi-turn prompting framework that leverages black-box Image Generation Models (IGMs) to perform physical simulation by sequentially generating intermediate visual frames as a proxy for visual working memory. Drawing inspiration from human mental simulation, the approach treats the internal activations of VLMs as implicit "de-rendering" to stabilize temporal predictions across dynamic physical domains without explicit physical engine integration, fine-tuning, or architectural modifications.
Critical strengths
The paper draws from cognitive psychology to naturally reframe complex, long-horizon physical prediction into an iterative, multi-step image generation task. This provides an intuitive paradigm for leveraging black-box IGMs as implicit physical simulators without additional training.
Critical weaknesses
- Lack of Technical Contribution and Insight:
The work presents a marginal technical contribution, relying primarily on a zero-shot prompting heuristic ("Chain-of-Time") applied to existing image generation models. Core concepts are insufficiently illustrated; for instance, "de-rendering" is loosely used to denote VLM internal activations without explicit explanation on the physical state. Furthermore, the lack of theoretical analysis or mechanistic investigation fails to explain how multi-step generation alters internal feature dynamics, reducing the contribution to an empirical observation rather than a principled algorithmic advance.
- Unfounded Claims on Physical Stabilization and Disregard for VLM Limitations:
The central assertion that multi-step prompting can "stabilize" physical simulations is fundamentally unconvincing and overlooks the inherent perceptual limitations of current VLM. The authors argue that chaining short-term predictions mimics human working memory, yet they completely ignore the well-documented vulnerabilities of VLMs and IGMs, such as severe visual hallucinations and cumulative error drift over time. Treating intermediate generated frames as reliable proxies for physical states merely patches superficial symptoms rather than addressing the base models' lack of true physical understanding. Consequently, the claim of enhanced stability lacks rigorous validation against the models' intrinsic failure modes.
Questions for authors
-
The paper claims de-rendering is extracting a low-level physical state. Can you give more explanation on what is captured by physical state more concretely with evidence. Since the tested models are black-box commercial image generation APIs, what empirical evidence guarantees that these models are genuinely performing internal physical state estimation? How accurate is such physical state estimation?
-
The manuscript would benefit significantly from a more comprehensive contextualization within contemporary literature. Specifically, the authors must directly discuss and contrast their sparse-prompt image generation paradigm with recent advancements in video generation and world models, established physical evaluation benchmarks (e.g., VideoPhy, PhysBench, Morpheus, CoPhyBench, PhysInOne), and recent explorations in physically plausible generative modeling incorporating physical laws or verifiable rewards.
-
In Appendix G, the comparison against native video generation models (e.g., LTX-Video) evaluates video models using sparse, low-frame inputs rather than their native continuous conditioning. Given that video diffusion models are inherently structured for dense, high-fps spatio-temporal prediction, is this a fair methodological comparison? Why were video models restricted from operating under conditions that leverage their native temporal architecture, and how can the authors claim that sparse IGM prompting holds a fundamental advantage without evaluating against state-of-the-art video world models under their optimal configurations?
-
The current empirical evaluation is heavily restricted to highly simplified, clean, and synthetic environments (e.g., pendulums, inclined planes, and basic bouncing balls) with minimal visual clutter. How does the proposed Chain-of-Time framework scale to real-world, unconstrained scenarios characterized by complex multi-body contact interactions, severe occlusions, non-rigid deformations, and dynamic lighting?
7 comments
RE:Official Review of Submission49658 by Reviewer ELDL Part 2
Treating intermediate generated frames as reliable proxies for physical states merely...
Another concern here appears to interpret “stability” as requiring the elimination of the base model’s perceptual limitations, and requiring the mechanistic interpretation of the model activations. This is certainly a worthwhile goal for some paper, but it was not the goal in our paper, as we explicitly reiterated. Our empirical claim concerns improved physical prediction over the evaluated horizons and conditions. Section 4 and Figure 4 evaluate predicted object positions or water levels against ground truth, using repeated trials and bootstrapped confidence intervals. Appendix H.3 additionally reports ten-second prediction experiments in Pendulum and Rolling on Track. These measurements permit a comparison between direct prediction and temporal decomposition even when both methods remain imperfect. The existence of residual errors does not establish that the measured reduction in error is invalid.
Beyond this, intermediate generated frames are predictions, not assumed ground-truth physical states. Their usefulness must be assessed against the external reference trajectories used in our evaluation. The method does not require each intermediate prediction to be error-free; whether shorter prediction steps provide sufficient benefit to offset accumulated errors is precisely an empirical question addressed by the rollout experiments. The working-memory analogy motivates this procedure, without establishing that the model implements human working memory or possesses a verified internal physics engine.
Moving beyond this, we believe that the description of the method as “patch[ing] superficial symptoms” does not identify which reported result or inference is inaccurate. An inference procedure can improve observable prediction accuracy, while leaving limitations in the underlying model unresolved. Our evidence supports conclusions about the evaluated physical observables, domains, and horizons; it does not establish unrestricted physical understanding or indefinite stability, and was not meant to establish that, nor does the work conclude that. In response to potential confusions here, we will make our scope more explicit in the wording of the manuscript.
Finally, we would like to refer the Reviewer to the failure mode that we tested in our paper: Direct Prediction baseline. This is the baseline where we showed that the model fails physical simulation tasks before chain-of-time is implemented, indicated by enormous error for each domain with each model. If the Reviewer considers the existing validation insufficient, we would deeply appreciate it if the Reviewer could specify the failure mode omitted from the analysis, and suggest how it could undermine the reported comparison, to identify the conclusion that consequently lacks support. That would provide a concrete methodological concern that we would then be happy to test and address, but at the moment we were unsure how to handle this specific comment.
The paper claims de-rendering is extracting a low-level physical state....
We would like to clarify that “physical state” refers to quantities needed to describe and predict a physical system. Depending on the domain, these can include object positions and velocities, pendulum angle and angular velocity, or fluid height. Predicting their evolution may additionally require physical parameters and constraints, such as gravity, flow rate, or the coupling between objects.
In terms of the empirical evidence, we again emphasize that model interals were not part of the stated investigation of our work. This is hardly rare. Several well-known methods like zero-shot learning never probe or change the internal of a LLM, but empirically they change and improve the behavior of a LLM. Our method stays at the behavioral level of the model, not the mechanistic level of the model. Section 2 expressly leaves open whether IGMs implement the proposed decomposition internally. As we said in reply to weakness 1 that we recognize that finding the underlying computation through mechanistic interpretation is an important direction, but it is not within the scope of our paper.
We believe the paper provides concrete behavioral evidence to support our stated conclusions. Section 4 and Figure 4 quantify predicted object positions or water levels against ground truth. Figures 6–7 examine intermediate trajectories in settings including nonlinear rolling, pendulum oscillation, bouncing, and coupled pulley motion, and they show that the model is able to correctly identify the underlying functional form of the physics dynamics. Appendix E.1 further varies initial water level and flow rate separately: predictions respond to the initial level but largely fail to distinguish flow rates. This identifies a specific sensitivity and failure in the generated predictions without establishing where that failure originates internally in their activation.
RE:Official Review of Submission49658 by Reviewer ELDL Part 3
Regarding accuracy, we report errors in observable physical predictions. We do not separately quantify the estimation error of an unobserved internal state. Prediction error can reflect errors in interpreting the input, evolving the estimated state, or rendering the output, and our measurements do not uniquely separate those sources. We will clarify this distinction wherever the schematic terminology could suggest a stronger mechanistic claim.
The manuscript would benefit significantly from a more comprehensive contextualization within contemporary literature.....
We agree that a more explicit comparison would improve the paper’s positioning. PhysBench, VideoPhy, and Morpheus are already cited in the manuscript; we will expand the discussion to explain their relationship to our experiments and include the additional relevant work identified by the reviewer.
However, the benchmarks address several related but distinct questions:
VideoPhy evaluates whether generated videos satisfy textual descriptions and physical commonsense across diverse material interactions. Our experiments instead condition on observed physical events and measure subsequent predictions against reference trajectories. These evaluations provide complementary information about generative physical prediction. PhysBench evaluates VLM understanding across physical properties, relationships, scenes, and dynamics. CoPhyBench tests reasoning from conditional video observations through prediction, physical calculation, and counterfactual questions. These are relevant assessments of physical reasoning, although their question-answering protocols differ from evaluating generated trajectories. Morpheus directly evaluates generated physical dynamics using measurements grounded in physical laws and invariants. This is particularly relevant: conservation-based evaluation can test plausibility beyond matching a particular reference trajectory, while our trajectory comparisons measure fidelity to the observed system’s continuation. We will make this relationship explicit. PhysInOne includes future-frame prediction, physical-property estimation, and physics-aware video generation with rich physical annotations. It is directly relevant to extending our evaluation to broader phenomena and more complex scenes. We will discuss that connection and the current limits of our evaluation coverage.
We will also distinguish our contribution from video world models and methods that incorporate physical constraints during training. Video foundation models such as LTX-video or Wan predict future world states through video generation with dedicated temporal architectures and extensive training. Our study asks how changing the inference procedure affects the predictions of a fixed pretrained image generator. It does not establish superiority over the broader class of video world models. Our analysis and comparison listed in Appendix G. and our response to reviewer eWVp also suggested that video generation model does not establish superiority over image generation models under Chain-of-time.
Similarly, approaches such as NewtonRewards use measurable physical constraints as rewards during post-training. Chain-of-Time changes inference without updating model weights or explicitly enforcing conservation laws. These approaches address different intervention points and may be complementary; our results do not establish that inference alone supplies the guarantees sought by physically constrained training.
In Appendix G, the comparison against native video generation models...
We would like to thank the Reviewer for raising this issue. However, we would also like to respectfully point out that the premise that the reported LTX-Video comparison restricts the model to sparse image inputs does not match the protocol described in Appendix G. Although we also explored sparse conditioning, the reported comparison uses all frames from the initial two-second segment as video conditioning and generates a two-second continuation at 30 FPS. LTX-Video therefore generates a dense video continuation using its temporal architecture. In the Appendix G., we write (additional emphasis added in CAPS here): “When we tested video generation models with the same sparse input data that we use for Chain-of-Time, their output was INCOHERENT, for example showing no motion at all. Therefore, in the following analysis we provided as input to LTX-video ALL FRAMES of the first 2 seconds of the video and prompted the model to generate 2 seconds into the future at 30 frames per second using a prompt similar to what we have in App. C”
RE:Official Review of Submission49658 by Reviewer ELDL Part 1
We would like to thank the Reviewer for the recognition of our work as a method to reframe a complex task as a solvable, theory-backed iterative method.
We also thank the Reviewer for their comments and questions, which we address in turn below:
We would like to thank the Reviewer for pointing this out. We agree with the Reviewer that our method relies primarily on a zero-shot prompting heuristics applied to existing image generation models. However, we would note that this does not take away from the novelty or the contribution that our method brings, which is proven empirically in our paper. Simplicity of implementation and the absence of additional training do not, by themselves, establish limited scientific contribution. If the concern is insufficient novelty, we would like to respectively ask the Reviewer to identify the prior work establishing the same substantive findings under comparable conditions, as we were not familiar with such work when we took on this investigation.
Regarding the use of “de-rendering” to denote VLM internal activations: we would like to clarify that our method is a computational model aimed to explain and improve the behavior of a model, and that our method does not touch on any VLM internal activations. We also specifically point this out in our paper: “However, even if current IGMs do not do (derendering, simulation, rendering) so implicitly, this framework suggests that if models can engage in short prediction steps, then with in-context reasoning we may be able to bring IGMs more in line with human-like physical simulation”.
Furthermore, the lack of theoretical analysis or mechanistic investigation fails to explain how multi-step generation alters internal feature dynamics, reducing the contribution to an empirical observation rather than a principled algorithmic advance.
We would like to emphasize that our model stays at the level of explaining the behavior of the image generation model using theories found in cognitive science. We do not claim our method refers to a mechanistic interpretation of any image generation model. We do agree that, after seeing our findings in the paper, a mechanistic interpretation on the internal activations of the VLM during the simulation would be a valuable thing to follow up on, but this is not within the intention and the scope of our paper.
To the point about algorithmic advances: in our view our method is not just an empirical observation, but a principled method, and this does not reduce its contribution. “Chain-of-time” proposes an algorithm on top of existing models, for initiating the image generation models to perform more human-like physical simulation.
We agree that perceptual errors, hallucinations, and accumulated prediction errors are important limitations of image generation models in general. However, we would like to respectfully push back on this being a specific issue for the work considered in paper, in several ways. First of all, many of these limitations are simply true of current models whether one considers our method or not, and our method does not set out to generally fix things like hallucinations. To our mind, pointing out that vision models suffer from hallucinations is similar to rejecting the original ‘chain of thought’ papers because they ignored the well-documented cases of hallucinations of LLMs. The fact that LLMs generally hallucinate is neither here nor there for the conclusions of that paper. Also, while a model's perceptual limitations might affect the accuracy we see in general, fundamentally it has nothing to do with whether a model could perform in a more human-like physical simulation under Chain-of-time. In other words, if a model has perceptual error PE without chain-of-time, and perceptual error PE with chain of time, and chain-of-time focuses on improving some performance metric M, then we would suggest that PE is not the relevant issue.
Further, and more directly to the Reviewer’s, the claim that “our paper ‘completely ignore[s]’ these limitations” is inconsistent with the analyses already included in the original manuscript. In particular, Section 5 and Figure 5 examine error accumulation during generated rollouts. Appendix E.1 provides a specific failure analysis in the Fluids domain: generated trajectories respond to initial water level but largely fail to distinguish different flow rates, and the resulting errors compound during rollout. So, we explicitly investigated cases where temporal decomposition fails to overcome a limitation in estimating physical parameters. In sum, we do not believe that it is accurate to characterize our paper as ignoring these issues.