acceptodds
Under review as a conference paper at ICLR 2027

Chain of Time: In-Context Physical Simulation with Image Generation Models

Abstract

We propose a novel cognitively-inspired prompting method to stabilize and interpret the physical simulation produced by autoregressive image generation models, called Chain-of-Time. Chain-of-Time involves generating a series of intermediate images during a simulation, and it is motivated by mental simulation in humans, as well as in-context reasoning in LLMs. Chain-of-Time is used at inference time, and requires no additional fine-tuning. We apply this method to synthetic and real-world domains, including 2-D graphics simulations and natural 3-D videos. These domains cover various physical dynamics, ranging from rigid-body rolling to harmonic oscillation and pulley systems. We found that Chain-of-Time simulation stabilizes the physics simulated by the state-of-the-art image generation models. Beyond examining performance, we also analyzed the specific states of the world simulated by an image model at each time step, which sheds light on the dynamics underlying these simulations. This analysis reveals insights that are hidden from traditional evaluations of physical reasoning, highlighting cases where image generation models can simulate physical properties such as velocity, gravity, oscillations, and pulley systems. Like humans, image generation models are able to accurately predict future world states over short time periods. By providing a mechanism analogous to working memory, Chain-of-Time enables a model to stitch together a series of short-term predictions into an accurate long-term simulation.

Primary area:
interpretability and explainable AI
Keywords:
In-context Learning, Physical Simulation, Intuitive Physics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ELDLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Chain-of-Time, a zero-shot multi-turn prompting framework that leverages black-box Image Generation Models (IGMs) to perform physical simulation by sequentially generating intermediate visual frames as a proxy for visual working memory. Drawing inspiration from human mental simulation, the approach treats the internal activations of VLMs as implicit "de-rendering" to stabilize temporal predictions across dynamic physical domains without explicit physical engine integration, fine-tuning, or architectural modifications.

Critical strengths

The paper draws from cognitive psychology to naturally reframe complex, long-horizon physical prediction into an iterative, multi-step image generation task. This provides an intuitive paradigm for leveraging black-box IGMs as implicit physical simulators without additional training.

Critical weaknesses
  1. Lack of Technical Contribution and Insight:

The work presents a marginal technical contribution, relying primarily on a zero-shot prompting heuristic ("Chain-of-Time") applied to existing image generation models. Core concepts are insufficiently illustrated; for instance, "de-rendering" is loosely used to denote VLM internal activations without explicit explanation on the physical state. Furthermore, the lack of theoretical analysis or mechanistic investigation fails to explain how multi-step generation alters internal feature dynamics, reducing the contribution to an empirical observation rather than a principled algorithmic advance.

  1. Unfounded Claims on Physical Stabilization and Disregard for VLM Limitations:

The central assertion that multi-step prompting can "stabilize" physical simulations is fundamentally unconvincing and overlooks the inherent perceptual limitations of current VLM. The authors argue that chaining short-term predictions mimics human working memory, yet they completely ignore the well-documented vulnerabilities of VLMs and IGMs, such as severe visual hallucinations and cumulative error drift over time. Treating intermediate generated frames as reliable proxies for physical states merely patches superficial symptoms rather than addressing the base models' lack of true physical understanding. Consequently, the claim of enhanced stability lacks rigorous validation against the models' intrinsic failure modes.

Questions for authors
  1. The paper claims de-rendering is extracting a low-level physical state. Can you give more explanation on what is captured by physical state more concretely with evidence. Since the tested models are black-box commercial image generation APIs, what empirical evidence guarantees that these models are genuinely performing internal physical state estimation? How accurate is such physical state estimation?

  2. The manuscript would benefit significantly from a more comprehensive contextualization within contemporary literature. Specifically, the authors must directly discuss and contrast their sparse-prompt image generation paradigm with recent advancements in video generation and world models, established physical evaluation benchmarks (e.g., VideoPhy, PhysBench, Morpheus, CoPhyBench, PhysInOne), and recent explorations in physically plausible generative modeling incorporating physical laws or verifiable rewards.

  3. In Appendix G, the comparison against native video generation models (e.g., LTX-Video) evaluates video models using sparse, low-frame inputs rather than their native continuous conditioning. Given that video diffusion models are inherently structured for dense, high-fps spatio-temporal prediction, is this a fair methodological comparison? Why were video models restricted from operating under conditions that leverage their native temporal architecture, and how can the authors claim that sparse IGM prompting holds a fundamental advantage without evaluating against state-of-the-art video world models under their optimal configurations?

  4. The current empirical evaluation is heavily restricted to highly simplified, clean, and synthetic environments (e.g., pendulums, inclined planes, and basic bouncing balls) with minimal visual clutter. How does the proposed Chain-of-Time framework scale to real-world, unconstrained scenarios characterized by complex multi-body contact interactions, severe occlusions, non-rigid deformations, and dynamic lighting?

7 comments
AuthorsOpenReview ↗

RE:Official Review of Submission49658 by Reviewer ELDL Part 1

We would like to thank the Reviewer for the recognition of our work as a method to reframe a complex task as a solvable, theory-backed iterative method.

We also thank the Reviewer for their comments and questions, which we address in turn below:

The work presents a marginal technical contribution...

We would like to thank the Reviewer for pointing this out. We agree with the Reviewer that our method relies primarily on a zero-shot prompting heuristics applied to existing image generation models. However, we would note that this does not take away from the novelty or the contribution that our method brings, which is proven empirically in our paper. Simplicity of implementation and the absence of additional training do not, by themselves, establish limited scientific contribution. If the concern is insufficient novelty, we would like to respectively ask the Reviewer to identify the prior work establishing the same substantive findings under comparable conditions, as we were not familiar with such work when we took on this investigation.

Core concepts are insufficiently illustrated...

Regarding the use of “de-rendering” to denote VLM internal activations: we would like to clarify that our method is a computational model aimed to explain and improve the behavior of a model, and that our method does not touch on any VLM internal activations. We also specifically point this out in our paper: “However, even if current IGMs do not do (derendering, simulation, rendering) so implicitly, this framework suggests that if models can engage in short prediction steps, then with in-context reasoning we may be able to bring IGMs more in line with human-like physical simulation”.

Furthermore, the lack of theoretical analysis or mechanistic investigation fails to explain how multi-step generation alters internal feature dynamics, reducing the contribution to an empirical observation rather than a principled algorithmic advance.

We would like to emphasize that our model stays at the level of explaining the behavior of the image generation model using theories found in cognitive science. We do not claim our method refers to a mechanistic interpretation of any image generation model. We do agree that, after seeing our findings in the paper, a mechanistic interpretation on the internal activations of the VLM during the simulation would be a valuable thing to follow up on, but this is not within the intention and the scope of our paper.

To the point about algorithmic advances: in our view our method is not just an empirical observation, but a principled method, and this does not reduce its contribution. “Chain-of-time” proposes an algorithm on top of existing models, for initiating the image generation models to perform more human-like physical simulation.

The central assertion that multi-step prompting can "stabilize" physical simulations is fundamentally unconvincing and overlooks the inherent perceptual limitations of current VLM....

We agree that perceptual errors, hallucinations, and accumulated prediction errors are important limitations of image generation models in general. However, we would like to respectfully push back on this being a specific issue for the work considered in paper, in several ways. First of all, many of these limitations are simply true of current models whether one considers our method or not, and our method does not set out to generally fix things like hallucinations. To our mind, pointing out that vision models suffer from hallucinations is similar to rejecting the original ‘chain of thought’ papers because they ignored the well-documented cases of hallucinations of LLMs. The fact that LLMs generally hallucinate is neither here nor there for the conclusions of that paper. Also, while a model's perceptual limitations might affect the accuracy we see in general, fundamentally it has nothing to do with whether a model could perform in a more human-like physical simulation under Chain-of-time. In other words, if a model has perceptual error PE without chain-of-time, and perceptual error PE with chain of time, and chain-of-time focuses on improving some performance metric M, then we would suggest that PE is not the relevant issue.

Further, and more directly to the Reviewer’s, the claim that “our paper ‘completely ignore[s]’ these limitations” is inconsistent with the analyses already included in the original manuscript. In particular, Section 5 and Figure 5 examine error accumulation during generated rollouts. Appendix E.1 provides a specific failure analysis in the Fluids domain: generated trajectories respond to initial water level but largely fail to distinguish different flow rates, and the resulting errors compound during rollout. So, we explicitly investigated cases where temporal decomposition fails to overcome a limitation in estimating physical parameters. In sum, we do not believe that it is accurate to characterize our paper as ignoring these issues.

AuthorsOpenReview ↗

RE:Official Review of Submission49658 by Reviewer ELDL Part 2

Treating intermediate generated frames as reliable proxies for physical states merely...

Another concern here appears to interpret “stability” as requiring the elimination of the base model’s perceptual limitations, and requiring the mechanistic interpretation of the model activations. This is certainly a worthwhile goal for some paper, but it was not the goal in our paper, as we explicitly reiterated. Our empirical claim concerns improved physical prediction over the evaluated horizons and conditions. Section 4 and Figure 4 evaluate predicted object positions or water levels against ground truth, using repeated trials and bootstrapped confidence intervals. Appendix H.3 additionally reports ten-second prediction experiments in Pendulum and Rolling on Track. These measurements permit a comparison between direct prediction and temporal decomposition even when both methods remain imperfect. The existence of residual errors does not establish that the measured reduction in error is invalid.

Beyond this, intermediate generated frames are predictions, not assumed ground-truth physical states. Their usefulness must be assessed against the external reference trajectories used in our evaluation. The method does not require each intermediate prediction to be error-free; whether shorter prediction steps provide sufficient benefit to offset accumulated errors is precisely an empirical question addressed by the rollout experiments. The working-memory analogy motivates this procedure, without establishing that the model implements human working memory or possesses a verified internal physics engine.

Moving beyond this, we believe that the description of the method as “patch[ing] superficial symptoms” does not identify which reported result or inference is inaccurate. An inference procedure can improve observable prediction accuracy, while leaving limitations in the underlying model unresolved. Our evidence supports conclusions about the evaluated physical observables, domains, and horizons; it does not establish unrestricted physical understanding or indefinite stability, and was not meant to establish that, nor does the work conclude that. In response to potential confusions here, we will make our scope more explicit in the wording of the manuscript.

Finally, we would like to refer the Reviewer to the failure mode that we tested in our paper: Direct Prediction baseline. This is the baseline where we showed that the model fails physical simulation tasks before chain-of-time is implemented, indicated by enormous error for each domain with each model. If the Reviewer considers the existing validation insufficient, we would deeply appreciate it if the Reviewer could specify the failure mode omitted from the analysis, and suggest how it could undermine the reported comparison, to identify the conclusion that consequently lacks support. That would provide a concrete methodological concern that we would then be happy to test and address, but at the moment we were unsure how to handle this specific comment.

The paper claims de-rendering is extracting a low-level physical state....

We would like to clarify that “physical state” refers to quantities needed to describe and predict a physical system. Depending on the domain, these can include object positions and velocities, pendulum angle and angular velocity, or fluid height. Predicting their evolution may additionally require physical parameters and constraints, such as gravity, flow rate, or the coupling between objects.

In terms of the empirical evidence, we again emphasize that model interals were not part of the stated investigation of our work. This is hardly rare. Several well-known methods like zero-shot learning never probe or change the internal of a LLM, but empirically they change and improve the behavior of a LLM. Our method stays at the behavioral level of the model, not the mechanistic level of the model. Section 2 expressly leaves open whether IGMs implement the proposed decomposition internally. As we said in reply to weakness 1 that we recognize that finding the underlying computation through mechanistic interpretation is an important direction, but it is not within the scope of our paper.

We believe the paper provides concrete behavioral evidence to support our stated conclusions. Section 4 and Figure 4 quantify predicted object positions or water levels against ground truth. Figures 6–7 examine intermediate trajectories in settings including nonlinear rolling, pendulum oscillation, bouncing, and coupled pulley motion, and they show that the model is able to correctly identify the underlying functional form of the physics dynamics. Appendix E.1 further varies initial water level and flow rate separately: predictions respond to the initial level but largely fail to distinguish flow rates. This identifies a specific sensitivity and failure in the generated predictions without establishing where that failure originates internally in their activation.

AuthorsOpenReview ↗

RE:Official Review of Submission49658 by Reviewer ELDL Part 3

Regarding accuracy, we report errors in observable physical predictions. We do not separately quantify the estimation error of an unobserved internal state. Prediction error can reflect errors in interpreting the input, evolving the estimated state, or rendering the output, and our measurements do not uniquely separate those sources. We will clarify this distinction wherever the schematic terminology could suggest a stronger mechanistic claim.

The manuscript would benefit significantly from a more comprehensive contextualization within contemporary literature.....

We agree that a more explicit comparison would improve the paper’s positioning. PhysBench, VideoPhy, and Morpheus are already cited in the manuscript; we will expand the discussion to explain their relationship to our experiments and include the additional relevant work identified by the reviewer.

However, the benchmarks address several related but distinct questions:

VideoPhy evaluates whether generated videos satisfy textual descriptions and physical commonsense across diverse material interactions. Our experiments instead condition on observed physical events and measure subsequent predictions against reference trajectories. These evaluations provide complementary information about generative physical prediction. PhysBench evaluates VLM understanding across physical properties, relationships, scenes, and dynamics. CoPhyBench tests reasoning from conditional video observations through prediction, physical calculation, and counterfactual questions. These are relevant assessments of physical reasoning, although their question-answering protocols differ from evaluating generated trajectories. Morpheus directly evaluates generated physical dynamics using measurements grounded in physical laws and invariants. This is particularly relevant: conservation-based evaluation can test plausibility beyond matching a particular reference trajectory, while our trajectory comparisons measure fidelity to the observed system’s continuation. We will make this relationship explicit. PhysInOne includes future-frame prediction, physical-property estimation, and physics-aware video generation with rich physical annotations. It is directly relevant to extending our evaluation to broader phenomena and more complex scenes. We will discuss that connection and the current limits of our evaluation coverage.

We will also distinguish our contribution from video world models and methods that incorporate physical constraints during training. Video foundation models such as LTX-video or Wan predict future world states through video generation with dedicated temporal architectures and extensive training. Our study asks how changing the inference procedure affects the predictions of a fixed pretrained image generator. It does not establish superiority over the broader class of video world models. Our analysis and comparison listed in Appendix G. and our response to reviewer eWVp also suggested that video generation model does not establish superiority over image generation models under Chain-of-time.

Similarly, approaches such as NewtonRewards use measurable physical constraints as rewards during post-training. Chain-of-Time changes inference without updating model weights or explicitly enforcing conservation laws. These approaches address different intervention points and may be complementary; our results do not establish that inference alone supplies the guarantees sought by physically constrained training.

In Appendix G, the comparison against native video generation models...

We would like to thank the Reviewer for raising this issue. However, we would also like to respectfully point out that the premise that the reported LTX-Video comparison restricts the model to sparse image inputs does not match the protocol described in Appendix G. Although we also explored sparse conditioning, the reported comparison uses all frames from the initial two-second segment as video conditioning and generates a two-second continuation at 30 FPS. LTX-Video therefore generates a dense video continuation using its temporal architecture. In the Appendix G., we write (additional emphasis added in CAPS here): “When we tested video generation models with the same sparse input data that we use for Chain-of-Time, their output was INCOHERENT, for example showing no motion at all. Therefore, in the following analysis we provided as input to LTX-video ALL FRAMES of the first 2 seconds of the video and prompted the model to generate 2 seconds into the future at 30 frames per second using a prompt similar to what we have in App. C”

Reviewer_eWVpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Chain-of-Time, a zero-shot prompting method for improving physical simulation capabilities of autoregressive image generation models (IGMs).

Critical strengths

This paper pioneers the study of physical reasoning in autoregressive image generation models, a rapidly emerging technology that major AI companies are adopting as the new standard for text-to-image generation.

Critical weaknesses
  1. The paper evaluates only three IGMs, with only two being fully-featured closed-source models from the same two companies. While the authors acknowledge this limitation and draw parallels to early chain-of-thought work, this comparison is problematic: Kojima et al. (2022) could test open-source models that, while weaker, demonstrated the phenomenon. Here, the technology is so new that no comparable alternatives exist, and one of the three models (HunyuanImage-3.0-Instruct) doesn't even fully support the required functionality (sequences of chat messages with images). This makes it premature to claim that Chain-of-Time is a general method rather than a potential artifact of specific model architectures or training procedures. The lack of ablation studies or analysis of why the method works further compounds this concern.

  2. The primary baseline (Direct Prediction) is relatively weak, a single-shot prediction without intermediate steps. While the authors mention a stronger baseline with self-consistency and CoT reasoning in the appendix, this should be a primary comparison given the method's CoT inspiration. More critically, the paper lacks comparison with state-of-the-art video generation models, which are the natural alternative for physical simulation tasks. The brief mention in Appendix G that Chain-of-Time performs "similarly" to video models raises questions about the practical utility: if specialized video models already solve this problem, what is the advantage of using IGMs with Chain-of-Time? The paper needs to articulate when and why one would choose this approach over existing alternatives.

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

RE:Official Review of Submission49658 by Reviewer eWVp Part 1

We would like to thank the Reviewer for their comment that sees our work as pioneering the study of physical reasoning in autoregressive image generation models.

We would also like to thank the Reviewer for pointing out challenges and raising questions about our studies. We address these in turn below:

The paper evaluates only three IGMs, with only two being fully-featured closed-source models from the same two companies....

We would like to thank the Reviewer for pointing out this important point. As we said in the paper, our work is indeed situated at a similar situation as the Kojima et al. (2022) paper. So, we would like to respectively push back on the point that Kojima et al. (2022) demonstrated that open-source models also show a (weaker) Chain-of-thought phenomenon. In the Kojima et al. (2022) paper, the models that obtained huge improvement in accuracy on several reasoning benchmarks were commercial, closed-source models (InstructGPT3, GPT3, and PaLM). While Kojima et al. did indeed test open-source models, none of the open-source models benefitted from Chain-of-thought, which was the finding we relied on in our paper. Specifically, consider appendix E, where they specifically investigated open source models, as well as the closed source models with a small parameter size. We cite their original results below:

Table 1: The Zero-Shot-CoT Results on Open Source Models From Kojima et al. 2022:

Open-source LLM AccuracySizeZero-shotZero-shot-CoT
GPT-21.5B3.22.2
GPT-Neo2.7B3.01.3
GPT-J6B2.72.5
T011B2.83.2
OPT13B3.72.2

We can see that in the Multiarith dataset, open source models like GPT-2, GPT-Neo, GPT-J, and OPT had no improvement under Chain-of-thought, and In OPT, GPT-Neo, and GPT-2, Chain-of-thought even harmed performance. Only on T0 the performance went up by a mere 0.4%, which could be attributed to noise.

We would like to emphasize that we generally agree with the more general point about testing open-source models. So, we decided to follow the path of Kojima et al. (2022), where they added substantial analysis focusing on the zero-shot Chain-of-thought context outputted by the model that works under their prompting method. We conducted a great deal of analyses on the intermediary frames generated by Chain-of-time, trying to show that the physical simulation is happening under the correct physics principle that generated the phenomenon shown in our stimuli. Especially for results like figure 16-27 in the Appendix E and figure 6-7 in our main paper, we specifically investigated the intermediary frames generated by Chain-of-time, and analyzed whether the underlying simulation is correct, and most importantly, if the simulation recovers the underlying functional form that determines the physical phenomenon shown in the stimuli. As shown in the figures, Chain-of-time indeed improves the physical simulation, and under our method, the model is able to trace the correct functional form and simulate with the correct physical principles.

This makes it premature to claim that Chain-of-Time is a general method rather than a potential artifact of specific model architectures or training procedures. The lack of ablation studies or analysis of why the method works further compounds this concern.

As in our response to Reviewer ELDL, to further trace why our methods work requires mechanistic interpretability levels of work. By looking at the activations in IGMs, we could potentially reconstruct hypotheses from the mechanistic aspect of the model to answer why our methods work. But since our method is a zero-shot learning method, and mostly serves to produce behaviorally correct outcomes and improve the model’s behavior, this direction was not our focus for this paper, and is left for future work. In the revised manuscript, we will specifically address the need and importance of mechanistic interpretability work on image generation models.

The primary baseline (Direct Prediction) is relatively weak…

We thank the Reviewer for bringing up this important point. We have moved the stronger self-consistency and CoT reasoning baseline to our main paper. Please see the table 1 in our response to reviewer c4wW.

AuthorsOpenReview ↗

RE:Official Review of Submission49658 by Reviewer eWVp Part 2

More critically, the paper lacks comparison with state-of-the-art video generation models….

In response to the comparison with the state-of-the-art video generation models, we ran additional experiments on two SOTA video generation models: LTX Video and Wan. We fully re-ran all domains on these two models. The mean RMSE of these two models in all domains, along with the statistical significance tests that whether the mean RMSE of the best video generation model is better than the mean RMSE of the best image generation model, are down below:

Table 2: Results of SOTA Open-Source Video Models On Our Tested Domains:

DomainLTX-VideoWan2.1-VACE
2D Gravity151 [131, 174]242 [200, 290]
Slope102 [83, 120]118 [74, 167]
Collision66 [48, 86]155 [86, 242]
Pulley46 [35, 58]57 [43, 74]
Bouncing21 [15, 28]25 [19, 32]
Track248 [119, 416]410 [291, 529]
Pendulum134 [80, 200]151 [95, 211]
Fluids14 [5, 26]18 [8, 33]

Table 3. Best video model vs best image model, per domain:

DomainVideo − image [95% CI]t statisticp
2D Gravity+86 [+62, +111]7.16<.001
Slope+56 [+34, +77]5.44<.001
Collision+8 [−16, +32]0.74>.99
Pulley−251 [−269, −233]−27.97<.001
Bouncing−8 [−16, −0.05]−2.00.39
Track−180 [−365, +5]−2.19.44
Pendulum−35 [−106, +37]−1.06>.99
Fluids−20 [−32, −9]−3.70.004

We can see that the video generation model does not generally perform better than the image generation model, and our statistical significance tests supports this. One outlier was the Fluid domain, for which the video generation model performed much better. In order to investigate this, we performed the same analysis that we did in Appendix E.1, where the examined if video generation models are sensitive to the initial water level and the flow rate, which are two critical physics parameters that dominantly impact the simulation accuracy. As the table shown below, the video generation model is not only sensitive to the initial water level changes we encoded in our stimuli, it is also sensitive to the flow rate changes, in contrast to the image generation model that is sensitive to the initial water level changes but not sensitive to the flow rate changes, as the results shown below:

Table 4: The perceived flow rate vs the ground truth flow rate:

Flow conditionActual rate (px/s)LTX-Video: perceived rate (px/s)Wan2.1-VACE: perceived rate (px/s)
252.40.7 [0.1, 1.3]2.7 [1.3, 4.2]
504.82.6 [1.7, 3.5]4.5 [−1.6, 9.1]
757.26.3 [4.4, 7.9]5.5 [2.4, 9.1]

Table 5: The perceived initial water level vs the ground truth water level

Starting fillActual level at 2 s (px above the mug bottom)LTX-Video: perceived levelWan2.1-VACE: perceived level
1/1238.3 [31.2, 43.8]34.2 [23.6, 43.6]26.3 [14.2, 37.5]
3/1264.6 [60.1, 68.8]68.5 [65.3, 72.1]61.8 [59.3, 64.2]
5/1295.6 [88.9, 103.1]96.8 [89.7, 104.3]90.8 [85.8, 96.3]
7/12122.2 [112.2, 132.9]124.2 [114.1, 134.8]118.8 [110.3, 127.9]
9/12147.2 [134.4, 160.9]148.9 [135.8, 162.4]145.1 [132.6, 158.4]

So, the video generation model may be doing than the image generation model for reasons that are unrelated to the main point of the Chain-of-Time method. If a model is fundamentally bad at generating physics, taking more and more sub-frames may not improve this issue.

Also, the performance of image generation model and video generation models on the domains that we tested are mixed. As we detailed in the Appendix G, the total of input frames that the video generation model needed to ensure an working and accurate simulation is EIGHT times more than what is needed in the image generation models, and that the output frame needed is FIFTEEN times more than what image generation models needed. Using the 5-input frame that we used in chain-of-time on the video model won’t even work, as it breaks the video generation model, and the model will start to produce uninterpretable content. We consider this as a major advantage to our image generation model, because being able to perform physical simulation at a similar precision and accuracy when compared to video generation model without needing so many input and output frames is important from the aspect of computational cost and accuracy tradeoff.

Furthermore, due to the fact that the video generation model receives EIGHT times more input frames, this comparison puts our method and image generation model at an enormous disadvantage and makes the comparison difficult to assess. Given this, we decide to keep this comparison in the Appendix. But we will make sure that in the revised manuscript we put every findings that we listed here into the Appendix.

AuthorsOpenReview ↗

RE:Official Review of Submission49658 by Reviewer eWVp Part 3

====

We again thank the Reviewer for their positive comments, and also the many constructive comments and questions. We believe we have addressed the issues raised, and that the paper is stronger as a result. If the Reviewer has additional remaining questions or issues, we would appreciate knowing what those are so that we can try to address those as well. If our response has indeed addressed the Reviewer’s issues, we would appreciate it if this is reflected in their score.

Reference:

Kojima, Takeshi, et al. "Large language models are zero-shot reasoners." Advances in neural information processing systems 35 (2022): 22199-22213.

AuthorsOpenReview ↗

Follow Up to the Rebuttal for Reviewer eWVp

Dear Reviewer eWVp,

With the deadline for the end of the discussion period fast approaching, we wanted to check whether there were any remaining comments that our response did not address (or whether our response requires further clarification). We appreciate how busy you are, and thank you for your time!

Authors

Reviewer_c4wWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether autoregressive image generation models (IGMs) can simulate physics. The setup: give the model five frames spanning two seconds, and ask for the scene k seconds in the future. Chain-of-Time (CoT) generates intermediate frames every 0.5s ("Quarter") or 1s ("Half") in a multi-turn loop, instead of predicting the final frame directly; the framing is borrowed from human mental simulation and from chain-of-thought prompting. The method is training-free. Nine domains are tested (three synthetic 2D settings, four taken from VLM physics benchmarks, two from real videos), with two closed IGMs (GPT Image 1.5, Nano Banana Pro), an open one (HunyuanImage-3.0, adapted because it lacks multi-turn chat), plus an LTX-video comparison, 10-second horizons, a VLM-critic variant and a self-consistency baseline. Evaluation tracks the focal object with SAM 3 (a hand-coded water-level detector for Fluids) and computes RMSE against ground-truth positions.

Critical strengths

As far as I know this is the first systematic study of physical simulation in this class of models, and the question is timely — IGMs are replacing diffusion decoders as the default text-to-image interface, and whether they carry anything like a world model is exactly what the community should be poking at. The mental-simulation framing is well-grounded in the cited cognitive-science literature without overloading the paper.

The step-wise trace is a real asset. Being able to watch per-frame accuracy accumulate, and to check whether the recovered trajectories have the right functional form — parabolic bouncing, sinusoidal pendulum motion, coupled pulley kinematics (Figs. 6-7) — is something outcome-only physical-reasoning benchmarks cannot offer, and I found these figures the most convincing part of the paper.

In the intended regime the method plainly works. In Table 8 (App. H.5), Quarter beats Direct Prediction in seven of eight domains for both closed models, often by 30-60% RMSE (e.g., Bouncing 93 -> 65, Rolling on Slope 98 -> 46 for Nano Banana Pro), and the ordering Quarter > Half > Direct is nearly monotone.

The breadth of follow-up experiments is good: input/output frame counts with a token-cost analysis, prompt variants, 10s horizons, a VLM critic (+18% on Pendulum), a self-consistency baseline, and a video-model comparison that is fair about its own limitations. The limitations section is honest about how few eligible IGMs currently exist.

Critical weaknesses

Three issues bear directly on the reliability and interpretation of the headline claims; the rest I class as lesser concerns.

The one consistent failure is downplayed. On Fluids, Chain-of-Time makes things substantially worse for all three IGMs: Nano Banana Pro 34 -> 43 -> 72, GPT Image 1.5 37 -> 47 -> 90, Hunyuan 92 vs 64 (Quarter vs Direct). The main text reports "seven of eight domains" without stating that the exception is a doubling of error; the abstract and conclusion do not mention it at all; and there is no analysis of whether the generator or the evaluator is at fault. Fluids is also the only domain not evaluated with SAM 3 — it uses a hand-coded detector written by ChatGPT-5 (App. D.4) — which makes disentangling the two even more necessary.

The quantitative evaluation layer cannot support the claims as stated. The main text contains no per-domain numbers (they live in App. Table 8); RMSE is in raw pixels with no normalization by frame size or travel distance; there are no significance tests; and the new-baseline rows lack CIs. The stimulus sets are small — 2D Motion has 3 stimuli, and the SAM 3 prompt lists in App. D suggest most domains have single-digit-to-low-teens counts — and the bootstrap unit (stimulus or repeat) is never stated. The pipeline itself is weakly validated: SAM 3 masks were checked by one human on subsampled frames at "the maximum possible score" (App. D.2), with no IoU and no numbers, and the Fluids detector has no quantitative validation at all. Several reported differences are plausibly of the same order as localization noise. No trivial baselines. Constant-velocity or constant-acceleration extrapolation of the tracked positions over a two-second horizon is cheap, deterministic, and is the null hypothesis for "physical simulation"; it is absent. Without it, the paper cannot say whether the IGM is doing more than kinematic extrapolation.

Lesser concerns, listed briefly: method-level novelty is modest — iterating intermediate image generations in context is a direct transfer of chain-of-thought to images, with Visual Sketchpad and the image-chain planners as close relatives (the paper discusses these fairly; the delta is the time-unit interpretation and the new subject population); the cognitive framing occasionally outruns the evidence ("Like humans, image generation models are able to accurately predict future world states over short time periods" — no human data was collected in this work); and there is no measured cost or latency for a full simulation in the main text (App. H.2 has a token analysis).

Questions for authors

What exactly fails on Fluids — liquid rendering degrading across iterations, or the hand-coded detector failing on generated images? Please show generated liquid frames and validate the detector on them.

Can you add a per-domain table with normalized errors (fraction of frame width or of object travel) and paired tests of CoT vs Direct Prediction, with exact stimulus counts and the bootstrap unit for the CIs stated?

How does CoT compare to constant-velocity / constant-acceleration extrapolation from the tracked input positions? This is the baseline that would establish whether the models are doing physics rather than motion continuation.

Can you provide quantitative validation of the localization pipeline (e.g., IoU of SAM 3 masks against annotated masks) and the corresponding RMSE noise floor?

7 comments
AuthorsOpenReview ↗

RE: Official Review of Submission49658 by Reviewer c4wW Part 1

We thank the Reviewer for characterizing our work as the first systematic study of a question that is timely and necessary. In addition, we appreciate their multiple positive comments, including about how the step-wise traces offer more than current mainstream benchmarks, the performance of our method, and the breadth of our experiments.

We also appreciate the Reviewer pointing out things that could improve about our paper, and the questions they raise. We believe these can be addressed in a way that makes the paper stronger and better. We consider them each below, in turn.

The one consistent failure is downplayed.…..

What exactly fails on Fluids….

We agree with the Reviewer that fluids are an interesting case, and did not mean to downplay them. In the edited version of our manuscript, we will add the relevant information regarding this important outlier in the introduction and conclusion.

We also agree with the Reviewer’s request to clarify the distinction between “failure to generate the right simulation” versus the “failure to detect the simulation in the right manner and method”. These two failure modes would result in a similar conclusion, which is that Chain-of-Time made fluid simulation substantially worse for all three IGMs. We consider adding an experiment to address this issue head-on.

To address whether it is the fault of the generator or detector, we recruited 150 human participants online via the Prolific platform, to participate in an online study (covered by our lab’s IRB protocol). In this study, we ask the participants to label the water level given a frame (ground truth + generated). In a power analysis, we found that when the number of participants exceeds 50 people, the mean RMSE noise floor starts to converge. So, our selection of 150 people is far above the sample size needed to draw our conclusions. Our experiment found that our detector’s RMSE noise floor is 11.5px [7.3,16.2], before normalization. We treat this as a golden standard as humans can easily tell whether a given height is in line with some other line (the detected water level). Combined with the RMSE finding that we have in the main paper, we can conclude that the detector is not at fault for our finding that Chain-of-Time did not help in the fluid domain. Rather, the issue is with the generator. We will put the details of this study and its conclusion into the revised paper, specifically specify the failure mode is in the generator using our finding from the experiment we conducted over our detector above, and provide the RMSE noise floor and example detector notated frames.

The quantitative evaluation layer cannot support the claims as stated...

We would like to thank the Reviewer for pointing out these issues. Before addressing the issue directly, we would like to note that even in principle, normalization should not affect our conclusions, since the resolution scale is consistent within a domain for all three methods (Chain-of-time quarter, Chain-of-time half, and Self-Consistency Baseline), so the results of all three methods are compared under the same resolution scale. For completeness, in the revised text, we added tables with per-domain numbers that were normalized by the length of frame diagonal into the main text. The formula is RMSEnormalized=(RMSEoriginal/lengthdiagonal)∗100RMSE_{normalized} = (RMSE_{original}/length_{diagonal}) * 100. As expected, and as described in more detail below, this does not alter our conclusions.

Also, we will provide the table that carries the new-baseline with bootstrapped CI. To make our conclusion stronger, we would like to officially add the self-consistency baseline into our main paper. All the tables mentioned above are attached below.

Table 1: The normalized RMSE error by frame diagonal in all domain for all models

DomainsRolling on TrackBouncingRolling on SlopePulley System2D GravityPendulumFluidCollision
Frame diagonal (px)188812987242203144811521024724
Nano-Banana Pro (Direct Prediction)35.3 ± 2.87.2 ± 0.413.5 ± 0.921.0 ± 0.515.6 ± 0.925.0 ± 4.13.3 ± 0.219.6 ± 1.7
Nano-Banana Pro (Self-Consistency Baseline)37.2 ± 4.38.3 ± 1.813.0 ± 1.622.8 ± 1.513.6 ± 2.822.0 ± 7.32.2 ± 0.721.0 ± 5.4
Nano-Banana Pro (Chain-of-Time Half)30.8 ± 2.75.5 ± 0.49.5 ± 0.918.7 ± 0.75.1 ± 0.916.2 ± 1.54.2 ± 0.214.4 ± 1.5
Nano-Banana Pro (Chain-of-Time Quarter)22.7 ± 1.45.0 ± 0.36.4 ± 0.715.4 ± 0.54.5 ± 0.614.7 ± 1.37.0 ± 0.28.0 ± 1.4
GPT-Image-1.5 (Direct Prediction)36.7 ± 5.15.7 ± 0.215.5 ± 1.317.2 ± 0.417.0 ± 0.826.6 ± 5.03.6 ± 0.221.0 ± 1.5
GPT-Image-1.5 (Self-Consistency Baseline)36.1 ± 16.24.4 ± 0.714.2 ± 4.216.1 ± 1.214.8 ± 2.627.5 ± 15.73.3 ± 0.618.1 ± 4.8
GPT-Image-1.5 (Chain-of-Time Half)29.8 ± 3.63.7 ± 0.312.2 ± 1.014.5 ± 0.86.4 ± 0.822.6 ± 2.14.6 ± 0.212.4 ± 1.7
GPT-Image-1.5 (Chain-of-Time Quarter)23.6 ± 1.62.2 ± 0.37.2 ± 0.913.5 ± 0.65.5 ± 0.621.1 ± 1.88.8 ± 0.39.4 ± 1.4
AuthorsOpenReview ↗

RE: Official Review of Submission49658 by Reviewer c4wW Part 2

Table 2: The bonferroni-adjusted t-test on normalized RMSE error by frame diagonal in all domains for all models. In addition, the monotonic ordering test.

DomainNano-Banana Pro (1): t, pNano-Banana Pro (2): t, pNano-Banana Pro (3): t, pNano-Banana Pro order: χ̄², pGPT-Image-1.5 (1): t, pGPT-Image-1.5 (2): t, pGPT-Image-1.5 (3): t, pGPT-Image-1.5 order: χ̄², p
2D Gravity−1.2, p = .745−6.3, p < .001−5.7, p < .001454, p < .001−1.7, p = .265−6.9, p < .001−6.1, p < .001528, p < .001
Bouncing−2.0, p = .134−3.5, p = .002−2.9, p = .01460, p < .001−6.7, p < .001−5.6, p < .001−1.8, p = .211335, p < .001
Rolling on Slope−5.6, p < .001−7.4, p < .001−3.7, p = .003161, p < .001−7.2, p < .001−3.2, p = .015−0.9, p = 1114, p < .001
Pendulum−1.6, p = .371−3.0, p = .020−2.3, p = .08723, p < .001−1.0, p = .905−0.8, p = 1−0.6, p = 15, p = .779
Collision−6.2, p < .001−4.6, p < .001−2.3, p = .095112, p < .001−2.8, p = .028−3.4, p = .008−2.2, p = .117130, p < .001
Rolling on Track−5.2, p < .001−6.3, p < .001−2.5, p = .06575, p < .001−3.0, p = .021−1.5, p = .471−0.7, p = 129, p < .001
Pulley System−7.3, p < .001−9.1, p < .001−4.9, p < .001233, p < .001−2.0, p = .160−3.9, p = .001−2.2, p = .106118, p < .001
Fluid21.7, p < .00112.8, p < .0015.2, p < .0010, p = 119.6, p < .00116.4, p < .0014.1, p < .0010, p = 1

Welch t for (1) Chain-of-Time Quarter vs Chain-of-Time Half, (2) Chain-of-Time Quarter vs New Baseline and (3) Chain-of-Time Half vs New Baseline. RMSEs are normalized by frame diagonal, and P-values are Bonferroni-adjusted within each model and domain over its 3 tests (1)–(3). "order" = χˉ2\bar\chi^2 of the ordered test Chain-of-Time Quarter ≤ Chain-of-Time Half ≤ Original Baseline. Bold = adjusted p < .05.

We can see that according to a (Bonferroni-corrected) t-test, at least one Chain-of-Time method beats the new baseline, which is the self-consistency baseline. And in most domain, a chi-square Cochran-Armitage test showed that the monotonic pattern (Chain-of-time Quarter <= Chain-of-time Half <= Self-consistency Baseline) observed is statistically significant.

The stimulus sets are small….

Regarding the stimuli count: we would like to kindly refer the Reviewer to Appendix B., where we showed the details of stimuli, data augmentation, and in the details of stimuli it includes the stimuli count. Across the domains, we believe our stimuli set is varied and generally adequate: In the 2D gravity domain, we have 24 stimuli. In the fluid domain, we have 45 stimuli. In the bouncing domain, we have 54 stimuli. In the pendulum, rolling on the slope, and collision domain, we have 18 stimuli. In the rolling on the track domain, we have 14 stimuli. In the pulley system domain, we have 26 stimuli. We agree that In the App.D the SAM 3 prompt is single-digit-to-low-teens counts, but the final stimuli are the data augmented based on the original videos, and only one SAM3 prompt is needed for each original video. The data augmentation is achieved by varying the physics parameters involved in the original video, for example, changing the flow rate of fluid, the velocity of the object in the video, etc.

The pipeline itself is weakly validated….

For the pipeline validation: We appreciate the Reviewer pointing out this, and we indeed value the validation here. In order to ground the credibility of our evaluator here, we again conducted experiments where we asked human annotators (online) to manually annotate the ground truth position of the objects we were interested in, using the stimuli we used in our experiment and images generated by the model. 150 people were recruited on Prolific to conduct such annotations. The power analysis we performed showed that ~100 people will stabilize the IoU and the RMSE noise floor we found in their data. So, our 150 people is far above the sample size needed to demonstrate the effect, and erring on the side of caution. The analysis showed that the SAM3 achieved a mean IoU of 0.89 [0.85,0.92] across the frames we tested. The RMSE noise floor is 4.4px [2.7,6.2], before normalization. We consider this to be a good IoU and RMSE noise floor, and with this RMSE noise floor it is not affecting the conclusion that Chain-of-time beats the baseline we considered in our paper. In our revised manuscript, like we did to the fluid detector, we will put the detail of the experiment, as well as the RMSE noise floor and IoU finding into our paper.

AuthorsOpenReview ↗

RE: Official Review of Submission49658 by Reviewer c4wW Part 3

No trivial baselines. Constant-velocity or constant-acceleration extrapolation….

How does CoT compare to constant-velocity / constant-acceleration extrapolation….

For the trivial baseline: We thank the Reviewer for pointing out this important null hypothesis out. We would emphasize even before addressing it though, that our main investigation had to do with whether Chain-of-Time is a useful method relative to direct prompting, and what the relationship is between added sub-frames and the resulting dynamics and accuracy. We see the general issue of whether an image model is doing linear kinematic extrapolation (or something else). However, to address this more directly, we conducted experiments trying to solve our stimulus with constant-velocity and constant-acceleration extrapolation. We found that none of the kinematic baseline consistently explains the image generation model’s behavior in all domains. Therefore, we should not conclude that the model is doing kinematic extrapolation.

The results are summarized below:

Table 3. Chain-of-Time Quarter vs each kinematic baseline

DomainNano-Banana Pro vs Constant velocity (least squares)Nano-Banana Pro vs Constant acceleration (least squares)GPT-Image-1.5 vs Constant velocity (least squares)GPT-Image-1.5 vs Constant acceleration (least squares)
2D Gravity−46.3, p < .0017.4, p < .001−41.7, p < .0019.1, p < .001
Bouncing5.0, p < .0013.1, p = .006−0.3, p = 1−1.0, p = .607
Rolling on Slope−7.3, p < .001−14.4, p < .001−6.7, p < .001−13.8, p < .001
Pendulum−1.3, p = .447−3.5, p = .0080.3, p = 1−2.8, p = .034
Collision−1.1, p = .579−2.2, p = .089−0.5, p = 1−2.0, p = .145
Rolling on Track−0.1, p = 10.6, p = 10.1, p = 10.8, p = .861
Pulley System33.3, p < .00141.6, p < .00127.0, p < .00133.7, p < .001
Fluid2.9, p = .013−0.5, p = 14.1, p < .001−0.1, p = 1

Welch t of Chain-of-Time Quarter vs Kinematic Baseline: negative = Chain-of-Time Quarter lower error, positive = the kinematic baseline is better. p is Bonferroni-adjusted within each model and domain over its 2 tests; bold = adjusted p < .05.

Lesser concerns, listed briefly: method-level novelty is modest — iterating intermediate image generations in context is a direct transfer of chain-of-thought to images, with Visual Sketchpad and the image-chain planners as close relatives (the paper discusses these fairly; the delta is the time-unit interpretation and the new subject population); the cognitive framing occasionally outruns the evidence ("Like humans, image generation models are able to accurately predict future world states over short time periods" — no human data was collected in this work); and there is no measured cost or latency for a full simulation in the main text (App. H.2 has a token analysis).

We would like to thank the Reviewer for these thoughtful comments. We recognize that our method might be similar to chain-of-thought for images, with visual sketchpad and the image-chain planners as close relatives. This is why in our introduction we specifically cited Visual Sketchpad and relevant work, and discussed the distinction between our method and their methods.

For the cognitive framing: we were referring to the generally accepted findings in cognitive science (and specifically the field of intuitive physics) that humans can reasonably predict future world states over short time periods, at least for many cases of everyday objects and everyday dynamics objects (see e.g. Battaglia et al. 2013 and the many papers that preceded or followed it). We do not take this to be a claim that needs additional empirical backing from studies in this paper. However, to not cause confusion for readers, we will remove this part in the paper, as it does not directly affect our claims and findings.

======

We again thank the Reviewer for their positive impression, and many constructive comments and questions. We believe we have addressed the issues raised, and that the paper is stronger as a result. If the Reviewer has additional remaining questions or issues, we would appreciate knowing what those are so that we can try to address those as well. If our response has indeed addressed the Reviewer’s issues, we would appreciate it if this is reflected in their score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.