acceptodds
Under review as a conference paper at ICLR 2027

Document-Grounded Generation of Verified, Interactive and Photoreal Physics Experiments

Abstract

Most scientific documents convey physical processes through static figures, which leave the dynamics for readers to figure out. Generative models can turn such documents into interactive simulations and videos. These outputs often look plausible, yet it is hard to ensure they obey the physical laws the document states. Our goal is to turn a document's figures into interactive simulations and videos that help readers understand the physics. These simulations must follow the physical laws stated in the document. We represent the document as a graph over its equations, quantities, and validity regimes. Retrieval on this graph continues until the collected equations form a closed, solvable system. This system is recorded as a physics scenario card. The card states what to solve, the governing dynamics, the quantities a reader controls, and their valid ranges. The same card drives generation and supplies tests derived from the document. Generated behavior is therefore checked against the document's own relations. Closure-terminated retrieval recovers substantially more of the required context than similarity search. An evaluator answers figure-specific questions from our interactive artifacts at 0.923, against 0.389 for single-call generation. In photorealistic videos, motion measured from the frames matches the closed form derived from the document. Without physical conditioning, the same video model produces no measurable period. This work enables a scientific document to demonstrate the physical behavior it describes. It opens a path from the static figures of education, science communication, and engineering to interactive, verifiable representations.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
retrieval-augmented generation, scientific document understanding, verified code generation, neuro-symbolic verification, controllable video diffusion, physical consistency in video generation, interactive simulations, AI for education

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rxrKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors explore the problem of converting turning static figures in printable textbooks into interactive experiences and video examples. To be faithful to the stated physical laws in the textbook, the proposed method first extracts relevant equations and provides them as context for code and video generation.

Critical strengths

The idea is simple but powerful. The method has the potential to speed up learning experience of textbooks and the included human evaluation shows this qualitatively. The method section tries to introduce the approach in a very detailed manner.

Critical weaknesses

That said, I don't think the paper is ready yet for publication and needs further revisions. First and foremost, the paper does provide almost no example outputs that would help to put the very dry methods section into context. Given the topic, I would have loved to inspect some outputs myself e.g. through the supplementary materials to deepen my understanding. Apart from this several aspects of the paper are either ambiguous, unclear or contradictory to me. Figure 1, for example, puts video generation as a pipeline step after interactive output generation, yet in Figure 2 the processes run in parallel. Some aspects of the approach also do not appear well discussed. The method seems to rely heavily on large language models, but how is not clear. Baselines, on the other hand, rely on much lighter embedding models which makes it difficult to assess if the observed improvements come from the approach specifically or from just using stronger models. The authors should be more clear and list which exact models they are employing in their experiments and why they think the comparison is fair. Other aspects of the evaluation are also unclear: How was the dataset collected? What are the eight multiple choice questions? How exactly does the degradation work? It sounds like the interactive artifacts are compared against the static image. How does this evaluate interactivity? And what value does the interactive figure provide if it can't outperform the static image baseline (Figure 4)? Last but not least, the authors should discuss the limitations of their approach, e.g., how does the runtime compare across approaches?

I am happy to increase my score if the authors can address my concerns. But as of now I would recommend the authors to revise their work and resubmit.

Questions for authors
  • l.27 to what metric do the reported score numbers belong?
Reviewer_JjHmOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes a pipeline for converting figures in scientific documents into interactive simulations and photorealistic videos. It constructs a graph of equations, quantities, and validity regimes, and retrieves dependencies until a physical-closure criterion is satisfied. The recovered specification is stored in a physics scenario card, which supports both artifact generation and executable verification. For video generation, verified simulation trajectories provide motion conditioning, and physical quantities are subsequently measured from the delivered frames.

Critical strengths

1.A useful formulation of scientific-document retrieval. Retrieving until the information required for a physical task is complete is a meaningful objective for scientific documents, where definitions and assumptions are often distributed across sections. The evaluation on 418 documents provides evidence that the graph and closure components improve dependency recovery. In particular, non-local recall increases from 0.105 for dense retrieval to 0.449, and the incremental improvement over graph reranking has a document-bootstrap confidence interval above zero. This is the most convincing empirical contribution.

2.Verification extends to the delivered output. The pipeline checks generated programs through execution and measures motion from the final video using two independently implemented trackers. This addresses a real failure mode: a correct driving simulation does not guarantee a physically faithful generated video. The instrument self-tests and unconditioned-video comparison strengthen the pendulum demonstration. This design could be valuable for educational and scientific visualization tools where plausible appearance alone is insufficient

Critical weaknesses

1.Clarification of the existing implementation: How are the apparently incompatible period relations in Eqs. (14) and (16) represented in the actual run? Please provide the scoped quantity identities, the final PSC, and the consistency-check result. Where do the downstream motion equation and initial conditions enter this specification?

2.Clarification of practical deployment: For the evaluated inputs, how many artifacts were attempted, passed without repair, passed after repair, or required manual intervention? What model versions, latency, and approximate computational cost were involved? These details would materially affect the assessment of the proposed authoring benefit.

3.Additional controlled analysis, if available: Does the interactive-artifact advantage remain when the baseline is explicitly required to expose the same equations, quantity labels, and evaluator interface? An ablation holding these factors fixed while disabling verification or repair would help determine what produces the reported improvement.

4.Clarification of the supported application claim: Which practical benefit does the current evidence establish most directly: reduced authoring effort, improved information accessibility, improved human understanding, or improved physical fidelity? Are there existing measurements supporting reduced authoring effort or usability beyond the interim comprehension study?

Questions for authors

See Weaknesses

Reviewer_tbhrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a training-free pipeline that converts static figures in physics documents into interactive simulations and physics-conditioned photorealistic videos, both verified against the relations stated in the source document. First, the document is parsed into a graph of equations and physical quantities. Starting from the figure’s objective, retrieval expands until the gathered equations form a closed, consistent, and well-posed system. LLM-proposed relations between equations are kept only if they can be symbolically re-derived. The closed system is recorded as a Physics Scenario Card (PSC), from which executable tests (conservation, dimensional, limiting-case, and validity-regime checks) are derived. A code-generation model builds the interactive artifact from the PSC alone, and the artifact is accepted only if it passes these tests. For photographic figures, the verified trajectory is rendered in MuJoCo as a control video that conditions an off-the-shelf video diffusion model. The conditioning is withdrawn partway through sampling, and the target invariant is re-measured from the output frames. Experiments show improved retrieval sufficiency over BM25 and dense retrieval (0.314 vs. 0.210 sufficiency@10 on 418 documents). On 26 figures, an automated evaluator answering figure-specific questions scores 0.923 on the generated artifacts versus 0.389 for single-call generation. In pendulum videos, periods measured from the frames deviate from T=2πL/gT = 2\pi\sqrt{L/g} by 1.1–4.5%.

Critical strengths

Using physical closure as the stopping rule for retrieval makes sense for this setting, and it clearly helps: non-local recall goes from 0.105 (dense) to 0.449. The ablation in Table 1 shows that the symbol graph, reranking, and the closure walk each contribute.

Relations between equations have to be re-derived by SymPy, and the generated code has to pass tests that come from the PSC rather than from another model’s opinion. The PSC also keeps a pointer back to the source page, so a reader or instructor can check where each relation came from.

For the video part, the authors measure the period directly from the generated frames with two separate trackers and compare it with T=2πL/gT = 2\pi\sqrt{L/g}. Most work on physics in video generation relies on human or VLM judgments, so this is a more convincing kind of evidence, even though it is only shown for one scenario.

Critical weaknesses

The main concern is the baseline for the interactive artifacts. The pipeline is compared only with single-call generation, yet the pipeline itself has access to the PSC, a test suite, and several rounds of repair. The large gap in Table 2 therefore says little about the PSC or the closure walk specifically, since a substantial part of it may come simply from allowing the model to execute and fix its code. At least one baseline in which the same LLM receives execution feedback and repair attempts, but no PSC, is needed to isolate the contribution of the proposed components.

The photoreal branch appears overstated. The title and abstract present it as a general capability, but the only end-to-end check, in which the period is measured from the frames and compared with the closed form, is a pendulum at three lengths. Table 7 lists seven families with a pre-render gate, but for six of them no measurement is reported on generated videos. The paper also states that each family defines its own invariant and tolerance, and apart from the pendulum example in Appendix A, it does not describe how the state-to-scene mapping Φ is constructed for a new figure. Since this directly affects how general the method is, it should be explained.

The human study is too small to support the claimed benefit. The intervals for page-only (0.554) and page + artifact (0.611) overlap almost entirely, and the authors note that the study is not yet complete. It would be more appropriate to present it as preliminary.

The end-to-end success rate of the pipeline is also unclear. With sufficiency@10 at 0.314, most queries do not reach a closed system. Reporting in the main text how many figures entered the pipeline and how many yielded an accepted artifact would make its practical value clearer.

Questions for authors

Could you compare against a stronger baseline on the 26 figures, for example the same LLM with execution feedback and self-repair but without the PSC? This would show how much of the gain comes from the PSC and closure retrieval rather than from iterative repair.

What happens downstream when the closure walk fails to reach a closed system? How many candidate figures were there before the exclusion rules reduced them to 26, and what fraction of figures in the corpus end up with an accepted artifact?

For the video branch, are the state-to-scene mapping, the invariant, and the probe written by hand for each scenario family? Do you have frame-level measurements for any family other than the pendulum?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.