acceptodds
Under review as a conference paper at ICLR 2027

Anatomy of a Jump

Abstract

Can a language model infer an explanatory concept that the observations do not name, rather than fit a familiar law to the state it is shown? We test this in 48 procedurally generated physical worlds. At the restructuring level, the best tested local patch stays below on the certification samples while the true law reaches 1.000; an ideal observer selects the true structural class after one rendered trajectory when that class is supplied among three candidates. Twelve model configurations face a capability check, and two run the full ladder, producing 1,872 answers, although only the frontier arm passes the check. The paper presents three results. First, all 893 parseable stated laws use only position, velocity and time, but the answer format required those variables. Without that restriction, four of seven models still produce no uncued law reading additional state; the other three produce one each, two by split votes. A cue elicits hidden state in 60 of 504 laws, none the world's governing law. Second, in 1,223 of 1,336 scored answers, the model's numerical prediction beats its own stated law, including 398 of 445 at retrieval, where the correct law fits the format. Third, a four-lag linear autoregression fitted to the same fifty prompt rows beats the frontier model on 124 of 153 powered restructuring trials, with roughly eighteen times smaller median error. What this control licenses is a narrower rule: predictive accuracy can support a world-model inference only where ontology-free predictors fail on the same regime. One of three families qualifies over the swept thresholds; its advantage has the same sign on fresh worlds, but both intervals include zero and the qualifying partition changes. Mechanistic probes at 1.7B yield bounded null results.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, scientific discovery, physical reasoning, dynamical systems, system identification, model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_41hROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper seems to propose an evaluation of LLMs on physical prediction tasks. It appears to place emphasis on the ability of models to generalize from their context to other scenarios.

Critical strengths

The paper seem to deal with an important topic that is not a new question but is still relevant as more evidence can be gathered to better understand how to place LLMs on the scale of "memorization" vs "generalization".

Critical weaknesses

First weakness:

The writing in the paper is hard to follow. Many arguments lack justification or explanation, and many terms are used without being defined.

Some examples from the abstract:

  • "run the full ladder" -- what is this ladder?
  • "frontier arm"

Some examples from the intro:

  • The second paragraph is hard to read. It packs the setup, the novelty ladder, the three hidden mechanisms, results (as numbers) and the "ideal observer" into one paragraph.
  • In the intro's second paragraph, it is not clear at all what observations and explanations are referred to.
  • Terms are often used before explaining what they are. For example, "arm" (first used in the abstract, defined only in Section 3) and "basic capability floor" (I couldn't understand it from the intro).
  • Some results are reported in the second paragraph (under "best local patch", whose definition appears much later in the paper), but then later on results are enumerated again without it being clear if they relate to the first results.
  • The task is described without any motivation for this specific task (both in the intro and in the abstract). It is not clear at all what setup was tested.

Those point are partial and are only about the abstract and the intro but the clarity of the rest of the paper is in similar or sometimes worse state.

Second weakness:

The novelty claim relative to prior work is overstated. Inferring hidden state from observed data with LLMs has been studied before. DiscoverPhysics (https://arxiv.org/abs/2605.26087) includes worlds with hidden structure that the model is not told about (dark_matter, three_species). The current setup seems to be covered by the claims in that paper.

Questions for authors

The paper is very hard to read to the point I had a difficult time assessing its contribution. The paper would benefit from professional language editing.

Reviewer_XUwQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses synthetic physical systems to test whether language models can infer unobserved state or history dependence from trajectories. No L2 law recovery is verified under the tested conditions. Direct predictions generally outperform predictions from the models’ stated laws, but autoregression performs better in most relevant trials, indicating that these prediction results alone do not establish recovery of the hidden mechanisms.

Critical strengths

1. A clear research question in a controlled setting.
Synthetic systems with known generative mechanisms, together with checks on the fit of a specified class of instantaneous-state laws, provide a concrete setting for testing whether models can propose missing state.

2. A substantively useful autoregressive control.
An autoregressive model fitted to the same observed trajectory history outperforms the primary model in most relevant trials. This control limits stronger interpretations of predictive performance and is one of the paper’s most valuable empirical analyses.

Critical weaknesses

1. The distinction between structural and parameter errors could be extended to L2.
Section 5 and Appendix E.1 already provide evidence of parameter-identification failures in the barely-novel regime. For L2 responses that introduce additional state, further distinguishing an unsuitable structure from inaccurate parameters would make the failure results more informative.

2. The success criteria and terminology need clarification.
The paper should clarify how explicit hidden variables, higher-order equations, and history-dependent recurrences are evaluated consistently. “Ontology-free” should not imply the absence of state or dynamical representations, and “floor” should refer to the empirical performance of the tested methods rather than a theoretical lower bound over all alternative predictors.

3. Label validation and model coverage remain limited.
Although multiple model-based judges reassess the labels, the key labels lack human validation, particularly for the rare responses involving additional state. The main full-depth capability results also rest on one model that passes the capability check, so the conclusions should remain scoped to the tested models and settings.

Questions for authors
  1. For fully specified L2 laws that introduce additional state, could you hold their structure fixed, re-estimate parameters and any required initial states using only the prompt data, and report which failures persist?
  2. Could you provide mathematically equivalent positive examples to check whether different dynamical representations receive consistent labels and execution-based verification results?
  3. Where does integration begin in Section 7.1? For laws in the original format, does the gap persist over the same prediction window when integration restarts from the last observed ground-truth state, with equations and parameters unchanged?
  4. Could you manually audit responses labeled as using additional state, borderline cases, and a sample of negatives, and report false positives and false negatives?
Reviewer_ZWRZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether LLMs can infer hidden states or history dependence from physical trajectories, going beyond fitting functions over the observed variables. The authors construct a three-level task ladder spanning familiar laws, recombinations of familiar laws, and dynamics involving hidden states. The tested models almost never recover the correct hidden mechanisms. Their numerical predictions generally outperform their own stated equations, but simple linear autoregression can also outperform the models, so this gap does not establish an internal world model that the models cannot verbalize. Representation probes and interventions likewise provide no positive evidence for a mechanism underlying conceptual restructuring.

Critical strengths
  • The research question is valuable. Distinguishing between fitting relationships among existing variables and independently proposing missing states offers a meaningful direction for evaluating scientific discovery capabilities.
  • Some controls are well designed. Comparing predictions and equations within the same response, together with the autoregressive baseline, demonstrates the risks of inferring internal world models from predictive accuracy.
  • The conclusions are appropriately cautious. The authors explicitly acknowledge output-format restrictions, insufficient baseline capabilities, and negative mechanistic results, and narrow their conclusions accordingly.
Critical weaknesses
  1. The task does not adequately isolate conceptual restructuring. These controlled dynamical systems can still be viewed as system identification problems with hidden states. Models already fail where the original function class remains sufficient to explain the data, making it difficult to distinguish parameter estimation difficulties from an inability to propose a new explanatory framework.

  2. The original instructions conflict with the target capability. The prompt requires equations to depend only on x,v,tx,v,t, whereas the true L2 laws require additional states or history. Failure therefore becomes confounded with instruction following. The subsequent format ablation partially addresses this issue, but does not make the main experiment’s null result independent evidence of limited conceptual restructuring.

  3. The model selection is insufficient to characterize the capability frontier. The full experiments primarily rely on Qwen3-1.7B and DeepSeek-V4-Pro, without systematic evaluation of newer, stronger reasoning models such as GPT6-astra or Claude-opus-5.5. Qwen3-1.7B fails the basic capability gate, while the main DeepSeek-V4-Pro configuration disables reasoning and imposes a 1,500-token output budget. The results may therefore reflect limitations in model capability and inference resources. Given claims of research-level mathematical capabilities and related case collections such as OpenAI Math, the authors should test whether stronger models still fail on this task, and whether the observed failures reflect a general bottleneck or limitations of particular configurations. Such external research examples do not establish success on this task, but make this evaluation more necessary.

  4. The evaluation does not adequately cover iterative scientific discovery. The main setting uses single-turn QA, requiring models to propose mechanisms, identify parameters, and formulate a final answer in one response, without isolating where failure occurs. Scientific investigation commonly involves fitting candidate models with tools, inspecting residuals, and revising hypotheses through repeated feedback. Code execution, numerical optimization, symbolic regression, and multiple rounds of reflection could help models recognize that the observed state description is insufficient and propose hidden variables or history dependence. Although the authors allow models to request additional trajectories, this does not constitute a full research harness. Whether the negative results extend to systems with tools and iterative feedback therefore remains untested.

Questions for authors
  1. How can failures in ordinary system identification or parameter estimation be distinguished from failures in conceptual restructuring that requires expanding the state space? Do the current experiments sufficiently support the latter interpretation?
  2. Do the main negative results persist when an output format allowing additional states and history dependence is used consistently?
  3. Do the conclusions hold for stronger reasoning models such as GPT6-astra and Claude-opus-5.5 with sufficient reasoning budgets? How do the authors delimit the capability claims supported by the tested configurations?
  4. With a harness supporting tool use, fitting feedback, and multiple rounds of trial, reflection, and revision, do models still fail to recover the hidden mechanisms? To what extent does the existing active trajectory acquisition experiment cover this iterative research process?

Comments on the paper

1 comment

The authors comment did not address the critical issues I raised in a satisfactory manner and I choose to keep my score.

Arguments that lack justification and terms that are used without being defined are very common in the paper, the specific examples i gave are only a few out of many. A revision of the paper had not been uploaded at the time of writing this reply but even if such revision fixes to those specified issues I believe the paper will still be difficult to read and assess its contribution. Also, fixing only those will not bring the paper close to the level of clarity and communication expected of an ICLR paper. Given this, I choose not to raise my score.

I am not certain that I fully understand the novelty over DiscoverPhysics. Based on the explanation provided, it seems mild.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.