acceptodds
Under review as a conference paper at ICLR 2027

Knowing When Not to Answer: Self-Assessed Measurability and the Limits of Physics Transfer

Abstract

A world model that has learned physics should carry something to physics it has not seen, and the field tests this by training on one system and evaluating on another. We show that this test usually does not measure what it is taken to. The reason is structural: a latent world model compresses a field, evolves it and decompresses, so its error contains both what the dynamics got wrong and what the round trip lost, and which term dominates depends on something rarely reported, namely how far the physics moves between the two frames being compared. When the field barely moves and the codec is lossy, the error measures the codec, and a model that has learned nothing transferable still scores well. The fix is a check the model runs on itself before acting, with no labels and no access to the future. Let ρ be its reconstruction error divided by the observed frame-to-frame change: below 1 the comparison is attributable to dynamics, at or above 1 it is not. The threshold is derived rather than fitted, and over 52 conditions the rule makes no errors; two stress tests and two weaker checks leave it intact. Applied to a standard PDE benchmark it disqualifies the protocol rather than the models: of 42 cross-family pairs across eight datasets, none is measurable at the single-step horizon those benchmarks use. This is not undertraining, nor simply a weak codec. Training the source to convergence improves in-domain attributability by over an order of magnitude and cross-domain attributability by under 2×, while refitting the codec on the target buys attributability only by destroying the skill being measured. What survives is narrower and real. Inside one physics family transfer occurs and is measurable, in three of six directions among three acoustic geometries, one beating codec-matched persistence by 0.568 [0.530, 0.606] at every horizon and replicating across seeds; across families, nowhere. The consequence is a world model that knows when not to answer, whose failure mode is decidable rather than probabilistic: we placed conditions on the boundary itself and the classes still did not overlap. It also predicts the size of its own forecast error well enough to be calibrated, at 90.6% coverage for a nominal 90% once the interval accounts for shift between families.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
world models, scientific machine learning, PDE surrogates, transfer learning, evaluation methodology, selective prediction, abstention, latent dynamics models, uncertainty quantification, benchmark analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DDnpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether field-space prediction errors really measure dynamics transfer in latent physics world models, since reconstruction error can drown out temporal changes. It proposes a reconstruction-to-change ratio and a two-frame decision-time approximation, with a fixed threshold of one to flag uninformative evaluations and trigger abstention. On LATH-v1 with The Well, decision-time and hindsight classifications agree across 52 conditions. All 42 cross-family settings fail the single-step criterion. Within-family acoustic transfer meets the criteria in three of six directions, and no cross-family setting shows significant positive skill. Family-held-out forecast-error intervals achieve 90.6% empirical coverage at a nominal 90% level after shift-aware widening.

Critical strengths

The paper points out a real limitation in physics-transfer evaluation, since reconstruction error can hide dynamical skill, and its focus on reconstruction quality and persistence baselines helps make sense of benchmark results. Also, the inclusion-to-maze transfer result holds across two seeds and all tested horizons, which strengthens the case for within-family transfer and gives a useful positive control.

Critical weaknesses

The paper needs to justify its key threshold better, because it's unclear why reconstruction error exceeding frame-to-frame change must mean prediction results can't reveal dynamical skill, and agreement between the two ratios only shows that the decision-time proxy reproduces hindsight classifications, not that the criterion establishes dynamical attributability. Also, the proposed ratio has no demonstrated edge over a simpler score, since normalizing reconstruction error by field amplitude gives the same decisions on all 52 conditions, so it's unclear what the temporal change actually adds.

Questions for authors

See weaknesses

Reviewer_AmudOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper raises an important question about the reliability of physics-transfer evaluation and presents several interesting empirical observations. However, its central contribution relies on a measurability criterion whose theoretical interpretation is not sufficiently justified. In particular, the reconstruction-to-change threshold does not establish the claimed impossibility of attributing prediction improvements to learned dynamics, while the reported perfect classification results largely validate agreement with the criterion's own definition rather than its underlying scientific meaning. Furthermore, the broad conclusions about cross-physics transfer are based on a single model architecture and a codec-dependent measurement.

Critical strengths
  1. The authors investigate reconstruction quality, temporal horizons, cross-family transfer, and within-family generalization, providing useful empirical evidence about the limitations of the evaluated world model.
  2. The exploration of self-assessed measurability, intervention prediction, and forecast-error estimation offers an interesting perspective on improving the reliability of physics-based predictive systems.
Critical weaknesses
  1. The reported perfect classification performance primarily measures agreement between the estimated ratio and labels defined by the same reconstruction-to-change threshold. This does not independently establish whether the rule correctly identifies cases where dynamical information is genuinely unmeasurable, making the claimed decisiveness difficult to substantiate.
  2. The main experiments rely on a single lightweight world-model architecture and its associated code, despite making broad claims about the limitations of standard physics-transfer evaluation.
Questions for authors
  1. Can the authors validate the measurability criterion against an independent operational measure of dynamics contribution, rather than labels derived directly from the proposed threshold?
  2. Does the proposed criterion remain effective across alternative encoder–decoder architectures, stronger pretrained physics models, and different reconstruction objectives?
  3. How reliably does the proposed decision rule operate at the individual trajectory-window level, rather than only across aggregated experimental conditions?
Reviewer_GqbDOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper discusses how to evaluate the transferability of physical prediction models. These models first compress the current state, then predict how the compressed state will change, and finally restore it to physical quantities such as temperature and velocity.

Critical strengths

None

Critical weaknesses

1.The “zero errors” results in Sections 4.3–4.4 demonstrate that the deployable estimate ρ^\hat{\rho} and the oracle ratio ρ\rho yield the same threshold-based decisions on the evaluated conditions. However, accurately estimating the representation-to-change ratio is distinct from establishing that ρ=1\rho=1 separates conditions in which forecast-error comparisons can or cannot be attributed to dynamics. The former is supported by the reported experiments; the latter requires independent justification beyond agreement with labels defined by the criterion itself.

2.The benchmark-level conclusion also exceeds the scope of the evaluation. The experiments use a single architecture, and Section 10 explicitly acknowledges that “a larger or differently designed encoder is untested.” Since the numerator of ρ\rho depends on the particular codec, the results characterize the evaluated model–dataset–horizon combinations. They do not, by themselves, establish that the benchmark protocol is generally incapable of measuring dynamics transfer. The conclusion should therefore be restricted to the tested architecture and evaluation settings.

Questions for authors

1.What assumptions are required for “reconstruction error is a lower bound on forecast error”? Does this require the encoder to find the best reconstruction achievable by the decoder? How are these conditions guaranteed in the model used in this paper? Are the labels for “correct decisions” determined entirely by whether the oracle ratio exceeds 1?

2.Is there validation independent of this definition, such as comparing known-correct, incorrect, and identity (no-update) dynamics modules under a fixed encoder and decoder?

3.How do the authors rule out the explanation that re-encoding changes the latent representation, making the original transition module incompatible with it? Keeping that module’s weights fixed does not guarantee that the meaning of the information it processes remains unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.