acceptodds
Withdrawn from ICLR 2027

Backward Latent Certainty: When More Certain Forecasts Lie Farther

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual world models predict how scenes evolve to support physical reasoning and planning. Forecasting in a frozen encoder's feature space lets them reuse pretrained representations without reconstructing future images. This forecast can follow the future motion of every object still in view. The view cannot keep every object inside it. After an object slips out and the image no longer captures it, the forecast continues. The predictor still reports a confidence with that continued forecast. That confidence does not identify an accurate physical position. The forecasts it treats as more certain lie farther from the true position. We call this failure backward latent certainty. We measure this on VideoSAUR and SAVi, encoders that world models widely use and that we keep frozen. On CLEVRER, take the objects that have left the view and keep the forecasts the model calls safest. However many are kept, their average position error is higher than if every such forecast had been kept. Averaged over every number kept, it is 19.2% higher on VideoSAUR and 23.9% higher on SAVi. Mix in the objects that are still visible, and the same rule keeps forecasts with lower position error. We split position error into the error of the predicted features and the error of reading a position from the true features. The published score ranks the first error in the right direction. The second error is the larger one after the object leaves the view, and the score does not contain it. A Gaussian example shows that encoder collapse is enough to make the more certain forecasts farther from the true position, even when the latent predictor is exactly right. We leave those position forecasts unchanged. Using the true positions, we correct that confidence. Averaged over every number kept, the forecasts the corrected confidence keeps have mean position error 19.4% lower on VideoSAUR and 19.5% lower on SAVi than the forecasts the original confidence keeps.

Primary area:
learning on time series and dynamical systems
Keywords:
World Models, Latent Uncertainty, Selective Prediction, Encoder Collapse

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aozYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies auto-regressive world models that predict how objects move and additionally estimate how confident they are in each prediction. The authors show that once an object leaves the camera view, this confidence can become misleading. The model is most confident about the predictions that are furthest from the object's real position(?). They argue the cause is that the model's internal representation gradually loses track of where hidden objects are, while its confidence only reflects how well it predicts that representation. On CLEVRER environments, the model's most confident out-of-view predictions are about 20% less accurate than average, and a correction trained with ground-truth positions largely fixes this.

Critical strengths
  • The general phenomenon is interesting. When an object is out of the field of view (or occluded), the model is most confident about exactly the predictions that are "most wrong". This can be easy to miss, because it disappears when you look at visible and hidden objects together. Splitting the error into parts helps explain where the problem comes from.
Critical weaknesses
  • The paper is very hard to follow. Without even considering experiments/results at this point, in the current form this paper is not appropriate for publication. While it is not particularly math-heavy, I repeatedly struggled to understand the content. Sentences are always very short and the terminology is confusing, which seems to me out of the scope of world models / model-based RL terminology/literature. E.g., what is "selection population", "world-error residual", "projected risk", "keep-all", "corrected confidence", "residual risk scoring", "world-error supervision", "forecasts", "clock term", "projected world score", "physical error", "published score", "readout", "scene/scene history", "risk ranking" ...? This is not an exhaustive list. Also risk, score, confidence and certainty seem to mean the same thing in this work. In my opinion, the paper needs a major rewrite ...

  • In essence, the paper is a limited empirical observation on a subset of world models. In particular, the observations hold more for slot-based models (VideoSAUR and SAVi on CLEVRER) rather than dense-based ones (DINOv2) where the evidence is too thin. For IntPhys, the slot-based observations also become weaker (in comparison to CLEVRER). Overall, different kind of WMs (as cited by the authors) need further investigation for broader conclusions.

  • Figure 1 right is unclear and does not present the idea. I can't make sense of the x- and y-axis labeling and when the peak in latent error occurs. What is the relation to the two images? What is the statement here? What is "true spread"?

  • The connection to some of the discussed world models (like DINO-WM and V-JEPA 2-AC) is not clear, since these models do not output covariances and they are also action-conditioned. So, the empirical evidence for broader justification is not given.

  • The AI use statement does not disclose everything correctly as described by the ICLR website.

Questions for authors
  • No questions.
Reviewer_MwFJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether uncertainty in a forecast of internal visual features helps select accurate predictions of an object's physical position. A fixed decoder maps these features to positions. After an object leaves view, the uncertainty score, called projected latent variance, can favor forecasts with larger position errors. The same score can still correctly rank the error between predicted and actual features after both are mapped to positions. The authors measure the harm of selecting these forecasts and train a better score using position-error labels offline, without changing the forecasts.

Critical strengths

The paper measures position error as different fractions of forecasts are kept, using mean squared error and AURC, a summary across those fractions. All scores use the same records. Fitted scores use labels from other videos, keeping the evaluated video's own labels out of fitting. Checks with alternative decoders, timing information, different losses and random seeds, along with assignments that use no future observations, address substantial alternative explanations. The comparison with a score trained directly on physical error is particularly useful.

The mathematics is valid and supports a coherent account of the target mismatch. The error decomposition keeps the interaction term between latent prediction error and decoder error. The Gaussian example shows that information loss can produce decreasing latent variance and increasing physical risk. The authors also distinguish offline supervision from runtime inputs and describe IntPhys as transfer with locally refitted decoders and risk scores. The citation check found no discrepancy among the 32 references. The main novelty is the measured target mismatch and its selection cost; the Gaussian identities and supervised correction use established ingredients.

Critical weaknesses

The empirical decoder failure does not identify irreversible loss of position information. Error when decoding the true representation can reflect both unavailable information and limitations of the fixed decoder, which was trained while objects were visible. A different decoder could recover some of what the current decoder misses. The Gaussian construction establishes a sufficient mechanism; it does not identify the mechanism operating in the tested encoders. The authors often qualify their interpretation, and checks with nonlinear decoders make a purely linear artifact less likely. Nevertheless, claims that the results show a “collapse of position information” should be narrowed. The correction is trained on total position error on a log scale, including the interaction term; it does not identify a separate decoder-error component.

The main evaluation selects videos offline, retaining the 75% with the lowest error when matching objects across the whole video. All methods use the same retained forecasts, making comparisons fair within this set. Object matches based on labeled positions while visible are held fixed during absence. A separate check using only the last visible context preserves negative correlations, but has not reproduced the main AURC selection curves. The proposed correction improves AURC by roughly 19% over the harmful original score, but only about 4.0% for VideoSAUR and 0.33% for SAVi relative to random selection. The stronger no-offset score, trained directly on position error, improves on random by about 11.7% and 3.0%, respectively. The corrected SAVi selector still has higher error than expected random selection at the first four reported low acceptance levels; paired intervals would clarify where selection helps.

Questions for authors

Can the existing assignment rule that uses only visible context reproduce the original, corrected, and no-offset selection curves on the main population?

Can you report paired improvements against random at useful acceptance levels and release per-record scores, errors, and cohort definitions? The abstract should also state that position labels train the score offline, while runtime uses the forecast and rollout index.

Reviewer_D5hfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper finds that a world model can become increasingly confident about its latent predictions while becoming increasingly inaccurate about the underlying physical state. In this case, uncertainty can become negatively correlated with physical error. The authors attribute this issue to a mismatch between latent space predictability and physical stage informativeness. Based on this, this paper introduces a lightweight post-hoc risk correction solution via the using of physical state prediction error to improve uncertainty based forecast selection.

Critical strengths
  1. The problem itself is interesting and quite important. This paper investigates whether a world model’s uncertainty about future latent representations reliably reflects the accuracy of its predictions of objects’ future physical positions. This is an interesting and very important research question for those uncertainty based forecast selection problems.

  2. The experimental results are convincing in demonstrating both the necessity of this research as well as in explaining the effectiveness of the proposed solution.

Critical weaknesses
  1. The requirement on physical state supervision might lower the potential of applying the proposed solution in real-world environments.

  2. The motivation is interesting, while the solution is quite simple, e.g. it’s definitely reasonable to rely on physical error as correction signal. However, the theory level contribution seems limited. The authors are expected to explain more on the alternative correction signals, or provide more structured analysis of this simple solution.

  3. The experimental setting is also limited, without real world environment related settings. Given the critical importance of uncertainty in robot related field, e.g. robot manipulation, VLA, world action models, etc, it’s strongly suggested to extend the experimental setting in a more general way to verify the broader impact of the proposed solution.

  4. Experimental results show that the correction improves selection. However, there is no evidence to explain if there is any possible solutions to repair prediction when it’s already wrong. Problem mitigation is suggested to come after issue auditing.

Questions for authors
  1. Is it possible to relax the requirements for physical state supervision? How?

  2. Is it possible to mathematically explain the intuition behind the physical error based correction technique?

  3. What are the main challenges of applying the solution to more general real-world settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.