Backward Latent Certainty: When More Certain Forecasts Lie Farther

Visual world models predict how scenes evolve to support physical reasoning and planning. Forecasting in a frozen encoder's feature space lets them reuse pretrained representations without reconstructing future images. This forecast can follow the future motion of every object still in view. The view cannot keep every object inside it. After an object slips out and the image no longer captures it, the forecast continues. The predictor still reports a confidence with that continued forecast. That confidence does not identify an accurate physical position. The forecasts it treats as more certain lie farther from the true position. We call this failure backward latent certainty. We measure this on VideoSAUR and SAVi, encoders that world models widely use and that we keep frozen. On CLEVRER, take the objects that have left the view and keep the forecasts the model calls safest. However many are kept, their average position error is higher than if every such forecast had been kept. Averaged over every number kept, it is 19.2% higher on VideoSAUR and 23.9% higher on SAVi. Mix in the objects that are still visible, and the same rule keeps forecasts with lower position error. We split position error into the error of the predicted features and the error of reading a position from the true features. The published score ranks the first error in the right direction. The second error is the larger one after the object leaves the view, and the score does not contain it. A Gaussian example shows that encoder collapse is enough to make the more certain forecasts farther from the true position, even when the latent predictor is exactly right. We leave those position forecasts unchanged. Using the true positions, we correct that confidence. Averaged over every number kept, the forecasts the corrected confidence keeps have mean position error 19.4% lower on VideoSAUR and 19.5% lower on SAVi than the forecasts the original confidence keeps.