- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Authors study when a learned probabilistic model remains useful after training when the graph composition, available evidence, or query changes. Its central theoretical message is sensible: good performance on the original prediction task does not guarantee that the learned representation contains the joint information required for later conditional queries. The paper formalizes this through identifiability and retention criteria and supports the theory with several controlled experiments.
Critical strengths
I find the theoretical perspective interesting and several controlled experiments convincing as demonstrations of the proposed information criteria. In particular, the distinction between information needed for current prediction and information needed for subsequent conditioning is useful. The authors report uncertainty across multiple generated worlds and disclose several negative or failed checks rather than presenting only successful experiments.
Critical weaknesses
However, the paper's practical contribution remains unclear. The motivating applications are broad, while the formal results assume known finite operators, declared future queries/evidence, and supplied structural interfaces, and most experiments are constructed probabilistic systems designed around those assumptions.
The paper motivates the work using world models, LLMs, compositional inference, and prediction under changing evidence, but the connection between these applications and the formal experimental setting remains abstract.
I also found the terminology around "prediction" insufficiently precise. Several experiments are more naturally understood as probabilistic classification or conditional-probability estimation followed by Bayesian updating. This matters because the paper's central statement that "prediction quality does not imply reusability" could be interpreted much more broadly than what is actually demonstrated. The paper would benefit from explicitly distinguishing these notions and narrowing its claims accordingly.
Questions for authors
I would ask the authors to explicitly define what they mean by prediction, how it differs from probabilistic classification in their experiments, and whether the results apply to arbitrary predictive models or specifically to models intended to represent coherent probabilistic laws.
I would like to see at least one experiment on an established world-model environment showing two models with comparable familiar-task prediction/classification performance but meaningfully different downstream reuse ability.
The conclusions should therefore distinguish more sharply between: what is proved for known finite probabilistic systems? what is demonstrated in synthetic learned mechanisms? and what is merely suggested for modern neural world models?
Thank you for asking how prediction, evidence updating and world models connect. We clarified the scope of the finite criteria and added the requested test on simulated walking (DeepMind Control Suite Walker).
Matched familiar prediction. The new simulated-walking comparison shows that models with similar familiar accuracy can differ after the same feedback. For two neural models trained independently on the same trajectories, all four paired 95% intervals for familiar-task Brier differences lay within ±0.005 on 8,192 fresh episodes. Their reversed-plan scores were close before feedback. Given the same step-four height, step-eight Brier was 0.1193 vs 0.1430, with paired gap 0.0237 [0.0216, 0.0258]. A directly supervised classifier for that query scored 0.1273. Both action plans occur in training. A separate added Gaussian control uses its own 2,048 episodes and holds the single-time distributions fixed while changing cross-time covariance (revised §8 and Appendix A). The supplement's
walker_md_aggregate/verify.pychecks the neural Brier point estimates from score sums.Prediction and the finite criteria. The revised Introduction defines a prediction as the expectation E[f], which for an indicator is the class or event probability used in probabilistic classification. Evidence with likelihood L changes it to E[fL]/E[L]. For coherent laws, known finite operators, strictly positive initial laws and linear expectation summaries, our exact criteria characterize when records and states determine both conditional answers and evidence probabilities. Revised §4 clarifies that a recursively updated state must also tell which evidence updates are available and determine the resulting summary (see also original §4).
Which records to collect. In original §6, the test identifies a triple interaction missing from pair records. In 32 fresh worlds, collecting relevant triples reduced mean excess Brier from 0.022783 to 0.002182 compared with equal-cost unrelated triples, with improvement in every world.
Reuse on unseen trees. In the original experiment on eight binary worlds (§5 and Figure 2), frozen local mechanisms with the supplied generating factorization transfer to unseen trees under the same interface used by the graph-conditioned neural comparison. Familiar-chain risks meet the prespecified matching margin. On unseen trees, excess Brier is 0.000320 for the local mechanisms and 0.021835 for the graph-conditioned model.
Additional neural evidence and next test. In a separate added 512-episode check, both released DreamerV3-port checkpoints improve their own Walker forecasts after noisy height evidence (revised Appendix A). A next question for general-purpose pretrained world models is whether they can identify which intermediate measurement will most improve a later forecast (Discussion).