acceptodds
Under review as a conference paper at ICLR 2027

Candidate-Dependent Margins in Latent World-Model Evaluation

Abstract

Candidate actions change the measured margins between fixed latent world models, even when their top-ranked predictor remains unchanged. We quantify this dependence through paired candidate sets at fixed checkpoints, initial states, goals, horizons, and candidate counts. Among seven predictors sharing an encoder on 64 Push-T families, all six mean gain advantages over a post-hoc logged-data comparator contract by –. Their raw prediction-error advantages also contract by –, so this sensitivity extends beyond reference normalization. These six paired gaps share one comparator and do not constitute independent replications. Twelve of 21 post-hoc score-response contrasts survive a studentized sign-flip test with Holm correction; paired tests retain the same findings. Separately, a public DINO-WM reproduction gains in persistence-relative score despite an increase in prediction error. Family-level accounting separates the opposing model-error and reference-error terms. A prospective PointMaze study identifies worse available task opportunity without resolving a change in within-set regret. We provide a reproducible audit of prediction error, reference difficulty, available opportunity, selected cost, regret, and cost discrimination. These measurements quantify candidate-dependent comparison margins and delimit their implications for action selection.

Primary area:
datasets and benchmarks
Keywords:
Latent world models, Model Evaluation, Candidate-action distributions, Model comparison, Model-based planning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fvTPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines how candidate action sets affect world-model evaluation. The authors fix the model, initial state, goal, and candidate count, then change the action sequences and compare prediction errors, reference-relative scores, and performance gaps between models. They also separate the selected action’s realized cost into the best available cost and the excess cost of selection. The experiments show that candidate composition changes scores and comparison margins, while the implications for practical model selection and planning remain unresolved.

Critical strengths

Holding the model and query conditions fixed while changing only the candidate set makes the comparison easier to interpret. The authors also examine raw prediction errors, showing that the observed margin changes are not entirely due to normalization. Unsupported findings are clearly reported.

Critical weaknesses
  1. I do not see a sufficiently substantial new insight. Evaluation depends on the test distribution, relative scores depend on their reference, and candidate-set quality differs from selection quality within that set. These distinctions are well established. Prior work has also discussed the difference between data-averaged prediction error and fidelity on the distributions queried by a planner. This paper adds controlled measurements of how candidate sets affect comparison margins, which have some value. However, it does not explain which training or model factors cause this sensitivity, or identify a concrete model-selection judgment that these findings would change. The top-ranked model in the shared-encoder comparison remains unchanged, and the planner extensions do not establish changes in between-model performance margins. I appreciate the authors’ careful qualification of their claims, but the resulting contribution remains narrow.

  2. The experiments do not form a clear overall argument. The strongest score and margin findings come from Push-T. The other environments use different candidate constructions and address different questions, so they do not directly replicate the same finding across environments. PointMaze does not support the anticipated regret change, and the released-planner experiments do not establish margin changes either. These results appropriately limit the scope of the findings, but do not clarify their broader significance. The presentation further interleaves protocol details, post-hoc analyses, and scope qualifications without clearly separating the central evidence from supporting observations. For a paper whose basic idea is relatively simple, this organization makes the contribution unnecessarily difficult to assess.

Questions for authors

How do the findings on constructed candidate sets relate to the distributions produced by an operational planner? Can the existing experiments explain why clear margin changes appear in the former but remain unresolved in the planner extensions?

Reviewer_9kT6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how changing candidate-action sets affects comparisons between fixed latent world models. Its main experiment evaluates seven shared-encoder predictors on 64 paired Push-T families, reporting 36-65% contractions in their mean gain advantages over a post-hoc logged-data comparator and 36-71% contractions in raw-error advantages. Twelve of 21 score-response contrasts survive Holm correction. A separate public DINO-WM reproduction shows increased persistence-relative gain despite increased prediction error. Further studies separate available task opportunity from within-set regret and identify candidate sets with no task-cost discrimination. The proposed audit reports these quantities jointly.

Critical strengths
  1. The main comparison goes beyond a denominator effect.

Section 4 and Table 23 provide the paper’s strongest argument. For example, BA7’s raw-error advantage over LD6 falls from 0.4990 to 0.3181, with a paired change interval excluding zero. The full analysis retains all 21 score-response contrasts, and the corrected sign-flip and paired-t procedures identify the same twelve findings. Thus, the result is not merely that a normalized score changes when its reference changes: the models respond differently to the candidate intervention in their common prediction space. The explicit treatment of the six LD6 comparisons as dependent is also important.

  1. The joint reporting supports more careful conclusions than any single metric would.

The public-reproduction accounting is performed before family aggregation, avoiding an invalid interpretation of pooled ratios. Likewise, the PointMaze result distinguishes worse available opportunity from an unresolved regret change, rather than attributing the selected-cost change to the model. These distinctions make the audit useful even when a study does not establish planning harm. The authors’ retention of these limits is a substantive strength.

Critical weaknesses
  1. The evidence for a sufficiently significant empirical contribution remains too concentrated in one candidate-bank comparison. The paper correctly treats sample-dependent skill scores and prediction-control mismatch as established concerns; prior work already supports those broader points. Its new contribution therefore rests mainly on the measured size of the comparison-margin changes, not the accounting identities themselves.

That central result uses one ordinary/rich bank construction in Push-T and seven configurations of one shared predictor design. The additional public-model and controlled studies mainly establish score-error co-increases, while PointMaze addresses opportunity and regret; they are not independent confirmations of the seven-model margin result. The operational planner extensions also leave their model-margin contrasts unresolved. Consequently, I am not yet persuaded that the reported magnitudes establish a broadly useful finding beyond this particular evaluation setup. A ranking reversal or improved control is not necessary, but stronger evidence connecting the chosen bank mixtures to evaluation conditions researchers would actually use would materially strengthen the significance argument.

  1. The public-checkpoint result has an unfinished measurement-validation step. Appendix E.2 explicitly states that an independent end-to-end comparison covering input tensors, action packing, predicted endpoints, and target alignment remains unavailable. Table 14’s encoder/reference trace covers one family and four endpoints; reaggregating stored predictions does not check their generation. The explanation for the original/reproduction input-resolution difference is reasonable and is not itself evidence of a defect. Nevertheless, the missing full-path check limits how much weight I place on this public checkpoint as evidence beyond the internal predictors. A small deterministic validation using the same reproduction checkpoint, rather than requiring agreement between different releases, would address this concern.
Questions for authors

a) How do the seven-model comparison margins vary under reasonable changes to the candidate-component weights in Tables 11–12? The ordinary and rich banks change Gaussian-proposal counts from 42 to four, alongside other composition changes. Table 20 reports one predictor’s generator-specific levels, not the corresponding between-model margins. A small reanalysis of those margins could show whether the observed scale has a broader basis than the selected mixture pair and would affect my significance assessment.

b) Can you complete the end-to-end check identified in Appendix E.2 on a fixed sample from both banks, comparing your measurement path with the native inference path for the same public reproduction checkpoint? Agreement on context, action packing, predicted latents, and aligned targets would substantially increase my confidence in Section 5’s external evidence.

Reviewer_H4PTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary
  • The paper examines the impact of the selected candidate action sequences or their distribution on the evaluation of latent world models, by evaluating prediction accuracy and planning performance when the candidate action sequences (or rather their distributions) change.
  • The authors found that changing the action sequences used for evaluation significantly changes the evaluation prediction accuracy between models. Based on this observation, they show an evaluation procedure that reports prediction errors, difficulty, task opportunity and selection regret.
Critical strengths
  • S1 Experimental Analysis
    • The empirical study is quite extensive with controlled comparisons across different candidate sets.
    • The paper provides a thorough statistical analysis of the considered models, including multiple training seeds, confidence intervals and corrections for duplicates.
  • S2 Potentially Useful Evaluation Perspective -The paper highlights that relative prediction scores can be misleading when the difficulty of the reference prediction changes. Therefore, reporting absolute prediction errors alongside normalized values might be a useful reminder for empirical evaluations.
    • The proposed evaluation approach also distinguishes between the quality of the available actions and the ability of the model to select among them. While the underlying metrics are not new, reporting them together may provide a more informative overall picture of model performance.
Critical weaknesses
  • W1 Presentation and Accessibility The paper is significantly difficult to follow, and identifying its actual contributions requires unnecessary effort from the reader. In my opinion, this is a considerable weakness, particularly for a paper whose main contribution is an empirical evaluation study.
    • Introduction: The paper starts with a very steep introduction without properly motivating or explaining the underlying setting. For example, the authors immediately discuss world models that "score candidate futures", without explaining what this means. It assumes that the reader is perfectly familiar with the relevant technical terms and concepts. Even the abstract focuses almost entirely on numerical results, rather than explaining the central idea and contribution. A more intuitive introduction to the problem and evaluation procedure would significantly improve accessibility.
    • Mathematical Formulation: Section 3 suffers from similar problems. The authors do not even introduce a world model explicitly in their mathematical notation, for example through a next-state prediction conditioned on an action sequence. Instead, several quantities are introduced without sufficient explanation. Examples include the candidate bank, the endpoint predictor, the frozen representation, the dimension dd in Eq. (1), and the goal representation zgz^g in line 140. While some of these quantities are defined or explained elsewhere, their presentation in the main text is insufficient to understand the setup without considerable effort. For readers familiar with the field of world models, some of these quantities might be clear from context. Nonetheless, the presentation makes it unnecessarily hard and confusing even for these readers.
    • General Readability: Throughout the paper, technical terminology is used without introduction or sufficient motivation, Generic or elaborate formulations often appear instead of precise explanations of the underlying, frequently simple, ideas. The paper would benefit from simpler language, clearer definitions, and a more focused narrative. In its current form, the presentation makes it unnecessarily difficult to assess the actual scientific contribution.
  • W2 Novelty and Contribution I have considerable concerns about the novelty and significance of the contribution. The central observation appears relatively straightforward, while the experiments do not convincingly demonstrate why the proposed evaluation procedure is necessary in practice.
    • Expected distribution dependence: The dependence of expected prediction error on the evaluation distribution is a basic property of statistical evaluation. It is not necessarily a methodological deficiency. In most applications, the distribution used for evaluation would depend on the specific application, with sufficiently many samples and appropriate uncertainty estimates. The paper does not clearly articulate what new insight its findings provide beyond this observation.
    • Limited Methodological Novelty: The proposed evaluation procedure combines largely established quantities, including prediction MSE, persistence-relative skill scores, and within-set regret. The authors acknowledge the existing literature on reference-relative evaluation. Consequently, the methodological contribution appears rather limited, and the main novelty rests on the empirical quantification of these effects.
    • Unclear Practical Consequences: Changes in measure performance may be quite large, but qualitatively, the strongest model seems to remain the same in the main experiments. The additional planning experiments also do not seem to show a harmful effect on model selection. It therefore remains unclear whether the observed sensitivity has meaningful practical consequences for practitioners.
    • Experimental Scope and Interpretation: The strongest findings come from seven configurations of the same architecture, not from independently established world models. Furthermore, these configurations differ simultaneously across training data, objectives, and training duration. While the authors acknowledge this limitation, it restricts the interpretation of the results. The proposed decompositions help separate the measured quantities, but do not explain the underlying causes of the observed changes. A demonstration that existing model comparisons lead to misleading conclusions would make the contribution more convincing.

Overall, I find the contribution difficult to assess in its current presentation. The paper would benefit from a rewrite that clearly introduces the problem and the challenges of world model evaluation, separates its contributions from existing evaluation metrics/practises, and provides a stronger motivation for the practical relevance of the problem. I would encourage such a rewrite to prepare the paper for resubmission.

Questions for authors
  • Q1 W1 What new scientific insight does the observed candidate dependence provide beyond the seemingly trivial fact? In particular, what limitation of standard evaluation using a clearly specified action distribution does the proposed method address?
  • Q2 W2 Could the authors clarify what practical consequences follow from their findings? In particular, can they identify a setting in which the proposed evaluation procedure leads to a different or more reliable model-selection conclusion than standard evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.