- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper examines how candidate action sets affect world-model evaluation. The authors fix the model, initial state, goal, and candidate count, then change the action sequences and compare prediction errors, reference-relative scores, and performance gaps between models. They also separate the selected action’s realized cost into the best available cost and the excess cost of selection. The experiments show that candidate composition changes scores and comparison margins, while the implications for practical model selection and planning remain unresolved.
Critical strengths
Holding the model and query conditions fixed while changing only the candidate set makes the comparison easier to interpret. The authors also examine raw prediction errors, showing that the observed margin changes are not entirely due to normalization. Unsupported findings are clearly reported.
Critical weaknesses
-
I do not see a sufficiently substantial new insight. Evaluation depends on the test distribution, relative scores depend on their reference, and candidate-set quality differs from selection quality within that set. These distinctions are well established. Prior work has also discussed the difference between data-averaged prediction error and fidelity on the distributions queried by a planner. This paper adds controlled measurements of how candidate sets affect comparison margins, which have some value. However, it does not explain which training or model factors cause this sensitivity, or identify a concrete model-selection judgment that these findings would change. The top-ranked model in the shared-encoder comparison remains unchanged, and the planner extensions do not establish changes in between-model performance margins. I appreciate the authors’ careful qualification of their claims, but the resulting contribution remains narrow.
-
The experiments do not form a clear overall argument. The strongest score and margin findings come from Push-T. The other environments use different candidate constructions and address different questions, so they do not directly replicate the same finding across environments. PointMaze does not support the anticipated regret change, and the released-planner experiments do not establish margin changes either. These results appropriately limit the scope of the findings, but do not clarify their broader significance. The presentation further interleaves protocol details, post-hoc analyses, and scope qualifications without clearly separating the central evidence from supporting observations. For a paper whose basic idea is relatively simple, this organization makes the contribution unnecessarily difficult to assess.
Questions for authors
How do the findings on constructed candidate sets relate to the distributions produced by an operational planner? Can the existing experiments explain why clear margin changes appear in the former but remain unresolved in the planner extensions?