- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces Retrospective Physical Process Reasoning (RPPR), a task that requires models to determine whether a physical transformation from an initial image to a target image is feasible and, if so, reconstruct the intermediate causal steps. This paper present RetroHolmes, a benchmark containing 7,227 image pairs across 13 domains and four categories of physical transformations. They further propose a Simulate-and-Verify framework consisting of an Initial Planner, a Transition Simulator, and a Verifier.
Critical strengths
- Retrospective reasoning about physical transformations is an interesting direction that goes beyond conventional image understanding and forward prediction. Evaluating whether two states can be connected by a physically plausible process is potentially valuable for assessing world-model capabilities.
- RetroHolmes spans 13 domains and multiple types of physical transformations. Its evaluation dimensions provide a useful starting point for analyzing different failure modes. The separation of planning, transition simulation, and verification offers a clear and interpretable pipeline.
Critical weaknesses
- Ambiguous definition of physical reachability. The paper does not sufficiently clarify the assumptions under which a transformation is considered reachable. Physical impossibility, missing resources, incomplete observations, and insufficient evidence may lead to different labels but appear difficult to distinguish under the current formulation. The human study in Appendix D.2 is insufficient, employing only three annotators without reporting inter-annotator agreement.
- The claimed 21.67% improvement over GPT-5.5 is misleading. Since Simulate-and-Verify is fine-tuned on 4,652 task-specific pairs (Qwen3-VL-8B and Wan2.2) while GPT-5.5 is evaluated in a zero-shot setting, this performance gap primarily reflects the benefit of supervised training on the benchmark dataset rather than the superiority of the proposed framework itself.
Questions for authors
What precise assumptions about available actions, resources, environmental conditions, and time horizon determine whether a target state is reachable? How are ambiguous cases handled?