acceptodds
Under review as a conference paper at ICLR 2027

RetroHolmes: When Semantic Plausibility Fails Retrospective Physical Process Reasoning

Abstract

Vision–Language Models (VLMs) are widely used for visual understanding, yet current evaluation protocols fail to assess whether these capabilities are grounded in physical reasoning. To address this gap, we introduce Retrospective Physical Process Reasoning, a new evaluation paradigm to reason backward from outcomes under explicit physical constraints. Building on the paradigm, we present RetroHolmes, the first real-world benchmark for Retrospective Physical Process Reasoning, comprising object-centric image pairs annotated with reachability labels and causal step sequences across diverse physical transitions. Using RetroHolmes, we analyze VLMs and uncover systematic failure modes, including judgment bias in reachability assessment and belief dominance over physical evidence, mirroring sycophancy behavior observed in large language models. Our quantitative analyses link these failures to reliance on linguistic priors and attention concentrated on visually invariant regions, suggesting limited physical simulation of the intermediate states connecting visual endpoints. To address these limitations, we propose Simulate-and-Verify, an analysis-by-synthesis framework that grounds reachability judgment and step reconstruction in visual simulation. Experiments show that Simulate-and-Verify improves judgment accuracy by 21.67 percentage points and reduces belief dominance by 10.39 percentage points compared with GPT-5.5, demonstrating the effectiveness of visual simulation in grounding physical reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Physical Process Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qseNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Retrospective Physical Process Reasoning (RPPR), a task that requires models to determine whether a physical transformation from an initial image to a target image is feasible and, if so, reconstruct the intermediate causal steps. This paper present RetroHolmes, a benchmark containing 7,227 image pairs across 13 domains and four categories of physical transformations. They further propose a Simulate-and-Verify framework consisting of an Initial Planner, a Transition Simulator, and a Verifier.

Critical strengths
  1. Retrospective reasoning about physical transformations is an interesting direction that goes beyond conventional image understanding and forward prediction. Evaluating whether two states can be connected by a physically plausible process is potentially valuable for assessing world-model capabilities.
  2. RetroHolmes spans 13 domains and multiple types of physical transformations. Its evaluation dimensions provide a useful starting point for analyzing different failure modes. The separation of planning, transition simulation, and verification offers a clear and interpretable pipeline.
Critical weaknesses
  1. Ambiguous definition of physical reachability. The paper does not sufficiently clarify the assumptions under which a transformation is considered reachable. Physical impossibility, missing resources, incomplete observations, and insufficient evidence may lead to different labels but appear difficult to distinguish under the current formulation. The human study in Appendix D.2 is insufficient, employing only three annotators without reporting inter-annotator agreement.
  2. The claimed 21.67% improvement over GPT-5.5 is misleading. Since Simulate-and-Verify is fine-tuned on 4,652 task-specific pairs (Qwen3-VL-8B and Wan2.2) while GPT-5.5 is evaluated in a zero-shot setting, this performance gap primarily reflects the benefit of supervised training on the benchmark dataset rather than the superiority of the proposed framework itself.
Questions for authors

What precise assumptions about available actions, resources, environmental conditions, and time horizon determine whether a target state is reachable? How are ambiguous cases handled?

Reviewer_TJckOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript introduces a benchmark called “RetroHolmes,” which evaluates whether VLMs can judge physical reachability between two states and reconstruct the intermediate steps. The authors further propose the Simulate-and-Verify framework, which performs well on this retrospective physical process reasoning benchmark.

Critical strengths
  1. The idea is interesting, and the dataset construction is thorough.
  2. The overall benchmark is well designed, and the authors propose five evaluation dimensions and three metrics.
  3. To the best of my knowledge, this is the first work to jointly evaluate reachability judgment and step reconstruction.
Critical weaknesses
  1. To my knowledge, the authors should cite [1] in their related work.
  2. Why is the random choice score on ED4 in Table 2 26.64 rather than around 50?
  3. What is the training target for unreachable samples?
  4. There are some presentation problems, including incorrect cross-references and inconsistencies between figure/table titles and their content.
  5. There are even several details where experimental values are mistyped or model names may be incorrect. For instance, the description of ED5 in Observation 1 is inconsistent with Table 2 (and this is not the only instance; the authors are requested to re-check), which suggests that this may not be a well-prepared manuscript.
  6. Although some annotation criteria are described (Appendix B.1), the paper does not report inter-annotator agreement, nor does it provide rubrics for the human judgments used in ED4 and for the manual curation of ED5.
  7. What videos is the verifier trained on?
  8. The web data in the manuscript were crawled from videos by 21 YouTube creators, but the paper does not mention licensing or usage permissions. Since Appendix D.1 refers to a “released” RetroHolmes benchmark, please clarify the data licensing and whether the released benchmark includes frames extracted from these videos.

Please revise the manuscript thoroughly so that the improvements merit a higher score.

[1] Chang, C. Y., Huang, D. A., Xu, D., Adeli, E., Fei-Fei, L., & Niebles, J. C. (2020, August). Procedure planning in instructional videos. In European Conference on Computer Vision (pp. 334–350). Cham: Springer International Publishing.

Questions for authors

As mentioned in the Weaknesses.

Reviewer_3DotOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RetroHolmes, a benchmark for retrospective physical process reasoning, where vision-language models (VLMs) must determine whether a target physical state is reachable from an initial state and reconstruct the corresponding causal steps. The benchmark contains 7,227 image pairs spanning various physical transformations, with evaluation dimensions covering causal modeling, temporal reversibility, irreversible state changes, and robustness to misleading human beliefs.

The authors further propose Simulate-and-Verify, a three-stage framework consisting of an initial planner, a video-based transition simulator, and a verifier. The framework uses a fine-tuned Wan2.2 video generation model to synthesize intermediate transitions between fixed endpoints, which are subsequently examined by a fine-tuned Qwen3-VL verifier. The paper reports substantial improvements over strong proprietary models, including a 21.67% gain in judgment accuracy over GPT-5.5.

Critical strengths

The paper addresses a potentially important distinction between semantic plausibility and physical feasibility. Evaluating whether models can reconstruct valid physical transitions, rather than merely describe visually plausible outcomes, is a meaningful research direction. Temporal irreversibility and belief-conflict sensitivity is interesting. These settings may expose weaknesses not captured by conventional visual question answering benchmarks. The attempt to incorporate visual simulation into retrospective reasoning is also potentially worth exploring.

Critical weaknesses

W1. Questionable baseline results and evaluation reliability. The reported performance of GPT-5.5 (63.36%) is substantially below GPT-4o (77.90%) and even Qwen3-VL-8B (65.40%), which is highly surprising for tasks involving relatively basic physical commonsense reasoning. The extremely poor temporal reversibility results are particularly difficult to reconcile with the capabilities of modern reasoning models. The paper provides insufficient qualitative evidence to substantiate these findings. Moreover, the class-wise recalls in Table S6 are inconsistent with some aggregate accuracies in Table 2, and the unusually low random baseline for ED4 (26.64%) raises additional questions about the evaluation protocol. These issues significantly undermine confidence in the main empirical claims.

W2. Missing strong reasoning baselines. The paper lacks comparisons against carefully prompted, reasoning-intensive VLMs or LLMs. Explicitly describing both visual states, reasoning through physical constraints, and iteratively verifying intermediate steps are natural and potentially effective alternatives. Without such baselines, the necessity of expensive video simulation is not established.

W3. Insufficient implementation details and reproducibility. Critical aspects of the training pipeline are underspecified, including supervision for unreachable Planner hypotheses, construction of Verifier training videos, and the adaptation of Wan2.2 to fixed first-and-last-frame conditioning. The main text and appendix also describe different simulator training objectives without sufficient clarification. These omissions make the central method difficult to reproduce or independently validate.

W4. Unclear physical grounding and limited practical utility. Endpoint-conditioned video generation does not guarantee physically valid transitions. The Verifier may exploit generation artifacts rather than genuine physical constraints, and the existing ablations do not adequately distinguish these possibilities. Furthermore, the reported inference cost of approximately 23.4 seconds on an H800 GPU raises serious concerns about practical deployment, especially when cheaper reasoning-based alternatives have not been adequately evaluated.

Questions for authors

Mostly same as the aforementioned weakness:

Q1. Can the authors provide representative GPT-5.5 failure cases for ED4, including the original image pairs, exact prompts, raw responses, parsed labels, and ground-truth annotations? How do they explain the substantial performance gap between GPT-5.5 and GPT-4o? Have the results been independently reproduced?

Q2. How does GPT-5.5 perform when explicitly prompted to describe both visual states, identify physical constraints, reason through intermediate steps, and iteratively verify its conclusions? Can the authors compare against such approaches under matched or reported inference budgets?

Q3. How do the authors reconcile the discrepancies between Table 2 and Table S6? Why does random choice achieve 26.64% on ED4? Is ED4 evaluated using ordinary binary accuracy or a different paired criterion?

Q4. How are Planner supervision targets constructed for unreachable examples? What videos are used to train the Verifier, and how are positive and negative examples generated? How exactly is Wan2.2 modified and fine-tuned for dual-endpoint conditioning? Will the authors release the full training-data construction and evaluation code?

Q5. What evidence demonstrates that the Verifier relies on physical transition consistency rather than artifacts of video generation? Have the authors tested temporally shuffled, mismatched, or otherwise corrupted simulated trajectories as controls?

Q6. The practical value of this work is questionable. What deployment scenarios justify approximately 23.4 seconds of inference on an H800 GPU for reachability classification? What advantages remain when compared with computationally cheaper reasoning-based alternatives?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.