- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on physical understanding of videos. Current VLMs may correctly understand a physical statement of an object, but cannot understand the full trajectory of object physical statement transition. To push current VLM physics understanding to fine-grained atomic physical statement, authors propose APT-Bench. Extensive zero shot evaluations including both commercial and open-source models show that current models suffer from this fine grained physics understanding. Authors further post-train open-source models and can make some improvement on this task.
Critical strengths
-
The motivation of this paper is clear, It focuses on evaluating fine-grained physical state transitions within video events, which are often overlooked by existing physical video understanding benchmarks.
-
The study is relatively comprehensive. This paper includes APT definition, Benchmark building, and zero shot evaluation on both commercial and public models, and post-training on open-source models.
Critical weaknesses
-
I am concerned about the claimed contribution of evaluating causal physical understanding. By defining fine-grained atomic physical state transitions and checking the correctness of model predictions, the current evaluation essentially measures transition type classification and temporal localization. It can be viewed as another temporal event localization task, just replacing atomic actions with atomic physical state transitions. The current APT-Bench evaluation does not explicitly assess causality. Although the authors acknowledge this limitation, I think this conceptual concern needs to be addressed during rebuttal: either reframe APT-Bench as a fine-grained, mechanism-typed temporal localization benchmark for physical state transitions, or add an explicit causality evaluation that checks the causal relationships between physical state transitions, as well as the structured representation of physical mechanisms, before/after states, and visual evidence, instead of the current evaluation, which only focuses on transition types and timestamps.
-
A natural follow-up question is how to ensure that the physical mechanisms covered by APT-Bench are visually identifiable. What if the same observed motion transition could be explained by multiple different physical mechanisms? Does APT-Bench suffer from such ambiguity? Does APT-Bench consistently measure visually identifiable physical transitions, or does it sometimes require information that is only accessible through simulator state traces?
Questions for authors
Please see the critical weaknesses section.
2 comments
Thanks for the response. Authors addressed my concerns and I maintain my positive rating.
Thank you for the two conceptual questions. Both are answered below, with new results for each. W1. What APT-Bench measures. We adopt your framing: APT-Bench is a fine-grained, mechanism-typed temporal localization benchmark for physical state transitions, the scope stated in Sec. 3.2. Wording changes:
Each type is one mechanism family plus one before/after state pair (Table 2), so a correct type and timestamp fix the mechanism, the state change and the time. A new entity-aware track on the simulator test split also requires the correct object or object pair: recall is 6.8% zero-shot, 23.1% for the submitted APT-Tune model and 31.0% with entity targets.
Difference from action localization.
W2. Visual identifiability.
inelastic_capture; one that decelerates while sliding issliding_arrest.New blind audit (additional annotation; benchmark labels unchanged). Two annotators who see only the video independently label the same 40 simulator test clips (744 trace-derived APTs). Under the benchmark's matching rule, they recover 89.2% of the trace-derived labels (90.1% and 88.3%), and 93.4% of their labels have a trace-derived counterpart. Median timing difference: 33 and 67 ms. Labels admitting a second plausible type: 4.1%.
Recall by label source (existing predictions of the Table 5 models). Human-labeled APTs (5,551): 8.0% → 38.2%. Simulator APTs (4,649): 12.4% → 38.0%. The tuned model, which sees only the video, recovers trace-derived labels at the same rate as human labels.
In the revision, we report the new blind audit in App. C.10 and the per-source results in Sec. 6.