- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies errors in Video-LLMs when observed events conflict with familiar expectations. It describes the rationalization of physical violations and the reporting of expected but absent events as Semantic Prior Dominance. The authors evaluate these errors through matched video pairs and propose PhyAR, which combines observation–attribution–verdict supervision with paired training. PhyAR improves paired accuracy on two backbones and transfers to independently constructed videos for event verification. On the physical reasoning benchmark IntPhys2, it modestly improves overall accuracy, with better detection of impossible events and lower accuracy on possible events.
Critical strengths
-
The proposed PhyAR achieves substantial gains across both physical-validity judgments and event verification on two backbones. The improvements on an external event-verification benchmark provide further evidence that the benefits extend beyond PriorPair.
-
The proposed PriorPair provides a useful way to test whether models distinguish observed events from familiar expectations. Its matched pairs require different judgments in similar scenes, making errors that would be hidden by high accuracy on ordinary events easier to identify.
Critical weaknesses
-
The SPD framing would become much more convincing if the benchmark could test it directly. At present, videos are labeled "prior-conflicting" by design and the prior itself is not measured. Prior conflict also coincides with the "no" answer and, in Anti-physics, with editing. Measuring prior strength and decoupling it from answer polarity and processing would let the results speak more clearly for SPD.
-
For PhyAR, it would benefit from showing that the gains come from better use of visual evidence. VARC asks the model to write observations before the verdict, but the order alone does not guarantee that the verdict depends on what was observed. PhyAR is also trained on data in which anomalies are frequent, so part of the improvement could come from expecting anomalies more often rather than seeing them better. Those checks would make the method's contribution much clearer.
Questions for authors
-
Since Anti-physics negatives are produced by editing or generation, a control set would help confirm that the benchmark rewards physical reasoning rather than sensitivity to processing.
-
SPD attributes failures to semantic priors, but prior strength is not measured. Estimating it per sample (e.g., having a text-only model predict the outcome from the scene description) and showing that errors on negatives increase with it would give direct support for SPD.
-
The joint-training result in Table 3b is interesting. It would help to discuss why single-stream training lowers performance on the other stream below the base model, since a fully shared mechanism might predict positive transfer. Any additional evidence of a common mechanism across the two streams would further support the unification claim.
-
Expert verification checks the full final videos, but models see 16 uniformly sampled frames. Was it checked that the decisive cue (contact, gap, cut, reversal) survives this sampling?
-
How do models respond when asked the physical-validity question on counterintuitive negatives, which are physically valid but unexpected? SPD and a default “valid” bias predict opposite answers here, so this result could provide clean support for the SPD interpretation.
7 comments
Rebuttal 2/5: Response to Q3
Q3: Cross-Stream Transfer and the Unifying Role of SPD
Fine-tuning on either stream improves performance on that task but lowers paired accuracy on the other. We interpret this pattern as task specialization. Relative to Base, training InternVL2.5 on Anti-physics alone reduces the number of correctly judged Counter-intuitive pairs from 15/41 to 12/41. Conversely, training on Counter-intuitive alone reduces the number of correct Anti-physics pairs from 7/127 to 2/127 (Table 3b). Both tasks require the model to follow visual evidence when it conflicts with a familiar expectation, but they ask different questions of that evidence. Anti-physics asks whether the observed sequence is physically plausible, even when a familiar explanation comes readily to mind. Counter-intuitive asks whether a particular event actually occurred, even when it could plausibly have happened. Training on only one stream reinforces one of these judgments without providing supervision for the other.
On the stream omitted from training, the model makes more errors consistent with the expected event. After Anti-physics-only training, InternVL2.5 still answers 40/41 Counter-intuitive positives correctly, but the number of negatives on which it reports that the target event occurred rises from 26/41 to 29/41. In the reverse direction, Counter-intuitive-only training increases the number of physically invalid Anti-physics videos judged plausible from 120/127 to 124/127.
Two cases that Base initially answered correctly illustrate how these errors arise after single-stream training.
- Anti-physics-only training, evaluated on Counter-intuitive. A hammer repeatedly strikes a pane of glass, but the glass remains intact. The model describes the glass shattering and fragments falling, then concludes that the target event occurred. It recognizes the blows but reports their expected consequence as something it saw.
- Counter-intuitive-only training, evaluated on Anti-physics. A cracker spontaneously turns into a miniature pizza. The model describes someone placing pepperoni on the cracker and judges the sequence physically plausible. It supplies a familiar cause for a transformation that the video shows happening on its own.
With joint training, PhyAR correctly describes both the intact glass and the cracker’s spontaneous transformation and gives the correct verdict in each case. The first error invents an expected outcome. The second invents a cause that would make the observed outcome plausible. In both cases, the model’s account departs from the visible facts in favor of a familiar sequence of events. This is the behavioral pattern that SPD unifies.
Two further results support this interpretation. In the text-only experiment reported in Q2, each backbone favored the positive event in over 90% of scenes in each stream, including Counter-intuitive scenes where both outcomes are physically plausible. Table 2b also helps explain why the two tasks benefit from different supervision. Among negative videos for which InternVL2.5 Base correctly described the decisive visual facts, it still gave the wrong verdict in 6/9 Anti-physics cases, compared with 1/15 Counter-intuitive cases. In the Counter-intuitive cases, an accurate description usually settled whether the target event occurred. In Anti-physics, the model could describe the facts correctly and still rationalize the violation.
Joint training improves physical-validity judgments while preserving nearly all of the single-stream gain in event verification. Compared with Anti-physics-only training, it raises the number of correct Anti-physics pairs from 33/127 to 52/127. It also yields 25/41 correct Counter-intuitive pairs, just one fewer than the 26/41 achieved with Counter-intuitive-only training. Together, these gains give PhyAR the highest overall PCA (45.83%).
Rebuttal 3/5: Responses to Q4 and Q5
Q4: Do the 16 sampled frames retain the decisive visual cues?
Yes. We randomly selected 100 videos and manually inspected the 16 uniformly sampled frames used as model input. The decisive visual cues remained visible in 98 of the 100 videos. PriorPair clips are trimmed around individual physical events and last about four seconds on average, so 16 frames correspond to roughly 4 fps for a typical clip. The decisive cue depends on the task. For example, for events involving contact, we checked whether the sampled frames showed the objects touching or a gap remaining between them. For causal reversals, we checked whether the sequence of sampled frames showed the effect occurring before its cause.
We also doubled the input from 16 to 32 frames on all 168 test pairs to assess whether denser sampling changed the main result for InternVL2.5. The base model’s overall PCA rose from 13.10% to 14.29%, a gain of only 1.19 percentage points. PhyAR achieved 45.83% with 16 frames and 43.45% with 32. Even at twice the sampling density, the base model still struggled to judge both videos in a pair correctly, while PhyAR retained a 29.17-percentage-point advantage.
Q5: How do models judge the physical validity of counter-intuitive negatives?
When asked directly about physical validity, both base models classify nearly all counter-intuitive negatives as physically valid. We posed the same physical-validity question for all 336 test videos. The table below reports results for all 41 Counter-intuitive pairs. The target event occurs in the positive video and is absent from its negative counterpart. Both videos obey physical laws, so “physically valid” is correct for both.
| Model | Positives judged physically valid | Negatives judged physically valid |
|---|---|---|
| InternVL2.5 Base | 39/41 (95.12%) | 39/41 (95.12%) |
| PhyAR (InternVL2.5) | 40/41 (97.56%) | 31/41 (75.61%) |
| VideoLLaMA3 Base | 41/41 (100%) | 41/41 (100%) |
| PhyAR (VideoLLaMA3) | 37/41 (90.24%) | 27/41 (65.85%) |
Both base models show a strong preference for the “physically valid” verdict. Each gives that verdict at the same rate on counter-intuitive positives and negatives. Even among the 127 Anti-physics videos containing genuine physical violations, InternVL2.5 and VideoLLaMA3 identify only 5 and 2 violations, respectively. Under the same prompt, their PhyAR counterparts identify 56 and 57 violations. PhyAR also makes more false judgments of invalidity on counter-intuitive negatives than on their equally valid positive counterparts. PhyAR (InternVL2.5) judges 10/41 negatives and 1/41 positives physically invalid. For PhyAR (VideoLLaMA3), the corresponding counts are 14/41 and 4/41. Thus, PhyAR detects more genuine violations but sometimes also treats an unexpected outcome as physically invalid.
SPD can lead a model to misreport what happened even when it answers the physical-validity question correctly. The lighter example in Appendix F illustrates why. The lighter is operated but produces no flame, yet the base model reports seeing one. Both ignition and failure to ignite are physically plausible. A model that accurately reports the absence of a flame and one that mistakenly reports the expected flame can therefore give the same “physically valid” verdict. SPD and a default preference for “valid” need not predict opposite answers to this question. The paired event question asks whether a flame actually appeared. A flame appears in only one video, so answering both questions correctly requires distinguishing the observed outcomes. The physical-validity question reveals the base models’ tendency to answer “valid,” while event verification directly exposes the counter-intuitive manifestation of SPD: even when the video clearly shows that the target event did not occur, the model describes the expected outcome as an observed fact.
Rebuttal 1/5: Responses to Q1 and Q2
Thank you for your thoughtful feedback. We respond to each point below.
Q1: Controls for Editing and Generation Artifacts
We conducted controls for editing and generation artifacts across all 127 Anti-physics test pairs. Both Base and PhyAR show little change in overall performance, and PhyAR’s advantage over Base is unchanged. These results support PriorPair as a test of physical reasoning rather than sensitivity to processing artifacts.
The controls cover all three construction routes:
We use the same evaluation settings as in the paper.
Across all 127 pairs, PhyAR retains its original advantages of 35.43 and 33.07 percentage points over Base on InternVL2.5 and VideoLLaMA3, respectively. The overall performance and the relative gains are preserved under the controls for editing and generation artifacts.
Q2: Measuring Prior Strength
We measured event priors for all 168 test pairs, independently using each backbone’s Base model. Without seeing the videos, InternVL2.5 and VideoLLaMA3 prefer the positive event in 163/168 and 164/168 cases, respectively. This also holds in the physically plausible Counter-intuitive stream: the positive event is preferred in 37/41 and 39/41 cases. These measurements show that the event expectations used to construct PriorPair are broadly reflected in the models’ own preferences.
For each pair, we provide the shared scene description without revealing the actual outcome and ask which of the positive and negative events would ordinarily be expected. Prior strength is the log-probability ratio favouring the positive event, averaged over both option orders.
PhyAR improves accuracy on negative videos even when the model strongly expects the positive event. The high-prior group comprises approximately the highest-scoring third of cases within each category, identified separately for each backbone. In this group, positive-event preference probabilities are at least 96.0% for InternVL2.5 and 92.4% for VideoLLaMA3.
We analysed the relationship between prior strength and the base models’ errors on negative videos across both backbones and streams. After controlling for category, we found no significant positive correlations (all p > 0.32). In Anti-physics, Base already misjudges 120/127 and 122/127 negatives, leaving little variation in errors. The score measures semantic expectations about events from text alone. Judgments about videos also depend on the strength of the visual evidence, the difficulty of understanding the event, and the model’s perceptual and physical reasoning abilities. These factors help explain why errors across videos need not increase monotonically with prior strength.