acceptodds
Under review as a conference paper at ICLR 2027

From Priors to Perception: Grounding Video-LLMs in Physical Reality

Abstract

Video Large Language Models (Video-LLMs) excel in general video understanding but often base physical judgments on event expectations rather than observations. We find that they not only rationalize physically impossible events, but also incorrectly report expected outcomes in physically plausible yet counter-intuitive scenarios, despite clear contradictory visual evidence. We provide the first unified account of these failures as Semantic Prior Dominance (SPD): semantic expectations override conflicting visual evidence. To diagnose and overcome these failures, we introduce PriorPair, a high-fidelity paired video benchmark covering both conflicts across eight physical categories. Its category-directed construction pairs prior-aligned events with semantically matched but prior-conflicting counterparts. We further propose Physics-Anchored Reasoner (PhyAR), a lightweight training framework. Its Visually Anchored Reasoning Chain grounds physical attribution and judgment in explicit visual observations, while Paired Data Binding strengthens learning of decisive event differences through matched-pair supervision. Standard LoRA fine-tuning with PhyAR substantially improves physical reasoning and reduces prior-driven judgment bias across different Video-LLM backbones without architectural modifications. These gains extend to an external physical benchmark while largely preserving general video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Large Language Models, Physical Reasoning, Semantic Prior Dominance, Paired Video Benchmark, Visually Grounded Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DafBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies errors in Video-LLMs when observed events conflict with familiar expectations. It describes the rationalization of physical violations and the reporting of expected but absent events as Semantic Prior Dominance. The authors evaluate these errors through matched video pairs and propose PhyAR, which combines observation–attribution–verdict supervision with paired training. PhyAR improves paired accuracy on two backbones and transfers to independently constructed videos for event verification. On the physical reasoning benchmark IntPhys2, it modestly improves overall accuracy, with better detection of impossible events and lower accuracy on possible events.

Critical strengths
  • The proposed PhyAR achieves substantial gains across both physical-validity judgments and event verification on two backbones. The improvements on an external event-verification benchmark provide further evidence that the benefits extend beyond PriorPair.

  • The proposed PriorPair provides a useful way to test whether models distinguish observed events from familiar expectations. Its matched pairs require different judgments in similar scenes, making errors that would be hidden by high accuracy on ordinary events easier to identify.

Critical weaknesses
  • The SPD framing would become much more convincing if the benchmark could test it directly. At present, videos are labeled "prior-conflicting" by design and the prior itself is not measured. Prior conflict also coincides with the "no" answer and, in Anti-physics, with editing. Measuring prior strength and decoupling it from answer polarity and processing would let the results speak more clearly for SPD.

  • For PhyAR, it would benefit from showing that the gains come from better use of visual evidence. VARC asks the model to write observations before the verdict, but the order alone does not guarantee that the verdict depends on what was observed. PhyAR is also trained on data in which anomalies are frequent, so part of the improvement could come from expecting anomalies more often rather than seeing them better. Those checks would make the method's contribution much clearer.

Questions for authors
  • Since Anti-physics negatives are produced by editing or generation, a control set would help confirm that the benchmark rewards physical reasoning rather than sensitivity to processing.

  • SPD attributes failures to semantic priors, but prior strength is not measured. Estimating it per sample (e.g., having a text-only model predict the outcome from the scene description) and showing that errors on negatives increase with it would give direct support for SPD.

  • The joint-training result in Table 3b is interesting. It would help to discuss why single-stream training lowers performance on the other stream below the base model, since a fully shared mechanism might predict positive transfer. Any additional evidence of a common mechanism across the two streams would further support the unification claim.

  • Expert verification checks the full final videos, but models see 16 uniformly sampled frames. Was it checked that the decisive cue (contact, gap, cut, reversal) survives this sampling?

  • How do models respond when asked the physical-validity question on counterintuitive negatives, which are physically valid but unexpected? SPD and a default “valid” bias predict opposite answers here, so this result could provide clean support for the SPD interpretation.

7 comments
AuthorsOpenReview ↗

Rebuttal 1/5: Responses to Q1 and Q2

Thank you for your thoughtful feedback. We respond to each point below.

Q1: Controls for Editing and Generation Artifacts

We conducted controls for editing and generation artifacts across all 127 Anti-physics test pairs. Both Base and PhyAR show little change in overall performance, and PhyAR’s advantage over Base is unchanged. These results support PriorPair as a test of physical reasoning rather than sensitivity to processing artifacts.

The controls cover all three construction routes:

  • Temporal editing, 63 pairs: Positive and negative inputs contain exactly the same 16 preprocessed frames. The positive retains the original order, while the negative uses segment shuffling, causal-stage exchange, or reversal.
  • Local editing/compositing, 36 pairs: We apply the corresponding editing pipeline to non-critical regions of the positive video, preserving its original physical interaction, and pair the edited positive with the original negative.
  • Generation, 28 pairs: We regenerate the positive using the same generator and scene conditions as for the negative, with a prompt that explicitly describes the positive’s physically valid event. Both videos are generated, but the positive depicts a physically valid process and the negative a physical violation.

We use the same evaluation settings as in the paper.

ControlBackboneBase PCA (%): Original → ControlledPhyAR PCA (%): Original → Controlled
Temporal editingInternVL2.51.59 → 1.5952.38 → 50.79
Temporal editingVideoLLaMA30.00 → 0.0044.44 → 47.62
Local editing/compositingInternVL2.55.56 → 8.3327.78 → 30.56
Local editing/compositingVideoLLaMA30.00 → 0.0027.78 → 22.22
GenerationInternVL2.514.29 → 10.7132.14 → 32.14
GenerationVideoLLaMA317.86 → 21.4332.14 → 35.71
All 127 pairsInternVL2.55.51 → 5.5140.94 → 40.94
All 127 pairsVideoLLaMA33.94 → 4.7237.01 → 37.80

Across all 127 pairs, PhyAR retains its original advantages of 35.43 and 33.07 percentage points over Base on InternVL2.5 and VideoLLaMA3, respectively. The overall performance and the relative gains are preserved under the controls for editing and generation artifacts.

Q2: Measuring Prior Strength

We measured event priors for all 168 test pairs, independently using each backbone’s Base model. Without seeing the videos, InternVL2.5 and VideoLLaMA3 prefer the positive event in 163/168 and 164/168 cases, respectively. This also holds in the physically plausible Counter-intuitive stream: the positive event is preferred in 37/41 and 39/41 cases. These measurements show that the event expectations used to construct PriorPair are broadly reflected in the models’ own preferences.

For each pair, we provide the shared scene description without revealing the actual outcome and ask which of the positive and negative events would ordinarily be expected. Prior strength is the log-probability ratio favouring the positive event, averaged over both option orders.

PhyAR improves accuracy on negative videos even when the model strongly expects the positive event. The high-prior group comprises approximately the highest-scoring third of cases within each category, identified separately for each backbone. In this group, positive-event preference probabilities are at least 96.0% for InternVL2.5 and 92.4% for VideoLLaMA3.

High-prior groupInternVL2.5 negative accuracy (%): Base → PhyARVideoLLaMA3 negative accuracy (%): Base → PhyAR
Anti-physics9.30 → 46.512.44 → 41.46
Counter-intuitive30.77 → 69.2346.15 → 69.23

We analysed the relationship between prior strength and the base models’ errors on negative videos across both backbones and streams. After controlling for category, we found no significant positive correlations (all p > 0.32). In Anti-physics, Base already misjudges 120/127 and 122/127 negatives, leaving little variation in errors. The score measures semantic expectations about events from text alone. Judgments about videos also depend on the strength of the visual evidence, the difficulty of understanding the event, and the model’s perceptual and physical reasoning abilities. These factors help explain why errors across videos need not increase monotonically with prior strength.

AuthorsOpenReview ↗

Rebuttal 2/5: Response to Q3

Q3: Cross-Stream Transfer and the Unifying Role of SPD

Fine-tuning on either stream improves performance on that task but lowers paired accuracy on the other. We interpret this pattern as task specialization. Relative to Base, training InternVL2.5 on Anti-physics alone reduces the number of correctly judged Counter-intuitive pairs from 15/41 to 12/41. Conversely, training on Counter-intuitive alone reduces the number of correct Anti-physics pairs from 7/127 to 2/127 (Table 3b). Both tasks require the model to follow visual evidence when it conflicts with a familiar expectation, but they ask different questions of that evidence. Anti-physics asks whether the observed sequence is physically plausible, even when a familiar explanation comes readily to mind. Counter-intuitive asks whether a particular event actually occurred, even when it could plausibly have happened. Training on only one stream reinforces one of these judgments without providing supervision for the other.

On the stream omitted from training, the model makes more errors consistent with the expected event. After Anti-physics-only training, InternVL2.5 still answers 40/41 Counter-intuitive positives correctly, but the number of negatives on which it reports that the target event occurred rises from 26/41 to 29/41. In the reverse direction, Counter-intuitive-only training increases the number of physically invalid Anti-physics videos judged plausible from 120/127 to 124/127.

Two cases that Base initially answered correctly illustrate how these errors arise after single-stream training.

  • Anti-physics-only training, evaluated on Counter-intuitive. A hammer repeatedly strikes a pane of glass, but the glass remains intact. The model describes the glass shattering and fragments falling, then concludes that the target event occurred. It recognizes the blows but reports their expected consequence as something it saw.
  • Counter-intuitive-only training, evaluated on Anti-physics. A cracker spontaneously turns into a miniature pizza. The model describes someone placing pepperoni on the cracker and judges the sequence physically plausible. It supplies a familiar cause for a transformation that the video shows happening on its own.

With joint training, PhyAR correctly describes both the intact glass and the cracker’s spontaneous transformation and gives the correct verdict in each case. The first error invents an expected outcome. The second invents a cause that would make the observed outcome plausible. In both cases, the model’s account departs from the visible facts in favor of a familiar sequence of events. This is the behavioral pattern that SPD unifies.

Two further results support this interpretation. In the text-only experiment reported in Q2, each backbone favored the positive event in over 90% of scenes in each stream, including Counter-intuitive scenes where both outcomes are physically plausible. Table 2b also helps explain why the two tasks benefit from different supervision. Among negative videos for which InternVL2.5 Base correctly described the decisive visual facts, it still gave the wrong verdict in 6/9 Anti-physics cases, compared with 1/15 Counter-intuitive cases. In the Counter-intuitive cases, an accurate description usually settled whether the target event occurred. In Anti-physics, the model could describe the facts correctly and still rationalize the violation.

Joint training improves physical-validity judgments while preserving nearly all of the single-stream gain in event verification. Compared with Anti-physics-only training, it raises the number of correct Anti-physics pairs from 33/127 to 52/127. It also yields 25/41 correct Counter-intuitive pairs, just one fewer than the 26/41 achieved with Counter-intuitive-only training. Together, these gains give PhyAR the highest overall PCA (45.83%).

AuthorsOpenReview ↗

Rebuttal 3/5: Responses to Q4 and Q5

Q4: Do the 16 sampled frames retain the decisive visual cues?

Yes. We randomly selected 100 videos and manually inspected the 16 uniformly sampled frames used as model input. The decisive visual cues remained visible in 98 of the 100 videos. PriorPair clips are trimmed around individual physical events and last about four seconds on average, so 16 frames correspond to roughly 4 fps for a typical clip. The decisive cue depends on the task. For example, for events involving contact, we checked whether the sampled frames showed the objects touching or a gap remaining between them. For causal reversals, we checked whether the sequence of sampled frames showed the effect occurring before its cause.

We also doubled the input from 16 to 32 frames on all 168 test pairs to assess whether denser sampling changed the main result for InternVL2.5. The base model’s overall PCA rose from 13.10% to 14.29%, a gain of only 1.19 percentage points. PhyAR achieved 45.83% with 16 frames and 43.45% with 32. Even at twice the sampling density, the base model still struggled to judge both videos in a pair correctly, while PhyAR retained a 29.17-percentage-point advantage.

Q5: How do models judge the physical validity of counter-intuitive negatives?

When asked directly about physical validity, both base models classify nearly all counter-intuitive negatives as physically valid. We posed the same physical-validity question for all 336 test videos. The table below reports results for all 41 Counter-intuitive pairs. The target event occurs in the positive video and is absent from its negative counterpart. Both videos obey physical laws, so “physically valid” is correct for both.

ModelPositives judged physically validNegatives judged physically valid
InternVL2.5 Base39/41 (95.12%)39/41 (95.12%)
PhyAR (InternVL2.5)40/41 (97.56%)31/41 (75.61%)
VideoLLaMA3 Base41/41 (100%)41/41 (100%)
PhyAR (VideoLLaMA3)37/41 (90.24%)27/41 (65.85%)

Both base models show a strong preference for the “physically valid” verdict. Each gives that verdict at the same rate on counter-intuitive positives and negatives. Even among the 127 Anti-physics videos containing genuine physical violations, InternVL2.5 and VideoLLaMA3 identify only 5 and 2 violations, respectively. Under the same prompt, their PhyAR counterparts identify 56 and 57 violations. PhyAR also makes more false judgments of invalidity on counter-intuitive negatives than on their equally valid positive counterparts. PhyAR (InternVL2.5) judges 10/41 negatives and 1/41 positives physically invalid. For PhyAR (VideoLLaMA3), the corresponding counts are 14/41 and 4/41. Thus, PhyAR detects more genuine violations but sometimes also treats an unexpected outcome as physically invalid.

SPD can lead a model to misreport what happened even when it answers the physical-validity question correctly. The lighter example in Appendix F illustrates why. The lighter is operated but produces no flame, yet the base model reports seeing one. Both ignition and failure to ignite are physically plausible. A model that accurately reports the absence of a flame and one that mistakenly reports the expected flame can therefore give the same “physically valid” verdict. SPD and a default preference for “valid” need not predict opposite answers to this question. The paired event question asks whether a flame actually appeared. A flame appears in only one video, so answering both questions correctly requires distinguishing the observed outcomes. The physical-validity question reveals the base models’ tendency to answer “valid,” while event verification directly exposes the counter-intuitive manifestation of SPD: even when the video clearly shows that the target event did not occur, the model describes the expected outcome as an observed fact.

Reviewer_dLMsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on one kind of reasoning failures of Video-LLMs: When making predictions, Video-LLMs tend to rely heavily on their learned “Common Sense”, so that overlooking the actual evidence in the video. The authors call this phenomenon "Semantic Prior Dominance". They introduce a new benchmark PriorPair to evaluate this problem with paired videos. They further proposed PhyAR, a training framework to enhance the model’s focus on visual evidence.

Critical strengths

This paper addresses a valuable research question in Video-LLMs. The authors construct semantically similar video pairs to evaluate “physically plausible but counter-intuitive events”, which offers a useful way to better understanding physical reasoning of Video-LLMs. The 828 carefully constructed video pairs are valuable to this field of research.

Critical weaknesses
  1. Previous works—such as VideoHallu, NOAH, QuantiPhy, and CDH-Bench—have already studied the phenomenon where visual evidence conflicts with common sense. VideoHallu has explored methods to mitigate a model's over-reliance on priors through training, and CDH-Bench has proposed the concept of using paired images to evaluate models. Although these works do not address exactly the same settings, they raise concern that this paper presents only a minor incremental contribution over them. I suggest that the authors include results for PhyAR on other benchmarks designed for this problem, as well as a comparison between PhyAR and other training methods (such as VideoHallu) on the same benchmark. This would help clarify the specific contributions of this paper.

  2. PriorPair contains eight categories, six of which concern anti-physics violations, while only two—Near-Miss and Consequence Arrest—cover counter-intuitive events that remain physically plausible. The latter cases are particularly important for the paper's claim that models may prioritize expected outcomes over actual visual evidence, even when no physical law is violated. However, the main experiments in Tables 2, 3(a), and 4 report aggregate metrics across these different categories. Such aggregation makes it difficult to determine the specific improvement of PhyAR. Separate results for the two streams, together with a more detailed analysis and more sub-categories of the counter-intuitive categories, would provide stronger evidence for the paper's central claim.

  3. PriorPair contains 828 video pairs, with approximately 660 pairs used for training and 168 for testing. Both splits are drawn from the same dataset and construction pipeline. Consequently, the reported improvements do not rule out the possibility that the model learns dataset-specific patterns, category cues, or recurring characteristics of the constructed videos.

  4. The paper reports improvements on IntPhys2 and largely preserved performance on Video-MME. I think these results is not sufficiently enough to support the external generalization capability of PhyAR. These datasets evaluate a model's video understanding and physical reasoning capabilities; the authors should also conduct evaluations using other relevant benchmarks focused on counter-intuitive scenarios.

  5. In the ablation study, training with a combination of randomly binding videos and VARC yields significantly better results than using either VARC or paired videos alone. This raises the question: does the model’s improvement stem merely from fitting this specific data distribution, rather than from genuinely learning to leverage the distinct visual evidence found in positive and negative sample pairs?

Questions for authors
  1. What is the main advantage of PriorPair over VideoHallu and other existing physical-reasoning benchmarks? Could you provide a direct comparison of their task coverage and explain which important failure cases can only be evaluated by PriorPair?

  2. Is it possible to evaluate PhyAR on other relevant benchmarks in this research question? If it cannot be evaluated on other benchmarks, could you please provide an explanation?

  3. Could you provide further explanation as to why the combination of VARC and random binding yields such strong results? I am concerned that this might actually indicate the model has learned the data distribution rather than the intended capability. Or that could you include a comparative experiment using "no-VARC + random binding"?

  4. Could you provide more evidence that supports the external generalization capability of PhyAR?

  5. Could you provide a more detailed analysis of the two counterintuitive categories within PriorPair? In my view, these two categories are the ones truly relevant to the research question of this paper, whereas the other six categories actually obscure the issue.

10 comments
AuthorsOpenReview ↗

Rebuttal 1/3: Response to Q1

Thank you for your thoughtful feedback. We respond to each point below.

Q1: PriorPair’s Advantages over Existing Benchmarks

PriorPair’s main advantage is its matched-pair design. The two videos in each pair evoke the same semantic expectation but require opposite answers to the same question. In the Anti-physics stream, we test whether models rationalize a visible physical violation. In the Counter-intuitive stream, both videos are physically plausible, and we test whether models report an expected target event even when it does not occur. Across both streams, we construct matched video pairs around predefined physical categories and provide complete Observation–Attribution–Verdict annotations for each video.

Related workResearch focus and designPriorPair’s distinctions and advantages
VideoHalluExpert-designed QA about alignment, physics, consistency, and commonsense abnormalities in generated videos. PaperConstructs matched videos of normal and conflicting physical interactions that require different answers to the same question, and explicitly includes physically plausible cases in which expected events do not occur.
NOAHControls narrative context through event insertion in multi-event videos to study hallucination and omission. PaperTargets physical facts within an interaction: whether trajectories establish contact and whether actions produce their expected consequences.
QuantiPhyEvaluates numerical kinematic inference and probes dependence on videos and supplied physical quantities. PaperCovers conflicts involving causal order, object persistence, physical constraints, contact, and consequences by changing the decisive physical facts in the video.
CDH-BenchUses matched commonsense/counterfactual images and identical questions to evaluate quantity, relation, and attribute conflicts. PaperCovers physical processes unfolding over time, including causal precedence, continuity, contact establishment, and consequence realization, with O–A–V annotations supporting analysis and training.

Among the compared benchmarks, only PriorPair provides systematic evaluation of the following important physical failure cases through explicitly defined categories and matched video controls:

  • Causal-stage inversion — Causal Reversal. An effect appears before its physical cause: a vase shatters before falling, or liquid accumulates before pouring begins. These cases expose models that replace the observed sequence with the familiar causal order.
  • Environmental–response incompatibility — Constraint Violation. An ordinary wood fire continues burning underwater without an additional oxygen supply, or a candle exposed to strong airflow retains an unchanged flame. These cases require judging whether the observed response is compatible with the environmental conditions.
  • Disproportionate interaction responses — Constraint Violation. A small, low-energy impact produces an enormous splash, or a massive object enters water without the disturbance its impact should produce. Recognizing the interaction is insufficient; its response must also be consistent with the physical conditions.
  • Failure to establish contact — Near-Miss. In Trajectory Miss, an object passes beside its target; in Abrupt Halt, it stops before contact. The model fails when it reports the expected interaction despite a visible gap or non-intersecting trajectory.
  • Failure to produce a consequence after an action — Consequence Arrest. In Insufficient Impact, contact occurs without the expected displacement or fracture; in Threshold Failure, an action does not reach the conditions needed for the outcome, such as tilting a cup without spilling. These cases expose models that treat an initiating action as sufficient evidence that its expected consequence occurred.

These matched contrasts directly support PhyAR’s structured, paired training. O–A–V annotations link visual facts, physical interpretations and final judgments; categories and construction targets are defined in Appendices A.1–A.2.

AuthorsOpenReview ↗

Rebuttal 2/3: Responses to Q2, Q3, and Q4

Q2 & Q4: External Generalization

PhyAR transfers zero-shot to NOAH Narrative QA, improving paired accuracy by 8.99 percentage points on InternVL2.5 and 4.58 points on VideoLLaMA3—relative gains of 30.4% and 20.0%, respectively. Both backbones improve on all three reported metrics.

NOAH is directly relevant to the prior–evidence conflict studied here. It tests whether models distinguish factual events from human-authored, narratively plausible but unshown events in independently constructed multi-event composite videos.

We evaluate the complete released Narrative QA set: 306 videos and 1,224 questions per model. Within each backbone, Base and PhyAR receive identical released questions and visual inputs and answer directly with Yes/No. Following the released grouping, we score 612 question pairs across the two question forms. A pair counts as correct only if the model answers both questions correctly, one about a factual event and one about a fabricated event.

ModelNarrative paired accuracy (%)Single-question accuracy (%)Videos with all four answers correct (%)
Base (InternVL2.5-8B)29.5863.895.23
PhyAR (InternVL2.5-8B)38.5668.5515.69
Base (VideoLLaMA3-7B)22.8861.0310.13
PhyAR (VideoLLaMA3-7B)27.4563.7313.40

Both backbones improve under the stricter requirement of four correct answers per video. The count triples from 16 to 48 on InternVL2.5 and rises from 31 to 41 on VideoLLaMA3.

PhyAR better distinguishes narratively plausible but unshown events from events actually depicted. Its substantial gains on NOAH provide direct evidence of transfer to an external prior-conflict task.

Q3: Random Binding and VARC

VARC + random binding benefits from structured supervision and same-category positive–negative co-training. The new no-VARC + random binding control isolates VARC’s contribution under random binding.

All six settings use InternVL2.5-8B, the same training data and category-balanced sampling. Without VARC, training answers contain only the final verdict, omitting observation and physical attribution; evaluation still uses the full O–A–V prompt on the same 168 video pairs.

Training supervisionBindingAnti-physics (127 pairs)Counter-intuitive (41 pairs)Overall (168 pairs)
Verdict onlyNone20.4756.1029.17
Verdict onlySame-category random (new)25.2060.9833.93
Verdict onlyMatched24.4165.8534.52
VARCNone29.9256.1036.31
VARCSame-category random37.0160.9842.86
VARCMatched (full PhyAR)40.9460.9845.83

Values are PCA (%).

Same-category positive–negative co-training improves performance even without true counterpart correspondence, explaining why random binding remains effective. Each random training unit still contains one positive and one negative video from the same physical category, jointly trained within the same parameter update. In settings with VARC, each video’s visual observations and physical attribution also receive supervision. Randomization disrupts true counterpart correspondence while retaining these training signals. Relative to no binding, PCA increases 29.17→33.93 without VARC and 36.31→42.86 with VARC.

With random binding held fixed, adding VARC raises overall PCA from 33.93% to 42.86%, a gain of 8.93 percentage points.

With the same VARC supervision, restoring true counterpart matching raises PCA from 42.86% to 45.83%, an additional 2.98 points and the highest overall PCA across all six settings. Matched counterparts highlight decisive physical differences within similar scenes and provide more compatible local supervision: mean gradient cosine similarity is 0.122, versus 0.010 for same-category random pairs (Appendix E).

PhyAR’s gains transfer beyond PriorPair’s construction pipeline. Both backbones improve on independently constructed NOAH Narrative QA (see Q2/Q4 above).

PhyAR’s gains persist when the same event is queried in opposite ways. We additionally tested Counter-intuitive videos with both “Is the target event present in the video?” and “Is the target event absent in the video?”, requiring all four answers per video pair to be correct. This rate increases 34.15%→43.90% on InternVL2.5 and 0%→12.20% on VideoLLaMA3.

AuthorsOpenReview ↗

Rebuttal 3/3: Responses to Q5 and Weakness 1

Q5: Counter-intuitive Categories

On the physically plausible Counter-intuitive stream, PhyAR raises PCA by 24.39 percentage points on InternVL2.5 and 12.20 points on VideoLLaMA3. Both backbones also improve on each of the two Counter-intuitive categories: Near-Miss and Consequence Arrest. PCA (%) requires correct verdicts for both videos in a pair.

Evaluation groupInternVL2.5-8B: Base → PhyARVideoLLaMA3-7B: Base → PhyAR
Anti-physics5.51 → 40.943.94 → 37.01
Counter-intuitive36.59 → 60.9843.90 → 56.10
Near-Miss18.18 → 45.4522.73 → 36.36
Consequence Arrest57.89 → 78.9568.42 → 78.95

Near-Miss and Consequence Arrest distinguish failure to establish an interaction from failure to produce a consequence after an action, covering four explicitly defined scenario types. Both counterparts remain physically valid, and the task is to determine whether the specified target event actually occurred. The four scenario types and their decisive visual evidence are summarized below (Appendix A.2):

CategoryScenario typeDecisive evidence
Near-MissTrajectory MissWhether the trajectory intersects the target contact region or opening
Near-MissAbrupt HaltWhether the approaching object makes contact or stops with a visible gap
Consequence ArrestInsufficient ImpactWhether contact produces the specified displacement, breakage, or other consequence
Consequence ArrestThreshold FailureWhether an executed action reaches the conditions required for the target outcome

In the Counter-intuitive negative examples, PhyAR corrects Base’s false reports of expected events. In the Near-Miss example, the hand stops before the cabinet handle, leaving a visible gap. Base nevertheless reports grasping; PhyAR identifies the gap and correctly judges that the target event did not occur. In the Consequence Arrest example, both videos show a lighter being operated, but only the positive counterpart produces a flame. Base invents a flame in the negative video; PhyAR correctly distinguishes successful ignition from operation without a flame and judges both counterparts correctly (Appendix F).

The six Anti-physics categories are an essential part of the same research question. They test forced rationalization of visible physical contradictions, such as inventing a collision to explain dominoes falling without ball contact; the Counter-intuitive stream tests whether models report expected events that are absent from physically plausible scenes. These are the two behavioral manifestations of SPD; separate stream results identify their respective gains, while studying them together covers both forms of prior–evidence conflict.

Weakness 1: Training-Method Comparison

We compare PhyAR with the GRPO training approach used in VideoHallu because it targets hallucinations on synthetic videos where physical or commonsense expectations conflict with visible abnormalities. VideoHallu, §3.2 We adapt this approach to InternVL2.5 using the same PriorPair training set and complete O–A–V reference annotations. GRPO generates multiple candidate responses per video-question and updates the model using their group-relative rewards; our adaptation uses ROUGE-based content rewards and the upstream reasoning-format reward.

PhyAR achieves substantially higher accuracy with approximately one-eleventh of the measured training cost. Both methods use four RTX A6000 GPUs:

MethodAnti-physics PCA (%)Counter-intuitive PCA (%)Overall PCA (%)GPU hours
GRPO (VideoHallu)2.3641.4611.90135.95
PhyAR40.9460.9845.8312.27

The GRPO adaptation leaves the central judgment bias unresolved: positive accuracy reaches 100%, while negative accuracy remains 11.90%. The adaptation scores each candidate against the O–A–V reference using ROUGE and separately rewards compliance with the reasoning format. Training uses neither token-level supervision on the reference answers nor matched-pair binding.

VARC supervises each token in the full Observation–Attribution–Verdict sequence, while matched binding ensures that both videos in a pair contribute to the same parameter update. The verdict-only SFT baseline without binding achieves 29.17% PCA. Adding VARC alone yields 36.31%, matched binding alone 34.52%, and both together 45.83%.

This targeted supervision makes standard SFT both effective and efficient. PhyAR requires no additional reward design, rollout-based optimization, or architectural modification. Its gains hold across both backbones and extend to zero-shot NOAH evaluation.

We hope these additional results resolve your main concerns. Please let us know if there is anything else we can clarify.

Reviewer_btqSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether Video-LLMs rely on visual evidence when observations conflict with common event expectations. The authors characterize two failure modes—rationalizing physically impossible events and reporting expected but absent events—as Semantic Prior Dominance (SPD). To evaluate these behaviors, they introduce PriorPair, a benchmark of 828 matched video pairs spanning eight physical categories. It contains an Anti-physics stream, which contrasts physically valid and invalid events, and a Counter-intuitive stream, which contrasts whether an expected target event occurs in otherwise plausible videos. The paper further proposes Physics-Anchored Reasoner (PhyAR), a LoRA-based fine-tuning framework combining structured Observation–Attribution–Verdict supervision with matched-pair batching. Experiments on VideoLLaMA3-7B and InternVL2.5-8B show substantial improvements in pair-wise consistency and negative-sample accuracy on PriorPair, while largely preserving performance on Video-MME. The authors also report reduced class imbalance and modest transfer to the external IntPhys2 benchmark.

Critical strengths
  1. The investigated problem is valuable: whether VLM really understands the visual content or just memorizes the prior knowledge.
  2. The design that assesses performance by evaluating a paired inverted question is solid and persuasive, and the proposed metrics can well evaluate model's ability.
  3. The proposed training methods improve performance by a large margin.
Critical weaknesses
  1. The frame sampling is 16 or 8. For temporal events that require dense perception, this sampling rate could cause information missing.
  2. The results in table 2 requires more discussion. It looks like the models are acapable of answering the positive questions in the first place, but after pyyAR, the pos acc drops. What is the reason behind this phenomenon? Is it because the models are guessing initially? Or phyAR damage the models' ability in positive questions when improving negative questions? Why phyAR cannot improve both? In the paper, the authors only mention "maintaining high positive accuracy", but no discussion about the 13.10pp drop on videollama3 and 6.55pp drop on internvl2.5.
  3. In table2 b, 66.67% is just 6/9. It can be easily noticed that there are 127 anti-physics negatives, where only 9 O are correct, and there are 6 out 9 V are incorrect. So there are still 114/127 where both O and V are wrong.
Questions for authors

As mentioned in the weaknesses, my main question is how to explain the degradation on positive questions after phyAR is applied.

4 comments
AuthorsOpenReview ↗

Rebuttal 1/2: Responses to Weaknesses 1 and 2

Thank you for your thoughtful feedback. We respond to each point below.

Weakness 1: Frame Sampling and Temporal Evidence

Both PhyAR backbones use 16 frames per video; the eight-frame setting applies only to Video-LLaVA-7B. PriorPair videos are manually trimmed around individual physical events and average approximately 4 seconds, so 16 frames correspond to roughly 4 fps and generally capture the event’s main temporal information. To further check whether sampling preserves the decisive temporal cues, we manually inspected the 16-frame inputs for 100 randomly sampled videos and confirmed that those cues were retained in 98 cases.

We further doubled the frame count to examine the effect of denser temporal sampling. We evaluated InternVL2.5-8B Base and PhyAR on the complete test set with 16 and 32 frames, keeping each model’s weights, task prompts, and preprocessing unchanged.

ModelFramesAnti-physics PCA (%)Counter-intuitive PCA (%)Overall PCA (%)
Base165.5136.5913.10
Base327.0936.5914.29
PhyAR1640.9460.9845.83
PhyAR3236.2265.8543.45

Doubling the frame count improves Base’s overall PCA by only 1.19 percentage points, while PhyAR retains a 29.17-point advantage at 32 frames. PhyAR’s Counter-intuitive PCA also increases from 60.98% to 65.85%. Thus, doubling the frame count does not resolve Base’s judgment difficulty, while PhyAR maintains a substantial advantage under denser temporal sampling.

Weakness 2: Positive-Accuracy Decrease

An important reason for the drop in positive accuracy is that PhyAR reduces the base models’ strong preference for positive answers. This substantially improves paired judgments, but it also reduces the advantage that this bias gives the models on positive videos. The decrease therefore does not, by itself, imply weaker recognition of positive events.

Because the base models favor positive answers, they often judge a positive video correctly yet give the same answer to its negative counterpart, overlooking the visual difference that calls for a different verdict. Indeed, InternVL2.5 and VideoLLaMA3 give positive verdicts to both videos in 144/168 and 141/168 pairs, respectively. In an additional text-only measurement, we give each backbone only the shared scene description and two event options, ask which event it would ordinarily expect, and swap the option order. Without seeing the videos, the two backbones already prefer the positive event in 163/168 and 164/168 scenes. Base can therefore score highly on positives by affirming the expected event, even when it fails to distinguish the two counterparts.

The InternVL2.5 and VideoLLaMA3 Base models already classify 166 and 164 of the 168 positive videos correctly, respectively, leaving little room for further gains in positive accuracy. PhyAR supervises the decisive differences between counterparts, teaching the model to reason from actual video evidence. However, the underlying model remains limited in visual recognition and reasoning. It can still misidentify the relevant facts and incorrectly reject some positive videos. This explains how learning to distinguish counterparts more accurately can coexist with a decrease in positive accuracy.

PCA tests whether models can use decisive visual evidence to distinguish semantically matched videos that require opposite answers to the same question. A pair counts as correct only when both judgments are correct.

BackbonePositive accuracy (%): Base → PhyARNegative accuracy (%): Base → PhyARPCA (%): Base → PhyAR
InternVL2.5-8B98.81 → 92.2613.10 → 50.6013.10 → 45.83
VideoLLaMA3-7B97.62 → 84.5214.88 → 51.1913.69 → 41.67

Across the 168 test pairs, the number with both verdicts correct increases from 22 to 77 on InternVL2.5 and from 23 to 70 on VideoLLaMA3.

AuthorsOpenReview ↗

Rebuttal 2/2: Response to Weakness 3

Weakness 3: Observation–Verdict Diagnosis

The 114/127 cases with both Observation and Verdict wrong show that most Anti-physics failures involve misreporting the visual evidence itself. The 66.67% (6/9) conditional error identifies a second failure: correctly reported facts followed by an incorrect judgment.

An Observation is correct when it accurately states the decisive visual facts (Appendix D). The complete counts for InternVL2.5-8B on the 127 Anti-physics negatives are shown below, where O denotes Observation and V denotes Verdict.

ModelO correct, V correctO correct, V wrongO wrong, V correctO wrong, V wrong
Base364114
PhyAR3452266

PhyAR improves both factual reporting and the reliability of judgments when the visual facts are stated correctly. Correct Observations increase from 9 to 39, and correct Verdicts from 7 to 56. Among responses with correct Observations, the proportion of correct Verdicts rises from 3/9 (33.33%) to 34/39 (87.18%), corresponding to conditional error rates of 6/9 and 5/39. Cases with both Observation and Verdict wrong decrease from 114 to 66. VARC supervises the full O–A–V sequence to jointly improve factual reporting and physical judgments based on those facts.

We hope these additional results resolve your main concerns. Please let us know if there is anything else we can clarify.

Dear Reviewer btqS,

We wanted to follow up on your main question about the drop in positive accuracy. In our response, we showed how often the base models gave a positive verdict to both videos in a pair and discussed how reducing this bias helps explain why positive accuracy falls even as more pairs are judged correctly. Even a brief note on whether this addresses your question would be very helpful to us.

Thank you for your time.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer btqS,

With the discussion period closing in a few hours, we wanted to briefly follow up on our responses and the additional experimental results. We have also incorporated these results into the revised manuscript. Even a brief indication of whether our responses address your concerns would be very helpful to us.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes in the Revised Manuscript

Dear Area Chair and Reviewers,

Thank you for your helpful feedback. We have updated the manuscript to incorporate the additional experiments and explanations from our responses, including the follow-up discussion with Reviewer DafB. The main revisions are:

  • External evaluation and training comparisons (Reviewer dLMs): Added NOAH Narrative QA results, the requested random-binding ablation, and a comparison with an adapted VideoHallu GRPO procedure (Sections 5.4–5.5; Appendix B.8).
  • Performance across the two streams (Reviewers dLMs and DafB): Expanded the Counter-intuitive results and explained why single-stream training can reduce performance on the other stream below Base, supported by qualitative examples (Sections 5.2 and 5.4; Appendix B.3.1).
  • Semantic priors and visual evidence (Reviewer DafB): Added measurements of model event expectations, controls for processing artifacts and answer polarity, and an intervention testing whether changing event expectations affects judgments of the same video. We also added matched-false-positive-rate comparisons showing improved discrimination between physically valid and invalid videos (Section 5.3; Appendices D.2–D.3).
  • Positive accuracy and observation–verdict consistency (Reviewers btqS and DafB): Expanded the discussion of positive-accuracy changes and provided the complete Observation–Verdict counts (Section 5.3; Appendix D.1).
  • Temporal sampling (Reviewers btqS and DafB): Added the sampled-frame inspection results and a comparison using 16 versus 32 frames (Section 5.1; Appendix B.4.1).

The new experiments specify the models, evaluation sets, and protocols used. We sincerely appreciate your time and the suggestions that helped us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.