- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies visual knowledge conflicts in VideoLLMs on counterfactual physical videos, where models may rely on parametric textual priors instead of visible motion evidence. The proposed method, VERTEX, has three stages. OKA tracks pixel motion with CoTracker, constructs motion masks, and pools motion-grounded vectors from the layer-zero visual embeddings. TSE extracts a textual-prior subspace from MLP activations obtained through text-only forward passes, and a kinematic subspace from attention outputs selected by the motion masks, then orthogonalizes the two subspaces. ODR suppresses the textual prior by projecting deep-layer MLP outputs and injects a projected motion-grounded vector into the attention residual. The method is evaluated on VideoHallu-Physics, VideoPHY2-Movement, SSV2-Reversed, and Real-Reversed using six VideoLLMs, with component ablations and an analysis of toxic prior alignment.
Critical strengths
-
The method design is closely tied to the paper’s mechanistic analysis. The authors use Logit Lens and Visual Knockout to locate the conflict in the imbalance between deep-layer MLP outputs and attention signals. They then design a two-stream intervention to suppress textual priors and strengthen motion evidence, providing an intuitive motivation for OKA and TSE.
-
VERTEX intervenes on the residual stream at inference time through orthogonal projection and signal injection. It requires neither fine-tuning nor paired counterfactual training data, making it applicable to several existing VideoLLMs, including both reasoning and standard models.
Critical weaknesses
-
The Qwen3-VL-8B baseline is inconsistent between Table 1 and Table 2. On VideoHallu-Physics with the standard input, Table 1 reports 65.31, whereas Table 2 reports 61.73 for what appears to be the same model and dataset. InternVL3.5-8B and LLaVA-NeXT have matching values across the two tables, so the discrepancy appears specific to Qwen3 or may reflect an unreported difference in evaluation settings. The authors should clarify which value is correct and explain the discrepancy.
-
The evaluation does not compare VERTEX with existing inference-time methods for mitigating hallucinations. Although the related work discusses STEAR, Nullu, VCD, and OPERA, the experiments compare VERTEX only with the original models and its own ablations. Without such comparisons, it is difficult to determine whether the gains arise specifically from the proposed geometric subspace alignment or reflect a more general effect of inference-time perturbation or regularization. The authors should include at least one relevant existing method under the same evaluation settings.
Questions for authors
-
Could the authors clarify whether the Qwen3-VL-8B baseline on VideoHallu-Physics is 65.31 or 61.73, and explain any differences in the evaluation settings between Tables 1 and 2? If the values come from different prompts or answer-parsing rules, please report the results under a unified evaluation protocol.
-
Table 2 reports 5.72 for Video-R1 and 0.24 for LLaVA-NeXT on SSV2-Reversed, a binary multiple-choice task for which chance accuracy is 50 percent. The extremely low baselines make the subsequent gains difficult to interpret and raise questions about the evaluation pipeline. What explains the far-below-chance baseline accuracy on SSV2-Reversed and Real-Reversed? Could the authors report the option distribution, answer-mapping procedure, and per-class accuracy?
4 comments
Response to reviewer Ldkw (Part 2)
Q2: Below-Chance Baseline Accuracy, Answer-Mapping Procedure, and Per-Class Breakdown
Near-zero accuracy on a binary task understandably looks like a parsing bug at first glance. However, on counterfactual benchmarks (SSV2-Reversed and Real-Reversed), these low scores happen because the models systematically default to forward-time language priors, not because of random guessing or formatting errors.
1. Answer-Mapping Procedure & Parsing Check:
- Prompt Setup: As shown in Figure 7, each prompt pairs a reversed video with two directional choices: Option A describes the normal forward-time event (e.g., an object falling down), while Option B describes the reversed motion actually shown in the video (e.g., an object rising upward into a hand).
- Answer Extraction: For standard instruction models (LLaVA-NeXT), we match the selected option letter directly from the output. For Chain-of-Thought models (Video-R1), we parse the final answer generated after the closing
</think>tag and verify text alignment when the option phrase is written out. Across all N=2,046 videos in SSV2-Reversed, 100% of baseline outputs map directly to Option A or Option B (0 unparsed or invalid responses), confirming there are no parser failures.
2. Option Distribution (SSV2-Reversed, N=2,046):
If the models were simply confused by the prompt format, their answers would split roughly 50% each. Instead, the predictions are heavily skewed toward Option A:
| Model | Option A (Incorrect) | Option B (Correct) | Unparsed / Invalid |
|---|---|---|---|
| Video-R1 | 1,929 (94.28%) | 117 (5.72%) | 0 (0.00%) |
| LLaVA-NeXT | 2,041 (99.76%) | 5 (0.24%) | 0 (0.00%) |
Rather than guessing at random, unmodified models pick standard forward-time physics on 94.28% to 99.76% of samples. In late decoder layers, MLP language priors override the visual tokens in the residual stream, leading the model to choose the plausible real-world event over what is shown in the frames. This is even stronger in LLaVA-NeXT (0.24%), which concatenates spatial frame features without explicit temporal tracking.
3. Per-Class Accuracy (SSV2-Reversed, Video-R1):
We show the results of Bbeaking down accuracy across the 8 action templates in SSV2-Reversed:
| SSV2-Reversed Action Category | Video-R1 Baseline Accuracy (%) |
|---|---|
| Letting [something] roll down a slanted surface | 0.00 |
| Pouring [something] into [something] | 0.00 |
| Squeezing [something] | 0.00 |
| Tearing [something] into two pieces | 0.19 |
| Spinning [something] that quickly stops spinning | 0.39 |
| Pushing [something] so that it falls off the table | 12.45 |
| Dropping [something] into [something] | 19.66 |
| Throwing [something] in the air and letting it fall | 36.22 |
| Overall | 5.72 |
Actions related to gravity or parabolic trajectories (throwing, dropping) retain moderate baseline accuracy, whereas actions related to thermodynamics or deformation (pouring, squeezing, tearing) are almost completely overridden by the language prior. We have added the answer-mapping details, option distributions, and per-class tables to Appendix H of our paper. We thank the reviewer again for their fruitful feedback and remain happy to address any additional questions.
Thank you for the detailed responses. These responses address my main concerns and provide useful additional evidence. I'll raise my rating from 2 to 3.
We deeply thank the reviewer for taking the time to review our rebuttal and for raising the score. We are happy that our additional experiments and theoretical clarifications addressed your concerns, and we again appreciate your constructive feedback, which has significantly strengthened our paper.
Response to reviewer Ldkw (Part 1)
W1 & Q1: Clarification on Qwen3-VL-8B Baseline Score in Table 1 vs. Table 2
We thank the reviewer for pointing out this mismatch. There is no difference in the underlying evaluation setup, prompt format, or frame sampling between Table 1 and Table 2. Both tables reference the exact same baseline run for Qwen3-VL-8B on the VideoHallu-Physics dataset.
The 65.31% entry in Table 1 was an accidental transcription error during writing, whereas 61.73% in Table 2 is the actual experimental score of the model. We have updated Table 1 in the revision of our paper to reflect 61.73%.
W2: Comparisons with Existing Inference-Time Mitigation Methods
We again thank the reviewer for raising this weakness. To test whether our improvements come from instance-specific subspace alignment rather than generic test-time perturbation, we benchmarked four inference-time baselines on Video-R1 using the exact same evaluation setup on VideoHallu-Physics and Real-Reversed:
VCD(Leng et al., 2024) andTriCD(Xing et al., 2026), which adjust output logits during decoding by contrasting against perturbed or counterfactual visual inputs.ITI(Li et al., 2023), which shifts hidden states along a fixed global direction, andNullu(Yang et al., 2025b), which projects out prior subspaces from MLP weights offline.(Note: We could not evaluate MASS or STEAR as neither has public code available, and OPERA's beam-search rollback mechanism becomes computationally intractable over Video-R1's long
<think>reasoning chains.)Here are the results:
Key Takeaways:
Static steering and offline weight editing (
ITI&Nullu): If our improvements simply came from generic activation noise or regularization,ITIandNulluwould also improve performance. Instead,ITIhurts accuracy on both datasets (↓ 10.20% and ↓ 4.00%) because a single fixed vector cannot represent sample-specific motion trajectories. WhileNullureaches a high score (36.00%) on Real-Reversed, it drops significnatly on VideoHallu-Physics (40.31%). On the binary Real-Reversed task, the unmodified baseline picks the forward-time textbook option 86% of the time (14.00%). Permanently projecting out MLP weights inNullubreaks the model's<think>reasoning chains and pushes predictions toward uncalibrated, near-random outputs.Limits of output-level logit contrasting (
TriCD&VCD):TriCDperforms well on VideoHallu-Physics (65.31%) but falls below the baseline on Real-Reversed (10.00%). Because contrastive decoding only adjusts logits at the final layer, when deep MLP blocks have already overwritten visual features in the residual stream, the input perturbations used byTriCDcorrupt the frame-to-frame motion cues needed to detect time reversal, causing the model to rely even more heavily on its language prior.Instance-specific subspace alignment (
VERTEX): Rather than using fixed global directions or late-stage logit penalties, VERTEX computes per-video subspaces to suppress prompt priors while injecting tracked kinematic features directly into the residual stream. AlongsideVCD,VERTEXis the only method that improves over the baseline in both regimes, achieving the top score on VideoHallu-Physics (66.33%) and the highest average overall (42.17%, ↑ 5.58%). We have added these results to Section 5.2 of the revised paper.