acceptodds
Under review as a conference paper at ICLR 2027

Starve the MLP, Feed the Attention: Resolving Vision-Knowledge Conflicts in VideoLLMs

Abstract

Modern video models have significantly advanced the field of video understanding. However, as these models generate responses, they overly depend on intrinsic textual priors rather than actual visual evidence, fundamentally restricting their performance. This over-reliance, called vision-knowledge conflicts, is especially evident in videos depicting counterfactual physics, where models hallucinate standard physics rules rather than processing the actual counterfactual visual input. Our mechanistic analyses reveal that as Transformer layers deepen, these language priors progressively dominate the residual stream, effectively overshadowing the visual tokens. To resolve this issue, we propose Vertex (Visual Evidence Restoration and Textual EXclusion), a zero-shot, inference-time intervention framework. Vertex first isolates physical motion to generate binary kinematic masks, which are directly utilized to construct distinct latent geometric subspaces. During inference, the framework actively rebalances the model by strengthening its focus on actual visual evidence while filtering out parametric textual biases. Evaluations on counterfactual physics benchmarks show that Vertex reduces vision-knowledge conflicts, outperforming standard baselines and providing a more robust framework for grounded video understanding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Video Large Language Models, Vision-Knowledge Conflicts

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LdkwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies visual knowledge conflicts in VideoLLMs on counterfactual physical videos, where models may rely on parametric textual priors instead of visible motion evidence. The proposed method, VERTEX, has three stages. OKA tracks pixel motion with CoTracker, constructs motion masks, and pools motion-grounded vectors from the layer-zero visual embeddings. TSE extracts a textual-prior subspace from MLP activations obtained through text-only forward passes, and a kinematic subspace from attention outputs selected by the motion masks, then orthogonalizes the two subspaces. ODR suppresses the textual prior by projecting deep-layer MLP outputs and injects a projected motion-grounded vector into the attention residual. The method is evaluated on VideoHallu-Physics, VideoPHY2-Movement, SSV2-Reversed, and Real-Reversed using six VideoLLMs, with component ablations and an analysis of toxic prior alignment.

Critical strengths
  1. The method design is closely tied to the paper’s mechanistic analysis. The authors use Logit Lens and Visual Knockout to locate the conflict in the imbalance between deep-layer MLP outputs and attention signals. They then design a two-stream intervention to suppress textual priors and strengthen motion evidence, providing an intuitive motivation for OKA and TSE.

  2. VERTEX intervenes on the residual stream at inference time through orthogonal projection and signal injection. It requires neither fine-tuning nor paired counterfactual training data, making it applicable to several existing VideoLLMs, including both reasoning and standard models.

Critical weaknesses
  1. The Qwen3-VL-8B baseline is inconsistent between Table 1 and Table 2. On VideoHallu-Physics with the standard input, Table 1 reports 65.31, whereas Table 2 reports 61.73 for what appears to be the same model and dataset. InternVL3.5-8B and LLaVA-NeXT have matching values across the two tables, so the discrepancy appears specific to Qwen3 or may reflect an unreported difference in evaluation settings. The authors should clarify which value is correct and explain the discrepancy.

  2. The evaluation does not compare VERTEX with existing inference-time methods for mitigating hallucinations. Although the related work discusses STEAR, Nullu, VCD, and OPERA, the experiments compare VERTEX only with the original models and its own ablations. Without such comparisons, it is difficult to determine whether the gains arise specifically from the proposed geometric subspace alignment or reflect a more general effect of inference-time perturbation or regularization. The authors should include at least one relevant existing method under the same evaluation settings.

Questions for authors
  1. Could the authors clarify whether the Qwen3-VL-8B baseline on VideoHallu-Physics is 65.31 or 61.73, and explain any differences in the evaluation settings between Tables 1 and 2? If the values come from different prompts or answer-parsing rules, please report the results under a unified evaluation protocol.

  2. Table 2 reports 5.72 for Video-R1 and 0.24 for LLaVA-NeXT on SSV2-Reversed, a binary multiple-choice task for which chance accuracy is 50 percent. The extremely low baselines make the subsequent gains difficult to interpret and raise questions about the evaluation pipeline. What explains the far-below-chance baseline accuracy on SSV2-Reversed and Real-Reversed? Could the authors report the option distribution, answer-mapping procedure, and per-class accuracy?

4 comments
AuthorsOpenReview ↗

Response to reviewer Ldkw (Part 1)

W1 & Q1: Clarification on Qwen3-VL-8B Baseline Score in Table 1 vs. Table 2

We thank the reviewer for pointing out this mismatch. There is no difference in the underlying evaluation setup, prompt format, or frame sampling between Table 1 and Table 2. Both tables reference the exact same baseline run for Qwen3-VL-8B on the VideoHallu-Physics dataset.

The 65.31% entry in Table 1 was an accidental transcription error during writing, whereas 61.73% in Table 2 is the actual experimental score of the model. We have updated Table 1 in the revision of our paper to reflect 61.73%.


W2: Comparisons with Existing Inference-Time Mitigation Methods

We again thank the reviewer for raising this weakness. To test whether our improvements come from instance-specific subspace alignment rather than generic test-time perturbation, we benchmarked four inference-time baselines on Video-R1 using the exact same evaluation setup on VideoHallu-Physics and Real-Reversed:

  1. Contrastive Decoding: VCD (Leng et al., 2024) and TriCD (Xing et al., 2026), which adjust output logits during decoding by contrasting against perturbed or counterfactual visual inputs.
  2. Activation & Weight Editing: ITI (Li et al., 2023), which shifts hidden states along a fixed global direction, and Nullu (Yang et al., 2025b), which projects out prior subspaces from MLP weights offline.

(Note: We could not evaluate MASS or STEAR as neither has public code available, and OPERA's beam-search rollback mechanism becomes computationally intractable over Video-R1's long <think> reasoning chains.)

Here are the results:

MethodVideoHallu-Physics (%)Real-Reversed (%)Average (%)
Baseline (Video-R1)59.1814.0036.59
ITI (Li et al., 2023)48.98 (↓ 10.20)10.00 (↓ 4.00)29.49 (↓ 7.10)
Nullu (Yang et al., 2025b)40.31 (↓ 18.87)36.00 (↑ 22.00)38.16 (↑ 1.57)
TriCD (Xing et al., 2026)65.31 (↑ 6.13)10.00 (↓ 4.00)37.66 (↑ 1.07)
VCD (Leng et al., 2024)62.76 (↑ 3.58)16.00 (↑ 2.00)39.38 (↑ 2.79)
VERTEX (Ours)66.33 (↑ 7.15)18.00 (↑ 4.00)42.17 (↑ 5.58)

Key Takeaways:

  • Static steering and offline weight editing (ITI & Nullu): If our improvements simply came from generic activation noise or regularization, ITI and Nullu would also improve performance. Instead, ITI hurts accuracy on both datasets (↓ 10.20% and ↓ 4.00%) because a single fixed vector cannot represent sample-specific motion trajectories. While Nullu reaches a high score (36.00%) on Real-Reversed, it drops significnatly on VideoHallu-Physics (40.31%). On the binary Real-Reversed task, the unmodified baseline picks the forward-time textbook option 86% of the time (14.00%). Permanently projecting out MLP weights in Nullu breaks the model's <think> reasoning chains and pushes predictions toward uncalibrated, near-random outputs.

  • Limits of output-level logit contrasting (TriCD & VCD): TriCD performs well on VideoHallu-Physics (65.31%) but falls below the baseline on Real-Reversed (10.00%). Because contrastive decoding only adjusts logits at the final layer, when deep MLP blocks have already overwritten visual features in the residual stream, the input perturbations used by TriCD corrupt the frame-to-frame motion cues needed to detect time reversal, causing the model to rely even more heavily on its language prior.

  • Instance-specific subspace alignment (VERTEX): Rather than using fixed global directions or late-stage logit penalties, VERTEX computes per-video subspaces to suppress prompt priors while injecting tracked kinematic features directly into the residual stream. Alongside VCD, VERTEX is the only method that improves over the baseline in both regimes, achieving the top score on VideoHallu-Physics (66.33%) and the highest average overall (42.17%, ↑ 5.58%). We have added these results to Section 5.2 of the revised paper.

AuthorsOpenReview ↗

Response to reviewer Ldkw (Part 2)

Q2: Below-Chance Baseline Accuracy, Answer-Mapping Procedure, and Per-Class Breakdown

Near-zero accuracy on a binary task understandably looks like a parsing bug at first glance. However, on counterfactual benchmarks (SSV2-Reversed and Real-Reversed), these low scores happen because the models systematically default to forward-time language priors, not because of random guessing or formatting errors.

1. Answer-Mapping Procedure & Parsing Check:

  • Prompt Setup: As shown in Figure 7, each prompt pairs a reversed video with two directional choices: Option A describes the normal forward-time event (e.g., an object falling down), while Option B describes the reversed motion actually shown in the video (e.g., an object rising upward into a hand).
  • Answer Extraction: For standard instruction models (LLaVA-NeXT), we match the selected option letter directly from the output. For Chain-of-Thought models (Video-R1), we parse the final answer generated after the closing </think> tag and verify text alignment when the option phrase is written out. Across all N=2,046 videos in SSV2-Reversed, 100% of baseline outputs map directly to Option A or Option B (0 unparsed or invalid responses), confirming there are no parser failures.

2. Option Distribution (SSV2-Reversed, N=2,046):

If the models were simply confused by the prompt format, their answers would split roughly 50% each. Instead, the predictions are heavily skewed toward Option A:

ModelOption A (Incorrect)Option B (Correct)Unparsed / Invalid
Video-R11,929 (94.28%)117 (5.72%)0 (0.00%)
LLaVA-NeXT2,041 (99.76%)5 (0.24%)0 (0.00%)

Rather than guessing at random, unmodified models pick standard forward-time physics on 94.28% to 99.76% of samples. In late decoder layers, MLP language priors override the visual tokens in the residual stream, leading the model to choose the plausible real-world event over what is shown in the frames. This is even stronger in LLaVA-NeXT (0.24%), which concatenates spatial frame features without explicit temporal tracking.

3. Per-Class Accuracy (SSV2-Reversed, Video-R1):

We show the results of Bbeaking down accuracy across the 8 action templates in SSV2-Reversed:

SSV2-Reversed Action CategoryVideo-R1 Baseline Accuracy (%)
Letting [something] roll down a slanted surface0.00
Pouring [something] into [something]0.00
Squeezing [something]0.00
Tearing [something] into two pieces0.19
Spinning [something] that quickly stops spinning0.39
Pushing [something] so that it falls off the table12.45
Dropping [something] into [something]19.66
Throwing [something] in the air and letting it fall36.22
Overall5.72

Actions related to gravity or parabolic trajectories (throwing, dropping) retain moderate baseline accuracy, whereas actions related to thermodynamics or deformation (pouring, squeezing, tearing) are almost completely overridden by the language prior. We have added the answer-mapping details, option distributions, and per-class tables to Appendix H of our paper. We thank the reviewer again for their fruitful feedback and remain happy to address any additional questions.

Thank you for the detailed responses. These responses address my main concerns and provide useful additional evidence. I'll raise my rating from 2 to 3.

We deeply thank the reviewer for taking the time to review our rebuttal and for raising the score. We are happy that our additional experiments and theoretical clarifications addressed your concerns, and we again appreciate your constructive feedback, which has significantly strengthened our paper.

Reviewer_kKniOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates vision-knowledge conflicts in VideoLLMs and provides behavioral and mechanistic evidence of models’ reliance on textual priors over visual evidence. It finds that in deeper layers, textual priors from MLP blocks can override visual signals retained by attention. It also proposes VERTEX, a training-free, test-time framework that mitigates this conflict by suppressing textual priors in MLP outputs and reinforcing motion-related visual evidence in attention outputs. The method robustly improves performance across multiple video (reasoning) models and counterfactual physics benchmarks.

Critical strengths
  1. Clear and important research question. The paper addresses a well-motivated problem that has been repeatedly observed in prior work. The literature review is thorough and clearly establishes its relevance. The experimental design is well suited to studying vision–knowledge conflicts, particularly by creating explicit disagreements between observed motion and familiar physical expectations.

  2. Solid behavioral and mechanistic analyses. The paper combines intuitive qualitative examples with quantitative analyses. Text-only, first-frame-only, shuffled, and reversed-video controls examine models’ dependence on visual input and temporal order, while logit-lens and visual-knockout analyses investigate how this dependence changes inside the model. These complementary perspectives provide substantial evidence of the phenomenon and a useful account of its possible mechanisms.

  3. Novel and clearly formulated method. The combination of motion-based grounding, input-specific subspace extraction, and dual-stream intervention is interesting and mathematically clear. The method also follows naturally from the analysis: suppressing textual-prior components in MLP outputs while reinforcing visual evidence in attention directly targets the identified imbalance. This connection between the diagnosis and the intervention makes the design particularly compelling.

  4. Elegant training-free framework. VERTEX operates at test time without updating the pretrained model weights or requiring additional task-specific training. Deriving the intervention subspaces from the current input is an appealing way to adapt the correction to individual examples.

  5. Well-chosen evaluation settings and reasonable model coverage. The benchmarks directly target the conflicts the method aims to resolve, and the evaluation covers both standard VideoLLMs and video reasoning models. The reported improvements across multiple models and datasets support the method’s effectiveness, while the component ablations provide further evidence about the contributions of the proposed interventions.

Critical weaknesses
  1. Missing comparisons with existing methods. Although the evaluation covers multiple model backbones, the comparisons mainly involve their unmodified versions. Comparisons with existing inference-time hallucination mitigation or activation intervention methods are needed to establish VERTEX’s advantage over other approaches.

  2. Ablation results do not consistently support the necessity of all components. In Table 5, the dense-mask and global-subspace variants outperform the full framework on VideoPHY2-Movement with Qwen3-VL-Thinking: 65.60% and 67.73%, respectively, versus 63.47%. These results suggest that the benefits of motion-based masking and instance-specific subspaces depend on the setting. The paper should clarify when these components help and moderate its claims about their necessity.

Questions for authors

Why is the evaluation limited to models with around 7B–8B parameters? Do the authors have evidence that the observed attention–MLP imbalance and the effectiveness of VERTEX also hold for substantially smaller or larger models?

3 comments
AuthorsOpenReview ↗

Response to reviewer kKni (Part 1)

W1: Comparisons with Existing Inference-Time Mitigation Methods

We appreciate this point from the reviewer. To directly compare VERTEX against existing test-time hallucination mitigation and activation intervention approaches, we evaluated four baselines on the Video-R1 backbone across VideoHallu-Physics and Real-Reversed: two contrastive decoding methods: VCD (Leng et al., 2024) and TriCD (Xing et al., 2026), and two internal weight intervention methods: ITI (Li et al., 2023) and Nullu (Yang et al., 2025b). We could not include recent hallucination mitigation methods MASS and STEAR as their official code has not been released, and OPERA's beam-search rollback is computationally complex over Video-R1's long <think> reasoning sequences.

Here are the results of the inferences:

MethodVideoHallu-Physics (%)Real-Reversed (%)Average (%)
Baseline (Video-R1)59.1814.0036.59
ITI (Li et al., 2023)48.98 (↓ 10.20)10.00 (↓ 4.00)29.49 (↓ 7.10)
Nullu (Yang et al., 2025b)40.31 (↓ 18.87)36.00 (↑ 22.00)38.16 (↑ 1.57)
TriCD (Xing et al., 2026)65.31 (↑ 6.13)10.00 (↓ 4.00)37.66 (↑ 1.07)
VCD (Leng et al., 2024)62.76 (↑ 3.58)16.00 (↑ 2.00)39.38 (↑ 2.79)
VERTEX (Ours)66.33 (↑ 7.15)18.00 (↑ 4.00)42.17 (↑ 5.58)

1. Performance on Standard Physical Dynamics (VideoHallu-Physics): Methods that rely on static or offline directions (ITI and Nullu) score relatively low here, dropping performance by 10.20% and 18.87% below the unmodified backbone. Because ITI shifts activations along a single fixed vector, it cannot represent video-specific motion trajectories and thus acts as out-of-distribution noise. Similarly, Nullu permanently orthogonalizes MLP weights against an offline subspace, which strips away deductive knowledge needed during the model's <think> reasoning step. While contrastive decoding methods (VCD and TriCD) improve over the baseline (62.76% and 65.31%), they require several forward passes per generated token and only adjust final logits. VERTEX achieves the highest accuracy (66.33%) in a single generative pass by intervening directly in the intermediate residual stream.

2. Performance under Temporal Reversal (Real-Reversed): First, both ITI and TriCD fall below the unmodified baseline on Real-Reversed (10.00%). ITI drops because a single static steering vector cannot represent sample-specific reversed trajectories, while TriCD, which only contrasts output logits after deep MLP blocks have already overwritten visual states in the residual stream, relies on strong visual perturbations that degrade the frame-to-frame motion cues needed to recognize reversed playback. While VCD avoids this drop by using milder Gaussian noise to penalize language priors, it yields only a modest gain on Real-Reversed (16.00%) at doubled inference cost, as adjusting final logits cannot restore kinematic features that were already lost in intermediate layers. Conversely, Nullu reaches 36.00% on Real-Reversed only at the cost of breaking general physical reasoning on VideoHallu-Physics. On the binary Real-Reversed task, the baseline selects the forward-time textbook option 86% of the time. Permanently projecting out MLP weights in Nullu disrupts the <think> reasoning chain and pushes predictions toward uncalibrated, near-random outputs.

3. Overall: By extracting sample-specific subspaces (Vmlp,WattnV_{mlp},W_{attn}) and balancing prior suppression with kinematic injection inside the residual stream, VERTEX avoids both failure modes. Alongside VCD, it is the only approach that improves over the baseline on both benchmarks, achieving the highest overall average (42.17%, ↑ 5.58%). We have added this comparison and analysis to Section 5.2 of the revised paper.

AuthorsOpenReview ↗

Response to reviewer kKni (Part 2)

W2: Component Necessity in CoT Architectures and Moderating Claims (Table 5)

We sincerely appreciate the reviewer for pointing this out. We fully agree that the effectiveness of motion-based masking (OKA) and instance-specific subspace extraction (TSE) depends on the underlying model architecture. We discussed this exact performance inversion in Appendix G of our original submission, and will now elaborate more about it here:

1. Why w/o OKA and w/o TSE outperform the full pipeline on Qwen3-VL-Thinking (VideoPHY2-Movement): As noted in Appendix G, this behavior stems from the constructive role of textual priors in Chain-of-Thought (CoT) architectures:

  • Destructive vs. Constructive Priors: In standard instruction-tuned VideoLLMs that output answers directly, deep MLP layers mostly act as textual priors that override visual evidence. However, Qwen3-VL-Thinking uses its <think> phase to work through multi-step physical deductions. On a physics benchmark like VideoPHY2-Movement, the MLP's parametric knowledge does more than just introducing bias: it also supplies the underlying physical principles (such as gravity, momentum, and collision dynamics) needed to evaluate the video. Because of this, applying full instance-level VERTEX suppression can unintentionally dampen valid reasoning steps during <think> generation.
  • Why Global Subspace (w/o TSE, 67.73%) helps CoT reasoning: Replacing instance-specific TSE with a Global Subspace computed by averaging singular vectors across an offline calibration set acts as a generalized low-pass filter. It removes broad, dataset-wide linguistic biases while leaving delicate, instance-specific logical chains intact during the <think> rollout.
  • Why Dense Masking (w/o OKA, 65.60%) helps on VideoPHY2-Movement: Using a dense mask grounds the Attention mechanism across the full visual scene (retaining static contextual cues such as ramps, support surfaces, and boundaries) without overly constraining the representation to sparse trajectories while the MLP carries out multi-step deduction.

2. Clarifying Why Full VERTEX is Necessary: While relaxing OKA or TSE benefits CoT models on scene-level physics verification, both ablated variants carry fundamental trade-offs that make the full VERTEX pipeline necessary for general zero-shot deployment:

  • Trade-off of w/o OKA (Dense Mask): Replacing kinematic tracking with a dense mask allows static background noise to leak into the anchor rather than isolating true temporal motion. When tasks require strict trajectory discrimination or when background clutter is high, removing OKA degrades grounding, such as dropping accuracy from 66.33% to 63.27% in Table 3 on VideoHallu-Physics.
  • Trade-off of w/o TSE (Global Subspace): Computing a Global Subspace requires prior access to an offline calibration dataset, which fundamentally breaks the strictly zero-shot nature of the framework. Instance-level TSE is required whenever offline calibration data from the target distribution is unavailable.
AuthorsOpenReview ↗

Response to reviewer kKni (Part 3)

Q1: Generalization Across Model Scales (2B–4B and 14B–32B)

We thank the reviewer for asking about this point. We focused our main evaluation on the 7B–8B parameter range because it is the shared scale where all major open-weight VideoLLM families (Video-R1, Qwen3-VL, InternVL3.5, and LLaVA-NeXT) provide both standard instruction-tuned and CoT checkpoints, allowing a controlled comparison across architectures.

To verify whether VERTEX's effectiveness hold across substantially smaller and larger scales, we expanded our evaluation across 2B, 4B, 14B, and 32B models on VideoHallu-Physics and Real-Reversed:

ModelVideoHallu-Physics (%)Real-Reversed (%)
Qwen3-VL-2B-Thinking67.3516.00
+ VERTEX (Ours)67.35 (0.00)34.00 (↑ 18.00)
InternVL3.5-4B67.8642.00
+ VERTEX (Ours)68.37 (↑ 0.51)48.00 (↑ 6.00)
InternVL3.5-14B-Thinking69.3934.00
+ VERTEX (Ours)69.90 (↑ 0.51)36.00 (↑ 2.00)
Qwen3-VL-32B-Instruct62.2412.00
+ VERTEX (Ours)65.31 (↑ 3.07)12.00 (0.00)

Key Findings Across Model Scales:

  1. Larger Models: Simply scaling up model capacity does not resolve vision-knowledge conflict. In fact, larger models encode even stronger prior knowledge in their expanded MLP blocks. For example, Qwen3-VL-32B-Instruct still drops to 12.00% on Real-Reversed and scores 62.24% on VideoHallu-Physics. Intervening with VERTEX improves accuracy at both the 14B and 32B scales (↑ 3.07% on VideoHallu-Physics for Qwen3-VL-32B-Instruct, and gains across both benchmarks on InternVL3.5-14B-Thinking).
  2. Smaller Models: Compact models have fewer attention heads to maintain fine-grained spatiotemporal tracking across frames, causing them to fall back heavily on linguistic shortcuts when visual motion is counterfactual. By anchoring the residual stream to tracked kinematic tokens and suppressing the MLP prior, VERTEX more than doubles reversal accuracy on Qwen3-VL-2B-Thinking (↑ 18.00%) and improves both benchmarks on InternVL3.5-4B (↑ 0.51% and ↑ 6.00%).

These results confirm that the dominance of deep MLP priors over visual attention is an architectural property of causal multimodal Transformers across scales (from 2B to 32B) rather than an artifact of 7B–8B models. We added this multi-scale analysis to Section 5.2 of our revised paper. We again deeply appreciate the insightful comments from the reviewer, and welcome additional discussions during the rebuttal period.

Reviewer_yyTYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies vision–knowledge conflicts in VideoLLMs, where textual priors override contradictory visual evidence. It proposes VERTEX, a training-free framework that uses motion tracking to identify visual evidence, extracts textual and visual subspaces, and suppresses MLP priors while reinforcing Attention representations. Experiments across six VideoLLMs and four physics-oriented benchmarks report improvements on most settings.

Critical strengths
  1. Addresses an important failure mode of VideoLLMs using counterfactual physical videos.
  2. Proposes a modular inference-time intervention without model retraining.
  3. Evaluates multiple model families and provides diagnostic experiments, ablations, and qualitative analyses.
Critical weaknesses
  1. Inconsistent results of Qwen3-VL-8B between table 1 and table 2. In table 1, it is 65.31 while in talbe 2, it is 61.73. And InternVL3.5-8B-Thinking performance is 65.31 in first-frame setting, maybe this is a transcription error?
  2. The OKA encodes motion in a norm format, does this strategy overlook the direction of the motion? How does such an embedding recognize similar motion but with a different direction? And in some anti-physical motion, this is a critical point.
  3. How to ensure the SVD after the mean-centering can obtain the motion component? It is not rigorous to directly claim that performing PCA with motion can obtain the principle component of "motion", it just obtain some high variance components that might also inlcude background, object id, position, etc.
  4. Is there any analysis of the kinematic truth vector? Does it really convey motion semantics? It is obtained via a mean pooling operation, which probably cannot guarantee a great portion of motion information intuitively.
  5. The method enforces geometric orthogonality, but does not establish semantic disentanglement between motion evidence and textual priors. In other words, the orthogonality could bring disentanglement, but does textual information and motion semantic are really orthogonal in embedding space?
Questions for authors

See the weakness above

3 comments
AuthorsOpenReview ↗

Response to reviewer yyTY (Part 1)

W1: Transcription Error in Table 1 and Table 2

We thank the reviewer for their thorough attention to detail in catching this inconsistency. The reviewer is absolutely correct that the baseline accuracy for Qwen3-VL-8B on the VideoHallu-Physics dataset must be the same across both tables.

This discrepancy was the result of a simple typographical transcription error in Table 1 during the final formatting. After rechecking the inference results, we confirm that the value reported in Table 2 (61.73%) is the correct empirical baseline accuracy. We have updated Table 1 in the revised paper to reflect this true value. We again greatly appreciate the reviewer's careful reading.


W2: Role of the Displacement Norm in OKA and Preservation of Motion Directionality

We thank the reviewer for this insightful theoretical question. Distinguishing between identical movements in opposite directions (e.g., falling downward vs. rising upward against gravity) is indeed critical for detecting anti-physical motion. We clarify that VERTEX never compresses motion into a directionless scalar norm. Instead, directional information is preserved through three coupled mechanisms:

1. The Euclidean Norm in Eq. (1) is a Binary Spatial Gate, Not a Feature Embedding: In OKA, the displacement norm in Eq. (1) is never fed into the model as a motion feature. Its only mathematical role is to serve as a binary threshold that constructs the 3D spatiotemporal mask Mkin\mathcal{M}_{kin}, separating moving foreground tokens from static background clutter and camera ego-motion. The active visual tokens (Mkin=1\mathcal{M}_{kin} = 1) retain their full dd-dimensional embeddings and trace inverted coordinate sequences for opposite motions, shifting from (tearlyt_{early}, ytopy_{top}) →\to (tlatet_{late}, ybottomy_{bottom}) for a falling object versus (tearlyt_{early}, ybottomy_{bottom}) →\to (tlatet_{late}, ytopy_{top}) for a rising one.

2. Asymmetric Temporal Encoding via 3D Positional Embeddings and Causal Self-Attention: Within the backbone, the tokens isolated by Mkin\mathcal{M}_{kin} encode motion directions through two architectural properties:

  • Spatiotemporal Positional Encoding: Visual tokens are processed via 3D patch tubelets and stamped with 3D positional encodings (e.g., 3D Multimodal RoPE across t,y,xt, y, x). Thus, an active trajectory from (t1t_1, ytopy_{top}) →\to (t2t_2, ybottomy_{bottom}) carries a distinct positional phase signature from its inverse (t1t_1, ybottomy_{bottom}) →\to (t2t_2, ytopy_{top}).
  • Temporal Non-Commutativity of Causal Attention: Under the decoder's strictly causal mask, tokens at frame tt attend only to preceding frames (t′≤tt' \le t). For motion A→BA \to B, tokens at BB (t2t_2) attend backward to AA (t1t_1), while for B→AB \to A, tokens at AA (t2t_2) attend backward to BB (t1t_1). Because causal attention is temporally asymmetric, opposite trajectories produce distinct deep Attention states ΔHattn\Delta H_{attn} at layer l≥Lcritl \ge L_{crit}.

3. Isolating Directional Variance via Mean-Centered SVD (TSE) and Subspace Projection (ODR): If the unweighted Layer-0 mean-pooled anchor hvis∗h_{vis}^{\ast} were injected in isolation, static object appearance would dominate because summation across frames is commutative. Crucially, VERTEX never injects raw hvis∗h_{vis}^{*} directly. Instead, ODR injects the subspace-projected vector β(WattnWattn⊤hvis∗)\beta (W_{attn}W_{attn}^\top h_{vis}^{\ast}) (Eq. 7). In TSE (Section 4.2), after projecting out the MLP prior subspace VmlpV_{mlp} from the active Mkin=1\mathcal{M}_{kin}=1 attention states, Eq. (5) applies mean-centered SVD:

ΔH^attn−Hˉ=\Delta \hat{H}_{attn} - \bar{H} = UattnU_{attn} Σattn\Sigma_{attn} Wattn⊤W_{attn}^\top

Subtracting the temporal mean Hˉ\bar{H} removes the time-invariant static appearance shared between opposite motions, ensuring Wattn∈Rd×mW_{attn} \in \mathbb{R}^{d \times m} spans only directional state transitions across the causal trajectory. Projecting hvis∗h_{vis}^{\ast} through WattnW_{attn} Wattn⊤W_{attn}^\top in Eq. (7) filters out direction-agnostic static appearance in the orthogonal complement, retaining only features aligned with the observed trajectory direction.

4. Existing Empirical Validation on Strictly Directional Benchmarks (Table 2): This is directly validated by our results on SSV2-Reversed and Real-Reversed (Table 2), where every video is a temporal reversal of a natural action—meaning object identity, background, and displacement norms are identical to forward physics, and the sole distinguishing signal is the 180-degree reversal of motion direction. Rather than overlooking direction, VERTEX yields consistent gains across all architectures (e.g., ↑ 12.90% on SSV2-Reversed with Video-R1 and ↑ 58.00% on Real-Reversed with LLaVA-NeXT). We have expanded Sections 4.1 and 4.2 in the revision to clarify this mechanism.

AuthorsOpenReview ↗

Response to reviewer yyTY (Part 2)

W3: How Filtered and Mean-Centered SVD Isolates Kinematic Variance from Background, Object ID, and Position

We agree with the reviewer's concern: applying PCA/SVD directly to raw, unmasked video tokens would not isolate motion, as the leading singular vectors would be dominated by static background clutter, object identity, and spatial layout.

In Targeted Subspace Extraction (TSE), VERTEX never applies SVD to raw video tokens, but representations pass through a three-stage filtration pipeline:

1. Eliminating Background Variance via Spatiotemporal Gating: Rather than using all NvisN_{vis} tokens, TSE constructs ΔHattn∈RP×d\Delta H_{attn} \in \mathbb{R}^{P \times d} exclusively from the PP active foreground tokens where the CoTracker mask is active (Mkin[j]=1\mathcal{M}_{kin}[j]=1). Because stationary background patches are excluded out in OKA, background tokens never enter ΔHattn\Delta H_{attn}.

2. Eliminating Linguistic Prior Variance via Orthogonal Projection (Eq. 4): Before computing SVD, Eq. (4) projects ΔHattn\Delta H_{attn} onto the orthogonal complement of the MLP prior subspace:

ΔH^attn=\Delta \hat{H}_{attn} = ΔHattn−\Delta H_{attn} - ΔHattn\Delta H_{attn} VmlpV_{mlp} Vmlp⊤V_{mlp}^\top

This strips out high-variance semantic shortcut features aligned with the text prompt, ensuring the remaining variance is strictly visually grounded.

3. Eliminating Static Object ID via Temporal Mean-Centering (Eq. 5): Because Mkin=1\mathcal{M}_{kin}=1 tracks the moving object across frames, each row j∈[1,P]j \in [1, P] of ΔH^attn\Delta \hat{H}_{attn} samples the same entity at different time steps tt. Decomposing a tracked token into a time-invariant object identity component μid\mu_{id} (category, color, texture) and a time-varying kinematic component δkin(t)\delta_{kin}(t) gives ΔH^attn(t)=\Delta \hat{H}_{attn}(t) = μid+\mu_{id} + δkin(t)\delta_{kin}(t). In Eq. (5), subtracting the trajectory mean Hˉ=1P∑j=1P\bar{H} = \frac{1}{P}\sum_{j=1}^P ΔH^attn[j]\Delta \hat{H}_{attn}[j] prior to SVD:

ΔH^attn−Hˉ=\Delta \hat{H}_{attn} - \bar{H} = UattnU_{attn} Σattn\Sigma_{attn} Wattn⊤W_{attn}^\top

cancels out the constant static identity μid\mu_{id}, leaving only zero-mean temporal fluctuations δkin(t)−\delta_{kin}(t) - δˉkin\bar{\delta}_{kin}.

4. Why Remaining "Position" Variance Encodes the Kinematic Trajectory: While positional variance in a static image merely reflects spatial layout, once tokens are restricted to a moving object track and static identity (Hˉ\bar{H}) is subtracted, the change in spatial position (xtx_t, yty_t) across time tt is the exact definition of a spatiotemporal trajectory (Δposition/Δt\Delta \text{position} / \Delta t). Capturing the principal axes of these positional transitions across frames is precisely what allows WattnW_{attn} to represent velocity and trajectory direction.

We have accordingly revised Section 4.2 of our paper.

AuthorsOpenReview ↗

Response to reviewer yyTY (Part 3)

W4: Analysis of the Kinematic Truth Vector (hvis∗h_{vis}^{\ast}) and Preservation of Motion Semantics

We thank the reviewer for raising this question. Unlike naïve global pooling that dilutes motion with background clutter, VERTEX prevents feature dilution via three coupled mechanisms:

1. Localized Spatiotemporal Gating (Eq. 2): Masked mean pooling aggregates exclusively over sparse foreground coordinates (Mkin[j]=1\mathcal{M}_{kin}[j]=1), filtering out stationary background and ego-motion:

hvis∗h_{vis}^{*} =1P∑j= \frac{1}{P} \sum_{j} Mkin(j)⋅\mathcal{M}_{kin}(j) \cdot Zvis(j)Z_{vis}(j)

2. 3D Tubelet Encoding in Layer-0 Tokens (ZvisZ_{vis}): Because Layer-0 tokens are generated by a 3D spatiotemporal vision encoder, active tokens inside Mkin=1\mathcal{M}_{kin}=1 already encode local motion gradients and optical flow prior to pooling.

3. Subspace Filtering via ODR (Eq. 7): VERTEX never injects raw hvis∗h_{vis}^{\ast} directly. Instead, hvis∗h_{vis}^{\ast} serves as an uncorrupted feature reservoir projected onto WattnW_{attn} Wattn⊤W_{attn}^\top in deep layers (l≥Lcritl \ge L_{crit}). Because WattnW_{attn} is extracted via mean-centered SVD after subtracting the static mean Hˉ\bar{H} and orthogonalizing against VmlpV_{mlp}, this projection filters out residual static appearance and retains only directional motion features.

4. Empirical Verification (Table 3 & Appendix G): Ablations confirm that hvis∗h_{vis}^{\ast} provides essential motion semantics: replacing masked pooling with dense unmasked pooling (w/o OKA) drops VideoHallu-Physics accuracy by ↓ 3.06%, while removing kinematic injection (w/o Injection, β=0\beta=0) fails to recover full accuracy.

We have updated Sections 4.1 and 5.4 in the revised paper accordingly.


W5: Geometric Orthogonality vs. Semantic Disentanglement in Embedding Space

We thank the reviewer for this insightful conceptual question. We agree that textual priors and visual motion semantics are not natively orthogonal in the shared multimodal embedding space—this native overlap is the root cause of vision-knowledge conflicts in deep layers. Rather than assuming native orthogonality, VERTEX constructs a disentangled subspace geometry:

1. Semantic Source Separation: Before enforcing orthogonality, VERTEX isolates the two semantic sources along distinct pathways: VmlpV_{mlp} is extracted strictly from text prompt tokens via MLP sub-layers, while candidate kinematics (ΔHattn\Delta H_{attn}) are gathered strictly from spatiotemporally active foreground tokens (Mkin=1\mathcal{M}_{kin}=1) in Self-Attention.

2. Orthogonal Projection (Eq. 4): Before SVD, ΔHattn\Delta H_{attn} is projected onto the orthogonal complement of VmlpV_{mlp}:

ΔH^attn=\Delta \hat{H}_{attn} = ΔHattn−\Delta H_{attn} - (ΔHattn(\Delta H_{attn} Vmlp)V_{mlp}) Vmlp⊤V_{mlp}^\top

This strips out prompt-aligned expectations, isolating the visual residual (counterfactual innovation).

3. Functional Disentanglement in the Residual Stream: Because sub-layer updates act as linear vector additions, enforcing strict mutual orthogonality (Wattn⊤Vmlp=0W_{attn}^\top V_{mlp} = 0) guarantees that:

  • Non-Destructive Suppression (Eq. 6): Suppressing the MLP prior via −αVmlpVmlp⊤Δhmlp(l)-\alpha V_{mlp} V_{mlp}^\top \Delta h_{mlp}^{(l)} never erases the kinematic trajectory encoded in WattnW_{attn}.
  • Uncontaminated Injection (Eq. 7): Injecting the kinematic anchor via β(Wattn\beta (W_{attn} Wattn⊤W_{attn}^\top hvis∗)h_{vis}^{*}) has zero projection onto VmlpV_{mlp}, preventing injected visual energy from reinforcing textual bias.

We have updated Sections 4.2 and 4.3 in the revised paper accordingly. We thank the reviewer again for their constructive feedback and welcome any further questions during the rebuttal period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary Comment

Summary of Revisions

We sincerely thank all reviewers for their constructive feedback and insightful questions. We have uploaded our revised paper that incorporates all requested experiments, results, and theoretical clarifications. Below is a summary of it:

  • Table 1 (Baseline Transcription Correction):

    • Corrected a transcription error for the unmodified Qwen3-VL-8B baseline accuracy on VideoHallu-Physics (61.73%) to ensure full consistency with Table 2.
  • Section 4.1 (Objective Kinematic Anchoring):

    • Role of the Displacement Norm (Eq. 1): Clarified that the frame-to-frame Euclidean displacement norm serves strictly as a binary spatial threshold to construct Mkin\mathcal{M}_{kin} rather than a directionless feature embedding, allowing active tokens to retain their full dd-dimensional representations and time-ordered directional coordinates.
    • Preservation of Motion Semantics in hvis∗h_{vis}^{\ast} (Eq. 2): Clarified that Layer-0 tokens (ZvisZ_{vis}) are generated by 3D spatiotemporal patch tubelets that already encode local motion gradients, and explained how masked foreground pooling coupled with downstream subspace projection (WattnWattn⊤W_{attn} W_{attn}^\top) avoids background dilution and strips out static appearance.
  • Section 4.2 (Targeted Subspace Extraction):

    • Constructed Disentanglement vs. Native Entanglement: Explicitly noted that visual motion cues and linguistic priors are natively entangled in the shared multimodal space, and detailed how TSE constructs mutually orthogonal subspaces (Wattn⊤Vmlp=0W_{attn}^\top V_{mlp} = 0) by separating visual and textual sources prior to orthogonal projection.
    • Three-Stage Filtration Prior to SVD (Eq. 4–5): Detailed how foreground masking (Mkin=1\mathcal{M}_{kin}=1) removes static background clutter, orthogonal projection against VmlpV_{mlp} (Eq. 4) strips out prompt-aligned semantic shortcuts to isolate the counterfactual visual residual, and temporal mean-centering (Eq. 5) cancels out time-invariant object identity so that WattnW_{attn} strictly spans directional spatiotemporal state transitions.
  • Section 4.3 (Orthogonal Dual-stream Rectification):

    • Functional Guarantees of Mutual Orthogonality (Eq. 6–7): Clarified how Wattn⊤Vmlp=0W_{attn}^\top V_{mlp} = 0 ensures two non-interfering guarantees in the additive residual stream: soft prior suppression (α<1\alpha < 1) attenuates the dominant hallucination direction while preserving syntactic fluency and leaving WattnW_{attn} intact, and kinematic injection (βWattnWattn⊤hvis∗\beta W_{attn} W_{attn}^\top h_{vis}^{\ast}) restores visual energy with zero projection onto VmlpV_{mlp}.
  • Section 5.2 (Quantitative Results — New Baselines & Multi-Scale Generalization):

    • Comparison with Existing Interventions (New Table 3): Added quantitative comparisons against inference-time activation steering (ITI), offline MLP weight orthogonalization (Nullu), and contrastive decoding methods (VCD, TriCD) on Video-R1 across VideoHallu-Physics and Real-Reversed.
    • Generalization Across Model Scales (New Table 4): Expanded evaluation to smaller (Qwen3-VL-2B-Thinking, InternVL3.5-4B) and larger (InternVL3.5-14B-Thinking, Qwen3-VL-32B-Instruct) backbones, confirming that deep MLP prior dominance and VERTEX's effectiveness hold across the 2B–32B parameter range.
  • Section 5.4 (Ablation Studies):

    • Role of Masked Pooling and Kinematic Injection: Expanded the analysis of w/o OKA and w/o Injection (β=0\beta=0) to highlight how localized kinematic pooling prevents background clutter dilution and supplies essential visual energy in deep layers.
    • Nuanced Analysis Across Reasoning Regimes (Appendix G): Moderated component necessity claims by discussing how, on scene-level physical verification with Chain-of-Thought (<think>) models (Qwen3-VL-Thinking), softer interventions (w/o OKA or calibration-averaged Global TSE) can better preserve valid multi-step logical deduction pathways, while full VERTEX remains the optimal strictly zero-shot configuration.
  • Appendix H (Option Distribution Analysis & Per-Option Accuracy Breakdown on Extreme Low-Accuracy Cases):

    • Added a fine-grained option distribution and per-option accuracy breakdown analyzing why unmodified baselines exhibit accuracies far below random chance on temporal reversal benchmarks.
    • Demonstrated that near-zero accuracies stem from two coupled failure modes rather than random guessing: systematic attraction to canonical forward-time distractors when visual tokens match the object identity of a natural forward action, and parametric option-letter bias (mode collapse) in deep MLP layers under counterfactual visual conflict. Showed how VERTEX breaks this collapse and restores balanced, visually grounded option distributions.

We again deeply appreciate the reviewers for their valuable feedbacks, and are welcome to answer any additional questions or discussions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.