acceptodds
Under review as a conference paper at ICLR 2027

Temporal Evidence Heads: From Temporal Sensitivity to Answer Relevance in Video-MLLMs

Abstract

Video multimodal large language models (Video-MLLMs) can misjudge event order even when they recognize the video content. We find that heads that respond strongly to temporal reordering are not always the heads that most affect support for the correct answer. We propose Temporal Evidence Heads (TEH), a method that requires no parameter training. TEH first identifies temporally sensitive heads and then ranks them through bidirectional representation swapping between original and temporally reordered videos. The resulting answer-support measurements guide head selection. At inference, TEH adjusts the selected heads along a sample-specific direction computed from the current video’s original–counterfactual representation difference. Experiments on TempCompass and VidHalluc show that TEH improves temporal reasoning across multiple Video-MLLMs. Head-selection ablations show that answer-support changes help select more effective heads than temporal sensitivity alone, and a fixed-head construction ablation shows that TEH retains its benefit under an alternative temporal permutation.

Primary area:
interpretability and explainable AI
Keywords:
video multimodal large language models, temporal reasoning, temporal hallucination, attention heads, activation patching, inference-time intervention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2xYZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Temporal Evidence Heads, a training free method to actively detect temporally sensitive head and use them to boost the temporal awareness of the multimodal foundation model. TEH significantly improves the ordering awareness of the model and boosts the benchmark performance.

Critical strengths
  1. The proposed idea about finding temporal evidence head to boost the model's temporal reasoning is very interesting.

  2. The targeted problem, the lack of temporal order awareness of current multimodal foundation model is critical and understudied in the current literature.

  3. The evaluation protocol is well-designed. Discovery, validation and test are split by video instead of video clips. Besides, I appreciate that the authors also report the negative or tie results.

Critical weaknesses
  1. Lack of evaluation on more advanced models. The proposed method is only evaluated on qwen3, SmolVLM2 and VideoLLaMA3. Expand the evaluation to more advanced models, such as qwen 3.8 series, and Kimi series.

  2. The evaluation questions is very simple and narrow, basically the same type of the questions. Both benchmarks only ask about order, and the counterfactual is exactly a reversed order.

  3. The evaluation of general video QA is not included. It's not clear whether this improvement would lead to better performace in general video QA.

Questions for authors
  1. Why does LLaVA-NeXT-Video-7B use the negative offset?

  2. There are different performance on the Qwen3 on temp compass benchmark. Could you kindly clarify how these different numbers are derived, and which one should we consider mainly?

4 comments

Thank you for your positive assessment of the idea and the evaluation protocol, and for pointing out issues in model coverage, task scope, general QA and the consistency of the reported numbers. We reply point by point below.

To W1 (more advanced models): We added VideoLLaMA3-7B and Qwen3-VL-8B. Both are run with the full main procedure: separate discovery, validation and test splits, fp32 patching of every head, and a single run on the test set. They are also compared with all six other methods under the same protocol (Table 1 and Table S14 of the revision).

ModelTempCompassVidHalluc-TSHVinoground clip accuracyVinoground text score
VideoLLaMA3-7B77.16 → 85.78 (p=3.6e-5)89.67 → 96.67 (p=1.0e-4)68.71 → 79.57 (p=6.8e-10)39.43 → 62.86 (p=9.4e-18)
Qwen3-VL-8B81.47 → 84.48 (p=0.28)86.00 → 94.67 (p=2.6e-6)65.71 → 73.00 (p=1.6e-5)36.00 → 54.57 (p=7.7e-14)

Cells show Vanilla → TEH, with p from an exact McNemar test against Vanilla.

  • Five of the six model–benchmark pairs improve significantly, by 7.00–10.86 points; Qwen3-8B on TempCompass is +3.02 and not significant.
  • In all six settings, TEH has the highest accuracy among the seven methods of Table 1, and the same holds for the Vinoground text score.
  • We chose these two models because they belong to the same families as the 2B models in the paper, so the same method can be compared directly across scales.
  • This revision does not cover the Kimi series or newer Qwen releases; we list them as future extensions.

To W2 (narrow question type; the counterfactual is exactly reversal): We respond on two fronts: the evaluation data and the counterfactual construction.

(1) We added Vinoground, built by another group. It tests matching between videos and captions: each pair of clips comes with two captions that differ only in event order, the data contain no reversed videos, and the correct answers are not produced by reversal. TEH improves both metrics significantly for all five models (Table 1 and Table S14):

ModelClip accuracyText score
Qwen3-VL-2B58.43 → 69.00 (p=1.9e-6)21.71 → 51.43 (p=5.3e-23)
SmolVLM2-2.2B46.14 → 56.86 (p=3.0e-7)5.43 → 30.86 (p=2.4e-22)
VideoLLaMA3-2B60.71 → 75.00 (p=1.6e-14)25.71 → 56.86 (p=3.0e-27)
VideoLLaMA3-7B68.71 → 79.57 (p=6.8e-10)39.43 → 62.86 (p=9.4e-18)
Qwen3-VL-8B65.71 → 73.00 (p=1.6e-5)36.00 → 54.57 (p=7.7e-14)
  • Clip accuracy rises by 7.29–14.29 points and text score by 18.57–31.15 points.
  • TEH has the highest clip accuracy for four of the five models (Qwen3-2B, SmolVLM2, VideoLLaMA3-7B and Qwen3-8B); MotionCD is 0.29 points higher for VideoLLaMA3-2B.

(2) The method does not depend on exact reversal. With the selected heads and (K, η) fixed and only the counterfactual changed, full reversal, 2-, 4- and 8-frame block reversal, a swap at the largest visual change, and a block shuffle all give positive gains in the six settings (+5 to +17 points); static frames and noise give no consistent gain. The full table is in our reply to m9a3 (W3) and in Table S18.

To W3 (general video QA): TEH targets event-order judgments, and we do not present it as a general enhancement for video QA. To answer your question, we evaluated Qwen3-VL-2B on MVBench and Video-MME:

BenchmarkSetupResult
MVBench (n=3413)Selected on TempCompass, applied as is57.78 → 55.35 (−2.43, p=6.1e-5)
MVBenchRandom heads, same K and η57.95 (+0.18)
MVBench (n=2381)Heads re-selected on MVBench58.13 → 57.71 (−0.42, p=0.395)
Video-MME (n=252)Selected on TempCompass, applied as is51.19 → 49.60 (−1.59, p=0.52)
  • Applying the TempCompass configuration to MVBench as is costs 2.43 points, mostly on the non-temporal tasks (−3.07, against −0.90 on the temporal ones).
  • When heads are re-selected on MVBench, accuracy shows no significant difference from Vanilla; the change on Video-MME is also not significant.
  • The gains of TEH are therefore concentrated on event-order judgments; for other question types, heads should be re-selected on the target task.

To Q1 (why LLaVA-NeXT-Video-7B uses a negative offset): Thank you for raising this. The result of the former Appendix H was not produced by the main TEH procedure, so it is not evidence about TEH at the 7B scale, and it should not have been included in the paper. That run differed from the main procedure in two ways:

  • heads were ranked by a sensitivity-style score that favors heads reacting strongly on wrong answers, rather than by the answer-support change c;
  • the sign and strength of the intervention were chosen on the same 600 items, without separate validation and test splits.

The negative offset found there therefore cannot be used to judge the direction of the TEH update. The revision removes the former Appendix H and re-evaluates the 7B/8B scale with the main procedure (see the table under W1):

  • with the configurations selected by the main procedure, VideoLLaMA3-7B and Qwen3-VL-8B improve significantly in five of the six settings;
  • with a negative update instead (η=−1, K of 1 or 2), accuracy is significantly below Vanilla in all four TempCompass/VidHalluc-TSH settings (−2.33 to −6.90 points).

Under the main procedure, updating along the original–counterfactual difference of the current video is therefore effective, and the opposite update lowers accuracy. We did not re-run the main procedure on LLaVA-NeXT-Video-7B; we used VideoLLaMA3-7B instead because its 2B counterpart is already in the paper, which allows a direct comparison across scales.

To Q2: Thank you for pointing out that these numbers are easy to confuse. We checked the corresponding configurations and run records; the table below explains the source and meaning of each.

NumberWhere (original submission)What it is
78.02 (181/232)Table 1Main configuration (K=16, η=4)
77.59 (180/232)Ablation text (Section 4.4, Table S10; Table S11 in the revision)The same configuration (K=16, η=4) in a separate run made for the direction ablation; one item differs
78.45 (182/232)Table 4A different configuration, with K and η tuned independently (K=8, η=4)

The result of the main configuration in the original submission is 78.02. To explain the difference from the direction-ablation result, we checked the run files of the two runs. The one-item difference comes from bf16 precision: in the ablation run, the correct option and another option of that item had exactly the same bf16 logit, and the scoring protocol counts exact ties as incorrect. Re-running the ablation with the main procedure gives 78.02 (181/232), and the revision reports this value.

In the revision, Table 1 reports the configuration selected with all-head discovery for every model (74.14 on Qwen3-2B TempCompass), while the analyses in Sections 4.3–4.4 keep the original configuration (78.02). All of Table 1 is scored from bf16 logits; evaluated with this code, the original configuration also scores 78.02 (Table S15). Vanilla for SmolVLM2 and VideoLLaMA3 differs from the original submission by one or two items. Table S15 places the two configurations side by side for checking.

Please let us know if any concerns remain; we are happy to discuss them.

Thanks for the detailed rebuttal! It addresses my concerns and I would like to raise my score accordingly!

Thank you very much for your encouraging feedback and for raising your score! Your suggestions have made our paper much stronger. Thank you again for your support!

Reviewer_72JkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Temporal Evidence Heads (TEH), a training-free intervention method to improve temporal reasoning in VideoLLMs. Its key claim is that heads responding strongly to temporal reordering are not necessarily those that matter for the answer, so heads should be selected by a causal signal rather than by sensitivity alone. Accordingly, it first selects candidate heads whose representations change most under frame-block reordering (measured at the answer position, before the attention output projection), and then re-ranks them by activation patching between the original and reordered videos, i.e., by how much swapping each head's representation changes the probability of the correct answer (answer relevance, denoted c). During inference, the representations of only these selected heads are shifted along the per-sample difference between the original and reordered representations. The method is validated on the Event-Order subset of TempCompass and the TSH subset of VidHalluc with three 2B-scale models; with heads selected on each benchmark's own discovery split (i.e., in-domain), it improves over Vanilla by 11–19 points and outperforms sensitivity-only head selection in most settings.

Critical strengths

Causally selected heads steered with a per-sample direction. Rather than relying only on how strongly heads respond to temporal perturbations, TEH ranks candidate heads with a more direct causal signal, i.e., how much swapping each head's representation between the original and reordered videos changes the probability of the correct answer, and then steers only these heads along a direction computed from the current sample and its temporal counterfactual. This places TEH between contrastive decoding (per-sample contrast, but on output logits) and representation steering (internal representations, but a fixed vector). The reported controls support both components: in matched comparisons, support-based selection generally outperforms sensitivity-only selection (Tables 3–4), and a fixed mean direction (82.3%) or random heads (82.4%) give no gain over Vanilla (82.0%), while TEH reaches 94.3% (SmolVLM2 on VidHalluc; Appendix J, Fig. 4).

Critical weaknesses
  1. Fragility of the head-selection signal.

    • The selection rules differ across settings, e.g., how a stability check on c is used (ranking priority, filter, or only a marker; Table S1) and the number of samples used to estimate c (32 for Qwen3 vs. 100–200 otherwise), and the 7B experiment uses a different heuristic with the opposite sign (Appendix H). This seems to stem from c itself being fragile: when computed on two halves of the discovery data, its head rankings are essentially uncorrelated for Qwen3 and SmolVLM2 on TempCompass (ρ_c ≈ 0; Table S7), which are also the settings with the largest gains over sensitivity-only selection (Table 3). As a result, it is unclear how to apply TEH to a new model.
    • The multi-stage filtering (sensitivity pre-filtering followed by patching) seems necessary mainly because activation patching is expensive. However, pre-filtering by sensitivity may exclude heads with high answer relevance but low sensitivity, contrary to the paper's own claim. Gradient-based approximations of activation patching, such as AtP* [1], could estimate c for all heads with a few forward and backward passes, which would remove the need for the pre-filter and allow more samples for estimating c.
  2. Limited evidence of generalization.

    • Tasks. The gains are shown only on two event-order subsets (TempCompass Event-Order, 232 questions; VidHalluc-TSH, 300 questions), where reordering the video typically flips the correct answer, so the per-sample contrast may directly encode the answer. It remains unverified whether TEH also improves general temporal reasoning benchmarks (e.g., MVBench, Video-MME) or causes degradation on other tasks.
    • Evaluation protocol. Activation-steering methods typically derive activations and search hyperparameters on a single set and then evaluate on other downstream tasks. Here, heads are instead selected on each benchmark's own labeled discovery split, while the steering baseline TA-AE derives its heads and directions from a separate dataset (Appendix G.2), which may favor TEH in Table 1. Notably, when transferred from TempCompass to VidHalluc, TEH fails for SmolVLM2 (−0.67; Table S11).
    • Model scale. The main experiments use only 2B-scale models, and the 7B experiment (Appendix H) does not use the proposed procedure, so the method itself remains to be validated at the 7B scale.

[1] J. Kramár et al. AtP*: An Efficient and Scalable Method for Localizing LLM Behaviour to Components. arXiv:2403.00745, 2024.

Questions for authors
  1. If a single selection rule is applied to all settings, with more samples for estimating c or with c estimated for all heads without the sensitivity pre-filter (e.g., using AtP* [1]), does c become stable across discovery halves on TempCompass, and do the gains over sensitivity-only selection still hold?
  2. When heads and hyperparameters are selected on one dataset and applied without reselection to other benchmarks, including general temporal reasoning benchmarks (e.g., MVBench, Video-MME) and non-temporal ones, does TEH still improve over Vanilla and over baselines evaluated under the same protocol (e.g., TA-AE), and does it degrade any task?
  3. What are the results at the 7B scale when using the main TEH procedure with separate discovery, validation, and test splits, rather than the different selection procedure and opposite sign used in Appendix H?
4 comments

Thank you for your comments. We unified the head-selection procedure for the main results and added stability and cross-benchmark experiments, as detailed below.

To W1 / Q1: To test whether the results depend on setting-specific rules, Table 1 uses one rule for every model and benchmark:

  • every head is patched during discovery;
  • c is computed in fp32 from the change in log-odds;
  • all models use the same discovery split (Qwen3 no longer uses 32 samples);
  • heads are ranked by c alone, and (K, η) is chosen on validation.

The method section of the revision states this rule explicitly (Eq. 7 and Section 3.3). The procedure no longer depends on setting-specific filters or markers, and applying it to a new model follows the same steps. The stability criteria of the original component analyses are described in Appendix A.

The near-zero split-half values in the original Table S7 come mainly from how c was estimated:

  • effects were measured on probabilities;
  • patching ran in bf16, and 42–51% of the effects on TempCompass were exactly zero;
  • Qwen3 used only 32 samples.

On the same data, switching the measure to log-odds alone raises the split-half correlation of Qwen3 from 0.031 to 0.602. With the configuration of Table 1, the split-half correlations of c are:

SettingAll headsTop-32 poolTop-8 overlap (Jaccard)
Qwen3-2B / SmolVLM2 / VideoLLaMA3-2B TC.504 / .327 / .535.969 / .621 / .940.60 / .60 / .778
Qwen3-2B / SmolVLM2 / VideoLLaMA3-2B VH.880 / .834 / .863.992 / .999 / .9981.00 / 1.00 / 1.00
VideoLLaMA3-7B TC / VH.449 / .827.918 / .998.455 / .778
Qwen3-8B TC / VH.516 / .873.965 / .997.778 / .778
  • Within the candidate pool, the split-half correlation is above .99 on VidHalluc and .62–.97 on TempCompass.
  • Over all heads it is lower on TempCompass (.33–.54), because most heads have near-zero effects.
  • The top-8 sets of the 2B models are identical across halves on VidHalluc. On TempCompass the top-8 sets overlap partially (Jaccard .46–.78 across the five models).

This table now replaces Table S7 in Appendix D.

With this rule, we compared c with sensitivity-only selection in all 15 settings:

  • in no setting is selection by c significantly worse than sensitivity-only selection;
  • c is significantly better in 4 settings (2 after Holm correction), all at 2B, and by 13–14 points in the two largest cases;
  • at 7B/8B the two rules do not differ significantly (Table S16).

On AtP*: we now patch every head exactly, which is the quantity AtP* approximates. Appendix K cites AtP* as a cheaper alternative for larger models. To W2 / Q2: We answer in three parts: transfer across benchmarks, the effect on other tasks, and the source of the contrast direction.

(1) Performance without in-domain selection.

First, we froze the configurations completely and applied them to Vinoground with no re-selection:

  • the paper's VidHalluc configurations give +13.14 clip accuracy for VideoLLaMA3-2B and +6.71 for SmolVLM2;
  • TempCompass configurations give Qwen3-2B and Qwen3-8B +4.29 and +6.00 on text score, with little change in clip accuracy. Second, we ranked heads on another benchmark and chose only K and η on the target validation set. TA-AE in Table 1 also selects its locations on external data (ShareGPT4Video, Appendix G.2) and tunes on target validation data, so the table below reports the two side by side for reference (Table S19):
SettingTA-AE (Table 1)TEH with transferred headsVanilla (Sections 4.3–4.4 / Table 1)
Qwen3-2B TempCompass60.7873.2859.48 / 59.48
SmolVLM2 TempCompass55.6056.9055.17 / 56.03
VideoLLaMA3-2B TempCompass53.4559.9153.45 / 53.88
Qwen3-2B VidHalluc80.6793.6782.00 / 82.00
SmolVLM2 VidHalluc83.3383.0082.00 / 82.67
VideoLLaMA3-2B VidHalluc82.3392.0082.67 / 82.33
  • TEH is 6 to 13 points higher in four of the six settings.
  • On SmolVLM2 the two are close and transfer brings no gain, consistent with Table S11 of the original submission that you point out (Table S12 in the revision).
  • The comparison is unpaired, and TEH's heads come from another event-order benchmark, so we report it as a reference for transfer.
  • For the larger models, transferred heads give VideoLLaMA3-7B +7.33 on TempCompass and +7.67 on VidHalluc.
  • In all six settings of this table, in-domain selection (Table 1) is higher than the transferred heads, so we recommend selecting heads on the target task.

(2) As the title, Temporal Evidence Heads: From Temporal Sensitivity to Answer Relevance in Video-MLLMs, indicates, our target is how video models use temporal evidence to judge the order of events; TempCompass and VidHalluc-TSH test this ability through question answering. The revision adds Vinoground, which tests the same question through matching natural videos with captions. On this benchmark, TEH raises clip accuracy by 7.29–14.29 points and text score by 18.57–31.15 points for all five models (Table 1, Table S14), and its clip accuracy is the highest among all methods for four of the five models. This shows that the gains TEH obtains on event-order question answering also appear in video–caption matching. Results on general video QA (MVBench, Video-MME) are in our reply to 2xYZ (W3).

(3) Does the contrast directly encode the answer? Here it is important to separate the labels used at discovery time from the source of the direction at inference time. In these questions, reordering the events usually changes which option is correct. This is the property TEH relies on, and it is why the claims concern event-order judgments. At inference, the contrast direction is built only from the input video and the model's own representations:

  • the counterfactual is built from the video alone, and no label is used at test time;
  • the contrast only helps if the model's own representations respond to order in the right direction;
  • static frames and noise give no consistent gain;
  • a negative update with η=−1 gives significantly lower accuracy than Vanilla in all four 7B/8B TempCompass/VidHalluc settings;
  • the gains come mainly from items where reversal does not change the model's output (+17.3 to +18.6 points on TempCompass).

To Q3: Following your suggestion on model scale, we evaluated VideoLLaMA3-7B and Qwen3-VL-8B with the main procedure: separate discovery, validation and test splits, all-head fp32 patching, and a single test run.

ModelTempCompassVidHalluc-TSHVinoground clip accuracy
VideoLLaMA3-7B77.16 → 85.78 (+8.62, p=3.6e-5)89.67 → 96.67 (+7.00, p=1.0e-4)68.71 → 79.57 (+10.86, p=6.8e-10)
Qwen3-VL-8B81.47 → 84.48 (+3.02, p=0.28)86.00 → 94.67 (+8.67, p=2.6e-6)65.71 → 73.00 (+7.29, p=1.6e-5)
  • Five of the six pairs are significant.
  • A negative update with η=−1 (K of 1 or 2) gives significantly lower accuracy than Vanilla in all four TempCompass/VidHalluc settings (−2.33 to −6.90).
  • Both models are compared with all six other methods in Table 1.

The revision removes the exploratory run of the former Appendix H.

Please let us know if any concerns remain; we are happy to discuss them.

I thank the authors for the extensive experiments. My original concerns were mostly about the inconsistent setups and the limited evaluation (benchmarks and models), which are largely addressed in the rebuttal and the updated paper. I will therefore raise my score from 2 to 3.

Thank you sincerely for your careful review and for engaging with our rebuttal. Your comments on consistent setups and broader evaluation have made the paper clearer and more convincing. We truly appreciate it.

Reviewer_m9a3OpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This work proposes a two-stage localization method: inexpensive input-sensitivity screening identifies candidate attention heads, and activation patching identifies the key heads with strong causal effects on the answer (TEH). It then applies sample-specific steering to these heads. The experiments demonstrate the effectiveness of this design.

Critical strengths
  1. In my view, coarse screening followed by causal-tracing-based reranking is a practical approach that may preserve localization quality while reducing computational cost. To my knowledge, it also offers novelty in localization and intervention for video language models. The experiments provide preliminary evidence of its effectiveness.
Critical weaknesses
  1. Input sensitivity measures changes in head representations induced by a particular input perturbation; it does not directly measure a head's influence on the correct answer. It is therefore unsurprising that heads with larger activation changes are not necessarily those that most affect the correct answer.

  2. The authors describe localization as a two-stage design, but in my view, the first stage primarily serves as a prefilter to avoid the cost of activation patching across all heads. If the authors consider this stage necessary for improving localization, beyond saving computation, they should provide evidence that the two-stage method outperforms activation patching across all heads. I also suggest analyzing how candidate-pool size affects localization performance.

  3. The counterfactual intervention based on reordering two-frame blocks is insufficiently motivated. Although it preserves within-block order, it remains a simple rule-based perturbation rather than a precise semantic counterfactual based on event boundaries. The limited ablation in Appendix C also provides insufficient justification for this design.

  4. MotionCD, TCD, and VCD already use differences between the current sample's original and perturbed branches for output-level contrastive adjustment. Sample-specific contrast is therefore not a new idea; this paper's specific distinction is applying it to the internal representations of selected attention heads. The authors need to clarify the added value of this shift relative to output-level contrast.

Questions for authors

None.

5 comments

Thank you for your comments. Below we address head selection, the counterfactual construction and the comparison with baselines.

To W1: We agree that temporal sensitivity alone does not show that a head is relevant to the answer. To address this, we compared the two rankings directly and tested whether the difference affects final accuracy.

  • Across all heads of the 2B models, sensitivity and c correlate at .21–.24 on TempCompass and .01–.07 on VidHalluc.
  • We compared the two rules in all 15 settings, each with (K, η) chosen on validation. Selecting by c is higher in 11, equal in 2, and lower in 2 (by at most 0.43 points); in one of the two equal settings, both rules select the same heads. It is significantly better in 4 settings (2 after Holm correction) and never significantly worse. All four are on 2B models; at 7B/8B the two rules do not differ significantly.
  • The two largest gains occur where the selected heads overlap least:
    • Qwen3-2B TempCompass: only 1 of the 4 c-selected heads is also selected by sensitivity, and c is 13.36 points higher.
    • SmolVLM2 TempCompass: none of the 8 heads is shared, and c is 14.22 points higher.

Section 4.3 of the revision reports this result (Table S16).

To W2: Following your suggestion, Table 1 patches every head during discovery, and we compared the candidate pool with all-head discovery on the original six settings:

SettingTop-32 poolAll headsAll − pool (p)
Qwen3-2B TC75.8674.14−1.72 (0.42)
SmolVLM2 TC66.3869.40+3.02 (0.21)
VideoLLaMA3-2B TC69.8369.830
Qwen3-2B VH94.6796.33+1.67 (0.063)
SmolVLM2 VH94.6794.670
VideoLLaMA3-2B VH95.0095.000
  • On these six settings, the Top-32 pool is not significantly different from all heads. Across the five models it needs 2.8–9.5% of the all-head patching runs.
  • Pools of 64 or 128 heads give the same accuracy as all heads.
  • A smaller pool does not always keep the full gain. On SmolVLM2 Vinoground, pools of 16 / 32 / 64 give 49.57 / 51.71 / 52.29 clip accuracy, against 56.86 with all heads.

We therefore treat the sensitivity screening of the first stage as an optional speed-up of discovery (Table S17).

To W3: To test how much the results depend on the counterfactual construction, we kept the selected heads and (K, η) fixed and changed only the counterfactual. We compared eight constructions on six settings, including one that cuts at the largest visual change as a proxy for an event boundary. The table shows the TEH gain over Vanilla, in points:

CounterfactualQwen3-TCSmol-TCVL3-TCQwen3-VHSmol-VHVL3-VH
2-frame block reversal (paper)+14.66+13.36+15.95+14.33+12.00+12.67
Full reversal+15.09+14.22+16.38+13.33+12.33+12.67
4-frame blocks+15.09+14.22+14.66+14.67+12.67+12.00
8-frame blocks+13.36+9.05+17.24+11.33+13.33+10.33
Swap at largest visual change+15.52+12.93+15.95+7.33+10.67+5.00
Block shuffle+10.78+7.33+13.36+9.67+8.67+6.00
Static frame−9.05−2.16−0.86+1.00+5.33+1.33
Noise (VCD-style)−2.59−2.59−0.86−8.00+2.670.00
  • Every tested construction that reorders events gives a positive gain in all six settings (+5 to +17). Static frames and noise give no consistent gain.
  • Blocks of 2, 4 and 8 frames all keep positive gains (+9.05 to +17.24).
  • Cutting at the largest visual change shows no consistent advantage across settings.
  • Full reversal is on par with the default: higher in four settings, equal in one and lower in one.

Based on these results, we keep block-pair reversal as a simple default that needs no extra annotation (Table S18).

To W4: Following your suggestion on the scope of the comparison, we evaluated every baseline on the two larger models and on Vinoground. Table 1 of the revision now reports all seven methods in all fifteen settings. VCD, MotionCD, TA-AE and SEASON are selected on validation data with the search spaces of Appendix G; DINO-HEAL keeps the fixed fusion weights of its public implementation. The comparison with VCD and MotionCD on the nine new settings:

SettingVanillaVCDMotionCDTEH
VideoLLaMA3-7B TempCompass77.1678.8883.1985.78
VideoLLaMA3-7B VidHalluc89.6789.3395.6796.67
Qwen3-8B TempCompass81.4779.7483.1984.48
Qwen3-8B VidHalluc86.0086.0089.3394.67
Qwen3-2B Vinoground58.43 / 21.7157.00 / 22.2963.00 / 36.8669.00 / 51.43
SmolVLM2 Vinoground46.14 / 5.4353.43 / 13.4353.14 / 32.8656.86 / 30.86
VideoLLaMA3-2B Vinoground60.71 / 25.7161.71 / 29.1475.29 / 59.1475.00 / 56.86
VideoLLaMA3-7B Vinoground68.71 / 39.4371.29 / 44.2978.14 / 59.7179.57 / 62.86
Qwen3-8B Vinoground65.71 / 36.0062.00 / 31.7164.14 / 36.8673.00 / 54.57

Vinoground cells show clip accuracy / text score.

  • Against MotionCD, the strongest baseline:
    • TEH is higher in 12 of the 15 settings. MotionCD is higher on TempCompass for SmolVLM2 and VideoLLaMA3-2B, and on Vinoground for VideoLLaMA3-2B.
    • TEH is significantly better on Qwen3-8B VidHalluc (+5.33, p=4.0e-4).
    • TEH is significantly better on Qwen3-2B Vinoground (+6.00 clip accuracy, p=0.0018; +14.57 text score, p=2.0e-7).
    • In Vinoground text score, MotionCD is higher for SmolVLM2 (+2.00) and VideoLLaMA3-2B (+2.28).
    • On Qwen3-8B Vinoground, MotionCD scores below Vanilla (64.14 vs. 65.71), while TEH reaches 73.00.

Why head-level intervention, and how it relates to output-level contrast. The paper addresses two questions: which heads that respond to temporal reordering actually affect answer support, and how to use these locations for an input-dependent intervention. Comparing output scores alone cannot separate the roles of different internal locations. We therefore use bidirectional activation patching to measure the answer-relevant effect of each head, and build the update direction from the difference between the original and counterfactual representations of the current video. The first separates temporal sensitivity from answer relevance; the second avoids applying one fixed offset to every video.

To further separate the counterfactual from the place where the contrast is applied, we added an OutCD control. It applies the contrast to the candidate-answer logits, z = (1+α)·z_orig − α·z_cf, using the same block-pair counterfactual as TEH; α is chosen on validation data and the test set is run once. Main metric (clip accuracy for Vinoground; TC: TempCompass, VH: VidHalluc-TSH, VG: Vinoground), Table S20 of the revision:

SettingVanillaOutCDTEHTEH − OutCD (p)
Qwen3-2B TC59.4875.8674.14−1.72 (0.388)
Qwen3-2B VH82.0095.3396.33+1.00 (0.508)
Qwen3-2B VG58.4369.8669.00−0.86 (0.55)
SmolVLM2 TC56.0370.6969.40−1.29 (0.701)
SmolVLM2 VH82.6793.3394.67+1.33 (0.344)
SmolVLM2 VG46.1452.5756.86+4.29 (0.054)
VideoLLaMA3-2B TC53.8867.6769.83+2.16 (0.227)
VideoLLaMA3-2B VH82.3395.0095.000.00 (1.00)
VideoLLaMA3-2B VG60.7173.1475.00+1.86 (0.079)
VideoLLaMA3-7B TC77.1685.3485.78+0.43 (1.00)
VideoLLaMA3-7B VH89.6797.0096.67−0.33 (1.00)
VideoLLaMA3-7B VG68.7181.2979.57−1.71 (0.275)
Qwen3-8B TC81.4784.0584.48+0.43 (1.00)
Qwen3-8B VH86.0094.3394.67+0.33 (1.00)
Qwen3-8B VG65.7171.2973.00+1.71 (0.57)

With the same counterfactual, OutCD and TEH reach similar accuracy: they differ by at most 4.29 points, none of the 15 paired differences is significant, and TEH is higher in 9 settings, lower in 5, and equal in 1. OutCD is also above MotionCD in 11 of the 15 settings (TEH: 12). These results support the effectiveness of the temporal counterfactual contrast.

The contribution of TEH is to ground this contrast in internal locations that can be measured and intervened on. We intervene on attention heads to answer two questions: which heads' responses to event order affect the model's judgment of the answer, and how to adjust these heads with information from the current video. To this end, TEH first measures each head's effect on answer support with bidirectional patching, and then sets the update direction from the representation difference between the current video and its temporal counterfactual.

In the 15 settings where both selection rules are tuned separately, selecting heads by the answer-support change c gives higher accuracy in 11; 4 of these differences are significant, and 2 remain significant after Holm correction (Table S16). The main experiments modify only 2–16 heads per setting. In the component analysis of SmolVLM2 on VidHalluc-TSH, for example, TEH reaches 94.33%, the same number of random heads averages 82.43%, and the base model scores 82.00%.

Section 4.4 further compares update directions. In the tested settings, the direction computed from the current video is more effective than a fixed direction averaged across videos. In the four TempCompass/VidHalluc-TSH settings of the 7B and 8B models, updating in the opposite direction lowers accuracy by 2.33–6.90 points relative to the base model. These experiments test two design choices that a head-level intervention makes explicit: where to update, and in which direction.

Please let us know if any concerns remain; we are happy to discuss them.

Comments on the paper

2 comments

General Response to All Reviewers

We sincerely thank all reviewers for their careful evaluations and constructive feedback. We have uploaded a revised manuscript, with additions and modifications highlighted in blue, and provided detailed point-by-point responses to each reviewer.

In response to the reviewers’ suggestions, we have substantially expanded our experiments and further strengthened the methodology and evaluation protocol. The main updates are as follows:

  1. More comprehensive evaluation and baseline comparisons. We expanded our evaluation to five Video-MLLMs (ranging from 2B to 8B parameters) and three event-order reasoning benchmarks, including the newly added Vinoground. We conducted a unified comparison of seven methods across all 15 model–benchmark settings. TEH improves over the base model in all 15 settings, with statistically significant gains in 14. Among the seven methods, TEH achieves the highest accuracy in 12 of the 15 settings.

  2. More extensive analyses of the proposed method. We added experiments on attention-head selection, selection stability, candidate-pool size, eight counterfactual constructions, cross-benchmark transfer, and a matched output-level contrast baseline (OutCD). These results provide further evidence for the effectiveness of answer-support-based head selection and offer more direct empirical support for sparse intervention at selected internal attention heads.

  3. A more consistent and transparent evaluation protocol. We unified the head-selection procedure across all models and benchmarks, using bidirectional activation patching over all heads, fp32 log-odds changes to measure answer support, and independent validation splits for selecting intervention parameters. We also clarified the implementation details and hyperparameter selection procedures for the baseline methods to improve the fairness, transparency, and reproducibility of our comparisons.

We believe these additional experiments and analyses substantially strengthen the empirical foundation of our work and provide clearer evidence of TEH’s effectiveness in identifying temporally relevant internal representations and improving event-order reasoning.

We sincerely appreciate the reviewers’ valuable suggestions and welcome further discussion.

AuthorsOpenReview ↗

Summary of Key Findings and Contributions

Dear Area Chair and Reviewers,

Thank you for your constructive feedback. We summarize the key findings and contributions supported by our revised manuscript.

Key Findings

  1. Temporal sensitivity does not imply answer relevance. Across all heads of the 2B models, the rank correlation between temporal sensitivity and answer-support change is only 0.21–0.24 on TempCompass and 0.01–0.07 on VidHalluc-TSH. With independently tuned configurations, answer-support-based selection is higher than sensitivity-only selection in 11/15 settings, tied in 2, and lower in 2 (by at most 0.43 points), with no significant decreases (Table S16). The largest gains occur when selected heads differ: +13.36 points on Qwen3-VL-2B TempCompass (1/4 shared heads) and +14.22 on SmolVLM2 TempCompass (0/8 shared).

  2. Sparse head intervention achieves performance comparable to output-level contrast. Steering only 2–16 selected heads yields accuracy comparable to output-level contrast using the same counterfactual; none of the 15 paired differences is statistically significant (Table S20). In the SmolVLM2–VidHalluc component analysis, matched random heads reach 82.43% (30-seed mean), close to Vanilla (82.00%), while TEH reaches 94.33% (Fig. 4). Reversing the direction reduces accuracy below Vanilla by 2.33–6.90 points in four tested 7B/8B settings. These controls support the relevance of the selected locations and directions.

  3. Intervention directions should be input-specific. On SmolVLM2–VidHalluc, sample-specific LocalContrast reaches 94.33%, compared with 82.33% for a cross-sample fixed-mean direction (Fig. 4). On Qwen3-VL-2B TempCompass, cached sample-specific intervention achieves 74.14%, versus 57.33% for a fixed mean at matched strength; online LocalContrast reaches 78.02% (Table S11). Thus, head locations can be shared while directions adapt to each video.

  4. Temporal reordering matters more than generic perturbation. All six tested event-reordering constructions improve accuracy across six settings (+5.00 to +17.24 points), including full reversal, varied block sizes, block shuffling, and a swap at the largest visual change. Static-frame and noise perturbations show no consistent improvement (Table S18).

  5. Gains extend to video–caption matching. On Vinoground, some base models score at or below the nominal 25% chance level in pairwise text score. TEH improves the text score of all five models by 18.57–31.15 points, reaching 30.86–62.86%. The gains thus extend beyond event-order QA to an independently constructed video–caption benchmark (Table 1).

Core Contributions

  1. Answer-relevance-based localization. We distinguish temporal sensitivity from answer relevance and use bidirectional activation patching to rank heads by their effect on correct-answer support.

  2. Input-specific, training-free intervention. TEH adjusts a small set of heads using each video's original–counterfactual representation difference, without updating model parameters. Selected heads and intervention strength are fixed after validation.

  3. Unified evaluation and controls. We evaluate five Video-MLLMs (2B–8B) on three event-order benchmarks under separate discovery, validation, and test splits, comparing with Vanilla and five inference-time baselines across 15 settings. TEH improves on Vanilla in all 15 and achieves the highest accuracy in 12 (using Vinoground clip accuracy). Controls address head selection, direction, counterfactual construction, and output-level contrast.

Our claims focus on event-order reasoning in Video-MLLMs. We hope these results clarify TEH's effectiveness and distinctive contribution.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.