- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Temporal Evidence Heads, a training free method to actively detect temporally sensitive head and use them to boost the temporal awareness of the multimodal foundation model. TEH significantly improves the ordering awareness of the model and boosts the benchmark performance.
Critical strengths
-
The proposed idea about finding temporal evidence head to boost the model's temporal reasoning is very interesting.
-
The targeted problem, the lack of temporal order awareness of current multimodal foundation model is critical and understudied in the current literature.
-
The evaluation protocol is well-designed. Discovery, validation and test are split by video instead of video clips. Besides, I appreciate that the authors also report the negative or tie results.
Critical weaknesses
-
Lack of evaluation on more advanced models. The proposed method is only evaluated on qwen3, SmolVLM2 and VideoLLaMA3. Expand the evaluation to more advanced models, such as qwen 3.8 series, and Kimi series.
-
The evaluation questions is very simple and narrow, basically the same type of the questions. Both benchmarks only ask about order, and the counterfactual is exactly a reversed order.
-
The evaluation of general video QA is not included. It's not clear whether this improvement would lead to better performace in general video QA.
Questions for authors
-
Why does LLaVA-NeXT-Video-7B use the negative offset?
-
There are different performance on the Qwen3 on temp compass benchmark. Could you kindly clarify how these different numbers are derived, and which one should we consider mainly?
4 comments
To Q1 (why LLaVA-NeXT-Video-7B uses a negative offset): Thank you for raising this. The result of the former Appendix H was not produced by the main TEH procedure, so it is not evidence about TEH at the 7B scale, and it should not have been included in the paper. That run differed from the main procedure in two ways:
- heads were ranked by a sensitivity-style score that favors heads reacting strongly on wrong answers, rather than by the answer-support change c;
- the sign and strength of the intervention were chosen on the same 600 items, without separate validation and test splits.
The negative offset found there therefore cannot be used to judge the direction of the TEH update. The revision removes the former Appendix H and re-evaluates the 7B/8B scale with the main procedure (see the table under W1):
- with the configurations selected by the main procedure, VideoLLaMA3-7B and Qwen3-VL-8B improve significantly in five of the six settings;
- with a negative update instead (η=−1, K of 1 or 2), accuracy is significantly below Vanilla in all four TempCompass/VidHalluc-TSH settings (−2.33 to −6.90 points).
Under the main procedure, updating along the original–counterfactual difference of the current video is therefore effective, and the opposite update lowers accuracy. We did not re-run the main procedure on LLaVA-NeXT-Video-7B; we used VideoLLaMA3-7B instead because its 2B counterpart is already in the paper, which allows a direct comparison across scales.
To Q2: Thank you for pointing out that these numbers are easy to confuse. We checked the corresponding configurations and run records; the table below explains the source and meaning of each.
| Number | Where (original submission) | What it is |
|---|---|---|
| 78.02 (181/232) | Table 1 | Main configuration (K=16, η=4) |
| 77.59 (180/232) | Ablation text (Section 4.4, Table S10; Table S11 in the revision) | The same configuration (K=16, η=4) in a separate run made for the direction ablation; one item differs |
| 78.45 (182/232) | Table 4 | A different configuration, with K and η tuned independently (K=8, η=4) |
The result of the main configuration in the original submission is 78.02. To explain the difference from the direction-ablation result, we checked the run files of the two runs. The one-item difference comes from bf16 precision: in the ablation run, the correct option and another option of that item had exactly the same bf16 logit, and the scoring protocol counts exact ties as incorrect. Re-running the ablation with the main procedure gives 78.02 (181/232), and the revision reports this value.
In the revision, Table 1 reports the configuration selected with all-head discovery for every model (74.14 on Qwen3-2B TempCompass), while the analyses in Sections 4.3–4.4 keep the original configuration (78.02). All of Table 1 is scored from bf16 logits; evaluated with this code, the original configuration also scores 78.02 (Table S15). Vanilla for SmolVLM2 and VideoLLaMA3 differs from the original submission by one or two items. Table S15 places the two configurations side by side for checking.
Please let us know if any concerns remain; we are happy to discuss them.
Thanks for the detailed rebuttal! It addresses my concerns and I would like to raise my score accordingly!
Thank you very much for your encouraging feedback and for raising your score! Your suggestions have made our paper much stronger. Thank you again for your support!
Thank you for your positive assessment of the idea and the evaluation protocol, and for pointing out issues in model coverage, task scope, general QA and the consistency of the reported numbers. We reply point by point below.
To W1 (more advanced models): We added VideoLLaMA3-7B and Qwen3-VL-8B. Both are run with the full main procedure: separate discovery, validation and test splits, fp32 patching of every head, and a single run on the test set. They are also compared with all six other methods under the same protocol (Table 1 and Table S14 of the revision).
Cells show Vanilla → TEH, with p from an exact McNemar test against Vanilla.
To W2 (narrow question type; the counterfactual is exactly reversal): We respond on two fronts: the evaluation data and the counterfactual construction.
(1) We added Vinoground, built by another group. It tests matching between videos and captions: each pair of clips comes with two captions that differ only in event order, the data contain no reversed videos, and the correct answers are not produced by reversal. TEH improves both metrics significantly for all five models (Table 1 and Table S14):
(2) The method does not depend on exact reversal. With the selected heads and (K, η) fixed and only the counterfactual changed, full reversal, 2-, 4- and 8-frame block reversal, a swap at the largest visual change, and a block shuffle all give positive gains in the six settings (+5 to +17 points); static frames and noise give no consistent gain. The full table is in our reply to m9a3 (W3) and in Table S18.
To W3 (general video QA): TEH targets event-order judgments, and we do not present it as a general enhancement for video QA. To answer your question, we evaluated Qwen3-VL-2B on MVBench and Video-MME: