Temporal Evidence Heads: From Temporal Sensitivity to Answer Relevance in Video-MLLMs

Video multimodal large language models (Video-MLLMs) can misjudge event order even when they recognize the video content. We find that heads that respond strongly to temporal reordering are not always the heads that most affect support for the correct answer. We propose Temporal Evidence Heads (TEH), a method that requires no parameter training. TEH first identifies temporally sensitive heads and then ranks them through bidirectional representation swapping between original and temporally reordered videos. The resulting answer-support measurements guide head selection. At inference, TEH adjusts the selected heads along a sample-specific direction computed from the current video’s original–counterfactual representation difference. Experiments on TempCompass and VidHalluc show that TEH improves temporal reasoning across multiple Video-MLLMs. Head-selection ablations show that answer-support changes help select more effective heads than temporal sensitivity alone, and a fixed-head construction ablation shows that TEH retains its benefit under an alternative temporal permutation.