- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper targets emotional video captioning (EVC) and argues that existing methods exploit background "confounders" instead of the true emotion trigger. Two contributions:
-
EVC-CauseGround. This is a benchmark of 2,000 videos from EmVidCap-L and MAFW, split 1,500 train / 500 eval. Each video has a spatio-temporal "cause tube." The tubes are produced by:
- prompting a frontier model with a 30-frame grid plus the ground-truth emotion and caption to pick a temporal span, capped at 30% of frames;
- having the same MLLM draw a box on the peak-optical-flow frame;
- propagating that box with SAM-3.
Humans then watch videos with the tube blacked out. If they can still guess the emotion, the video is labeled "Causal-Redundant"; otherwise, it is "Causal-Faithful," which yields a 176-video subset. The paper also defines an ST-IoU metric.
-
Causal-EVC. This model combines:
- MCSL, a motion prior from adjacent-frame patch-feature dissimilarity, used to bias temporal and axial spatial attention, plus box/span regression heads;
- ISER, top-K sparse routing from masked visual features to GloVe+VAD emotion prototypes;
- a two-way contrastive loss L_cf, contrasting foreground-masked (V⊙M_c) against background-masked (V⊙(1−M_c)) features;
- a Qwen2.5-VL-7B decoder.
Critical strengths
- Shortcut reliance on scene context in affective video understanding is a real problem. Spatio-temporal grounding of emotion triggers would be a useful resource if done carefully.
- The human masking test, which asks whether annotators can still infer the emotion with the tube removed, is a sensible idea for diagnosing context leakage. It could be the most valuable part of the paper with better execution and reporting.
- The annotation pipeline is practical and cheap: an MLLM proposes the span and box, and a tracker densifies it. The paper also includes ablations of the main components and of top-K.
Critical weaknesses
A. Conceptual problems with the "causal" framing
-
"Cause" vs. "confounder" is defined by fiat, and the definition is often wrong. The paper treats the localized foreground action as the cause and everything else as a confounder. In much affective video, the context is the cause: a party causes surprise, a disaster scene causes fear, a funeral causes grief. MAFW is explicitly said to include natural disasters. Fig. 1(a) seems to label the party itself as the "contextual confounder" for a surprised man. The Causal-Redundant criterion (annotators can guess the emotion from the masked video) does not separate "background is a spurious correlate" from "background is a genuine cause." It also does not separate either from "the mask failed to remove the agent," discussed in A3 below. The theoretical framing needs a definition of cause that survives these cases.
-
Cause is conflated with expression. In the main case study (Fig. 6), the model is praised for attending to "the disgust expressions of the girl" rather than the ice cream. A facial expression is a manifestation of the emotion, not its cause; the cause is the bad taste. Rewarding the model for attending to expressions makes this a facial-expression saliency task, not emotion-cause reasoning. This is a core conceptual issue for a paper whose central claim is genuine emotion-cause attribution.
-
The counterfactual is not an intervention. The do-operator and "backdoor" language (App. B: severing "the backdoor path V→U→Y," anchoring to P(Y|do(C))) is not supported.
- The counterfactual video blacks out the tube only within [t_start, t_end], which is at most 30% of frames. The agent remains fully visible in the other 70%, so the "background-only" video often still shows the person whose emotion is being judged.
- At the model level, Ĉ = V⊙(1−M_c) masks patch tokens after the Qwen2.5-VL vision encoder. Those tokens have already mixed global context through self-attention, so zeroing them removes neither foreground information from "background" tokens nor the reverse.
- V→U→Y is not a backdoor path; a backdoor path enters the treatment. U as described is a co-occurring variable, not a confounder of C and Y.
- No structural causal model is stated in the main text, and no identification argument is given.
-
The information-theoretic "guarantee" (App. B) is not valid.
- Eq. 24 claims that minimizing a two-element softmax loss "tightly bounds" I(C;E) − I(U;E). No derivation is given, and with one negative the InfoNCE bound saturates at log 2.
- L_cf only compares the GT-class cosine for C vs. Ĉ (Eq. 20). It does not drive I(U;E) toward zero in any formal sense.
- L_cf also has a degenerate solution: grow M_c toward all-ones so that Ĉ ≈ 0 and z_fake becomes constant. The only counterweight is box/span supervision. Please report mask coverage statistics; claims like "absorb all emotion-relevant semantics into C" suggest this is happening.
-
Circularity between annotation and model. The ground-truth tubes are anchored on the peak-optical-flow frame, and the span is forced into a short burst (≤30%). The model's key inductive bias is a motion prior that boosts high-motion frames and regions. The grounding gains may largely reflect matching the annotation heuristic rather than emotion-cause reasoning. A motion-only baseline (argmax-motion span plus the moving region) on EVC-CauseGround is essential. The paper's own Table 3 shows that motion alone is a weak mask for captioning, but it never reports grounding numbers for that baseline.
-
The t-SNE argument is circular. Fig. 7 shows Ĉ spread out and C clustered by emotion. L_cf optimizes exactly that, so the figure confirms the loss was minimized. It does not show that predictions rely on true causes.
B. Benchmark validity
-
The headline evaluation is missing. The abstract and intro claim "significant advantages in the causal-faithfulness subset," and the 176-video subset is presented as the key contribution. No table reports any result on that subset, or a Faithful-vs-Redundant split. Table 2 and Fig. 2 report the full 500-video eval set only. This is the main evidence the paper promises, and it is absent.
-
Label provenance and family bias.
- Cause labels come from a GPT model that is given the GT emotion and caption, so the label is "what GPT thinks justifies this emotion."
- GPT-4o and Gemini are then evaluated zero-shot against these labels, while Causal-EVC is trained on 1,500 videos labeled the same way.
- The annotating model is called GPT-5.6 (main text) and GPT-5.5 (appendix), with a GPT-5 system card citation.
- Missing details: the number of human annotators, how many boxes and spans were corrected, inter-annotator agreement, pipeline-vs-human agreement, and the guessing protocol for the masking test (forced choice among how many classes, chance level, majority rule).
- It is unclear whether the 176 Faithful videos come from the 500 eval videos or from all 2,000. If the latter, about 91% of videos are "redundant," which undercuts the premise that context is mostly spurious.
-
Granularity. Videos average ~100 s, but only 30 frames are sampled, about 3.3 s apart. "Motion" between frames 3 s apart is mostly scene or shot change rather than the kinematic signal the method describes. ST-IoU is also computed at this coarse frame resolution. The metric also assumes a single cause per video. Eq. 1 switches between T_interest and T_intersect without defining either.
C. Experimental and reporting issues
-
Possible label leakage at inference. Sec. 3 says the model is "given the input video and emotion category." Eq. 19 computes logits against E_gt, "the prototype embedding of the ground-truth emotion labels." If the GT emotion, or anything derived from it, is available at test time, emotion accuracy (Acc_sw, Acc_c) and the hybrid metrics are invalid. Please state exactly what the model receives at inference and how the emotion word is chosen.
-
Where do tubes come from on EVC-MSVD / EVC-VE? All ablations (Tables 3–6) are on EVC-MSVD and include L_temp and L_spa. These require spatio-temporal tube annotations, which EVC-MSVD does not have. Was the model pretrained on EVC-CauseGround and transferred, or were tubes pseudo-labeled? This affects every number in Tables 1 and 3–6.
-
The implementation section does not match the described model.
- Qwen2.5-VL-7B is cited as Raffel et al. 2020 (T5).
- The vocabulary size is 32,128, which is T5's vocabulary; Qwen's is roughly 150k.
- D = 1408 matches EVA-ViT-g (BLIP-2), not the Qwen2.5-VL vision encoder.
- Input is 224×224 center crops, whereas Qwen2.5-VL uses native dynamic resolution.
- Maximum caption length is 15.
- Hyperparameters λ_e, λ_cls, and θ are listed but never appear in the method.
- GloVe is described as "Transformer-based."
This looks like configuration inherited from a T5-based predecessor. I could not determine what was actually trained.
-
The equations are dimensionally inconsistent.
- Eq. 7 adds a T×T diagonal to an N_w×(T·P) score matrix and claims an output in R^{T×P×D}, but N_w queries yield N_w×D.
- Eq. 9 has similar issues.
- Eq. 4 claims to remove camera motion by adding λ_m times the global mean to the per-patch score. Adding a constant does not remove global motion; subtracting it would. Under the per-frame max normalization in Eq. 5, the term is close to a no-op.
-
Unfair and selectively framed comparisons.
- The contributions cite "+17.3% mIoU over zero-shot Qwen2.5-VL-7B." That is 6.9 points absolute, against the untuned version of the model's own backbone, after training on 1,500 in-domain annotated videos. The necessary baseline is Qwen2.5-VL-7B LoRA-tuned on the same data with the same heads but without MCSL/ISER/L_cf. Fine-tuned grounding baselines (VTimeLLM, TimeSuite, etc.) on the 1,500 training videos are also needed.
- In Table 1, the "+15.7% BFS" is relative and measured against SAGML (71.5). The strongest baseline, MM-ECPE, is at 81.8 vs. 82.7. MM-ECPE also beats the proposed method on BLEU-4 (71.4 vs. 71.2).
- Relative and absolute gains are mixed throughout without labels.
- It is not explained how MM-ECPE, which produces no boxes, gets an mIoU in Table 2.
-
Statistical weight. The EVC-MSVD test set has 134 videos, EVC-CauseGround eval has 500, and the Faithful subset has 176. Many reported differences are around 1 point, with no seeds, variance, or significance tests. The top-K sensitivity is also extreme: K=1 drops Acc_sw from 94.2 to 49.6 and CIDEr by 54. That calls for explanation rather than a fault-tolerance narrative.
-
Human evaluation. It uses 20 raters, 50 pairs, and a 1–10 scale, with no agreement statistics, blinding, or significance tests. VideoBLIP appears in the human study and in the baseline list but not in Table 1, while SGN appears in Table 1 but not in the baseline list.
D. Related work and novelty
-
Missing causal video captioning and reasoning work.
- NarrativeBridge (Nadeem et al., ICLR 2025) introduced LLM-generated causal-temporal narrative captions for MSVD/MSRVTT and a Cause-Effect Network with separate cause and effect encoders. This is the closest prior work on causality-aware captioning and should be discussed and compared.
- MECD (Chen et al., NeurIPS 2024) frames multi-event causal discovery in video. It uses mask-based event prediction (a Granger-style test that compares predictions with premise events masked vs. unmasked) together with front-door adjustment and counterfactual inference. That is the same masking-as-counterfactual idea, executed with an actual causal-inference treatment.
The claim in App. A.2 to be "the first work to notice the visual causes of emotions" is also contradicted by the paper's own baseline MM-ECPE.
-
Missing background-debiasing work. Motion-guided foreground/background separation to stop video models from cheating on static backgrounds is well established. FAME (Ding et al., CVPR 2022) extracts the moving foreground from frame differences and pastes it onto other videos' backgrounds so contrastive learning cannot rely on the background. Related work includes SDN (Choi et al., NeurIPS 2019) and Background Erasing (Wang et al., CVPR 2021). The deconfounded and counterfactual VL literature (deconfounded captioning, counterfactual VQA, CSS) is also relevant. Against this background, MCSL plus L_cf amounts to a motion-weighted attention mask with a single-negative foreground/background contrast, and the novelty is modest. The emotion-reasoning MLLMs (e.g., Emotion-LLaMA, EMER/explainable multimodal emotion recognition) are also absent from Table 2.
-
Related work is in the appendix only. It cites several clearly unrelated works as multimodal emotion-cause extraction: radiology report generation, news captioning, and time-series anomaly detection.
Minor
- "Cofounder" → "confounder" (multiple places); "convinced metric"; "throwed"; "sicken bread."
- "Pixel-level motion correlation" is computed on ViT patch features, not pixels.
- Ethics, licensing, and release terms for MAFW (which includes movie clips) and EmVidCap-derived faces are not discussed. Release of the benchmark itself is not promised; only code is.
Sources:
Questions for authors
- At inference, does the model receive the GT emotion category or E_gt in any form?
- How were L_temp and L_spa supervised on EVC-MSVD for Tables 3–6?
- Can you report Table 2 on the 176 Faithful videos and on the Redundant videos separately? Can you include a fine-tuned Qwen2.5-VL baseline and a motion-only grounding baseline?
- What is the average mask coverage of M_c, and how does it change with and without L_cf?
- How do you distinguish background-as-confounder from background-as-cause? How do you handle the agent remaining visible outside the masked span?
- What exactly was trained: which encoder, which vocabulary, what feature dimension?
- What are the annotation statistics: number of annotators, correction rates, agreement, and the masking-test protocol and chance level?
- How does your approach relate to NarrativeBridge, MECD, and FAME, and how does it perform against NarrativeBridge's CEN?
Reviewer comment
I have not received any response from the authors. But I would recommend that they consider the weaknesses mentioned in my review.