acceptodds
Withdrawn from ICLR 2027

Causal-EVC: Breaking Emotional Spurious Causality via Spatiotemporal Grounding and Counterfactual Intervention in MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Emotional Video Captioning aims to generate factually accurate and emotionally empathetic descriptions. While recent methods have recognized the importance of visual causes to guide emotion perception and caption generation, they fundamentally rely on simple attention matching, which inevitably suffers from causal redundancy and spurious correlations in co-occurrence bias (e.g., misclassifying “sadness” as “joy” on a sunny beach), leading to severe shortcut learning from confusing backgrounds. Furthermore, existing evaluations fail to verify whether models have genuinely mastered causal reasoning or merely exploited background confounders. To address these limitations, we first construct EVC-CauseGround, a comprehensive benchmark with dense spatio-temporal causal annotations. Crucially, it introduces a carefully selected Causal-Faithfulness Subset to explicitly quantify genuine emotion-cause attribution. Second, we propose Causal-EVC, an emotion-grounding captioning framework, which introduces a Motion-guided Causal Spatiotemporal Localization module to precisely decouple causal triggers from background confounders. Besides, we introduce an Interpretable Sparse Emotion Routing module. By synthesizing counterfactual representations and formulating a novel counterfactual contrastive objective, we enforce the model to anchor its emotion predictions strictly on authentic causal triggers instead of confusing background. Extensive experiments show that Causal-EVC not only achieves the best performance on semantic metrics but also exhibits significant advantages in the causal-faithfulness subset, which demonstrates that our model could mine emotional cues from genuine visual causes and mitigate co-occurrence bias for interpretable multimodal emotion understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Emotional Understanding, Causal Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nb4FOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper targets emotional video captioning (EVC) and argues that existing methods exploit background "confounders" instead of the true emotion trigger. Two contributions:

  1. EVC-CauseGround. This is a benchmark of 2,000 videos from EmVidCap-L and MAFW, split 1,500 train / 500 eval. Each video has a spatio-temporal "cause tube." The tubes are produced by:

    • prompting a frontier model with a 30-frame grid plus the ground-truth emotion and caption to pick a temporal span, capped at 30% of frames;
    • having the same MLLM draw a box on the peak-optical-flow frame;
    • propagating that box with SAM-3.

    Humans then watch videos with the tube blacked out. If they can still guess the emotion, the video is labeled "Causal-Redundant"; otherwise, it is "Causal-Faithful," which yields a 176-video subset. The paper also defines an ST-IoU metric.

  2. Causal-EVC. This model combines:

    • MCSL, a motion prior from adjacent-frame patch-feature dissimilarity, used to bias temporal and axial spatial attention, plus box/span regression heads;
    • ISER, top-K sparse routing from masked visual features to GloVe+VAD emotion prototypes;
    • a two-way contrastive loss L_cf, contrasting foreground-masked (V⊙M_c) against background-masked (V⊙(1−M_c)) features;
    • a Qwen2.5-VL-7B decoder.
Critical strengths
  1. Shortcut reliance on scene context in affective video understanding is a real problem. Spatio-temporal grounding of emotion triggers would be a useful resource if done carefully.
  2. The human masking test, which asks whether annotators can still infer the emotion with the tube removed, is a sensible idea for diagnosing context leakage. It could be the most valuable part of the paper with better execution and reporting.
  3. The annotation pipeline is practical and cheap: an MLLM proposes the span and box, and a tracker densifies it. The paper also includes ablations of the main components and of top-K.
Critical weaknesses

A. Conceptual problems with the "causal" framing

  1. "Cause" vs. "confounder" is defined by fiat, and the definition is often wrong. The paper treats the localized foreground action as the cause and everything else as a confounder. In much affective video, the context is the cause: a party causes surprise, a disaster scene causes fear, a funeral causes grief. MAFW is explicitly said to include natural disasters. Fig. 1(a) seems to label the party itself as the "contextual confounder" for a surprised man. The Causal-Redundant criterion (annotators can guess the emotion from the masked video) does not separate "background is a spurious correlate" from "background is a genuine cause." It also does not separate either from "the mask failed to remove the agent," discussed in A3 below. The theoretical framing needs a definition of cause that survives these cases.

  2. Cause is conflated with expression. In the main case study (Fig. 6), the model is praised for attending to "the disgust expressions of the girl" rather than the ice cream. A facial expression is a manifestation of the emotion, not its cause; the cause is the bad taste. Rewarding the model for attending to expressions makes this a facial-expression saliency task, not emotion-cause reasoning. This is a core conceptual issue for a paper whose central claim is genuine emotion-cause attribution.

  3. The counterfactual is not an intervention. The do-operator and "backdoor" language (App. B: severing "the backdoor path V→U→Y," anchoring to P(Y|do(C))) is not supported.

    • The counterfactual video blacks out the tube only within [t_start, t_end], which is at most 30% of frames. The agent remains fully visible in the other 70%, so the "background-only" video often still shows the person whose emotion is being judged.
    • At the model level, Ĉ = V⊙(1−M_c) masks patch tokens after the Qwen2.5-VL vision encoder. Those tokens have already mixed global context through self-attention, so zeroing them removes neither foreground information from "background" tokens nor the reverse.
    • V→U→Y is not a backdoor path; a backdoor path enters the treatment. U as described is a co-occurring variable, not a confounder of C and Y.
    • No structural causal model is stated in the main text, and no identification argument is given.
  4. The information-theoretic "guarantee" (App. B) is not valid.

    • Eq. 24 claims that minimizing a two-element softmax loss "tightly bounds" I(C;E) − I(U;E). No derivation is given, and with one negative the InfoNCE bound saturates at log 2.
    • L_cf only compares the GT-class cosine for C vs. Ĉ (Eq. 20). It does not drive I(U;E) toward zero in any formal sense.
    • L_cf also has a degenerate solution: grow M_c toward all-ones so that Ĉ ≈ 0 and z_fake becomes constant. The only counterweight is box/span supervision. Please report mask coverage statistics; claims like "absorb all emotion-relevant semantics into C" suggest this is happening.
  5. Circularity between annotation and model. The ground-truth tubes are anchored on the peak-optical-flow frame, and the span is forced into a short burst (≤30%). The model's key inductive bias is a motion prior that boosts high-motion frames and regions. The grounding gains may largely reflect matching the annotation heuristic rather than emotion-cause reasoning. A motion-only baseline (argmax-motion span plus the moving region) on EVC-CauseGround is essential. The paper's own Table 3 shows that motion alone is a weak mask for captioning, but it never reports grounding numbers for that baseline.

  6. The t-SNE argument is circular. Fig. 7 shows Ĉ spread out and C clustered by emotion. L_cf optimizes exactly that, so the figure confirms the loss was minimized. It does not show that predictions rely on true causes.

B. Benchmark validity

  1. The headline evaluation is missing. The abstract and intro claim "significant advantages in the causal-faithfulness subset," and the 176-video subset is presented as the key contribution. No table reports any result on that subset, or a Faithful-vs-Redundant split. Table 2 and Fig. 2 report the full 500-video eval set only. This is the main evidence the paper promises, and it is absent.

  2. Label provenance and family bias.

    • Cause labels come from a GPT model that is given the GT emotion and caption, so the label is "what GPT thinks justifies this emotion."
    • GPT-4o and Gemini are then evaluated zero-shot against these labels, while Causal-EVC is trained on 1,500 videos labeled the same way.
    • The annotating model is called GPT-5.6 (main text) and GPT-5.5 (appendix), with a GPT-5 system card citation.
    • Missing details: the number of human annotators, how many boxes and spans were corrected, inter-annotator agreement, pipeline-vs-human agreement, and the guessing protocol for the masking test (forced choice among how many classes, chance level, majority rule).
    • It is unclear whether the 176 Faithful videos come from the 500 eval videos or from all 2,000. If the latter, about 91% of videos are "redundant," which undercuts the premise that context is mostly spurious.
  3. Granularity. Videos average ~100 s, but only 30 frames are sampled, about 3.3 s apart. "Motion" between frames 3 s apart is mostly scene or shot change rather than the kinematic signal the method describes. ST-IoU is also computed at this coarse frame resolution. The metric also assumes a single cause per video. Eq. 1 switches between T_interest and T_intersect without defining either.

C. Experimental and reporting issues

  1. Possible label leakage at inference. Sec. 3 says the model is "given the input video and emotion category." Eq. 19 computes logits against E_gt, "the prototype embedding of the ground-truth emotion labels." If the GT emotion, or anything derived from it, is available at test time, emotion accuracy (Acc_sw, Acc_c) and the hybrid metrics are invalid. Please state exactly what the model receives at inference and how the emotion word is chosen.

  2. Where do tubes come from on EVC-MSVD / EVC-VE? All ablations (Tables 3–6) are on EVC-MSVD and include L_temp and L_spa. These require spatio-temporal tube annotations, which EVC-MSVD does not have. Was the model pretrained on EVC-CauseGround and transferred, or were tubes pseudo-labeled? This affects every number in Tables 1 and 3–6.

  3. The implementation section does not match the described model.

    • Qwen2.5-VL-7B is cited as Raffel et al. 2020 (T5).
    • The vocabulary size is 32,128, which is T5's vocabulary; Qwen's is roughly 150k.
    • D = 1408 matches EVA-ViT-g (BLIP-2), not the Qwen2.5-VL vision encoder.
    • Input is 224×224 center crops, whereas Qwen2.5-VL uses native dynamic resolution.
    • Maximum caption length is 15.
    • Hyperparameters λ_e, λ_cls, and θ are listed but never appear in the method.
    • GloVe is described as "Transformer-based."

    This looks like configuration inherited from a T5-based predecessor. I could not determine what was actually trained.

  4. The equations are dimensionally inconsistent.

    • Eq. 7 adds a T×T diagonal to an N_w×(T·P) score matrix and claims an output in R^{T×P×D}, but N_w queries yield N_w×D.
    • Eq. 9 has similar issues.
    • Eq. 4 claims to remove camera motion by adding λ_m times the global mean to the per-patch score. Adding a constant does not remove global motion; subtracting it would. Under the per-frame max normalization in Eq. 5, the term is close to a no-op.
  5. Unfair and selectively framed comparisons.

    • The contributions cite "+17.3% mIoU over zero-shot Qwen2.5-VL-7B." That is 6.9 points absolute, against the untuned version of the model's own backbone, after training on 1,500 in-domain annotated videos. The necessary baseline is Qwen2.5-VL-7B LoRA-tuned on the same data with the same heads but without MCSL/ISER/L_cf. Fine-tuned grounding baselines (VTimeLLM, TimeSuite, etc.) on the 1,500 training videos are also needed.
    • In Table 1, the "+15.7% BFS" is relative and measured against SAGML (71.5). The strongest baseline, MM-ECPE, is at 81.8 vs. 82.7. MM-ECPE also beats the proposed method on BLEU-4 (71.4 vs. 71.2).
    • Relative and absolute gains are mixed throughout without labels.
    • It is not explained how MM-ECPE, which produces no boxes, gets an mIoU in Table 2.
  6. Statistical weight. The EVC-MSVD test set has 134 videos, EVC-CauseGround eval has 500, and the Faithful subset has 176. Many reported differences are around 1 point, with no seeds, variance, or significance tests. The top-K sensitivity is also extreme: K=1 drops Acc_sw from 94.2 to 49.6 and CIDEr by 54. That calls for explanation rather than a fault-tolerance narrative.

  7. Human evaluation. It uses 20 raters, 50 pairs, and a 1–10 scale, with no agreement statistics, blinding, or significance tests. VideoBLIP appears in the human study and in the baseline list but not in Table 1, while SGN appears in Table 1 but not in the baseline list.

D. Related work and novelty

  1. Missing causal video captioning and reasoning work.

    • NarrativeBridge (Nadeem et al., ICLR 2025) introduced LLM-generated causal-temporal narrative captions for MSVD/MSRVTT and a Cause-Effect Network with separate cause and effect encoders. This is the closest prior work on causality-aware captioning and should be discussed and compared.
    • MECD (Chen et al., NeurIPS 2024) frames multi-event causal discovery in video. It uses mask-based event prediction (a Granger-style test that compares predictions with premise events masked vs. unmasked) together with front-door adjustment and counterfactual inference. That is the same masking-as-counterfactual idea, executed with an actual causal-inference treatment.

    The claim in App. A.2 to be "the first work to notice the visual causes of emotions" is also contradicted by the paper's own baseline MM-ECPE.

  2. Missing background-debiasing work. Motion-guided foreground/background separation to stop video models from cheating on static backgrounds is well established. FAME (Ding et al., CVPR 2022) extracts the moving foreground from frame differences and pastes it onto other videos' backgrounds so contrastive learning cannot rely on the background. Related work includes SDN (Choi et al., NeurIPS 2019) and Background Erasing (Wang et al., CVPR 2021). The deconfounded and counterfactual VL literature (deconfounded captioning, counterfactual VQA, CSS) is also relevant. Against this background, MCSL plus L_cf amounts to a motion-weighted attention mask with a single-negative foreground/background contrast, and the novelty is modest. The emotion-reasoning MLLMs (e.g., Emotion-LLaMA, EMER/explainable multimodal emotion recognition) are also absent from Table 2.

  3. Related work is in the appendix only. It cites several clearly unrelated works as multimodal emotion-cause extraction: radiology report generation, news captioning, and time-series anomaly detection.

Minor

  • "Cofounder" → "confounder" (multiple places); "convinced metric"; "throwed"; "sicken bread."
  • "Pixel-level motion correlation" is computed on ViT patch features, not pixels.
  • Ethics, licensing, and release terms for MAFW (which includes movie clips) and EmVidCap-derived faces are not discussed. Release of the benchmark itself is not promised; only code is.

Sources:

Questions for authors
  1. At inference, does the model receive the GT emotion category or E_gt in any form?
  2. How were L_temp and L_spa supervised on EVC-MSVD for Tables 3–6?
  3. Can you report Table 2 on the 176 Faithful videos and on the Redundant videos separately? Can you include a fine-tuned Qwen2.5-VL baseline and a motion-only grounding baseline?
  4. What is the average mask coverage of M_c, and how does it change with and without L_cf?
  5. How do you distinguish background-as-confounder from background-as-cause? How do you handle the agent remaining visible outside the masked span?
  6. What exactly was trained: which encoder, which vocabulary, what feature dimension?
  7. What are the annotation statistics: number of annotators, correction rates, agreement, and the masking-test protocol and chance level?
  8. How does your approach relate to NarrativeBridge, MECD, and FAME, and how does it perform against NarrativeBridge's CEN?
1 comment
Reviewer_nb4FOpenReview ↗

Reviewer comment

I have not received any response from the authors. But I would recommend that they consider the weaknesses mentioned in my review.

Reviewer_n69POpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

this paper addresses emotional video captioning, arguing that attention-based cause attribution learns background shortcuts and that existing benchmarks cannot detect this. It constructs EVC-CauseGround, with frame-level temporal spans and dense spatial tubes for visual causes, a 176-video Causal-Faithfulness Subset, and an ST-IoU metric, and proposes Causal-EVC, which combines motion-guided spatio-temporal localization, top-K sparse routing over VAD-augmented emotion prototypes, and a counterfactual contrastive loss between causal and background representations. Experiments report state-of-the-art captioning/emotion results on EVC-MSVD and EVC-VE and large cause-grounding gains over zero-shot multimodal LLMs and grounding models on EVC-CauseGround.

Critical strengths
  • benchmark idea addresses a real gap. dense spatio-temporal tubes for emotional causes, a subset designed to remove background leakage, and a metric that scores temporal and spatial localization together would let the field test whether a model used the trigger or the scene.

  • the method is a plausible combination. the method is a sensible combination of motion priors, sparse prototype routing, and counterfactual contrast, and the ablations show each module changes the metrics in the expected direction on the authors' training setup. the approach is reasonable but what is missing is evidence that it supports the paper’s central claim.

Critical weaknesses
  • the headline claim about the Causal-Faithfulness Subset has no results. The abstract says the model “exhibits significant advantages in the causal-faithfulness subset,” but no table, figure, or number reports any metric on the 176-video subset.

  • the main comparison is trained-vs-untrained, and the evaluation target comes from the same automated annotation pipeline used to create the training targets. Table 2 compares a model trained on the benchmark’s 1,500 training videos against baselines that are not trained on EVC-CauseGround (the paper describes Qwen2.5-VL as zero-shot). more importantly, ground-truth tubes are generated by an MLLM prompted with ground-truth emotion and caption and tracked by SAM-3, with human checking limited to boundary frames. So no independent human annotation or inter-annotator agreement is reported. Those tubes are the training targets for Ltemp/Lspa and the evaluation targets for ST-IoU. the cause-grounding gaps therefore reflect training and supervision effects, and ST-IoU measures agreement with the automated annotator rather than human causal attribution.

Questions for authors
  1. What are the results on the 176-video Causal-Faithfulness Subset?
  2. How were the reference tubes validated beyond boundary checking. is there a human-drawn subset, and what is the inter-annotator agreement?
  3. given that the annotation MLLM is prompted with the ground-truth emotion, how do you ensure ST-IoU measures causal attribution rather than label-conditioned agreement?
Reviewer_H23vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the reliance on background correlations in emotional video captioning. The authors introduce EVC-CauseGround, which provides temporal spans and spatial annotations for emotion-related visual regions, together with a Causal-Faithfulness subset of 176 videos. The proposed method combines motion-guided spatiotemporal localization, sparse emotion routing, and foreground–background contrastive training. Evaluation covers two emotional captioning datasets and the new grounding benchmark, with ablations, qualitative examples, and human evaluation. The reported results show improvements in captioning and localization, although whether these improvements demonstrate reduced reliance on background shortcuts remains unclear.

Critical strengths
  1. The dataset makes the visual basis of emotion predictions more inspectable. Caption quality and emotion-word accuracy alone provide limited information about why a model makes a particular prediction. Adding temporal spans and spatial regions allows predictions to be checked against specific video content. This is a useful contribution and provides a more detailed evaluation setting for subsequent work.

  2. The reported results and ablations support the practical value of the approach. Table 1 reports generally favorable results on both captioning datasets. For example, on EVC-VE, emotion-word accuracy increases from 80.2 to 82.8 and CIDEr from 71.8 to 74.0 compared with SAGML. Tables 3–6 examine the motion prior, localization components, training objectives, and sparse routing. These results support further investigation of the approach, particularly the use of motion information alongside semantic localization.

Critical weaknesses
  1. The central causal claims lack direct supporting evidence.

The paper repeatedly emphasizes performance on the Causal-Faithfulness subset, but Table 2 reports only aggregate EVC-CauseGround results. Separate results for the 176 videos are not provided, and their distribution across training and evaluation splits is unclear. Better agreement with annotated regions does not, by itself, establish reduced reliance on background shortcuts. The foreground–background objective encourages the foreground branch to score the target emotion higher than the background branch. Appendix B goes further by claiming that this removes background information and provides strict causal guarantees, without sufficient justification. The caption decoder also receives the complete video features, leaving a route through which background information could influence the generated emotion words. Since reduced background dependence is central to the paper, it needs direct evaluation through the dedicated subset and controlled background interventions.

  1. The source of the performance gains and the independence of the evaluation data need clarification.

The paper describes the main Qwen2.5-VL-7B comparison as zero-shot, whereas the proposed method uses temporal and spatial supervision together with parameter fine-tuning. The difference in Table 2 therefore reflects both additional supervision and architectural changes. A comparison using the same backbone, training data, and grounding supervision is needed to assess the contribution of the proposed components.

In addition, the new benchmark includes the 1,523 EmVidCap-L videos and introduces a new training/evaluation split. The paper also reports results on EVC-VE, which uses these source videos, without explaining how the splits and training stages correspond. This does not establish that leakage occurred, but it leaves open whether original test videos were used during grounding training. These details directly affect the interpretation and comparability of the reported results.

Questions for authors
  1. How many of the 176 Causal-Faithfulness videos belong to the held-out evaluation set? Does Table 2 evaluate the full test set or this subset? Please provide separate results for the subset.

  2. Which training data are used for each reported experiment? How do the EVC-CauseGround splits overlap with the original EVC-VE splits? Were any original EVC-VE test videos used at any training stage for the model evaluated on EVC-VE?

  3. How much improvement remains when the baseline uses the same backbone, training data, and spatiotemporal supervision as the proposed method?

  4. When foreground content is held fixed and the background is changed, how stable are the emotion predictions and generated captions? How does this stability change when foreground–background contrastive training is removed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.