acceptodds
Under review as a conference paper at ICLR 2027

EventTriggerBench: Discovering Plausible Trigger Events for Explanatory Video Understanding

Abstract

Understanding videos requires more than recognizing what happens; it also requires identifying which earlier events make a later event plausible. We introduce EventTriggerBench, a benchmark for target-conditioned trigger attribution in videos. Given an ordered sequence of localized events and a target event that may appear anywhere in the timeline, models must identify earlier events that provide plausible explanatory support for the target. EventTriggerBench is designed to test a key gap in current video understanding: models often capture temporal co-occurrence, but struggle to distinguish events that merely precede a target from those that meaningfully explain it. To study this gap, we provide auto-verified trigger labels and evaluate a broad range of heuristic, sequential, graph-based, multimodal, causal, and LLM baselines. We further introduce EventGraph-Trigger, a multimodal event-graph reasoning framework for target-aware trigger attribution. Our experiments show that trigger attribution remains challenging even for strong video-language and LLM-based models, especially when explanations require long-range context, multi-step event chains, or distinguishing direct triggers from incidental earlier events. These findings suggest that target-conditioned trigger discovery offers a useful diagnostic lens for moving video understanding beyond event recognition toward structured explanatory reasoning.

Primary area:
datasets and benchmarks
Keywords:
anguage Modeling, Multimodality and Language Grounding to Vision, Dataset

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_VRseOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces EventTriggerBench to evaluate whether models can identify earlier events that plausibly explain a target event in an ordered video sequence. The target may occur anywhere in the timeline, and the task includes non-adjacent triggers, multi-step explanatory chains, and multiple triggers. The benchmark contains 1,240 videos, 8,936 event segments, and 3,482 target-conditioned samples. Trigger labels are proposed by a language model and automatically verified, and the paper also reports a human audit of 300 test samples. The authors further propose EventGraph-Trigger, which uses an event graph, multimodal features, and target-conditioned message passing to predict triggers and explanatory chains. The experiments compare a range of baselines and include ablation, difficulty, and robustness analyses.

Critical strengths
  1. The task allows the target event to appear anywhere in the video timeline, enabling evaluation of non-adjacent triggers, multi-step explanations, and multiple triggers rather than focusing only on the final event. The reported statistics show that many triggers are not immediately adjacent to their targets, and a simple previous-event heuristic performs poorly.

  2. The paper describes the data construction and label-quality checks in detail. It reports human acceptance rates before and after automatic verification, inter-annotator agreement, and comparisons between independently collected human trigger labels and automatic labels. The model evaluation also includes a parameter-matched multimodal Transformer, component ablations, and results across different difficulty conditions.

Critical weaknesses
  1. The trigger events and explanatory chains are proposed and verified by language models that receive only event captions. For videos without existing event captions, the paper uses Video-LLaVA-7B to generate them. If a caption omits or misrepresents a key detail, the automatic label cannot assess the trigger relation based on that visual evidence. In Table 7, removing text features reduces EventGraph-Trigger’s accuracy by 9.0 percentage points, while removing video features reduces it by only 3.3 points. In Table 15, masking captions reduces EventGraph-Trigger’s accuracy from 72.7 to 49.6 and GPT-4o’s from 64.3 to 43.5. These results show that the models rely substantially on captions, but they do not establish whether visual evidence itself is weak or whether captions or automatically generated labels omit important visual information. Although the paper includes a human audit of 300 test samples in which annotators view the videos, labels for the remaining samples rely primarily on the caption-based automatic pipeline.
Questions for authors
  1. Could the authors report the accuracy of each model on a visually diagnostic subset under three caption conditions, namely complete captions, captions with 50 percent of their content words randomly removed, and fully masked captions? In particular, how much does EventGraph-Trigger improve over text-only baselines on samples where visual evidence is necessary to identify the trigger?
2 comments
AuthorsOpenReview ↗

Response to Reviewer VRse

We thank the reviewer for requesting a visual-evidence-specific evaluation.

1. Visual evidence under three caption conditions

On the 60-example human-defined visually diagnostic subset, video improves EventGraph-Trigger (EGT) by 23.3 pp under full captions, 25.0 pp under 50% content-word deletion, and 23.3 pp when captions are fully masked, relative to its matched no-video variant. The much larger gain than on the full test set (+3.3 pp; manuscript Table 7) shows the value of video for these cases.

The modest overall video gain may have two causes: video evidence may add little beyond captions, or caption-derived labels may omit visually important details. To distinguish these, annotators first select triggers using captions alone and then re-evaluate the same 300 audited examples with the original clips. The 60 examples in which visual evidence changes the supported attribution form the visually diagnostic subset, selected independently of model predictions. We evaluate both automatic-label agreement and model accuracy against the video-informed human reference.

Model on visual subset (n=60), Acc.Full captions50% content-word deletionFully masked
GPT-4 CoT (text only)30.023.3—
EGT without video (text+time; time only when masked)33.323.316.7
CERN (video+text)46.740.033.3
GPT-4o (video+text)51.741.733.3
EGT (video+text)56.748.340.0
Video gain over matched EGT no-video (pp)+23.3+25.0+23.3

The original caption-derived labels agree with the video-informed reference on 32/60 (53.3%) visual cases, versus 201/240 (83.8%) remaining cases. This supports the role of both additional visual evidence and caption-label omissions.

Reference-label auditCasesAutomatic–human matchesAgreement
Visually diagnostic603253.3
Remaining audited cases24020183.8
All audited cases (manuscript)30023377.7

The full model's 56.7% accuracy on this difficult subset also shows that visually grounded attribution remains challenging despite the substantial relative gain.

Thank you for your response. My concerns are mostly addressed, and I maintain my overall positive rating.

Reviewer_fAgxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EventTriggerBench, a benchmark for target-conditioned trigger attribution in videos. Given localized events, their captions and clips, and a target event, models identify earlier events that plausibly explain the target and optionally predict an explanatory chain. The dataset contains 1,240 videos and 3,482 target-conditioned samples, with labels produced by an LLM proposal-and-verification pipeline. The authors also propose EventGraph-Trigger, which combines text, video, and temporal features in a target-conditioned event graph. Evaluation includes diverse baselines, matched architectural controls, independent human labels, cross-family re-annotation, and ablations. The task concerns plausible explanatory attribution rather than causal identification.

Critical strengths
  1. A useful benchmark with substantial validation. Target-conditioned attribution and explanatory-chain prediction provide a meaningful way to study relationships between video events. Independent human annotation on 300 test samples and cross-family re-annotation strengthen the evaluation beyond agreement with a single automatic labeling pipeline. These analyses make the benchmark a potentially useful resource despite the remaining ambiguity in explanatory labels.

  2. Consistent improvements with informative controls. EventGraph-Trigger achieves 72.7% accuracy, compared with 67.4% for ReAct, 66.9% for CERN, and 65.3% for the relation-biased Transformer. On independent human labels, it achieves 69.7%, compared with 64.0% for CERN. The matched-input, supervision, and capacity comparisons are particularly useful, and the advantage under independent human evaluation supports the practical value of the method. These strengths lead me to lean toward acceptance, while the concerns below limit the strength of my recommendation.

Critical weaknesses
  1. The annotation and evaluation choices limit how broadly the results can be interpreted. The main pipeline uses gpt-4-turbo-2024-04-09 for both proposal and verification, and the general-purpose LLM/VLM baselines primarily include GPT-4 Turbo, GPT-4o, Qwen2-VL-7B, and Video-LLaVA-7B. Model age alone does not invalidate the results, and the human and Qwen2.5 analyses are valuable. However, the latter changes only test labels, while training still relies on GPT-4 annotations. Comparisons with more recent strong annotators and evaluation models would help establish the stability of the labels and the benchmark's remaining difficulty. Moreover, removing video features reduces accuracy only from 72.7% to 69.4%. This indicates that much of the benchmark can be addressed through caption semantics, and a human-validated subset requiring visual evidence would better establish its contribution to video understanding.

  2. The necessity of branching graph reasoning and the semantics of the explanatory labels remain only partly established. The model does construct a directed graph, but supervision and evaluation primarily concern a single annotated explanatory path, and decoding returns one path from the top-ranked trigger to the target. The relation-biased Transformer is a useful control, but a matched chain-reasoning baseline would more directly test whether branching graph structure is necessary. Label ambiguity also affects this interpretation: the first-event convention can exclude reasonable proximal explanations, while multi-trigger labels combine jointly necessary events, individually sufficient events, and alternative explanations. Independent human annotation does not impose the same first-event convention. Consequently, some disagreement reflects differences in interpretation, and Hit@1 does not demonstrate recovery of a complete explanatory structure. The authors acknowledge these limitations, but they still constrain the conclusions supported by the benchmark.

Questions for authors
  1. Clarification on data provenance: Which source datasets or video collections contribute to the 1,240 videos, and in what proportions? Please provide source/domain counts, the proportion of human versus generated captions, and source-level split statistics. I could not identify a precise breakdown in the current description.

  2. Evidence that could strengthen the evaluation: On a representative fixed subset, how do labels and retention decisions change with a more recent strong annotator, and how well do those labels agree with independent human judgments? Would the method's advantage remain against a recent strong reasoning LLM/VLM under comparable input and inference-budget settings?

  3. Clarification and targeted comparison: How often do examples require branching or converging dependencies rather than one explanatory chain? Can a matched chain-reasoning baseline, with the same inputs and supervision, clarify what is gained specifically from the graph structure?

  4. Clarification of task validity: Can the authors quantify a human-validated subset where captions alone are insufficient? How would evaluation change if annotators followed the same trigger-granularity convention and multi-trigger cases were separated into jointly necessary triggers, individually sufficient triggers, and alternative explanations? The current chain-lenient and set-level results are useful, but do not fully resolve these distinctions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer fAgx[1/2]

We thank the reviewer for the detailed questions about annotation robustness, structural reasoning, and trigger semantics.

1. New annotators and training-label dependence

The advantage of EventGraph-Trigger (EGT) persists across annotation sources. The manuscript's paired Qwen2.5-72B evaluation (Table 9; 607 jointly retained examples) shows +5.9 pp over CERN under GPT-4 labels and +5.6 pp under Qwen2.5 labels. In the additional 300-example annotator audit, Qwen3.8-27B attains 77.6% human-majority agreement versus 78.8% for GPT-4 on the same 255 retained examples; the pipelines agree on 207/255 (81.2%) labels.

Annotation pipelineAcceptReviseDiscardRetained on fixed 300 panelHuman agreement on paired retention
Qwen3.8-27B (additional 300-panel run)63.321.715.0255/30077.6
GPT-4 labels on same retained cases———255/30078.8

For Qwen3.8-27B, the accept/revise/discard distribution corresponds to the fixed 300-example panel. Training-label dependence is evaluated against the same human-reference set:

Training labelsCERN human Acc.EventGraph-Trigger human Acc.Gap (pp)
Original GPT-4 labels64.069.7+5.7
Independent Qwen2.5 labels61.767.0+5.3

The EGT–CERN margin remains similar after changing training labels (5.7 → 5.3 pp), although both absolute scores decrease.

2. Video-dependent examples

Video provides a substantial benefit on visually diagnostic examples, while caption-derived annotations also miss some visually justified triggers. On the 60-example human-defined visual subset, EventGraph-Trigger outperforms its matched no-video variant by 23.3 pp with full captions, 25.0 pp with half the content words removed, and 23.3 pp with all captions masked. Against independent video-informed human judgments, the original automatic labels agree on only 53.3% of these visual cases, compared with 83.8% outside the subset.

The modest overall video gain may have two causes: video evidence may add little beyond captions, or caption-derived labels may omit visually important details. To distinguish these, annotators first select triggers using captions alone and then re-evaluate the same 300 audited examples with the original clips. The 60 examples in which visual evidence changes the supported attribution form the visually diagnostic subset, selected independently of model predictions. We evaluate both automatic-label agreement and model accuracy against the video-informed human reference.

Reference-label auditCasesAutomatic–human matchesAgreement
Visually diagnostic603253.3
Remaining audited cases24020183.8
All audited cases (manuscript total)30023377.7

Of the 28 disagreements on visual cases, 19 involve omitted visual details, 6 involve misleading captions, and 3 reflect trigger-granularity ambiguity. The disagreement pattern indicates that caption-derived annotations sometimes miss evidence visible in the clips.

Model on visual subset , Acc.Full captions50% content-word deletionFully masked
GPT-4 CoT (text only)30.023.3—
EGT without video (text+time; time only when masked)33.323.316.7
CERN (video+text)46.740.033.3
GPT-4o (video+text)51.741.733.3
EGT (video+text)56.748.340.0
Video gain over matched EGT no-video (pp)+23.3+25.0+23.3

On the full test set, the manuscript reports a +3.3 pp video gain (72.7 vs. 69.4) and a chain-edge F1 increase from 77.7 to 82.6 (Table 21).

AuthorsOpenReview ↗

Response to Reviewer fAgx[2/2]

3. Branching evidence and matched one-parent control

The graph advantage is larger on multi-trigger samples, and restricting event aggregation to one incoming edge reduces that advantage. Manuscript Table 14 reports +13.1 pp over the matched MM Transformer on multi-trigger cases (68.5 vs. 55.4) versus +10.0 pp on single-trigger cases (73.7 vs. 63.7). Under a matched k=1 in-degree restriction, EGT falls from 68.5 to 58.5 on multi-trigger samples (−10.0 pp), compared with 73.7 to 71.1 on single-trigger samples (−2.6 pp).

Graph restrictionSingle-trigger Acc.Multi-trigger Acc.Overall Acc.
Matched MM Transformer (manuscript)63.755.462.1
EGT with one incoming edge (k=1)71.158.568.7
EGT, full (manuscript)73.768.572.7

Existing labels also show that 82/130 (63.1%) multi-trigger cases have at least one annotated trigger outside the primary chain, indicating that a single linear path does not capture every annotated initiating event.

4. Trigger granularity and multi-trigger semantics

The first-event convention accounts for some human–automatic disagreement, and performance differs across multi-trigger meanings. In the submitted audit, 21/67 disagreements select a chain mediator rather than its initiating event; allowing mediators increases agreement from 77.7% to 84.7%. Under an explicit initiating-trigger instruction, an additional paired annotation analysis gives 84.0% agreement, compared with 77.7% under unconstrained instructions. The paired comparison quantifies how trigger granularity affects human agreement.

Multi-trigger semantic category (n=130)CasesEGT Acc.CERN Acc.EGT Set F1EGT Set Exact Match
Jointly necessary3663.952.863.936.1
Individually sufficient2774.159.373.848.1
Alternative plausible explanations4971.463.371.244.9
Ambiguous / unresolved1861.166.761.033.3
All multi-trigger cases (manuscript aggregate)13068.560.068.941.5

EGT gains the most on individually sufficient triggers (+14.8 pp Acc.). On ambiguous cases, CERN is higher by one correct example (12/18 vs. 11/18), corresponding to 66.7% versus 61.1%. The per-type Set F1 and Set Exact Match further distinguish identifying one plausible trigger from recovering the annotated trigger set.

5. Video source

The 1,240 source videos comprise 1,062 videos from ActivityNet Captions[1] and 178 from MEVA[2]. The submitted Section 3.1 describes the event-acquisition pipeline but omits this source breakdown. Available dense event annotations are cleaned and retained; videos without event annotations are segmented with TransNetV2 and captioned with Video-LLaVA-7B. We will add the source breakdown to Section 3.1 and Table 4. Generated captions also undergo human review during data processing. Table 15 separately compares original human-written captions and generated captions on the same annotated videos. EventTriggerBench adds target selection, auto-verified plausible trigger labels, multi-trigger sets, and explanatory chains.

SourceVideosEvent segmentsTrigger samples
ActivityNet Captions1,06278643173
MEVA1781072309
Total1,2408,9363,482

The benchmark uses video-disjoint 70/10/20 train/validation/test splits as reported in Appendix D.

[1] Krishna, Ranjay, et al. "Dense-captioning events in videos." Proceedings of the IEEE International Conference on Computer Vision. 2017.

[2] Corona, Kellie, et al. "Meva: A large-scale multiview, multimodal video dataset for activity detection." 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021.

Reviewer_apD2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EventTriggerBench, a benchmark for target-conditioned plausible trigger attribution in videos. Given an ordered sequence of captioned event clips and a target event at any position, a model must select the earlier event(s) that plausibly explain the target.

The paper claims three contributions: (1) the task formulation, which allows targets anywhere in the timeline and labels multi-trigger sets and chains; (2) the benchmark; and (3) EventGraph-Trigger, a target-conditioned event-graph model with temporal, semantic, visual, and learned edges and an auxiliary chain head as a baseline for their benchmark.

Critical strengths
  1. The paper is well written and easy to follow.
  2. Independent human annotations on 300 samples agree with the automatically generated trigger labels in 77.7% of cases.
Critical weaknesses
  1. My main concern is the limited novelty beyond what CARVE[1] presents. I think the paper should better clarify what new reasoning capability the benchmark evaluates beyond CARVE.

  2. The authors do not discuss what is the initial real world video dataset that they use to create the benchmark from it.

  3. I appreciate the authors introducing EventGraph-Trigger, a baseline for their benchmark, but this baseline has very limited novelty compared to CERN[1], which already addresses trigger attribution through temporal-semantic event graphs. It is unclear what specific limitations of existing approaches EventGraph-Trigger is intended to address. Also without explanatory-chain supervision, EventGraph-Trigger achieves 69.0% accuracy compared with CERN’s 66.9%, raising questions about how much of the reported improvement comes from the architectural changes rather than additional supervision.

  4. In the benchmark generation pipeline labels are generated entirely from captions, and the experiments show that text contributes substantially more than visual information. Removing video features reduces accuracy by 3.3 points, compared with 9.0 points when removing text features. This is concerning for me since this benchmark is mainly for evaluating video understanding capabilities of models.


[1] Le, T. M., Le, V., Do, K., Gupta, S., Venkatesh, S., & Tran, T. (2025). Finding the Trigger: Causal Abductive Reasoning on Video Events.

Questions for authors
  1. Have the authors considered validating the annotation pipeline against intervention-derived ground truth, for example by applying it to CARVE's synthetic split? I think this would help assess whether the LLM-generated trigger labels align with counterfactual evidence rather than only plausible explanations.

  2. What is the initial real world video dataset that the authors built the benchmark from?

3 comments
AuthorsOpenReview ↗

Response to Reviewer apD2[1/3]

We thank the reviewer for the detailed comments on benchmark novelty, model comparisons, and annotation quality.

1. Distinction from CARVE

EventTriggerBench introduces a complementary real-video evaluation setting for trigger attribution and explanatory paths. CARVE studies arbitrary target events and causal-chain reasoning, including intervention-grounded synthetic examples. EventTriggerBench instead evaluates open-vocabulary explanatory relationships in real videos, with event-index chain annotations and separately evaluated explanatory paths. These relationships include game rules, intentions, and social responses that are not reducible to synthetic physical interactions. In Fig. 1, an opponent's missed return supports the player's win, celebration, and mocking—an explanatory sequence whose meaning depends on the game and social context.

ComparisonCARVEEventTriggerBench
Targets at arbitrary positionsYesYes
Trigger and chain reasoningYesYes
Label evidenceIntervention-derived synthetic labels and model-derived real-video labelsCaption-proposed plausible explanations; automatic verification and 300-sample human audit
Explicit event-index path evaluationNot the primary evaluationEdge F1, Path Exact Match, Path Recall, Chain-length Acc.
Real-video rule-, intent-, and social-context distractorsNot the focus of its synthetic intervention settingEvaluated within open-vocabulary event narratives

The submitted model reaches 82.6 Edge F1 and 48.6 Path Exact Match (Table 19), while distractor injection reduces Acc. from 72.7 to 68.6 (Table 20). We clarify the CARVE row in Table 1 to distinguish causal-chain reasoning from separately evaluated event-index chain annotations.

2. Original video sources

The 1,240 source videos comprise 1,062 videos from ActivityNet Captions[1] and 178 from MEVA[2]. The submitted Section 3.1 describes the event-acquisition pipeline but omits this source breakdown. Available dense event annotations are cleaned and retained; videos without event annotations are segmented with TransNetV2 and captioned with Video-LLaVA-7B. We will add the source breakdown to Section 3.1 and Table 4. Generated captions also undergo human review during data processing. Table 15 separately compares original human-written captions and generated captions on the same annotated videos. EventTriggerBench adds target selection, auto-verified plausible trigger labels, multi-trigger sets, and explanatory chains.

SourceVideosEvent segmentsTrigger samples
ActivityNet Captions1,06278643173
MEVA1781072309
Total1,2408,9363,482

The benchmark uses video-disjoint 70/10/20 train/validation/test splits as reported in Appendix D.

[1] Krishna, Ranjay, et al. "Dense-captioning events in videos." Proceedings of the IEEE International Conference on Computer Vision. 2017.

[2] Corona, Kellie, et al. "Meva: A large-scale multiview, multimodal video dataset for activity detection." 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021.

AuthorsOpenReview ↗

Response to Reviewer apD2[2/3]

3. What EventGraph-Trigger improves over CERN

EventGraph-Trigger (EGT) improves upon CERN primarily through additional visual and learned event relations, together with explanatory-chain supervision. The controlled comparison separates these effects: equipping CERN with the same chain head increases Acc. from 66.9 to 69.2 (+2.3 pp), while full EGT reaches 72.7 (+3.5 pp). With temporal+semantic edges alone, EGT is comparable to CERN (66.7 vs. 66.9 without chain supervision; 69.4 vs. 69.2 with it). Adding visual and learned relations raises EGT from 69.4 to 72.7 (+3.3 pp) under chain supervision.

Architecture and supervisionAcc.
CERN66.9
Matched Multimodal Transformer, with chain loss62.1
Relation-biased Transformer, with chain loss65.3
EGT, without chain loss69.0
CERN + matched chain head69.2
EGT, temporal+semantic edges, without chain loss66.7
EGT, temporal+semantic edges, with chain loss69.4
EGT, full72.7

For the CERN + chain head control, the event encoders, input modalities, chain annotations, loss weight, and training/evaluation protocol are matched to EGT; the underlying event-relation architecture remains CERN's. The comparison isolates chain-supervision gains from the remaining differences in relational modeling.

The improvements are particularly clear on more demanding attribution cases in manuscript Table 14: relative to CERN, EGT gains +9.6 pp for long-distance triggers (68.1 vs. 58.5), +9.1 pp for chains of length ≥4 (68.9 vs. 59.8), and +8.5 pp on multi-trigger examples (68.5 vs. 60.0). The respective gains for short-distance triggers, length-2 chains, and single-trigger examples are +3.8, +5.1, and +5.3 pp. These comparisons support the value of richer event relations for non-adjacent and multi-step explanations.

Chain-path evaluationEdge F1Path Exact MatchPath RecallChain-length Acc.
Matched MM Transformer68.932.865.358.6
EGT without chain loss71.336.368.060.5
CERN + matched chain head77.543.074.267.0
EGT full82.648.679.471.6

On annotated paths, EGT exceeds CERN with the matched chain head by 5.1 pp Edge F1 and 5.6 pp Path Exact Match.

AuthorsOpenReview ↗

Response to Reviewer apD2 [3/3]

4. Visual evidence versus caption-derived label omissions

Video provides a substantial benefit on visually diagnostic examples, while caption-derived annotations also miss some visually justified triggers. On the 60-example human-defined visual subset, EventGraph-Trigger outperforms its matched no-video variant by 23.3 pp with full captions, 25.0 pp with half the content words removed, and 23.3 pp with all captions masked. Against independent video-informed human judgments, the original automatic labels agree on only 53.3% of these visual cases, compared with 83.8% outside the subset.

The modest overall video gain may have two causes: video evidence may add little beyond captions, or caption-derived labels may omit visually important details. To distinguish these, annotators first select triggers using captions alone and then re-evaluate the same 300 audited examples with the original clips. The 60 examples in which visual evidence changes the supported attribution form the visually diagnostic subset, selected independently of model predictions. We evaluate both automatic-label agreement and model accuracy against the video-informed human reference.

Reference-label auditCasesAutomatic–human matchesAgreement
Visually diagnostic603253.3
Remaining audited cases24020183.8
All audited cases (manuscript total)30023377.7

Of the 28 disagreements on visual cases, 19 involve omitted visual details, 6 involve misleading captions, and 3 reflect trigger-granularity ambiguity. The disagreement pattern indicates that caption-derived annotations sometimes miss evidence visible in the clips.

Model on visual subset, Acc.Full captions50% content-word deletionFully masked
GPT-4 CoT (text only)30.023.3—
EGT without video (text+time; time only when masked)33.323.316.7
CERN (video+text)46.740.033.3
GPT-4o (video+text)51.741.733.3
EGT (video+text)56.748.340.0
Video gain over matched EGT no-video (pp)+23.3+25.0+23.3

On the full test set, the manuscript reports a +3.3 pp video gain (72.7 vs. 69.4) and a chain-edge F1 increase from 77.7 to 82.6 (Table 21).

5. Validation against CARVE intervention labels

On a fixed subset of CARVE synthetic cases, automated verification increases agreement with intervention-derived trigger labels. The same examples are scored before and after verification, avoiding an artificial gain from selective retention.

CARVE synthetic, fixed comparison subsetAcc.Set F1Set Exact Match
GPT-4 Turbo proposal68.464.040.2
GPT-4 Turbo proposal + verifier75.671.149.0
Independent proposal70.266.343.3
Independent proposal + verifier77.172.850.7

Verification improves agreement with intervention-derived labels by 7.2 pp Acc. and 7.1 pp Set F1, supporting the validity of the annotation filter on synthetic cases with intervention-based references.

Reviewer_zLiwOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the challenge of explanatory video understanding, moving beyond simple event recognition to identifying earlier events that make a later event plausible. The authors introduce EventTriggerBench, a benchmark designed for target-conditioned trigger attribution in videos. Given an ordered sequence of localized events and a target event at any position in the timeline, models are required to identify earlier events that provide plausible explanatory support for the target, rather than merely capturing temporal co-occurrence. The benchmark provides auto-verified trigger labels and evaluates a wide range of heuristic, sequential, graph-based, multimodal, causal, and LLM baselines. Furthermore, the authors propose EventGraph-Trigger, a multimodal event-graph reasoning framework for target-aware trigger attribution. Experimental results demonstrate that trigger attribution remains challenging for strong video-language and LLM-based models—especially in scenarios involving long-range context, multi-step event chains, or incidental earlier events—and show that the proposed framework serves as an effective baseline for structured explanatory reasoning.

Critical strengths
  1. Meaningful task: The studied task is a relatively high-level video understanding task. I believe this type of task is quite challenging for current LLMs and therefore holds high research value.
  2. Rigorous dataset annotation: The proposed dataset annotation pipeline is rigorous, which makes the quality of the dataset convincing. Moreover, the proposed evaluation metrics are rich and comprehensive.
  3. Detailed modality ablations: The ablation studies on different modality inputs are very detailed, covering both the proposed model and the baseline models.
  4. Well-motivated model: The proposed model is straightforward and well-motivated, as it naturally formulates trigger attribution through a multimodal event graph with target-conditioned message passing and explanatory-chain supervision.
Critical weaknesses
  1. Missing recent comparisons: The paper lacks comparisons with some more recent works, such as MECD+ (TPAMI 2025). In addition, the current evaluation focuses on using a single event as the target; I am curious whether the evaluation can be conducted at a multi-event level or at the whole-video level.
  2. Lacking comparison with the latest LLMs: The paper lacks some comparison with the latest LLM baselines. It would be more helpful if the authors could test larger or newer open-source or closed-source LLMs (such as Qwen3.8, etc.). I believe this would help readers better understand the capabilities of more recent models in this field on this task.
  3. Unclear video sources: The dataset lacks a description of the video data sources. I think this description is necessary, or the authors should clarify whether the dataset consists of video data collected by them.
Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zLiw

We thank the reviewer for recognizing the value of the task and the quality of the annotation pipeline.

1. MECD+ comparison, multi-event and whole-video evaluation

EventTriggerBench evaluates multiple targets per video. For whole-video evaluation, we combine the predicted trigger–target links across all annotated targets in each video. Micro Edge F1 measures link-prediction accuracy across all videos together; Macro Edge F1 averages each video's link F1; Video Exact Match measures the fraction of videos for which every annotated trigger–target link is predicted exactly. EventGraph-Trigger (EGT) obtains 68.3 Micro Edge F1, compared with 65.2 for adapted MECD+ and 61.7 for CERN. Its 29.4% Video Exact Match highlights the difficulty of recovering all annotated dependencies within a video.

Video-level evaluationMicro Edge F1Macro Edge F1Video Exact Match
CERN61.759.822.2
MECD+ (adapted)65.263.425.8
EventGraph-Trigger68.365.929.4

2. Comparisons with recent models

Recent models improve trigger attribution, but performance remains far from perfect. In the additional comparison, Qwen3.8-27B exceeds Qwen3-VL-8B-Instruct by 5.5 percentage points (pp) in Acc.; GPT-6.2 Sol performs best at 75.6% Acc., followed by Gemini 3.1 Pro at 73.2%.

Additional modelAcc.Top-3MRRMAPNDCG
Qwen3-VL-8B-Instruct65.986.276.769.981.2
Qwen3.8-27B71.489.781.475.185.4
GPT-6.2 Sol75.692.085.479.489.3
Gemini 3.1 Pro73.291.584.178.287.8

3. Original video sources

The 1,240 source videos comprise 1,062 videos from ActivityNet Captions[1] and 178 from MEVA[2]. The submitted Section 3.1 describes the event-acquisition pipeline but omits this source breakdown. Available dense event annotations are cleaned and retained; videos without event annotations are segmented with TransNetV2 and captioned with Video-LLaVA-7B. We will add the source breakdown to Section 3.1 and Table 4. Generated captions also undergo human review during data processing. Table 15 separately compares original human-written captions and generated captions on the same annotated videos. EventTriggerBench adds target selection, auto-verified plausible trigger labels, multi-trigger sets, and explanatory chains.

SourceVideosEvent segmentsTrigger samples
ActivityNet Captions1,06278643173
MEVA1781072309
Total1,2408,9363,482

The benchmark uses video-disjoint 70/10/20 train/validation/test splits as reported in Appendix D.

[1] Krishna, Ranjay, et al. "Dense-captioning events in videos." Proceedings of the IEEE International Conference on Computer Vision. 2017.

[2] Corona, Kellie, et al. "Meva: A large-scale multiview, multimodal video dataset for activity detection." 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021.

Thanks for the rebuttal. My concerns are addressed; I will maintain my original positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.