acceptodds
Under review as a conference paper at ICLR 2027

Event-STVG: Spatio-Temporal Video Grounding for Long-Form Event Reasoning

Abstract

Spatio-Temporal Video Grounding (STVG) aims to jointly localize a target in space and time given a natural-language query. Despite continued advances, existing STVG benchmarks remain largely centered on localizing targets through instance-specific descriptions, leaving them largely decoupled from long-form reasoning increasingly emphasized in multimodal large language models (MLLMs). In this paper, we introduce Event-STVG, a task that shifts STVG from locating what is described toward event-level reasoning by unifying long-form context, repeated-event reasoning, event localization, role inference, and multi-target grounding. To facilitate research on Event-STVG, we construct a large-scale benchmark comprising 4,016 annotated event occurrences across soccer and seven additional domains. Experiments with state-of-the-art MLLMs reveal substantial difficulty, with the strongest model achieving only 5.4% vIoU at the 10-minute horizon. Across domains, we consistently observe confusion among repeated same-type events, spatial grounding bottlenecks, and rare success in jointly grounding all required roles. Event-STVG establishes a new challenge for visual understanding in MLLMs, calling for future models that can jointly reason about complex events across space and time.

Primary area:
datasets and benchmarks
Keywords:
Spatio-Temporal Video Grounding, Event Understanding, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Kc7VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Event-STVG, a task and benchmark for locating events and their participants in long videos. Given an event type, its order of occurrence, and a set of participant roles, models must locate the event in time and track each participant in space. The benchmark contains 4,016 annotated events from soccer and seven other domains, covering 26 event types and 30 roles. The paper evaluates nine general vision-language models and one specialized STVG model across different video lengths. It finds that models often confuse repeated events and struggle to locate the correct participants, with the best model achieving only 5.4% mean video IoU on 10-minute videos.

Critical strengths
  1. Novel task design. Event-STVG extends traditional video grounding to event reasoning. It requires models to identify specific event occurrences and locate multiple participants in space and time.

  2. Detailed benchmark and analysis. The benchmark covers eight domains with human-verified annotations. The experiments provide detailed analysis of model failures in event selection, spatial grounding, and participant localization.

Critical weaknesses
  1. Imbalanced domain coverage. Soccer accounts for 3,516 of the 4,016 annotated events, while the other seven domains contain only 500 events.

  2. Inconsistent evaluation settings. Different models use different video input settings. For example, GPT-5.6 Sol receives JPEG frames sampled at 2 FPS, while Qwen, GLM, and MiniMax receive native MP4 videos. Since video sampling settings can be controlled through model APIs, a more consistent evaluation would provide a fairer comparison.

  3. Limited baseline comparison. The paper evaluates only one specialized STVG model, PTD. Its adapted input does not preserve the requested event order, which limits the comparison. Moreover, recent agent-based methods for video temporal grounding are not included.

Questions for authors
  1. Could the authors provide results under consistent video input settings across models?

  2. Could the authors include more specialized STVG models and recent agent-based VTG methods as baselines?

4 comments

We thank the reviewer for recognizing the novelty of Event-STVG and the value of its annotations and analysis. We have revised the dataset and evaluation to address the concerns about domain coverage, input consistency, and baseline comparisons.

W1. Domain coverage

We have updated the non-soccer data and unified the context settings across all domains. The revised benchmark contains 4,036 events, 45 event types, and 27 semantic-role labels, comprising 7,342 event–role pairs. The full dataset remains soccer-heavy. However, our revised long-context evaluation and analysis now use 300 soccer events and all 520 non-soccer events, with the following composition:

DomainEventsProportion (%)
Soccer30036.59
Basketball15018.29
Ice hockey15018.29
American football10012.20
Esports303.66
Engineering303.66
Food preparation303.66
Entertainment303.66
Total820100.00

Non-soccer events therefore constitute 63.41% of this cohort, increasing the domain diversity of the evidence supporting our long-context findings.

W2 and Q1. Consistent video inputs

We now provide explicit input configurations in Appendix B.2 and conduct paired input-alignment experiments for both short and long contexts.

For short contexts, we re-evaluate seven native-video models on the same 100 queries using 2-FPS, 512×288 frames under Oracle and one-minute settings (Appendix B.7). Entries below show native-video → aligned-frame m-vIoU (%). An asterisk indicates that the paired change’s source-clustered bootstrap 95% confidence interval excludes zero.

ModelOracle m-vIoU1-min m-vIoU
Qwen3.8-Max12.87 → 27.81*3.36 → 6.43*
MiniMax-M31.08 → 8.51*0.02 → 1.19*
Qwen3.7-Plus3.60 → 21.83*1.07 → 1.99*
GLM-4.6V12.73 → 10.680.83 → 0.74
GLM-5V-Turbo9.60 → 14.19*1.40 → 1.26
GLM-5.3-Flash11.40 → 12.131.31 → 1.45
Qwen3.5-9B3.36 → 3.590.11 → 0.34

Input alignment improves Oracle grounding for four models and one-minute grounding for three; the remaining m-vIoU changes are inconclusive. Nevertheless, all six models excluded from the main long-context evaluation remain below 2% one-minute m-vIoU after alignment. For temporal localization, MiniMax improves by 22.19 pp; the other models’ m-tIoU changes are inconclusive.

For long contexts, we evaluate Qwen3.8-Max on a separate 100-query subset using Gemini’s exact frame protocol: 512×288 frames at 2 FPS for three and five minutes, and 900 uniformly sampled frames for ten minutes (Appendix B.8).

DurationNative m-vIoU (%)Aligned m-vIoU (%)Change (pp)95% CI
3 min2.884.37+1.49[−0.61, +3.69]
5 min3.113.72+0.61[−2.18, +3.28]
10 min1.912.16+0.25[−2.00, +2.76]

All long-context m-vIoU and m-tIoU differences have confidence intervals containing zero. Joint grounding remains limited under aligned inputs. Moreover, neither input format achieves complete grounding of all roles on the 40 multi-role queries in this subset at three, five, or ten minutes.

W3 and Q2. Additional specialized and agentic baselines

We add STVG-R1-7B, VideoMind-7B, and VideoTemp-o3, evaluating all three on the full 4,036 events under Oracle and one-minute settings. All receive complete event definitions, occurrence indices, and role specifications (Appendix B.12).

STVG-R1 selects role identities from tracked candidates. VideoMind retains its Planner/Grounder/Verifier, and VideoTemp retains its native agent loop. Both agentic methods use a shared YOLOv12x–SAM2.1–Qwen2-VL spatial backend to produce role trajectories.

MethodOracle m-vIoU (%)1-min m-tIoU (%)1-min m-vIoU (%)
STVG-R1-7B15.498.820.84
VideoMind-7B6.6811.610.32
VideoTemp-o36.6811.790.37

The two agentic methods share the supplied interval and spatial backend under Oracle, explaining their identical Oracle scores.

To verify agent execution, we additionally test VideoTemp-o3’s tool workflow, since it did not invoke optional tools during Event-STVG evaluation. Under the same frozen deployment, we evaluate 30 Video-MME questions through each prompt path:

Prompt pathQuestions invoking toolsSuccessful cropsCrop feedback processed
Oracle path28/3041/4541/41
1-minute path26/3028/2928/28

These checks confirm that autonomous tool invocation, cropping, and feedback processing are functional in the evaluated deployment.

We also directly test PTD’s interface limitation on 100 soccer queries (Appendix B.11). Providing full task instructions reduces one-minute m-vIoU from 2.10% to 1.22%, while valid native outputs fall from 180/183 to 51/183. We retain its native referring interface while acknowledging this limitation.

Dear Reviewer Kc7V,

We have uploaded a revised version of the manuscript addressing the questions and concerns raised in your review. Thank you for your thoughtful feedback. We hope these revisions address your concerns and strengthen the evaluation of Event-STVG. We welcome any further comments or suggestions.

I appreciate the authors' rebuttal, which clarifies several points. Overall, I will keep my positive score of a weak accept.

Dear Reviewer Kc7V,

Thank you for following up after reading our rebuttal. We’re glad our responses helped clarify several points, and we appreciate your helpful feedback.

Reviewer_vhwfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Event-STVG, a task in which a templated query specifies an event type, an ordinal occurrence, and a set of semantic roles. The model must return the event interval and one box track per role. The benchmark contains:

  • 3,516 soccer events from SoccerNet-v2, evaluated both with the event window given (Oracle) and in 1-minute clips;
  • a 300-event subset re-cut at 3, 5, and 10 minutes;
  • 500 events in 100 videos from seven other domains.
Critical strengths

S1. The annotation resource. It comprises 4,016 events with human-verified role trajectories (59,381 boxes, 7,323 event–role pairs). If released, occurrence-indexed, multi-role tracks in broadcast sports would be reusable. This is the paper’s main asset.

Critical weaknesses

W1. The benchmark does not mainly measure what it claims (long-form, repeated-event, and role reasoning), and its delta from existing sports spatio-temporal benchmarks is unstated.

  • Long-form: 3,516 of the 4,516 evaluation videos (78%) are 1-minute clips. Table 1 reports a mean duration of 121.5 s, below the 180 s it lists for LF-STVG.
  • Repeated-event reasoning: only 13.68% of 1-minute soccer contexts contain a repeated same-type event (Sec. 3.3). About 86% of main-benchmark queries therefore ask for an event that occurs once.
  • Role inference is never measured on its own. The score is box overlap only (App. A), so correctly identifying the shooter but boxing him loosely scores the same as picking the wrong player. The pipeline already tracks every player (SoccerMaster DETR, StrongSORT; Sec. 3.2), so asking models to choose the role filler from these tracks would isolate it.
  • Positioning: spatio-temporal action detection in the same sports is not discussed. MultiSports (arXiv:2105.07404) localizes actions in time and space across basketball, volleyball, football, and aerobic gymnastics, and SportsHHI (arXiv:2404.04565) grounds interacting player pairs in basketball and volleyball. The claim that no prior benchmark jointly grounds an event in time and its role fillers in space (l.151–152) needs its actual delta stated: ordinal selection, multiple named roles, and an MLLM interface.
  • Why consequential: the task’s value rests on its claimed construct. By the paper’s own statistics, most of the benchmark tests single-occurrence, 1-minute event localization plus box drawing.

W2. The evaluation cannot support the diagnostic conclusions, and the findings that remain are expected.

  • Baselines: all ten systems generate boxes as text from raw frames. There is no detector- or tracker-assisted zero-shot pipeline, e.g., Grounding DINO (arXiv:2303.05499) or the authors’ own detector and tracker for players, with the MLLM selecting the role identity as STVG-R1 (arXiv:2602.11730) does by choosing among SAM2-tracked object IDs. There is also no trained STVG model, since there is no training split, and PTD receives role queries without the ordinal (App. C.5).
  • Inputs:
    • GPT and Gemini receive 512×288 frames (Table C.1). At that size, the paper’s own Fig. 2 example boxes measure about 18×41 px (shooter) and 15×30 px (goalkeeper).
    • Resolution is never varied; App. C.7 toggles only GPT’s detail flag.
    • Gemini’s input at 3, 5, and 10 minutes is unspecified, and Qwen, GLM, and MiniMax receive provider-sampled MP4.
  • Bottleneck I: Sec. 4.3 concludes that the primary long-context bottleneck is resolving which occurrence is meant (l.415–417). App. D.1.1 shows that from 1 to 10 minutes, same-type mismatches rise by 13.40/15.79/19.62 pp while hit rates fall by 43.54/21.05/44.98 pp. Confusion therefore accounts for at most 31%/75%/44% of lost hits. App. D.4 adds that only 23.91%/54.93%/38.05% of valid 10-minute misses overlap another same-type event. The claim holds for GPT only.
  • Bottlenecks II–III: GapSG mixes wrong-entity errors with imprecise-box errors, as App. D.5 acknowledges. What remains is already established: longer videos are harder (Table 3), and text-box MLLMs localize poorly in video. STVG-o1 (arXiv:2511.21375), for instance, reports GPT-4o at 7.9 mvIoU on HCSTVG-v1.
  • No ceiling: there is no human performance on the task as posed. Sec. 3.2’s statement that the authors “independently cross-annotate” (l.254–255) is contradicted by App. E.5.5, where reviewers saw the original labels and the same track candidates. Why consequential: without detector-assisted baselines, a resolution control, or a human ceiling, the low scores cannot be attributed to event reasoning, which is the benchmark’s stated purpose.
Questions for authors

Experiments that could help clarify more about the work:

  • What do detector- or tracker-assisted zero-shot baselines achieve, along with an identity-selection variant built on the provided player tracks?
  • What is human performance when independent annotators are given only the query and the clip?
  • Please run at least one model at higher input resolution, and report frames and resolution per model per duration, including Gemini at 3, 5, and 10 minutes.

Clarifications:

  • Where is the target placed within each 1/3/5/10-minute context: at a fixed or a random offset? How many unique matches and unique contexts underlie the 4,516 evaluation videos?
  • Are category definitions given to models? For example, do blocked shots or scoring free kicks count as shots on target (App. E.2.3)? Will a training split be released?
5 comments

We thank the reviewer for their careful reading and critical assessment of our work.

Most importantly, we clarify the distinction between the full benchmark and the experimental subsets, which is central to W1.1 and W1.2. For soccer, the current benchmark has already included all 3,516 events under all four context durations: 1, 3, 5, and 10 minutes (see Appendices E.3.3 and C.1 of the original submission). To align other domains with soccer, our full multi-domain benchmark is updated to 4,036 events across these four settings: (1) it supports STVG evaluation of all events from 1 to 10 minutes, with a mean context duration of 285 seconds; (2) 57.11% of events require multi-role grounding; and (3) the proportions of contexts containing repeated same-type events are 15.63%, 34.56%, 47.84%, and 68.68% for the 1-, 3-, 5-, and 10-minute settings, respectively.

These statistics describe the full benchmark resource, rather than the subset selected for a particular experiment in this paper. We have revised the relevant descriptions and statistics to make both scopes explicit, and hope this clarification helps reassess the benchmark’s coverage of long-context and repeated-event grounding.

We thank the reviewer for W1.4.

We have added MultiSports, SportsHHI, and other work to Table 1 and narrowed our claim to: “To our knowledge, no prior benchmark requires models to reason over repeated event occurrences and jointly localize a selected event and its multiple semantic-role trajectories in minute-scale video contexts.” This clarified positioning, together with the full-benchmark statistics reported above, supports Event-STVG’s claimed contribution.

In response to W1.3, W2.1, and Q1, we further evaluate track-assisted methods and directly assess semantic-role identification.

Track-assisted baselines. We evaluate three additional methods on all 4,036 events in the oracle and 1-min settings.

MethodGrounding pipeline
STVG-R1-7BPredicts event boundaries and role IDs; selected SAM2 tracks provide box trajectories.
VideoMind-7BRetains its Planner/Grounder/Verifier; adds a YOLOv12x–SAM2.1–Qwen2-VL spatial backend.
VideoTemp-o3Retains its temporal agent; uses the same spatial backend as VideoMind.

Results (%):

MethodOracle m-vIoU1-min m-tIoU1-min m-vIoU
STVG-R1-7B15.498.820.84
VideoMind-7B6.6811.610.32
VideoTemp-o36.6811.790.37

The two agentic methods share their Oracle spatial backend, explaining their identical Oracle scores. Low grounding performance therefore also occurs with these track-assisted pipelines.

Role identification through candidate-ID selection. We evaluate a fixed, event-stratified cohort of 300 soccer queries covering 17 event categories and 48 matches. Models receive Oracle clips and numbered SoccerMaster candidates, manually supplemented where necessary, and select each role’s ID every 0.5 seconds instead of generating boxes. We also evaluate three human participants on a matched subset of 30 queries.

ModelID Accuracy (%)Joint ID Accuracy (%)
Gemini-3.7 Flash62.9548.56
GPT-5.6 Sol45.0329.90
Qwen3.8-Max47.2633.77

ID Accuracy measures correct selections over GT-visible role–frame pairs; Joint ID Accuracy requires all visible roles at a timestamp to be correct. On a matched 30-query subset, humans achieve 96.80% ID Accuracy, compared with Gemini’s 69.65% (Appendix B.14). Notably, role-identification errors remain even when temporal retrieval and free-form box generation are removed.

In response to W2.2 and Q3, we clarify the input settings and add resolution and input-format controls.

Input settings. Appendix B.2 in the updated paper now specifies each model’s configuration. GPT uses 512×288 frames at 2 FPS: 120/360/600/1,200 frames for 1/3/5/10 minutes, with detail=low. Gemini uses the same resolution and 120/360/600/900 frames; its ten-minute setting uses approximately 1.5 FPS to accommodate input limits. Qwen, GLM, and MiniMax originally receive native 1280×720 MP4 inputs with provider-side preprocessing.

Higher resolution. We evaluate Gemini on the same 100 one-minute queries, stratified by target-box size across all eight domains (Appendix B.5). Scores are percentages; confidence intervals use source-clustered bootstrap.

Metric512×2881280×720Change (pp)95% CI
m-tIoU45.0450.46+5.42[−0.09, +10.88]
m-vIoU13.2416.48+3.24[−1.05, +7.54]

Higher resolution improves the point estimates, yet both confidence intervals include zero. Joint grounding remains limited at the higher resolution.

Input alignment. We re-evaluate seven native-video models on the same 100 queries using 2-FPS, 512×288 frames (Appendix B.7). Entries show native-video → frame-input m-vIoU (%).

ModelOracle1 minute
Qwen3.8-Max12.87 → 27.813.36 → 6.43
MiniMax-M31.08 → 8.510.02 → 1.19
Qwen3.7-Plus3.60 → 21.831.07 → 1.99
GLM-4.6V12.73 → 10.680.83 → 0.74
GLM-5V-Turbo9.60 → 14.191.40 → 1.26
GLM-5.3-Flash11.40 → 12.131.31 → 1.45
Qwen3.5-9B3.36 → 3.590.11 → 0.34

Input format affects performance, particularly under Oracle. Nevertheless, all six models excluded from our long-context evaluation remain below 2% 1-minute m-vIoU after alignment.

We also evaluate Qwen3.8-Max on 100 long-context queries using Gemini’s exact frame-input protocol (Appendix B.8):

DurationNative m-vIoUAligned m-vIoU95% CI for change (pp)
3 min2.884.37[−0.61, +3.69]
5 min3.113.72[−2.18, +3.28]
10 min1.912.16[−2.00, +2.76]

The paired differences are inconclusive, and long-context joint grounding remains difficult under aligned inputs.

We tested PTD with full task instructions on 100 soccer queries as well (Appendix B.11). Its one-minute m-vIoU decreased from 2.10% to 1.22%, while valid native outputs fell from 180/183 to 51/183. We therefore retain its native referring-query interface while acknowledging this limitation.

In response to the suggestions in W2.3–W2.4, we have reframed Section 4.3 as a diagnostic analysis of m-vIoU. Following prior work that distinguishes temporal and spatial grounding, we further separate whether the requested event is reached from how accurately it is temporally aligned once reached. This yields the exact decomposition:

m-vIoU=H⋅TH⋅Sw,\mathrm{m\text{-}vIoU}=H\cdot T_H\cdot S_w,

where HH is the target-hit rate, defined as the proportion of predictions with nonzero temporal overlap with the ground-truth target event (tIoU>0\mathrm{tIoU}>0), THT_H is temporal alignment conditional on a hit, and SwS_w is temporally weighted joint-spatial quality. This preserves the original metric without assuming independent cognitive components.

On the targets across long durations, the decomposition reveals a pronounced decline in target-hit rate and comparatively smaller changes in conditional temporal alignment. Same-type confusion is analyzed separately as one source of target misses, rather than equated with all misses.

We further complement this decomposition with complete multi-role grounding and candidate-ID selection. The latter directly evaluates role identification with event intervals and candidate boxes supplied, addressing the ambiguity between incorrect entity selection and imprecise box generation. Together, these diagnostics examine distinct aspects of occurrence-indexed, multi-role grounding beyond aggregate low scores.

In response to W2.5 and Q2, we have corrected the description of annotation agreement: it measures consistency under assisted review, not fully independent annotation agreement (Appendix C.8.1).

Besides the human baseline of ID selection above, we additionally conduct an independent end-to-end human evaluation (Appendix B.9). We select 30 ten-minute queries spanning different occurrence orders and role counts, independently of model predictions or scores. Six annotators independent of the research team participate, with three per resolution setting. Each evaluates the same queries without access to ground-truth event intervals or candidate tracks.

Input resolutionMean hit rate (%)Mean m-tIoU (%)Mean m-vIoU (%)
Original95.5684.9066.85
512×28897.7884.5068.47

These results provide a direct human-performance reference for the task and show that humans can successfully perform long-context event and role grounding even at the reduced resolution.

We address Q4–Q5 below:

Context construction and counts. For soccer, crop starts are sampled uniformly from the feasible range containing the complete target event, separately for each duration. Targets are not fixed at the center, and windows need not be nested. Other domains use randomized starts subject to event-completeness and domain-specific ordinal constraints. The revised benchmark uses 50 soccer matches (100 half-match timelines) and 56 additional source recordings. Deduplicating by source-timeline ID and crop boundaries yields the following counts, indicating a high proportion of unique contexts:

DurationQuery-input recordsUnique contexts
1 min4,0364,034
3 min4,0364,032
5 min4,0364,033
10 min4,0364,024
Total16,14416,123

Category definitions. Yes. General-purpose VLMs receive event definitions, role definitions, semantic boundary rules, and the requested occurrence index. Context prompts do not provide ground-truth event timestamps or instance descriptions. We verified identical task text across GPT, Gemini, and Qwen for the original 300-query ten-minute evaluation. A blocked shot qualifies as Shot on Target if it meets our on-target criterion. A scoring direct free kick qualifies as both Shot on Target and Goal. Occurrence indices are computed separately for each category (original Appendix E.2.3).

Training split. Event-STVG is an evaluation-only benchmark. We do not plan to provide a training split.

Reviewer_rr7ROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on the task of spatio-temporal video grounding (STVG), specifically moving beyond queries that describe the target directly toward queries that require reasoning about events in long videos. To this end, the authors introduce Event-STVG, a task where the input query specifies only an event type, which occurrence is wanted, and which semantic roles to ground. Models must return the time span of that occurrence along with a box trajectory for each role. The benchmark contains 4,016 human-verified event occurrences, and experiments with frontier multimodal models and an STVG specialist show the task is quite challenging, with the best model reaching only 5.4% m-vIoU at 10 minutes. An error analysis finds that models often confuse repeated occurrences of the same event, and they rarely ground all of its roles correctly even when they find the right one.

Critical strengths
  • The task is well motivated. Asking for event-role grounding without descriptions requires telling repeated events apart and inferring who plays each role, and is something I have not seen in other similar work.
  • Tracker-proposed boxes are human-verified and overall seem of high quality, with the cross-annotated sample achieving higher than 0.90 for both temporal and spatial IoU.
Critical weaknesses
  • The low scores may partly reflect the input setup rather than a lack of event reasoning. GPT and Gemini receive frames at 512×288 (Table C.1), so the spatial grounding bottleneck in Section 4.3 may be partly a resolution effect. The detail=low comparison in Appendix C.7 keeps the resolution fixed, so the authors don’t seem to have tested this.
Questions for authors
  • The example in Figure 2 seems tractable for a person to do, if somewhat time-intensive, but did the authors test how well humans can perform this task? Given the extremely low scores it seems reasonable as a sanity check.
4 comments

We thank the reviewer for recognizing the motivation and annotation quality of Event-STVG. We agree that input resolution and human performance are important controls for interpreting the model scores.

W1. Effect of input resolution

We agree that our original detail=low comparison kept frame resolution fixed and therefore did not directly test resolution. We now evaluate Gemini-3.7 Flash at 512×288 and 1280×720 on the same 100 one-minute queries, stratified by target-box size across all eight domains (Appendix B.5). We report source-clustered bootstrap 95% confidence intervals.

Metric512×2881280×720Change (pp)95% CI
m-tIoU (%)45.0450.46+5.42[−0.09, +10.88]
m-vIoU (%)13.2416.48+3.24[−1.05, +7.54]

Higher resolution improves both point estimates, suggesting that resolution may contribute to performance. However, both confidence intervals include zero, and m-vIoU remains 16.48% at 1280×720. We therefore acknowledge input sensitivity while observing that substantial joint-grounding difficulty persists at higher resolution on this subset.

Q1. Human performance

We now include an independent end-to-end human evaluation (Appendix B.9). We select 30 ten-minute queries spanning different occurrence indices and role counts, independently of model predictions and scores. Six annotators independent of the research team evaluate the same queries without access to ground-truth event intervals or candidate tracks, with three annotators per resolution setting.

Results are averaged across the three annotators in each setting.

Input resolutionMean hit rate (%)Mean m-tIoU (%)Mean m-vIoU (%)
Original95.5684.9066.85
512×28897.7884.5068.47

These results provide the requested human-performance reference and show that humans can successfully perform occurrence-indexed event and role grounding in ten-minute videos, including at 512×288.

Accordingly, Section 4.3 now presents a diagnostic analysis rather than attributing all low scores to event reasoning alone. The new controls show that resolution affects the point estimates, while substantial gaps remain in event retrieval, role identification, and spatial grounding.

Dear Reviewer rr7R,

We have uploaded a revised version of the manuscript addressing the question raised in your review. Thank you for your thoughtful feedback, and we welcome any further comments or suggestions.

Thank you for your response, and for your detailed response to the other reviewers. Upon review I will keep my positive score of a weak accept.

Dear Reviewer rr7R,

Thank you for taking the time to read our rebuttal and our responses to the other reviewers. We truly appreciate your recognition of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Clarifications

Dear AC and all reviewers,

Here is a clarification of the key differences between the original manuscript and the revised version:

Key revisionDescription of the changesLocation in the revised manuscript
1. Dataset composition and reconstructionThe revised benchmark contains 4,036 unique target events: 3,516 soccer events and 520 events across basketball, ice hockey, American football, engineering, food preparation, entertainment, and esports. The non-soccer component has been reconstructed, with changes to its domain composition, sources, and event/role definitions; this is more than a net increase of 20 events. All context videos now use contiguous source-video crops, whereas the original construction allowed concatenated clips. Table 1 reports the same 4,036 targets across four context-duration settings (1/3/5/10 minutes), yielding 16,144 context-specific instances, not 16,144 independent target events.Table 1, p. 3; §§3.2–3.3, pp. 4–5; Appendix C, pp. 42–48
2. Evaluation sets and updated resultsThe main short-context evaluation now covers all 4,036 events, compared with 3,516 soccer events in the original main table. We also expand this evaluation from ten to thirteen models and methods by adding three baselines: STVG-R1-7B, VideoMind-7B, and VideoTemp-o3. The long-context evaluation expands from 300 soccer events to 820 events, comprising 300 soccer events and 520 events from the additional domains. Table 4 reports results for the same 820 targets across Oracle and all four context durations. These evaluated subsets should be distinguished from the full benchmark construction reported in Table 1. Scores computed on different evaluation sets are not directly comparable between versions. The original ten models’ one-minute soccer m-vIoU results are retained in Table 3.Tables 2–4, pp. 6–7; §B.1, p. 19
3. Annotation statistics and quality-control updatesWe have updated the statistics to describe the revised benchmark: event types increase from 26 to 45, role labels change from 30 to 27, and annotated bounding boxes change from 59,381 to 58,915. The revised consistency assessment covers 406 events, including 354 soccer events and 52 events from the additional domains. We also clarify the assisted-review procedure and distinguish jointly visible box IoU from frame-level micro IoU that counts one-sided missing boxes as zero. Differences in review samples and aggregation methods mean that the original and revised agreement scores should not be interpreted as directly comparable measures of improvement.§3.3, p. 5; Appendix C, particularly Table C.5, p. 47
4. Changes to the diagnostic analysisWe have replaced the original GapSG-based analysis with an exact decomposition of the existing m-vIoU metric into target-hit rate, temporal alignment conditional on a hit, and temporally weighted joint-spatial grounding quality. This decomposition preserves the metric definition and supports a more detailed analysis of long-context degradation, confusion between repeated event occurrences, and semantic-role grounding.§4.3, pp. 8–10
5. Additional experimentsWe have added a frame-resolution ablation (512×288 vs. 1280×720), input-format alignment experiments comparing native video and standardized frame-based inputs in both short- and long-context settings, a candidate-ID role-selection experiment, and human-performance baselines. These experiments examine sensitivity to resolution and input format, the difficulty of semantic-role identification, and the gap between model and human performance. Detailed experimental settings and analyses are provided in Appendix B.§4; §§B.5, B.7–B.9, B.14–B.15

The revised manuscript preserves the original task formulation and core research focus while addressing the reviewers’ feedback through clearer descriptions of the benchmark construction, additional experiments, and more detailed analyses. We hope these revisions clarify the novelty and contributions of Event-STVG and make our analysis more robust and insightful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.