acceptodds
Withdrawn from ICLR 2027

When, Where and What to Comment: A Clue-grounded Benchmark for Sports Video Commentary

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-world sports commentary requires more than generating descriptions from video clips. A competent commentator must continuously determine when an event deserves commentary, identify where the supporting visual evidence occurs, and decide what to say based on the event and its competitive significance. Existing benchmarks largely reduce this process to text generation from pre-segmented clips, implicitly specifying commentary timing and weakening the need for fine-grained visual grounding. We introduce CG-SC-Bench, a clue-grounded benchmark for sports commentary that evaluates these three capabilities in a unified framework. Each commentary instance is associated with explicit visual clues, enabling the benchmark to assess whether a model identifies commentary-worthy moments (When), localizes the evidence supporting the commentary (Where), and generates factually consistent commentary conditioned on the observed clues (What). CG-SC-Bench covers 8 major sports categories and 27 subcategories, comprising 2,684 commentary segments with fine-grained clue annotations. Based on these annotations, we construct three complementary evaluation tasks for commentary timing, visual clue localization, and clue-conditioned commentary generation. We evaluate a broad range of closed- and open-source vision-language models on CG-SC-Bench. The results reveal substantial limitations of current models in temporal decision-making, fine-grained visual grounding, and evidence-grounded factual generation, showing that fluent sports descriptions do not necessarily translate into reliable continuous commentary. CG-SC-Bench provides a diagnostic evaluation framework for developing sports video models that can comment at the right time, ground their statements in the right evidence and say the right thing.

Primary area:
datasets and benchmarks
Keywords:
Video Understanding, Benchmark, Sports Video Commentary

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_4SutOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CG-SC-Bench evaluates sports video commentary through commentary-to-spatiotemporal grounding, causal streaming commentary decisions, and visual-clue-conditioned generation. It connects commentary segments to timestamps and bounding boxes across eight sports categories. Experiments examine the grounding, timing, and factual-generation limitations of multimodal models.

Critical strengths
  1. Evaluating when commentary is warranted, alongside visual grounding and generated content, addresses a meaningful limitation of pre-segmented commentary tasks.

  2. Fine-grained links between commentary, timestamps, and bounding boxes across sports provide a useful resource for diagnosing temporal, spatial, and factual errors.

Critical weaknesses
  1. Trigger reference. Human-commentary onset measures agreement with a narration schedule, not every reasonable commentary-worthy moment. Valid alternative timings or omissions should be distinguished from errors when interpreting the When capability.

  2. MTE definition. Section 3.3.1 describes tolerance-constrained matching, yet Table 5 reports MTE above the largest tolerance. This would be inconsistent under that same matching set; a separate unconstrained rule could explain it but must be specified.

  3. Commentary-judge validity. Human checks of dataset annotations do not establish judge reliability for generated commentary. Independent agreement on factuality and visual support would directly support the benchmark's conclusions.

  4. Linked-task analysis. The aligned resource is useful. Showing which findings require linked timing, grounding, and generation annotations would make its diagnostic contribution more concrete than separate task scores.

Questions for authors
  1. How does the trigger reference handle plausible alternative commentary timings?
  2. Which matching set and denominator produce the reported MTE?
  3. What independent agreement supports generated-commentary judgments?
  4. Which diagnostic findings rely on the linked annotations?
Reviewer_Kr5ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CG-SC-Bench, a clue-grounded benchmark for evaluating sports video commentary across three tasks: commentary-to-spatiotemporal grounding, causal streaming commentary decision-making, and visual-clue-conditioned commentary generation. The benchmark covers eight major sports categories and 27 subcategories, comprising 72 videos, 2,684 commentary segments, and 4,943 visual evidence bounding boxes. The authors evaluate 12 vision-language models and report substantial limitations in temporal decision-making, spatial grounding, and factual consistency. The benchmark addresses an important gap in sports commentary evaluation, but its limited video coverage, evaluation reliability, and task-design assumptions require further examination.

Critical strengths
  1. The paper addresses an important gap by jointly evaluating when to comment, where the supporting visual evidence occurs, and what commentary should be generated.
  2. The dataset provides fine-grained temporal and spatial annotations, linking commentary segments to relevant athletes, officials, scoreboards, and other visual evidence.
  3. The three evaluation settings disentangle visual grounding, commentary triggering, and evidence-conditioned generation, enabling more informative failure analysis than text-only evaluation.
  4. The benchmark covers eight sports categories and evaluates 12 open- and closed-source vision-language models, providing a useful initial assessment of current capabilities.
  5. The results reveal substantial gaps between linguistic fluency and factual grounding, highlighting important challenges for real-time sports commentary systems.
Critical weaknesses
  1. The benchmark contains only 72 videos, with highly uneven coverage across sports, including just one pole-vault video, limiting the reliability of sport-specific comparisons and claims of broad generalization.
  2. The evaluation relies heavily on GPT-based judges for commentary quality and factuality, but the paper provides insufficient validation against independent human assessments or inter-annotator agreement.
  3. Uniform sampling at 1 FPS may miss brief actions and rapid interactions, potentially limiting the validity of fine-grained temporal grounding and commentary-timing evaluations.
  4. Some reference commentaries contain historical records, athlete background information, or retrospective descriptions that cannot necessarily be inferred from the provided visual evidence, potentially confounding visual grounding with access to external knowledge.
  5. The human preference study uses four evaluators and assigns each pair to a single evaluator, leaving inter-rater reliability and the statistical uncertainty of the reported preference margins unclear.
  6. The streaming evaluation uses previously generated commentary as context, but the effects of accumulated generation errors and differences in historical context on subsequent triggering and content quality are not sufficiently analyzed.
Questions for authors
  1. Can the authors provide additional videos for underrepresented sports and report uncertainty estimates for sport-specific results, particularly for categories represented by very few videos?
  2. How well do the GPT-based commentary-quality and factuality scores correlate with independent human judgments, and can the authors report inter-annotator agreement on a representative subset?
  3. Can the authors evaluate whether higher frame-sampling rates improve temporal and spatial grounding, especially for fast actions and brief athlete interactions?
  4. How are retrospective commentary, historical facts, and statements requiring external knowledge handled when evaluating factual grounding from visual evidence alone?
  5. Can the authors quantify the effect of previously generated commentary on subsequent triggering decisions and evaluate robustness to accumulated errors in long streaming sequences?
  6. Can the authors clarify the distinction between commentary timing accuracy and evidence availability, particularly for commentary segments whose supporting visual evidence appears before or after the annotated utterance?
Reviewer_rEoZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether AI models can give reliable sports commentary by choosing when to speak, finding the right visual evidence, and describing events correctly. Many earlier benchmarks use short, selected clips, which already tell the model when something important happens. They also do not fully check whether the commentary is supported by the video. To address these gaps, the paper introduces CG-SC-Bench, which includes over 12 hours of video from eight sports and 2,684 commentary segments. It tests whether models can find the evidence for a sentence, decide when and what to say while watching a video, and describe specific people or objects marked by boxes. The best tested model achieves 16.98% on the combined time and box matching score, 33.48% F1 for commentary timing, and 21.30% on an LLM-based factuality score. These results suggest that current models still have difficulty giving accurate commentary at the right time.

Critical strengths
  1. The paper studies a useful problem: giving correct sports commentary at the right time.
  2. The three tasks help show whether a model struggles with timing, finding evidence, or describing events.
  3. The dataset covers eight sports and links commentary to specific times and image regions.
Critical weaknesses
  1. The timing score depends on when the human commentator speaks. A model may choose another reasonable time and still lose points
  2. The paper uses an LLM to judge content quality but provides limited evidence that its scores agree with human judgements.
  3. Finding evidence for a given sentence does not prove that the model used that evidence when writing its own commentary.
Questions for authors
  1. Can the evaluation accept several reasonable times to describe the same event?
  2. How closely do the LLM's scores agree with human scores?
Reviewer_MsWVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CG-SC-Bench for sports commentary, covering 72 videos across eight sports categories and 27 subcategories. The dataset contains 2,684 timestamped commentary segments, of which 2,168 have visual evidence annotations comprising 4,943 boxes. Three settings evaluate commentary-to-visual grounding, causal commentary triggering and generation, and generation conditioned on specified visual boxes. The paper evaluates twelve VLMs and reports substantial limitations in temporal/spatial grounding, trigger precision/recall, and factual commentary generation. The intended contribution is a diagnostic benchmark connecting when to comment, where the evidence occurs, and what can be said from that evidence.

Critical strengths

The three settings usefully separate evidence localization, commentary triggering, and evidence-conditioned generation. Timestamped bounding boxes add a valuable spatiotemporal grounding dimension beyond conventional commentary-quality evaluation.

The benchmark covers diverse competition structures and twelve models. Reporting triggering performance alongside matched timing error and quality-aware metrics supports more informative comparisons.

Critical weaknesses

1.Expected knowledge may exceed the available inputs. Fig.1 includes “They ran beautifully yesterday,” which cannot be inferred from current footage alone. It is unclear whether sufficient background information is provided or whether such claims are excluded from scoring requirements. Consequently, low factuality scores may reflect unavailable knowledge rather than poor understanding. The discussion should better connect to specialized sports understanding and knowledge-enhanced commentary, e.g. MatchVision [1] and GameSight [2], to clarify the benchmark’s scope.

2.The experiments provide limited diagnosis of model failures. Uniform 1 FPS sampling may discard crucial motion and identity cues. Frame-rate ablations and a high-frame-rate baseline such as F-16 [3] would help separate input limitations from model limitations. Moreover, only 72 videos are included, with uneven coverage across sports, making video-level confidence intervals important.

3.The evaluation protocol needs stronger validation. Tab. 5’s MTE values exceed the stated grounding tolerances, requiring clarification of the matching procedure. The streaming window and allowed output timestamps are also insufficiently specified to verify causality. Representative evidence frames and individual human commentary onsets may penalize alternative valid predictions; annotation agreement and a human audit would strengthen the conclusions.

4.The title of Sec 3.3.3 incorrectly describes commentary-to-grounding rather than evidence-conditioned commentary generation.

References:

[1] Jiayuan Rao, Haoning Wu, Hao Jiang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards Universal Soccer Video Understanding. CVPR, 2025. arxiv.org

[2] Zeyu Jin, Xiaoyu Qin, Songtao Zhou, Kaifeng Yun, and Jia Jia. Towards Automatic Soccer Commentary Generation with Knowledge-Enhanced Visual Reasoning. ICME, 2026. arxiv.org

[3] Yixuan Li, Changli Tang, Jimin Zhuang, Yudong Yang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. Improving LLM Video Understanding with 16 Frames Per Second. ICML, 2025.

Questions for authors
  1. How are historical or statistical claims handled when their supporting information is absent from the model inputs?
  2. How do results change with higher frame rates, and are model rankings stable under video-level uncertainty estimates?
  3. Can you clarify MTE matching and streaming input/output boundaries, and quantify how often unmatched predictions are nevertheless valid?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.