acceptodds
Withdrawn from ICLR 2027

Towards Comprehensive Basketball Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding a basketball game requires recognizing events, localizing actions, identifying players, and relating these to structured game knowledge. Existing benchmarks primarily evaluate these abilities one at a time, leaving the interactions among these abilities under-explored. We introduce BasketballBench, a multimodal benchmark comprising 7,980 questions across ten tasks in text, image, and video. It is built from the 2025-2026 NBA season and includes official playby-play, rosters and profiles for 530 active players, and 2,501 possession-level broadcast clips. We further propose BasketballSkills, an agent that composes eight basketball-specific perception and retrieval tools under four reusable skills that specify tool order, evidence bindings, and stopping conditions. Experiments show that current MLLMs struggle particularly on questions requiring the integration of multiple capabilities, whereas BasketballSkills outperforms them, highlighting the effectiveness of explicitly composing domain-specific capabilities for comprehensive basketball understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Reasoning, Video Understanding, Multimodal Benchmarks, Sports Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DvjrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BasketballBench, a benchmark of 7,600 questions in ten basketball tasks over text, images and videos. The data come from NBA broadcast clips, a database of NBA records, and player photos. The tasks test basketball knowledge, perception of players and broadcasts, and understanding of events in space and time. The paper evaluates nine MLLMs and finds that they can often recognize events but fail to identify which player performed them. It also proposes BasketballAgent, which uses vision tools, database queries and manually written skills, and reports the best results on seven of the ten tasks.

Critical strengths
  1. Broad task coverage. The benchmark covers different types of basketball tasks, ranging from basic knowledge to visual and event understanding. The tasks are well organized under a unified evaluation framework.

  2. Strong agent baseline. BasketballAgent achieves good performance across the proposed tasks and obtains the best results on seven of the ten tasks.

Critical weaknesses
  1. Limited comparison with existing benchmarks. The paper lacks a systematic comparison with other basketball and sports benchmarks in terms of dataset size, task types, and coverage.

  2. Limited value of some tasks. Q4 and Q9 mainly test visual text recognition, and many models already achieve high scores, especially on Q9. Moreover, Q9 uses player crops rather than requiring actual video understanding, which makes its inclusion as a VideoQA task less convincing.

  3. Lack of agent efficiency analysis. BasketballAgent relies on complex tools and manually designed skills, but its execution efficiency remains unclear.

  4. Limited depth of basketball understanding. Most tasks focus on factual knowledge or basic visual perception. Even Q7, Q8, and Q10 mainly involve event localization, player identification, and knowledge retrieval. In contrast, SoccerBench (Rao et al., 2025) also covers action classification and commentary generation. The lack of tasks involving more complex actions, tactics, and strategy weakens the claim of comprehensive basketball understanding.

Questions for authors
  1. Could the authors provide a more complete efficiency analysis of BasketballAgent, including latency, token usage, and inference cost?

  2. Could the authors provide a systematic comparison with existing basketball and sports benchmarks in terms of scale and task coverage?

  3. Could the authors evaluate general agent systems, such as DeepSeek-V4-Flash with OpenCode or DeepseekHarness, on BasketballBench? A comparison under similar tool and database access would help clarify the benefits of the custom agent design.

Reviewer_fwK8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a basketball benchmark that includes 7600 questions from 2501 broadcast clips and records. It also proposed that BasketballAgent combines specialized perception and knowledge retrieval to achieve the best available results on seven of the ten benchmark tasks.

Critical strengths

1 The paper proposes a basketball benchmark includes 7600 questions.

2 The proposed BasketballAgent performs well on the introduced benchmark, showing the usefulness of the agent framework.

Critical weaknesses

1 The whole paper structure, including the benchmark and agent framework, is really similar to the existing work [1], which also proposed a benchmark called SoccerBench and a multi-agent system called SoccerAgent, but for soccer. The authors should emphasize and elaborate on the technical differences and novelty between these two papers. In my opinion, this paper just applies a similar structure from the mentioned paper[1] to another domain, basketball.

2 Existing benchmarks such as SVI-Bench [2] already cover basketball video understanding with similar task categories. The authors should clarify why BasketballBench is better or more unique than the basketball-related tasks in SVI-Bench.

[1] Rao, Jiayuan, et al. "Multi-agent system for comprehensive soccer understanding." Proceedings of the 33rd ACM International Conference on Multimedia. 2025.

[2] Pan, Yulu, et al. "SVI-Bench: A Dynamic Microworld for Strategic Video Intelligence." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

Questions for authors

In my view, BasketballAgent is designed as a domain-specific agent for basketball. I would expect the agent suite to be more comprehensively designed. I was wondering whether the authors have considered adding a commentary generation task to the agent suite.

Reviewer_FGp4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a benchmark for “understanding” Basketball games. The authors collect game video snippets and play-by-play information from existing resources (several websites and the like) and propose 10 tasks. All the tasks are QA, several are multimodal, and they target different information from the multimodal input and different aspects of the game of basketball. The paper also includes experimental results with a combination of existing tools.

Critical strengths
  • Multimodal “understanding” is certainlu an interesting problem.
  • The proposed benchmark is relatively large.
  • The paper includes results with several (I think 9) LLMs. These are all quantitative.
Critical weaknesses
  • The main issue here is that I fail to see why we need yet another benchmark. Multimodal reasoning is a well established problem with many benchmarks already published. The authors themselves list with no details or effort to many compare previous research on “sports understanding (041).

  • The very same limitation the authors point out about previous work and benchmarks (045) are present in their benchmark. They have 10 individual tasks (different problems, evaluated independently, and so on). The sentence starting with “This motivates …” is unsupported.

  • The paper refers to “annotated instances” and “annotations”, but there is no explanation of what they mean by annotation. I think (but cannot tell from the paper) that these come from merging existing online resources (e.g., play-by-play information). They do not come from humans. Section 3.2 does not provide anywhere near what I would expect in a scientific paper to explain the benchmark construction.

  • The 10 tasks (Q1-Q10) are presented without any justification. I can only guess, but I cannot tell for sure, that they were defined because they are the easiest to map to the information they could gather from the existing resources in 3.2.

  • Following with the tasks, I just cannot tell where the ground truth comes from. Where do the court regions come from and why? I would expect most shots/plays to be made from the same court regions (based on my limited knowledge of basketball). Is Q4 simply an OCR task (I am guessing by “scorebug” they mean a picture?). If so, that has little to do with basketball and should be trivial with a sota vision model. In Q8, what is a “visible event” (and more importantly, what would an “invisible” event be, and who gets to decide which events are important or worth extracting? The bottom line is that the reader cannot tell what the benchmark is about, how it was created, and why is it needed given to the many many benchmarks already out there.

  • The experiments results appear valid at face value, but they are using existing tools without any justification. There are no baselines outside prompting a bunch of LLMs. All the games they work with were in the past, thus LLMs are likely to have been pre-trained on information about these games. It is likely that parametric knowledge can already answer many of their questions, and that a RAG-based system would excel (I am basing this comment on Figure 3 and the examples in the appendices).

  • The paper does not describe the LLMs used for their BasketballAgent. Comparisons with their LLM baselines are therefore unsound. Any differences may be due to using different LLMs. There is no mention of additional compute needs, model sizes, or amount of pretraining that could justify the different numbers.

  • The qualitative analysis in 5.6 is 3.5 lines. I can only state that it is not serious enough for a research paper.

Given the lack of motivation for the benchmark and comparison with previous multimodal efforts (do these previous proposals outperform BasketballAgent?; comparing with off-the-shelf LLMs is unfair), and the relatively high numbers they get in most tasks (Table 2), I don’t recommend for publication this benchmark, the main contribution of the paper.

Questions for authors

None. The paper is easy to read.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.