- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces BasketballBench, a benchmark of 7,600 questions in ten basketball tasks over text, images and videos. The data come from NBA broadcast clips, a database of NBA records, and player photos. The tasks test basketball knowledge, perception of players and broadcasts, and understanding of events in space and time. The paper evaluates nine MLLMs and finds that they can often recognize events but fail to identify which player performed them. It also proposes BasketballAgent, which uses vision tools, database queries and manually written skills, and reports the best results on seven of the ten tasks.
Critical strengths
-
Broad task coverage. The benchmark covers different types of basketball tasks, ranging from basic knowledge to visual and event understanding. The tasks are well organized under a unified evaluation framework.
-
Strong agent baseline. BasketballAgent achieves good performance across the proposed tasks and obtains the best results on seven of the ten tasks.
Critical weaknesses
-
Limited comparison with existing benchmarks. The paper lacks a systematic comparison with other basketball and sports benchmarks in terms of dataset size, task types, and coverage.
-
Limited value of some tasks. Q4 and Q9 mainly test visual text recognition, and many models already achieve high scores, especially on Q9. Moreover, Q9 uses player crops rather than requiring actual video understanding, which makes its inclusion as a VideoQA task less convincing.
-
Lack of agent efficiency analysis. BasketballAgent relies on complex tools and manually designed skills, but its execution efficiency remains unclear.
-
Limited depth of basketball understanding. Most tasks focus on factual knowledge or basic visual perception. Even Q7, Q8, and Q10 mainly involve event localization, player identification, and knowledge retrieval. In contrast, SoccerBench (Rao et al., 2025) also covers action classification and commentary generation. The lack of tasks involving more complex actions, tactics, and strategy weakens the claim of comprehensive basketball understanding.
Questions for authors
-
Could the authors provide a more complete efficiency analysis of BasketballAgent, including latency, token usage, and inference cost?
-
Could the authors provide a systematic comparison with existing basketball and sports benchmarks in terms of scale and task coverage?
-
Could the authors evaluate general agent systems, such as DeepSeek-V4-Flash with OpenCode or DeepseekHarness, on BasketballBench? A comparison under similar tool and database access would help clarify the benefits of the custom agent design.