- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a Basketball-QA benchmark comprising over 8K questions, including both MCQs and open-ended QA pairs, grounded in 209 clips from professional basketball broadcasts. A comprehensive evaluation of 20 MLLMs shows that the best model achieves a multiple-choice accuracy of 76.0%, and the best open-ended score is 2.87/5, revealing substantial limitations in complex multiplayer basketball scenes.
Critical strengths
- The evaluation is comprehensive, with multiple backbones for both open-source and closed-source models.
- The benchmark has some novel tasks, such as Court spatial grounding, that other benchmarks lack.
- The paper is well-written and easy to follow.
Critical weaknesses
- Although the authors provide atomic perception evaluations, more detailed error analysis is needed to show which perception errors lead to incorrect QA answers. For example, are the errors mainly due to player identification or temporal and spatial grounding? This would provide insight into downstream tasks, since the authors have already defined a taxonomy.
- The difficulty design is unclear to me. The authors classify questions into different difficulty levels based on the number of core evidence types. However, I am not sure whether all these dimensions are actually required to answer the questions. For example, if an event can be easily identified by its action, a model may answer a question about where a specific player performed that action without identifying that player. Could the authors clarify this?
- Table 6 shows that some models achieve higher accuracy on hard questions than on medium questions, which also suggests that the difficulty design based on the core evidence type may have some flaws.
- The paper has a human verification process, but no inter-annotator agreement scores are reported.
Questions for authors
- I have a question regarding the hallucination taxonomy. Based on the example in Figure 6, the hallucination task seems to involve questions about players whose jersey numbers are not clearly visible in the video (according to the option context). However, if MLLMs can recognize these players by appearance and infer their jersey numbers from prior knowledge, how do the authors address this? How do the authors ensure that the models rely on visual evidence rather than prior knowledge?
- The paper aims to diagnose both perception and reasoning. However, in my view, most of the tasks in MCQs are perception-heavy, and only open-ended tactical analysis seems to clearly require basketball-specific reasoning. Could the authors clarify which tasks require reasoning beyond combining perceptual facts?
- Following on the previous question, could the authors provide some visualized examples of tactical analysis questions, since this seems to be an important reasoning task in the benchmark?
7 comments
supplement the rebuttal
Q1 Hallucination items and prior knowledge
Reviewer (summarized): the hallucination items seem to involve players whose jersey numbers are not clearly visible, so a model could recognise the player by appearance and recover the number from prior knowledge; how do the authors keep the task grounded in visual evidence?
We already control for this in constructing the benchmark: the number of a player whose jersey number is not visible is recorded as -1, and such players are avoided during generation; clips with fewer than 10 observable numbers do not generate number-hallucination items; #-1 is rejected by the validator; and Figure 6 has been replaced with an example in which the numbers are visible. For items where the number is visible, prior knowledge can help recognize the person, but this is still a form of person perception, and we do not consider it cheating.
This point requires no corresponding change in the paper (the relevant figure is Figure 7, L842–L843).
Q2 Which tasks genuinely require reasoning
Reviewer (summarized): most MCQs look perception-heavy and only open-ended tactical analysis seems to need basketball-specific reasoning; which tasks really require reasoning beyond combining perceptual facts?
We agree that "perception-dominated" needs to be distinguished from "requiring composition". Below are several examples that require reasoning over composed perceptual information:
basketball_12: "which player completes a drive with the ball into the interior?" This requires combining "holding the ball" with "moving quickly into the interior" to determine who is the driver.basketball_19: "during the time White team #5 holds the ball, how many times does White team #32 complete a pick-and-roll screen?" This requires locating the interval in which White #5 holds the ball, filtering the screen events within that interval, and counting them.basketball_11: "which happens first: Black team #3's pass or Black team #17's shot?" This requires locating the time of each of the two events and comparing their order.
None of these can be answered by "reading one more fact", but neither are they like tactical analysis, which requires very deep understanding and reasoning.
Q3 Visualized examples of tactical analysis questions
Reviewer (summarized): since tactical analysis is a flagship reasoning task, could the authors provide visualized examples of such questions?
Visualized examples already exist in the paper: Appendix Figure 8 (fig:tactical-instance) draws the question text, the expert reference answer, and the pass, defender movement and offensive-player movement arrows on the keyframes together; the tactical paragraph in the main text now carries a pointer to it. The sample clips in the supplementary material also contain complete tactical QA, for example:
-
basketball_3.mp4Q: What tactic was Green team originally running in this possession? A: In this possession, Green team originally ran a pick-and-roll between the center and the guard: Green team #14 sets a pick-and-roll screen for Green team #1, Green team #1 can choose to drive or to shoot, and Green team #14 also has the chance to roll to the basket. -
basketball_10.mp4Q: What is the key step in Purple team's offense? A: It was Purple team #1's screen for Purple team #4 on the off-ball side that prevented Gray team #3 from keeping up in time.
Thank you for the authors' response. W1 and W4 have been addressed. However, I still have further questions regarding W2 and W3.
Thanks for the additional audit, but I am still not fully convinced that all annotated dimensions are necessary. For example, the hard counting question (middle of the third row) in Figure 6 is labeled with player matching, action, counting, and temporal. However, it seems that the answer could be inferred by counting actions within the given time interval, without explicitly identifying players. Could the authors clarify why player matching is required here?
I also appreciate the analysis explaining on W3. My concern is more about the difficulty definition itself. I am not convinced that more evidence dimensions necessarily make a question harder. I think the current taxonomy measures compositionality rather than actual question difficulty, and these two concepts should be distinguished and disentangled more clearly.
In addition, I still have some concerns about its unique contribution to video understanding and the sports domain. The authors conclude that current models are primarily limited by fine-grained visual perception rather than reasoning over correctly provided facts. However, challenges in action recognition, counting, temporal, and spatial grounding have already been studied in existing general and sports video understanding benchmarks. Figure 2 shows that action and player matching each appear in over 6K questions, suggesting that much of the benchmark focuses on combining a relatively fixed set of actions with player matching and other perceptual dimensions. I would appreciate further clarification on what unique challenges the basketball setting introduces and what new insights this benchmark provides. Nowadays, I would expect a single-sport benchmark focused on evaluating sport-specific reasoning that requires understanding its unique rules, tactics, and player interactions.
Tactical analysis is a good example in this work, but it includes only 84 questions out of more than 8K, which seems quite limited. This brings me to the next questions.
Could the authors provide a more detailed taxonomy of these questions, including the types of basketball tactics covered and their distribution? I think this would help clarify how well the benchmark evaluates basketball-specific reasoning.
Rebuttal by Authors
W1 More fine-grained attribution from perception errors to QA errors
We thank the reviewer for the suggestion, and we have run two additional analyses. First, we performed content-level error attribution on the open-ended answers scored 3–4: spatial grounding is the dominant source of error (64%), player identification accounts for only 10%, and trajectory nodes and region details are the main sub-classes. This is consistent with the atomic-perception results, where trajectory/timing is weakest and player matching is strongest. Second, we turned the correlation into an intervention via dimension swap: we replaced the model's own atomic report, one dimension at a time, with verified facts along player / event timing / event content / trajectory, and measured the QA drop caused by each type of perception error. The conclusion is that current QA errors come mainly from spatial/temporal binding and event ordering rather than from player identification; player matching is the strongest dimension in the atomic diagnostic. (L452–L458)
Details are in the revision at L1281–L1290 and L1296–L1297.
W2 Whether every dimension of the difficulty design is necessary
This is a crucial suggestion. Prompted by the reviewer, we audited the full benchmark item by item, re-labeled items in which a cited dimension "does not work at all" according to the dimensions that are effective, and updated the difficulty distribution in Fig. 2 and the per-difficulty numbers in Table 6 accordingly. The audit criteria are: whether the action is unique within the game; whether the time/region anchor discriminates the answer; whether, for point-in-time questions, the distractor event is truly absent from the whole game; and whether the number of action occurrences corresponding to an ordinal word is really 1. The decision looks only at whether the answer side still requires identifying that dimension. Hallucination items are a special case: the answer itself only requires identifying the negated dimension, so we did not downgrade them for "dimension redundancy" and still report them separately in the Hal column. (Fig. 2: L172–L176; Table 6: L1134–L1157)
W3 Hard above medium in the difficulty table
We agree that this inversion needs an explanation, but it does not show that the difficulty design is flawed. The three tiers are grades of compositional complexity at construction time (2/3/4+ evidence types), not an equal-interval difficulty scale, and no conclusion in the paper depends on every model being monotonic across every adjacent tier. In the original table, 8 of 20 models nominally have hard > medium; for 7 of them the gap is at most 2.1 points and the confidence interval contains zero, and only Gemini-3.8 is significant (+6.9). In aggregate, 19 of 20 models drop from easy to hard and 17 of 20 drop significantly from easy to medium, with OR = 0.74 per tier. The single significant inversion comes from the composition of medium: 528 items require identity, space and time but no action cue, and removing that block gives medium .80 > hard .76. The atomic-perception experiments likewise show that event content is a strength of Gemini-3.8. (L1109–L1112, L1115–L1128)
Details are in the revision at L1115–L1128.
W4 Missing inter-annotator agreement (IAA)
This point needs clarification: the current process is serial review, not independent double annotation. One expert checks and edits the structured facts and a second expert reviews those edits; the tactical subset is written by one person and reviewed by a second; generated items are audited by sampling against the rubric. We have revised this sentence in §3.2 to state the serial process explicitly, so that "annotators" is not read as double annotation (L274–L286).