acceptodds
Under review as a conference paper at ICLR 2027

Basketball-QA: Diagnosing Perception and Reasoning in Multi-Player Basketball Video Understanding

Abstract

Multimodal large language models (MLLMs) show increasingly strong capabilities in sports video analysis, including question answering, commentary generation, and event recognition. However, existing evaluations rarely combinemultiple interacting players, dense sequences of events, and fine-grained spatialand temporal relations within a single video understanding task. We introduceBasketball-QA, a basketball video understanding benchmark with over 8K questions covering player matching, spatial grounding, action recognition, temporalgrounding, event relations, counting, hallucination and tactical analysis. Solving these questions requires models to jointly identify relevant players, determine theiractions and locations, and associate events across time. In end-to-end evaluation of video-input MLLMs, the best multiple-choice accuracy is 76.0% and the best open-ended score is 2.87/5, revealing substantial limitations in complex multi-player basketball scenes. We then localize the source of these failures througha series of increasingly targeted experiments. Probing atomic perception testsshow that errors already arise when recovering the underlying facts. When correct perceptual facts are instead provided in text, a small language model without any task-specific fine-tuning achieves 78.5% multiple-choice accuracy, surpassing the best video-input model we evaluate. Together, these experiments indicatethat current MLLMs are primarily limited by the acquisition and association offine-grained visual evidence on Basketball-QA, rather than by reasoning over correctly provided perceptual information. We hope Basketball-QA will support thesystematic evaluation and improvement of MLLMs for complex sports video understanding.

Primary area:
datasets and benchmarks
Keywords:
Basketball Video Understanding, Multimodal Large Model, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vhVBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a Basketball-QA benchmark comprising over 8K questions, including both MCQs and open-ended QA pairs, grounded in 209 clips from professional basketball broadcasts. A comprehensive evaluation of 20 MLLMs shows that the best model achieves a multiple-choice accuracy of 76.0%, and the best open-ended score is 2.87/5, revealing substantial limitations in complex multiplayer basketball scenes.

Critical strengths
  1. The evaluation is comprehensive, with multiple backbones for both open-source and closed-source models.
  2. The benchmark has some novel tasks, such as Court spatial grounding, that other benchmarks lack.
  3. The paper is well-written and easy to follow.
Critical weaknesses
  1. Although the authors provide atomic perception evaluations, more detailed error analysis is needed to show which perception errors lead to incorrect QA answers. For example, are the errors mainly due to player identification or temporal and spatial grounding? This would provide insight into downstream tasks, since the authors have already defined a taxonomy.
  2. The difficulty design is unclear to me. The authors classify questions into different difficulty levels based on the number of core evidence types. However, I am not sure whether all these dimensions are actually required to answer the questions. For example, if an event can be easily identified by its action, a model may answer a question about where a specific player performed that action without identifying that player. Could the authors clarify this?
  3. Table 6 shows that some models achieve higher accuracy on hard questions than on medium questions, which also suggests that the difficulty design based on the core evidence type may have some flaws.
  4. The paper has a human verification process, but no inter-annotator agreement scores are reported.
Questions for authors
  1. I have a question regarding the hallucination taxonomy. Based on the example in Figure 6, the hallucination task seems to involve questions about players whose jersey numbers are not clearly visible in the video (according to the option context). However, if MLLMs can recognize these players by appearance and infer their jersey numbers from prior knowledge, how do the authors address this? How do the authors ensure that the models rely on visual evidence rather than prior knowledge?
  2. The paper aims to diagnose both perception and reasoning. However, in my view, most of the tasks in MCQs are perception-heavy, and only open-ended tactical analysis seems to clearly require basketball-specific reasoning. Could the authors clarify which tasks require reasoning beyond combining perceptual facts?
  3. Following on the previous question, could the authors provide some visualized examples of tactical analysis questions, since this seems to be an important reasoning task in the benchmark?
7 comments
AuthorsOpenReview ↗

Rebuttal by Authors

W1 More fine-grained attribution from perception errors to QA errors

Reviewer (summarized): the atomic perception evaluation is not enough; the authors should show which perception errors actually lead to wrong QA answers — player identification, or temporal and spatial grounding — since the taxonomy already exists.

We thank the reviewer for the suggestion, and we have run two additional analyses. First, we performed content-level error attribution on the open-ended answers scored 3–4: spatial grounding is the dominant source of error (64%), player identification accounts for only 10%, and trajectory nodes and region details are the main sub-classes. This is consistent with the atomic-perception results, where trajectory/timing is weakest and player matching is strongest. Second, we turned the correlation into an intervention via dimension swap: we replaced the model's own atomic report, one dimension at a time, with verified facts along player / event timing / event content / trajectory, and measured the QA drop caused by each type of perception error. The conclusion is that current QA errors come mainly from spatial/temporal binding and event ordering rather than from player identification; player matching is the strongest dimension in the atomic diagnostic. (L452–L458)

Details are in the revision at L1281–L1290 and L1296–L1297.

W2 Whether every dimension of the difficulty design is necessary

Reviewer (summarized): it is unclear whether every dimension of the difficulty design is truly needed to answer a question; e.g., an action may be identified from the action alone, so a question about where a player performed it might be answered without identifying that player.

This is a crucial suggestion. Prompted by the reviewer, we audited the full benchmark item by item, re-labeled items in which a cited dimension "does not work at all" according to the dimensions that are effective, and updated the difficulty distribution in Fig. 2 and the per-difficulty numbers in Table 6 accordingly. The audit criteria are: whether the action is unique within the game; whether the time/region anchor discriminates the answer; whether, for point-in-time questions, the distractor event is truly absent from the whole game; and whether the number of action occurrences corresponding to an ordinal word is really 1. The decision looks only at whether the answer side still requires identifying that dimension. Hallucination items are a special case: the answer itself only requires identifying the negated dimension, so we did not downgrade them for "dimension redundancy" and still report them separately in the Hal column. (Fig. 2: L172–L176; Table 6: L1134–L1157)

W3 Hard above medium in the difficulty table

Reviewer (summarized): some models score higher on hard than on medium questions in Table 6, which may indicate a flaw in the difficulty design built on core evidence types.

We agree that this inversion needs an explanation, but it does not show that the difficulty design is flawed. The three tiers are grades of compositional complexity at construction time (2/3/4+ evidence types), not an equal-interval difficulty scale, and no conclusion in the paper depends on every model being monotonic across every adjacent tier. In the original table, 8 of 20 models nominally have hard > medium; for 7 of them the gap is at most 2.1 points and the confidence interval contains zero, and only Gemini-3.8 is significant (+6.9). In aggregate, 19 of 20 models drop from easy to hard and 17 of 20 drop significantly from easy to medium, with OR = 0.74 per tier. The single significant inversion comes from the composition of medium: 528 items require identity, space and time but no action cue, and removing that block gives medium .80 > hard .76. The atomic-perception experiments likewise show that event content is a strength of Gemini-3.8. (L1109–L1112, L1115–L1128)

Details are in the revision at L1115–L1128.

W4 Missing inter-annotator agreement (IAA)

Reviewer (summarized): the paper describes a human verification process but reports no inter-annotator agreement.

This point needs clarification: the current process is serial review, not independent double annotation. One expert checks and edits the structured facts and a second expert reviews those edits; the tactical subset is written by one person and reviewed by a second; generated items are audited by sampling against the rubric. We have revised this sentence in §3.2 to state the serial process explicitly, so that "annotators" is not read as double annotation (L274–L286).

AuthorsOpenReview ↗

supplement the rebuttal

Q1 Hallucination items and prior knowledge

Reviewer (summarized): the hallucination items seem to involve players whose jersey numbers are not clearly visible, so a model could recognise the player by appearance and recover the number from prior knowledge; how do the authors keep the task grounded in visual evidence?

We already control for this in constructing the benchmark: the number of a player whose jersey number is not visible is recorded as -1, and such players are avoided during generation; clips with fewer than 10 observable numbers do not generate number-hallucination items; #-1 is rejected by the validator; and Figure 6 has been replaced with an example in which the numbers are visible. For items where the number is visible, prior knowledge can help recognize the person, but this is still a form of person perception, and we do not consider it cheating.

This point requires no corresponding change in the paper (the relevant figure is Figure 7, L842–L843).

Q2 Which tasks genuinely require reasoning

Reviewer (summarized): most MCQs look perception-heavy and only open-ended tactical analysis seems to need basketball-specific reasoning; which tasks really require reasoning beyond combining perceptual facts?

We agree that "perception-dominated" needs to be distinguished from "requiring composition". Below are several examples that require reasoning over composed perceptual information:

  • basketball_12: "which player completes a drive with the ball into the interior?" This requires combining "holding the ball" with "moving quickly into the interior" to determine who is the driver.
  • basketball_19: "during the time White team #5 holds the ball, how many times does White team #32 complete a pick-and-roll screen?" This requires locating the interval in which White #5 holds the ball, filtering the screen events within that interval, and counting them.
  • basketball_11: "which happens first: Black team #3's pass or Black team #17's shot?" This requires locating the time of each of the two events and comparing their order.

None of these can be answered by "reading one more fact", but neither are they like tactical analysis, which requires very deep understanding and reasoning.

Q3 Visualized examples of tactical analysis questions

Reviewer (summarized): since tactical analysis is a flagship reasoning task, could the authors provide visualized examples of such questions?

Visualized examples already exist in the paper: Appendix Figure 8 (fig:tactical-instance) draws the question text, the expert reference answer, and the pass, defender movement and offensive-player movement arrows on the keyframes together; the tactical paragraph in the main text now carries a pointer to it. The sample clips in the supplementary material also contain complete tactical QA, for example:

  • basketball_3.mp4 Q: What tactic was Green team originally running in this possession? A: In this possession, Green team originally ran a pick-and-roll between the center and the guard: Green team #14 sets a pick-and-roll screen for Green team #1, Green team #1 can choose to drive or to shoot, and Green team #14 also has the chance to roll to the basket.

  • basketball_10.mp4 Q: What is the key step in Purple team's offense? A: It was Purple team #1's screen for Purple team #4 on the off-ball side that prevented Gray team #3 from keeping up in time.

Thank you for the authors' response. W1 and W4 have been addressed. However, I still have further questions regarding W2 and W3.

Thanks for the additional audit, but I am still not fully convinced that all annotated dimensions are necessary. For example, the hard counting question (middle of the third row) in Figure 6 is labeled with player matching, action, counting, and temporal. However, it seems that the answer could be inferred by counting actions within the given time interval, without explicitly identifying players. Could the authors clarify why player matching is required here?

I also appreciate the analysis explaining on W3. My concern is more about the difficulty definition itself. I am not convinced that more evidence dimensions necessarily make a question harder. I think the current taxonomy measures compositionality rather than actual question difficulty, and these two concepts should be distinguished and disentangled more clearly.

In addition, I still have some concerns about its unique contribution to video understanding and the sports domain. The authors conclude that current models are primarily limited by fine-grained visual perception rather than reasoning over correctly provided facts. However, challenges in action recognition, counting, temporal, and spatial grounding have already been studied in existing general and sports video understanding benchmarks. Figure 2 shows that action and player matching each appear in over 6K questions, suggesting that much of the benchmark focuses on combining a relatively fixed set of actions with player matching and other perceptual dimensions. I would appreciate further clarification on what unique challenges the basketball setting introduces and what new insights this benchmark provides. Nowadays, I would expect a single-sport benchmark focused on evaluating sport-specific reasoning that requires understanding its unique rules, tactics, and player interactions.

Tactical analysis is a good example in this work, but it includes only 84 questions out of more than 8K, which seems quite limited. This brings me to the next questions.

Could the authors provide a more detailed taxonomy of these questions, including the types of basketball tactics covered and their distribution? I think this would help clarify how well the benchmark evaluates basketball-specific reasoning.

Reviewer_1STZOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Basketball-QA, a benchmark containing over 8K question-answer pairs derived from 209 professional basketball video clips. The benchmark evaluates fine-grained multi-player video understanding, including player identification, spatial and temporal grounding, action recognition, event relations, and tactical analysis.

The authors evaluate 20 multimodal large language models (MLLMs) and further investigate whether their performance limitations arise from visual perception or downstream reasoning. Their experiments suggest that accurate structured visual evidence substantially improves question-answering performance, indicating that visual evidence acquisition and association remain major bottlenecks.

Critical strengths
  • The paper addresses an important and challenging problem in fine-grained sports video understanding, particularly multi-player interactions and temporal associations.

  • The evaluation covers a diverse collection of recent MLLMs, offering useful empirical observations about their performance on basketball-specific understanding tasks.

  • The attempt to distinguish visual perception from downstream reasoning through atomic-perception and structured-facts experiments is interesting and provides insights beyond conventional end-to-end benchmark evaluations.

Critical weaknesses
  • No critical weakeness spotted in the paper. Raise a few questions in the next sector.
Questions for authors
  • How does new benchmark (i.e., scale, sizing, variety, task supported etc.,) look like in comparison to 'Sports Video Analysis on Large-Scale Data' ECCV 2024?

  • How sensitive are the reported temporal grounding and trajectory results to the adopted 2 FPS sampling strategy?

  • Given that over 8K questions originate from only 209 clips, how robust are the results when evaluated across independent games or seasons?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Q1 Comparison with NSVA in scale, composition, diversity, and task coverage

Reviewer comment

How does new benchmark (i.e., scale, sizing, variety, task supported etc.,) look like in comparison to 'Sports Video Analysis on Large-Scale Data' ECCV 2024?

Response

NSVA ("Sports Video Analysis on Large-Scale Data", ECCV 2024) pairs each clip with a single templated narrative transcribed from the official play-by-play, and its evaluation is built around narrative generation and video–text retrieval, with action recognition and key-player identification as auxiliary tasks. On the language side it therefore offers one fixed narrative per clip rather than MLLM-style question answering, which is the main difference from Basketball-QA. That difference in task form is also why NSVA is not a row of Table 1: Table 1 compares sports-video question-answering benchmarks, whereas NSVA is discussed as related work (L154–161).

Q2 Sensitivity of temporal-grounding and trajectory results to the 2 FPS sampling strategy

Reviewer comment

How sensitive are the reported temporal grounding and trajectory results to the adopted 2 FPS sampling strategy?

Response

We thank the reviewer for this suggestion, which was very helpful for our experiments.

Starting from the 2 fps of the main protocol, we raised sampling to 4 fps and re-ran inference for the seven frontier API models in the main evaluation, all of which accept long-context input. Open-ended performance improves markedly: the average score rises from 2.24 to 2.92 (+0.68). Among the per-dimension results we currently report, the two dimensions most closely tied to trajectory — spatial grounding (+0.69) and temporal sequence (+0.83) — both improve clearly. Higher frame rates do help models recover finer details of play. Multiple-choice accuracy, in contrast, is essentially unchanged. Notably, even at 4 fps the absolute level remains below that of the structured channel: denser sampling improves evidence recovery but does not overcome the perception bottleneck, and structured facts plus an LLM remains the stronger configuration. (L376–L397, L1127–L1187).

Q3 Robustness across independent games or seasons

Reviewer comment

Given that over 8K questions originate from only 209 clips, how robust are the results when evaluated across independent games or seasons?

Response

The reviewer's concern is fair: there are 209 clips, so although the item count is large, the video sources are limited. We therefore recomputed the end-to-end results separately by source. The conclusion is that scores change only modestly across seasons. The table below reports averages over 20 models:

Question setClipsMCQOpen-ended
2024 season5143.4%1.76
2025 season13645.8%1.71
SportsMOT2244.0%1.76

In both ranking and average score, the seasons are best regarded as parallel versions of the same difficulty rather than as distinct benchmarks. Across seasons and sources, the model ranking, the top-five set, and the ordering of stronger and weaker models are unchanged; the overall change is mainly a small shift in absolute scores.

Thanks for the rebuttall. It well addresses the previous concern on work novelty as compared to the ECCV work and some impact of configs, i.e., low frame rate. This reviwer would like to remain a weak acceptance score.

Thank you very much for your feedback and recognition. We are glad that our rebuttal has addressed your concerns regarding the novelty compared with the ECCV work and the impact of low frame rate configurations. We sincerely appreciate your valuable suggestions and constructive comments that help improve our paper.

Reviewer_9wSWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Basketball-QA, a benchmark for fine-grained multi-player basketball video understanding built on possession-level clips from professional broadcasts. It introduces two key components: human-verified structured facts extracted by a basketball perception pipeline, which record each player's identity, actions, court regions, and timing, and a question set generated from these facts by templates and an LLM and complemented by expert-written tactical questions, which covers player matching, spatial and temporal grounding, action recognition, event relations, counting, hallucination, and tactical analysis. The paper further combines end-to-end video QA, a direct perception diagnostic, and a structured-facts input evaluation to separate evidence acquisition from reasoning. Experiments report that current video MLLMs perform poorly on the benchmark and that supplying verified facts as text substantially improves accuracy, which the authors interpret as a perception rather than reasoning bottleneck.

Critical strengths

S1. Meaningful research problem. Jointly binding player identity, court location, action timing, and simultaneous multi-player states is under-covered by existing video benchmarks, and asking whether failures stem from perception or reasoning is a useful diagnostic angle.

S2. Reasonable diagnostic design. Combining end-to-end QA, direct perception reports, and structured-facts input examines model failures from complementary angles and is more informative than reporting end-to-end accuracy alone.

S3. Fine-grained annotations and broad evaluation. Per-player action timelines with canonical court regions, the evaluation of 20 models with capability and difficulty breakdowns, and the partial release of videos and facts provide a basis for future research and external verification.

Critical weaknesses

W1. Potential answer-position and surface-form shortcuts. Counting the released multiple-choice questions that contain a negation-style option (e.g., "There is no …", "never …", "does not …"), excluding "which happens first" questions, I find that option D is a negation-style option in 828 of the 829 such hallucination-style items, 357 of which concern "#99", a jersey number used in no other multiple-choice question, and 355 contain only one negation-style option, although L251–253 states that, after enhancement, a question contains two; all three hallucination examples in Figure 6 are also answered D. On the 249 items from the ten clips with released answers, a video-free rule ("choose D if a negation-style option exists, otherwise A") reaches 49.8%, above the overall accuracy of 12 of the 20 models in Table 2 (albeit on a different subset), so the capability comparisons, especially Hal and the SFT results, may partly reflect format artifacts.

W2. Insufficient support for the perception-rather-than-reasoning conclusion. This conclusion rests mainly on the large accuracy gain obtained when verified facts replace the video, but rule-based questions are instantiated directly from these facts (L263–264), so their answers correspond directly to records in the facts, and the experiment acts more as an oracle upper bound on whether the questions can be answered from the facts than as a test of reasoning. Moreover, the high scores after SFT may stem from template and position regularities (over three quarters of the released multiple-choice questions share a template with questions from other clips even when only jersey numbers and team colors are masked), while tactical analysis, which requires multi-player reasoning, even drops after SFT for the 4B model (2.24 → 1.80 in Table 3), and the paper itself notes that multi-player reasoning remains a challenge (L468–469).

W3. Incomplete comparison with prior basketball datasets. Table 1 presents jersey-number recognition and multi-player states as unique to Basketball-QA, but omits closely related basketball datasets such as NSVA [1], which annotates NBA broadcast videos for captioning, action recognition, and player identification, and SportsHHI [2], which targets multi-player interactions in basketball and volleyball.

W4. Limited novelty of the perception–reasoning decoupling. Separating perception from reasoning by supplying visual information as text to a language model, or by evaluating perception and reasoning at different levels on the same visual content, has been explored in general multimodal evaluation, e.g., Prism [3], LENS [4], and Do You See Me [5], which the paper does not discuss; the authors should clarify what their diagnostic adds beyond these frameworks besides the basketball domain.

[1] Wu D, Zhao H, Bao X, et al. Sports video analysis on large-scale data[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 19-36.

[2] Wu T, He R, Wu G, et al. SportsHHI: A dataset for human-human interaction detection in sports videos[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[3] Qiao Y, Duan H, Fang X, et al. Prism: A framework for decoupling and assessing the capabilities of VLMs[C]//Advances in Neural Information Processing Systems. 2024.

[4] Yao R, Zhang B, Huang J, et al. LENS: Multi-level evaluation of multimodal reasoning with large language models[C]//International Conference on Learning Representations. 2026.

[5] Kanade A, Ganu T. Do You See Me: A multidimensional benchmark for evaluating visual perception in multimodal LLMs[C]//Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Questions for authors

Q1. Are options shuffled at evaluation time, and how do Tables 2 and 3, especially the Hal column, change under circular evaluation or against a question-only baseline on the full set?

Q2. Why do hallucination items repeatedly use "#99", and why do 355 of them still contain only one negation-style option?

Q3. Is the 28.8% video-only result of Qwen3.5-4B in Table 2 computed on the same held-out test split as the 78.5% in Table 3? In addition, how do the facts-input results differ between rule-based and LLM-generated questions, and what accuracy is obtained when each model's own direct-perception report (Section 4.3), rather than the verified facts, is supplied as text?

Q4. Figure 1 and L85–86 organize the benchmark around five basic evidence types (team, identity, action, time, and space), whereas difficulty is defined by the number of core evidence types (L206–207), which the difficulty prompt in Appendix C lists as four (action, id, temporal, and space); is team membership counted as a core evidence type when assigning difficulty?

Q5. The abstract reports over 8K questions while the released data contain 7,945, Figure 5 shows 11 models although its caption and the text refer to 12, and the split ratio is given as 7:1:7 (L425); which values are correct? In addition, the Judge mean in Table 5 is exactly 3.00 for most models; is this a consequence of sampling 20 responses per score level (L954–956), and can agreement be reported on responses sampled from the natural score distribution?

Q6. The main text uses several names for the same concepts, e.g., "direct perception diagnostic", "atomic perception tests", "direct atomic-perception evaluation", and "direct fine-grained perception" for the perception experiment; "structured match facts", "human-verified records", "evidence reports", "structured textual descriptions", and "structured text" (Figure 5) for the structured facts; and "temporal grounding", "temporal reasoning", and "temporal localization", as well as "hallucination detection", "false-premise", and "hallucination and negation detection", for the capabilities. Could the terminology be unified? In addition, the abbreviation BVPT does not match "Basketball Video Perception Pipeline", and there are typos such as "lead to an error answer" (L215), "procedures.In" (L268), and "Qwen-Plus(Qwen" (L269).

Q7. Can the authors discuss the datasets and evaluation frameworks above and clarify how Basketball-QA differs from them?

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors

W1

We thank the reviewer for this observation. The concentration of answers on D is an artifact of how our prompt examples are laid out: every example places the correct negation option at D and runs through #99 throughout. #99 is nevertheless not a serious error; see Q2 for details. Distractor enhancement is subject to certain conditions, which we explain at L261–L266. At evaluation time, the evaluation script automatically shuffles the distribution of the real options. The SFT issue is answered separately in W2.

W2

We thank the reviewer for the question. First, the benchmark does not consist only of rule-based questions, and even the rule-based ones are not simple look-and-say: the rule generator instantiates human-verified structured facts into questions, but answering requires binding team, number, action, time and space to the correct entities and composing them (ordering, aggregation, across entities). On easy questions most items can be answered from perception alone; overall, however, for the medium and hard questions the facts experiment measures whether a model can reason once correct evidence is given, not whether it can copy ground truth into the options. Second, we thank the reviewer for pointing out the SFT issue: after shuffling the options and re-testing the original SFT models, accuracy does indeed drop by 5–6 points, and we have re-fine-tuned on the shuffled dataset and report the updated results at L492–L493. Third, tactical reasoning requires strong reasoning ability together with prior knowledge of the game of basketball, and our tactical-reasoning data are not yet large enough for models to learn such hard problems through SFT; this is indeed a current limitation.

W3

Both NSVA and SportsHHI are now introduced in the related-work section of the revision, together with their differences from Basketball-QA: NSVA pairs NBA clips with templated narratives derived from the official play-by-play and targets narrative generation and retrieval, while SportsHHI detects tubes of two-person interactions. Neither evaluates models in the form of a question plus reference answer, so adding them to Table 1 would only produce N/A cells. The caption of Table 1 already restricts its scope to sports-video question-answering benchmarks, and our coverage claim should be read within that scope. Details are at L155–L161 and L149–L151.

W4

Supplying visual information as text to separate perception from reasoning, or evaluating perception and reasoning at different levels on the same content, has indeed been done in Prism, LENS, and Do You See Me; the revision introduces them in related work and restates our contribution as an instantiation of this methodology in the basketball setting. What we add beyond them is that §4.3 delivers a checkable ground-truth intermediate state, which lets us measure intermediate-state error directly, run controlled contrasts in which only the correctness of the content varies, and correlate per-clip perception quality with QA score, rather than merely replacing components and reading off a joint score; the domain conclusions that follow depend on domain-grounded facts, which general frameworks do not provide. They are not sports-domain benchmarks, however, so they are not added to Table 1. Details are at L138–L144.

Q1

Options are stored in generation order, but the evaluation script automatically shuffles the option order. We regret that we cannot run the circular-evaluation experiments during the rebuttal period: with video input, the tokens and processing time required by large models exceed our budget. We have added video-free experiments and experiments that split hallucination items into those with and without distractors. Accuracy drops substantially for all models overall, and with distractors the hallucination items also greatly reduce the tendency of models to choose the negation option directly (L1221-L1229).

Q2

Both stem from our generation pipeline. #99 comes from the prompt examples: every example in the hallucination and distractor prompts used a nonexistent #99 for illustration, and the generator imitated it. Half of the single-negation items are by design, and the paper now states why single-negation items exist (L261–L266). #99 is in fact an existing number; it is simply absent from the videos corresponding to these hallucination items. Furthermore, the evaluation script scores each question independently, so there is no situation in which a model infers regularities from a large number of items.

AuthorsOpenReview ↗

supplement the rebuttal

Q3

They are not the same subset. Table 2 uses video input on the evaluation intersection of the 20 models; Table 3 uses structured-facts input on a held-out test split with disjoint clips. To remove this confound, the video results of the 20 models on the same held-out split are now in the appendix (L1350–L1368), in the same format as Table 2, and accuracy is almost identical to that on the full set.

Facts-input results for rule-based versus LLM-generated questions:

Question sourceMCQ accuracyOpen-ended mean
Rule-based45.2%1.90
LLM-generated44.7%1.70

The two sources are comparable in difficulty, with LLM-generated items slightly harder.

We do not consider the run that replaces facts with each model's own perception report worth doing, because the value of this diagnostic lies precisely in the ground-truth intermediate state: with a checkable intermediate state we can measure the model's own perception error directly (atomic F1 in §4.3), fix the textual content as correct and let reasoning be the only variable (facts input), and correlate per-clip perception quality with QA score, turning "perception errors cause QA errors" from an assumption into a measurable quantitative relation. Feeding the model's own report back in measures the joint effect of the description channel and its errors, with no reference truth to compare against, so the error still cannot be attributed to perception or to reasoning; that is exactly the joint score that frameworks such as Prism and LENS, which have no checkable intermediate state, can only produce. This condition would therefore add one more joint score without adding attribution information. Should the reviewer consider it necessary, the per-clip outputs are all saved, and the extra cost is a single text-only inference pass, but it would not change the conclusion.

Q4

Team membership is one of the five basic evidence types in the capability taxonomy, but it is not counted as an independent core dimension when assigning difficulty: jersey identity and team membership are resolved jointly, and a player-matching fact is itself team plus number, so the difficulty counter treats identity as a single core dimension that subsumes team. The four core dimensions are action, identity, temporal, and space, and easy/medium/hard correspond to two, three, and at least four of them co-occurring.

Q5

We accept all four points. The first is an export bug on our side: the multiple-choice items from the two clips basketball9 and basketball171 were not imported, so the v1 supplement contained only 7,945 items. We have fixed the export and re-uploaded it, and the supplement now matches the full set. As for the train/validation/test split, 7:1:7 is an approximate ratio; the actual split is 95:13:101 clips. The "12 models" is a writing error and has been corrected to 11 (L517, L522). The Judge mean being exactly 3.00 is an artifact of stratified sampling (20 responses per score level per model); the "exact" column of the human ratings reports agreement. Details are in Table 5.

Q6

Terminology has been unified throughout: the perception experiment is now consistently "atomic-perception evaluation", the textual evidence is consistently "structured facts", capability names follow Table 1 (temporal grounding, hallucination detection), and the abbreviation is now BVPP. The three spelling and punctuation errors are fixed, and we also corrected non-standard hyphens and comma splices along the way. Text inside both figures has been re-rendered from the corrected sources. Details are at L25, L88, L92, L96, L104, L409, L244–L246, L447, and L861.

Q7

See W3 and W4: the datasets (NSVA, SportsHHI) and the decoupling frameworks (Prism, LENS, Do You See Me) are all introduced in the related work of the revision with their differences from Basketball-QA; the scope of Table 1 is restricted to sports-video question-answering benchmarks, and within that scope the capabilities unique to Basketball-QA still hold. Details are at L155–L161, L149–L151, and L138–L144.

Thanks for the detailed response and the effort put into the revision. I appreciate the updates that address several of my earlier concerns (W3, W4, Q4 to Q7, and parts of W1 and W2). However, three concerns remain unresolved and make me keep my rating:

(i) In Q3, the numbers reported as "facts-input results" (45.2% / 1.90 for rule-based and 44.7% / 1.70 for LLM-generated questions) cannot mathematically come from the structured-facts setting. These numbers really surprise me. In revised Table 3, the four models score 38.8% / 2.38, 78.5% / 4.55, 91.0% / 4.69, and 95.5% / 4.79: even the weakest model (Qwen3.5-0.8B base) has an overall open-ended mean of 2.38 (with TA at 1.38), so none of the models in Table 3, nor their average, can have open-ended means of 1.90 and 1.70. Could the authors clarify this discrepancy and report the actual structured-facts breakdown?

(ii) Revised Table 9 confirms that single-negation ("1 distractor option") hallucination items admit strong video-free shortcuts, reaching up to 73.8% for frontier models and 97.7% for local models under Q-only input (versus 0.0% to 13.5% on two-distractor items). Moreover, Q2 acknowledges that only half of these single-negation items are by design, and while zero-shot inference scores items independently, SFT trains across clips and can directly learn cross-item regularities such as the repeated "#99" false-premise cue. Why not fix the unintended single-negation items and diversify "#99" in the dataset itself, and report updated Hal and SFT results? I believe only a properly cleaned dataset can make a solid contribution to the field.

(iii) Since the per-clip Section 4.3 perception outputs are already saved in the same structured format (Appendix B) and require only a single text-only inference pass, I do consider this experiment necessary. Feeding several representative models' own Section 4.3 reports into a fixed text reasoner (e.g., Qwen3.5-4B base) and comparing directly against human-verified structured facts holds both the schema and the reasoner constant while varying only perception quality, directly quantifying how much QA performance is lost to the Section 4.3 perception errors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.