- Rating
- 1
- Confidence
- 3(−1)
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces SportD, a benchmark for evaluating VLM decision-making in soccer scenarios. The authors evaluate several VLMs on passing and shooting decisions and analyze their risk preferences and action valuation.
Critical strengths
The paper attempts to study VLM decision-making in real-world sports scenarios.
Critical weaknesses
The overall contribution is limited. The experimental setup is relatively simple, and most analyses provide limited insight beyond showing that VLMs make suboptimal decisions. The figures are also poorly presented and do not effectively communicate the significance of the findings. The paper lacks sufficient methodological depth.
Questions for authors
- What is the main scientific contribution beyond evaluating existing VLMs on a relatively simple decision-making task?
- How do the findings provide actionable insights beyond the observation that VLMs often make suboptimal decisions?
- What makes this benchmark sufficiently novel and useful compared with existing sports-understanding and strategic-reasoning benchmarks?
7 comments
Dear Reviewer UUc2,
Thank you again for your feedback on SportD. In our response, we make clearer what the benchmark adds beyond existing work, and we provided new evidence for the main finding: models favor safer, lower-value actions, while a targeted prompt improves their choices beyond what simple changes in action frequency can explain. We also explored how their estimates of action value relate to this pattern. We hope the added evidence makes the contribution clearer, and we'd be grateful if you would consider raising your score.
Sincerely,
The Authors
Dear Reviewer UUc2,
Can you please take a look at the authors' response and provide your further input? Thank you.
AC.
Response to reviewer
We thank the reviewer for their feedback. SportD addresses a fundamental challenge in physical intelligence: Can multimodal AI systems understand real-world environments, reason strategically about the consequences of their actions, and make effective decisions under uncertainty? This question extends beyond soccer to embodied AI, robotics, and real-world decision-making, where recognizing a scene is not sufficient; systems must strategically evaluate possible actions and their consequences. We make three connected contributions.
1. Aggregate finding: A systematic physical decision-making failure
SportD uses real World Cup video and spatial scene representations. Models must interpret player positions and movements, anticipate uncertain physical outcomes, and choose between consequential actions. Across scenarios, VLMs systematically favor higher-probability, lower-variance actions even when alternatives have greater expected value. This is a directional strategic bias, not merely low accuracy: models struggle to translate multimodal physical evidence into valuable actions.
2. Belief inconsistency: Success probability is not action value
We independently elicit probability and expected-value rankings. Models recover success-probability rankings relatively well (83–92% recall) but perform substantially worse on value rankings (56–66% recall). Crucially, models exhibit a positive correlation between perceived success probability and expected value (Spearman ρ = +0.30 to +0.52), despite essentially no correlation in the ground truth (ρ = −0.08). This suggests that models systematically conflate an action's likelihood of succeeding with its strategic value, offering a specific, testable explanation for their preference for safer but lower-value actions without claiming definitive access to internal mechanisms.
Elicited valuations also relate to actual choices: GPT 5.6 Sol chooses its own top-value-ranked action on 54.3% of events and Gemini 3.5 Flash on 45.9%, versus 18.4% by chance. Thus, we connect the behavioral failure to models' stated assessments, rather than simply reporting mistakes.
3. Predictable steering: Better choices, not indiscriminate shooting
Our SEEK prompt directs attention to expected value and improves decision quality. It does not simply make models shoot more. We compare SEEK with randomly switching passes to shots at the same increase in shooting frequency. For GPT 5.6 Sol, SEEK reaches 0.365 skill versus 0.272 for this matched intervention: +0.093 [0.052, 0.132]. The gain therefore cannot be explained by shooting more blindly; SEEK changes which situations elicit different actions.
Novelty relative to existing benchmarks
Sports understanding: SPORTU (ICLR 2025) and SportR (ICLR 2026) emphasize multimodal sports understanding, rules, and tactical questions. SportD instead evaluates strategic decision-making in real physical environments, requiring models to choose actions that maximize expected value under uncertainty, rather than merely answer questions about a scene. Decisions are evaluated using counterfactual action values rather than textual-answer correctness.
Strategic games: VS-Bench (CVPR 2026 Oral) studies visual strategy in multi-agent games through action prediction and episode returns; BALROG (ICLR 2025) studies agents in game environments. SportD complements these with actual physical scenes, multimodal evidence, and explicit counterfactual valuations of competing real-world actions.
In short, SportD identifies a systematic physical decision-making bias, connects it to a measurable probability–value discrepancy in elicited beliefs, and demonstrates predictable steering beyond simple action-frequency changes. We will improve the figures and exposition to make these contributions clearer.