acceptodds
Under review as a conference paper at ICLR 2027

SportD: How do VLMs physically strategize?

Abstract

Vision-language models (VLMs) can describe a scene, but can they act well within one? We study whether VLMs can make sound strategic decisions, using soccer as an objective testbed with quantifiably-valued actions. We introduce SportD, a dataset and evaluation consisting of 1421 decision scenarios across professional men's and women's soccer games, where a VLM must decide what action to take next. Models on average select the optimal action around 27% of the time, less often than the professional players, and capture markedly less of the value at stake. Furthermore, they exhibit a clear preference for safer actions, favoring lower-variance, lower-value choices that also make less physical progress toward goal. Frontier VLMs are better at estimating whether an action will succeed, placing the highest-success-probability action among their top choices in 72–85% of cases. Yet VLMs systematically conflate likelihood with value, assigning higher value to actions that are more likely to succeed ( to ), despite no such relationship in the ground truth (). Modifying the deliberation instructions to encourage risk-taking brings the frontier models closer to the players’ skill levels. SportD opens a new direction for rigorously evaluating physical strategic decision-making in VLMs, showing that careful decomposition of their choices can reveal the mechanisms underlying systematic biases such as risk aversion.

Primary area:
datasets and benchmarks
Keywords:
VLM, Strategy, Cognitive Science, Evaluation, Sports

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_UUc2OpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces SportD, a benchmark for evaluating VLM decision-making in soccer scenarios. The authors evaluate several VLMs on passing and shooting decisions and analyze their risk preferences and action valuation.

Critical strengths

The paper attempts to study VLM decision-making in real-world sports scenarios.

Critical weaknesses

The overall contribution is limited. The experimental setup is relatively simple, and most analyses provide limited insight beyond showing that VLMs make suboptimal decisions. The figures are also poorly presented and do not effectively communicate the significance of the findings. The paper lacks sufficient methodological depth.

Questions for authors
  1. What is the main scientific contribution beyond evaluating existing VLMs on a relatively simple decision-making task?
  2. How do the findings provide actionable insights beyond the observation that VLMs often make suboptimal decisions?
  3. What makes this benchmark sufficiently novel and useful compared with existing sports-understanding and strategic-reasoning benchmarks?
7 comments
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for their feedback. SportD addresses a fundamental challenge in physical intelligence: Can multimodal AI systems understand real-world environments, reason strategically about the consequences of their actions, and make effective decisions under uncertainty? This question extends beyond soccer to embodied AI, robotics, and real-world decision-making, where recognizing a scene is not sufficient; systems must strategically evaluate possible actions and their consequences. We make three connected contributions.

1. Aggregate finding: A systematic physical decision-making failure

SportD uses real World Cup video and spatial scene representations. Models must interpret player positions and movements, anticipate uncertain physical outcomes, and choose between consequential actions. Across scenarios, VLMs systematically favor higher-probability, lower-variance actions even when alternatives have greater expected value. This is a directional strategic bias, not merely low accuracy: models struggle to translate multimodal physical evidence into valuable actions.

2. Belief inconsistency: Success probability is not action value

We independently elicit probability and expected-value rankings. Models recover success-probability rankings relatively well (83–92% recall) but perform substantially worse on value rankings (56–66% recall). Crucially, models exhibit a positive correlation between perceived success probability and expected value (Spearman ρ = +0.30 to +0.52), despite essentially no correlation in the ground truth (ρ = −0.08). This suggests that models systematically conflate an action's likelihood of succeeding with its strategic value, offering a specific, testable explanation for their preference for safer but lower-value actions without claiming definitive access to internal mechanisms.

Elicited valuations also relate to actual choices: GPT 5.6 Sol chooses its own top-value-ranked action on 54.3% of events and Gemini 3.5 Flash on 45.9%, versus 18.4% by chance. Thus, we connect the behavioral failure to models' stated assessments, rather than simply reporting mistakes.

3. Predictable steering: Better choices, not indiscriminate shooting

Our SEEK prompt directs attention to expected value and improves decision quality. It does not simply make models shoot more. We compare SEEK with randomly switching passes to shots at the same increase in shooting frequency. For GPT 5.6 Sol, SEEK reaches 0.365 skill versus 0.272 for this matched intervention: +0.093 [0.052, 0.132]. The gain therefore cannot be explained by shooting more blindly; SEEK changes which situations elicit different actions.

Novelty relative to existing benchmarks

Sports understanding: SPORTU (ICLR 2025) and SportR (ICLR 2026) emphasize multimodal sports understanding, rules, and tactical questions. SportD instead evaluates strategic decision-making in real physical environments, requiring models to choose actions that maximize expected value under uncertainty, rather than merely answer questions about a scene. Decisions are evaluated using counterfactual action values rather than textual-answer correctness.

Strategic games: VS-Bench (CVPR 2026 Oral) studies visual strategy in multi-agent games through action prediction and episode returns; BALROG (ICLR 2025) studies agents in game environments. SportD complements these with actual physical scenes, multimodal evidence, and explicit counterfactual valuations of competing real-world actions.

In short, SportD identifies a systematic physical decision-making bias, connects it to a measurable probability–value discrepancy in elicited beliefs, and demonstrates predictable steering beyond simple action-frequency changes. We will improve the figures and exposition to make these contributions clearer.

Dear Reviewer UUc2,

Thank you again for your feedback on SportD. In our response, we make clearer what the benchmark adds beyond existing work, and we provided new evidence for the main finding: models favor safer, lower-value actions, while a targeted prompt improves their choices beyond what simple changes in action frequency can explain. We also explored how their estimates of action value relate to this pattern. We hope the added evidence makes the contribution clearer, and we'd be grateful if you would consider raising your score.

Sincerely,

The Authors

Dear Reviewer UUc2,

Can you please take a look at the authors' response and provide your further input? Thank you.

AC.

Reviewer_ATpxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SportD, a VLM benchmark that includes 1421 decision-making scenarios for men's and women's World Cup ball handling. The model selects to shoot or pass to a designated teammate based on the pre decision screen, decision moment screen, and player position map. The author uses VAEP based counterfactual action valuation to evaluate decisions based on optimal action accuracy, normalized skills, and variance of returns. The experiment shows that the summary performance of the tested model under default prompts is lower than that of real people; Models are usually better at identifying high success probability actions rather than high-value actions. The author further proposes that the model tends to confuse success probability with value through independent probability and value ranking inquiries, as well as risk oriented prompt interventions.

Critical strengths
  1. The task design connects real visual scenes with action value evaluation. Compared to only judging event categories or reproducing real person choices, SportD allows for the comparison of estimated benefits for different candidate actions, enabling the study of the gap between "understanding the scene" and "making valuable choices". The coverage and public implementation of men's and women's competitions also contribute to subsequent research.

  2. Analysis goes beyond a single accuracy rate. The author attempts to explain the erroneous structure by combining value loss, normalization skills, probability and value ranking, prompt intervention, and alternative value models.

Critical weaknesses
  1. The prior of strong actions on the benchmark and the effectiveness of the scoring system have not been fully controlled. According to the public candidate action table and scoring rules, the accuracy of SHOOT in all 1421 scenarios is approximately 49.8%, and the mean skill is approximately 0.361, which is higher than the reported real person accuracy of 35.1% and 0.335, respectively. On the men's set, the skill of this baseline is about 0.480, which is also higher than the GPT of about 0.366 under SEEK conditions. This does not mean that the benchmark is invalid, but it indicates that the summary score can be obtained through strategies that do not rely on visual input. Therefore, it is necessary to report the performance under this baseline and action type condition in order to determine whether the model has situational decision-making ability.
  2. The paper uses "belief ranking obtained from separate questioning" and "behavioral changes after changing prompt words" to argue that "the model mistook probability for value", but these pieces of evidence are all relevant. For examlpe, They cannot be distinguished from the alternative explanation that the model is only inherently inclined towards safe actions, and the paper did not conduct intervention experiments that can distinguish between the two, so the mechanism claim is stronger than the evidence.
Questions for authors
  1. What evidence distinguishes active risk aversion from errors in value estimation and action-category heuristics? In particular, do the gains from SEEK exceed what can be explained by simple baselines such as always shooting, passing forward, or selecting high-risk actions? Is there verifiable consistency between the independently elicited rankings and the models’ actual choices?
  2. Please explain the discrepancies between the values plotted in Figure 5 and those stated in its caption, and between the default-condition results in Table 8 and Table 2. Please also provide paired uncertainty estimates for the model–human comparisons and prompt interventions, appropriately accounting for correlations among scenarios from the same match.
9 comments
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for their thoughtful feedback. All intervals below are 95% match-clustered bootstrap intervals, resampling matches within tournaments.

Weakness 1: Always-SHOOT baseline

We would appreciate clarification of the reported 49.8% accuracy and 0.361 skill, as neither figure is established in our submission. The review refers to a "public candidate action table and scoring rules," but we cannot identify the materials or calculation used. Could the reviewer specify the source and methodology?

Weakness 2 / Q1: Risk preference versus value estimation

We have run additional tests. Among passes matched on approximate expected value and distance, players choose the higher-probability pass 50.5% of the time, versus 71.6% for GPT 5.6 Sol and 67.6% for Claude Opus 4.8. This controls for the shoot/pass category and large value differences, revealing a stronger preference for likely passes.

Q1: Does SEEK outperform simple changes in action frequency?

On the men's set, SEEK increases frontier-model shot rates by only 1.4–4.0 percentage points. Compared to an initial skill baseline of 0.298, randomly converting passes to shots at the same rate yields expected skill 0.272 for GPT 5.6 Sol, versus 0.365 under SEEK: a gain of 0.093 [0.052, 0.132]. Gemini 3.5 Flash and Claude Opus 4.8 also exceed their shot-rate-matched baselines. All three outperform redirecting the same number of decisions to the most forward pass. Thus, SEEK gains are not explained by these simple action heuristics.

Q1: Do elicited rankings predict choices?

Yes. GPT 5.6 Sol chooses its own top-ranked value action on 54.3% of events, and Gemini 3.5 Flash on 45.9%, versus 18.4% by chance. When the top-value and top-probability actions differ, choices favor the top-value action by 27.2 and 34.1 percentage points, respectively. These results show meaningful correspondence between elicited rankings and behavior.

Q2: Figure and table clarifications

Figure 5's labels pool both tournaments; we corrected the caption to report the pooled values. On this basis, the spread across model generations narrows from 12.6 to 7.9 percentage points. Table 8 uses the 529 men's events with a labeled frame, whereas Table 2 uses all 726 men's events. We have clarified the subsets.

Q2: Paired uncertainty

After reweighting to the source distribution, model-minus-player accuracy (percentage points) and skill differences are:

ModelAccuracySkill
GPT 5.6 Sol−6.9 [−10.0, −4.0]−0.103 [−0.145, −0.059]
Gemini 3.5 Flash−12.8 [−16.0, −9.6]−0.162 [−0.207, −0.117]
Claude Opus 4.8−18.5 [−22.0, −15.1]−0.273 [−0.322, −0.221]
Qwen3-VL-8B−22.1 [−25.8, −18.3]−0.373 [−0.426, −0.316]
Llama-4-Maverick−22.3 [−26.0, −18.5]−0.347 [−0.397, −0.292]

So, all directional results are the same. SEEK improves GPT 5.6 Sol's skill over the neutral prompt by 0.097 [0.053, 0.141], with improvements also observed for the other frontier models.

We thank the reviewer again. The new interventions, matched comparisons, ranking-consistency analyses, and paired intervals strengthen our central conclusion: VLMs make context-sensitive decisions but still struggle to estimate the value of physical actions from visual evidence.

Dear Reviewer ATpx,

Thank you again for the helpful comments. We followed up with risk comparisons that match expected value and pass distance, tested whether our prompt improvements could be explained by simple action-frequency changes, and checked how elicited rankings relate to the models' actual choices. We also added paired uncertainty estimates for the key comparisons. These additions give us much firmer evidence for the paper's conclusions, and we'd be grateful if you would consider raising your score.

Sincerely,

The Authors

Thank you for the follow-up. I find the premise of SportD promising, since using real match footage to ask which action a player should take gives the work a clear purpose. The rebuttal substantially improves the paper.

To clarify the always shoot calculation, I used the released men's and women's options tables and the repository's scoring rule. For each event I selected shoot, counted it as correct if its expected value matched the maximum among the listed options, and computed skill as (EV of SHOOT − mean option EV) / (maximum option EV − mean option EV). Shooting was optimal on 415 of 726 men's events and 293 of 695 women's events, for 49.8% pooled accuracy and 0.361 mean skill.

I will maintain my Weak Reject recommendation. The rebuttal gives evidence for a preference for lower-variance passes and for the effect of seek in this setting, but my remaining concern is how much the current evaluation reveals about strategic reasoning from visual evidence. It is still unclear how much the benchmark scores reflect visual context beyond simple action and spatial heuristics. An always shoot policy also scores higher than both the evaluated models and professional players on overall accuracy and mean skill, which makes the aggregate scores hard to read as evidence of situational use.

I encourage the authors to continue developing this work. A revised submission that establishes what the models gain from visual context and offers a more mature evaluation would make a stronger case for the benchmark and for the conclusions it can support.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for their constructive feedback. All confidence intervals are 95% bootstrap intervals, resampling matches within each tournament and pairing differences by event.

Q1. Validity of counterfactual action values

How counterfactuals are constructed. The value models train on Euro 2020 and 2024 (men) and Euro 2022 and 2025 (women) and score the 2022 and 2023 World Cups, which they never saw. Each action's value is V = p·v⁺ + (1−p)·v⁻. Pass completion p comes from a gradient-boosted classifier on the freeze frame (pass length, progression, distance to goal, and opponents in the lane, near the target, and goal-side). Shot p comes from an xG model on distance, angle, and header status.

A completed pass has v⁺ equal to VAEP's predicted probability of scoring minus conceding after a synthesized pass to the teammate's freeze-frame position, and a goal has v⁺ = 1. v⁻ is the mean outcome after real failed passes or missed shots in the training data, binned by target position or shot distance. Outcomes cover the next nine on-ball actions by either team, so possession changes, rebounds, and corners count. In the Figure 1 event, the optimal pass is worth 0.899 × 0.145 + 0.101 × 0.006 = 0.131 and the player's shot 0.041, a regret of 0.090.

Validation. On held-out real passes, aimed at the nearest teammate's freeze-frame position as for the candidates, pass prediction reaches AUC 0.838 (men) and 0.773 (women), with calibration errors of 0.051 and 0.075. Shot prediction reaches AUC 0.785 and 0.732, with calibration errors of 0.025 and 0.015. Both models predict success accurately on unseen matches.

Passes in the top fifth of predicted value lead to goals 3.02% (men) and 2.50% (women) of the time, against 0.71% and 0.49% in the bottom fifth (net-goal gap 0.022 [0.014, 0.030] and 0.020 [0.013, 0.027]). Within deciles of state value, a real pass that matches the best pass adds 0.010 [0.004, 0.016] and 0.009 [0.005, 0.014] net goals. This validation uses observed outcomes and does not depend on xT.

Q2. Risk Preference and Value Estimation

Does risk preference persist after controlling for expected value? Yes. We compare each chosen pass against alternatives within 0.01 expected value and 5 meters of distance, measuring how often the chosen pass has a higher success probability (0.5 indicates no preference). GPT scores 0.716 [0.673, 0.754] and Claude 0.676 [0.632, 0.720], versus 0.505 [0.454, 0.554] for players. Both model–player differences are statistically significant, demonstrating a stronger preference for safer passes even when expected value and distance are matched.

Does belief elicitation remain consistent with matched visual inputs? Yes. We repeat the experiment on 198 events using the same run-up frames, final frame, and spatial map as the decision task, explicitly defining expected value on the signed [-1, 1] scale. The probability–value recall gap persists: 21.7 percentage points [13.3, 30.1] for GPT and 19.7 [11.9, 27.5] for Gemini. Models also continue to associate success probability with expected value: correlations are +0.326 for GPT and +0.397 for Claude, compared with −0.023 in the reference value model.

Q3. Uncertainty and reporting discrepancies

Statistical uncertainty. When reweighted to the original source-pool frequencies, the top model lags behind players by 6.9 percentage points

−10.0,−4.0-10.0, -4.0

in accuracy and 0.103

−0.145,−0.059-0.145, -0.059

in skill. Every VLM falls below players, with confidence intervals excluding zero.

Under SportD's regret-tier sampling, GPT is only 0.6 percentage points below players [-3.2, 2.0], which is not statistically significant. However, players still lead when we consider accuracy to be in the top-20% (+3.5 points [0.8, 6.3]) and top-30% (+3.3 points [0.5, 6.2]) of actions.

SEEK also significantly improves skill over NEUTRAL: +0.097 [0.053, 0.141] for GPT, +0.058 [0.006, 0.102] for Gemini, and +0.129 [0.079, 0.178] for Claude.

Table 2 vs. Table 8. Table 2 includes all 726 men's events, whereas Table 8 uses the 529 events with labeled gameplay frames. We will clarify this distinction.

Figure 5. The plotted values pool both tournaments and are correct. The corresponding numbers in Section 3.2 and the caption are inconsistent and will be corrected.

SEEK more than doubles Claude's skill from 0.085 to 0.214 and brings GPT to 0.050 above players, although this difference is not statistically significant [-0.012, 0.111].

We thank the reviewer again. Their suggestions strengthened the validation, the reading of risk preference, and the reporting of uncertainty.

AuthorsOpenReview ↗

Follow-up response

We thank the reviewer for their follow-up. We address and resolve their remaining concerns.

1. Counterfactual action rankings

Label sensitivity. We re-scored all 1,421 events under seven variations of the two counterfactual estimates identified by the reviewer. For the post-pass state (v⁺), we shifted the reception point 2 m toward goal or 2 m toward the passer, or applied random jitter (SD 2 m per axis, five draws), re-scoring each state with VAEP. For the failure outcome (v⁻), we set it to zero, used finer bins (lane and depth; 6 m shot bins), refit it on 200 bootstrap resamples of training matches, or replaced it with a synthesized turnover state scored by VAEP.

VariationLabel unchangedDecisive eventsOriginal optimum in top two
Reception 2 m toward goal88.1% [86.4, 89.9]94.0%98.1%
Reception 2 m toward passer86.4% [84.4, 88.4]91.8%95.8%
Reception jittered85.1% [83.8, 86.4]91.3%95.5%
v⁻ = 091.9% [90.6, 93.2]97.2%98.9%
v⁻ finer bins96.3% [95.2, 97.2]98.8%99.5%
v⁻ bootstrap refit97.7% [97.3, 98.1]99.4%99.8%
v⁻ synthesized turnover70.9% [68.6, 73.2]80.3%96.2%

Decisive events are those where the optimum leads the runner-up by at least one standard deviation of the event's candidate action values. When a perturbation changes the label, the original optimum trails the new one by a median of only 0.1–0.6 value spreads. The synthesized turnover state represents a structurally different failure model. Even under this variation, the original optimum remains in the top two for 96.2% of events.

All principal conclusions hold under every variation. After source-pool reweighting, every VLM remains significantly below players in accuracy and skill (GPT 5.6 Sol: −4.7 to −7.6 percentage points and −0.060 to −0.134 skill). The optimum remains at the 74th–85th risk percentile, players at the 59th–60th, and every VLM at the 45th–54th. SEEK significantly improves skill over NEUTRAL for GPT 5.6 Sol (+0.067 to +0.099) and Claude Opus 4.8 (+0.106 to +0.132). These results complement the appendix noise analysis, which perturbs outcome spread rather than counterfactual construction.

Figure 1 candidates (variance as in Eq. (5)):

Optionpv⁺v⁻EVVariance
D (optimal)0.8990.14450.00630.13060.0017
A0.9000.08170.00150.07370.0006
G0.4860.14450.00630.07350.0048
B0.9660.05960.00630.05780.0001
E0.8310.06280.00630.05330.0005
C0.9380.04820.00630.04560.0001
SHOOT (player)0.03310.00780.04070.0316
F0.2150.12420.00630.03160.0023

D and G share v⁺ and v⁻, so their value difference arises entirely from completion probability. We will add both tables to the appendix.

2. Risk preference and its explanation

Stated risk metric. We repeated the matched-pass comparison using the payoff variance defined in Eq. (5), and the result persists. Among alternatives within 0.01 EV and 5 m of pass distance, the chosen pass has lower variance than 0.674 [0.634, 0.716] of alternatives for GPT 5.6 Sol and 0.587 [0.542, 0.629] for Claude Opus 4.8, compared with 0.496 [0.443, 0.551] for players. The model–player differences are +0.178 [0.107, 0.245] and +0.091 [0.016, 0.163], respectively. The result persists at EV tolerances of 0.005 and 0.02 and responds consistently to risk-oriented prompts. On the men's set, AVOID increases the measure (GPT 0.631→0.723; Claude 0.554→0.648), while SEEK brings it close to the players' level (0.500 and 0.522, vs. 0.521).

Belief results. We will revise the conclusions to present probability–value conflation as an explanation consistent with conservative choices, rather than an established causal mechanism.

Reporting. We will distinguish the two sampling schemes explicitly. Under regret-tier sampling, GPT 5.6 Sol's accuracy is not significantly below players'; after source-pool reweighting, every VLM's accuracy is significantly lower. We will describe SEEK as a significant improvement over NEUTRAL, without claiming an advantage over players.

We thank the reviewer again for their constructive suggestions.

We thank the reviewer again for their thoughtful feedback! We hope our follow-up has addressed the two remaining concerns:

  1. Counterfactual reliability: Our principal findings remain robust across seven alternative constructions of counterfactual action values, including substantially different assumptions about unsuccessful actions.
  2. Risk preference: Using the paper's stated payoff-variance metric, we confirm that models consistently prefer lower-risk actions even after controlling for expected value and pass distance.

Together with our earlier responses and the revisions we have committed to making, we hope these additional analyses resolve the reviewer's remaining concerns. We would be grateful if the reviewer would consider raising their score.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.