acceptodds
Withdrawn from ICLR 2027

WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear prediction before the answer is available. We present WorldCupArena, a dynamic benchmark for language models and deep-research agents. The 2026 FIFA World Cup is its first evaluation, and the same process can be reused for future leagues and cups. Before each match, a model either receives a common evidence package or searches for information itself. It predicts the result and score, likely players and events, match statistics, and the outcome of the competition. After the match, these predictions are compared with the recorded result. We report result accuracy, exact-score accuracy, and a scoreline score that gives some credit when a predicted score is close but not exact, together with scores for the other prediction tasks. Across systems, similar result accuracy can mask larger differences in detailed predictions. Four systems predicted champion Spain, and two of them also recovered the exact final pairing. Compared with betting-market and human-fan baselines, the best system shows only small gains in result and exact-score accuracy, but a clearer gain in Scoreline. New schedules can be added as they begin, allowing the benchmark to evaluate future models without using outcomes that are already known. Code, predictions and evaluation scripts will be publicly released.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Benchmarks, Football Forecasting, Deep-Research Agents

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_i6kFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

WorldCupArena evaluates LLMs and agents capability on forecasting 2026 World Cup. It compares configurations across match outcomes, scorelines, players, events, statistics, and tournament predictions. The baselines includes market and fan data. Results suggest that detailed metrics distinguish systems with similar outcome accuracy, at the same time, search doesn't seem to show significant improvements.

Critical strengths

I do think the most important strength lies in showing that evaluating several related prediction goals can reveal more in-depth differences in forecasts that outcome accuracy alone misses, and creates opportunities to study forecasts at greater depth.

Critical weaknesses
  1. Predicting lineups, goal times, and statistics creates more evaluation tasks, but the paper does not establish that these measurements reveal meaningful agent capabilities beyond sports knowledge, statistical regularities, or access to better evidence. For a paper that entirely discusses benchmarking a model on one sports event, it needs to convince us what each granular evaluation metric proposed is meant to demonstrate. Of course, for any forecasting task, one can create many intermediate proxies and subtasks to demonstrate the differences between the forecasters, but I believe we need to at least have a good argument on exactly what those subtasks and proxies are measuring: models' logical capability? models' statistical intuition? models' risk parameters?

  2. Model rankings, inconsistent search gains, and shared failures on upsets are qualitative observations. This work does not adequately isolate why systems differ or demonstrate that the evaluation changes our understanding of forecasting agents. Claims about broader applicability remain unvalidated.

Combined together, my main concerns for this work is, what scientific properties of agents can we learn from this work? How can we generalize any of these findings and frameworks to beyond the 2026 World Cups?

Questions for authors

What do the granular metrics measure? For each evaluation, what specific capability or forecasting behavior is it intended to measure? In particular, why should differences on these subtasks be informative about forecasting agents more generally?

What scientific insight does the evaluation establish? Beyond ranking the evaluated systems, what is the main finding about forecasting agents that the benchmark enables?

What generalizes beyond this tournament? Which conclusions or metric interpretations should hold beyond the 2026 World Cup, under what assumptions, and what evidence in the current study supports that expectation?

Reviewer_R8PiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper deals with LLM evaluation through football forecasting, especially for the 2026 World Cup. For this, the authors performed the prediction 24 hours before kickoff. The evaluation was conducted across five different layers, and the focus was on comparing the effectiveness of general systems (S1) and search-oriented systems (S2). The authors insist that there were no consistent differences between S1 and S2, and the best system demonstrates a gain in terms of scoreline compared with the betting market and human-fan baseline.

Critical strengths
  • No clear strengths
Critical weaknesses
  • I am not sure if the scoreline metric is a widely used one. The weights in equation 1 seem a bit arbitrary. If the authors themselves designed this metric, then the insisted gain against the baselines seems less meaningful.
  • The authors insisted that the inference was conducted 24 hours before the kickoff, which might imply that the experiment cannot be reproduced based on its characteristics. For instance, since each game is now publicly accessible, search-based S2 predictions would be significantly affected (although one can adjust the search dates). While I understand the structure and characteristics of this, I am unsure whether we can call it science without a method to reproduce the results. Note that the authors did not submit the code or raw result.
  • The authors mention that "fixture intersections differ". What is the meaning of this? Does each model use different games for evaluation? In this case, I believe we cannot interpret current results.
  • Are the current results statistically significant? I couldn't find the part to ensure this or the p-value.
  • In B.1, the authors stated that they use "pre-lock market odds" for S1. What does this mean? What is the difference between this and Polymarket/BetVictor?
  • Unclear information about "internal football-fan group"
Questions for authors

See Weaknesses

Reviewer_KtWjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors explore the task of predicting outcomes of soccer matches, where they focus on the WC2026 and explore not just the outcome itself, but also various subtasks to predict various aspects of the game, and also the final outcome of the tournament. To do so, they provide an evaluation framework which can be used to compare various predictors in a fair manner.

Critical strengths
  • Interesting topic being explored.
  • The authors contribution seems useful for the stated purpose.
  • Interesting evaluation presented in the experiments.
Critical weaknesses
  • The novelty is quite limited, where the authors provide a framework to run eval and not much beyond that.
  • It is a useful platform, however in the context of ICLR conference and added value the contribution seems quite below the bar.
  • The experiments are very weak, with very limited to no deeper insights being shared.
Questions for authors
  • The authors explore a single data set, making the generalizability and wider usefulness of the eval platform questionable.
  • "from an internal football-fan group", this is a strange baseline to include, with very vague explanations that are not adding a lot of value.
  • The authors use public data and compare a number of existing models, where they only provide an evaluation platform. There is very limited noveltly presented, and the authors should have at least shown some more interesting insights, results, recommendations, or anything like that. This is all missing, and the results in general are presented in a very black-box manner.
  • More qualitative analysis should also be provided, which would reduce this issue of black-box approach and strengthen the experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.