acceptodds
Under review as a conference paper at ICLR 2027

Cup Arena: Can Large Language Models Beat the Market? A Live 2026 FIFA World Cup Benchmark

Abstract

Can large language models beat betting markets in live football forecasting, and improve as match outcomes accumulate? We investigate this research question with CupArenaBench, a benchmark built from the 2026 FIFA World Cup. Across 103 matches, we require 10 frontier LLMs to submit real-time scoreline and win–draw–loss forecasts before kickoff. Each resolved match provides new evidence for subsequent forecasts. We find that most frontier LLMs outperform conventional forecasting baselines. This pattern persists when market odds are withheld. We then study whether LLM forecasters can improve as new match outcomes become available. Dynamic memory allows agents to retain lessons from resolved matches. Our Forecast Harness enables them to execute forecasting code. As matches resolve, the forecasting code evolves from the newly observed outcomes. This continual adaptation leads to progressively better forecasts as the tournament unfolds. We further introduce Football-RL, which updates model parameters from the chronological match stream. The trained policy also transfers zero-shot to the Premier League, improving over the base policy. Together, these results show that LLM forecasters can continually improve from one-shot real-world feedback as new matches resolve. We release the benchmark, data, and code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pZnaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CupArena, a chronological football forecasting benchmark built around the World Cup, together with the CupGym environment. It investigates whether large language models can outperform market and crowd forecasts and improve through sequential outcome feedback. Models commit scoreline predictions and win/draw/loss probabilities before kickoff, while progressively richer configurations incorporate dynamic memory, executable forecasting programs, and market information. Forecast Harness Evolution (FHE) generates and executes program variants before a match, selects a variant after the outcome is resolved for use in subsequent matches, and promotes code fragments to a skill library when evidence and reward thresholds are met. The paper also proposes Football-RL, which combines current and recent rewards to construct advantage estimates and adaptively allocates sampling resources, updating model parameters after each matchday for subsequent evaluation. Experiments compare against statistical, market, and crowd baselines, assess probabilistic and betting performance, and examine different model sizes and retrospective transfer to Premier League matches.

Critical strengths
  1. The paper addresses the practically relevant problem of forecasting unresolved events and adapting to sequential feedback. Committing official forecasts before kickoff establishes a clear temporal boundary between prediction and outcome observation. FHE also explicitly distinguishes the incumbent forecast used for the current match's evaluation from candidate forecasts used to select the next program, avoiding the use of the retrospectively best candidate as the official prediction for that match.

  2. The progression from direct prediction to dynamic memory, executable programs, and market access provides a useful framework for studying different forms of adaptation. Team-specific and global memories, traceable code fragments, and parameter updates offer complementary mechanisms for retaining experience. The analysis of draw underprediction is particularly informative, connecting aggregate performance changes to a concrete forecasting behavior that feedback may help correct.

  3. The evaluation includes statistical, market, and crowd baselines, compares Football-RL with GRPO and DAPO at different model scales, and adds a cross-competition transfer study. Assessing Brier scores and betting performance alongside accuracy provides several perspectives on forecast quality and practical value. The appendix also includes paired confidence intervals for selected comparisons and repeated training runs, offering some evidence about the uncertainty and variability of the reported gains.

Critical weaknesses
  1. FHE lacks a sufficiently reliable mechanism for program selection and fragment-level credit assignment. Program variants are selected according to their Brier scores on a single realized match outcome. Given the stochastic nature of football, the winning variant may reflect outcome noise rather than better expected forecasting performance. The skill library then treats rewards of selected programs as evidence for the fragments they contain, without isolating a fragment's contribution from that of other components or favorable outcomes. This creates a risk of reinforcing programs and fragments that were merely fortunate. Committing candidates before kickoff preserves the temporal validity of the current match's evaluation, but does not resolve selection noise or attribution during evolution. Validation across multiple matches and controlled fragment removal experiments would help establish whether the accumulated skills provide transferable predictive value.

  2. The connection between Football-RL's proposed mechanisms and the learning difficulties they are intended to address is insufficiently established. Historical rewards come from different fixtures and policy versions; pooling them with current rewards for normalization mixes variation in match difficulty, outcome uncertainty, and policy quality. The resulting advantage shifts therefore need not reflect improvements in the current forecasting behavior. Similarly, increasing the number and length of rollouts provides more candidate responses but no additional independent match outcomes. Low reward dispersion alone does not establish a need for further exploration, since it could also reflect agreement among predictions. The paper does not adequately explain when these mechanisms should improve learning under noisy feedback, and lacks separate ablations of the historical reference and adaptive sampling under matched computational budgets.

  3. The methodological novelty and the independent contribution of the proposed mechanisms remain insufficiently demonstrated. Prior work, including Outcome-based Reinforcement Learning to Predict the Future, has already studied training forecasters with realized event outcomes and Brier rewards, including forecasting-specific modifications to GRPO advantage normalization. Football-RL's methodological contribution therefore rests on the particular historical reference and resource allocation mechanisms, whose benefits require more direct comparison with existing forecasting RL approaches. Attribution is also difficult for FHE: moving from S2 to S3 introduces retrieval, computational tools, and program evolution together. Without fixed-program baselines using the same information and tools, simple online probability calibration, and more closely related RL baselines, the results provide stronger support for the benchmark and integrated system than for the necessity and distinct advantages of the new learning mechanisms.

Questions for authors
  1. Is any consistency requirement imposed between the predicted scoreline and the win/draw/loss probabilities? If the outcome implied by the scoreline differs from the highest-probability outcome, how are betting decisions and the accuracy component of the training reward determined? Could the authors clarify how the composite Brier-and-accuracy reward affects probabilistic calibration and report diagnostics for output consistency and forecast reliability?

  2. How does the memory system determine whether a resolved match supports or contradicts a stored lesson? The occurrence of an unlikely outcome does not, by itself, invalidate the underlying probability estimate or reasoning. How do the Beta count updates account for this outcome noise, and how is correlated evidence handled when the same match contributes to team memories, global memory, or similar records?

  3. How are FHE updates, memory writes, and matchday-level Football-RL updates scheduled when fixtures overlap or kick off simultaneously? Could the authors specify the program, memory, and model checkpoint used for each official forecast, and clarify whether repeated training runs were conducted concurrently during the live tournament or through retrospective replay? This would help distinguish strictly prospective results from replay-based evaluations.

Reviewer_s36dOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper performs an out-of-sample evaluation of a series of LLM forecasting methods on 2026 soccer World Cup games. They claim their proposed methods enable the model to substantially outperform human and market forecasts. Their first method (S2) is to equip the models with external memory, their second method (S3) enables the models to create and edit forecasting programs that they use to submit predictions, and their third method (S4) is S3 provided with market odds. They also introduce an RL algorithm that they use to finetune an LM on these ~100 examples from the World Cup, which they claim generalizes to a different sport.

Critical strengths

The topic area is interesting and timely, and the paper introduces a large number of potential methods for improving performance on this problem.

Critical weaknesses
  • One of their main contributions is a live benchmark (CupArenaBench). This benchmark is not live or reusable under the same assumptions as when they did their evaluation; frontier models now have access to the outcomes. Moreover, for a benchmark that is susceptible to leakage, it has a relatively small number of examples.
  • Their primary method, the Forecast Harness Evolution (FHE) is underspecified. It's not clear what the programs look like, how they are mutated, and no qualitative analysis that shows how or why the method works.
  • Their Football-RL method is poorly specified and generally not given sufficient real estate in the paper given the importance of the contribution. It seems to just be GRPO with minor changes. The novelty of those minor changes is not clear.
  • Their evaluation on premier league games is susceptible to leakage since agents are given access to web search and their method for dealing with this sounds fragile
Questions for authors
  • How does the football RL algo compare to prior work?
Reviewer_UhDZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CupArenaBench asks whether LLMs can outperform football forecasting baselines and improve from sequential outcomes. Its progression is direct prediction, dynamic memory executable, evolving forecasting programs, to final market-aware forecasting. It uses the past 2026 Football World Cup results (103 matches) as the benchmark and tested 10 frontier LLMs. The paper also develops of its own harness for forecasting football games and proposes Football-RL as a reinforcement learning recipe that helps them beat other baseline RL methods in forecasting accuracy.

Critical strengths

Sequential forecasting setup: most existing work on LLM forecasting focuses on setups when the forecasting questions and problem context are independent -- so evaluation and learning cannot borrow information from previously finished (and relevant) events. The sequential setup that's fundamental to this work opens up new possibility measuring the adaptivity and continual-learning ability of LLM models (with an agentic harness).

Solid comparisons: 10 models are evaluated in the benchmark against human/market baseline. In RL training, the paper addresses two model sizes, include GRPO/DAPO comparisons, paired intervals, and repeated runs provide more support of the reported results.

Critical weaknesses

mechanism attribution (ablation): supplied experiments do not isolate FHE, temporal calibration, and rollout allocation under matched resources. This limits algorithmic conclusions without negating system-level gains

Questions for authors
  1. For Figure 5: Which models/fixtures enter each market result, and why do the denominators differ; how are timestamped market probabilities constructed?
  2. With matched retrieval and rollout budgets, how much improvement remains when FHE or each Football-RL component is disabled? (related to the weakness above)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.