acceptodds
Withdrawn from ICLR 2027

Towards a General AI Prediction Arena

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Prediction, the ability to gather evidence and reason under uncertainty before an outcome is known, is a cornerstone of human intelligence and a critical frontier for artificial intelligence. Evaluating this capability requires testing models against true future events to prevent data contamination, and modern benchmarks employ live, dynamic arenas to achieve this. However, these systems suffer from three critical limitations. First, temporal asymmetry leads to unfair comparisons between early forecasts made under severe uncertainty and late predictions that benefit from accumulated evidence. Second, their question pools rely heavily on popular prediction markets while neglecting specialized, long-tail domains. Third, models cannot be scored until real-world events finally resolve, stalling iterative development. We introduce Echo, a live and general AI prediction arena that addresses these challenges. Echo implements a multi-point aligned Elo rating system that schedules predictions across an event's lifecycle and compares models only at the exact same prediction point. We propose a data acquisition pipeline combining prediction markets, automated trending question synthesis, and domain expert annotation for broad coverage. Echo further introduces narrative rubric evaluation, where rubrics are algorithmically evolved to score reasoning trajectories independently of the final outcome, providing immediate feedback for model iteration. In robustness experiments, even when half the participants miss prediction days, time-weighted Brier rankings have up to the rank standard deviation of aligned Elo. In a revised seven-model rubric evaluation, scores computed without the recorded outcomes correlate with Elo rankings at .

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Agents, Agentic Prediction, Online Arena

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_jp6kOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a platform for continuously and fairly evaluating the capabilities of agents in AI forecasting. Specifically, when comparing agents, the platform compares only those that made predictions at the same time and uses an Elo score based on comparisons across multiple prediction time points to assign ratings. Furthermore, this paper devises a method to score the reasoning process behind predictions and, by comparing these reasoning scores with the prediction scores themselves, explores where agents that make excellent predictions excel in their reasoning process.

Critical strengths

The attempt to compare agent performance using predictions made at multiple time points for the same forecasting questions represents a contribution to this field. This approach aims to provide a fairer comparison than existing forecasting benchmarks.

Critical weaknesses

The evaluation metric used in this paper is not a proper scoring rule, raising doubts about its validity as an evaluation metric. For example, given a true distribution p=(0.6, 0.3, 0.1) and a predicted probability distribution q=(2/3, 1/3, 0), a higher score is assigned to q than to p, even though q differs from the true distribution. Consequently, the analyses and interpretations in the paper based on this metric remain questionable.

Questions for authors

Although the appendix also uses the standard multi-class Brier score, is there a reason for adopting an evaluation metric that is not a proper scoring rule?

Reviewer_jZgTOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Echo, an arena for evaluating LLM forecasting agents. Questions are derived from several diverse sources and a pipeline then converts these questions into a standardized question database. Models are ranked based on Elo rating.

Critical strengths

A key critical strength is that it is taking a key problem (forecasting real-world events) and trying to standardize the framework for evaluating models on this key question, using time-locked systems and corpuses.

Critical weaknesses

I have several major concerns:

  1. The framework ranks models based on an Elo rating, which is not in any way a proper scoring rule. A model maximizing Elo rating on tasks like this will have clear incentives not to report its true forecasts on a set of questions.

  2. I felt that I needed significantly more information about the data (that seeds the arena). In other words, how many questions are there, what is the structure of 'battles', and so on.

  3. I do not trust this timelocked approach to data leakage. It seems that the best way to run this project is to have a live set of short-run questions on which it is impossible for a model to snoop/peek at the answers.

Questions for authors
  1. How do Echo's rankings compare to classic proper scoring rules (Brier scores would work).

  2. Please consider better tests of leakage from test to training data, especially as models are re-trained or released using more up-to-date future models.

  3. Can you do a better job of motivating the value of this type of framework vs. the forecasting-related benchmarks that already exist outside of this paper.

Reviewer_tVzMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors introduce Echo, a live benchmark for testing how well AI agents perform forecasting over real-world events. Instead of averaging scores across predictions made at different times, it compares models only when they answered the same question at the same moment, and ranks them with an Elo-style system. They also score unresolved questions using an LLM judge to grade the reasoning behind each forecast. The authors claim that Elo rankings are more stable than standard Brier scores but it is unclear whether Elo recovers the same ranking as Brier. Further, evaluating the reasoning of the models based on rubric sounds promising as an approach but is tested on only seven models and requires further justification.

Critical strengths

The paper tries tackles a real problem in live forecasting evaluation with a simple, well-motivated fix: compare models only when they answered the same question at the same time and develop a Elo-based metric which seems to provide more stable rankings. The question pipeline is a noteworthy contribution, broadening coverage beyond popular markets to tail events through news-driven synthesis, expert input, and a detailed taxonomy.

Critical weaknesses
  1. The formatting at times has been squeezed a lot to fit the text which is not recommended and goes against the ICLR guidelines I believe. (For example, see lines 089-091 on Page 2 where there is no newline despite new paragraphs).

  2. The scoring rule used in the paper is improper for multi-option questions. The authors use only (1 − p_correct)², which matches Brier for binary questions but not otherwise. Example: if the true probabilities are (0.5, 0.3, 0.2), reporting roughly (0.61, 0.36, 0.03) gets a lower expected loss than reporting the truth. This incentivizes incorrect reporting! What is the rationale behind not using a proper scoring rule?

  3. The rubric methodology seems flawed to me. For example, the sample shown in the Appendix with low (zero) rubric score is actually correct. The model had made the right prediction with the question having been resolved before the resolution deadline but gets 0 based on the rubric given that the model is able to find the answer. Often, questions essentially resolve earlier than the planned resolution date in which case the model might be able to arrive at the answer easily by looking up the latest developments. This is indeed the correct thing to do in my opinion but this would be penalized by the rubric. Another issue with the rubric is its (fixed) axes. For example, one of the axes is "Historical Base Rate Analysis" but even this might not be calculable if the question itself is about a one off event, for example say "When will the covid lockdown be over". Covid only happened once in the world and it's hard to calculate a base rate for such questions. Given that rubrics have pre-defined axes, it seems easily gameable if the model just pretends to show "probability deviation", "causal analysis" and "uncertainty awareness" for the sake of doing it but not because it wants to make the best calibrated guess. The judge only seems to check whether these concepts were covered in the response but that does not say anything about whether the reasoning itself seems promising or not.

  4. The paper shows Elo ranking but never shows how does it compare to a ranking based on Brier computed only on the aligned prediction points. It’s unclear whether the benefit comes from the time alignment or from the Elo model itself. Further, apart from stability, what is the benefit of Elo over normal ranking based on Brier score computed only on the aligned points?

  5. No details is given on how the web search is provided to the models so that they only have access to information until the prediction point on which they are being tested?

  6. The authors claim in L712 that ".. removes questions where one option is already near-certain, since such questions provide little signal for comparison.", yet both samples shown in the Appendix showcase examples that are truly near-certain (with the model getting both correct with 98% and 99% probability). The question pipeline does not seem to be working properly.

  7. The robustness experiments in Section 4.3 show that Elo rankings move less than Brier rankings, but each method is compared only against its own full-data ranking. This measures stability rather than correctness, since a ranking can be consistent and still wrong. However, the authors never show whether Elo recovers the true ordering from Brier score more accurately?

  8. I don't see any supplementary material being shared. I would appreciate if the code and the dataset was shared by the authors. How big is the dataset? What is the composition breakdown percentages? More details on all of this would be useful.

Questions for authors
  1. Can the authors share the control experiment -- how does their Elo compare with Brier differences computed only on aligned points?

  2. Can the authors explain what is the point of expert annotation paragraph if they are not being used in the system?

  3. How was the weight for the rubric criteria determined?

  4. σ seems to move some of the ranks a lot (Fig. 8). For example, sweeping σ from 0.01 to 0.5 sends GPT-5.2 anywhere from #2 to #13. Also, why does the rubric–Elo comparison use different settings (σ = 0.5, γ = 0) from the leaderboard (σ = 0.1, γ = 1)?

  5. No details are provided on how many questions is the Fig.1 based on? What is the dataset size and composition breakdown? Appendix Section A repeats what is said in the main paper and never provides concrete details like which outlets are the questions sourced form, which prediction markets, etc.? I request the authors to please provided more details on the pipeline.

  6. There are no error bars reported in Table 1? Can the authors share how noisy was the ranking estimation from rubric? Why is rubric reported only on 7 models in Table 1 whereas Elo is reported for many more models in Fig.1?

  7. No details are provided on how the experts were recruited -- how did they help, how were they paid, and did the study go through an Ethics board to recruit?

Reviewer_yAnPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Echo, a live forecasting benchmark for evaluating AI systems on unresolved real-world events. It addresses three limitations of existing forecasting benchmarks: temporal asymmetry, narrow question coverage, and delayed feedback. The proposed system combines time-aligned Elo ranking, a question acquisition pipeline drawing from multiple sources, and an outcome-free rubric evaluation that provides feedback before events resolve.

Critical strengths

Forecasts made at different times have access to different information, making temporal alignment an important consideration for fair model comparison. Echo integrates question acquisition, prediction scheduling, ranking, and pre-resolution feedback rather than focusing on a single evaluation metric.

Critical weaknesses
  • Echo combines time alignment, matched pairwise comparisons, soft labels, and Elo/Bradley--Terry aggregation, but the main baseline uses Brier-based ranking. It is therefore unclear which component drives the reported robustness gains. A matched-point Brier baseline using exactly the same question--time pairs would be particularly useful. The paper also does not quantify the actual overlap in questions and prediction points across models on the live leaderboard, making it difficult to assess how serious the temporal mismatch is in practice.
  • The question acquisition pipeline aims to improve coverage, but expert-authored questions are not included in the resolved evaluation, and results are not sufficiently broken down by question source. For the outcome-free rubric evaluation, the main evidence is its correlation with Elo. However, the authors describe this result as exploratory because the model set was revised after an earlier evaluation. A held-out evaluation with a fixed protocol would provide stronger evidence for this component.
  • Better comparison with related works is needed.
Questions for authors
  • What is the actual overlap in questions and exact prediction points across models on the live leaderboard? How large is the temporal mismatch that the proposed ranking method aims to address?

  • In the ranking robustness experiments, how does Brier-based ranking compare with Elo when both are evaluated on the same question--time points? This would help separate the benefit of temporal alignment from that of the ranking method itself.

  • Could the authors report results separately for resolved prediction-market and synthesized questions? It would also be helpful to report the size and coverage of the expert-authored question pool, since these questions are not yet included in the resolved evaluation.

  • Could the rubric--Elo correlation be validated on held-out questions or models, with the model set, rubric, judge, and evaluation protocol fixed in advance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.