- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a platform for continuously and fairly evaluating the capabilities of agents in AI forecasting. Specifically, when comparing agents, the platform compares only those that made predictions at the same time and uses an Elo score based on comparisons across multiple prediction time points to assign ratings. Furthermore, this paper devises a method to score the reasoning process behind predictions and, by comparing these reasoning scores with the prediction scores themselves, explores where agents that make excellent predictions excel in their reasoning process.
Critical strengths
The attempt to compare agent performance using predictions made at multiple time points for the same forecasting questions represents a contribution to this field. This approach aims to provide a fairer comparison than existing forecasting benchmarks.
Critical weaknesses
The evaluation metric used in this paper is not a proper scoring rule, raising doubts about its validity as an evaluation metric. For example, given a true distribution p=(0.6, 0.3, 0.1) and a predicted probability distribution q=(2/3, 1/3, 0), a higher score is assigned to q than to p, even though q differs from the true distribution. Consequently, the analyses and interpretations in the paper based on this metric remain questionable.
Questions for authors
Although the appendix also uses the standard multi-class Brier score, is there a reason for adopting an evaluation metric that is not a proper scoring rule?