Towards a General AI Prediction Arena

Prediction, the ability to gather evidence and reason under uncertainty before an outcome is known, is a cornerstone of human intelligence and a critical frontier for artificial intelligence. Evaluating this capability requires testing models against true future events to prevent data contamination, and modern benchmarks employ live, dynamic arenas to achieve this. However, these systems suffer from three critical limitations. First, temporal asymmetry leads to unfair comparisons between early forecasts made under severe uncertainty and late predictions that benefit from accumulated evidence. Second, their question pools rely heavily on popular prediction markets while neglecting specialized, long-tail domains. Third, models cannot be scored until real-world events finally resolve, stalling iterative development. We introduce Echo, a live and general AI prediction arena that addresses these challenges. Echo implements a multi-point aligned Elo rating system that schedules predictions across an event's lifecycle and compares models only at the exact same prediction point. We propose a data acquisition pipeline combining prediction markets, automated trending question synthesis, and domain expert annotation for broad coverage. Echo further introduces narrative rubric evaluation, where rubrics are algorithmically evolved to score reasoning trajectories independently of the final outcome, providing immediate feedback for model iteration. In robustness experiments, even when half the participants miss prediction days, time-weighted Brier rankings have up to $$\begin{gathered} 2.18 \\ times \end{gathered}$$ the rank standard deviation of aligned Elo. In a revised seven-model rubric evaluation, scores computed without the recorded outcomes correlate with Elo rankings at $rho=0.857$.