- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FinVerse, a large-scale benchmark for financial time-series forecasting that organizes series by their economic meaning and evaluates models through point-wise forecasting, cross-sectional ranking, and portfolio backtesting. The full collection contains 116,897 series and 171.1M observations, with a selected evaluation set of 60,232 series. The study evaluates 43 public time-series foundation models in a zero-shot setting. Overall, I find the paper interesting and the benchmark potentially useful. Although there are already many benchmarks for time-series foundation models, FinVerse's relatively low ranking correlation with GIFT-Eval is an important result: a model's performance on an established general-purpose leaderboard may leave meaningful domain- and objective-specific blind spots.
Critical strengths
-
The benchmark reveals potentially important blind spots in existing leaderboards with modest ranking correlations (Pearson , Spearman , and Kendall ). Reverso Small, for example, ranks first on FinVerse despite a weaker GIFT-Eval rank. These differences could help researchers identify where models have room to improve and motivate more targeted training and evaluation.
-
Evaluation connects forecasting to downstream objectives. The particularly weak relationship between point-wise and portfolio rankings () reinforces the importance of not treating forecasting accuracy as synonymous with investment utility. I also appreciate that the paper explicitly distinguishes proxy metrics from realized outcomes under a particular portfolio strategy.
-
FinVerse extends beyond equity-only evaluation to include currencies, commodities, crypto assets, fixed income, macroeconomic indicators, ETFs, and fundamentals, across multiple frequencies. Evaluating 43 models provides a substantial empirical comparison. The category- and metric-specific results are useful because they show that no model is uniformly best and encourage model selection based on the intended task rather than a single generic leaderboard.
Critical weaknesses
-
Major weakness: the paper does not explain why FinVerse differs from GIFT-Eval. The benchmark reveals a meaningful difference in model rankings, but offers little analysis of what makes FinVerse distinctive. In particular, it does not compare the benchmarks' data statistics, frequency distributions, or other time-series attributes to identify what may drive the ranking differences. Nor does it separate the effect of changing the data domain from the effect of changing the evaluation metrics: FinVerse and GIFT-Eval differ along both dimensions. Analyses with matched metrics and data, and performance stratified by measurable series characteristics, would help explain the observed blind spots and make the paper more useful for guiding model development. Without such analysis, the work establishes that rankings differ, but gives limited insight into why.
-
Ranking robustness and data availability deserve clearer validation. The main-text results would be stronger with uncertainty estimates and sensitivity analyses for the rank aggregation and context-window choices. This matters when drawing conclusions from leaderboard differences and cross-benchmark correlations. The acknowledged possibility of pretraining overlap and imperfect point-in-time consistency also limits how confidently the results can be interpreted as strictly unseen, deployable forecasting performance. These limitations do not negate the benchmark's value, but should remain explicit in its claims.
-
The benchmark omits intraday frequencies. FinVerse covers daily and lower-frequency data, but does not include minute- or second-level financial time series. These high-frequency settings are important in many financial applications and have distinct data characteristics and forecasting requirements. Their absence limits the benchmark's coverage of financial forecasting, so claims about its breadth should be scoped accordingly.
Questions for authors
-
Can the authors compare FinVerse and GIFT-Eval in terms of data statistics, sampling frequencies, and other time-series attributes, and analyze which differences are associated with model ranking reversals?
-
Can the authors report model-ranking correlations between FinVerse and a broader set of established time-series benchmarks such as fev-bench/TIME/BOOM, rather than GIFT-Eval alone? It would also be useful to compare correlations across FinVerse's evaluation aspects and, where possible, use matched metrics to help separate differences due to the data domain from those due to the evaluation objective.
-
How stable are the main rankings and correlations across evaluation periods, aggregation choices, and context lengths?
13 comments
Rebuttal [2/6]
[Q3] Imperfect point-in-time consistency
[A3] Because publicly available point-in-time (PIT) data could not be obtained for all datasets, the original forecasting experiments did not account for PIT constraints. To address the reviewer's concern, we conducted the following PIT-aware evaluation. For fundamentals collected from the SEC, release dates and revision histories are available and can be used to construct point-in-time (PIT) inputs. We therefore report PIT-aware results for selected leading models on Fundamentals. The Spearman correlation between the original and PIT-aware Fundamentals rankings was 0.8908. This high correlation of approximately 0.89 suggests that, although PIT issues remain in the benchmark, their impact on the ranking patterns examined in this experiment is limited.
Experiment: PIT-Aware Fundamentals Evaluation
Using the original SEC data, we constructed inputs from the historical vintages available at each forecast origin. For each company, the forecast origin was the release date of the latest quarterly financial results included in the input, and the model predicted the next four quarters. We first report results for nine models selected from the leading models in the original Fundamentals evaluation.
| Model | Original FinVerse Fundamentals Rank | PIT-Aware FinVerse Fundamentals Rank |
|---|---|---|
| TiRex 1.1 | 3 | 2 |
| TiRex 2 ZS | 2 | 4 |
| VisionTS | 1 | 1 |
| Super Linear | 6 | 5 |
| Timer-S1 | 11 | 8 |
| TiRex 2 Pretrain | 4 | 2 |
| Reverso Small | 8 | 5 |
| Chronos-2 | 5 | 7 |
| TimesFM 1.0 200M | 12 | 9 |
[Q4] possibility of pretraining overlap.
[A4] To assess the concern about pretraining contamination, we reevaluated nine models uploaded to Hugging Face before January 1, 2025, restricting the evaluation period to 2025–2026. Among the same nine models, the Spearman correlation between the full-period FinVerse ranking and the 2025–2026 ranking was 0.7915. Comparing the 2025–2026 ranking with the ranking for 2020–2024, a period that may overlap with the models' pretraining data, also yielded a correlation of 0.7143. In light of the year-to-year ranking variation reported in Table 18, these correlations indicate substantial agreement. Although this analysis does not establish that the original evaluation was entirely unaffected by pretraining overlap, it shows that the relative performance of the models is largely preserved when evaluation is restricted to observations from after the models' public release.
Experiment: Overall Rankings Among the Same Nine Models
| Model | Original FinVerse (2020–2026) | 2020–2024 | 2025–2026 |
|---|---|---|---|
| Chronos-Bolt Base | 1 | 1 | 1 |
| TimesFM 1.0 200M | 2 | 2 | 4 |
| Chronos-Bolt Small | 3 | 3 | 6 |
| TimesFM 2.0 500M | 3 | 4 | 1 |
| Moirai 1.1-R Large | 5 | 6 | 4 |
| Moirai 1.1-R Small | 5 | 5 | 3 |
| Moirai 1.1-R Base | 7 | 9 | 7 |
| TTM R2 | 7 | 7 | 8 |
| TTM R1 | 9 | 8 | 9 |
Rebuttal [3/6]
[Q5] uncertainty estimates and sensitivity analyses for the rank aggregation.
[A5] We conducted a range of sensitivity analyses using alternative rank aggregation approaches.
In Experiment 1, varying the weights assigned to the three evaluation aspects (point-wise / IC / portfolio) yielded Spearman correlations of 0.961–0.987 with the original overall ranking.
In Experiment 2, we addressed the potential imbalance in contributions to the final ranking when some categories are evaluated under only a subset of the three aspects. We ranked models by the equally weighted arithmetic mean of the ten category-level aggregate K values in Table 3. The Spearman correlation with the original overall ranking was 0.8780.
In Experiment 3, we replaced the geometric mean of individual metric ranks within each aspect with an arithmetic mean. The Spearman correlation with the original overall ranking was 0.9496.
These results support the relative robustness of the overall model ranking structure across the aggregation alternatives examined.
Experiment 1: Varying Aspect Weights
The weights are listed in the order point-wise / IC / portfolio. Top-5 and top-10 overlap indicate the number of models shared with the corresponding sets in the original ranking. Ties at the selection boundary were broken by model ID so that exactly five or ten models were compared.
| Weights | Spearman | Top-5 Overlap | Top-10 Overlap | Top-Ranked Model |
|---|---|---|---|---|
| 1/3, 1/3, 1/3 | 1.000 | 5/5 | 10/10 | Reverso Small |
| 0.50, 0.25, 0.25 | 0.987 | 5/5 | 10/10 | Reverso Small |
| 0.25, 0.50, 0.25 | 0.980 | 5/5 | 9/10 | Reverso Small |
| 0.25, 0.25, 0.50 | 0.961 | 4/5 | 8/10 | Reverso Small |
Experiment 2: Equally Weighted Mean of Category Scores in Table 3
To give each category equal weight, we computed the arithmetic mean of the ten category-level aggregate K values in each row of Table 3 for all 43 models.
| Model | Original FinVerse Rank | Category-Balanced Rank |
|---|---|---|
| Reverso Small | 1 | 2 |
| Chronos-2 Synth | 2 | 1 |
| TimesFM 2.5 200M | 3 | 5 |
| Chronos-2 | 4 | 8 |
| TiRex 1.1 | 4 | 7 |
| Chronos-Bolt Base | 6 | 11 |
| Chronos-Bolt Small | 7 | 13 |
| T0 Alpha | 8 | 18 |
| TiRex 2 ZS | 8 | 16 |
| TiRex 2 Pretrain | 10 | 12 |
| Toto 2.0 2.5B | 11 | 3 |
| VisionTS | 11 | 20 |
| TimesFM 1.0 200M | 13 | 19 |
| TimesFM 2.0 500M | 14 | 26 |
| PatchFM | 15 | 10 |
| Kairos-50M | 16 | 17 |
| Toto 2.0 4M | 17 | 15 |
| Toto 2.0 1B | 18 | 4 |
| Granite FlowState R1 | 19 | 22 |
| TempoPFN | 20 | 6 |
| Kairos-10M | 21 | 21 |
| Toto 2.0 22M | 22 | 14 |
| FlowState | 23 | 24 |
| Toto 2.0 313M | 24 | 9 |
| Super Linear | 25 | 25 |
| Granite PatchTST-FM R1 | 26 | 38 |
| PatchTST-FM R1 | 27 | 35 |
| Timer-S1 | 27 | 28 |
| Toto Open Base 1.0 | 29 | 23 |
| Moirai 1.1-R Large | 30 | 29 |
| Moirai 1.1-R Base | 31 | 33 |
| Kairos-23M | 32 | 27 |
| Moirai 1.1-R Small | 33 | 32 |
| Sundial Base 128M | 33 | 31 |
| Moirai 2.0-R Small | 35 | 30 |
| TTM R3 | 36 | 37 |
| TTM R1 | 37 | 36 |
| CleanTS-65M | 38 | 39 |
| TTM R2 | 39 | 34 |
| YingLong 50M | 40 | 40 |
| YingLong 300M | 41 | 42 |
| YingLong 110M | 42 | 43 |
| YingLong 6M | 43 | 41 |
Experiment 3: Changing the Aggregation Strategy
We replaced the geometric mean of individual metric ranks within each aspect with an arithmetic mean.
| Model | Original FinVerse Rank | Arithmetic-Mean Rank |
|---|---|---|
| Reverso Small | 1 | 1 |
| Chronos-2 Synth | 2 | 2 |
| TimesFM 2.5 200M | 3 | 4 |
| Chronos-2 | 4 | 6 |
| TiRex 1.1 | 4 | 3 |
| Chronos-Bolt Base | 6 | 5 |
| Chronos-Bolt Small | 7 | 10 |
| T0 Alpha | 8 | 11 |
| TiRex 2 ZS | 8 | 7 |
| TiRex 2 Pretrain | 10 | 9 |
| Toto 2.0 2.5B | 11 | 13 |
| VisionTS | 11 | 20 |
| TimesFM 1.0 200M | 13 | 7 |
| TimesFM 2.0 500M | 14 | 15 |
| PatchFM | 15 | 16 |
| Kairos-50M | 16 | 18 |
| Toto 2.0 4M | 17 | 11 |
| Toto 2.0 1B | 18 | 17 |
| Granite FlowState R1 | 19 | 26 |
| TempoPFN | 20 | 22 |
| Kairos-10M | 21 | 21 |
| Toto 2.0 22M | 22 | 13 |
| FlowState | 23 | 30 |
| Toto 2.0 313M | 24 | 19 |
| Super Linear | 25 | 26 |
| Granite PatchTST-FM R1 | 26 | 35 |
| PatchTST-FM R1 | 27 | 32 |
| Timer-S1 | 27 | 24 |
| Toto Open Base 1.0 | 29 | 28 |
| Moirai 1.1-R Large | 30 | 28 |
| Moirai 1.1-R Base | 31 | 31 |
| Kairos-23M | 32 | 23 |
| Moirai 1.1-R Small | 33 | 25 |
| Sundial Base 128M | 33 | 32 |
| Moirai 2.0-R Small | 35 | 32 |
| TTM R3 | 36 | 36 |
| TTM R1 | 37 | 38 |
| CleanTS-65M | 38 | 37 |
| TTM R2 | 39 | 39 |
| YingLong 50M | 40 | 39 |
| YingLong 300M | 41 | 41 |
| YingLong 110M | 42 | 43 |
| YingLong 6M | 43 | 42 |
Rebuttal [1/6]
We thank the reviewer for the insightful feedback. Your questions have helped us clarify and strengthen our evaluation, and we provide our responses below.
[Q1] Can the authors compare FinVerse and GIFT-Eval in terms of data statistics, sampling frequencies, and other time-series attributes.
[A1]
We provide the comparison below.
[Q2] Analyze which differences are associated with model ranking reversals
[A2] We first examined this question in Appendix E.2 of the original manuscript. Holding the FinVerse data fixed, we analyzed the correlations between rankings under the conventional MASE metric used in GIFT-Eval and those under FinVerse's RelMAE, HR, IC, and portfolio metrics. RelMAE yields rankings similar to those under classical MASE, suggesting that the difference between these two metrics alone is unlikely to explain the ranking reversals between GIFT-Eval and FinVerse. Agreement progressively weakens for HR, IC, and portfolio metrics, indicating that these views capture aspects of forecasting performance not directly assessed by GIFT-Eval.
We conducted additional experiments to further address the reviewer's concern.
In the first additional experiment, rather than holding the data fixed and varying the metrics, we held the metric fixed at MASE and compared model rankings on GIFT-Eval and FinVerse data. The Spearman correlation was approximately 0.36 when using all GIFT-Eval domains and increased to approximately 0.42 when restricting GIFT-Eval to its Econ/Fin domain. This modest increase suggests that domain composition is associated with differences between the rankings.
In the second additional experiment, we further investigated the limited agreement that remained after restricting GIFT-Eval to Econ/Fin. We compared rankings across matched sampling-frequency groups. Agreement was relatively higher for the combined weekly and monthly group, at approximately 0.46, but lower for daily/business-daily data (approximately 0.32) and quarterly/annual data (approximately −0.03). In FinVerse, weekly and monthly data primarily comprise macroeconomic indicators; daily and business-daily data include crypto, stock, and ETF prices; and quarterly and annual data primarily comprise company fundamentals and macroeconomic indicators. Although M4 (which constitutes GIFT-Eval's Econ/Fin domain) provides broad domain labels, the specific underlying indicators are not identified in sufficient detail to attribute these differences precisely. Differences in data composition may therefore contribute to the lower agreement in the daily/business-daily and quarterly/annual groups.
In the final additional experiment, we examined the correlation with GIFT-Eval's Econ/Fin rankings separately for each FinVerse evaluation year, using MASE on both benchmarks. Agreement was notably lower in 2020 than in the other years. M4, which constitutes GIFT-Eval's Econ/Fin domain, was assembled for the 2018 competition and does not include the late-2019–2020 pandemic period. FinVerse includes this period of substantial market disruption. The lower agreement in 2020 is consistent with the possibility that pandemic-related changes in economic and market conditions contributed to the ranking differences. By covering multiple years, FinVerse enables evaluation across a range of economic and market conditions.
Taken together, these results suggest that model ranking differences between FinVerse and GIFT-Eval are associated not only with evaluation metrics, but also with domain composition, the specific data included within a domain, and the evaluation period.