acceptodds
Under review as a conference paper at ICLR 2027

FinVerse: Toward More Realistic Evaluation for Financial Time-Series Forecasting

Abstract

Time-series foundation models (TSFMs) are increasingly evaluated on broad benchmarks that apply uniform conventional forecast evaluation metrics across heterogeneous time-series. However, in the financial domain, decision-relevant evaluation requires going beyond conventional metrics to assess forecasts in terms of directional accuracy, cross-sectional ranking, and portfolio outcomes. To this end, we introduce FinVerse, a financial time-series benchmark, collecting a broad collection of 116,897 series with 171.1M observations. FinVerse organizes financial data according to their economic semantics and evaluates target series across point-wise forecasting, cross-sectional ranking, and portfolio backtesting. Across 43 public TSFMs, model rankings on FinVerse show only moderate agreement with those on the existing benchmark, GIFT-Eval (r=0.43), suggesting that generic forecasting benchmarks do not fully capture the aspects of forecast quality that matter for financial decision-making. Furthermore, substantial variation in model performance across financial time-series categories and evaluation metrics provides a more informative basis for model selection, enabling practitioners to identify suitable models according to their target series and downstream objectives.

Primary area:
datasets and benchmarks
Keywords:
Time-Series Forecasting, Foundation Models, Financial Time Series, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VP7qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FinVerse, a large-scale benchmark for financial time-series forecasting that organizes series by their economic meaning and evaluates models through point-wise forecasting, cross-sectional ranking, and portfolio backtesting. The full collection contains 116,897 series and 171.1M observations, with a selected evaluation set of 60,232 series. The study evaluates 43 public time-series foundation models in a zero-shot setting. Overall, I find the paper interesting and the benchmark potentially useful. Although there are already many benchmarks for time-series foundation models, FinVerse's relatively low ranking correlation with GIFT-Eval is an important result: a model's performance on an established general-purpose leaderboard may leave meaningful domain- and objective-specific blind spots.

Critical strengths
  1. The benchmark reveals potentially important blind spots in existing leaderboards with modest ranking correlations (Pearson r=0.43r=0.43, Spearman ρ=0.41\rho=0.41, and Kendall τ=0.27\tau=0.27). Reverso Small, for example, ranks first on FinVerse despite a weaker GIFT-Eval rank. These differences could help researchers identify where models have room to improve and motivate more targeted training and evaluation.

  2. Evaluation connects forecasting to downstream objectives. The particularly weak relationship between point-wise and portfolio rankings (r=0.14r=0.14) reinforces the importance of not treating forecasting accuracy as synonymous with investment utility. I also appreciate that the paper explicitly distinguishes proxy metrics from realized outcomes under a particular portfolio strategy.

  3. FinVerse extends beyond equity-only evaluation to include currencies, commodities, crypto assets, fixed income, macroeconomic indicators, ETFs, and fundamentals, across multiple frequencies. Evaluating 43 models provides a substantial empirical comparison. The category- and metric-specific results are useful because they show that no model is uniformly best and encourage model selection based on the intended task rather than a single generic leaderboard.

Critical weaknesses
  1. Major weakness: the paper does not explain why FinVerse differs from GIFT-Eval. The benchmark reveals a meaningful difference in model rankings, but offers little analysis of what makes FinVerse distinctive. In particular, it does not compare the benchmarks' data statistics, frequency distributions, or other time-series attributes to identify what may drive the ranking differences. Nor does it separate the effect of changing the data domain from the effect of changing the evaluation metrics: FinVerse and GIFT-Eval differ along both dimensions. Analyses with matched metrics and data, and performance stratified by measurable series characteristics, would help explain the observed blind spots and make the paper more useful for guiding model development. Without such analysis, the work establishes that rankings differ, but gives limited insight into why.

  2. Ranking robustness and data availability deserve clearer validation. The main-text results would be stronger with uncertainty estimates and sensitivity analyses for the rank aggregation and context-window choices. This matters when drawing conclusions from leaderboard differences and cross-benchmark correlations. The acknowledged possibility of pretraining overlap and imperfect point-in-time consistency also limits how confidently the results can be interpreted as strictly unseen, deployable forecasting performance. These limitations do not negate the benchmark's value, but should remain explicit in its claims.

  3. The benchmark omits intraday frequencies. FinVerse covers daily and lower-frequency data, but does not include minute- or second-level financial time series. These high-frequency settings are important in many financial applications and have distinct data characteristics and forecasting requirements. Their absence limits the benchmark's coverage of financial forecasting, so claims about its breadth should be scoped accordingly.

Questions for authors
  1. Can the authors compare FinVerse and GIFT-Eval in terms of data statistics, sampling frequencies, and other time-series attributes, and analyze which differences are associated with model ranking reversals?

  2. Can the authors report model-ranking correlations between FinVerse and a broader set of established time-series benchmarks such as fev-bench/TIME/BOOM, rather than GIFT-Eval alone? It would also be useful to compare correlations across FinVerse's evaluation aspects and, where possible, use matched metrics to help separate differences due to the data domain from those due to the evaluation objective.

  3. How stable are the main rankings and correlations across evaluation periods, aggregation choices, and context lengths?

13 comments
AuthorsOpenReview ↗

Rebuttal [1/6]

We thank the reviewer for the insightful feedback. Your questions have helped us clarify and strengthen our evaluation, and we provide our responses below.


[Q1] Can the authors compare FinVerse and GIFT-Eval in terms of data statistics, sampling frequencies, and other time-series attributes.

[A1]

We provide the comparison below.

Comparison ItemFinVerseGIFT-Eval
Data CoverageFinance-focused; 13 categories in the full collection23 datasets across 7 domains
Total Number of Time Series116,897Approximately 144,000
Total Number of ObservationsApproximately 171.1MApproximately 177M
Sampling FrequenciesDaily·Business-daily·Weekly·Monthly·Quarterly·Annual10-second·5-minute·10-minute·15-minute·Hourly·Daily·Weekly·Monthly·Quarterly·Annual
Forecast LengthApproximately one year, depending on frequency, with evaluation at multiple economically meaningful horizonsDataset- and frequency-specific. Examples: minute/hourly data: Short 48 / Medium 480 / Long 720 steps; 10-second data: 60 / 600 / 900 steps.
Evaluation MetricsHR, RelMAE, IC, Portfolio (Return, Sharpe, MDD, Volatility)Primarily MASE and CRPS; additional metrics include MSE, MAE, MAPE, sMAPE, RMSE, NRMSE, ND, and MSIS

[Q2] Analyze which differences are associated with model ranking reversals

[A2] We first examined this question in Appendix E.2 of the original manuscript. Holding the FinVerse data fixed, we analyzed the correlations between rankings under the conventional MASE metric used in GIFT-Eval and those under FinVerse's RelMAE, HR, IC, and portfolio metrics. RelMAE yields rankings similar to those under classical MASE, suggesting that the difference between these two metrics alone is unlikely to explain the ranking reversals between GIFT-Eval and FinVerse. Agreement progressively weakens for HR, IC, and portfolio metrics, indicating that these views capture aspects of forecasting performance not directly assessed by GIFT-Eval.

We conducted additional experiments to further address the reviewer's concern.

In the first additional experiment, rather than holding the data fixed and varying the metrics, we held the metric fixed at MASE and compared model rankings on GIFT-Eval and FinVerse data. The Spearman correlation was approximately 0.36 when using all GIFT-Eval domains and increased to approximately 0.42 when restricting GIFT-Eval to its Econ/Fin domain. This modest increase suggests that domain composition is associated with differences between the rankings.

In the second additional experiment, we further investigated the limited agreement that remained after restricting GIFT-Eval to Econ/Fin. We compared rankings across matched sampling-frequency groups. Agreement was relatively higher for the combined weekly and monthly group, at approximately 0.46, but lower for daily/business-daily data (approximately 0.32) and quarterly/annual data (approximately −0.03). In FinVerse, weekly and monthly data primarily comprise macroeconomic indicators; daily and business-daily data include crypto, stock, and ETF prices; and quarterly and annual data primarily comprise company fundamentals and macroeconomic indicators. Although M4 (which constitutes GIFT-Eval's Econ/Fin domain) provides broad domain labels, the specific underlying indicators are not identified in sufficient detail to attribute these differences precisely. Differences in data composition may therefore contribute to the lower agreement in the daily/business-daily and quarterly/annual groups.

FrequencySpearman
Daily + Business-daily0.324
Weekly + Monthly0.462
Quarterly + Annual−0.034

In the final additional experiment, we examined the correlation with GIFT-Eval's Econ/Fin rankings separately for each FinVerse evaluation year, using MASE on both benchmarks. Agreement was notably lower in 2020 than in the other years. M4, which constitutes GIFT-Eval's Econ/Fin domain, was assembled for the 2018 competition and does not include the late-2019–2020 pandemic period. FinVerse includes this period of substantial market disruption. The lower agreement in 2020 is consistent with the possibility that pandemic-related changes in economic and market conditions contributed to the ranking differences. By covering multiple years, FinVerse enables evaluation across a range of economic and market conditions.

FinVerse Evaluation YearSpearman
Full Period0.425
20200.204
20210.458
20220.443
20230.370
20240.510
20250.332

Taken together, these results suggest that model ranking differences between FinVerse and GIFT-Eval are associated not only with evaluation metrics, but also with domain composition, the specific data included within a domain, and the evaluation period.


AuthorsOpenReview ↗

Rebuttal [2/6]

[Q3] Imperfect point-in-time consistency

[A3] Because publicly available point-in-time (PIT) data could not be obtained for all datasets, the original forecasting experiments did not account for PIT constraints. To address the reviewer's concern, we conducted the following PIT-aware evaluation. For fundamentals collected from the SEC, release dates and revision histories are available and can be used to construct point-in-time (PIT) inputs. We therefore report PIT-aware results for selected leading models on Fundamentals. The Spearman correlation between the original and PIT-aware Fundamentals rankings was 0.8908. This high correlation of approximately 0.89 suggests that, although PIT issues remain in the benchmark, their impact on the ranking patterns examined in this experiment is limited.

Experiment: PIT-Aware Fundamentals Evaluation

Using the original SEC data, we constructed inputs from the historical vintages available at each forecast origin. For each company, the forecast origin was the release date of the latest quarterly financial results included in the input, and the model predicted the next four quarters. We first report results for nine models selected from the leading models in the original Fundamentals evaluation.

ModelOriginal FinVerse Fundamentals RankPIT-Aware FinVerse Fundamentals Rank
TiRex 1.132
TiRex 2 ZS24
VisionTS11
Super Linear65
Timer-S1118
TiRex 2 Pretrain42
Reverso Small85
Chronos-257
TimesFM 1.0 200M129

[Q4] possibility of pretraining overlap.

[A4] To assess the concern about pretraining contamination, we reevaluated nine models uploaded to Hugging Face before January 1, 2025, restricting the evaluation period to 2025–2026. Among the same nine models, the Spearman correlation between the full-period FinVerse ranking and the 2025–2026 ranking was 0.7915. Comparing the 2025–2026 ranking with the ranking for 2020–2024, a period that may overlap with the models' pretraining data, also yielded a correlation of 0.7143. In light of the year-to-year ranking variation reported in Table 18, these correlations indicate substantial agreement. Although this analysis does not establish that the original evaluation was entirely unaffected by pretraining overlap, it shows that the relative performance of the models is largely preserved when evaluation is restricted to observations from after the models' public release.

Experiment: Overall Rankings Among the Same Nine Models

ModelOriginal FinVerse (2020–2026)2020–20242025–2026
Chronos-Bolt Base111
TimesFM 1.0 200M224
Chronos-Bolt Small336
TimesFM 2.0 500M341
Moirai 1.1-R Large564
Moirai 1.1-R Small553
Moirai 1.1-R Base797
TTM R2778
TTM R1989

AuthorsOpenReview ↗

Rebuttal [3/6]

[Q5] uncertainty estimates and sensitivity analyses for the rank aggregation.

[A5] We conducted a range of sensitivity analyses using alternative rank aggregation approaches.

In Experiment 1, varying the weights assigned to the three evaluation aspects (point-wise / IC / portfolio) yielded Spearman correlations of 0.961–0.987 with the original overall ranking.

In Experiment 2, we addressed the potential imbalance in contributions to the final ranking when some categories are evaluated under only a subset of the three aspects. We ranked models by the equally weighted arithmetic mean of the ten category-level aggregate K values in Table 3. The Spearman correlation with the original overall ranking was 0.8780.

In Experiment 3, we replaced the geometric mean of individual metric ranks within each aspect with an arithmetic mean. The Spearman correlation with the original overall ranking was 0.9496.

These results support the relative robustness of the overall model ranking structure across the aggregation alternatives examined.

Experiment 1: Varying Aspect Weights

The weights are listed in the order point-wise / IC / portfolio. Top-5 and top-10 overlap indicate the number of models shared with the corresponding sets in the original ranking. Ties at the selection boundary were broken by model ID so that exactly five or ten models were compared.

WeightsSpearmanTop-5 OverlapTop-10 OverlapTop-Ranked Model
1/3, 1/3, 1/31.0005/510/10Reverso Small
0.50, 0.25, 0.250.9875/510/10Reverso Small
0.25, 0.50, 0.250.9805/59/10Reverso Small
0.25, 0.25, 0.500.9614/58/10Reverso Small

Experiment 2: Equally Weighted Mean of Category Scores in Table 3

To give each category equal weight, we computed the arithmetic mean of the ten category-level aggregate K values in each row of Table 3 for all 43 models.

ModelOriginal FinVerse RankCategory-Balanced Rank
Reverso Small12
Chronos-2 Synth21
TimesFM 2.5 200M35
Chronos-248
TiRex 1.147
Chronos-Bolt Base611
Chronos-Bolt Small713
T0 Alpha818
TiRex 2 ZS816
TiRex 2 Pretrain1012
Toto 2.0 2.5B113
VisionTS1120
TimesFM 1.0 200M1319
TimesFM 2.0 500M1426
PatchFM1510
Kairos-50M1617
Toto 2.0 4M1715
Toto 2.0 1B184
Granite FlowState R11922
TempoPFN206
Kairos-10M2121
Toto 2.0 22M2214
FlowState2324
Toto 2.0 313M249
Super Linear2525
Granite PatchTST-FM R12638
PatchTST-FM R12735
Timer-S12728
Toto Open Base 1.02923
Moirai 1.1-R Large3029
Moirai 1.1-R Base3133
Kairos-23M3227
Moirai 1.1-R Small3332
Sundial Base 128M3331
Moirai 2.0-R Small3530
TTM R33637
TTM R13736
CleanTS-65M3839
TTM R23934
YingLong 50M4040
YingLong 300M4142
YingLong 110M4243
YingLong 6M4341

Experiment 3: Changing the Aggregation Strategy

We replaced the geometric mean of individual metric ranks within each aspect with an arithmetic mean.

ModelOriginal FinVerse RankArithmetic-Mean Rank
Reverso Small11
Chronos-2 Synth22
TimesFM 2.5 200M34
Chronos-246
TiRex 1.143
Chronos-Bolt Base65
Chronos-Bolt Small710
T0 Alpha811
TiRex 2 ZS87
TiRex 2 Pretrain109
Toto 2.0 2.5B1113
VisionTS1120
TimesFM 1.0 200M137
TimesFM 2.0 500M1415
PatchFM1516
Kairos-50M1618
Toto 2.0 4M1711
Toto 2.0 1B1817
Granite FlowState R11926
TempoPFN2022
Kairos-10M2121
Toto 2.0 22M2213
FlowState2330
Toto 2.0 313M2419
Super Linear2526
Granite PatchTST-FM R12635
PatchTST-FM R12732
Timer-S12724
Toto Open Base 1.02928
Moirai 1.1-R Large3028
Moirai 1.1-R Base3131
Kairos-23M3223
Moirai 1.1-R Small3325
Sundial Base 128M3332
Moirai 2.0-R Small3532
TTM R33636
TTM R13738
CleanTS-65M3837
TTM R23939
YingLong 50M4039
YingLong 300M4141
YingLong 110M4243
YingLong 6M4342

Reviewer_4vceOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing time-series foundation models (TSFMs) are commonly evaluated on general benchmarks such as GIFT-Eval, mainly using forecasting accuracy metrics like MASE and CRPS. The authors argue that these metrics may not fully capture practical value in financial applications. For example, stock forecasting requires predicting price directions and relative asset performance; macroeconomic forecasting may focus on whether inflation is accelerating or slowing down; and portfolio decisions depend on investment returns and risks.

To address this, the authors introduce FinVerse, a financial time-series benchmark that organizes financial data based on economic semantics and evaluates models from three perspectives: point-wise forecasting, cross-sectional ranking, and portfolio performance.

Critical strengths
  1. Traditional time-series benchmarks mainly focus on prediction accuracy, while forecasting is often only one step toward real-world decisions. This distinction is particularly important in finance, where lower forecasting errors do not necessarily lead to better investment outcomes.

  2. FinVerse includes diverse financial time series, such as foreign exchange, commodities, cryptocurrencies, fixed income, macroeconomic indicators, equities, ETFs, and company fundamentals.

  3. The authors evaluate 43 publicly available TSFMs, including TimesFM, Chronos, Moirai, TiRex, and Toto, providing a useful comparison across different financial tasks and evaluation metrics.

Critical weaknesses
  1. Lack of strict point-in-time consistency. The paper aims to provide more realistic financial forecasting evaluation, but it is unclear whether all information used at each forecast origin was actually available at that time. For example, CPI and GDP figures are typically released after their reference periods, while company fundamentals such as EPS and revenue become available only after earnings announcements. Moreover, historical macroeconomic data may be revised afterward. These issues could introduce future information leakage and affect the reliability of the evaluation.

  2. Potential pretraining data contamination. The evaluation period starts in 2020, while many of the 43 evaluated foundation models were released in 2025–2026. Therefore, some evaluation data may have already appeared in their pretraining datasets. This raises concerns about whether the reported zero-shot results truly reflect out-of-sample generalization.

  3. Unclear justification for rank aggregation. The authors aggregate rankings across three evaluation perspectives with equal weights, but the motivation for this choice is not sufficiently explained. Furthermore, many macroeconomic tasks are evaluated only through point-wise metrics, while equity-related tasks also contribute to cross-sectional and portfolio evaluations. This may introduce an imbalance in the overall ranking. I would like to see a sensitivity analysis using alternative aggregation weights and strategies.

Questions for authors

please check weakness

4 comments
AuthorsOpenReview ↗

Rebuttal [1/2]

We thank the reviewer for the insightful feedback. Your questions have helped us clarify and strengthen our evaluation, and we provide our responses below.


[Q1] Lack of strict point-in-time consistency

[A1] Because publicly available point-in-time (PIT) data could not be obtained for all datasets, the original forecasting experiments did not account for PIT constraints. To address the reviewer's concern, we conducted the following PIT-aware evaluation. For fundamentals collected from the SEC, release dates and revision histories are available and can be used to construct point-in-time (PIT) inputs. We therefore report PIT-aware results for selected leading models on Fundamentals. The Spearman correlation between the original and PIT-aware Fundamentals rankings was 0.8908. This high correlation of approximately 0.89 suggests that, although PIT issues remain in the benchmark, their impact on the ranking patterns examined in this experiment is limited.

Experiment: PIT-Aware Fundamentals Evaluation

Using the original SEC data, we constructed inputs from the historical vintages available at each forecast origin. For each company, the forecast origin was the release date of the latest quarterly financial results included in the input, and the model predicted the next four quarters. We first report results for nine models selected from the leading models in the original Fundamentals evaluation.

ModelOriginal FinVerse Fundamentals RankPIT-Aware FinVerse Fundamentals Rank
TiRex 1.132
TiRex 2 ZS24
VisionTS11
Super Linear65
Timer-S1118
TiRex 2 Pretrain42
Reverso Small85
Chronos-257
TimesFM 1.0 200M129

[Q2] Potential pretraining data contamination.

[A2] To assess the concern about pretraining contamination, we reevaluated nine models uploaded to Hugging Face before January 1, 2025, restricting the evaluation period to 2025–2026. Among the same nine models, the Spearman correlation between the full-period FinVerse ranking and the 2025–2026 ranking was 0.7915. Comparing the 2025–2026 ranking with the ranking for 2020–2024, a period that may overlap with the models' pretraining data, also yielded a correlation of 0.7143. In light of the year-to-year ranking variation reported in Table 18, these correlations indicate substantial agreement. Although this analysis does not establish that the original evaluation was entirely unaffected by pretraining overlap, it shows that the relative performance of the models is largely preserved when evaluation is restricted to observations from after the models' public release.

Experiment: Overall Rankings Among the Same Nine Models

ModelOriginal FinVerse (2020–2026)2020–20242025–2026
Chronos-Bolt Base111
TimesFM 1.0 200M224
Chronos-Bolt Small336
TimesFM 2.0 500M341
Moirai 1.1-R Large564
Moirai 1.1-R Small553
Moirai 1.1-R Base797
TTM R2778
TTM R1989

AuthorsOpenReview ↗

Rebuttal [2/2]

[Q3] Unclear justification for rank aggregation

[A3] We conducted a range of sensitivity analyses using alternative rank aggregation approaches.

In Experiment 1, varying the weights assigned to the three evaluation aspects (point-wise / IC / portfolio) yielded Spearman correlations of 0.961–0.987 with the original overall ranking.

In Experiment 2, we addressed the potential imbalance in contributions to the final ranking when some categories are evaluated under only a subset of the three aspects. We ranked models by the equally weighted arithmetic mean of the ten category-level aggregate K values in Table 3. The Spearman correlation with the original overall ranking was 0.8780.

In Experiment 3, we replaced the geometric mean of individual metric ranks within each aspect with an arithmetic mean. The Spearman correlation with the original overall ranking was 0.9496.

These results support the relative robustness of the overall model ranking structure across the aggregation alternatives examined.

Experiment 1: Varying Aspect Weights

The weights are listed in the order point-wise / IC / portfolio. Top-5 and top-10 overlap indicate the number of models shared with the corresponding sets in the original ranking. Ties at the selection boundary were broken by model ID so that exactly five or ten models were compared.

WeightsSpearmanTop-5 OverlapTop-10 OverlapTop-Ranked Model
1/3, 1/3, 1/31.0005/510/10Reverso Small
0.50, 0.25, 0.250.9875/510/10Reverso Small
0.25, 0.50, 0.250.9805/59/10Reverso Small
0.25, 0.25, 0.500.9614/58/10Reverso Small

Experiment 2: Equally Weighted Mean of Category Scores in Table 3

To give each category equal weight, we computed the arithmetic mean of the ten category-level aggregate K values in each row of Table 3 for all 43 models.

ModelOriginal FinVerse RankCategory-Balanced Rank
Reverso Small12
Chronos-2 Synth21
TimesFM 2.5 200M35
Chronos-248
TiRex 1.147
Chronos-Bolt Base611
Chronos-Bolt Small713
T0 Alpha818
TiRex 2 ZS816
TiRex 2 Pretrain1012
Toto 2.0 2.5B113
VisionTS1120
TimesFM 1.0 200M1319
TimesFM 2.0 500M1426
PatchFM1510
Kairos-50M1617
Toto 2.0 4M1715
Toto 2.0 1B184
Granite FlowState R11922
TempoPFN206
Kairos-10M2121
Toto 2.0 22M2214
FlowState2324
Toto 2.0 313M249
Super Linear2525
Granite PatchTST-FM R12638
PatchTST-FM R12735
Timer-S12728
Toto Open Base 1.02923
Moirai 1.1-R Large3029
Moirai 1.1-R Base3133
Kairos-23M3227
Moirai 1.1-R Small3332
Sundial Base 128M3331
Moirai 2.0-R Small3530
TTM R33637
TTM R13736
CleanTS-65M3839
TTM R23934
YingLong 50M4040
YingLong 300M4142
YingLong 110M4243
YingLong 6M4341

Experiment 3: Changing the Aggregation Strategy

We replaced the geometric mean of individual metric ranks within each aspect with an arithmetic mean.

ModelOriginal FinVerse RankArithmetic-Mean Rank
Reverso Small11
Chronos-2 Synth22
TimesFM 2.5 200M34
Chronos-246
TiRex 1.143
Chronos-Bolt Base65
Chronos-Bolt Small710
T0 Alpha811
TiRex 2 ZS87
TiRex 2 Pretrain109
Toto 2.0 2.5B1113
VisionTS1120
TimesFM 1.0 200M137
TimesFM 2.0 500M1415
PatchFM1516
Kairos-50M1618
Toto 2.0 4M1711
Toto 2.0 1B1817
Granite FlowState R11926
TempoPFN2022
Kairos-10M2121
Toto 2.0 22M2213
FlowState2330
Toto 2.0 313M2419
Super Linear2526
Granite PatchTST-FM R12635
PatchTST-FM R12732
Timer-S12724
Toto Open Base 1.02928
Moirai 1.1-R Large3028
Moirai 1.1-R Base3131
Kairos-23M3223
Moirai 1.1-R Small3325
Sundial Base 128M3332
Moirai 2.0-R Small3532
TTM R33636
TTM R13738
CleanTS-65M3837
TTM R23939
YingLong 50M4039
YingLong 300M4141
YingLong 110M4243
YingLong 6M4342

We are providing these responses first and plan to update the manuscript by November 15, indicating the corresponding revisions in our responses. For experiments in which we have reported results for only a subset of models due to time constraints, we plan to complete the evaluations for all models.

AuthorsOpenReview ↗

Revised Paper

The corresponding sections of the revised manuscript for each response are listed below. Revisions are highlighted in red.

[A1] The PIT-aware Fundamentals experiment is still being completed. We will add the results to the revised manuscript once the remaining predictions and the full evaluation are complete.

[A2] Appendix E.6.

[A3] Appendix E.10

As the discussion deadline approaches, we would like to express our gratitude. Thank you for your insightful comments and valuable time. Your feedback has helped us clarify the presentation and strengthen the analysis in our paper. We will ensure that the revisions presented during the discussion are faithfully incorporated into the final manuscript.

Reviewer_z8YCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FinVerse, a benchmark for evaluating time-series foundation models on financial forecasting tasks using criteria tailored to the economic roles of different series. The full collection contains 116,897 series and 171.1 million observations, while the evaluation subset comprises 60,232 target series and 13.6 million observations across 10 categories and 67 detail categories. Evaluation combines point-wise forecasting metrics, cross-sectional ranking, and portfolio outcomes under a specified equal-weight, long-only strategy. The authors evaluate 43 pretrained model checkpoints without FinVerse-specific fine-tuning. They report moderate agreement with GIFT-Eval rankings and substantial variation across financial categories and evaluation objectives. The main contribution is a broad evaluation resource and an empirical characterization of how model assessments change with financial objectives.

Critical strengths

Broad coverage organized around economically distinct forecasting targets

FinVerse brings together financial series spanning multiple domains with explicit category-specific assignments of metrics and forecasting horizons. The evaluation subset’s coverage of 10 categories and 67 detail categories. This breadth is consequential because the paper evaluates whether model suitability varies with the economic role of the target, rather than reporting only a single pooled forecasting score. Together with the evaluation of 43 checkpoints, the collection has substantial potential as a reusable resource for studying financial transfer and model selection.

Critical weaknesses

1. Temporal validity remains insufficiently verified. The paper explicitly acknowledges that observation timestamps may differ from actual release times (§5), and that some evaluation-period observations may have appeared in model pretraining (§4.1). However, it does not provide a documented evaluation subset that controls for these issues or quantify their effects on the reported rankings. Because the paper’s central motivation is more realistic financial evaluation, these are substantive benchmark-validity concerns rather than peripheral implementation details. The authors’ transparency is valuable, but acknowledgment alone does not establish the robustness of the conclusions.

2. Incremental economic value is not established. The portfolio evaluation uses a fully invested, long-only selection strategy. Appendix C.6 transparently documents omitted transaction costs and several asset-specific simplifications, including omitted FX carry and futures roll returns. Nevertheless, the paper does not compare the forecast-driven portfolios with matched equal-weight-universe, random-selection, or simple momentum policies.

Questions for authors

1. Clarification — historical information availability and asset eligibility. Are macroeconomic and fundamental inputs aligned to their actual release dates and historical data vintages, or to observation periods using subsequently revised values? Please also clarify when and how the cryptocurrency “Top 100” universe was selected, and how an existing holding is valued or liquidated if its price series ends before the scheduled exit. These details would materially affect my assessment of historical feasibility.

2. Validation of the current claims — pretraining overlap. Can the main conclusions be reproduced on a documented subset of model–data pairs for which pretraining overlap is excluded, for example through verified training cutoffs or models with documented disjoint training data? A credible subset analysis would help establish whether the reported comparisons reflect generalization to unseen financial observations.

3. Validation of the current claims — economic value and ranking uncertainty. On the equity and ETF subsets, how do the forecast-driven portfolios compare with equal-weight-universe and simple momentum policies using the same dates and eligible instruments? Do the leading-model comparisons remain meaningful with uncertainty estimates that account for temporal dependence and with modest transaction costs based on holdings turnover? Evidence that the principal conclusions survive these controls would substantially improve my assessment.

11 comments
AuthorsOpenReview ↗

Rebuttal [1/5]

We thank the reviewer for the insightful feedback. Your questions have helped us clarify and strengthen our evaluation, and we provide our responses below.


[Q1] Are macroeconomic and fundamental inputs aligned to their actual release dates and historical data vintages, or to observation periods using subsequently revised values?

[A1] We support alignment with actual release dates and historical data vintages where the data source makes this information available. However, because publicly available point-in-time (PIT) data could not be obtained for all datasets, the original forecasting experiments did not account for PIT constraints. To address the reviewer’s concern, we report results on the following data groups, which are unaffected by the PIT issues considered here. We also report a PIT-aware reforecasting experiment for selected leading models on Fundamentals, for which PIT information is available.

In Experiment 1, the Spearman correlation between the original FinVerse ranking and the ranking obtained using only the PIT-free data groups is 0.8252. In Experiment 2, the Spearman correlation between the original FinVerse Fundamentals ranking and the ranking obtained from PIT-aware Fundamentals forecasts is 0.8908. These correlations, both exceeding 0.8, suggest that although PIT issues remain in the benchmark, their impact on the overall ranking patterns examined in these experiments is limited.

Experiment 1: PIT-Free Data Groups

Evaluation targets

  • FX Pair
  • Future Front
  • Spot
  • Commodity ETF Single
  • Commodity ETF Broad
  • Commodity ETF Thematic
  • Crypto Pair Close (Top 100)
  • Country Index
  • Regional Index
  • Implied Volatility
  • US Equity Index
  • ETF OHLCV Close
  • Equity OHLCV Close
ModelOriginal FinVerse RankPIT-Free FinVerse Rank
Reverso Small12
Chronos-2 Synth21
TimesFM 2.5 200M37
Chronos-246
TiRex 1.1412
Chronos-Bolt Base611
Chronos-Bolt Small77
T0 Alpha819
TiRex 2 ZS828
TiRex 2 Pretrain1024
Toto 2.0 2.5B113
VisionTS1126
TimesFM 1.0 200M1321
TimesFM 2.0 500M1414
PatchFM1516
Kairos-50M1610
Toto 2.0 4M175
Toto 2.0 1B184
Granite FlowState R11919
TempoPFN2016
Kairos-10M2115
Toto 2.0 22M229
FlowState2318
Toto 2.0 313M2412
Super Linear2533
Granite PatchTST-FM R12631
PatchTST-FM R12729
Timer-S12735
Toto Open Base 1.02923
Moirai 1.1-R Large3022
Moirai 1.1-R Base3127
Kairos-23M3225
Moirai 1.1-R Small3329
Sundial Base 128M3339
Moirai 2.0-R Small3532
TTM R33637
TTM R13734
CleanTS-65M3835
TTM R23937
YingLong 50M4039
YingLong 300M4141
YingLong 110M4242
YingLong 6M4342

Experiment 2: PIT-Aware Fundamentals Evaluation

Using the original SEC data, we constructed inputs from the historical vintages available at each forecast origin. For each company, the forecast origin was the release date of the latest quarterly financial results included in the input, and the model predicted the next four quarters. We first report results for nine models selected from the leading models in the original Fundamentals evaluation.

ModelOriginal FinVerse Fundamentals RankPIT-Aware FinVerse Fundamentals Rank
TiRex 1.132
TiRex 2 ZS24
VisionTS11
Super Linear65
Timer-S1118
TiRex 2 Pretrain42
Reverso Small85
Chronos-257
TimesFM 1.0 200M129

AuthorsOpenReview ↗

Rebuttal [2/5]

[Q2] Can the main conclusions be reproduced on a documented subset of model–data pairs for which pretraining overlap is excluded, for example through verified training cutoffs or models with documented disjoint training data?

[A2] We reevaluated nine models uploaded to Hugging Face before January 1, 2025, restricting the evaluation period to 2025–2026. Among the same nine models, the Spearman correlation between the full-period FinVerse ranking and the 2025–2026 ranking was 0.7915. Comparing the 2025–2026 ranking with the ranking for 2020–2024, a period that may overlap with the models' pretraining data, also yielded a correlation of 0.7143. In light of the year-to-year ranking variation reported in Table 18, these correlations indicate substantial agreement. Although this analysis does not establish that the original evaluation was entirely unaffected by pretraining overlap, it shows that the relative performance of the models is largely preserved when evaluation is restricted to observations from after the models' public release.

Experiment: Overall Rankings Among the Same Nine Models

ModelOriginal FinVerse (2020–2026)2020–20242025–2026
Chronos-Bolt Base111
TimesFM 1.0 200M224
Chronos-Bolt Small336
TimesFM 2.0 500M341
Moirai 1.1-R Large564
Moirai 1.1-R Small553
Moirai 1.1-R Base797
TTM R2778
TTM R1989

[Q3] clarify when and how the cryptocurrency “Top 100” universe was selected?

[A3] We selected the top 100 cryptocurrencies by market capitalization using the April 26, 2026 snapshot of CoinMarketCap.


AuthorsOpenReview ↗

Rebuttal [3/5]

[Q4] How is an existing holding valued or liquidated if its price series ends before the scheduled exit?

[A4] Appendix C.6 of the manuscript describes the portfolio evaluation settings in detail. Although the original settings were chosen for simplicity, we conducted an additional experiment with more realistic trading rules. When an asset becomes unavailable for trading, operationalized as the end of its observed price series, the position is sold at its last observed closing price and replaced with the next candidate in the original forecast ranking. We also apply proportional transaction costs based on the amount traded. On the equity and ETF subsets, the Spearman correlations between the original portfolio ranking and the rankings under these additional rules are 0.9964, 0.9899, and 0.9497 at transaction costs of 0, 10, and 20 basis points, respectively. Thus, while these rules lead to some changes, the relative model rankings remain largely consistent with those obtained under the original simplified settings.

Experiment: Adding More Realistic Trading Rules

If a held asset's price series ends, we sell it at its last observed closing price and buy the next candidate in the fixed forecast ranking established when the portfolio was initially constructed, excluding assets already held. A replacement asset must have a positive observed price at the replacement time, and assets whose price series end at that same time are excluded. Only the proceeds from the affected position, net of transaction costs, are reinvested. Transaction costs of 0, 10, or 20 basis points are applied separately to purchases and sales, including initial purchases, scheduled rebalancing trades, early exits, replacement purchases, and final liquidation. Trades are settled under a self-financing constraint so that investments do not exceed the assets available after costs. Using the same eligible universe, we also evaluated a comparison condition that carries forward the last observed price without replacement during the holding period.

ModelOriginal FinVerse (Equities and ETFs)0bp10bp20bp
Chronos-2 Synth2112
TimesFM 2.0 500M1234
Moirai 1.1-R Large3321
T0 Alpha4443
Reverso Small5555
Moirai 1.1-R Base6666
Chronos-Bolt Base10779
Toto Open Base 1.0981213
Kairos-50M119810
Chronos-2810911
Toto 2.0 2.5B7111014
TimesFM 1.0 200M13121112
Toto 2.0 1B12131617
Chronos-Bolt Small14141315
TiRex 1.115151416
PatchTST-FM R11616158
Granite PatchTST-FM R11717177
Toto 2.0 313M19182120
TimesFM 2.5 200M18191818
Moirai 2.0-R Small21201922
Toto 2.0 4M20212325
Kairos-23M23222021
Kairos-10M22232219
Toto 2.0 22M24242629
PatchFM25252730
VisionTS26262427
Moirai 1.1-R Small27272524
TTM R128282932
FlowState30293033
Super Linear29302831
Granite FlowState R131313234
TempoPFN32323535
CleanTS-65M33333326
TTM R334343123
TTM R235353428
YingLong 300M36363636
TiRex 2 Pretrain38373941
TiRex 2 ZS37383738
YingLong 110M39393837
YingLong 6M40404140
YingLong 50M41414039
Sundial Base 128M42424242
Timer-S143434343

Comments on the paper

1 comment

Dear reviewers,

The authors have now submitted their responses. Please review them and share your thoughts on whether your concerns have been addressed. I encourage you to engage with one another’s assessments, discuss any remaining disagreements, and update your reviews and scores as appropriate.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.