acceptodds
Under review as a conference paper at ICLR 2027

PPD-SOTA: Recasting State-of-the-Art Evaluation for AI-Driven Financial Forecasting

Abstract

AI-driven financial forecasting has advanced rapidly, but state-of-the-art evaluation still relies heavily on metric wins over selected baselines. In financial forecasting, however, predictive performance must pass through portfolio construction and deployment conditions before it translates into practical value, making a single leaderboard rank difficult to interpret. We introduce PPD-SOTA, an evaluation framework that separates forward model selection from the scope of a SOTA claim. For deployment, PPD-SOTA evaluates forecasting models through Prediction, Portfolio, and Deployability (PPD). For comparison, it tests a comparator set using practical margins and paired simultaneous inference. On a unified benchmark of recent stock-forecasting methods, PPD achieves lower mean regret than conventional metrics-win selection and remains effective across changing conditions, while superiority support narrows as the comparator set expands and no model clears the full set. These results motivate evaluating financial forecasting models by both forward deployment performance and the scope of superiority supported by the evidence, rather than by isolated metric wins alone.

Primary area:
datasets and benchmarks
Keywords:
financial forecasting, machine learning, benchmarking, state-of-the-art evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SBfwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PPD-SOTA separates two questions in financial forecasting evaluation: which model to deploy next, and where the evidence supports a superiority claim. It combines Prediction, Portfolio, and Deployability criteria for forward model selection, and uses practical margins with paired simultaneous inference to qualify comparisons across models and environments. Experiments adapt seven forecasting methods to a common protocol, evaluate quarterly selection and deployment conditions, and report lower forward regret for the proposed rule. The paper also presents a component study, an inference-calibration exercise, and an audit of evaluation practices in 65 papers.

Critical strengths

Separating a deployment decision from a superiority claim is useful. A researcher may need to choose a model even when the evidence does not establish that it is uniformly better. The paper makes this distinction explicit and connects predictive metrics to execution and deployment assumptions, which are often left implicit in forecasting comparisons.

The forward evaluation is more informative than another same-window leaderboard. Figure 4 reports a reduction in mean next-quarter regret from 1.898 to 1.283, and the additional selector comparisons and component grid go beyond a single favorable baseline. I also appreciate the explicit distinction between failing to establish superiority and establishing equivalence.

Critical weaknesses

The statistical procedure is not specified well enough to assess its central guarantee. Proposition 1 is valid conditional on simultaneous interval coverage, but that condition is where much of the difficulty lies. Appendix D lists the block length, bootstrap repetitions, confidence level, and practical margin without specifying the actual simultaneous-interval construction or the full comparison family. Figure 8 is encouraging, but the simulation setup and criterion behind "18/18 cells pass" are insufficiently detailed. Since reliable inference is a central contribution, these are necessary methodological details rather than optional implementation information.

The evidence that the proposed selection rule is reliably better is limited. Figure 4 reports 6 better, 13 tied, and 1 worse outcome across 20 dependent pool-events. The four component-removal intervals in Figure 5 all cross zero. This does not show that the components are ineffective, but it leaves their benefits uncertain. Appendix Table 4 also shows that Full PPD is best on primary regret, but not on deployment-average or worst-environment regret. I would therefore distinguish a promising result under the primary protocol from a broadly established advantage of this particular aggregation rule.

Questions for authors
  1. Please specify the simultaneous confidence-interval algorithm, the family of comparisons it covers, and how joint resampling preserves dependence across models, pools, and environments. For Figure 8, what are the data-generating settings, Monte Carlo sample sizes, and pass criteria?

  2. Can you provide uncertainty for the main paired regret improvement using the dependent selection events as the basis for inference? How sensitive is the improvement to individual quarters? The 24 deployment environments should not be treated as independent repetitions of the selection experiment.

  3. Among comparisons that fail to establish superiority, how many support equivalence and how many remain inconclusive? Reporting interval widths relative to the practical margin would help determine whether the absence of a full-scope winner reflects genuine trade-offs or limited statistical resolution.

Reviewer_jx4kOpenReview ↗
Rating
2(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces a framework called PPD-SOTA to evaluate models on financial forecasting. It constructs a comprehensive score based on three dimensions including forecasting quality, portfolio performance, and deployability. It defines the scope of model superiority claims through unified trading backtesting rules, as well as statistical inference that accounts for practical significance thresholds and multiple comparisons. Experiments conducted on two U.S. stock pools and seven forecasting models show that PPD reduces average regret in the following quarter.

Critical strengths
  • It employs a historical window model selection approach followed by quarterly evaluation, while clearly defining the trading sequence, fees, and handling of missing quotes. Maintaining the model selection while altering the deployment conditions also helps assess the adaptability of the selected models to changes in trading settings.

  • It compares seven forecasting models under a unified framework and includes single-metric selection, average ranking, turnover-related controls, component ablation, and weight sensitivity analysis, providing useful evidence for understanding the impact of different evaluation choices.

Critical weaknesses
  • PPD consists primarily of components such as multi-index weighting, forward out-of-sample evaluation, unified backtesting, and statistical comparisons. Although integrating these components into an evaluation protocol has some value, the paper has not sufficiently explained what new capabilities or insights this integration offers compared to existing evaluation and statistical tools.

  • In Table 4, some simplified configurations perform very similarly to the full PPD and are even slightly better in terms of average regret under deployment conditions. Furthermore, the simultaneous confidence intervals for all ablation effects in Figure 5 span zero. Therefore, the current results are insufficient to conclusively demonstrate a consistent advantage of the full set of metrics and the hierarchical design over simpler alternatives.

Questions for authors

See weaknesses above.

Reviewer_gqCdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a criticial yet overlooked problem in financial AI settings in terms of selection criteria. The paper proposes a holistic selection process compared with previous solutions. The paper is evaluated on US equity with various combinations against different baselines.

Critical strengths
  1. The problem is a realistic problem with meaningful practitional outcome. The paper summarizes all the existing FinAI papers and spots the problem.
  2. The paper is very easy to follow, with each RQ associated with the supporting experiment and the concrete takeaway.
Critical weaknesses
  1. The consideration for deployment could be relatively oversimplified.
  2. The paper is missing case study. In practice, the practioners in finance will never conduct selection based on single metric. They tend to weight various criteria with different weights and select certain plateo optimal. For instance, strategy A is Sharpe 1 with maxDD 0.05, strategy B is Sharpe 2 with maxDD 0.3, strategy C is Sharpe 1.9 with maxDD 0.08. Any practitioner will go with C but C could never be selected by any baseline in this paper. A case study would help them appreciate the necessaity of this paper.
Questions for authors

Please see Cons

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.