- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PPD-SOTA separates two questions in financial forecasting evaluation: which model to deploy next, and where the evidence supports a superiority claim. It combines Prediction, Portfolio, and Deployability criteria for forward model selection, and uses practical margins with paired simultaneous inference to qualify comparisons across models and environments. Experiments adapt seven forecasting methods to a common protocol, evaluate quarterly selection and deployment conditions, and report lower forward regret for the proposed rule. The paper also presents a component study, an inference-calibration exercise, and an audit of evaluation practices in 65 papers.
Critical strengths
Separating a deployment decision from a superiority claim is useful. A researcher may need to choose a model even when the evidence does not establish that it is uniformly better. The paper makes this distinction explicit and connects predictive metrics to execution and deployment assumptions, which are often left implicit in forecasting comparisons.
The forward evaluation is more informative than another same-window leaderboard. Figure 4 reports a reduction in mean next-quarter regret from 1.898 to 1.283, and the additional selector comparisons and component grid go beyond a single favorable baseline. I also appreciate the explicit distinction between failing to establish superiority and establishing equivalence.
Critical weaknesses
The statistical procedure is not specified well enough to assess its central guarantee. Proposition 1 is valid conditional on simultaneous interval coverage, but that condition is where much of the difficulty lies. Appendix D lists the block length, bootstrap repetitions, confidence level, and practical margin without specifying the actual simultaneous-interval construction or the full comparison family. Figure 8 is encouraging, but the simulation setup and criterion behind "18/18 cells pass" are insufficiently detailed. Since reliable inference is a central contribution, these are necessary methodological details rather than optional implementation information.
The evidence that the proposed selection rule is reliably better is limited. Figure 4 reports 6 better, 13 tied, and 1 worse outcome across 20 dependent pool-events. The four component-removal intervals in Figure 5 all cross zero. This does not show that the components are ineffective, but it leaves their benefits uncertain. Appendix Table 4 also shows that Full PPD is best on primary regret, but not on deployment-average or worst-environment regret. I would therefore distinguish a promising result under the primary protocol from a broadly established advantage of this particular aggregation rule.
Questions for authors
-
Please specify the simultaneous confidence-interval algorithm, the family of comparisons it covers, and how joint resampling preserves dependence across models, pools, and environments. For Figure 8, what are the data-generating settings, Monte Carlo sample sizes, and pass criteria?
-
Can you provide uncertainty for the main paired regret improvement using the dependent selection events as the basis for inference? How sensitive is the improvement to individual quarters? The 24 deployment environments should not be treated as independent repetitions of the selection experiment.
-
Among comparisons that fail to establish superiority, how many support equivalence and how many remain inconclusive? Reporting interval widths relative to the practical margin would help determine whether the absence of a full-scope winner reflects genuine trade-offs or limited statistical resolution.