- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This study examines whether two models that produce similar forecasts for the same variable (in this case, one-step ahead equity volatility) can be considered functionally and behaviorally identical. In order to assess this question, the researchers used a historical S&P 500 panel provided by CRSP to develop a model that predicts Garman-Klass log-variance based upon 100 previous data points. They tested three different types of neural networks (SGD, Adam, Muon) and compared them to a linear baseline. Each neural net type was fitted 13 times and evaluated using 564,795 out-of-sample observations.
While many of the configurations produced very similar test NMSE values, the determination of statistical equivalence depends on the chosen tolerance level. A major finding of this study is that there are significant differences between these models even though they have similar forecasting errors. The authors measured the distance in the prediction-space of each model with respect to its within-configuration variability. They also conducted additional trials using matched initialization and data order.
Additionally, they investigated the impact of model architecture on impulse response and SHAP lag-attribution. The authors explored how learning rate and batch size affect curvature dynamics as a potential explanation for why these models behave differently.
Lastly, they demonstrated how the models vary in terms of target-allocation turnover when constructing volatility-sorted portfolios. The authors argue that training configurations should be assessed based on their functional and downstream behavior rather than simply on their scalar forecasting loss.
Critical strengths
This work’s main advantage is that it will never draw conclusions about functional differences based solely upon visual distinctions in the generated response surfaces. Instead, this paper assesses the degree to which there are disagreements among various optimizations when run on the same set of held-out data; then, it compares those disagreements to the variation that occurs when multiple optimizations of the same model are performed. The use of the "stronger" normalization (i.e., using the larger within-optimizer distance) is particularly beneficial. As an illustration of this, the MLP Adam-Muon comparison results in a cross-optimizer distance of 0.0539 compared to within-optimizer distances of 0.0400 and 0.0119, with a pointwise interval for the "stronger" ratio excluding unity.
In addition, the seed-level jackknife accounts for the dependencies created due to distances being shared among different optimizations. By combining the seed-level jackknife with the differentiation between non-rejection and tolerance-based equivalence, this work provides a great deal more descriptive evidence than a traditional optimizer-leaderboard study would.
Critical weaknesses
The comparative analysis of optimizer performance has the potential to influence the overall optimization update rule more than the individual optimization update rule itself. In Table 4, the weight-decay values are varied among the different optimizers. Appendix E.2 clearly indicates that the paired experiments control for initialization and the order of the data; however, they do not control for regularization or computation time.
While the CNN-SGD optimizer is trained for seven epochs, all other CNN optimizers are trained for 50 epochs. Similarly, the paired LSTM-SGD experiment has a different configuration than the main 13-run cohort.
Although matching initialization and batches is helpful, this approach does not provide the means to distinguish implicit optimization biases from explicit regularization and finite-training effects.
Thus, the evidence demonstrates how differences exist between entire training configurations rather than providing additional support for the optimizer-specific interpretations that are presented in the title and conclusions. A controlled comparison of optimizers without explicit decay, or with an explicitly regularized and comparably sized training budget would substantially improve the interpretation of these findings. Curvature-based interventions do not address this gap in identifying causality. Appendix F provides useful information regarding how interventions affect both predictions and curvature.
However, altering learning rate does not isolate curvature as the causal factor behind functional movement.
Moreover, Appendix F.1 acknowledges that contractions occurring within a non-quadratic region could also result in changes to curvature. Such acknowledgment directly contradicts the more definitive claims made in Sections 4.4 and 6 concerning the fitted solution being a minimum. The interventions provide support for a possible dynamic explanation of the observed phenomena, but do not provide support for excluding alternative explanations.
Questions for authors
Will the main differences in function space continue to exist if we control for explicit regularization (i.e., disable weight decay) and have the same amount of time to train in each architecture, and use appropriate learning rates for each optimizer? A single, well-defined example will allow us to determine if an optimizer-specific implicit bias is present as opposed to the overall effects of a properly tuned training setup. Is it possible to limit your conclusions to only those cases where the entire training setup has been optimized?
Can Table 12 be expanded to include the distribution of held out turnover (turnover in the testing set) and the forecast error for each of the retained repeated-fit checkpoints? The main issue with expanding Table 12 is whether the difference between configuration allocations continue to be significantly greater than the difference that results from running the same configuration again. It would also be helpful to compare HAR to other methods in this downstream comparison because HAR produced the lowest NMSE in Table 1 and is a good example of why neural predictors should be retained even though they do not improve forecasting error.
3 comments
Thank you for the detailed rebuttal and clarifications. The responses have adequately addressed my main concerns and improved my assessment of the paper. I have raised my score.
Thanks a lot!
Good luck with your papers!
Best,
The authors
Thank you for recognizing the value of our paper. We have completed the zero-decay, matched-budget experiment and repeated-fit downstream analysis you requested. Supporting pictures.
Critical Weaknesses and Question 1: regularization and training budget.
Thank you for suggesting this control. We are sorry the role of the validation-selected settings was not clearer. Below we explain the choices and the additional experiment.
Original hyperparameters. Learning rate and native weight decay were selected for each optimizer by validation loss. Table 4 therefore reports validation-selected decay values. We tested whether separation persists without decay and with equal duration.
Additional control disabling weight decay. Transformer–SGD and Adam use the same architecture, final-fit data (train+validation), preprocessing and batch size, with zero decay in every parameter group. Both train for exactly 50 epochs without early stopping, over 13 paired initializations. Starting weights, minibatch order and stochastic-layer seeds are paired. Learning rates remain optimizer-specific and previously validation-selected. We compare epoch-50 checkpoints.
Results. Mean test NMSE is 0.57445 ± 0.00139 (SGD) and 0.57613 ± 0.00139 (Adam), mean ± seed SD. Group mean errors are equivalent within the existing exploratory tolerance 0.005 NMSE (paired TOST, p=0.0000136). Paired prediction distance is 0.06309, versus 0.03974 within SGD and 0.05100 within Adam: 1.24 times the larger within baseline, with exploratory pointwise 95% interval [1.08, 1.41]. Thus separation persists with equivalent group errors under the controls you requested. We will include all seed results in the revision.
CNN duration and paired LSTM configuration. Thank you for identifying these issues. CNN–SGD retained its selected seven-epoch recipe for compute budget but we can repeat it at matched horizons in the final version. We corrected the three paired LSTM–SGD fits using the selected configuration and saved starts/data order. Test NMSE is 0.57662 ± 0.00016 (SGD), 0.57541 ± 0.00056 (Adam), 0.58189 ± 0.00050 (Muon). Ratios to the larger within baseline, with paired distances in parentheses, are SGD–Adam: 2.87 (0.04683); SGD–Muon: 1.40 (0.08726); Adam–Muon: 1.31 (0.08188). All three exceed both within baselines. We will update Appendix E.2.
Curvature interventions: interpretation and scope.
Thank you for highlighting where the interpretation needs clarification.
What the intervention tests. A curvature change alone would not establish constrained dynamics; we appreciate the opportunity to clarify this. Appendix F.1 asks a more specific question. At a fixed quadratic minimum, LR/10 reduces fluctuations without changing the Hessian. In Appendix F.3, for instance MLP–SGD instead shows sustained sharpening and further loss reduction relative to a stable control. This sustained contrast supports a learning-rate-dependent curvature constraint.
Causal interpretation. You are right that LR/10 does not uniquely identify curvature as the mediator of functional movement. Appendix F.1 explicitly discusses the nonquadratic alternative. We will make this scope equally clear in Sections 4.4 and 6: the interventions support constrained dynamics without excluding every alternative explanation.
Question 2: held-out turnover across repeated fits and HAR.
Thank you for suggesting this check across initializations, the results further strengthen our previous conclusion.
Expanded Table 12. We evaluated all 13 retained fits of each of the 12 paper configurations, reporting held-out turnover and gross Sharpe as mean ± seed SD. Companion tables give every checkpoint and its forecast error. All methods use the same test dates, assets and portfolio rule.
Results across initializations. Across-optimizer allocation distance exceeds both within baselines in all 24 architecture/pair/quintile comparisons for Q1/Q5, with exploratory pointwise 95% ratio intervals above one. For MLP Adam–Muon Q5, turnover is 14.97 ± 0.25% versus 11.31 ± 0.14%, with gross Sharpe 0.841 ± 0.007 versus 0.845 ± 0.005. Allocation distance is 1.99 [1.90, 2.08] times the larger baseline. This directly addresses your question: allocation differences exceed rerun variability, with substantially different turnover at very similar gross Sharpe. We also added HAR on the same panel: Q1/Q5 turnover is 26.00%/18.78%, with gross Sharpe 1.176/0.833. As set forth in the introduction of the paper and Question 1, our contribution concerns optimizer-dependent functions and decisions, rather than forecasting superiority over established methods.
Please let us know what doubts remain. We are happy to provide clarification or targeted experiments and hope this evidence gives you grounds to reconsider your assessment and score.