- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focues on the impact of reward function selection for RL-based financial trading algorithms. Six reward families are considered across various types of RL algorithms. Through experiments, the authors find many interesting observations in this domain.
Critical strengths
- The general motivation of this paper is interesting. Reward function design is very important for the design of RL-based trading algorithms.
- The flow of this work is clear and easy to follow.
Critical weaknesses
- Although the perspective of this paper is interesting, the depth of the research is poor. Overall, this paper is more like a course project rather than a submission of top-tier conference such as ICLR.
- Theoritical results is required for this kind of analysis. Theoretical bounds can significantly help on the understanding of reward's impact.
- I don't think the experiments are comprehensive. For instance, the reward design of RL-based algorithms in this setting can be more complex such as the combination of the six reward families.
- For RL algorithms, several classic RL algorithms are not enough. More advanced RL-based algorithms for trading need to be compared.
- The plots and figures in this work need to be improved. The current version is quite big with limited useful information.
- The conclusion derived from the experiments does not make sense to me. Observations on some specific datasets and RL algorithms can not fully support the conclusions.
Questions for authors
See weakness above.
Author Response — Experimental Depth, Statistical Evidence, and Research Scope
We thank the reviewer for the detailed assessment. We recognize the concerns regarding experimental depth, theoretical justification, composite rewards, algorithm coverage, and presentation. We conducted additional controlled and historical experiments to strengthen the empirical evidence.
Our contribution is an empirical investigation of the reliability of reward-objective comparisons under training stochasticity, rather than a new reward function or reinforcement-learning algorithm.
The controlled extension examines six reward objectives, four configurations, and ten matched seeds, producing 240 PPO training jobs and 720 held-out evaluations.
The configurations include the baseline, additional causal historical state features, modified critic-loss scaling, and both modifications. Each trained policy is evaluated using three checkpoint-selection rules: maximum validation annualized return, maximum validation internal reward, and the final checkpoint.
Reward-dependent checkpoint-selection sensitivity remains significant across all four configurations (Friedman tests, Holm-adjusted p values from 2.10 × 10⁻⁷ to 0.000500).
These findings strengthen the evidence for evaluation-protocol sensitivity without establishing a significant reward-by-control interaction or eliminating every possible confounder.
We evaluated two additional historical periods, 2018–2020 and 2021–2022, alongside 2023–2025.
These extensions add 120 training jobs and 360 held-out evaluations, bringing the supplementary experiments to 360 training jobs and 1,080 evaluations.
The reward-dependent checkpoint-selection contrast is significant across all three periods (Holm-adjusted p = 0.00234, 0.000349, and 1.57 × 10⁻⁷).
Excluding the negative-maximum-drawdown objective removes statistical significance in the earliest period (p = 0.9724), illustrating the influence of individual objectives on aggregate conclusions.
These findings provide temporal evidence within one asset, not cross-asset validation.
We agree that theoretical bounds could strengthen the work. However, our contribution is empirical rather than a theoretical characterization of reward-dependent generalization.
The statistical analyses quantify observed variability and sensitivity under the evaluated conditions. They do not establish universal performance bounds or guarantees concerning reward rankings.
We will clarify this distinction and avoid interpreting empirical confidence intervals as theoretical guarantees.
Composite objectives combining returns, risk, and drawdown penalties are practically relevant.
Our study examines six specified reward objectives separately to characterize the reliability of their comparisons. Composite rewards would introduce additional weighting and tuning decisions requiring a separate experimental design.
We therefore identify composite reward evaluation as future work rather than claim that it is covered by the present experiments.
We acknowledge that additional algorithms, including off-policy methods such as SAC, would strengthen external validity. We do not claim algorithm-independent conclusions.
We also clarify that 250 refers to outer PPO rollout/update iterations, each containing ten optimization epochs, rather than 250 optimization epochs in total. Summary
The additional experiments strengthen the empirical contribution through controlled comparisons, historical evaluation, and checkpoint-selection sensitivity analysis. Theoretical guarantees, composite objectives, additional assets, and broader algorithm coverage remain outside the demonstrated scope. The Supplementary Experimental Analysis provides additional empirical evidence relevant to the reviewer's concerns, particularly Sections 2–5 and 8 (Questions 1–4 and 7). Figures 1–3 summarize the controlled experiments, reward-dependent checkpoint-selection effects, and historical evaluations. Sections 6–7 provide further behavioral and seed-stability diagnostics. Question 7 clarifies the study's limitations, including the absence of theoretical guarantees, composite-reward experiments, and broader algorithmic validation.
We respectfully ask the reviewer to consider this additional evidence when reassessing the paper. Anonymous reproducibility repository: https://anonymous.4open.science/r/reward-conclusion-reliability-C444/