acceptodds
Under review as a conference paper at ICLR 2027

How Reliable Are Reward-Objective Conclusions in Financial Reinforcement Learning?

Abstract

Financial reinforcement learning (RL) studies use materially different reward objec- tives, while separate work shows that repeated RL training can yield substantially different outcomes across random seeds. Neither observation is treated as new here. Instead, this study asks how reliable conclusions attributed to reward choice are when evaluated across independent stochastic retraining and different subsets of observed runs. Six reward families—portfolio-value change, log wealth growth, differential Sharpe ratio, rolling Sortino ratio, negative maximum drawdown, and Sharpe minus conditional-value-at-risk penalty—are evaluated with Proximal Pol- icy Optimization (PPO) on a controlled single-asset task using 10 matched seeds per reward. Omnibus tests do not detect a significant reward effect on annualized return, Sharpe ratio, Sortino ratio, or maximum drawdown; reward-associated differences are detected for turnover and long exposure, although most policies remain predominantly long. Counterfactual single-seed comparisons produce dif- ferent apparent reward leaders. Robust summaries reinforce that caution: replacing the mean with the median or interquartile mean (IQM) substantially compresses the apparent performance separation among rewards, and matched-seed bootstrap resampling does not yield a stable IQM leader. Exhaustive enumeration of matched seed subsets further shows that the non-significant return and Sharpe conclusions are stable, whereas the long-exposure conclusion becomes increasingly stable as more runs are included: 99.2% of seven-seed subsets and all subsets of eight or more seeds yield a significant Friedman reward effect. Turnover stabilizes more slowly. Reward rankings also become progressively more similar to the 10-seed reference as additional runs are included, while winner claims remain fragile when leading estimates are close. No significant performance reward effect is detected across three temporal windows or with a second actor–critic algorithm. The contribution is therefore a reward-specific conclusion-reliability analysis: the results support reporting full run distributions and directly examining whether conclusions attributed to reward choice remain stable under stochastic retraining

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Financial Reinforcement Learning, Reward Design, Reward Sensitivity, Training Stochasticity, Random Seeds, Empirical Evaluation, Portfolio Management

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HDjLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focues on the impact of reward function selection for RL-based financial trading algorithms. Six reward families are considered across various types of RL algorithms. Through experiments, the authors find many interesting observations in this domain.

Critical strengths
  • The general motivation of this paper is interesting. Reward function design is very important for the design of RL-based trading algorithms.
  • The flow of this work is clear and easy to follow.
Critical weaknesses
  • Although the perspective of this paper is interesting, the depth of the research is poor. Overall, this paper is more like a course project rather than a submission of top-tier conference such as ICLR.
  • Theoritical results is required for this kind of analysis. Theoretical bounds can significantly help on the understanding of reward's impact.
  • I don't think the experiments are comprehensive. For instance, the reward design of RL-based algorithms in this setting can be more complex such as the combination of the six reward families.
  • For RL algorithms, several classic RL algorithms are not enough. More advanced RL-based algorithms for trading need to be compared.
  • The plots and figures in this work need to be improved. The current version is quite big with limited useful information.
  • The conclusion derived from the experiments does not make sense to me. Observations on some specific datasets and RL algorithms can not fully support the conclusions.
Questions for authors

See weakness above.

1 comment
AuthorsOpenReview ↗

Author Response — Experimental Depth, Statistical Evidence, and Research Scope

We thank the reviewer for the detailed assessment. We recognize the concerns regarding experimental depth, theoretical justification, composite rewards, algorithm coverage, and presentation. We conducted additional controlled and historical experiments to strengthen the empirical evidence.

  1. Research contribution and experimental depth

Our contribution is an empirical investigation of the reliability of reward-objective comparisons under training stochasticity, rather than a new reward function or reinforcement-learning algorithm.

The controlled extension examines six reward objectives, four configurations, and ten matched seeds, producing 240 PPO training jobs and 720 held-out evaluations.

The configurations include the baseline, additional causal historical state features, modified critic-loss scaling, and both modifications. Each trained policy is evaluated using three checkpoint-selection rules: maximum validation annualized return, maximum validation internal reward, and the final checkpoint.

Reward-dependent checkpoint-selection sensitivity remains significant across all four configurations (Friedman tests, Holm-adjusted p values from 2.10 × 10⁻⁷ to 0.000500).

These findings strengthen the evidence for evaluation-protocol sensitivity without establishing a significant reward-by-control interaction or eliminating every possible confounder.

  1. Historical evaluation and statistical robustness

We evaluated two additional historical periods, 2018–2020 and 2021–2022, alongside 2023–2025.

These extensions add 120 training jobs and 360 held-out evaluations, bringing the supplementary experiments to 360 training jobs and 1,080 evaluations.

The reward-dependent checkpoint-selection contrast is significant across all three periods (Holm-adjusted p = 0.00234, 0.000349, and 1.57 × 10⁻⁷).

Excluding the negative-maximum-drawdown objective removes statistical significance in the earliest period (p = 0.9724), illustrating the influence of individual objectives on aggregate conclusions.

These findings provide temporal evidence within one asset, not cross-asset validation.

  1. Theoretical justification and bounds

We agree that theoretical bounds could strengthen the work. However, our contribution is empirical rather than a theoretical characterization of reward-dependent generalization.

The statistical analyses quantify observed variability and sensitivity under the evaluated conditions. They do not establish universal performance bounds or guarantees concerning reward rankings.

We will clarify this distinction and avoid interpreting empirical confidence intervals as theoretical guarantees.

  1. Composite reward objectives

Composite objectives combining returns, risk, and drawdown penalties are practically relevant.

Our study examines six specified reward objectives separately to characterize the reliability of their comparisons. Composite rewards would introduce additional weighting and tuning decisions requiring a separate experimental design.

We therefore identify composite reward evaluation as future work rather than claim that it is covered by the present experiments.

  1. Algorithm coverage The original study includes PPO and a limited A2C robustness comparison. The supplementary experiments focus on PPO to preserve matched conditions while examining state information, critic-loss scaling, and checkpoint selection.

We acknowledge that additional algorithms, including off-policy methods such as SAC, would strengthen external validity. We do not claim algorithm-independent conclusions.

  1. Figures and presentation We prepared supplementary figures showing controlled checkpoint-selection effects, reward-specific differences, historical comparisons, negative-maximum-drawdown cash exposure, and seed-subset stability.

We also clarify that 250 refers to outer PPO rollout/update iterations, each containing ten optimization epochs, rather than 250 optimization epochs in total. Summary

The additional experiments strengthen the empirical contribution through controlled comparisons, historical evaluation, and checkpoint-selection sensitivity analysis. Theoretical guarantees, composite objectives, additional assets, and broader algorithm coverage remain outside the demonstrated scope. The Supplementary Experimental Analysis provides additional empirical evidence relevant to the reviewer's concerns, particularly Sections 2–5 and 8 (Questions 1–4 and 7). Figures 1–3 summarize the controlled experiments, reward-dependent checkpoint-selection effects, and historical evaluations. Sections 6–7 provide further behavioral and seed-stability diagnostics. Question 7 clarifies the study's limitations, including the absence of theoretical guarantees, composite-reward experiments, and broader algorithmic validation.

We respectfully ask the reviewer to consider this additional evidence when reassessing the paper. Anonymous reproducibility repository: https://anonymous.4open.science/r/reward-conclusion-reliability-C444/

Reviewer_rXuBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper attempts to answer whether researchers should trust their reward design when compared to others based on a market metric in financial reinforcement learning. The authors conduct three sets of experiments to understand among six reward designs if they show relevant performance difference, if those differences are stable across multiple runs, and if the performance pattern persists across a second algorithm, temporal windows, and state-information. The main contribution of the paper is a deeper understanding on how reward design evaluation in financial RL can be statistically unreliable. Among the results, the paper presents that due to wide difference in performance across different seeds, most metrics are statistically unreliable. turnover and long exposure present themselves as the only metrics that consistently rank reward designs, yet reliable identification of specific pairs is limited to one long-exposure only. Subsequent testing mostly confirms their initial findings but provides useful explanations into how the metrics behave when ranking reward designs.

Critical strengths
  1. The paper is quite clear in its writing and experimental design. Most of the terminology is properly introduced and carefully brought over from RL, statistics, and financial backgrounds. The writing is straightforward and easy to follow. The research questions build on one another so the reader is taken along to learn more about the performance metrics behavior with reward designs.
Critical weaknesses
  1. Specificity to financial market applications. The paper is designed around analyzing metrics and reward designs for financial market applications only. It is understandable that this field is highly dependent on historical data, and counterfactual scoring is indeed a significant problem, which is likely why this kind of analysis is important. If a reward reflects a real life metric, then it should do so reliably. However, this problem is not specific to financial reinforcement learning. The RL field has struggled with reward design and statistical significance of its findings, as referenced by the authors themselves. I completely understand that overgeneralization of findings is irresponsible and dangerous, however, discussion around possible implications for the broader field are still valid and connect the paper's findings to future work.
  2. Insufficient discussion on reward design. The paper selects six reward objectives to compare. Results show that most rewards produce statistically similar performance metrics and only a single pair remains significant, negative-MDD and log-return. However, the authors do not discuss why this pair specifically shows distinction, or furthermore, why all objectives, despite having very different designs still cannot be told apart from each other. The paper does a great job showing us how unreliable these comparisons are but provides little explanation as to why, even if hypothesized.
Questions for authors
  1. Line 188 says that training runs for 250 epochs, does it mean 250 PPO iterations or actual 250 epochs resulting 25 iterations with 10 epochs per iteration?
  2. Have he authors considered running more than 10 seeds?
  3. Could the authors explain their choice for the second algorithm being A2C instead of the more established SAC?
1 comment
AuthorsOpenReview ↗

Author Response - Generalizability, Reward Behavior, and Training Protocol Clarifications

We thank the reviewer for the constructive assessment and for recognizing the importance of evaluating reward-objective conclusions under training stochasticity. We address the questions concerning generalizability, reward behavior, training budget, seed count, and algorithm selection.

  1. Implications beyond financial reinforcement learning

Our central question is whether comparative conclusions about reward objectives remain reliable under stochastic training and alternative evaluation choices. This question is relevant to other reinforcement-learning applications, although our empirical evidence is restricted to one financial asset.

We extended the evaluation to two historical periods (2018–2020 and 2021–2022), in addition to 2023–2025. These experiments examine temporal robustness within the same asset, not generality across assets or domains.

  1. Why does negative maximum drawdown behave differently?

The negative-maximum-drawdown objective penalizes declines from the portfolio's previous peak, potentially encouraging conservative policies that avoid market exposure.

Our additional experiments provide behavioral evidence. Under checkpoint selection by validation internal reward, all ten policies for this objective in each historical period exhibited at least 95% cash exposure during held-out testing.

Under validation annual-return selection, the corresponding counts were 0/10, 3/10, and 0/10 for 2018–2020, 2021–2022, and 2023–2025, respectively.

Thus, the observed cash-heavy behavior depends on both the reward objective and checkpoint-selection criterion.

  1. Why are other reward objectives difficult to distinguish?

Statistically nonsignificant differences do not establish equivalence. Several rewards may produce similar held-out performance despite differing mathematically, while finite training repetitions limit statistical resolution.

We therefore distinguish statistically unresolved comparisons from evidence of equivalent performance. Our matched-seed and seed-subset analyses help characterize the stability of these conclusions.

  1. Clarification of 250 training iterations

The earlier terminology "250 epochs" was imprecise. Each job uses 250 outer PPO rollout/update iterations, with ten optimization epochs per update. Validation occurs every ten outer iterations, yielding 25 checkpoints.

The controlled extension comprises six rewards, four configurations, and ten matched seeds: 240 training jobs. Three checkpoint-selection rules produce 720 held-out evaluations.

Two historical extensions add 60 training jobs and 180 evaluations each, yielding 360 training jobs and 1,080 held-out evaluations overall.

  1. Why ten seeds?

Ten matched seeds provide repeated stochastic outcomes and enable paired reward comparisons. However, ten seeds cannot reliably detect every small effect or rare failure.

The original study therefore includes seed-subset and ranking-stability analyses. We do not treat multiple checkpoint evaluations of the same trained policy as independent training repetitions.

  1. Why A2C rather than SAC?

A2C provides a second actor-critic algorithm with a different update mechanism, enabling a limited robustness comparison with PPO.

We agree that this does not establish algorithm-independent conclusions. SAC would provide a useful off-policy, entropy-regularized comparison, but was not evaluated. We will clarify this limitation rather than imply broader algorithmic validation.

Additional evidence

The supplementary experiments also show significant reward-dependent checkpoint-selection sensitivity in all four controlled configurations (Holm-adjusted p values ranging from 2.10 × 10⁻⁷ to 0.000500).

The historical baseline comparisons are significant across all three evaluation periods (Holm-adjusted p = 0.00234, 0.000349, and 1.57 × 10⁻⁷).

These findings strengthen the evidence for evaluation-protocol sensitivity while remaining specific to the studied environment. Further evidence and methodological clarifications are provided in the Supplementary Experimental Analysis, particularly Sections 5–8 (Questions 4–7). Figure 4 illustrates the negative-maximum-drawdown cash-policy behavior, while Figure 5 examines seed-subset ranking stability. Section 2 and Question 7 clarify the PPO training budget, checkpoint evaluations, and remaining algorithmic limitations.

The experimental code, results, and analysis scripts are available at:

https://anonymous.4open.science/r/reward-conclusion-reliability-C444/

Reviewer_NgYdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

There could be different reward design choices for training a trading Ai agent including making money, avoiding loss or balance between returns against risk. the paper studies six reward designs and investigates if the conclusions about best reward remains consistent across runs. The empirical results show that there is no statistically differences in finance performance, while some trading behaviors share some common patterns.

Critical strengths

In general, the conducted experiments appear to be considerate, as it provides matched comparison, summaries, transparent checks of how conclusions change with the available runs, etc.

Critical weaknesses
  1. The paper did cite some related works that study how different reward designs might affect the empirical performance, but how those works relate to the trading behavior/finance domain is very limited. Importantly, I would like to see what is really discovered to be new; what is merely a specific known observation in the finance domain.

  2. The experiments setting provides limited opportunity to distinguish meaningful reward preferences, it is not clear if the findings persist across different assets/task where different reward produce substantially different strategies.

  3. In the empirical design, the reward effects remain entangled with information, model optimization and selection. For example, those history-dependent rewards might need some history-dependent statistics that are included in the state, and critic targets remain differently scaled and checkpoint is selected using annualized return even for risk-sensitive training objectives. I think these limit what final behavior consequences can really be attributed to the reward objectives themselves.

  4. There is a solid reason to doubt the significance of the work: the scope of the experiments merely make it a case study, and there is no broadly validated diagnostic rules for reward comparison.

Questions for authors

see weakness.

1 comment
AuthorsOpenReview ↗

Author Response - Clarification of Contributions and Additional Controlled Experimental Evidence

We thank the reviewer for the constructive feedback. We conducted additional experiments to address concerns about novelty, generalizability, experimental confounding, and diagnostic recommendations.

  1. Novelty and contribution

Our contribution is not that reward functions affect trading performance, but that conclusions comparing reward objectives may be unreliable under training stochasticity and evaluation choices. We examine the stability of comparative conclusions rather than identifying a universally superior reward.

We will clarify this distinction from existing reward-design literature.

  1. Generalizability

We acknowledge that studying one exchange-traded fund limits external validity. We extended the evaluation to two historical periods (2018–2020 and 2021–2022), alongside 2023–2025.

Reward-dependent checkpoint-selection sensitivity was significant across all three periods (Holm-adjusted p = 0.00234, 0.000349, and 1.57 × 10⁻⁷).

These findings strengthen temporal evidence within the studied asset but do not establish cross-asset or cross-domain generality.

  1. Experimental confounding

We conducted controlled experiments involving six reward objectives, four configurations, and ten matched seeds, totaling 240 PPO training jobs and 720 held-out evaluations.

The configurations include the baseline, causal historical state features, modified critic-loss scaling, and both modifications. Each policy was evaluated under three checkpoint-selection rules: validation annualized return, validation internal reward, and final checkpoint.

Reward-dependent checkpoint-selection sensitivity remained significant across all configurations:

Baseline: Friedman χ² = 42.25, Holm-adjusted p = 2.10 × 10⁻⁷. Historical state features: χ² = 22.11, p = 0.000500. Critic-loss scaling: χ² = 28.09, p = 0.0000698. Both modifications: χ² = 31.99, p = 0.0000179.

These findings show that selection sensitivity persists under the examined controls. However, history augmentation does not establish Markov sufficiency, critic-loss scaling is not complete reward normalization, and no significant reward-by-control interaction was established.

  1. Diagnostic recommendations

We propose a reproducible evaluation procedure combining matched-seed comparisons, checkpoint-selection sensitivity, seed-subset stability, and reward-specific behavioral diagnostics.

The importance of examining individual rewards is illustrated by the historical analysis: excluding negative maximum drawdown eliminates statistical significance in the earliest period (p = 0.9724).

We present these as practical evaluation recommendations, not universally validated rules.

Summary

Additional evidence addressing the reviewer's concerns is provided in the Supplementary Experimental Analysis, particularly Sections 2–5 and 8 (Questions 1–4 and 7). Figures 1–3 present the controlled configurations, reward-dependent checkpoint-selection sensitivity, and historical robustness. These analyses clarify the effects of state-history augmentation, critic-loss scaling, and checkpoint selection, while acknowledging the single-asset limitation.

Anonymous reproducibility repository:

https://anonymous.4open.science/r/reward-conclusion-reliability-C444/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.