acceptodds
Under review as a conference paper at ICLR 2027

Beyond Expected Returns: Time-Average Regularized Policy Optimization for Trajectory-Level Performance

Abstract

Standard reinforcement learning learns policies by maximizing the expected return, which averages performance over many possible trajectories. However, this objective may not fully capture how individual trajectories behave over time. For instance, the expected return can be heavily influenced by rare outcomes and may therefore differ substantially from the behavior observed along most individual trajectories. An alternative perspective is to consider time-average growth, which measures how the return of a single trajectory evolves over time. To incorporate such information into policy optimization, we propose a variant of the well-known Box–Cox transformation from statistics. In particular, along each trajectory, we transform returns and use increments between consecutive transformed returns as an auxiliary signal for policy improvement. We implement this growth signal in proximal policy optimization (PPO) via an actor-only update, while the critic, advantage estimation, and value learning remain unchanged. This design preserves conventional value learning while allowing policy improvement to exploit more trajectory-level information to complement conventional expected-return optimization. We provide theoretical conditions under which this transformation yields a well-defined long-run growth rate for a broad class of return dynamics, including additive and multiplicative special cases. Experiments across continuous-control and Atari benchmarks show that suitable transformation and regularization settings can improve learning in several tasks. These results suggest that transformed-growth signals can provide useful complementary information to expected-return optimization in policy learning.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, policy optimization, time-average growth rate, non-ergodicity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ckRZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper augments PPO with a trajectory-dependent growth signal obtained from increments of a signed Box–Cox transformation of cumulative rewards. The signal is mixed with the conventional advantage for actor updates, while the critic and GAE remain unchanged. The paper characterizes compatible return dynamics and establishes conditional long-run growth and variance-contraction results. Validation includes a multiplicative coin-toss problem, six MuJoCo tasks, and four Atari games, with improvements reported for selected configurations.

Critical strengths
  1. The actor-only modification is simple and easy to integrate into existing PPO implementations. Retaining conventional value learning makes the approach practically appealing.

  2. The experiments cover both continuous and discrete control, with promising reported improvements on tasks such as BeamRider and Reacher. Providing complete parameter sweeps also helps assess sensitivity and limitations.

Critical weaknesses
  1. The practical update is not adequately connected to the proposed objective. Section 4.1 derives a policy gradient using the trajectory-level statistic Gu,TG_{u,T}, but the algorithm replaces it with the immediate increment gtg_t. Although acknowledged as a local surrogate, this replacement generally omits an action’s influence on future growth. For example, an action yielding an immediate cost followed by a larger delayed reward can receive a negative local growth signal despite improving total transformed return. No conditions or bias analysis establish when this surrogate supports the intended optimization.

  2. The reported evaluation statistics are internally inconsistent under the stated definitions. Appendix C.2 defines IQM as the average of the middle three of five per-seed scores. However, the BeamRider baseline reports mean 2919.4, median 2608.0, standard deviation 568.8, and IQM 3144.6. These imply that the two largest scores are at least 4217.8, requiring a population standard deviation of at least 821.2. Thus, these statistics cannot all describe the same five scores. This may reflect a reporting or aggregation error, but it needs resolution before the empirical evidence can be assessed reliably.

Questions for authors
  1. What objective does the implemented local growth update optimize? Under what assumptions can gtg_t replace the trajectory-level growth statistic without losing essential temporal credit assignment?

  2. How is the infinite-horizon growth objective reconciled with episodic resets? Without resets, bounded rewards imply that the proposed growth rate vanishes for all policies when 0≤α<10\leq\alpha<1.

  3. Could you provide the five BeamRider baseline scores and the statistics-generation procedure, reconcile the discrepancy above, and verify the remaining tables?

4 comments
AuthorsOpenReview ↗

Part 1: Growth objective and local approximation

We thank the reviewer for the careful assessment and for recognizing the simplicity and practical appeal of the actor-only modification. We address the related weaknesses and questions together below.

1. Local growth objective and approximation (Weakness 1 and Question 1).

Our proposed objective combines expected additive return with policy-level growth: Jλg(π)=(1−λg)Jadd(π)+λgGπ.J_{\lambda_g}(\pi) = (1-\lambda_g)J_{\mathrm{add}}(\pi) + \lambda_g G^\pi. Simply adding the same policy-level growth scalar value to both QπQ^\pi and VπV^\pi gives

+\lambda_g G^\pi\bigr] - \bigl[(1-\lambda_g)V_{\mathrm{add}}^\pi(s) +\lambda_g G^\pi\bigr] =(1-\lambda_g)A_{\mathrm{add}}^\pi(s,a).$$ The shared growth term cancels in their advantage difference and provides no action-specific growth signal. We will explain this observation in a short appendix remark. We then use transformed-return increments as auxiliary feedback directly in the actor update. For nonlinear $u$, the same immediate reward can produce different growth signals depending on the return accumulated so far. This allows trajectory information to influence the weighting of actions during policy learning. The implementation uses PPO's clipped surrogate with a mixture of this signal and the original reward-based GAE, while retaining the original critic. Our parameter sweep includes $\alpha=1$, for which $U_1(R)=R$ and the auxiliary increment reduces to the original reward before normalization. This provides a control for adding the original reward as an auxiliary signal. For example, at the same growth weight $\lambda_g=0.10$ in Hopper, the mean return is $2892.2$ for $\alpha=1$ and $3476.6$ for $\alpha=0.25$, compared with $2915.8$ for baseline PPO. We will highlight this comparison in the revised results discussion. We thank the reviewer for highlighting the need to analyze the approximation introduced by using the local increment in the growth component. The newly added Appendix A.4 derives $$\nabla_\theta J_{u,T} = D_{\mathrm{local}}+B_T,$$ where $B_T$ captures the current action's influence on subsequent transformed increments. The local direction equals the full growth gradient if, given the current history, expected future transformed growth does not depend on the current action. More generally, under compatible transformed dynamics, geometric mixing, and uniformly bounded second moments of the policy score, we obtain $$\|B_T\| \leq KC \operatorname{span}(a)\frac{\rho}{1-\rho},$$ with the constants defined in Appendix A.4. For a fixed policy and transformation, this bound does not grow with the horizon. The analysis concerns the growth contribution before normalization and PPO clipping. We will summarize these conditions and results in Section 4.1.
AuthorsOpenReview ↗

Part 2: Episodic resets, evaluation statistics, and additional experiments

2. Episodic resets and the long-run criterion (Question 2).

As specified in Algorithm 1, each rollout is collected under the current behavior policy and may contain several episodes or episode segments. Cumulative return is tracked within each episode. After including the final reward of an episode, it is reset to its initial value for the next episode. Thus, each raw growth increment is computed from consecutive cumulative returns within the same episode. The reset itself is not counted as a growth increment. We will clarify this distinction between rollouts and episodes in the main text. Growth increments are standardized using the mean and standard deviation of the current rollout, without pooling statistics from previous rollouts since the time-average can change as the policy is updated. This controls the scale of the auxiliary signal under the same policy, while the original GAE retains its minibatch normalization.

Our theoretical analysis does not assume uniformly bounded raw rewards. The coefficient conditions concern a(Zt)a(Z_t) and b(Zt)b(Z_t) of the transformed dynamics, allowing compatible return processes with unbounded reward increments. For a continuing process with bounded rewards, however, the reviewer is correct that RT=O(T)R_T=O(T) and the criterion vanishes for 0≤α<10\leq\alpha<1. The identity endpoint α=1\alpha=1 retains the rate cc for linear accumulation RT=R0+cTR_T=R_0+cT, which our dynamic family covers in Sections 4.2--4.3. Therefore, our method is particularly suited to settings where accumulated return affects subsequent reward magnitudes, such as compounding wealth dynamics. Our parameter sweep also includes α=1\alpha=1, where the auxiliary increment reduces to the original reward, allowing us to compare identity and nonlinear transformations at matched growth weights. We will discuss these intended application settings in Section 4.3 and explicitly highlight the role of the α=1\alpha=1 control in Section 5 of the revised manuscript.

In the episodic benchmarks, we use transformed increments as finite-horizon auxiliary feedback. A zero asymptotic limit does not imply that different finite-horizon returns become indistinguishable: at any fixed finite horizon and with a common initial return, applying the same strictly increasing transformation preserves the ordering of trajectory returns [1, Eqs.(5)--(7)]. We will make this distinction in the revised manuscript.

3. Evaluation statistics and additional experiments (Weakness 2 and Question 3).

We thank the reviewer for highlighting the ambiguity in our statistical definitions. The five per-training-seed mean episodic returns for the BeamRider baseline are 1805.72,3046.90,3096.24,3290.76,3357.42.1805.72,\quad 3046.90,\quad 3096.24,\quad 3290.76,\quad 3357.42. These give a mean of 2919.42919.4, a standard deviation of 568.8568.8 (using divisor 55), and an IQM of 3144.63144.6, matching the reported values. Their median is 3096.243096.24. The reported median of 2608.02608.0 instead summarizes the 500 pooled episodic returns, with 100 evaluation episodes per training seed. It therefore describes a different aggregation level from the five seed means used for Std and IQM.

We have rechecked the reported statistics against the underlying evaluation data and confirmed that the numerical values are correct. The issue lies in our description of the aggregation levels, rather than in the numerical calculations. We retain the pooled episodic median and explicitly stated the aggregation level of each statistic in the metric definitions and table captions in Appendix C.2 of the revised manuscript.

We have also added Survival Ratio in Appendix C.2 which is the fraction of episodes reaching the episode-length cap in MuJoCo, and the IQM of per-training-seed mean episode lengths relative to PPO in Atari. For example, for the selected Hopper configuration, this fraction increases from 53.6%53.6\% for PPO to 97.8%97.8\% for BoxCoxPPO. The selected BeamRider configuration achieves a ratio of 1.4101.410, corresponding to a 41.0%41.0\% increase in this episode-length statistic.

Furthermore, the added Appendix D presents a 30-stock FinRL experiment with compounding portfolio wealth dynamics, including a mean--variance PPO comparison. Across five training seeds, the selected BoxCoxPPO configuration achieves a mean cumulative return of 22.82%22.82\%, compared with 7.21%7.21\% for PPO, 17.33%17.33\% for MV-PPO with λMV=0.5\lambda_{\mathrm{MV}}=0.5, and 12.57%12.57\% with λMV=1\lambda_{\mathrm{MV}}=1. Its standard deviation across training seeds is also lower than PPO's that 6.686.68 versus 15.0415.04 percentage points.

Reference

[1] Peters, O., and Adamou, A. (2018). The time interpretation of expected utility theory. arXiv:1801.03680.

Thank you for the detailed response and additional experiments. Your clarifications have helped address my main concerns, and I have raised my rating accordingly.

Thank you for your engagement during the discussion and for raising your score. We will incorporate the clarifications from our discussion into the final version.

Reviewer_SE2ROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers the problem that the expected return by averaging over trajectories may not reflect the long-run behavior of individual trajectories, particularly under non-ergodic dynamics. To tackle this problem, the proposed method introduces the growth increments of the reward as an auxiliary learning signal when updating the policy and incorporates this signal into the PPO framework. Experiments on multiplicative coin-toss problem, MuJoCo, and Atari environments demonstrate performance improvements over standard PPO in several tasks.

Critical strengths

The proposed method is simple and easy to incorporate with existing actor critic framework. Experiments in multiple MuJoCo and Atari domains demonstrate performance improvements over standard PPO.

Critical weaknesses
  1. Presentation. While I understand the intuition behind the problem this paper aims to address, I find the proposed method insufficiently motivated. Several key design choices lack clear justification. For instance, why is the transformed growth increment defined as in line 129? Why is the signed Box–Cox transformation in Eq. (3) chosen? More importantly, why should incorporating this growth signal into the PPO advantage effectively address the problem? The authors use statements such as "this motivates applying a transformation (u)" (line 127) and "motivated by the Box–Cox transformation" (line 185), but do not sufficiently explain how these choices follow from the problem formulation. Furthermore, the problem considered in this paper is closely related to Sheng & Baumann (2025) (Beyond Expected Value: Geometric Mean Optimization for Long-Term Policy Performance in Reinforcement Learning). Although this work is cited, the manuscript lacks a detailed discussion of how the proposed approach differs from it and what additional contributions are made.

  2. Policy update. In Section 4.1 (line 262), the policy gradient is derived using Gu,TG_{u,T}, which represents the average transformed growth over a trajectory. However, as acknowledged in line 279, the practical algorithm replaces Gu,TG_{u,T} with the local increment gtug_t^u. These two quantities are generally not equivalent, and the manuscript does not provide sufficient justification for this approximation or analyze the potential bias it introduces into the policy gradient. I also have concerns about the trajectory-level time-average growth rate defined in line 137. By definition, Gu(τ)=lim⁡T→∞1T(U(RT)−U(R0))G_u(\tau) = \lim_{T\rightarrow \infty} \frac{1}{T}(U(R_T) - U(R_0)). Consider a simple example where R0=0R_0=0 and rt=1r_t=1 for all tt, so that RT=TR_T=T. When α=0\alpha=0 in UU, the proposed transformation yields Gu(τ)=lim⁡T→∞log⁡(1+T)T=0G_u(\tau)=\lim_{T \rightarrow \infty} \frac{\log(1+T)}{T}=0. Similarly, the limit is zero for any 0<α<10<\alpha<1. This raises a fundamental question about whether the proposed growth objective can meaningfully distinguish different policies. I believe the choice of UU requires more careful justification, particularly regarding the relationship between the transformation, the underlying reward dynamics, and the resulting long-run growth criterion.

  3. Experiments. I feel the experiments do not sufficiently validate the central motivation of this paper. The authors argue that expected return may fail to capture the long-run performance of individual trajectories. But the experiments on MuJoCo and Atari mainly report mean return, median, and IQM. It is unclear whether the considered environments exhibit the discrepancy between ensemble-average and time-average performance that motivates the proposed method. I believe authors should provide more direct evidence of this discrepancy and demonstrate how the proposed method addresses it.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Part 1: Design Choices and Local Approximation

We thank the reviewer for the careful assessment and for recognizing the simplicity of our method and the improvements observed over PPO.

Our central question is: without first identifying the return dynamics or separately fitting an ergodicity transformation, can we directly use accumulative return information to learn policies that perform well on typical trajectories?

With this research question in mind, we address the reviewer's concerns point by point below.

To Weakness 1. Design choices and relation to prior work.

The increment gtu=u(Rt+1)−u(Rt)g_t^u=u(R_{t+1})-u(R_t) follows the transformed-increment formulation in [1]. These increments sum to u(RT)−u(R0)u(R_T)-u(R_0), providing an additive measure of changes in transformed accumulated return.

For nonlinear uu, the increment depends on both the current reward and the accumulated return, making the actor update sensitive to accumulated return. The signed Box--Cox family, linked to estimating ergodicity transformations in [6] and used to model different accumulation dynamics in decision-making research [3], connects identity and signed-logarithmic transformations and supports zero and negative returns.

We thank the reviewer for highlighting the need to analyze the local approximation. Appendix A.4 of the revised manuscript now identifies the future-growth contribution omitted by the local update and derives a horizon-independent bias bound under the stated assumptions. We discuss this analysis in our response to Weakness 2.

Compared with [5], which introduces a modified geometric-mean regularizer in Q-learning, we develop a parameterized transformed-increment signal for PPO's actor update, support continuous actions, and characterize the return dynamics compatible with the transformation family.

We will add a related-work discussion to clarify these distinctions, refine the motivation in Section 2, and summarize the new gradient analysis in Section 4.1 with a reference to Appendix A.4.

To Weakness 2. Local approximation and the long-run criterion.

We have added Appendix A.4 to explicitly analyze the approximation in the actor update. For a fixed transformation and finite horizon, the growth gradient decomposes as ∇θJu,T=Dlocal+BT\nabla_\theta J_{u,T} = D_{\mathrm{local}}+B_T where BTB_T captures the current action's influence on subsequent transformed increments. Under compatible transformed dynamics, geometric mixing, and uniformly bounded second moments of the log-policy gradient, we obtain

\leq KC\operatorname{span}(a)\frac{\rho}{1-\rho},$$ with the constants defined in Appendix A.4. For a fixed policy and transformation, this bound does not grow with the horizon, although the bias need not vanish as the horizon increases. The counterexample illustrates why the transformation must be considered together with the dynamics [2]. For $R_0=0$ and $R_T=cT$ with $c>0$, the limit is zero for $0\leq\alpha<1$, while $\alpha=1$ recovers $c$, indicating $\alpha=1$ gives the correct ergodicity transformation. So this example is also covered by our transformation family. For unknown dynamics, we keep a chosen transformation fixed across policy updates without separately identifying the return dynamics or fitting an ergodicity transformation for each policy. The parameter sweep examines whether this common transformation can provide useful auxiliary feedback from finite-horizon trajectories. It reports both improvements and degradations across configurations, showing that the choice of transformation matters. Developing principled parameter-selection rules is a direction for further work. At a common horizon and initial return, applying the same strictly increasing transformations preserve the ordering of trajectory returns [2, Eqs. (5)--(7)]. And for a fixed policy, $\alpha$ also controls the variance of the transformed statistic. We have revised Proposition 4.4 to make the variance bound's dependence on $\alpha$ explicit. Thus, a zero asymptotic limit does not rule out a useful finite-horizon auxiliary signal. The long-run interpretation requires compatible dynamics.
AuthorsOpenReview ↗

Part 2: Experiments and evidence for trajectory-level performance.

Response to Weakness 3

In the coin-toss setting, the expected-wealth optimum a=1a=1 makes expected wealth increase while wealth tends to zero along almost every trajectory under repeated play (see [1, Fig. 1(a)]). Our reported configuration learns actions close to a=0.25a=0.25, the optimum for long-run growth along individual trajectories [1, Sec. 3].

We additionally report the Survival Ratio to complement the episodic-return statistics. In MuJoCo, this measures the fraction of evaluation episodes reaching the environment's maximum episode length. For the selected configurations, the fraction reaching the 10001000-step limit increases from 53.6%53.6\% to 97.8%97.8\% in Hopper, and from 66.6%66.6\% to 96.6%96.6\% in Walker2d (Table 1 in Section 5 and Table 3 in Appendix C.2). For Atari, Survival Ratio measures episode length relative to PPO. For example, the selected BeamRider configuration achieves a ratio of 1.4101.410 (Table 3).

We also add a 30-stock FinRL task [4] over 2025, where portfolio gains and losses depend on the capital invested and trading decisions, while wealth changes affect the capital available for subsequent investment. Across five training seeds, the reported BoxCoxPPO configuration achieves a mean return of 22.82%22.82\% (Std 6.686.68), compared with 7.21%7.21\% (Std 15.0415.04) for PPO baseline. The setup, full results, and portfolio-return curves are provided in Appendix D.

We will clarify the roles of these experiments: coin toss demonstrates the ensemble--time discrepancy in a setting with known dynamics, while the benchmarks evaluate the practical effect of the auxiliary signal.

References

[1] Baumann et al. (2025).
Reinforcement learning with non-ergodic reward increments: robustness via ergodicity transformations.
Transactions on Machine Learning Research.

[2] Peters et al. (2018).
The time interpretation of expected utility theory.
arXiv preprint arXiv:1801.03680.

[3] Price et al. (2023).
Distinguishing risk preferences using repeated gambles.
arXiv preprint arXiv:2308.07054.

[4] Liu et al. (2021).
FinRL: Deep reinforcement learning framework to automate trading in quantitative finance.
Proceedings of the Second ACM International Conference on AI in Finance

[5] Sheng et al. (2025).
Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learning.
IEEE 64th Conference on Decision and Control.

[6] Britto, A. (2026).
Likelihood-based ergodicity transformations in time series analysis.
arXiv preprint arXiv:2601.11237.

Thanks for the reply.

Policy gradient approximation. I appreciate the additional analysis in Appendix A.4. However, a horizon-independent bias bound does not necessarily imply that the bias is small or that the resulting update is an ascent direction for the proposed objective.

Long-run growth objective. The counter example I mentioned seems to hold as long as the reward is bounded, i.e., for any MDP with bounded rewards, the proposed long-run growth rate is zero for all policies whenever 0≤α<10\leq\alpha<1. Sorry that due to the latex support here, I can not provided detailed derivations, but authors can validate by themselves. Interestingly, the best-performing configuration reported in multiplicative coin-toss setting uses α=0.25\alpha=0.25. Thus, even in this controlled setting, it is unclear how the empirical success of the proposed method follows from the theoretical growth objective.

We thank the reviewer for the follow-up and for generalizing the counterexample, which helped us state the scope of our results more precisely. We respond to the two points below.

Local approximation. We agree that a horizon-independent bias bound alone does not guarantee ascent. Appendix A.5 gives a sufficient condition for the mixed objective with finite-horizon growth. For the population update before signal normalization and PPO clipping, with an exact additive-return gradient, dλgd_{\lambda_g} is a strict ascent direction whenever λgMT<∥dλg∥\lambda_g M_T<\|d_{\lambda_g}\|, where MTM_T bounds the omitted future-growth contribution.

Long-run growth criterion. We agree that for uniformly bounded rewards and 0≤α<10\leq\alpha<1, the long-run growth criterion is zero for every policy. Indeed, ∣RT∣≤∣R0∣+cT|R_T|\leq |R_0|+cT and UαU_\alpha grows sublinearly, so the infinite-horizon mixed objective reduces to (1−λg)Jadd(1-\lambda_g)J_{\mathrm{add}}. However, at a fixed finite horizon and a common initial return, a nonlinear signed Box–Cox transformation preserves the ordering of trajectory returns while changing the gaps between them. Two policies with the same expected cumulative return can therefore have different transformed returns. To see how this affects the actor's feedback, consider two trajectories with different nonnegative accumulated returns that receive the same positive reward at the same time step. At α=1\alpha=1, both receive the same auxiliary increment rtr_t. For 0≤α<10\leq\alpha<1, that reward produces a larger increment on the trajectory that has accumulated less return and a smaller one on the trajectory that has accumulated more. This reflects our motivation in Section 1. A few atypically high-return rollouts can dominate the mean, so we want the actor's feedback to reflect progress along each individual trajectory.

Sections 4.2–4.3 identify a class of return dynamics for which, under a fixed policy and the stated ergodicity assumptions, the time average of these increments converges along individual trajectories. When the return dynamics are unknown, the actor can still use the increments observed in the current rollout as finite-horizon auxiliary feedback, without estimating a long-run growth rate for each policy. The critic uses the original rewards. Our episodic experiments test whether this auxiliary feedback improves the learned policy.

Coin toss. In our multiplicative coin-toss model, the reward on each toss scales with current wealth. Maximizing expected log-wealth growth gives the optimal betting fraction a⋆=0.25a^\star=0.25, which gives us a criterion for evaluating the learned actions. During training, α\alpha determines the local feedback mixed with the original reward advantage, so the log-growth calculation alone does not determine which training configuration will recover a⋆a^\star most closely. Table 2 shows that the logarithmic endpoint of our family also yields bets close to a⋆a^\star. At λg=0.25\lambda_g=0.25, the two means are (0.204,0.216)(0.204,0.216) for α=0\alpha=0, and (0.281,0.294)(0.281,0.294) for α=0.25\alpha=0.25, compared with (0.676,0.520)(0.676,0.520) for PPO. At this common weight (λg=0.25\lambda_g=0.25), both the logarithmic endpoint (α=0\alpha=0) and the power transformation (α=0.25\alpha=0.25) yield mean bets within 0.050.05 of the growth-optimal fraction a⋆=0.25a^\star=0.25 in each state. Thus, for suitable growth weights, Box–Cox PPO approximates the growth-optimal bet with both α=0\alpha=0 and α=0.25\alpha=0.25.

Reviewer_hU76OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies time-average growth, a new perspective into the evaluation of RL policies. For this purpose, it introduces a singed Box-Cox transformation of cumulative returns, and uses the transformed increment as an auxiliary actor signal of PPO. Preliminary theoretical analysis and empirical evaluation results are presented to justify the method.

Critical strengths
  1. The actor-only modification of PPO is simply and straight-forward, preserving most known structures of PPO and thus making the proposed method relatively easy to understand and implement.
  2. Section 4.2 provides an explanation of the Box-Cox transformation, connecting it to explicit return dynamics and thus providing a hint of theoretical motivation on using the transformation.
Critical weaknesses
  1. The research topic of this paper seems under-motivated. It is unclear why we should be interested in the reward increments, why introducing a transformation uu is necessary, and why Box-Cox transformation is a good candidate for uu. All the technical innovations seem to be coming out of nowhere.
  2. The experiments need better design to support the claims.
    • Conventional episodic returns are reported, which is inconsistent with the "long-run growth" motivation.
    • The setting in Section 5 (1.5a1.5a and 0.6a0.6a multipliers) is inconsistent with the analysis in Appendix B ((1+0.5a)(1+0.5a) and (1−0.4a)(1-0.4a) multipliers), making it hard to make sense of.
    • Ablation studies are missing. Comparisons with simpler auxiliary reward and/or risk-sensitive variants may help to isolate what the Box-Cox transformed increment signal contributes.
Questions for authors

Why should people be interested in the new setting, especially with the Box-Cox transformation?

3 comments
AuthorsOpenReview ↗

1. Response to Weakness 1 and the Question for Authors: Motivation and Choice of the Box–Cox Transformation

We thank the reviewer for carefully reading the paper and for the positive evaluation of the technical soundness. We will now answer the questions one by one.

For motivation and ergodicity transformations.

Our motivation is that the RL objective is usually the expected value. However, if the return dynamics are non-ergodic, then the ensemble average (expectation) does not capture the performance of individual realizations/trajectories. We will make this distinction explicit through an analytical coin-toss example.

Consider a fair coin toss with a constant betting fraction a∈[0,1]a\in[0,1] and initial wealth W0>0W_0>0. Each toss multiplies wealth by either 1+0.5a1+0.5a or 1−0.4a1-0.4a, with equal probability. With rewards defined as wealth increments, the expected return is

= \mathbb{E}[W_T] = W_0(1+0.05a)^T,$$ which is maximized at $a=1$. However, repeatedly following this policy without resets causes wealth to approach $0$ along almost every trajectory as the number of tosses increases. Meanwhile, expected wealth continues to grow, driven by increasingly rare trajectories with exceptionally large wealth. This illustrates how maximizing expected return can favor a policy that produces poor outcomes on most individual trajectories over long horizons, as described in [4, Fig. 1(a)]. In this example, taking logarithmic increments removes the multiplicative dependence on current wealth and turns compounded wealth changes into additive increments. For a fixed betting fraction, these increments are independent and identically distributed, so their time average converges to their expectation. We can therefore optimize the expected log increment to maximize long-run wealth growth along individual trajectories, obtaining $a=0.25$. This illustrates the role of an ergodicity transformation that lets an expectation-based criterion characterize growth along individual trajectories [1, 2]. This criterion can also be interpreted as expected utility with logarithmic utility; see Puterman [3, Sec. 4.1] for the general expected-utility framework in MDPs. The distinction lies in the motivation for the transformation. Expected utility reflects the decision maker's attitude toward risk, while ergodicity transformations (e.g., the logarithm) follow from the dynamics (e.g., multiplicative dynamics) and capture long-term growth along individual trajectories, without assuming psychological risk aversion. **Transformed increments and choice of Box--Cox.** Following the ergodicity-transformation approach [4], we model the cumulative return $R_t$ as a stochastic process and use \$ g_t^u = u(R_{t+1}) - u(R_t), \$ with \$ \frac{1}{T}\sum_{t=0}^{T-1} g_t^u = \frac{u(R_T)-u(R_0)}{T}. \$ Thus, the increments measure transformed-return changes along each trajectory. Because return dynamics are generally unknown and policy-dependent, we use a signed, shifted Box--Cox family connecting the identity ($\alpha=1$) and signed logarithm ($\alpha=0$), with intermediate power transformations and support for zero and negative returns. Section 4.2 connects $\alpha$ to diffusion coefficients proportional to $(1+|R|)^{1-\alpha}$, together with compatible drift conditions. Related power transformations appear in decision making under risk [5] and in the estimation of ergodicity transformations within the Box--Cox family [6]. We incorporate the resulting increments into the PPO actor while retaining the original critic and GAE. We will revise the Introduction and Section 2 to include the analytical coin-toss example and clarify the motivation for transformed-return increments and the choice of the Box--Cox transformation.
AuthorsOpenReview ↗

2. Response to Weakness 2 about Experiments

Evaluation beyond episodic returns. We agree that episodic returns alone provide limited evidence about sustained trajectory performance. Therefore, for Ant, Hopper, and Walker2d, we additionally report the fraction of evaluation episodes reaching 1,0001{,}000 steps (the environmental default) as a finite-horizon measure of trajectory persistence. For selected configurations, this fraction increases from 53.6%53.6\% to 97.8%97.8\% in Hopper and from 66.6%66.6\% to 96.6%96.6\% in Walker2d (Table 3, Appendix C.2).

For Atari, episode length complements game score by measuring how long the agent continues playing. We use the interquartile mean (IQM) of the five per-training-seed mean episode lengths to reduce the influence of extreme seed outcomes, and report it relative to the corresponding PPO value. Selected configurations achieve ratios of 1.4101.410 in BeamRider and 1.1211.121 in Qbert. Full results and definitions are provided in Appendix C.2.

We also evaluate the method on a 30-stock FinRL task [7] over 2025, where gains and losses depend on the capital invested and trading decisions, while wealth changes affect the capital available for subsequent investment. Across five training seeds, the reported BoxCoxPPO configuration achieves a mean return of 22.82%22.82\% (standard deviation (Std) 6.686.68), compared with 7.21%7.21\% (Std 15.0415.04) for the PPO baseline. We provide the setup, full results, and portfolio-return curves in Appendix D.

Coin-toss dynamics. Thank you for indicating the typos. We corrected the Section 5 multipliers to 1+0.5a1+0.5a and 1−0.4a1-0.4a in the revised manuscript, as stated in Appendix B and used in the implementation. This does not affect the experimental results.

Auxiliary-reward and risk-sensitive controls. Our parameter sweep includes an auxiliary-reward control: at α=1\alpha=1, U1(R)=RU_1(R)=R and gt=rtg_t=r_t. In Hopper at λg=0.10\lambda_g=0.10, mean returns are 2892.22892.2 for α=1\alpha=1 and 3476.63476.6 for α=0.25\alpha=0.25, compared with 2915.82915.8 for the PPO baseline (Table 6 in the appendix). This comparison supports a contribution from the nonlinear transformation beyond adding the original reward.

Our transformation is motivated by return dynamics and long-term growth along individual trajectories, rather than by prescribing a degree of risk aversion. Baumann et al. [4] discuss this distinction and the connections between the two perspectives.

To address the reviewer's suggestion, we additionally evaluate mean--variance PPO (MV-PPO) on the FinRL task. Across five training seeds, MV-PPO achieves a mean return of 17.33%17.33\% (Std 14.2014.20) at λMV=0.5\lambda_{\mathrm{MV}}=0.5 and 12.57%12.57\% (Std 4.004.00) at λMV=1\lambda_{\mathrm{MV}}=1. The reported BoxCoxPPO configuration achieves a higher mean return (22.82%22.82\% versus 17.33%17.33\%) and a lower across-seed Std (6.686.68 versus 14.2014.20) than MV-PPO with λMV=0.5\lambda_{\mathrm{MV}}=0.5. MV-PPO with λMV=1\lambda_{\mathrm{MV}}=1 achieves a smaller Std (4.004.00), but also a lower mean return (12.57%12.57\%), than the reported BoxCoxPPO configuration. These results illustrate different trade-offs between mean return and variability across training runs.

Full sweeps and portfolio-return curves are provided in Appendix D.3.

References

[1] Peters et al. (2016). Evaluating gambles using dynamics. Chaos: An Interdisciplinary Journal of Nonlinear Science, 26(2).

[2] Peters et al. (2018). The time interpretation of expected utility theory. arXiv preprint arXiv:1801.03680.

[3] Puterman, M. L. (2014). Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons.

[4] Baumann et al. (2025). Reinforcement learning with non-ergodic reward increments: robustness via ergodicity transformations. Transactions on Machine Learning Research.

[5] Price et al. (2023). Distinguishing risk preferences using repeated gambles. arXiv preprint arXiv:2308.07054.

[6] Britto, A. (2026). Likelihood-based ergodicity transformations in time series analysis. arXiv preprint arXiv:2601.11237.

[7] Liu et al. (2021). FinRL: Deep reinforcement learning framework to automate trading in quantitative finance. Proceedings of the Second ACM International Conference on AI in Finance.

Dear Reviewer,

As the discussion period ends soon, we wanted to follow up on our response. If you have any remaining concerns, we would be happy to discuss them further.

Thank you again for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.