- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper augments PPO with a trajectory-dependent growth signal obtained from increments of a signed Box–Cox transformation of cumulative rewards. The signal is mixed with the conventional advantage for actor updates, while the critic and GAE remain unchanged. The paper characterizes compatible return dynamics and establishes conditional long-run growth and variance-contraction results. Validation includes a multiplicative coin-toss problem, six MuJoCo tasks, and four Atari games, with improvements reported for selected configurations.
Critical strengths
-
The actor-only modification is simple and easy to integrate into existing PPO implementations. Retaining conventional value learning makes the approach practically appealing.
-
The experiments cover both continuous and discrete control, with promising reported improvements on tasks such as BeamRider and Reacher. Providing complete parameter sweeps also helps assess sensitivity and limitations.
Critical weaknesses
-
The practical update is not adequately connected to the proposed objective. Section 4.1 derives a policy gradient using the trajectory-level statistic , but the algorithm replaces it with the immediate increment . Although acknowledged as a local surrogate, this replacement generally omits an action’s influence on future growth. For example, an action yielding an immediate cost followed by a larger delayed reward can receive a negative local growth signal despite improving total transformed return. No conditions or bias analysis establish when this surrogate supports the intended optimization.
-
The reported evaluation statistics are internally inconsistent under the stated definitions. Appendix C.2 defines IQM as the average of the middle three of five per-seed scores. However, the BeamRider baseline reports mean 2919.4, median 2608.0, standard deviation 568.8, and IQM 3144.6. These imply that the two largest scores are at least 4217.8, requiring a population standard deviation of at least 821.2. Thus, these statistics cannot all describe the same five scores. This may reflect a reporting or aggregation error, but it needs resolution before the empirical evidence can be assessed reliably.
Questions for authors
-
What objective does the implemented local growth update optimize? Under what assumptions can replace the trajectory-level growth statistic without losing essential temporal credit assignment?
-
How is the infinite-horizon growth objective reconciled with episodic resets? Without resets, bounded rewards imply that the proposed growth rate vanishes for all policies when .
-
Could you provide the five BeamRider baseline scores and the statistics-generation procedure, reconcile the discrepancy above, and verify the remaining tables?
4 comments
Part 2: Episodic resets, evaluation statistics, and additional experiments
2. Episodic resets and the long-run criterion (Question 2).
As specified in Algorithm 1, each rollout is collected under the current behavior policy and may contain several episodes or episode segments. Cumulative return is tracked within each episode. After including the final reward of an episode, it is reset to its initial value for the next episode. Thus, each raw growth increment is computed from consecutive cumulative returns within the same episode. The reset itself is not counted as a growth increment. We will clarify this distinction between rollouts and episodes in the main text. Growth increments are standardized using the mean and standard deviation of the current rollout, without pooling statistics from previous rollouts since the time-average can change as the policy is updated. This controls the scale of the auxiliary signal under the same policy, while the original GAE retains its minibatch normalization.
Our theoretical analysis does not assume uniformly bounded raw rewards. The coefficient conditions concern and of the transformed dynamics, allowing compatible return processes with unbounded reward increments. For a continuing process with bounded rewards, however, the reviewer is correct that and the criterion vanishes for . The identity endpoint retains the rate for linear accumulation , which our dynamic family covers in Sections 4.2--4.3. Therefore, our method is particularly suited to settings where accumulated return affects subsequent reward magnitudes, such as compounding wealth dynamics. Our parameter sweep also includes , where the auxiliary increment reduces to the original reward, allowing us to compare identity and nonlinear transformations at matched growth weights. We will discuss these intended application settings in Section 4.3 and explicitly highlight the role of the control in Section 5 of the revised manuscript.
In the episodic benchmarks, we use transformed increments as finite-horizon auxiliary feedback. A zero asymptotic limit does not imply that different finite-horizon returns become indistinguishable: at any fixed finite horizon and with a common initial return, applying the same strictly increasing transformation preserves the ordering of trajectory returns [1, Eqs.(5)--(7)]. We will make this distinction in the revised manuscript.
3. Evaluation statistics and additional experiments (Weakness 2 and Question 3).
We thank the reviewer for highlighting the ambiguity in our statistical definitions. The five per-training-seed mean episodic returns for the BeamRider baseline are These give a mean of , a standard deviation of (using divisor ), and an IQM of , matching the reported values. Their median is . The reported median of instead summarizes the 500 pooled episodic returns, with 100 evaluation episodes per training seed. It therefore describes a different aggregation level from the five seed means used for Std and IQM.
We have rechecked the reported statistics against the underlying evaluation data and confirmed that the numerical values are correct. The issue lies in our description of the aggregation levels, rather than in the numerical calculations. We retain the pooled episodic median and explicitly stated the aggregation level of each statistic in the metric definitions and table captions in Appendix C.2 of the revised manuscript.
We have also added Survival Ratio in Appendix C.2 which is the fraction of episodes reaching the episode-length cap in MuJoCo, and the IQM of per-training-seed mean episode lengths relative to PPO in Atari. For example, for the selected Hopper configuration, this fraction increases from for PPO to for BoxCoxPPO. The selected BeamRider configuration achieves a ratio of , corresponding to a increase in this episode-length statistic.
Furthermore, the added Appendix D presents a 30-stock FinRL experiment with compounding portfolio wealth dynamics, including a mean--variance PPO comparison. Across five training seeds, the selected BoxCoxPPO configuration achieves a mean cumulative return of , compared with for PPO, for MV-PPO with , and with . Its standard deviation across training seeds is also lower than PPO's that versus percentage points.
Reference
[1] Peters, O., and Adamou, A. (2018). The time interpretation of expected utility theory. arXiv:1801.03680.
Thank you for the detailed response and additional experiments. Your clarifications have helped address my main concerns, and I have raised my rating accordingly.
Thank you for your engagement during the discussion and for raising your score. We will incorporate the clarifications from our discussion into the final version.
Part 1: Growth objective and local approximation
We thank the reviewer for the careful assessment and for recognizing the simplicity and practical appeal of the actor-only modification. We address the related weaknesses and questions together below.
1. Local growth objective and approximation (Weakness 1 and Question 1).
Our proposed objective combines expected additive return with policy-level growth: Jλg(π)=(1−λg)Jadd(π)+λgGπ. Simply adding the same policy-level growth scalar value to both Qπ and Vπ gives
+\lambda_g G^\pi\bigr] - \bigl[(1-\lambda_g)V_{\mathrm{add}}^\pi(s) +\lambda_g G^\pi\bigr] =(1-\lambda_g)A_{\mathrm{add}}^\pi(s,a).$$ The shared growth term cancels in their advantage difference and provides no action-specific growth signal. We will explain this observation in a short appendix remark. We then use transformed-return increments as auxiliary feedback directly in the actor update. For nonlinear $u$, the same immediate reward can produce different growth signals depending on the return accumulated so far. This allows trajectory information to influence the weighting of actions during policy learning. The implementation uses PPO's clipped surrogate with a mixture of this signal and the original reward-based GAE, while retaining the original critic. Our parameter sweep includes $\alpha=1$, for which $U_1(R)=R$ and the auxiliary increment reduces to the original reward before normalization. This provides a control for adding the original reward as an auxiliary signal. For example, at the same growth weight $\lambda_g=0.10$ in Hopper, the mean return is $2892.2$ for $\alpha=1$ and $3476.6$ for $\alpha=0.25$, compared with $2915.8$ for baseline PPO. We will highlight this comparison in the revised results discussion. We thank the reviewer for highlighting the need to analyze the approximation introduced by using the local increment in the growth component. The newly added Appendix A.4 derives $$\nabla_\theta J_{u,T} = D_{\mathrm{local}}+B_T,$$ where $B_T$ captures the current action's influence on subsequent transformed increments. The local direction equals the full growth gradient if, given the current history, expected future transformed growth does not depend on the current action. More generally, under compatible transformed dynamics, geometric mixing, and uniformly bounded second moments of the policy score, we obtain $$\|B_T\| \leq KC \operatorname{span}(a)\frac{\rho}{1-\rho},$$ with the constants defined in Appendix A.4. For a fixed policy and transformation, this bound does not grow with the horizon. The analysis concerns the growth contribution before normalization and PPO clipping. We will summarize these conditions and results in Section 4.1.