- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes QARPO, a PPO variant that replaces ratio clipping with a quadratic penalty whose per-sample weight depends on the advantage magnitude. As a result, the target policy change grows with the advantage but stays bounded. The authors show that the method matches PPO’s gradient at the old policy and that its surrogate lower-bounds PPO’s. Experiments on MuJoCo and Atari report improvements over PPO and several of its variants.
Critical strengths
None
Critical weaknesses
First, the objective is not motivated. The quadratic penalty is simply posited, not derived from any policy-improvement bound or trust-region argument. Because its weights depend on the advantage, it is no longer a divergence between the new and old policies, unlike the KL term in TRPO or the term in PPO-RPE. It is therefore unclear why it should keep updates proximal. With weights equal to the absolute advantage, the objective appears to reduce to SPO; the contribution then amounts to a reweighting of SPO’s penalty, which is not discussed.
Second, the theory is shallow. The results are elementary properties of a quadratic function: its per-sample maximizer, its gradient and Hessian at the old policy, and a pointwise inequality. None concerns policy improvement, convergence, or performance. The relation to PPO only orders surrogate values and provides no improvement guarantee.
Third, key quantities are not defined:
The advantage estimate (Eq. 3) and its normalized version (Sec. 4.1) are only named. The main text specifies neither the estimator nor the normalization, and it is unclear whether is a deterministic function of or a random sample estimate. The curvature is used in Eqs. (5)–(7) before it is defined in Eq. (12). The analysis treats , and as fixed population quantities. Algorithm 1 instead uses minibatch-normalized advantages and the minibatch scale . Consequently, the objective analyzed in Eq. (13) differs from the one optimized.
Minor: the presentation is verbose and obscures simple ideas, and the number of seeds in the ablations is inconsistent between Sec. 6.2 and App. C.2.
Questions for authors
None
4 comments
W1: Objective Motivation, the Relationship to SPO, and Proximal Constraints
(3) Core distinction from SPO: QARPO changes both the penalty weights and the allocation of target responses across advantage magnitudes. In the general quadratic surrogate framework, the target response magnitude of SPO is independent of nonzero advantage magnitude. Through piecewise curvature design, QARPO instead makes target responses strictly increase with relative advantage magnitude while maintaining a finite upper bound in the high-advantage region. Specifically, shared curvature in the weak-advantage region makes responses approach zero as advantage magnitude decreases; increasing curvature in the high-advantage region preserves strictly increasing responses while preventing unbounded growth. The distinction from SPO thus lies in both the numerical penalty coefficients and the allocation of target responses across advantage samples. Figures 7 and 11 compare actual training performance under different curvature structures and functional forms, respectively, providing experimental support for the mechanisms underlying these design differences. We will further clarify the theoretical connection, structural differences, and corresponding experimental results for SPO and QARPO in the revised manuscript.
W2: Theoretical Contributions and the Connection to Policy-Update Distance
We thank the reviewer for the detailed comments on the theoretical contributions and the scope of the conclusions. We clarify the main roles of the theoretical analysis from two perspectives.
(1) Theoretical contributions: General quadratic-function properties establish the analytical foundation, while the core results concern the structural relationship between advantage responses and curvature allocation. The gradient consistency and Hessian correction in Section 5.1 characterize the common local structure of a class of quadratic penalty objectives: first-order gradient consistency at the old-policy point, with a sample-wise second-order correction that modifies the local optimization geometry. The more direct methodological contribution appears in Section 4, where an analytical mapping between curvature and independent scalar optimal responses is used to derive the QARPO curvature function from response requirements and prove strictly increasing and bounded advantage responses. This analysis also explains the structural differences among quadratic penalty designs including SPO, shared curvature, and QARPO.
(2) Connection to actual policy updates: The new TV upper bound extends the scalar-response analysis to policy distributions. To connect independent scalar target analysis to changes in actual policy distributions, we have added the theoretical result presented in W1(2). This result shows that the curvature lower bounds in QARPO determine the response structure of independent scalar targets and also enable the weighted quadratic penalty to control the average squared TV distance between the new and old policies. Under the additional condition that the population surrogate objective is nondecreasing, we obtain
\left[D_{\mathrm{TV}}\bigl(\pi(\cdot\mid s),\pi_{\mathrm{old}}(\cdot\mid s)\bigr)\right] \leq\frac{\epsilon}{\sqrt{\lambda}}.$$ Unlike an analysis limited to the optimum of a quadratic function, this result directly concerns the distance between normalized policy distributions and explicitly relates $\epsilon$ and $\lambda$ to actual policy-update magnitude. The new analysis therefore extends the theory from response properties of independent scalar targets to conditional control of policy distribution shift, providing a more direct explanation of the proximal update mechanism in QARPO.W3: Definitions of Key Variables and Consistency Between Theory and Implementation
(1) Advantage estimation: The theoretical analysis uses a fixed advantage function, whereas practical training uses random advantage estimates generated from old-policy trajectories. QARPO follows the PPO advantage-estimation procedure and computes sample advantages using generalized advantage estimation (GAE). The experiments use a discount factor of and a GAE parameter of , as specified in Appendix B.1. In practical training, advantage estimates depend on sampled trajectories and the value function, so they are generally random sample estimates and are not uniquely determined by the state–action pair alone. For the theoretical analysis, the old-policy distribution and advantage function are fixed to study how curvature affects target responses and the local optimization structure. In the implementation, raw advantages are computed after trajectory collection and then normalized within each minibatch.
(2) Curvature notation: In Section 4.1, is the curvature variable for the general quadratic surrogate objective, whereas Equation (13) gives the specific QARPO curvature design. Section 4.1 first defines a general quadratic surrogate objective with an arbitrary positive sample-wise curvature coefficient , establishing the common relationship among advantages, curvature, and target responses. Section 4.2 then analyzes different curvature-allocation rules using this relationship. Equation (13) in Section 4.3 subsequently gives the specific curvature function satisfying the QARPO response-design requirements. Equations (6)–(8) and Equation (13) therefore correspond to two levels: the general theoretical analysis and the specific method instantiation.
(3) Theoretical and practical objectives: QARPO follows the same paradigm of population surrogate analysis and stochastic minibatch optimization as PPO and SPO. First, the population objective used in theoretical analysis should be distinguished from the empirical objective used in training. This distinction is not specific to QARPO; it is common to PPO and related methods. For example, the theoretical PPO-Clip surrogate is typically defined as an expectation under the old-policy state–action distribution, whereas practical training constructs an empirical loss from sampled trajectories and optimizes the policy through minibatch updates over multiple epochs. SPO similarly defines the quadratic penalty surrogate under the old-policy distribution and uses sampled advantages and an empirical loss for policy optimization. Population expectations and minibatch training thus correspond to theoretical modeling and numerical implementation, respectively; an exact identity between the two is not required on every finite batch.
QARPO follows this paradigm. Equation (14) defines the population quadratic surrogate objective under the old-policy distribution to analyze the effects of advantage-dependent curvature on target responses and the local optimization structure. In practical training, Algorithm 1 in Appendix B.2 normalizes sampled advantages within each minibatch, computes the minibatch mean absolute advantage , and uses this value to construct sample-wise curvature coefficients. The policy is then updated through probability ratios using a quadratic penalty surrogate of the same form. Equation (14) and Algorithm 1 therefore share the same objective structure and optimization mechanism; the main difference is that the advantage reference scale and curvature coefficients are computed from minibatch data rather than population definitions. Moreover, the core response properties of QARPO follow directly from the algebraic relationship between advantages and curvature. For each minibatch, provided that the normalized advantages and corresponding curvature coefficients are held fixed during the current gradient update, the sample-wise curvature still satisfies:
The structural properties therefore continue to hold, including independent scalar target responses, local gradient properties and Hessian corrections, and pointwise comparisons with the PPO clipping gap. Conclusions involving population policy distributions, such as the new upper bound on average TV distance, must instead be interpreted under the relevant population surrogate objective and the associated assumptions; they are not unconditional guarantees for arbitrary finite-sample gradient steps. QARPO thus follows the same population-modeling and sample-based optimization paradigm as PPO and SPO, while using minibatch advantage statistics for data-adaptive curvature allocation.
W4: Presentation and Seed Counts in the Ablations
We thank the reviewer for the suggestions on clarity and consistency of the experimental settings.
(1) Presentation: We will further emphasize the core reasoning from SPO to QARPO and streamline repeated theoretical explanations. We have added an introduction to SPO in the Preliminaries and highlighted the progression from advantage-response requirements to curvature-function design and policy target responses in the method section. We will also harmonize notation and terminology and reduce repeated explanations of general quadratic-function properties to make the connections among the motivation, core design, and theoretical conclusions more direct.
(2) Seed counts: The discrepancy between Section 6.2 and Appendix C.2 is a textual error in the statistical settings reported in the main text. The main comparisons use 10 independent random seeds per MuJoCo task and 5 per Atari task, with aggregate metrics and 95% confidence intervals. The ablations actually use 5 seeds per MuJoCo task and 3 per Atari task. The statement in Section 6.2 that all ablations used 3 seeds was a textual error; Appendix C.2 describes the settings actually used. We will harmonize the statistical descriptions in the main text, appendix, and relevant captions to ensure that the experimental settings are accurate and consistent.
We thank the reviewer for the thorough and detailed evaluation of the method design, theoretical analysis, and experimental implementation. The comments address the motivation for QARPO, the relationship to SPO, the practical significance of the theoretical results, and the consistency between population-level analysis and sampled implementation. These issues are important for understanding the core contributions and the scope of the theory. We address them point by point below and will further improve the manuscript through the new theoretical analysis and revisions to the main text.
W1: Objective Motivation, the Relationship to SPO, and Proximal Constraints
We thank the reviewer for the comments on the rationale for the QARPO objective, the policy-update constraint mechanism, and the relationship to SPO. We clarify these issues from three perspectives.
(1) Objective motivation: QARPO starts from the quadratic penalty objective of SPO and derives the curvature structure from advantage-response requirements. SPO uses a penalty coefficient proportional to the absolute advantage, so the independent scalar optimal responses for different nonzero advantages have the same displacement scale. In the unified analysis developed in this paper, SPO corresponds to Constant Response (SPO). QARPO further introduces differentiated responses to different advantage magnitudes. Specifically, we first establish an analytical relationship among advantage magnitude, curvature coefficients, and independent scalar optimal responses, and then specify three response-design requirements: responses for weak advantages approach zero as the advantage magnitude tends to zero; different advantage magnitudes retain strict response ordering; and responses under extreme advantages have a finite upper bound. Within the continuous piecewise-affine curvature family considered in the paper, these requirements jointly determine the QARPO curvature structure. QARPO thus extends the quadratic penalty structure of SPO through an explicitly motivated advantage-response design. We have added an introduction to SPO in the Preliminaries of the revised manuscript and further emphasized the progression from SPO to QARPO in Section 4.
(2) Policy-update constraints: The curvature lower bounds in QARPO establish a quantitative relationship between the weighted quadratic penalty and the TV distance between the new and old policies. Because the penalty coefficient depends on the advantage, the weighted quadratic penalty in QARPO generally cannot be directly interpreted as the KL divergence in TRPO or a standard χ2 divergence. The proximal effect can instead be characterized through quantitative control of policy distribution shift. To clarify this relationship, we have added a theoretical analysis relating the QARPO penalty to policy TV distance.
Define the weighted quadratic penalty as
=\mathbb E_{(s,a)\sim\mu_{\mathrm{old}}} \left[c(s,a)(r(s,a)-1)^2\right].$$ For the QARPO curvature design, we have $$c(s,a)\geq\kappa_A,\qquad c(s,a)\geq\lambda|\widehat A(s,a)|,\qquad \kappa_A=\mathbb E_{\mu_{\mathrm{old}}}[|\widehat A|]>0.$$ When the new policy is absolutely continuous with respect to the old policy and the relevant expectations are finite, we can show that $$P_c(\pi;\pi_{\mathrm{old}}) \geq4\kappa_A\mathbb E_{s\sim d_{\mathrm{old}}} \left[D_{\mathrm{TV}}^2\bigl(\pi(\cdot\mid s),\pi_{\mathrm{old}}(\cdot\mid s)\bigr)\right].$$ Furthermore, when the population surrogate objective with fixed advantages and curvature coefficients satisfies $L_c(\pi)\geq L_c(\pi_{\mathrm{old}})$, we obtain the following conditional upper bound on average TV distance: $$\mathbb E_{s\sim d_{\mathrm{old}}} \left[D_{\mathrm{TV}}\bigl(\pi(\cdot\mid s),\pi_{\mathrm{old}}(\cdot\mid s)\bigr)\right] \leq\frac{\epsilon}{\sqrt{\lambda}}.$$ This result shows that, although the weighted quadratic penalty in QARPO is not a standard divergence, it admits a lower bound in terms of the average squared TV distance under the old-policy state distribution. This establishes a connection between penalty strength and actual changes in the policy distribution.