acceptodds
Under review as a conference paper at ICLR 2027

Quadratic Advantage Response Policy Optimization

Abstract

Policy optimization methods such as trust region policy optimization (TRPO) and proximal policy optimization (PPO) improve training stability through policy change constraints. However, their surrogate objectives mainly regulate policy update magnitudes, without explicitly characterizing the relationship between advantage magnitude and policy response. We derive an analytical mapping from sample-wise quadratic curvature to advantage-conditioned responses under a general smooth quadratic probability-ratio surrogate objective. Based on this analysis, we propose Quadratic Advantage Response Policy Optimization (QARPO). QARPO assigns quadratic curvature according to relative advantage magnitude, suppressing response growth under extreme advantages. It preserves response ordering across different advantage magnitudes while retaining a smooth first-order optimization form. Theoretical analysis shows that QARPO preserves first-order gradient consistency at the old-policy point and characterizes the effect of policy normalization on the resulting scalar targets. Meanwhile, its advantage-dependent curvature reshapes the local second-order optimization structure. The quadratic restoration mechanism further provides analytical control of the PPO-Clip clipping gap, establishing an ordering between the smooth quadratic and clipped surrogates. Experiments on continuous-control and discrete-control benchmarks show that QARPO effectively regulates advantage-conditioned responses through advantage-dependent curvature design. Compared with representative policy optimization methods, QARPO achieves response differentiation and competitive performance.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Gradient, Proximal Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5jEkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes QARPO, a PPO variant that replaces ratio clipping with a quadratic penalty whose per-sample weight depends on the advantage magnitude. As a result, the target policy change grows with the advantage but stays bounded. The authors show that the method matches PPO’s gradient at the old policy and that its surrogate lower-bounds PPO’s. Experiments on MuJoCo and Atari report improvements over PPO and several of its variants.

Critical strengths

None

Critical weaknesses

First, the objective is not motivated. The quadratic penalty is simply posited, not derived from any policy-improvement bound or trust-region argument. Because its weights depend on the advantage, it is no longer a divergence between the new and old policies, unlike the KL term in TRPO or the χ2\chi^2 term in PPO-RPE. It is therefore unclear why it should keep updates proximal. With weights equal to the absolute advantage, the objective appears to reduce to SPO; the contribution then amounts to a reweighting of SPO’s penalty, which is not discussed.

Second, the theory is shallow. The results are elementary properties of a quadratic function: its per-sample maximizer, its gradient and Hessian at the old policy, and a pointwise inequality. None concerns policy improvement, convergence, or performance. The relation to PPO only orders surrogate values and provides no improvement guarantee.

Third, key quantities are not defined:

The advantage estimate A^t\hat A_t (Eq. 3) and its normalized version A^i=A^(si,ai)\hat A_i = \hat A(s_i,a_i) (Sec. 4.1) are only named. The main text specifies neither the estimator nor the normalization, and it is unclear whether A^i\hat A_i is a deterministic function of (s,a)(s,a) or a random sample estimate. The curvature cic_i is used in Eqs. (5)–(7) before it is defined in Eq. (12). The analysis treats A^i\hat A_i, cic_i and κA=Eμold[∣A^∣]\kappa_A = \mathbb E_{\mu_{\text{old}}}[|\hat A|] as fixed population quantities. Algorithm 1 instead uses minibatch-normalized advantages A^iM\hat A^{\mathcal M}_i and the minibatch scale κ^A,M\hat\kappa_{A,\mathcal M}. Consequently, the objective analyzed in Eq. (13) differs from the one optimized.

Minor: the presentation is verbose and obscures simple ideas, and the number of seeds in the ablations is inconsistent between Sec. 6.2 and App. C.2.

Questions for authors

None

4 comments

We thank the reviewer for the thorough and detailed evaluation of the method design, theoretical analysis, and experimental implementation. The comments address the motivation for QARPO, the relationship to SPO, the practical significance of the theoretical results, and the consistency between population-level analysis and sampled implementation. These issues are important for understanding the core contributions and the scope of the theory. We address them point by point below and will further improve the manuscript through the new theoretical analysis and revisions to the main text.

W1: Objective Motivation, the Relationship to SPO, and Proximal Constraints

We thank the reviewer for the comments on the rationale for the QARPO objective, the policy-update constraint mechanism, and the relationship to SPO. We clarify these issues from three perspectives.

(1) Objective motivation: QARPO starts from the quadratic penalty objective of SPO and derives the curvature structure from advantage-response requirements. SPO uses a penalty coefficient proportional to the absolute advantage, so the independent scalar optimal responses for different nonzero advantages have the same displacement scale. In the unified analysis developed in this paper, SPO corresponds to Constant Response (SPO). QARPO further introduces differentiated responses to different advantage magnitudes. Specifically, we first establish an analytical relationship among advantage magnitude, curvature coefficients, and independent scalar optimal responses, and then specify three response-design requirements: responses for weak advantages approach zero as the advantage magnitude tends to zero; different advantage magnitudes retain strict response ordering; and responses under extreme advantages have a finite upper bound. Within the continuous piecewise-affine curvature family considered in the paper, these requirements jointly determine the QARPO curvature structure. QARPO thus extends the quadratic penalty structure of SPO through an explicitly motivated advantage-response design. We have added an introduction to SPO in the Preliminaries of the revised manuscript and further emphasized the progression from SPO to QARPO in Section 4.

(2) Policy-update constraints: The curvature lower bounds in QARPO establish a quantitative relationship between the weighted quadratic penalty and the TV distance between the new and old policies. Because the penalty coefficient depends on the advantage, the weighted quadratic penalty in QARPO generally cannot be directly interpreted as the KL divergence in TRPO or a standard χ2\chi^2 divergence. The proximal effect can instead be characterized through quantitative control of policy distribution shift. To clarify this relationship, we have added a theoretical analysis relating the QARPO penalty to policy TV distance.

Define the weighted quadratic penalty as

=\mathbb E_{(s,a)\sim\mu_{\mathrm{old}}} \left[c(s,a)(r(s,a)-1)^2\right].$$ For the QARPO curvature design, we have $$c(s,a)\geq\kappa_A,\qquad c(s,a)\geq\lambda|\widehat A(s,a)|,\qquad \kappa_A=\mathbb E_{\mu_{\mathrm{old}}}[|\widehat A|]>0.$$ When the new policy is absolutely continuous with respect to the old policy and the relevant expectations are finite, we can show that $$P_c(\pi;\pi_{\mathrm{old}}) \geq4\kappa_A\mathbb E_{s\sim d_{\mathrm{old}}} \left[D_{\mathrm{TV}}^2\bigl(\pi(\cdot\mid s),\pi_{\mathrm{old}}(\cdot\mid s)\bigr)\right].$$ Furthermore, when the population surrogate objective with fixed advantages and curvature coefficients satisfies $L_c(\pi)\geq L_c(\pi_{\mathrm{old}})$, we obtain the following conditional upper bound on average TV distance: $$\mathbb E_{s\sim d_{\mathrm{old}}} \left[D_{\mathrm{TV}}\bigl(\pi(\cdot\mid s),\pi_{\mathrm{old}}(\cdot\mid s)\bigr)\right] \leq\frac{\epsilon}{\sqrt{\lambda}}.$$ This result shows that, although the weighted quadratic penalty in QARPO is not a standard divergence, it admits a lower bound in terms of the average squared TV distance under the old-policy state distribution. This establishes a connection between penalty strength and actual changes in the policy distribution.

W1: Objective Motivation, the Relationship to SPO, and Proximal Constraints

(3) Core distinction from SPO: QARPO changes both the penalty weights and the allocation of target responses across advantage magnitudes. In the general quadratic surrogate framework, the target response magnitude of SPO is independent of nonzero advantage magnitude. Through piecewise curvature design, QARPO instead makes target responses strictly increase with relative advantage magnitude while maintaining a finite upper bound in the high-advantage region. Specifically, shared curvature in the weak-advantage region makes responses approach zero as advantage magnitude decreases; increasing curvature in the high-advantage region preserves strictly increasing responses while preventing unbounded growth. The distinction from SPO thus lies in both the numerical penalty coefficients and the allocation of target responses across advantage samples. Figures 7 and 11 compare actual training performance under different curvature structures and functional forms, respectively, providing experimental support for the mechanisms underlying these design differences. We will further clarify the theoretical connection, structural differences, and corresponding experimental results for SPO and QARPO in the revised manuscript.

W2: Theoretical Contributions and the Connection to Policy-Update Distance

We thank the reviewer for the detailed comments on the theoretical contributions and the scope of the conclusions. We clarify the main roles of the theoretical analysis from two perspectives.

(1) Theoretical contributions: General quadratic-function properties establish the analytical foundation, while the core results concern the structural relationship between advantage responses and curvature allocation. The gradient consistency and Hessian correction in Section 5.1 characterize the common local structure of a class of quadratic penalty objectives: first-order gradient consistency at the old-policy point, with a sample-wise second-order correction that modifies the local optimization geometry. The more direct methodological contribution appears in Section 4, where an analytical mapping between curvature and independent scalar optimal responses is used to derive the QARPO curvature function from response requirements and prove strictly increasing and bounded advantage responses. This analysis also explains the structural differences among quadratic penalty designs including SPO, shared curvature, and QARPO.

(2) Connection to actual policy updates: The new TV upper bound extends the scalar-response analysis to policy distributions. To connect independent scalar target analysis to changes in actual policy distributions, we have added the theoretical result presented in W1(2). This result shows that the curvature lower bounds in QARPO determine the response structure of independent scalar targets and also enable the weighted quadratic penalty to control the average squared TV distance between the new and old policies. Under the additional condition that the population surrogate objective is nondecreasing, we obtain

\left[D_{\mathrm{TV}}\bigl(\pi(\cdot\mid s),\pi_{\mathrm{old}}(\cdot\mid s)\bigr)\right] \leq\frac{\epsilon}{\sqrt{\lambda}}.$$ Unlike an analysis limited to the optimum of a quadratic function, this result directly concerns the distance between normalized policy distributions and explicitly relates $\epsilon$ and $\lambda$ to actual policy-update magnitude. The new analysis therefore extends the theory from response properties of independent scalar targets to conditional control of policy distribution shift, providing a more direct explanation of the proximal update mechanism in QARPO.

W3: Definitions of Key Variables and Consistency Between Theory and Implementation

(1) Advantage estimation: The theoretical analysis uses a fixed advantage function, whereas practical training uses random advantage estimates generated from old-policy trajectories. QARPO follows the PPO advantage-estimation procedure and computes sample advantages using generalized advantage estimation (GAE). The experiments use a discount factor of γ=0.99\gamma=0.99 and a GAE parameter of λGAE=0.95\lambda_{\mathrm{GAE}}=0.95, as specified in Appendix B.1. In practical training, advantage estimates depend on sampled trajectories and the value function, so they are generally random sample estimates and are not uniquely determined by the state–action pair (s,a)(s,a) alone. For the theoretical analysis, the old-policy distribution and advantage function are fixed to study how curvature affects target responses and the local optimization structure. In the implementation, raw advantages are computed after trajectory collection and then normalized within each minibatch.

(2) Curvature notation: In Section 4.1, cic_i is the curvature variable for the general quadratic surrogate objective, whereas Equation (13) gives the specific QARPO curvature design. Section 4.1 first defines a general quadratic surrogate objective with an arbitrary positive sample-wise curvature coefficient ci>0c_i>0, establishing the common relationship among advantages, curvature, and target responses. Section 4.2 then analyzes different curvature-allocation rules using this relationship. Equation (13) in Section 4.3 subsequently gives the specific curvature function satisfying the QARPO response-design requirements. Equations (6)–(8) and Equation (13) therefore correspond to two levels: the general theoretical analysis and the specific method instantiation.

(3) Theoretical and practical objectives: QARPO follows the same paradigm of population surrogate analysis and stochastic minibatch optimization as PPO and SPO. First, the population objective used in theoretical analysis should be distinguished from the empirical objective used in training. This distinction is not specific to QARPO; it is common to PPO and related methods. For example, the theoretical PPO-Clip surrogate is typically defined as an expectation under the old-policy state–action distribution, whereas practical training constructs an empirical loss from sampled trajectories and optimizes the policy through minibatch updates over multiple epochs. SPO similarly defines the quadratic penalty surrogate under the old-policy distribution and uses sampled advantages and an empirical loss for policy optimization. Population expectations and minibatch training thus correspond to theoretical modeling and numerical implementation, respectively; an exact identity between the two is not required on every finite batch.

QARPO follows this paradigm. Equation (14) defines the population quadratic surrogate objective under the old-policy distribution to analyze the effects of advantage-dependent curvature on target responses and the local optimization structure. In practical training, Algorithm 1 in Appendix B.2 normalizes sampled advantages within each minibatch, computes the minibatch mean absolute advantage κA,M\kappa_{A,M}, and uses this value to construct sample-wise curvature coefficients. The policy is then updated through probability ratios using a quadratic penalty surrogate of the same form. Equation (14) and Algorithm 1 therefore share the same objective structure and optimization mechanism; the main difference is that the advantage reference scale and curvature coefficients are computed from minibatch data rather than population definitions. Moreover, the core response properties of QARPO follow directly from the algebraic relationship between advantages and curvature. For each minibatch, provided that the normalized advantages and corresponding curvature coefficients are held fixed during the current gradient update, the sample-wise curvature still satisfies:

ciM>0,ciM≥λ∣A^iM∣.c_i^M>0,\qquad c_i^M\geq\lambda|\widehat A_i^M|.

The structural properties therefore continue to hold, including independent scalar target responses, local gradient properties and Hessian corrections, and pointwise comparisons with the PPO clipping gap. Conclusions involving population policy distributions, such as the new upper bound on average TV distance, must instead be interpreted under the relevant population surrogate objective and the associated assumptions; they are not unconditional guarantees for arbitrary finite-sample gradient steps. QARPO thus follows the same population-modeling and sample-based optimization paradigm as PPO and SPO, while using minibatch advantage statistics for data-adaptive curvature allocation.

W4: Presentation and Seed Counts in the Ablations

We thank the reviewer for the suggestions on clarity and consistency of the experimental settings.

(1) Presentation: We will further emphasize the core reasoning from SPO to QARPO and streamline repeated theoretical explanations. We have added an introduction to SPO in the Preliminaries and highlighted the progression from advantage-response requirements to curvature-function design and policy target responses in the method section. We will also harmonize notation and terminology and reduce repeated explanations of general quadratic-function properties to make the connections among the motivation, core design, and theoretical conclusions more direct.

(2) Seed counts: The discrepancy between Section 6.2 and Appendix C.2 is a textual error in the statistical settings reported in the main text. The main comparisons use 10 independent random seeds per MuJoCo task and 5 per Atari task, with aggregate metrics and 95% confidence intervals. The ablations actually use 5 seeds per MuJoCo task and 3 per Atari task. The statement in Section 6.2 that all ablations used 3 seeds was a textual error; Appendix C.2 describes the settings actually used. We will harmonize the statistical descriptions in the main text, appendix, and relevant captions to ensure that the experimental settings are accurate and consistent.

Reviewer_s8c6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes QARPO, an on-policy policy optimization algorithm that extends existing surrogate loss functions by introducing advantage-dependent curvature. The paper provides theoretical results relating the proposed method to PPO, and compares QARPO against PPO and other relevant baselines on 6 MuJoCo tasks and 12 Atari tasks.

Critical strengths
  • [S1]: The paper proposes an intuitive advantage-dependent weighting supported by sound derivations, which strengthens the correlation between advantage magnitude and update magnitude relative to PPO.
  • [S2]: Experiments compare against a range of relevant baselines that consider modifications to PPO.
Critical weaknesses
  • [W1]: There has been a lot of similar research proposing modifications to PPO. The loss used in this work is an extension to the loss function considered in SPO [1] with a different weight function on the quadratic objective. The connection between QARPO and SPO should also be made more clear throughout the paper.
  • [W2]: Experiments only consider 6 MuJoCo tasks and 12 Atari tasks. The paper does not consider more recent relevant applications of PPO such as massively parallel training in Isaac Lab or LLM finetuning.
  • [W3]: The notation could be simplified to make the methodology easier to follow. Currently, the paper introduces a lot of inter-connected notation (cc, α\alpha, ϕ\phi, qq).

References:

[1] Xie et al, “Simple Policy Optimization.” In ICML 2025.

Questions for authors
  • [Q1]: Please demonstrate the impact of QARPO on more recent benchmarks where PPO is currently widely used as the default solution (e.g., Isaac Lab, LLM finetuning).
  • [Q2]: How does the proposed loss function impact the magnitude of policy updates (e.g., TV distance) as a function of ϵ\epsilon?
4 comments

We thank the reviewer for the thorough evaluation and constructive comments. We especially appreciate the recognition of the clear intuition and sound theoretical derivation behind advantage-magnitude-based curvature allocation in QARPO, as well as the comparisons with several related PPO methods. We address the concerns point by point below and will further clarify the main contributions through manuscript revisions and additional experiments.

W1: The Relationship Between QARPO and SPO, and Novelty

We thank the reviewer for the comments on the relationship and differences between QARPO and SPO. We clarify these issues from three perspectives.

(1) Relationship to SPO: SPO is the constant-response special case of the general quadratic surrogate framework in this paper and an important starting point for curvature design in QARPO. SPO and QARPO use the same smooth quadratic penalty objective form, with the main difference lying in sample-wise curvature allocation. Specifically, SPO uses a quadratic penalty coefficient proportional to the absolute advantage, so the independent scalar optimal responses for different nonzero advantages have the same displacement scale. Within the curvature–advantage-response framework developed in this paper, SPO therefore corresponds to Constant Response (SPO): the target response magnitude does not vary with advantage magnitude. Building on this quadratic penalty framework, QARPO studies how curvature allocation can produce controlled variation in target responses across advantage magnitudes. QARPO thus extends the quadratic penalty principle of SPO, shifting the focus from the specific form of the penalty coefficient to the systematic relationship between curvature and advantage responses.

(2) Novelty: The key distinction is that curvature allocation is derived from explicit advantage-response requirements, rather than an arbitrary adjustment to the penalty weights in SPO. We first establish an analytical relationship between quadratic penalty curvature and independent scalar optimal responses, showing that target responses depend on both advantage magnitude and the relative variation between advantage magnitude and the corresponding curvature. Based on this relationship, we specify three requirements: responses for weak advantages approach zero as the advantage magnitude tends to zero; different advantage magnitudes retain strict response ordering; and responses under extreme advantages have a finite upper bound. Within the continuous piecewise-affine curvature family considered in the paper, these requirements jointly determine the curvature structure in QARPO and yield response characteristics that differ from SPO across advantage regions. For weak advantages, QARPO uses shared curvature so that target responses approach zero as advantage magnitudes decrease, whereas SPO assigns the same response scale to different nonzero advantages. For high advantages, QARPO increases curvature with advantage magnitude so that target responses remain strictly increasing and bounded, avoiding the unbounded response growth induced by shared curvature. Thus, SPO produces a constant response independent of advantage magnitude, while QARPO achieves strictly increasing and bounded advantage responses. This distinction changes both the penalty coefficients and the allocation of target responses across advantage samples. In addition, the curvature design in QARPO satisfies explicit lower bounds that allow us to establish a quantitative relationship between the weighted quadratic penalty and average TV distance. This extends the curvature analysis from independent scalar target responses to policy-distribution-level update constraints, further explaining the proximal update mechanism in QARPO.

(3) Presentation in the paper: We have further emphasized the progression from SPO to QARPO and used the existing experiments to illustrate the practical effects of different curvature designs. First, we have added an introduction to the quadratic penalty objective of SPO in the Preliminaries of the revised manuscript to clarify the connection to PPO-Clip and QARPO. Second, the method analysis in Section 4 now more clearly highlights the differences in advantage responses among SPO, shared curvature, and QARPO. We have also clarified the previously unqualified constant-response label by identifying it as Constant Response (SPO), the curvature structure corresponding to SPO. This makes the connection to Figure 2, Figure 7, and the theoretical analysis clearer.

W2/Q1: Scope of Experimental Validation and New Application Settings

We thank the reviewer for the constructive suggestion to further evaluate QARPO in emerging application settings. We explain the experimental design and scope of validation from two perspectives.

(1) Experimental coverage: We have systematically evaluated the effectiveness of QARPO in both continuous and discrete action spaces. We conduct experiments on 6 MuJoCo continuous-control tasks and 12 Atari discrete-control tasks, with 10 and 5 independent random seeds per task, respectively, and report aggregate metrics including Median, IQM, Mean, and POI with confidence intervals. These benchmarks differ in action-space structure and task characteristics: MuJoCo primarily evaluates policy optimization with continuous actions, whereas Atari involves discrete-action decisions from high-dimensional visual observations. The results show competitive aggregate performance for QARPO in both action-space settings. We also use the same default curvature parameter across environments and keep the main training configurations consistent across algorithms to evaluate advantage-dependent curvature design across tasks and action spaces. The experiments therefore provide cross-task evidence for applicability to continuous and discrete policy optimization, beyond performance on a single control task.

(2) Isaac Lab and LLM fine-tuning: Large-scale parallel training and evaluation in new application settings are promising extensions of the research framework. We agree with the suggestion. Large-scale parallel reinforcement learning in Isaac Lab and PPO-based fine-tuning of large language models (LLMs) are important settings for evaluating policy optimization methods across task scales and training paradigms. These settings also introduce training characteristics that differ from the MuJoCo and Atari benchmarks. For example, large-scale parallel environments involve higher-throughput trajectory collection and different optimization configurations, while LLM fine-tuning additionally involves sequence generation, reward modeling, and policy regularization mechanisms. Evaluating QARPO in these settings therefore requires targeted experiments that account for the relevant task characteristics. The experimental conclusions in this paper concern online policy optimization in continuous and discrete action spaces; evaluation in large-scale parallel training and LLM fine-tuning remains a direction for future work. We will clarify this scope in the revised manuscript and explicitly identify Isaac Lab and LLM fine-tuning as important application settings for future investigation.

W3: Key Notation and the Curvature–Response Relationship

We thank the reviewer for the constructive suggestions on readability. The symbols cc, α\alpha, ϕ\phi, and qq describe different stages of the curvature–advantage-response mechanism. Specifically, α\alpha is the relative advantage magnitude normalized by the reference scale and characterizes the advantage strength of a sample; ϕ\phi is the curvature-allocation function that determines relative curvature at different advantage magnitudes; cc is the sample-wise curvature coefficient applied in the quadratic penalty objective; and qq is the dimensionless response function describing the independent scalar target response induced by curvature allocation. The dependencies are explicit: the relative advantage α\alpha is the input, the curvature function ϕ\phi determines the actual penalty coefficient cc, and qq describes the corresponding target response. The structure can therefore be expressed as α=∣A^∣/κA\alpha=|\widehat A|/\kappa_A, c=κAϕ(α)c=\kappa_A\phi(\alpha), and q(α)=α/ϕ(α)q(\alpha)=\alpha/\phi(\alpha). Together, these symbols describe the progression from relative advantage to curvature allocation and target response, providing the basis for a unified analysis of quadratic surrogate objectives including SPO, shared curvature, and QARPO. To simplify the notation, we will further distinguish symbols needed for the theoretical analysis from variables needed in the implementation and emphasize the core design logic.

Q2: The Effect of ϵ\epsilon on Policy-Update Magnitude

We thank the reviewer for the comments on policy-update magnitude in QARPO and the relationship to ϵ\epsilon. We have added theoretical analysis and new experiments to clarify two aspects.

(1) Theoretical effect of ϵ\epsilon: Under explicit conditions, the curvature design in QARPO yields an upper bound on policy TV distance that scales linearly with ϵ\epsilon. In the QARPO objective, ϵ\epsilon determines the response scale of the independent scalar targets and affects actual policy-update magnitude through the quadratic penalty coefficient. To clarify this relationship, we have added a theoretical analysis relating the weighted quadratic penalty in QARPO to policy TV distance. Using the curvature bounds c(s,a)≥κAc(s,a)\geq\kappa_A and c(s,a)≥λ∣A^(s,a)∣c(s,a)\geq\lambda|\widehat A(s,a)|, we can show that, for the population surrogate objective with fixed advantages and curvature coefficients, the following holds when the new policy is absolutely continuous with respect to the old policy, the relevant expectations are finite, and the population surrogate objective is nondecreasing:

\left[D_{\mathrm{TV}}\bigl(\pi(\cdot\mid s),\pi_{\mathrm{old}}(\cdot\mid s)\bigr)\right] \leq\frac{\epsilon}{\sqrt{\lambda}}$$ This result gives an $O(\epsilon)$ upper bound on average TV distance under the stated conditions. With the curvature parameter $\lambda$ fixed, decreasing $\epsilon$ tightens the theoretical update bound and quantifies the relationship between $\epsilon$ and policy distribution shift. The result applies to population policy updates satisfying the relevant conditions and is not an unconditional guarantee for every stochastic gradient update. **(2) Measuring actual update magnitude: We have added KL divergence and mean absolute probability-ratio deviation to examine policy-update behavior under different $\epsilon$ settings.** The theoretical result relates $\epsilon$ to policy-update magnitude, but actual training updates also depend on policy parameterization, gradient optimization, and sampling. We therefore record KL divergence and mean absolute probability-ratio deviation $|r-1|$ in the new $\epsilon$ sensitivity experiments to assess the actual changes between the new and old policies from two complementary perspectives. KL divergence measures the overall difference between the two policy distributions, while mean absolute probability-ratio deviation has a direct mathematical relationship to TV distance: $$\mathbb E_{s\sim d_{\mathrm{old}}} \left[D_{\mathrm{TV}}\bigl(\pi(\cdot\mid s),\pi_{\mathrm{old}}(\cdot\mid s)\bigr)\right] =\frac12\mathbb E_{(s,a)\sim\mu_{\mathrm{old}}}\left[|r(s,a)-1|\right].$$ Mean absolute probability-ratio deviation thus directly reflects average TV distance under the old-policy state distribution, while KL divergence provides a complementary measure of distributional difference. Together with training returns under different $\epsilon$ settings and the newly added KL comparisons, these metrics allow us to examine the relationship between the theoretical update bound and actual policy changes using training data, and to analyze how the advantage-response mechanism in QARPO affects policy optimization performance.

W3: Key Notation and the Curvature–Response Relationship

We thank the reviewer for the constructive suggestions on readability. The symbols cc, α\alpha, ϕ\phi, and qq describe different stages of the curvature–advantage-response mechanism. Specifically, α\alpha is the relative advantage magnitude normalized by the reference scale and characterizes the advantage strength of a sample; ϕ\phi is the curvature-allocation function that determines relative curvature at different advantage magnitudes; cc is the sample-wise curvature coefficient applied in the quadratic penalty objective; and qq is the dimensionless response function describing the independent scalar target response induced by curvature allocation. The dependencies are explicit: the relative advantage α\alpha is the input, the curvature function ϕ\phi determines the actual penalty coefficient cc, and qq describes the corresponding target response. The structure can therefore be expressed as α=∣A^∣/κA\alpha=|\widehat A|/\kappa_A, c=κAϕ(α)c=\kappa_A\phi(\alpha), and q(α)=α/ϕ(α)q(\alpha)=\alpha/\phi(\alpha). Together, these symbols describe the progression from relative advantage to curvature allocation and target response, providing the basis for a unified analysis of quadratic surrogate objectives including SPO, shared curvature, and QARPO. To simplify the notation, we will further distinguish symbols needed for the theoretical analysis from variables needed in the implementation and emphasize the core design logic.

Reviewer_VysFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies smooth quadratic probability-ratio surrogates of the form Lc(θ)=E[riA^i−ci2ϵ(ri−1)2]L_c(\theta)=\mathbb{E}[r_i\hat A_i - \tfrac{c_i}{2\epsilon}(r_i-1)^2] and observes that, treating each ratio as an independent scalar, the maximizer satisfies ∣ri∘−1∣=ϵ∣A^i∣/ci|r_i^\circ-1| = \epsilon|\hat A_i|/c_i. Parameterizing the curvature as ci=κAϕ(αi)c_i=\kappa_A\phi(\alpha_i) with αi=∣A^i∣/κA\alpha_i=|\hat A_i|/\kappa_A and κA=E∣A^∣\kappa_A=\mathbb{E}|\hat A| gives a "response function" q(α)=α/ϕ(α)q(\alpha)=\alpha/\phi(\alpha) that unifies several designs (constant response ≈ PPO's clip scale, shared curvature, min/max bounds). The proposed QARPO uses a piecewise-affine curvature, ϕλ(α)=1\phi_\lambda(\alpha)=1 for α≤1\alpha\le1 and 1+λ(α−1)1+\lambda(\alpha-1) for α>1\alpha>1, which yields a response that is linear for weak advantages, strictly increasing for strong ones, and bounded by 1/λ1/\lambda (2ϵ2\epsilon at the default λ=1/2\lambda=1/2).

Theory: (i) the gradient at θold\theta_{old} equals the unclipped PG gradient and the curvature only adds a negative-semidefinite Hessian correction; (ii) for λ≥1/2\lambda\ge1/2, LQARPO≤LPPO≤LunclipL_{QARPO}\le L_{PPO}\le L_{unclip}; (iii) under policy normalization, fitting the independent targets is equivalent (up to a θ\theta-independent constant) to fitting the normalized targets.

Experiments: 6 MuJoCo tasks (10 seeds) and 12 Atari games (5 seeds) vs. PPO-Clip, PPO-Penalty, PPO-RB, TR-PPO, SPO, reporting POI ≈ 0.71–0.91 and improved IQM/median; ablations (3 seeds) on curvature structure, λ\lambda, ϵ\epsilon, and piecewise vs. globally-affine curvature.

Critical strengths
  1. A clean, unifying lens on quadratic surrogates. Reducing the design space to a single response function q(α)=α/ϕ(α)q(\alpha)=\alpha/\phi(\alpha) (Fig. 2) is a simple but useful way to see PPO-Penalty/SPO-style objectives and the constant-response ("PPO-like") objective as special cases, and it makes the design rationale for Eq. (11)–(12) transparent. This is the main reason I would consider acceptance.

  2. Reasonably thorough main comparison. 6 MuJoCo × 10 seeds and 12 Atari × 5 seeds, rliable-style aggregate metrics with CIs, identical training pipeline across methods, a single λ\lambda across all environments, and anonymous code. The empirical protocol for the headline comparison is above the typical bar for PPO-variant papers.

Critical weaknesses
  1. The hyperparameter-robustness claim is contradicted by the paper's own Figure 9, and the ablations are underpowered. The text states QARPO "remains robust to moderate changes in λ\lambda and ϵ\epsilon", but in Fig. 9 HalfCheetah-v4 reaches ~6000 at λ=0.5\lambda=0.5 versus ~2500 at λ∈{0.25,0.75}\lambda\in\{0.25,0.75\}, and ~6000 at ϵ=0.2\epsilon=0.2 versus ~3000 at ϵ∈{0.1,0.3}\epsilon\in\{0.1,0.3\}; Humanoid drops to ~1500 at ϵ=0.3\epsilon=0.3. That is roughly a 2× swing from a single hyperparameter step, i.e., high sensitivity, not robustness. All ablations use only 3 seeds, so the differences in Fig. 7/10 (and whether the λ=0.5\lambda=0.5 default was effectively selected on the test environments) cannot be assessed. This matters because the method's advantage over SPO (POI 0.72/0.71, with CIs that appear close to 0.5 in Fig. 4) is modest, so the result hinges on whether the default setting is a tuned sweet spot.

  2. Incremental novelty and thin theory relative to the claims; a possible confound in the comparison. Relative to SPO/PPO-Penalty-type quadratic restoration, the contribution is a specific sample-wise curvature schedule. The "theoretical results" are largely immediate: Sec. 5.1 holds for any penalty centered at r=1r=1 (including PPO-Penalty and SPO); Sec. 5.2 shows QARPO is a lower bound of the PPO surrogate, which by itself is not a performance guarantee (PPO's clipped surrogate is itself not a lower bound on the true objective without the KL term), and the justification of λ=1/2\lambda=1/2 as "minimum coverage" rests on the loose bound max⁡(x−ϵ,0)≤x2/4ϵ\max(x-\epsilon,0)\le x^2/4\epsilon. More importantly, with λ=1/2\lambda=1/2 the admissible independent displacement is 2ϵ=0.42\epsilon=0.4, i.e., QARPO at ϵ=0.2\epsilon=0.2 effectively permits twice the ratio excursion of PPO-Clip at ϵ=0.2\epsilon=0.2. Without a PPO-Clip sweep over ϵ\epsilon (e.g., 0.3–0.4) or a matched-KL comparison, it is unclear how much of the gain comes from the advantage-dependent shape versus simply a larger effective trust region.

Questions for authors
  1. Fig. 9: please reconcile the "robust" claim with the ~2× gaps on HalfCheetah/Humanoid. Was λ=0.5\lambda=0.5 (and ϵ=0.2\epsilon=0.2) chosen before or after seeing results on the 6 MuJoCo / 12 Atari tasks?
  2. Please add PPO-Clip with ϵ∈{0.3,0.4}\epsilon\in\{0.3,0.4\} (and/or report realized mean ∣r−1∣|r-1| or KL per update for all methods). Does QARPO's advantage persist when the effective update magnitude is matched?
  3. Fig. 5 says returns are "normalized with PPO-Clip as the reference", yet PPO-Clip's median/IQM read ~0.65 rather than 1.0. What exactly is the normalization (random/PPO min–max? per-seed?)? Fig. 7 also lacks PPO-Clip as a reference curve; the "Constant Response" variant (the quadratic analogue of PPO's scale) sits at ~1000 on Humanoid, far below PPO-Clip's main-table performance — why would the quadratic analogue be so much worse than clipping?
  4. How is κA\kappa_A computed in practice: per rollout batch or per minibatch, before or after the standard advantage standardization, and is the gradient stopped through it? Since A^\hat A is already standardized, κA≈0.8\kappa_A\approx0.8 — is the piecewise breakpoint at ∣A^∣≈0.8|\hat A|\approx0.8 sensitive to this choice?
  5. Can you run the ablations (Figs. 7–10) with ≥5 seeds, or at least report CIs comparable to Fig. 5/6?
  6. Sec. 5.3: the normalization equivalence is stated at the population level with Ea∼πold(⋅∣s)\mathbb{E}_{a\sim\pi_{old}(\cdot|s)} at a fixed state; in practice each state has a single sampled action. What is the practical implication, if any, of the "truncation inactive" condition for the sampled objective?
8 comments

We thank the reviewer for the thorough evaluation and constructive comments. We especially appreciate the recognition of the unified analytical perspective introduced in this paper as an important contribution that merits consideration for acceptance. We address the concerns point by point below and will revise the relevant statements accordingly.

W1/Q1: Hyperparameter Sensitivity and Statistical Reliability

We thank the reviewer for the careful consideration of hyperparameter sensitivity, statistical reliability, and the rationale for the default parameter settings. We clarify these issues from four perspectives.

(1) Hyperparameter robustness: Figure 9 reveals task-dependent sensitivity to curvature allocation and response scale, and we have clarified the scope of the corresponding conclusions accordingly. The substantial performance differences across parameter settings on HalfCheetah-v4 and Humanoid-v4 indicate that different tasks have different requirements for curvature allocation and response scale, and that hyperparameter choices can materially affect final training performance. Based on this observation, we have replaced the broad characterization of hyperparameter robustness in the main text with a description of task-dependent sensitivity. This revision allows Figure 9 to convey more directly how curvature allocation and response scale affect optimization behavior across tasks.

(2) Hyperparameter selection: The default configuration has a theoretical basis and supports comparable experiments; the parameters are not tuned separately to optimize performance on each task. For the ablation parameters λ\lambda and ϵ\epsilon, the choice λ=0.5\lambda=0.5 follows from the theoretical analysis in Section 5.2: within this curvature family, it is the minimum threshold that uniformly covers the PPO clipping gap for all advantage magnitudes. The value ϵ=0.2\epsilon=0.2 was selected based on empirical evaluation and is consistent with the PPO setting. The hyperparameter ablations in Figure 9 therefore analyze how different parameter values affect training performance, rather than selecting an optimal configuration separately for each test task.

(3) Statistical reliability: The main comparisons and ablations serve different purposes, so the statistical settings and evaluation criteria should be distinguished. The main comparisons use 10 seeds per MuJoCo task and 5 seeds per Atari task, with aggregate metrics and 95% confidence intervals. The ablations examine the effects of curvature design and parameter settings; the actual seed counts are 5 for MuJoCo and 3 for Atari. The statement in Section 6.2 that both domains used 3 seeds was a textual error; the ablation settings reported in Appendix C.2 are correct. We will harmonize the statistical descriptions in the main text, captions, and appendix to ensure that the experimental basis is accurate and can be checked.

(4) Comparison with SPO: The advantage of QARPO is reflected in aggregate performance across tasks, rather than the highest return on an individual environment. The probability of improvement (POI) comparison between QARPO and SPO measures improvement across tasks. The POI of QARPO over SPO is 0.72 on MuJoCo and 0.71 on Atari, indicating estimated probabilities of approximately 72% and 71%, respectively, that QARPO achieves a higher return in the corresponding cross-task comparisons. These values do not measure return differences within a single environment. We also report aggregate metrics such as Median, IQM, and Mean, together with per-environment results, to evaluate performance from multiple perspectives. The primary performance assessment of QARPO is therefore based on aggregate results across tasks, rather than a peak return on a single task under a particular default configuration in Figure 9.

W2/Q2: Novelty, Theoretical Contributions, and the Update Mechanism

We thank the reviewer for the detailed comments on novelty, theoretical analysis, and the fairness of the experimental comparisons. We clarify the core contributions of QARPO and the relationship to existing methods from three perspectives.

(1) Novelty: This paper develops a curvature-design framework driven by advantage-response requirements, enabling quadratic penalty objectives to assign differentiated responses to different advantage magnitudes.

First, SPO is a special case of the general quadratic surrogate framework developed in this paper, but SPO and QARPO have different advantage-response properties. SPO uses a quadratic penalty coefficient proportional to the absolute advantage, so the independent scalar optimal responses for different nonzero advantages have the same displacement scale. QARPO instead studies how curvature allocation can make target responses strictly increase with advantage magnitude while preventing unbounded response growth under extreme advantages. This analysis shows that the target response is determined jointly by advantage magnitude and the relative variation between advantage magnitude and the corresponding curvature. Different quadratic surrogate objectives can therefore be analyzed and compared through the induced response functions within a unified framework.

Second, the curvature function in QARPO is derived from explicit response-design requirements, rather than arbitrarily specified sample weights. We seek three properties simultaneously: responses for weak-advantage samples approach zero as the advantage magnitude tends to zero; different advantage magnitudes retain strict response ordering; and responses remain bounded under extreme advantages. Within the class of continuous piecewise-affine curvature functions considered in the paper, these requirements lead to the specific QARPO design.

QARPO therefore achieves strictly increasing and bounded advantage responses, in contrast to the constant response of SPO. This changes both the numerical form of the penalty coefficients and the mechanism for allocating target responses across advantage samples. We have added an introduction to SPO in the Preliminaries of the revised manuscript and explicitly presented the response differences among SPO, shared curvature, and QARPO in Section 4 to make this line of development clearer.

(2) Theoretical contributions: The paper establishes the relationship between advantage responses and curvature allocation and further provides quantitative constraints on actual policy-update distances through curvature.

First, the basic properties of general quadratic surrogate objectives should be distinguished from the properties induced by the specific curvature design in QARPO. Although the gradient consistency and negative-semidefinite Hessian correction in Section 5.1 hold for a class of quadratic penalty objectives, these results primarily establish the local optimization structure. The core methodological contribution is the analytical relationship between curvature and independent scalar optimal responses, from which we derive a strictly increasing and bounded advantage-response mechanism. Section 5.2 additionally establishes the comparison with the PPO clipping gap and provides a theoretical basis for the default choice λ=0.5\lambda=0.5.

To clarify how curvature design affects actual policy updates, we have further added a theoretical analysis relating the weighted quadratic penalty in QARPO to the policy TV distance. The curvature in QARPO satisfies the following lower bounds: c(s,a)≥κA,c(s,a)≥λ∣A^(s,a)∣.c(s,a)\geq\kappa_A,\qquad c(s,a)\geq\lambda|\widehat A(s,a)|. These bounds imply that the weighted quadratic penalty admits a lower bound in terms of the average squared TV distance under the old-policy state distribution. When the population surrogate objective with fixed advantages and curvature coefficients is nondecreasing, and the relevant absolute-continuity and integrability conditions hold, we further obtain:

\left[ D_{\mathrm{TV}}\bigl(\pi(\cdot\mid s),\pi_{\mathrm{old}}(\cdot\mid s)\bigr) \right] \leq\frac{\epsilon}{\sqrt{\lambda}}$$ This result extends the curvature–advantage-response analysis to policy distributions by establishing a quantitative relationship among the curvature parameter, target response scale, and actual policy-update magnitude. Rather than only characterizing the optimum of an independent scalar objective, the result directly describes the distribution shift between the new and old policies and further explains the proximal update mechanism in QARPO. The bound applies to population surrogate objectives satisfying the stated conditions; it does not directly provide an unconditional constraint on stochastic gradient updates or a guarantee of monotonic improvement in actual environment return.

W2/Q2: Novelty, Theoretical Contributions, and the Update Mechanism

(3) Effective policy-update range: The advantage-response mechanism in QARPO differs from simply enlarging the clipping range of PPO-Clip, and the existing curvature ablations provide direct evidence for this design. First, QARPO assigns differentiated target responses according to relative advantage magnitude. With the default λ=0.5\lambda=0.5, the independent scalar target responses have the following properties: the response magnitude is smaller than ϵ\epsilon for weak-advantage samples; it equals ϵ\epsilon at the reference scale; and for high-advantage samples, it increases strictly with a supremum of 2ϵ2\epsilon. QARPO thus uses smaller target responses in the weak-advantage region while allowing larger but bounded target responses in the high-advantage region.

By contrast, increasing ϵ\epsilon in PPO-Clip uniformly changes the clipping thresholds for all nonzero-advantage samples, without inducing the strictly increasing advantage-response relationship of QARPO. Moreover, PPO-Clip produces a flat region in the single-sample objective beyond the clipping threshold, whereas the quadratic surrogate in QARPO has a well-defined independent scalar optimum. Thus, even if the two methods have similar maximum response scales, the optimization structures of the objectives are not equivalent.

Second, the existing ablations examine the practical role of curvature allocation from different perspectives. Figure 7 compares training performance under different curvature structures; Figure 8 further examines the relationship between advantage magnitude and realized responses during training; and Figure 11 compares piecewise-affine and globally affine curvature. In particular, the two curvature designs in Figure 11 have the same functional form in the high-advantage region and differ primarily in the weak-advantage region. This experiment therefore specifically examines the effect of curvature allocation in the weak-advantage region on training performance, providing experimental evidence for the mechanism underlying the piecewise design in QARPO.

Finally, we regard the suggested comparisons with PPO-Clip at ϵ=0.3/0.4\epsilon=0.3/0.4 and with matched actual KL divergence as valuable directions for further quantifying the sources of performance differences among response mechanisms. In the revised manuscript, we have added experiments with PPO-Clip and SPO at ϵ=0.3\epsilon=0.3 and 0.40.4 on 6 MuJoCo environments and 12 Atari environments, compared with QARPO at ϵ=0.2\epsilon=0.2; the advantage of QARPO persists when the effective update magnitudes are matched. In addition, the new bound on average TV distance establishes a theoretical relationship among the curvature parameter, response scale, and policy-update magnitude in QARPO. This result complements the matched-KL comparisons above and further clarifies the advantage-response mechanism in QARPO and the effect on actual policy updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.