acceptodds
Under review as a conference paper at ICLR 2027

Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control

Abstract

Reward shaping is fundamental to modern robotic control with deep reinforcement learning (RL), yet practitioners still rely heavily on heuristic principles borrowed from classical optimal control and trajectory optimization. Existing methods rarely distinguish reward terms that are intrinsic to the control objective from numerical regularizers, leading to brittle hyperparameter tuning. To determine which quantities a reward must contain, we study the stabilization control problem with a focus on zeroth-order (configuration) and first-order (velocity) information. We theoretically and empirically demonstrate that policy gradient methods can successfully solve stabilization tasks without first-order reward terms, adding such terms can instead introduce severe sensitivity as their scale grows. Conversely, our findings confirm that reward functions must be zeroth-order complete over goal-relevant coordinates, while the first-order state remains necessary in the policy observation under our low-dissipation assumptions. Overall, these results provide actionable and principled guidance for reward design in robotic RL.

Primary area:
reinforcement learning
Keywords:
Policy Gradient, Deep RL, Reward Shaping, Robot Learning, Stabilization Control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RSVoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper provides theoretical and empirical evidence that velocity-based reward terms are unnecessary for policy-gradient RL on stabilization control tasks. The authors show that learning performance can be sensitive to the coefficient of the velocity-based term in the reward function, and the PPO can learn stabilization control tasks without the velocity-based reward term with minimal or no performance degradation.

Critical strengths

The illustrative example in Figure 1 clearly demonstrates the sensitivity of performance to the coefficient of the velocity-based reward term. I appreciate the details of the reward functions of all tested tasks in table 1. Results in table 2 and Figure 5 provide insight why capping the value of the velocity-based term can recover some loss of performance, particularly for large coefficient values.

Critical weaknesses

The second part of section 4 provides theoretical explanation why velocity-based terms remain necessary in the agent’s observation. However, the explanation in lines 216–255 is unclear, hard to understand, and the writing clarity can be improved significantly. The following are a list of unclear technical phrases used in the explanation that have not been defined anywhere: “canonical phase-space state vector, dynamics seen from the policy action interface, dissipative force outside the policy, divergence in canonical phase space, force field, contract phase-space volume, full-state stabilizability, asymptotic attraction, passive damping, lower-level PD control, dissipative contact, dominance condition, structural obstruction, zeroth-order actor.”

Since Figure 3 shows the results of an ablation, the plot should include velocity-observation results to make comparison with no-velocity observation easier.

It is not clear why changing eq 11 to eq 12 makes it partial. The provided explanation is hard to understand: "Partial explicitly constrains the complete end-effector pose (position and orientation) but not the remaining null-space coordinate. Thus, “Partial” is complete in task space but incomplete in the robot’s generalized configuration."

The abstract sentence “We theoretically and empirically demonstrate that policy gradient methods can successfully solve stabilization tasks without first-order reward terms” is an overclaim. The paper empirically demonstrates only with PPO not all policy gradient methods.

The conclusion sentence “Our work provides a principled framework for reward design in deep reinforcement learning for robotics” is an overclaim. The scope of the work is much narrower.

Questions for authors

No questions.

2 comments

Thank you for the specific list.

Lines 216-255 and the undefined terms. This passage was too compressed. The next version will state the system as an explicit equation, and each term will either be defined at first use or replaced:

TermMeaning
canonical phase-space state vectorx=(q,p)x=(q,p): configuration and momentum p=M(q)q˙p=M(q)\dot q
dynamics seen from the policy action interfaceThe plant between the policy's action and the state, including any fixed low-level controller
dissipative force outside the policyA velocity-dependent force that removes energy and is not produced by the policy, such as cart damping
divergence in canonical phase space∇⋅l=∑i∂li/∂xi\nabla\cdot l=\sum_i\partial l_i/\partial x_i: how fast a small volume of states shrinks or grows along the flow
force fieldThe position-dependent force B(q)g(q)B(q)g(q) from a position-only policy
contract phase-space volumeMake a small set of states shrink over time, which any attracting target requires
full-state stabilizabilitySome feedback from all of (q,p)(q,p) makes the target locally stable. It only excludes impossible tasks and is not used in the proof
asymptotic attractionNearby states stay near the target and converge to it
passive dampingPhysical friction-like force with no control involved
lower-level PD controlA fixed servo below the policy, τ=Kp(qcmd−q)−Kdq˙\tau=K_p(q_{\mathrm{cmd}}-q)-K_d\dot q, whose −Kdq˙-K_d\dot q term supplies damping (Franka)
dissipative contactEnergy lost through foot-ground contact and friction (Humanoid)
dominance conditionAn extra condition for a discounted value function to act as a Lyapunov function
structural obstructionA limit set by the form of the policy, whatever the reward or training
zeroth-order actorA policy that sees only configuration, a=g(q)a=g(q)

The argument in plain words: a position-only policy can push the system with forces that depend on where it is, but it cannot remove energy according to how fast it is moving. Without friction-like forces, a set of nearby states therefore keeps its volume over time, and it cannot collapse onto the target. With damping, the convergence rate is limited by the damping itself.

Fig. 3 should include velocity-observation results. The next version will use paired bars for every task. For the tasks below, dwell in seconds:

TaskFull observationNo velocity
Cartpole7.810.63
Double Cart Pendulum8.880.02
Quadcopter10.090.00
Franka14.8611.54
Humanoid6.745.28

The comparison also shows that dwell understates what these policies do. On Double Cart Pendulum, the no-velocity policy holds the configuration (configuration-only dwell about 9.7 s) and fails only the velocity check. On Franka it keeps the pose (pass rate 0.975) while its velocity pass rate falls from 0.609 to 0.046 between the best and final checkpoints.

Why Eq. 12 is "partial". The Franka arm has 7 joints, while the end-effector pose has 6 degrees of freedom. Fixing the hand's position and orientation therefore still leaves one free direction: the elbow can swing while the hand stays exactly still. Eq. 12 rewards only the hand, so that motion costs nothing and the reward never distinguishes a still arm from a moving one. Eq. 11 also rewards the joint posture q∗q^\ast, so all 7 joints are pinned and elbow motion lowers the reward. "Partial" means the reward constrains the hand but not the whole arm. This matches Fig. 6: Partial holds the pose, but its joint-speed pass rate is 0.088, compared with 0.759 for Full. We will use this explanation and state it as a short proposition.

Abstract and conclusion overclaim. Only PPO was tested. The sentence will read: "We show that a configuration-only reward can be sufficient for stabilization, and test this with PPO on simulated robotic tasks, where large velocity weights can cause collapse." The sufficiency theorem does not depend on the algorithm, but every empirical statement will now say PPO.

Reviewer_EmGLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines reward shaping and argues that, in physical tasks, velocity penalization is appropriate in classical optimal control but can be detrimental in reinforcement learning. The authors also argue that full configuration and velocity information is needed for RL to solve the optimization problem, and that the reward term should be "configuration-complete" to avoid drifts in the configuration space.

The authors show that RL can optimize the control tasks with zero-velocity penalization (but full observation for both actor and critic) in seven Isaac Lab environments, and that zero-velocity penalization causes IPOPT and CasADi to fail on Acrobot and Pendubot.

Critical strengths

Challenging Common Reward-Shaping Practices: Including velocity penalization terms is common. The paper challenges this practice and explains why such practice is not only unnecessary, but can also be harmful to policy optimization. I find both arguments (although overgeneralized; see critical weaknesses) and experiments convincing.

Broad Empirical Evaluation: The authors' empirical analysis is well done, and the results support the theoretical claim well. I particularly appreciated the return tails and MSE diagnostic (Table 2, Figure 5), which show why clipping the velocity term improves RL performance.

Critical weaknesses

Overgeneralization: "Velocity terms are needed in trajectory optimization" (line 064) is false. Dynamic programming-based methods like LQR, iLQR, and DDP essentially work like RL and solve for the optimal action sequence recursively; thus, all the arguments made about RL remain valid for this class of optimizers.

Lack of Novelty Regarding Full-Observation: The fact that we need both configuration and velocity to control a dynamical system is true for both RL and optimal control systems, and is motivated by the fact that dynamics are expressed as x¨=f(x,x˙,u)\ddot{x} = f(x, \dot{x}, u), and, therefore, the Markov policy needs to know xx and x˙\dot{x}, otherwise, the state information becomes ambiguous. The swing-up pendulum is a didactic example where the optimal policy needs to know not only the rotation angle but also the angle's velocity to perform the swings.

Trivial State-Based Reward Cancellation: The "cancellation" mentioned in 204--215 is also a known result in RL, i.e., Q(s,a)=r(s)+γEs′,a′∣s,a[Q(s′,a′)]Q(s, a) = r(s) + \gamma E_{s', a' | s, a}[ Q(s', a')], and thus E[Q(s,a)∇θlog⁡π(a∣s)]=E[r(s)∇θlog⁡π(a∣s)+γEs′,a′∣s,a[Q(s′,a′)∇θlog⁡π(a∣s)]]=γE[Q(s′,a′)∇θlog⁡π(a∣s)]E[Q(s, a) \nabla_\theta\log\pi(a|s)] = E[r(s)\nabla_\theta\log\pi(a|s) + \gamma E_{s', a' | s, a}[ Q(s', a')\nabla_\theta\log\pi(a|s)]] = \gamma E[Q(s', a')\nabla_\theta\log\pi(a|s)] (because the state-dependent reward acts as a baseline subtraction).

Poor Clarity of Exposition. Many parts of the paper are hard to parse, i.e.,

  • 165--170 (which is one of the core arguments that highlights the importance of velocity penalization in IPOPT-like methods, but is exposed very densely)
  • 172--215 seems unnecessarily complicated

Confounded Reward Shaping. It seems that (almost) all configuration-space rewards are bounded by constructions (cosine angle, tanh, logistic function), while the velocity rewards use squared penalization. This makes credit assignment for whether the velocity term is detrimental a bit harder, since these two reward terms are constructed differently.

Questions for authors

Clarification questions

  • what are α\alpha and bb in the humanoid reward, Table 1?

Suggestions for the future

  • J(Θ)J(\Theta) is defined ambiguously in lines 118 and 122.
  • in equation 1, s_0 should not be part of the set of optimization variables
  • 165--170 is a bit too quick and dense, hard to parse.
  • 192--197 is a bit too quick and dense, hard to parse.
  • SuccessRate and Dwell could be defined early on, since they are mentioned in several places before page 6
4 comments

Thank you for the careful review.

Confounded reward shaping. A velocity penalty acts through its size relative to the bounded configuration term, so collapse could come from the square rather than from velocity itself. We ran controls at β=1\beta=1 on one swing-up task and one manipulation task, with 5 seeds each:

  • Bounded velocity term: ev→ctanh⁡(ev/c)e_v\to c\tanh(e_v/c) with c=4c=4, the same bounded family as the configuration terms (Cartpole, Franka).
  • Both terms unbounded: squared angle error on Cartpole and squared position error on Franka, each with the raw squared velocity term.

Results at β=1\beta=1 (5 seeds; dwell in seconds):

TaskRawBounded velocity termUnbounded configuration term
Cartpole0.009.803.56 (2 of 5 seeds succeed)
Franka1.468.411.53

For reference, β=0\beta=0 gives 7.81 / 3.32 on Cartpole and 14.86 / 14.71 on Franka. With the squared position term, Franka at β=0\beta=0 gives 8.78 / 8.18, so that term is a weaker reward even before any velocity penalty is added.

Bounding the velocity term removes the collapse on Cartpole entirely. On Franka it removes it in part, matching our hard cap (8.73) with similar seed variance. Making the configuration term unbounded instead helps on only 2 of 5 Cartpole seeds, and not at all on Franka. The collapse therefore comes from the unbounded size of the velocity penalty relative to the bounded configuration term, not from penalizing velocity as such. We will state the claim that way, and add these controls to the paper.

On Cartpole, the bounded term at β=1\beta=1 even does better than β=0\beta=0, which fits the ripple described in our reply to Reviewer PVAR. Near the target, ctanh⁡(ev/c)≈evc\tanh(e_v/c)\approx e_v, so the term still damps the ripple. Away from it, the cost of motion is at most βc\beta c per step, so reaching the target stays worthwhile.

"Velocity terms are needed in trajectory optimization" (line 064).

We ran control-limited iLQR on the Acrobot and Pendubot problems of Fig. 1 (same dynamics, horizon, torque bound and objective; 32 cold starts per setting). With a suitable velocity weight, iLQR matches IPOPT's objective almost exactly (Acrobot β=0.025\beta=0.025: 4212 vs 4214; Pendubot β=2\beta=2: 3686 vs 3682). At β=0\beta=0, Acrobot ends upright on all 32 samples, so the position-only objective does drive it to the target. But it had not converged after 1500 iterations, and its cost was still about three times IPOPT's (12044 vs 3827). Pendubot converges to a non-reaching local optimum on all 32 (5110 vs 2991). So a position-only cost is a valid specification, as you say, but from a cold start a velocity cost makes these local methods far more reliable.

Your summary reads it as IPOPT failing at β=0\beta=0, and the paper invites that reading. Rechecking the solver logs, IPOPT reported success on all 128 β=0\beta=0 solves for both tasks. On Acrobot, 99% of solves reach the upright configuration, but many any of them miss the velocity check only because of a period-2 ripple. Pendubot stops at a local optimum from a cold start: warm-starting the β=0\beta=0 problem from the β=2\beta=2 solution lowers the cost from 2960-3490 to 1020-1124 (8 samples). So a velocity cost can help a local solver avoid poor optima, but it is not needed to specify the problem. This is a weaker and more accurate claim than ours. The RL-versus-classical contrast will go, and lines 165-170 with it.

Novelty of the full-observation result.

We will keep the following statement: under position-only feedback the closed-loop divergence is exactly −tr(DM−1)-\mathrm{tr}(DM^{-1}) for any feedback, which bounds the decay rate by tr(DM−1)/(2n)\mathrm{tr}(DM^{-1})/(2n), and we measure which tasks this actually constrains. That measurement limits our own claim: it does not cover Double Cart Pendulum, Franka or Humanoid.

α\alpha and bb in Table 1. We apologize for the unclear notation.

  • α(dxy)\alpha(d_{xy}) is the weight of the position term. It equals 3 at all distances, so the dependence on dxyd_{xy} in our notation is unnecessary and will be removed.
  • b(dxy)=1[dxy≤0.55] clip((0.55−dxy)/0.20, 0, 1)b(d_{xy})=\mathbb 1[d_{xy}\le0.55]\,\mathrm{clip}((0.55-d_{xy})/0.20,\,0,\,1) is a braking gate. It is 0 beyond 0.55 m and rises linearly to 1 at 0.35 m, so the standing term estande_{\mathrm{stand}} applies only near the target.

The other constants in that row are sp=2.0s_p=2.0, sω=0.25s_\omega=0.25 and sq=0.1s_q=0.1. rdefault,ZOr_{\mathrm{default,ZO}} is the Isaac Lab default reward with velocity and action terms removed: alive 2.0, plus upright 0.1 when the up-projection exceeds 0.93, plus a heading term with weight 0.5 saturating at projection 0.8, minus the count of joints at their limits.

J(Θ)J(\Theta) at lines 118 and 122. Agreed. It denotes the MDP return on line 118 and the policy-gradient surrogate on line 122. We will use JJ for the return and L\mathcal L for the surrogate, and a different symbol for the constraint Jacobian in Section 4.

Eq. 1. You are right. s0s_0 is given, not optimized. We optimize over s1:Ts_{1:T} and a0:T−1a_{0:T-1} with s0s_0 fixed.

Success Rate and Dwell. We will define these in Section 3, before their first use.

Reviewer_EmGLOpenReview ↗

Thanks for your reply

Thanks, I've read your answer carefully, and I'll take into consideration for the discussion with the other reviewers and AC and for updating my review.

Reviewer_PVAROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The writing of the paper is very unclear. It is not rigorous enough to support any formal claims.

Two examples:

  1. Equation (4) is wrong. r∗r^* is a local maximum, so r(qt+)−r0(qt−)≥r(qt+)−r∗+ϵ0(δ)r(q_t^+) - r_0(q_t^-) \geq r(q_t^+) - r^* + \epsilon_0(\delta). Some of the claims in the paper do not hold because of this.

  2. The main theorem of the paper (Theorem 4.2) is not stated precisely enough to be verifiable. It never states which system the assumptions and theorem concern. If it only concerns (7), this has to be stated explicitly. It is also not obvious that this theorem covers the systems reported in the experiments at all.

Overall, the paper is not very readable. Its main message is also vague and does not appear to be significant.

Critical strengths

The question being asked is of practical relevance.

Critical weaknesses

Soundness of claims is questionable.

Questions for authors
  1. What does "sufficiency" even mean in the title? Please give a precise definition.

  2. Perhaps related to 1), what is the main message of the paper?

  3. Equation (4) is wrong as stated. What claims does it affect precisely? Can it be fixed?

2 comments

Thank you for your review. Your two technical points are valid but both are narrower than the review suggests: Eq. 4 is one inequality that nothing else in the paper depends on, and Theorem 4.2 concerns Eq. 7 and has a complete proof in Appendix F.

Q3. Eq. 4 is wrong. What does it affect, and can it be fixed?

The gap only gives r0(qt+)−r0(qt−)≥r0(qt+)−r∗+ϵ0(δ)r_0(q_t^+)-r_0(q_t^-)\ge r_0(q_t^+)-r^\ast+\epsilon_0(\delta), and r0(qt+)r_0(q_t^+) can lie below r∗r^\ast. For example, with r0(q)=−q2r_0(q)=-q^2, δ=1\delta=1, q+=0.99q^+=0.99 and q−=1.01q^-=1.01, the difference is 0.040.04 while ϵ0(1)=1\epsilon_0(1)=1.

What it affects. Eq. 4 and the sentence after it, and nothing else. No theorem, proof or experiment uses it. It is a motivating step in Section 4.

The fix. Two radii are needed. Let η(δ′)=sup⁡∥q−q∗∥<δ′(r∗−r0(q))\eta(\delta')=\sup_{\lVert q-q^\ast\rVert<\delta'}(r^\ast-r_0(q)). For δ′<δ\delta'<\delta with η(δ′)<ϵ0(δ)\eta(\delta')<\epsilon_0(\delta),

∥q+−q∗∥<δ′, ∥q−−q∗∥≥δ ⟹ r0(q+)−r0(q−)≥ϵ0(δ)−η(δ′)>0.\lVert q^+-q^\ast\rVert<\delta',\ \lVert q^--q^\ast\rVert\ge\delta \ \Longrightarrow\ r_0(q^+)-r_0(q^-)\ge\epsilon_0(\delta)-\eta(\delta')>0.

The two affected sentences then hold with ϵ0(δ)−η(δ′)\epsilon_0(\delta)-\eta(\delta') in place of ϵ0(δ)\epsilon_0(\delta).

Q1. What does "sufficiency" mean?

Actions are held over a control period hh, rewards are read at samples qk=q(kh)q_k=q(kh), vk=q˙(kh)v_k=\dot q(kh), and ∥q¨∥≤L\lVert\ddot q\rVert\le L on the region of interest. Let c(q)=r∗−r0(q)≥0c(q)=r^\ast-r_0(q)\ge0.

Definition. A reward with cost cc is sufficient for stabilization at velocity tolerance εv\varepsilon_v if every admissible trajectory with ∑kc(qk)<∞\sum_k c(q_k)<\infty satisfies qk→q∗q_k\to q^\ast and lim sup⁡k∥vk∥≤εv\limsup_k\lVert v_k\rVert\le\varepsilon_v.

This is a property of the reward and the dynamics, not of an algorithm. In control terms it is a detectability condition. Whether a learner finds such a trajectory is a separate, empirical question.

Theorem. Assume ϵ0(δ)>0\epsilon_0(\delta)>0 for every δ>0\delta>0.

  • (a) If ∥qk−q∗∥≤δ\lVert q_k-q^\ast\rVert\le\delta and ∥qk+m−q∗∥≤δ\lVert q_{k+m}-q^\ast\rVert\le\delta, then ∥vk∥≤2δ/(mh)+Lmh/2\lVert v_k\rVert\le 2\delta/(mh)+Lmh/2.
  • (b) r0r_0 is sufficient at tolerance εv=Lh/2\varepsilon_v=Lh/2.
  • (c) This is tight under exact zero-order hold: qk≡q∗q_k\equiv q^\ast, vk=(−1)kv0v_k=(-1)^kv_0, ∥v0∥=Lh/2\lVert v_0\rVert=Lh/2 is a zero-cost trajectory. This is the sampling zero at z=−1z=-1.
  • (d) For a continuous-time reward, ∫0∞c(q(t)) dt<∞\int_0^\infty c(q(t))\,dt<\infty implies q(t)→q∗q(t)\to q^\ast and v(t)→0v(t)\to0.

Part (a) follows from Taylor's formula with integral remainder, (b) from (a) with m=1m=1, and (d) from Barbalat's lemma.

Q2. What is the main message?

Velocity belongs in the observation but need not be in the reward. In a second-order system, sampled configuration alone already pins velocity down to a ripple of size Lh/2Lh/2. This matters because velocity penalties are a default, hand-weighted term in robotic RL rewards, a practical question you also recognize. On the five tasks below, their weight β\beta shows three regimes under PPO (5 seeds; dwell in seconds):

Taskβ=0\beta=0β=0.01\beta=0.01β=1\beta=1
Cartpole7.819.570.00
Double Cart Pendulum8.888.830.00
Quadcopter10.097.448.78
Franka14.8617.551.46
Humanoid6.745.952.26
  1. β=0\beta=0: on these tasks PPO stabilizes without any velocity reward. A control-rate ripple may remain. Part (c) gives the exact case, and in the simulator it appears as a bang-bang limit cycle with small residual motion.
  2. Small β\beta: it can remove the ripple. It helps on Cartpole and Franka, is similar on Double Cart Pendulum, and is lower but within seed variance on Quadcopter and Humanoid.
  3. Large β\beta: four of the five tasks degrade sharply. For swing-up tasks we can say why: if β\beta exceeds (J0∗−Jstay)/Eδ(J_0^\ast-J_{\mathrm{stay}})/E_\delta, where EδE_\delta is the least discounted velocity cost of entering the δ\delta-ball, then staying put strictly beats every trajectory that reaches the target. The penalty then changes the task rather than regularizing it. Franka instead looks like an optimization failure.

Theorem 4.2 is not stated precisely, and may not cover the experiments?

The theorem concerns Eq. 7, the system displayed immediately before Assumption 4.1, under that assumption. The proof in Appendix F is complete: a converse Lyapunov function gives a compact sublevel set whose boundary flux is strictly negative, which contradicts the divergence theorem because Eq. 8 gives zero divergence. So the statement is verifiable, and we stand by it. Section 5 also says that Franka and Humanoid violate the assumption, so we did not claim it covers every task. But your question showed we had not been precise about the others, so we made the statement stronger and measured the coverage.

The stronger statement allows dissipation:

q˙=∇pH0,p˙=−∇qH0−D(q)M(q)−1p+B(q)a,H0=12p⊤M(q)−1p+U(q),\dot q=\nabla_pH_0,\qquad \dot p=-\nabla_qH_0-D(q)M(q)^{-1}p+B(q)a,\qquad H_0=\tfrac12p^\top M(q)^{-1}p+U(q),

with M≻0M\succ0, D⪰0D\succeq0, and M,U∈C2M,U\in C^2, B∈C1B\in C^1. For a C1C^1 feedback a=g(q)a=g(q) with equilibrium x∗=(q∗,0)x^\ast=(q^\ast,0):

  • (a) The closed-loop divergence is −tr(D(q)M(q)−1)-\mathrm{tr}(D(q)M(q)^{-1}), independent of gg.
  • (b) If D≡0D\equiv0, then x∗x^\ast is not locally asymptotically stable for any gg. This is Theorem 4.2.
  • (c) In general, any exponentially stable x∗x^\ast has decay rate at most tr(D(q∗)M(q∗)−1)/(2n)\mathrm{tr}(D(q^\ast)M(q^\ast)^{-1})/(2n), because the Jacobian eigenvalues sum to the divergence.

Stabilizability by full-state feedback plays no role in the proof. It only lets a failure be attributed to the missing observation.

We then measured from simulator-identified dynamics whether each task satisfies these hypotheses:

TaskDivergence (1/s)Status under the theorem
Acrobot, Pendubot−0.038-0.038Near-conservative. Decay rate at most 0.0095 /s, negligible over a 10 s episode. Covered by (c).
Quadcopternear 0Near-conservative. Decay rate about 0.01 /s or less. Covered by (c).
Cartpole−7.80-7.80Bound (c) is too weak (1.95 /s). A separate argument applies: the s1s^1 coefficient of the closed-loop characteristic polynomial is −d mgl<0-d\,mgl<0 for every position gain, so no position-only linear gain stabilizes it.
Double Cart Pendulum−7.64-7.64Not covered. A stabilizing position-only gain exists (spectral radius 0.985). Its no-velocity result is an empirical observation about PPO, not a consequence of the theorem.
Frankan/aOutside the hypotheses: a PD servo sits below the policy, and our no-velocity observation keeps the last action, so the policy is not memoryless.
Humanoidn/aOutside the hypotheses: contact dynamics.

So the theorem constrains four of the seven tasks directly or through the coefficient argument, and we will state that scope in the paper.

Readability. We accept that Section 4 was too compressed, and the next version will be built around the definition and statements above. Reviewer RSVo's list of undefined terms has been especially useful here, and our reply to them gives each term a plain definition.

We hope this answers your three questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.