acceptodds
Under review as a conference paper at ICLR 2027

Constraint-Enhanced Reinforcement Learning Based on Dynamic Decoupled Spherical Radial Squashing

Abstract

When deploying reinforcement learning policies to physical robots, actuator rate constraints—hard limits on how fast each joint command may change per control step—must be respected. These limits can vary substantially across joints, producing a high-dimensional box in action-increment space. Existing solver-based projections introduce runtime optimization and training–execution inconsistency, whereas single-radius spherical parameterizations inscribe an isotropic ball and increasingly under-cover the true feasible set as dimensionality and rate-limit heterogeneity grow. This paper proposes Dynamic Decoupled Spherical Radial Squashing (DD-SRad), which computes a position-adaptive effective radius independently for each actuator and tightly aligns the closure of its reachable set with the heterogeneous feasible box. For every latent action, DD-SRad deterministically satisfies the per-step rate and position constraints. Its analytic, per-dimension mapping has a diagonally positive Jacobian at finite latent values, polynomial gradient decay, and requires no runtime optimization solver. Across MuJoCo benchmarks, DD-SRad attains the highest return among the constrained methods with zero constraint violations and improves constraint-space utilization by 30%–50% over spherical baselines. High-fidelity IsaacLab experiments on Unitree H1 and G1, using specification-consistent rate limits and native task reward configurations, provide hardware-relevant simulation evidence while not constituting physical-robot validation. Code and rollout videos are available at https://anonymous.4open.science/r/DD-SRad.

Primary area:
reinforcement learning
Keywords:
Constrained Reinforcement Learning, Action-Space Parameterization, Constrained Control, Rate Constraints, Spherical Radical Squashing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_MB1xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DD-SRad, which converts a policy's outputs into actions that satisfy per-joint rate and position limits. The allowed range changes with the previous action. The paper proves constraint and coverage properties, then tests the method with SAC and TD3 in MuJoCo and IsaacLab simulations.

Critical strengths
  1. The action map enforces each joint's command limits without solving an optimization problem.
  2. The experiments use five seeds. The D-Tanh comparison helps show that the choice of squashing function affects learning even when the allowed action range is the same.
Critical weaknesses
  1. It is unclear what the direction-dependent radii add over a simpler interval map. One could compute each joint's allowed interval and map into it using its midpoint and half-width. D-Tanh compares squashing functions within DD-SRad, but does not test this simpler alternative.
  2. Section 3.1 says Post(QP) may violate rate limits because its critic does not account for projection. But projecting into the correct feasible box should enforce the limits regardless of the critic. The paper needs to separate poor learning from violations in the actions actually executed.
  3. Equations 15-16 need to address position boundaries. At the upper limit, the positive-direction radius is zero, so all positive latent values map to the same action. The map cannot be inverted and log R_eff is undefined. It is unclear how the SAC implementation handles this case.
Questions for authors

Table 4 lists Tanh for the actor mean. If TD3's latent output is therefore limited to [-1,1], the polynomial map can reach only 1/sqrt(2) of each available radius. The full-coverage proof assumes unbounded latent outputs. Which output is bounded in the actual implementation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer MB1x

Thank you for suggesting the midpoint control and for the precise questions about projection and reachability. We have completed the requested TD3 comparison on all four environments.

W1. Midpoint/half-width comparison

For previous action p=at−1p=a_{t-1}, define

Li=max⁡(li,pi−δi),Hi=min⁡(hi,pi+δi),mi=(Li+Hi)/2,hiF=(Hi−Li)/2.L_i=\max(l_i,p_i-\delta_i),\qquad H_i=\min(h_i,p_i+\delta_i),\qquad m_i=(L_i+H_i)/2,\qquad h_i^F=(H_i-L_i)/2.

We implemented the suggested map as

ψi(ui)=mi+hiFui1+ui2.\psi_i(u_i)=m_i+h_i^F\frac{u_i}{\sqrt{1+u_i^2}}.

Both maps use the polynomial squash and cover the same feasible interval in closure. Their zero-latent anchors satisfy

ϕi(0)=pi,ψi(0)=mi,mi−pi=Ri+−Ri−2.\phi_i(0)=p_i,\qquad \psi_i(0)=m_i,\qquad m_i-p_i=\frac{R_i^+-R_i^-}{2}.

For a symmetric interval, the maps coincide. Near a position bound, DD-SRad preserves the previous command as the zero-latent action; the midpoint map shifts it toward the interval center. This directly tests the role of directional anchoring.

EnvironmentTD3 DD-SRadMidpoint-PolyRelative mean gain
Ant4344.7 ± 200.34076.5 ± 517.5+6.58%
Hopper3233.9 ± 601.02438.0 ± 556.8+32.65%
HalfCheetah4122.7 ± 591.93047.1 ± 786.8+35.30%
Humanoid5498.9 ± 238.65220.2 ± 229.4+5.34%

Each seed contributes its mean over the final 20 evaluation points; entries are mean ± sample standard deviation across seeds. DD-SRad uses five historical TD3 seeds per environment. Midpoint uses five seeds on Ant/Hopper and three on HalfCheetah/Humanoid, under severe heterogeneous constraints.

DD-SRad achieves higher final means on all four environments. The largest gains are on Hopper (+32.65%) and HalfCheetah (+35.30%), supporting the practical value of the previous-command anchor. Figure R3 provides both learning curves, and the supplementary tables provide the individual seed values.

W2. Projection feasibility

Exact projection onto the current feasible interval is

ΠFt(v)i=min⁡(Hi,max⁡(Li,vi)).\Pi_{\mathcal{F}_t}(v)_i=\min(H_i,\max(L_i,v_i)).

This enforces the modeled rate and position limits regardless of the critic's input. The retained benchmark summaries report Post(QP) CVR as 0.0% at their recorded precision. We will correct Section 3.1 to state this explicitly.

The optimization distinction concerns how the projected action enters critic learning and policy differentiation. Ordinary clipping has zero derivative outside the feasible interval, whereas DD-SRad has a positive derivative on every positive-radius branch at finite nonzero latent values. We will keep the feasibility statement and optimization discussion separate.

W3. Position boundaries and the action density

At the exact upper position bound, Ri+=0R_i^+=0 and positive latent values map to that boundary. The action remains feasible; the collapsed stochastic branch contributes a point mass. Eqs. (15)–(16) give the continuous density on positive-radius branches:

log⁡πa(a∣s~)=log⁡πu(u∣s~)−∑ilog⁡Risgn⁡(ui)+32∑ilog⁡(1+ui2).\log\pi_a(a\mid\tilde{s})=\log\pi_u(u\mid\tilde{s})-\sum_i\log R_i^{\operatorname{sgn}(u_i)}+\frac32\sum_i\log(1+u_i^2).

The stochastic implementation uses an interior margin of 10−610^{-6} times the native action range, initializes within it and keeps executed actions strictly inside that support. The directional radii used for density and entropy evaluation are therefore positive. At the latent sign switch, unequal directional radii give one-sided derivatives. We will state these domains alongside the closed-box feasibility theorem.

Q. TD3 output layer and full coverage

The TD3 latent head is linear and unbounded. The standalone actor returns its final linear output; TD3ActorNetwork.forward in algorithms/common/networks.py also returns the final linear layer directly. The action transformation then enforces the bounds.

Table 4's actor-output row will be made backbone-specific: TD3 has an unbounded latent output followed by the feasible-action map. For any normalized interior increment zi∈(−1,1)z_i\in(-1,1), the inverse is

ui=zi1−zi2.u_i=\frac{z_i}{\sqrt{1-z_i^2}}.

Every interior action therefore has a finite latent preimage, and interval endpoints are reached in closure. A Tanh-bounded latent vector would give the Ri/2R_i/\sqrt{2} restriction identified in your question; the implemented linear TD3 head supplies the unbounded domain used in the proof.

Code, comparison tables and Figure R3:

https://anonymous.4open.science/r/DD-SRad-DE86/

https://anonymous.4open.science/r/DD-SRad-DE86/supplementary/rebuttal/README.md

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.

Thank you for your time and consideration!

Best regards, The Authors

Reviewer_iN5POpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper enforces hard per-joint rate and position limits in off-policy reinforcement learning. Each joint has its own rate limit. The proposed action layer smoothly squashes the policy output for each joint into an allowed step around the previous action, shrinking the step near the joint's position bounds. The authors prove that the resulting actions are always feasible, that each joint's gradient stays nonzero, and that the whole feasible region is reachable. Experiments on MuJoCo with SAC and TD3, and on two Unitree humanoids in IsaacLab, report the highest return among constrained baselines with zero violations.

Critical strengths

A practically relevant problem with a simple, solver-free and deployable layer that plugs into SAC/TD3. The D-Tanh control (same box geometry, different squashing) is a sensible ablation, and the evaluation covers MuJoCo and two humanoids.

Critical weaknesses
  1. Limited novelty and weak gains. Joint by joint, the method is the standard bounded delta-action design with a different squashing function, and its theorems are straightforward consequences. Against the strongest baselines (D-Tanh, BoxPre+), most improvements are not statistically significant, and established alternatives are not compared.

  2. The paper and the code do not match. The released code omits the entropy correction described in the paper, adds undisclosed adaptive and jerk regularizers, uses different rate limits, and lacks baseline implementations. Table 2 also contains duplicated values. It is unclear which algorithm produced the results.

Questions for authors
  1. Which code produced the reported results? Was the change-of-variables correction used? Was the regularization weight fixed or adaptive, and was the jerk penalty active?
  2. Which rate limits were actually used, those in Table 5 or those in the code?
  3. Did all baselines receive the same regularization, the norm rescaling and the final clipping?
  4. Why does Post(QP), which projects onto the feasible set, violate constraints, and how often does it do so?
  5. How does DD-SRad compare with an action-rate penalty plus clipping, and with FlowPG or NFWPO?
3 comments
AuthorsOpenReview ↗

Response to Reviewer iN5P (1/2)

Thank you for recognizing the practical value of the action layer and the D-Tanh control. We have completed two additional comparisons and address the implementation and constraint questions below.

W1. Contribution and additional experimental evidence

The construction combines heterogeneous directional budgets, the previous-command anchor and a polynomial-tail map. These give coordinatewise feasibility, full feasible-box coverage in closure and explicit gradient sensitivity. The MuJoCo D-Tanh control retains the same geometry and tests squashing-function choice; its reported median gain is 6.6%, with positive mean differences in all 16 configurations.

We also implemented the simpler midpoint/half-width polynomial map. For the same feasible interval, its zero-latent action is the midpoint mim_i; DD-SRad's is the previous action pip_i. Their offset is

mi−pi=Ri+−Ri−2.m_i-p_i=\frac{R_i^+-R_i^-}{2}.

The two maps coincide when both directional margins are equal. The comparison therefore tests the anchor under position-induced interval asymmetry.

EnvironmentTD3 DD-SRadMidpoint-PolyRelative mean gain
Ant4344.7 ± 200.34076.5 ± 517.5+6.58%
Hopper3233.9 ± 601.02438.0 ± 556.8+32.65%
HalfCheetah4122.7 ± 591.93047.1 ± 786.8+35.30%
Humanoid5498.9 ± 238.65220.2 ± 229.4+5.34%

Each seed contributes its final-20-evaluation mean; entries are mean ± sample standard deviation across seeds. The DD-SRad reference uses five historical seeds per environment. Midpoint uses five on Ant/Hopper and three on HalfCheetah/Humanoid.

The completed SAC comparison directly tests a dynamic truncated Gaussian on the same moving feasible intervals:

EnvironmentDD-SRadDynamic truncated GaussianRelative mean gain
Ant3818.8 ± 453.53588.1 ± 475.5+6.43%
Hopper3098.9 ± 245.23309.2 ± 80.2-6.36%
HalfCheetah3996.9 ± 58.73576.6 ± 368.8+11.75%
Humanoid6061.1 ± 248.45863.1 ± 304.8+3.38%

SAC uses three seeds per method and environment. Both distributions record zero evaluated rate and position violations at action tolerance 10−610^{-6}. DD-SRad achieves higher final means in three environments and reduces measured update time by 5.3–15.6% and policy-sampling time by 27.3–49.4% across all four. The tables report the observed effects and their seed-level variability.

Q1. Entropy convention, regularization and experimental code

The earlier released SAC/FastSAC paths use latent-Gaussian entropy. The completed additional SAC experiments use the action-space density in Eqs. (15)–(16):

log⁡πa=log⁡πu−∑ilog⁡Risgn⁡(ui)+32∑ilog⁡(1+ui2).\log\pi_a=\log\pi_u-\sum_i\log R_i^{\operatorname{sgn}(u_i)}+\frac32\sum_i\log(1+u_i^2).

Both actor and critic receive (st,at−1)(s_t,a_{t-1}). Entropy gradients include the analytical sign-probability contribution. The common fixed penalty is

0.005d∑i(μi2+σi2).\frac{0.005}{d}\sum_i(\mu_i^2+\sigma_i^2).

Adaptive and jerk terms are disabled in this comparison. The executable files are under experiments/rebuttal/, with the distribution, actor objective and update path contained in each standalone script.

The robot scripts specify their effective objectives directly:

RobotEntropyDD-SRad latent regularizationRobot Tanh regularization
H1Gaussian latent entropyAdaptive, initialized at 0.005; range [0.001, 0.02]Fixed 0.005
G1Gaussian latent entropyFixed 0.05Fixed 0.05

In H1, the replay-only acceleration term is independent of the current actor parameters, so its actor gradient is zero. The adaptive multiplier on the current latent penalty affects the actor gradient and will be specified explicitly. The complete robot training and reward sources are under experiments/isaaclab/benchmarks/.

For the supplied standalone MuJoCo scripts, command-line defaults disable the adaptive and acceleration options. The completed additional SAC comparison uses the fixed protocol stated above.

Code and supplementary results:

https://anonymous.4open.science/r/DD-SRad-DE86/

https://anonymous.4open.science/r/DD-SRad-DE86/supplementary/rebuttal/README.md

AuthorsOpenReview ↗

Response to Reviewer iN5P (2/2)

Q2. Rate vectors and table entries

The completed severe-constraint comparison uses the following action-coordinate budgets:

EnvironmentRate vector
Antfour at 0.2; four at 0.5
Hopper0.2, 0.5, 0.5
HalfCheetahthree at 0.2; three at 0.5
Humanoidsix at 0.8; six at 0.5; five at 0.2

The standalone scripts declare these vectors and record native action bounds. The saved robot records give nineteen H1 actions with a max/min budget ratio of 2.2222 and thirteen G1 actions with a ratio of 4.1494. DD-SRad and the robot Tanh baseline use identical recorded vectors for each robot. G1's final budget is the minimum of its configured rate and 0.8 times the action bound; its smallest recorded budget is 0.0964. We will align the profile tables with the effective vectors and coordinate units.

The duplicated D-Tanh utilization entry is localized: the retained benchmark summary gives Humanoid-SAC 0.199 ± 0.041 and HalfCheetah-SAC 0.662 ± 0.069. The Humanoid cell will be corrected accordingly. The corrected values are also listed in the supplementary result table.

Q3. Regularization, norm rescaling and clipping

The additional SAC pair uses identical network and optimizer settings, fixed Gaussian-parameter regularization, entropy targets and numerical support conventions. The robot settings are listed above. G1 shares the reward implementation and fixed regularization between the two methods; H1 uses the explicitly stated adaptive/fixed choices. The revised protocol will tabulate regularization and execution maps per baseline.

DD-SRad enforces the rate constraint coordinatewise:

∣Δai∣≤Risgn⁡(ui)≤δi.|\Delta a_i|\leq R_i^{\operatorname{sgn}(u_i)}\leq\delta_i.

It follows that

∣Δa∣2≤∣δ∣2,∣Δa∣∞≤max⁡iδi.\\|\Delta a\\|_2\leq\\|\delta\\|_2,\qquad \\|\Delta a\\|_\infty\leq\max_i\delta_i.

Aggregate norm safeguards at these thresholds leave an already feasible increment unchanged in exact arithmetic. The directional map supplies the per-coordinate rate guarantee, and its effective radii also enforce position bounds. The robot Tanh training map is a rate-scaled Tanh followed by position clipping. Its evaluator has been aligned with this trained map, and the corresponding robot rollout entries will be regenerated for revision. The matched dynamic-radius D-Tanh implementation uses the shared ActionDistribution interface in training and playback.

Q4. Post(QP) feasibility and violation frequency

With Li=max⁡(li,pi−δi)L_i=\max(l_i,p_i-\delta_i) and Hi=min⁡(hi,pi+δi)H_i=\min(h_i,p_i+\delta_i), exact projection is

ai=min⁡(Hi,max⁡(Li,vi)).a_i=\min(H_i,\max(L_i,v_i)).

This guarantees executed-action feasibility independently of the critic. The retained benchmark summaries report Post(QP) CVR as 0.0% at their recorded precision. We will correct Section 3.1 accordingly and discuss critic/projection mismatch as an optimization issue. Under ordinary differentiation through clipping, the derivative is zero outside the interval and one inside it; this explains the gradient distinction from a smooth interior parameterization.

Q5. Rate penalties, FlowPG and NFWPO

A rate penalty adds an optimization preference. Projection onto the current feasible box supplies a hard execution guarantee. BoxPre+ is the paper's critic-aware clipping control. Its execution and regularization details will be stated explicitly alongside the direct TG and midpoint comparisons above.

FlowPG learns a feasible-action transformation using normalizing flows. NFWPO uses state-wise Frank-Wolfe improvement and regression-based policy updates. Both are relevant approaches to action-constrained RL and can be considered with the previous action included in the state. DD-SRad exploits the separable moving-box structure to obtain an explicit feasible map and density, with linear cost in action dimension and no learned feasibility model. The additional experiments compare two direct alternatives for this structure: dynamic truncated Gaussian and midpoint polynomial mapping. FlowPG and NFWPO were not evaluated in these experiments.

Code, per-seed tables and Figures R1–R6:

https://anonymous.4open.science/r/DD-SRad-DE86/

https://anonymous.4open.science/r/DD-SRad-DE86/supplementary/rebuttal/README.md

References:

https://papers.neurips.cc/paper_files/paper/2023/hash/3fd9fe8ec6d7238bf71784797399bb61-Abstract-Conference.html

https://proceedings.mlr.press/v161/lin21b.html

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.

Thank you for your time and consideration!

Best regards, The Authors

Reviewer_TeTFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DD-SRad is proposed in this paper. It establishes dynamic decoupled spherical radial squashing. A position-adaptive effective radius is computed independently for each actuator and tightly aligns the closure of its reachable set. Experiments give somewhat gains than compared baselines. However, the code cannot be found in the released github link.

Critical strengths

1.The layout looks good. However, the implementation details are not good.

Critical weaknesses
  1. The novelty is largely dependent on the particular parameterization and optimization settings. It is incremental work, not for ICLR publication.
  2. The improvements shown in the experimental tables do not support the claim of novelty.
  3. The repository is expired. Open source code not found
Questions for authors
  1. What is novel about only computing a position-adaptive effective radius independently for each actuator? This work is not novel enough for ICLR.
  2. Can the authors model the gains from polynomial gradient decay?
  3. How do Eqs. (15)–(16) guarantee zero radius?
  4. Why is no open-source code provided to guarantee a fair comparison with baselines under matched hyperparameters and implementation details?
  5. Can the authors explain more experimental results? The inconsistent G1 rate limits and H1 joint counts are not combined well with your work.
3 comments
AuthorsOpenReview ↗

Response to Reviewer TeTF (1/2)

Thank you for the questions. We have restored anonymous code access and completed comparisons against dynamic truncated Gaussian and midpoint/half-width polynomial mapping.

Q1. Contribution of the parameterization

DD-SRad combines a previous-command anchor with heterogeneous directional radii and a closed-form polynomial-tail transformation. For the feasible interval [Li,Hi][L_i,H_i] and previous command pip_i, its radii are Ri+=Hi−piR_i^+=H_i-p_i and Ri−=pi−LiR_i^-=p_i-L_i.

This construction provides three concrete properties: coordinatewise feasibility, full feasible-box coverage in closure, and polynomially decaying sensitivity to large latent outputs. Relative to a single-radius spherical map, its geometric benefit grows with dimension and heterogeneity. Away from position boundaries,

VDDVSRad=2dΓ(d/2+1)πd/2∏iδi(min⁡iδi)d.\frac{V_{\mathrm{DD}}}{V_{\mathrm{SRad}}}=\frac{2^d\Gamma(d/2+1)}{\pi^{d/2}}\frac{\prod_i\delta_i}{(\min_i\delta_i)^d}.

The midpoint control tests the previous-command anchor directly. Its zero-latent action is mi=(Li+Hi)/2m_i=(L_i+H_i)/2, whereas DD-SRad gives pip_i. The offset is

mi−pi=Ri+−Ri−2.m_i-p_i=\frac{R_i^+-R_i^-}{2}.

Thus, both constructions coincide for symmetric intervals, while DD-SRad preserves the previous command as the zero-latent action when position margins become asymmetric.

Q2. Modeling the gradient effect and testing the alternatives

For a shared latent coordinate and directional radius,

JDD=R(1+u2)−3/2,Jtanh⁡=R,sech⁡2(u).J_{\mathrm{DD}}=R(1+u^2)^{-3/2},\qquad J_{\tanh}=R\\,\operatorname{sech}^2(u).

Their tails decay as O(∣u∣−3)O(|u|^{-3}) and O(e−2∣u∣)O(e^{-2|u|}). A finite-latent bound also makes the sensitivity explicit: if ∣ui∣≤U|u_i|\leq U and all active radii lie in [Rmin⁡,Rmax⁡][R_{\min},R_{\max}] with Rmin⁡>0R_{\min}>0, then, away from the sign-switch points,

σmin⁡(J)≥Rmin⁡(1+U2)3/2,cond⁡2(J)≤Rmax⁡Rmin⁡(1+U2)3/2.\sigma_{\min}(J)\geq\frac{R_{\min}}{(1+U^2)^{3/2}},\qquad \operatorname{cond}_2(J)\leq\frac{R_{\max}}{R_{\min}}(1+U^2)^{3/2}.

These expressions quantify how latent magnitude and directional-radius imbalance affect gradient transmission. Figure R1 presents the analytical comparisons, while Figures R4–R5 report actor-gradient norms and local policy derivatives during training.

The additional experiments compare DD-SRad with two expressive alternatives under severe constraints:

EnvironmentSAC DD-SRadSAC dynamic TGTD3 DD-SRadTD3 Midpoint-Poly
Ant3818.8 ± 453.53588.1 ± 475.54344.7 ± 200.34076.5 ± 517.5
Hopper3098.9 ± 245.23309.2 ± 80.23233.9 ± 601.02438.0 ± 556.8
HalfCheetah3996.9 ± 58.73576.6 ± 368.84122.7 ± 591.93047.1 ± 786.8
Humanoid6061.1 ± 248.45863.1 ± 304.85498.9 ± 238.65220.2 ± 229.4

Entries are mean ± sample standard deviation of seed-level final-20-evaluation averages. SAC uses three seeds per method. TD3 uses five historical DD-SRad seeds per environment; Midpoint-Poly uses five on Ant/Hopper and three on HalfCheetah/Humanoid.

DD-SRad achieves higher final means than Midpoint-Poly on all four environments, with gains of 32.65% on Hopper and 35.30% on HalfCheetah. It also exceeds dynamic TG on three environments and has lower measured SAC update and sampling costs on all four. These comparisons connect the parameterization choices to final return and practical computation cost.

Code and supplementary results:

https://anonymous.4open.science/r/DD-SRad-DE86/

https://anonymous.4open.science/r/DD-SRad-DE86/supplementary/rebuttal/README.md

AuthorsOpenReview ↗

Response to Reviewer TeTF (2/2)

Q3. Zero directional radii

Feasibility follows from the action map. At the exact upper position bound, Ri+=0R_i^+=0 and outward latent outputs map to the boundary action. For stochastic policies, that branch contributes a point mass.

Eqs. (15)–(16) describe continuous densities on positive-radius branches. The stochastic implementation uses an interior margin of 10−610^{-6} times each native action range and keeps execution strictly inside that support, ensuring positive radii for Jacobian and entropy evaluation. We will state the closed-box feasibility domain and positive-radius density domain explicitly.

Q4. Code access and matched settings

The original anonymous link expired. The working repository is:

https://anonymous.4open.science/r/DD-SRad-DE86/

It contains SAC/TD3 implementations, D-Tanh and Midpoint options, the eight standalone SAC comparison scripts, per-seed results and Figures R1–R6. The SAC pair shares architecture, learning rates, budgets, seeds, rate vectors, native rewards, action-space entropy and fixed Gaussian-parameter regularization. Both actor and critic receive the previous executed action.

Q5. Robot action dimensions and effective rate vectors

The saved benchmark records give the following effective policy-action budgets:

RobotControlled actionsMinimum budgetMaximum budgetMax/min ratio
H1190.180.402.2222
G1130.09640.404.1494

DD-SRad and the robot Tanh baseline have identical recorded vectors for each robot. Both available evaluation-record sets agree on these vectors. H1's ten-joint description refers to its lower-body group; the full policy controls nineteen joints. G1 controls twelve leg joints plus the torso.

The G1 budget construction applies

δieff=min⁡(δiconfigured,0.8bi),\delta_i^{\mathrm{eff}}=\min(\delta_i^{\mathrm{configured}},0.8b_i),

where bib_i is the configured action bound. The recorded minimum becomes 0.0964, giving 0.40/0.0964≈4.14940.40/0.0964\approx4.1494. Evaluation reads the final vector from each checkpoint. We will use the explicit action dimensions and κ≈4.15\kappa\approx4.15 label in the revised tables.

These recorded budgets are in policy-action coordinates. For qicmd=qiref+ciaiq_i^{\mathrm{cmd}}=q_i^{\mathrm{ref}}+c_i a_i, their joint-command increments satisfy

∣Δqicmd∣≤∣ci∣δi(a).|\Delta q_i^{\mathrm{cmd}}|\leq |c_i|\delta_i^{(a)}.

The reference adapter records the actual controlled-joint order, action scales, offsets and rate units. The standalone benchmark sources and reward definitions are provided under experiments/isaaclab/benchmarks/.

Code and supplementary results:

https://anonymous.4open.science/r/DD-SRad-DE86/supplementary/rebuttal/README.md

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.

Thank you for your time and consideration!

Best regards, The Authors

Reviewer_Htq6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Dynamic Decoupled Spherical Radial Squashing (DD-SRad), an action parameterization for reinforcement learning with heterogeneous per-dimension action-rate and position constraints. It computes direction-dependent feasible radii from the previous action and applies an elementwise squashing function with polynomially decaying derivatives. The theoretical analysis establishes constraint satisfaction, coverage of the feasible box in closure, and Jacobian and Lipschitz properties. The method is integrated into SAC and TD3 and evaluated on four MuJoCo locomotion benchmarks, with additional FastSAC experiments on Unitree H1 and G1 in IsaacLab. Comparisons include clipping, projection, spherical parameterizations, and a matched D-Tanh ablation. The authors report improved average returns while satisfying the modeled constraints, supported by utilization analyses and regularization sensitivity experiments.

Critical strengths
  1. A lightweight construction with explicit feasibility guarantees. DD-SRad directly encodes heterogeneous action-rate and position bounds through a closed-form, per-dimension mapping. Its reachable-set closure matches the feasible box, avoiding the geometric conservatism of an inscribed spherical parameterization. This is practically useful because it enforces the modeled constraints without runtime optimization.

  2. Relevant ablation helps isolate the design contribution. The D-Tanh comparison preserves the dynamic radii and feasible-set geometry while changing the squashing function. Together with evaluations across multiple environments and actor–critic backbones, this provides evidence beyond comparisons against geometrically restrictive spherical baselines. This strengthens the empirical motivation for the proposed squashing function, although the implementation-consistency concerns limit the conclusions currently supported.

Critical weaknesses
  1. Substantive discrepancies between the stated method and the supplied implementation undermine the experimental evidence. Equations (15)–(16) require a Jacobian correction for the executed-action log probability, but the supplied SAC and FastSAC implementations use the latent Gaussian log probability without this correction. The MuJoCo actor and critic also receive the original observations rather than the augmented state specified in the paper. Separately, the IsaacLab scripts modify reward configurations despite Appendix H stating that native rewards and weights are retained. These discrepancies leave unclear whether the reported results evaluate the stated algorithm and tasks. Resolving them requires identifying the exact experimental implementation and potentially rerunning key comparisons.

  2. The distinct advantage over equally expressive box parameterizations remains insufficiently established. The matched D-Tanh ablation is a relevant comparison: Table 9 reports positive mean-return differences in all 16 configurations, a median improvement of 6.6%, and a significant aggregate Wilcoxon test. These results support a consistent empirical benefit, although only five individual configurations reach significance under the reported Welch tests. What remains insufficiently established is the attribution of this benefit specifically to polynomial gradient decay, rather than latent scaling, entropy treatment, or regularization. This distinction is consequential because the feasible-box coverage itself is shared by simpler dynamic interval mappings, and the implementation discrepancies leave the controlled nature of the comparison uncertain.

Questions for authors
  1. Equations (15)–(16) specify a Jacobian correction, whereas the supplied SAC and FastSAC implementations appear to use only the latent Gaussian log probability. Which objective generated the reported results? If latent-space entropy was intentional, how does this affect the formulation and the comparison with D-Tanh?
  2. The supplied MuJoCo actor and critic appear to receive only the original environment observations, although the formulation augments the state with the previous action. Was this augmentation used in the reported experiments? Please identify the corresponding code implementation.
  3. How can Appendix H’s statement that native rewards and weights are retained be reconciled with the reward overrides in the supplied H1/G1 scripts, particularly the explicit zeroing of reward weights in the G1 configuration? Please provide the effective reward configurations and code version used for Table 3.
  4. Were DD-SRad and D-Tanh evaluated with identical entropy conventions, regularization, and tuning budgets? Given their shared feasible-set geometry, what evidence from the existing experiments distinguishes the proposed polynomial-decay mechanism from differences in latent scaling or optimization settings?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Htq6 (1/2)

Thank you for the implementation-level questions. We have completed a four-environment SAC comparison using executed-action entropy and augmented inputs, and have checked the standalone robot training scripts against their reward and constraint records.

Q1. Entropy objective and Jacobian correction

The supplied standalone training implementations use Gaussian latent entropy. Their actor objective has the form

Lπ=E[αlog⁡πu(u∣s~)−Q(s~,ϕ(u))+λeff∣u∣22/d].\mathcal{L}_{\pi}=\mathbb{E}[\alpha\log\pi_u(u\mid\tilde{s})-Q(\tilde{s},\phi(u))+\lambda_{\mathrm{eff}}\\|u\\|_2^2/d].

Eqs. (15)–(16) give the executed-action density on positive-radius branches:

log⁡πa=log⁡πu−∑ilog⁡Risgn⁡(ui)+32∑ilog⁡(1+ui2).\log\pi_a=\log\pi_u-\sum_i\log R_i^{\operatorname{sgn}(u_i)}+\frac32\sum_i\log(1+u_i^2).

We will state the entropy coordinate system alongside each experimental protocol. The completed additional SAC comparison uses the action-space density above in the actor objective, soft Bellman target and temperature update.

Its entropy gradient also accounts for directional-branch probabilities. With Pi=Φ(μi/σi)P_i=\Phi(\mu_i/\sigma_i),

E[log⁡Risgn⁡(ui)]=Pilog⁡Ri++(1−Pi)log⁡Ri−.\mathbb{E}[\log R_i^{\operatorname{sgn}(u_i)}]=P_i\log R_i^+ +(1-P_i)\log R_i^-.

Differentiating this expectation retains the contribution from changing sign probabilities. Stochastic execution uses an interior margin of 10−610^{-6} times each native action range, ensuring positive radii for continuous density evaluation.

Q2. Previous-action inputs and the completed comparison

In the original MuJoCo implementation, the previous action enters the action map and replay, while actor and critic inputs contain the environment observation. In the completed SAC comparison, both networks explicitly receive (st,at−1)(s_t,a_{t-1}). SAC.select_action and SAC.update in experiments/rebuttal/ construct the current and next augmented inputs and use the corresponding action densities throughout learning.

Both distributions share native rewards, architecture, learning rates, severe rate vectors, training budgets and three seeds. The common fixed Gaussian-parameter penalty is 0.005∑i(μi2+σi2)/d0.005\sum_i(\mu_i^2+\sigma_i^2)/d. Adaptive and jerk terms are disabled.

Entries are mean ± sample SD across training seeds, using each seed's average over the final 20 evaluation points.

EnvironmentDD-SRadDynamic truncated GaussianRelative mean gain
Ant3818.8 ± 453.53588.1 ± 475.5+6.43%
Hopper3098.9 ± 245.23309.2 ± 80.2-6.36%
HalfCheetah3996.9 ± 58.73576.6 ± 368.8+11.75%
Humanoid6061.1 ± 248.45863.1 ± 304.8+3.38%

Both methods record zero evaluated rate and position violations at action tolerance 10−610^{-6}. DD-SRad achieves higher final mean return on three environments, and its measured update and policy-sampling times are lower on all four. These completed runs establish effective learning with the action-density and augmented-state formulation.

Code and supplementary results:

https://anonymous.4open.science/r/DD-SRad-DE86/

https://anonymous.4open.science/r/DD-SRad-DE86/supplementary/rebuttal/README.md

AuthorsOpenReview ↗

Response to Reviewer Htq6 (2/2)

Q3. Effective H1/G1 rewards and executable sources

The standalone G1 scripts first disable the environment reward terms, then instantiate HolosomaRewardComputerV3. Each collection step calls its step() method; the returned reward is stored in replay and accumulated into perf/TAR. The DD-SRad and robot Tanh scripts contain identical reward-computer implementations and default weights.

The G1 reward includes velocity tracking, survival, foot height, pose, gait, foot separation, anti-crossing, torso, ankle and slip terms listed in Appendix H. The action-rate term is

rrate=−0.005cf∑i(ati−at−1i)2,r_{\mathrm{rate}}=-0.005c_f\sum_i(a_t^i-a_{t-1}^i)^2,

and will be included in the complete reward table. H1 uses the configured environment reward plus a 0.1 survival bonus; its configured tracking weights are 2.0 and 1.0, with the remaining effective weights listed in the repository.

The executable sources are under experiments/isaaclab/benchmarks/: train_ddsrad_h1.py, train_ddsrad_g1.py, train_tanh_clip_h1.py and train_tanh_clip_g1.py. The accompanying settings table identifies the reward functions, coefficients and recorded constraint vectors. Appendix H will use these effective task definitions.

Training Return is recorded through perf/TAR. The rollout evaluators export VTE, fall rate, episode length and utilization. The robot Tanh evaluator now uses its trained action map; the corresponding rollout entries will be regenerated for the revision.

Q4. D-Tanh controls and optimization settings

The paper's MuJoCo D-Tanh control is defined using the same dynamic directional radii. At a shared latent coordinate,

JDD=R(1+u2)−3/2,Jtanh⁡=R,sech⁡2(u).J_{\mathrm{DD}}=R(1+u^2)^{-3/2},\qquad J_{\tanh}=R\\,\operatorname{sech}^2(u).

The tails decay polynomially and exponentially, respectively. The reported MuJoCo comparison has positive mean differences in all 16 configurations, with median gain 6.6% and five individual Welch tests reaching the reported threshold. Figures R1, R4 and R5 provide analytical sensitivities, actor-gradient trajectories and local policy derivatives.

The robot Tanh baseline uses rate-scaled Tanh followed by position clipping. G1 shares reward, architecture, update logic, latent entropy and fixed regularization coefficient 0.05 between the two methods. H1 DD-SRad uses adaptive latent regularization initialized at 0.005 and bounded to [0.001, 0.02]; H1 Tanh uses fixed 0.005. We will distinguish these complete robot parameterizations from the dynamic-radius squashing control.

For the matched reference implementation, --method ddsrad and --method dtanh select the same ActionDistribution interface, augmented inputs, action-space entropy and fixed regularization. Training and playback restore the same method from the checkpoint.

Code and results:

https://anonymous.4open.science/r/DD-SRad-DE86/

https://anonymous.4open.science/r/DD-SRad-DE86/supplementary/rebuttal/README.md

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.

Thank you for your time and consideration!

Best regards, The Authors

Reviewer_wV4mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Decoupled Spherical Radial Squashing (DD-SRad), a differentiable action parameterization for reinforcement learning under heterogeneous, state-dependent joint-command rate constraints.

Unlike conventional spherical squashing methods that use a shared radius across action dimensions, DD-SRad introduces independent, position-adaptive effective radii for each dimension. The results show consistent improvements over several constrained-action baselines, particularly under tight heterogeneous constraints.

Critical strengths
  • A principled and useful action parameterization for heterogeneous constraints.
  • A reasonably comprehensive theoretical and experimental evaluation.
Critical weaknesses
  • Insufficient comparison with alternative feasible-action distributions. an important comparison with truncated Gaussian policies is missing.Such a comparison would help establish whether DD-SRad provides a meaningful optimization advantage over alternative approaches that directly model constrained action distributions.
  • The performance improvements are not uniformly substantial across all environments. For example, the improvement on HalfCheetah-SAC is relatively small compared with the reported variability.

Although the authors report favorable aggregate statistical results, the experimental configurations are not necessarily independent. Additional analysis of gradient norms, gradient behavior near saturation, and optimization stability would strengthen the claim that polynomial gradient decay is responsible for the observed improvements.

Questions for authors
  • Why is a truncated Gaussian policy not included as a baseline? Given that the previous action is already incorporated into the augmented state, could the authors implement a truncated Gaussian policy with dynamically recomputed per-joint feasible intervals? How would DD-SRad compare in terms of return, optimization stability, and computational cost?
  • Can the authors provide additional empirical evidence isolating the contribution of polynomial gradient decay? For example, could they report gradient norms and gradient distributions as latent actions approach saturation, and demonstrate how these properties correlate with optimization performance?
  • when the previous action approaches a position boundary, one of the effective radii can approach zero. How does this affect Jacobian computation, log-probability estimation, and entropy optimization?
3 comments
AuthorsOpenReview ↗

Response to Reviewer wV4m (1/2)

Thank you for the constructive suggestions. We have completed the requested dynamic truncated-Gaussian comparison on all four environments, together with gradient diagnostics and computation-time measurements.

Q1. Dynamic truncated Gaussian: return, stability and cost

At every step, the baseline recomputes each coordinate's feasible interval from the previous executed action p=at−1p=a_{t-1}:

Li=max⁡(li,pi−δi),Hi=min⁡(hi,pi+δi).L_i=\max(l_i,p_i-\delta_i),\qquad H_i=\min(h_i,p_i+\delta_i).

It samples a truncated Gaussian within this interval, accounting for the truncation normalizer and interval transformation in the executed-action density. Both methods use SAC with explicit (st,at−1)(s_t,a_{t-1}) actor and critic inputs, native rewards, identical architecture, learning rates, severe rate vectors and training budgets. The three seeds are 12345, 32345 and 52345. Both use action-space entropy, normalized-interval temperature targets and a fixed Gaussian-parameter energy penalty with coefficient 0.005. Adaptive and jerk terms are disabled.

Each seed contributes its average over the final 20 evaluation points. Entries are mean ± sample standard deviation across training seeds; CV is the seed-level coefficient of variation.

EnvironmentDD-SRadDynamic TGRelative mean gainCV, DD / TG (%)
Ant3818.8 ± 453.53588.1 ± 475.5+6.43%11.88 / 13.25
Hopper3098.9 ± 245.23309.2 ± 80.2-6.36%7.91 / 2.42
HalfCheetah3996.9 ± 58.73576.6 ± 368.8+11.75%1.47 / 10.31
Humanoid6061.1 ± 248.45863.1 ± 304.8+3.38%4.10 / 5.20

DD-SRad achieves higher final mean returns and lower seed-level CV on Ant, HalfCheetah and Humanoid. The HalfCheetah result is particularly informative: +11.75% mean return, with CV 1.47% versus 10.31%. TG achieves the higher return on Hopper. Both methods record zero evaluated rate and position violations at an absolute action tolerance of 10−610^{-6}, excluding episode resets.

EnvironmentSAC update, DD / TG (ms)Policy sampling, DD / TG (ms)
Ant22.940 / 24.9530.754 / 1.424
Hopper23.576 / 25.5450.776 / 1.068
HalfCheetah19.706 / 23.3440.639 / 1.262
Humanoid30.285 / 31.9810.624 / 1.051

DD-SRad reduces measured SAC update time by 5.3–15.6% and batch-one policy-sampling time by 27.3–49.4% across the four environments. Sampling time includes policy computation and density evaluation. These results demonstrate competitive feasible-policy learning with consistently lower measured computation cost. Figure R2 provides the learning curves; Figure R6 provides the timing comparison.

Code and supplementary results:

https://anonymous.4open.science/r/DD-SRad-DE86/

https://anonymous.4open.science/r/DD-SRad-DE86/supplementary/rebuttal/README.md

AuthorsOpenReview ↗

Response to Reviewer wV4m (2/2)

Q2. Polynomial decay and the added gradient diagnostics

The geometry-matched MuJoCo D-Tanh control retains the same directional radius RR. Its comparison with DD-SRad gives

JDD(u)=R(1+u2)−3/2,Jtanh⁡(u)=R,sech⁡2(u).J_{\mathrm{DD}}(u)=R(1+u^2)^{-3/2},\qquad J_{\tanh}(u)=R\\,\operatorname{sech}^2(u).

As latent magnitude grows, these decay as O(∣u∣−3)O(|u|^{-3}) and O(e−2∣u∣)O(e^{-2|u|}), respectively. Polynomial decay therefore preserves greater tail sensitivity to large latent outputs. The original D-Tanh ablation reports positive mean-return differences in all 16 configurations, with a median gain of 6.6%.

Figure R1 also gives the complementary comparison at shared action utilization z=∣a−p∣/Rz=|a-p|/R:

JDD/R=(1−z2)3/2,Jtanh⁡/R=1−z2.J_{\mathrm{DD}}/R=(1-z^2)^{3/2},\qquad J_{\tanh}/R=1-z^2.

This identifies the coordinates in which the tail advantage operates. Figures R4 and R5 add pre-clipping actor-gradient norms throughout training and local action derivatives with respect to Gaussian means near interval saturation. The relationship to policy learning follows directly from

∇θ[−Q(s~,ϕ(uθ))]=−(Dθuθ)⊤Jϕ⊤∇aQ.\nabla_\theta[-Q(\tilde{s},\phi(u_\theta))]=-(D_\theta u_\theta)^\top J_\phi^\top\nabla_a Q.

The map Jacobian controls transmission of the critic gradient; the measured actor gradient also includes the latent network, entropy and regularization. We will present the analytical tail result, D-Tanh ablation and measured optimization diagnostics together. Configuration-level aggregate results will be reported as consistency across the tested settings, alongside the individual mean and variability estimates.

Q3. Position boundaries and entropy computation

For a positive directional radius, the action-space density is

log⁡πa=log⁡πu−∑ilog⁡Risgn⁡(ui)+32∑ilog⁡(1+ui2).\log\pi_a=\log\pi_u-\sum_i\log R_i^{\operatorname{sgn}(u_i)}+\frac32\sum_i\log(1+u_i^2).

The stochastic implementation uses an interior margin of 10−610^{-6} times each native action range, initializes within that support and keeps executed floating-point actions strictly inside it. This maintains positive directional radii for density and entropy evaluation.

At an exact physical upper bound, Ri+=0R_i^+=0: outward latent outputs map to the boundary action, preserving feasibility. That branch contributes a point mass under a stochastic policy. We will specify the two mathematical domains explicitly: deterministic feasibility on the closed box, and continuous action densities on positive-radius branches. Entropy gradients account analytically for the probabilities of selecting the positive and negative branches.

Code, final tables and Figures R1–R6:

https://anonymous.4open.science/r/DD-SRad-DE86/supplementary/rebuttal/README.md

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.

Thank you for your time and consideration!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.