- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DD-SRad, which converts a policy's outputs into actions that satisfy per-joint rate and position limits. The allowed range changes with the previous action. The paper proves constraint and coverage properties, then tests the method with SAC and TD3 in MuJoCo and IsaacLab simulations.
Critical strengths
- The action map enforces each joint's command limits without solving an optimization problem.
- The experiments use five seeds. The D-Tanh comparison helps show that the choice of squashing function affects learning even when the allowed action range is the same.
Critical weaknesses
- It is unclear what the direction-dependent radii add over a simpler interval map. One could compute each joint's allowed interval and map into it using its midpoint and half-width. D-Tanh compares squashing functions within DD-SRad, but does not test this simpler alternative.
- Section 3.1 says Post(QP) may violate rate limits because its critic does not account for projection. But projecting into the correct feasible box should enforce the limits regardless of the critic. The paper needs to separate poor learning from violations in the actions actually executed.
- Equations 15-16 need to address position boundaries. At the upper limit, the positive-direction radius is zero, so all positive latent values map to the same action. The map cannot be inverted and log R_eff is undefined. It is unclear how the SAC implementation handles this case.
Questions for authors
Table 4 lists Tanh for the actor mean. If TD3's latent output is therefore limited to [-1,1], the polynomial map can reach only 1/sqrt(2) of each available radius. The full-coverage proof assumes unbounded latent outputs. Which output is bounded in the actual implementation?
2 comments
Follow-up on Our Response
Dear Reviewer,
Thank you again for your thoughtful review and valuable feedback.
As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.
Thank you for your time and consideration!
Best regards, The Authors
Response to Reviewer MB1x
Thank you for suggesting the midpoint control and for the precise questions about projection and reachability. We have completed the requested TD3 comparison on all four environments.
W1. Midpoint/half-width comparison
For previous action p=at−1, define
Li=max(li,pi−δi),Hi=min(hi,pi+δi),mi=(Li+Hi)/2,hiF=(Hi−Li)/2.
We implemented the suggested map as
ψi(ui)=mi+hiF1+ui2ui.
Both maps use the polynomial squash and cover the same feasible interval in closure. Their zero-latent anchors satisfy
ϕi(0)=pi,ψi(0)=mi,mi−pi=2Ri+−Ri−.
For a symmetric interval, the maps coincide. Near a position bound, DD-SRad preserves the previous command as the zero-latent action; the midpoint map shifts it toward the interval center. This directly tests the role of directional anchoring.
Each seed contributes its mean over the final 20 evaluation points; entries are mean ± sample standard deviation across seeds. DD-SRad uses five historical TD3 seeds per environment. Midpoint uses five seeds on Ant/Hopper and three on HalfCheetah/Humanoid, under severe heterogeneous constraints.
DD-SRad achieves higher final means on all four environments. The largest gains are on Hopper (+32.65%) and HalfCheetah (+35.30%), supporting the practical value of the previous-command anchor. Figure R3 provides both learning curves, and the supplementary tables provide the individual seed values.
W2. Projection feasibility
Exact projection onto the current feasible interval is
ΠFt(v)i=min(Hi,max(Li,vi)).
This enforces the modeled rate and position limits regardless of the critic's input. The retained benchmark summaries report Post(QP) CVR as 0.0% at their recorded precision. We will correct Section 3.1 to state this explicitly.
The optimization distinction concerns how the projected action enters critic learning and policy differentiation. Ordinary clipping has zero derivative outside the feasible interval, whereas DD-SRad has a positive derivative on every positive-radius branch at finite nonzero latent values. We will keep the feasibility statement and optimization discussion separate.
W3. Position boundaries and the action density
At the exact upper position bound, Ri+=0 and positive latent values map to that boundary. The action remains feasible; the collapsed stochastic branch contributes a point mass. Eqs. (15)–(16) give the continuous density on positive-radius branches:
logπa(a∣s~)=logπu(u∣s~)−∑ilogRisgn(ui)+23∑ilog(1+ui2).
The stochastic implementation uses an interior margin of 10−6 times the native action range, initializes within it and keeps executed actions strictly inside that support. The directional radii used for density and entropy evaluation are therefore positive. At the latent sign switch, unequal directional radii give one-sided derivatives. We will state these domains alongside the closed-box feasibility theorem.
Q. TD3 output layer and full coverage
The TD3 latent head is linear and unbounded. The standalone actor returns its final linear output; TD3ActorNetwork.forward in algorithms/common/networks.py also returns the final linear layer directly. The action transformation then enforces the bounds.
Table 4's actor-output row will be made backbone-specific: TD3 has an unbounded latent output followed by the feasible-action map. For any normalized interior increment zi∈(−1,1), the inverse is
ui=1−zi2zi.
Every interior action therefore has a finite latent preimage, and interval endpoints are reached in closure. A Tanh-bounded latent vector would give the Ri/2 restriction identified in your question; the implemented linear TD3 head supplies the unbounded domain used in the proof.
Code, comparison tables and Figure R3:
https://anonymous.4open.science/r/DD-SRad-DE86/
https://anonymous.4open.science/r/DD-SRad-DE86/supplementary/rebuttal/README.md