- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors first analyze the trade-off between commitment loss and amplification of value-estimation error under hard persistence, and show that a shared holding duration is restricted by a shared-commitment bottleneck in high-dimensional systems with heterogeneous actuators. They then define the feasible control horizon, and prove that bounded action revision extends the feasible horizon beyond hard holding. Building on this, they propose SOPER, which learns action dynamics under a conditional-drift budget. Finally, they discretized this objective into a regularizer with an adaptive Lagrange multiplier integrated into standard actor-critic training. Compared against FlashSAC, TaCoS, and ASAP, the proposed method shows better sample-efficiency on several benchmarks and a real humanoid robot task.
Critical strengths
- The continuous-time trade-off analysis offers a clear theoretical perspective. Proposition 2 characterizes via the feasible control horizon how relaxing the drift budget monotonically extends the feasible horizon, and explains the bottleneck of shared holding durations under heterogeneous high-dimensional actuators. This provides an analytical framework for softening persistence into controlled action revision, and is the paper's most substantive contribution.
- The experimental coverage is broad and the algorithm analysis does demonstrate that, compared with the baseline algorithms, the proposed method exhibits certain favorable properties, such as a broader distribution of action timescales.
Critical weaknesses
-
The ablation study is insufficient. Although the theoretical motivation is sound, apart from augmenting the policy input with the previous action, the only formal difference between SOPER and existing algorithms with action constraints, such as CAPS (Mysore et al., 2021) and ASAP (Kwak & Hwang, 2026), is that fixed coefficients are replaced with adaptive ones. It therefore remains unclear whether the observed performance gains genuinely stem from the theoretical insights derived from continuous-time RL, as the possibility that the improvement is driven by the adaptive weight coefficients has not been ruled out.
-
The final practical algorithm contains no module that allocates the drift budget on a per-actuator basis. Although Figures 4(b) and 4(c) show that the algorithm produces a broader distribution of action timescales, it remains unclear which factor actually gives rise to this distribution.
Questions for authors
- Could additional ablation experiments be provided to verify that the performance gains do not stem solely from the adaptive weight coefficients?
- Could the direct factor responsible for the broader action timescale distribution be further clarified?
- Could further analysis be conducted on the impact of discretization on the theoretical guarantees?
4 comments
Rebuttal (2/2)
W2: The final practical algorithm contains no module that allocates the drift budget on a per-actuator basis. Although Figures 4(b) and 4(c) show that the algorithm produces a broader distribution of action timescales, it remains unclear which factor actually gives rise to this distribution.
Q2: Could the direct factor responsible for the broader action timescale distribution be further clarified?
Reply to W2 & Q2
SoPer permits different actuator rates without explicitly favoring them. The budget constrains the expected discounted average of the total squared revision, . SoPer allocates the revision of different dimension to optimize task reward while satisfying the budget. Therefore it can learns different actuator timescales based on the both task requirement and budget constraint.
Q3: Could further analysis be conducted on the impact of discretization on the theoretical guarantees?
Reply to Q3
Corollary 3 admits an correction. Suppose a budget-feasible controller has task gap at most at each control step. Under Assumption 2, the state moves by at most during the control interval, so the task gap remains at most while the command is held. The discounted return gap of the optimal feasible discrete controller is therefore at most .
We consider short control intervals are practical in real-time robotics e.g. h=10-20 ms. If is also small relative to the task-gap tolerance , discretization adds only a small relative correction to the continuous-time return bound.
Official Comment by Reviewer MrCe
I broadly agree that, compared with existing action smoothing techniques, the overall action-drift budget constraint proposed in this paper offers a more flexible and more effective way of constraining multiple actuators, and that it can effectively give rise to heterogeneous behavioral patterns. However, from the standpoint of the continuous-time RL motivation, there remains a noticeable disconnect between the theory and the actual implementation, and I therefore maintain my rating.
Thanks for your feedback. Below we clarify your main concerns. Please refer to the revised pdf to see our additional experimental results.
Main concerns 1: However, from the standpoint of the continuous-time RL motivation, there remains a noticeable disconnect between the theory and the actual implementation.
Reply to main concerns 1
The continuous-time formulation directly determines SoPer’s nominal displacement cost and regularized critic target. Our new experiments further examine the resulting behavior during stochastic training.
SoPer implements the drift objective through an explicit discretization of continuous-time action dynamics. For control interval , discretizing gives and therefore , with budget (Eqs. 14–16). Feedback at every step is compatible with these dynamics: it determines how to revise the current command within the drift budget.
Figure 13 provides an empirical diagnostic of how SoPer obtains robustness to critic error motivated by our analysis. SoPer learns drift-constrained action dynamics, promoting persistence that can temper the influence of critic errors. To examine this empirically, we save checkpoints throughout training and perform actor updates under controlled critic-gradient perturbations (Figure 13, Appendix F). We evaluate the updated policy using each method’s frozen, unperturbed critic on fixed held-out states. SoPer consistently exhibits less predicted-value degradation than FlashSAC as perturbation strength increases. This diagnostic supports our motivation of reducing the influence of value-estimation errors during learning.
We shows that soft persistence extends to stochastic training. In Figure 14 (Appendix F), the 99th percentile of squared sampled displacement is consistently lower for SoPer than for FlashSAC across all three tested budgets. Although the constraint applies to nominal displacement, smaller large-action revisions are also observed during stochastic exploration, supporting soft persistence in the implemented algorithm.
We hope the new explanations have addressed your main concerns.
Rebuttal (1/2)
Thanks for your insightful feedback. SoPer makes continuous-time control practical for high-dimensional dynamcial systems. Motivated by our analysis, SoPer employs a simple and general regularizer to learn heterogeneous action dynamics based on task reward and a single drift budget. Our experiments show consistent gains in sample efficiency across various high-dimensional embodiments with different actuator types, ranging from humanoid to 700-actuator musculoskeletal model. The learned actuator timescales of SoPer provides a quantitative way to inspect the intrinsic temporal structure of tasks during embodied control.
Below we clarify your concerns. Please refer to the revised pdf to see our additional experimental results.
Reply to W1 & Q1
We clarify the difference of SoPer and conventional action smoothing with empirical comparison showing that the performance gains do not stem solely from the adaptive weight coefficients.
SoPer v.s. conventional action smoothing
SoPer constrains how far the current policy revises the previously executed action. This choice of reference is independent of whether the penalty weight is fixed or learned. Let at denote the action executed at step t and aˉθ,t the current policy’s nominal output at the corresponding input.
Figure 7 in Appendix F compares SoPer with conventional action smoothing strategies. We sweep four fixed weights for each smoothing baseline and three budgets for adaptive policy-output smoothing. SoPer achieves the highest evaluation return across the tested settings. The gap persists even when the smoothing weight is learned. The performance gap highlight the difference of SoPer and conventional action smoothing.
[1] S. Mysore et al. Regularizing Action Policies for Smooth Control with Reinforcement Learning. ICRA, 2021.
[2] T. Kobayashi. L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learning. IROS, 2022.
[3] I. Lee et al. Gradient-based Regularization for Action Smoothness in Robotic Control with Reinforcement Learning. IROS, 2024.
[4] K. Kwak et al. Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States. AAAI, 2026.
[5] Y. Xue et al. A Unified and General Humanoid Whole-Body Controller for Versatile Locomotion. Robotics: Science and Systems, 2025.
[6] N. Rudin et al. Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Conference on Robot Learning, PMLR 164, pp. 91–100, 2022.