acceptodds
Under review as a conference paper at ICLR 2027

Scalable Continuous-Time Reinforcement Learning for High-Dimensional Control via Soft Persistence

Abstract

Lots of real-world physical systems evolve continuously over time and require control across diverse timescales. Standard reinforcement learning typically represents continuous control as instantaneous state-to-action decisions, leaving the temporal structure of actions implicit. Continuous-time approaches introduce persistence through adaptive decision timing. However, committing the entire action vector delays feedback-based revisions, which become restrictive in high-dimensional systems with heterogeneous actuator requirements. We analyze the trade-off between commitment loss and value-estimation error amplification, and show how heterogeneous temporal requirements can restrict shared commitment as action dimensionality increases. Our derivation shows that bounded action revision can extend the feasible control horizon beyond hard holding. Motivated by the analysis, we propose Soft Persistence Regularization (SoPer), which learns action dynamics by optimizing task return under a conditional-drift budget. Discretizing this objective yields a practical regularizer that integrates directly into standard actor-critic learning, encouraging temporal coherence while allowing action components to evolve at different rates. Experiments on diverse high-dimensional continuous control benchmarks show sample efficiency advantages of SoPer over discrete-time and continuous-time RL baselines. It can support distinct temporal behavior across actuators in different task and regularization strength. We also demonstrate the success of SoPer in a real humanoid robot experiment. The results suggest soft persistence as a scalable way to incorporating continuous-time action dynamics into efficient learning of control.

Primary area:
reinforcement learning
Keywords:
continuous-time reinforcement learning, high-dimensional continuous control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MrCeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors first analyze the trade-off between commitment loss and amplification of value-estimation error under hard persistence, and show that a shared holding duration is restricted by a shared-commitment bottleneck in high-dimensional systems with heterogeneous actuators. They then define the feasible control horizon, and prove that bounded action revision extends the feasible horizon beyond hard holding. Building on this, they propose SOPER, which learns action dynamics under a conditional-drift budget. Finally, they discretized this objective into a regularizer with an adaptive Lagrange multiplier integrated into standard actor-critic training. Compared against FlashSAC, TaCoS, and ASAP, the proposed method shows better sample-efficiency on several benchmarks and a real humanoid robot task.

Critical strengths
  1. The continuous-time trade-off analysis offers a clear theoretical perspective. Proposition 2 characterizes via the feasible control horizon how relaxing the drift budget monotonically extends the feasible horizon, and explains the bottleneck of shared holding durations under heterogeneous high-dimensional actuators. This provides an analytical framework for softening persistence into controlled action revision, and is the paper's most substantive contribution.
  2. The experimental coverage is broad and the algorithm analysis does demonstrate that, compared with the baseline algorithms, the proposed method exhibits certain favorable properties, such as a broader distribution of action timescales.
Critical weaknesses
  1. The ablation study is insufficient. Although the theoretical motivation is sound, apart from augmenting the policy input with the previous action, the only formal difference between SOPER and existing algorithms with action constraints, such as CAPS (Mysore et al., 2021) and ASAP (Kwak & Hwang, 2026), is that fixed coefficients are replaced with adaptive ones. It therefore remains unclear whether the observed performance gains genuinely stem from the theoretical insights derived from continuous-time RL, as the possibility that the improvement is driven by the adaptive weight coefficients has not been ruled out.

  2. The final practical algorithm contains no module that allocates the drift budget on a per-actuator basis. Although Figures 4(b) and 4(c) show that the algorithm produces a broader distribution of action timescales, it remains unclear which factor actually gives rise to this distribution.

Questions for authors
  1. Could additional ablation experiments be provided to verify that the performance gains do not stem solely from the adaptive weight coefficients?
  2. Could the direct factor responsible for the broader action timescale distribution be further clarified?
  3. Could further analysis be conducted on the impact of discretization on the theoretical guarantees?
4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thanks for your insightful feedback. SoPer makes continuous-time control practical for high-dimensional dynamcial systems. Motivated by our analysis, SoPer employs a simple and general regularizer to learn heterogeneous action dynamics based on task reward and a single drift budget. Our experiments show consistent gains in sample efficiency across various high-dimensional embodiments with different actuator types, ranging from humanoid to 700-actuator musculoskeletal model. The learned actuator timescales of SoPer provides a quantitative way to inspect the intrinsic temporal structure of tasks during embodied control.

Below we clarify your concerns. Please refer to the revised pdf to see our additional experimental results.

W1: The ablation study is insufficient. Although the theoretical motivation is sound, apart from augmenting the policy input with the previous action, the only formal difference between SOPER and existing algorithms with action constraints, such as CAPS (Mysore et al., 2021) and ASAP (Kwak & Hwang, 2026), is that fixed coefficients are replaced with adaptive ones. It therefore remains unclear whether the observed performance gains genuinely stem from the theoretical insights derived from continuous-time RL, as the possibility that the improvement is driven by the adaptive weight coefficients has not been ruled out.

Q1: Could additional ablation experiments be provided to verify that the performance gains do not stem solely from the adaptive weight coefficients?

Reply to W1 & Q1

We clarify the difference of SoPer and conventional action smoothing with empirical comparison showing that the performance gains do not stem solely from the adaptive weight coefficients.

SoPer v.s. conventional action smoothing

SoPer constrains how far the current policy revises the previously executed action. This choice of reference is independent of whether the penalty weight is fixed or learned. Let ata_t denote the action executed at step tt and aˉθ,t\bar a_{\theta,t} the current policy’s nominal output at the corresponding input.

  • SoPer places a budget on the expected squared displacement from the previously executed action to the current nominal output, ct=∥aˉθ,t−at−1∥22c_t=\|\bar a_{\theta,t}-a_{t-1}\|_2^2.
  • Policy-output smoothing regularizes variation in policy outputs across inputs [1][2][3][4]: e.g. ct=∥aˉθ,t−aˉθ,t−1∥22c_t=\|\bar a_{\theta,t}-\bar a_{\theta,t-1}\|_2^2. Both outputs are computed with the current policy, so its reference can differ from the action actually executed at the previous step.
  • Reward-side smoothing subtracts a penalty for changes between consecutive executed actions from the task reward, rt=rttask−β∥at−at−1∥22r_t=r_t^{\mathrm{task}}-\beta\|a_t-a_{t-1}\|_2^2 [5][6].

Figure 7 in Appendix F compares SoPer with conventional action smoothing strategies. We sweep four fixed weights for each smoothing baseline and three budgets for adaptive policy-output smoothing. SoPer achieves the highest evaluation return across the tested settings. The gap persists even when the smoothing weight is learned. The performance gap highlight the difference of SoPer and conventional action smoothing.

[1] S. Mysore et al. Regularizing Action Policies for Smooth Control with Reinforcement Learning. ICRA, 2021.

[2] T. Kobayashi. L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learning. IROS, 2022.

[3] I. Lee et al. Gradient-based Regularization for Action Smoothness in Robotic Control with Reinforcement Learning. IROS, 2024.

[4] K. Kwak et al. Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States. AAAI, 2026.

[5] Y. Xue et al. A Unified and General Humanoid Whole-Body Controller for Versatile Locomotion. Robotics: Science and Systems, 2025.

[6] N. Rudin et al. Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Conference on Robot Learning, PMLR 164, pp. 91–100, 2022.

AuthorsOpenReview ↗

Rebuttal (2/2)

W2: The final practical algorithm contains no module that allocates the drift budget on a per-actuator basis. Although Figures 4(b) and 4(c) show that the algorithm produces a broader distribution of action timescales, it remains unclear which factor actually gives rise to this distribution.

Q2: Could the direct factor responsible for the broader action timescale distribution be further clarified?

Reply to W2 & Q2

SoPer permits different actuator rates without explicitly favoring them. The budget constrains the expected discounted average of the total squared revision, ct=∑i=1d(aˉθ,t,i−at−1,i)2c_t=\sum_{i=1}^{d} (\bar{a}^{\theta,t,i}-a^{t-1,i})^2. SoPer allocates the revision of different dimension to optimize task reward while satisfying the budget. Therefore it can learns different actuator timescales based on the both task requirement and budget constraint.

Q3: Could further analysis be conducted on the impact of discretization on the theoretical guarantees?

Reply to Q3

Corollary 3 admits an O(h)O(h) correction. Suppose a budget-feasible controller has task gap at most δ\delta at each control step. Under Assumption 2, the state moves by at most FhFh during the control interval, so the task gap remains at most δ+LℓFh\delta+L_\ell Fh while the command is held. The discounted return gap of the optimal feasible discrete controller is therefore at most (δ+LℓFh)/ρ(\delta+L_\ell Fh)/\rho.

We consider short control intervals are practical in real-time robotics e.g. h=10-20 ms. If LℓFhL_\ell Fh is also small relative to the task-gap tolerance δ\delta, discretization adds only a small relative correction to the continuous-time return bound.

Reviewer_MrCeOpenReview ↗

Official Comment by Reviewer MrCe

I broadly agree that, compared with existing action smoothing techniques, the overall action-drift budget constraint proposed in this paper offers a more flexible and more effective way of constraining multiple actuators, and that it can effectively give rise to heterogeneous behavioral patterns. However, from the standpoint of the continuous-time RL motivation, there remains a noticeable disconnect between the theory and the actual implementation, and I therefore maintain my rating.

Thanks for your feedback. Below we clarify your main concerns. Please refer to the revised pdf to see our additional experimental results.

Main concerns 1: However, from the standpoint of the continuous-time RL motivation, there remains a noticeable disconnect between the theory and the actual implementation.

Reply to main concerns 1

The continuous-time formulation directly determines SoPer’s nominal displacement cost and regularized critic target. Our new experiments further examine the resulting behavior during stochastic training.

SoPer implements the drift objective through an explicit discretization of continuous-time action dynamics. For control interval hh, discretizing a˙=uθ(z)\dot a=u_\theta(z) gives aˉθ(z)=a−+huθ(z)\bar a_\theta(z)=a^{-}+hu_\theta(z) and therefore cθ(z)=∥aˉθ(z)−a−∥2=h2∥uθ(z)∥2c_\theta(z)=\lVert\bar a_\theta(z)-a^{-}\rVert^2=h^2\lVert u_\theta(z)\rVert^2, with budget b=h2Bb=h^2B (Eqs. 14–16). Feedback at every step is compatible with these dynamics: it determines how to revise the current command within the drift budget.

Figure 13 provides an empirical diagnostic of how SoPer obtains robustness to critic error motivated by our analysis. SoPer learns drift-constrained action dynamics, promoting persistence that can temper the influence of critic errors. To examine this empirically, we save checkpoints throughout training and perform actor updates under controlled critic-gradient perturbations (Figure 13, Appendix F). We evaluate the updated policy using each method’s frozen, unperturbed critic on fixed held-out states. SoPer consistently exhibits less predicted-value degradation than FlashSAC as perturbation strength increases. This diagnostic supports our motivation of reducing the influence of value-estimation errors during learning.

We shows that soft persistence extends to stochastic training. In Figure 14 (Appendix F), the 99th percentile of squared sampled displacement is consistently lower for SoPer than for FlashSAC across all three tested budgets. Although the constraint applies to nominal displacement, smaller large-action revisions are also observed during stochastic exploration, supporting soft persistence in the implemented algorithm.

We hope the new explanations have addressed your main concerns.

Reviewer_FqAoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies action persistence in continuous time RL for high dimensional control. Through Prop.1, it first shows a trade-off between commitment loss and estimation error amplification under hard holding. Then it argues with a tracking example that shared holding duration is restrictive when actuators need different timescales (figure 1). Proposed SOPER models actions as variables with bounded drift and maximizes return under a drift budget. In practice, this becomes a Lagrangian penalty on the change of the mean action. Experiments show improved sample efficiency than the baseline.

Critical strengths
  • The motivation is clear, figure 1 explains it well, proofs are clean
  • The method is simple, can be plugged into most actor-critic algorithms
  • Experiments cover a wide range of embodiments
Critical weaknesses
  • Prop 1 show sthe estimation benefit only for hard holding, where the benefit comes from making fewer decisions. Prop 2 and Cor 3 only show that the constraint does not hurt much; SOPER still decides at every step, so it is not clear why the benefit from prop 1 carries over.
  • While the final algorithm is close to a Lagrangian actor-rate penalty, comparison or discussion with the related works seems to be missing: “Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls” (JMLR 2021), “Regularizing Action Policies for Smooth Control with Reinforcement Learning” (ICRA 2021), “ODE-based Smoothing Neural Network for Reinforcement Learning Tasks” (ICLR 2025), “Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning”, CoRL 2021), etc. Also, per-dimension or learned persistence alreadyh exists, e.g., “Reinforcement Learning for Control with Multiple Frequencies” (NeurIPS 2020), "Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning” (ICLR 2017), “TempoRL: Learning When to Act” (ICML 2021), etc. With per-dimension persistence, the sec. 4 example can be easily solved, so the claim that hard persistence must share the same duration should be reframed.
  • About baselines; ASAP stays near zero on all tasks, which looks like an implementation issue. Real robot experiment has no baseline.
  • A scalar lambda with an isotropic budget does not explicitly encourage different rates per dimension; heterogeneity across actuators are not explicitly shown.
Questions for authors
  1. How does SOPER compare with 1. reward level action-rate penalty, 2. fixed lambda, and 3. removing the penalty from critic?
  2. Could you show the trajectory of lambda and the ratio c/b during training?
  3. Can you report the timescale distribution per actuator group (legs / arms / hands)?
6 comments
AuthorsOpenReview ↗

Rebuttal (1/4)

Thanks for your insightful feedback. SoPer makes continuous-time control practical for high-dimensional dynamcial systems. Motivated by our analysis, SoPer employs a simple and general regularizer to learn heterogeneous action dynamics based on task reward and a single drift budget. Our experiments show consistent gains in sample efficiency across various high-dimensional embodiments with different actuator types, ranging from humanoid to 700-actuator musculoskeletal model. The learned actuator timescales of SoPer provides a quantitative way to inspect the intrinsic temporal structure of tasks during embodied control.

Below we clarify your concerns. Please refer to the revised pdf to see our additional experimental results.

W1: Prop 1 show sthe estimation benefit only for hard holding, where the benefit comes from making fewer decisions. Prop 2 and Cor 3 only show that the constraint does not hurt much; SOPER still decides at every step, so it is not clear why the benefit from prop 1 carries over.

Reply to W1

SoPer learns drift-constrained action dynamics while retaining feedback at every control step. The continuous-time controller a˙=uθ(x,a)\dot a=u_\theta(x,a) itself uses state-action feedback. Its discretized implementation therefore evaluates the controller at each step, while the drift budget regulates the magnitude of revisions relative to the previously executed action.

Figure 6(a) in Appendix F shows that the policy thus learns early to track the prescribed drift budget while continuing to optimize task return, consistent with the action dynamics in our analysis. By making large revisions costly, soft persistence can temper the influence of value-estimation errors on action selection.

AuthorsOpenReview ↗

Rebuttal (2/4)

W2: While the final algorithm is close to a Lagrangian actor-rate penalty, comparison or discussion with the related works seems to be missing: “Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls” (JMLR 2021), “Regularizing Action Policies for Smooth Control with Reinforcement Learning” (ICRA 2021), “ODE-based Smoothing Neural Network for Reinforcement Learning Tasks” (ICLR 2025), “Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning”, CoRL 2021), etc. Also, per-dimension or learned persistence alreadyh exists, e.g., “Reinforcement Learning for Control with Multiple Frequencies” (NeurIPS 2020), "Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning” (ICLR 2017), “TempoRL: Learning When to Act” (ICML 2021), etc. With per-dimension persistence, the sec. 4 example can be easily solved, so the claim that hard persistence must share the same duration should be reframed.

Reply to W2

SoPer constrains how far the current nominal policy output aˉθ,t\bar a_{\theta,t} revises the previously executed command at−1a_{t-1}, with cost ct=∥aˉθ,t−at−1∥22c_t=\lVert\bar a_{\theta,t}-a_{t-1}\rVert_2^2. All actuators share one drift budget, and the policy learns from task return how quickly to revise each component. Below we discuss the works mentioned by the reviewer.

  • “Hamilton-Jacobi...” (JMLR 2021) uses a pointwise action-rate bound to derive Q-learning for deterministic continuous-time control. We start from the similar bounded-rate model for different purpose: to study heterogeneous actuator dynamics, then learn those dynamics under a budget on expected discounted squared drift.
  • “Regularizing...” (ICRA 2021) compares policy outputs at consecutive states , e.g. ct=∥aˉθ,t−aˉθ,t−1∥22c_t=\lVert\bar a_{\theta,t}- \bar a_{\theta,t-1}\rVert_2^2. Both outputs are evaluated under the current policy, so the reference can differ from the command actually executed at the previous step. Figure 7 (a) and (b) in Appendix F shows that SoPer outperforms this policy-output smoothing strategy under both fixed weight and adaptive weight setting.
  • “ODE-based...” (ICLR 2025) smooths the policy’s hidden states through ODE neurons with learned time constants and regularized state bounds. SoPer applies different constraint directly to action revisions.
  • “Learning to Walk...”, CoRL 2021) includes an action-rate penalty in the reward, −β∥at−at−1∥22-\beta\|a_t-a_{t-1}\|_2^2. Figure 7 (c) in Appendix F shows that SoPer outperforms this reward-side smoothing strategy.
  • “Reinforcement Learning...” (NeurIPS 2020) enables per-dimensional level persistence by predefining the actuator control frequency instead of learned. It can solve the sec. 4 example only when having the prior knowledge of how each actuator evolves. SoPer uses one drift budget and learns actuator-specific command timescales that vary with the task and budget (Figure 4 (b) and (c)).
  • "Learning to Repeat...” (ICLR 2017) and “TempoRL...” (ICML 2021) learn how long to repeat a selected action. The duration can vary across decisions, but within each decision it applies to the whole action vector as shared hard persistence. Their method cannot learn per-dimension actuator timescale like SoPer.

We thank the reviewer for pointing out the insufficient discussion of previous works, and we will include the clarification in the revised version.

AuthorsOpenReview ↗

Rebuttal (3/4)

W3: About baselines; ASAP stays near zero on all tasks, which looks like an implementation issue. Real robot experiment has no baseline.

Reply to W3

We validated our ASAP adaptation against the official implementation and conduct broad sweep of hyperparameters. Our controlled hardware experiment demonstrates SoPer achieves efficient and better control learning than PPO with and without action-rate penalties.

ASAP experiment

We implemented ASAP on the FlashSAC backbone and checked its penalty calculations against the official implementation. On Ant-v5, the highest benchmark shown in the ASAP paper, our implementation with FlashSAC normalization achieves improved performance under aligned experimental condition (Figure 9 (a)).

For the main comparisons, we transferred the coefficients used on Ant-v5 to our higher-dimensional tasks (Table 2). We also conducted a wide range of hyperparameter sweep of ASAP by downscaling the orginal penalty weight combination, and observe consistent low performances (Figure 9 (b)). ASAP introduces prediction loss to encourage policy to visit similar states. On H1hand-Run, the prediction loss was approximately 400x larger than prediction loss on Ant-v5. This change in loss scale offers one possible explanation for why ASAP performs poorly in high-dimensional control.

Hardware baseline

We conduct a new hardware experiment to make the humanoid robot track full-body dancing movement. We remove action-rate penalty from the original reward design. Please see Figure W7 in project page for the sim-to-real performances of SoPer.

We compare the learning performances of SoPer against standard PPO baseine, and PPO variants with different level of action rate penalty. Our proposed method demonstrate superior learning efficiency and control performances (Figure 10 in Appendix F).

W4: A scalar lambda with an isotropic budget does not explicitly encourage different rates per dimension; heterogeneity across actuators are not explicitly shown.

Q3: Can you report the timescale distribution per actuator group (legs / arms / hands)?

Reply to W4 & Q3

SoPer allocates actuator’s revision within scalar budget, while the task reward can favor faster changes in some commands than in others. We show the learned timescale distributions across the actuator groups under different budgets and tasks.

How scalar budget encourage different actuator timescales

SoPer permits different actuator rates without explicitly favoring them. The budget constrains the expected discounted average of the total squared revision, ct=∑i=1d(aˉθ,t,i−at−1,i)2c_t=\sum_{i=1}^{d}(\bar{a}^{\theta,t,i}-a^{t-1,i})^2. SoPer allocates the revision of different dimension to optimize task reward while satisfying the budget. Therefore it can learns different actuator timescales based on the both task requirement and budget constraint.

Timescale distribution per actuator group

Figure 11 in Appendix F shows that SoPer learns different command timescales across actuator groups under a shared drift budget.

On Run (panel a), the leg and torso distributions have narrow peaks at short timescales. Tightening the budget shifts these peaks toward slower changes. The arms and hands show a broad tail toward slower changes.

At the same drift budget (panel b), all four groups peak at longer timescales on Balance Simple than on Run. The torso shows the largest shift and has the longest typical timescale on Balance Simple. This is consistent with the task demands: running requires repeated adjustments within each stride, while balancing on a tilting board favors a sustained upright posture. The hand distribution also becomes more concentrated on Balance Simple. FlashSAC shows smaller shifts across tasks, particularly for the torso, where its two curves nearly overlap.

Reviewer_CTKnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies high-dimensional continuous control in which different actuators require different temporal responses. It proposes Soft Persistence Regularization (SOPER), which encourages temporally coherent actions while allowing individual action components to change at different rates. The authors first analyze the trade-off between action commitment loss and value-estimation error amplification. A command-tracking example illustrates how a shared action-holding duration can be limited by the fastest-changing component. Motivated by this analysis, the paper formulates action evolution as a continuous-time process subject to a drift budget and derives a discrete action-displacement regularizer compatible with actor–critic learning. In the practical implementation, the policy conditions on the current state and previously executed action, and the regularizer penalizes the squared difference between its deterministic output and the previous action. A primal–dual procedure adapts the penalty coefficient, with displacement costs incorporated into both policy optimization and value learning. Experiments evaluate SOPER on HumanoidBench, LocoMuJoCo imitation tasks, and a 700-actuator musculoskeletal control task, comparing it with FlashSAC, TaCoS, and ASAP. The paper reports sample-efficiency improvements across multiple tasks, analyzes the resulting actuator-level action timescales, and demonstrates deployment of a simulation-trained PPO variant on a humanoid robot performing upper-body dance motions while maintaining balance.

Critical strengths
  1. The evaluation covers several challenging control settings, including humanoid control and a 700-actuator musculoskeletal system. The main simulation experiments use five seeds and a common actor–critic backbone. The reported results provide evidence that the proposed regularizer can improve learning across different actuator types.
  2. The practical method is relatively simple and compatible with existing actor–critic algorithms. Conditioning on the previous action and adaptively penalizing action displacement could be useful for high-dimensional control.
Critical weaknesses
  1. The connection between the continuous-time formulation and the implemented stochastic algorithm is incomplete. The theoretical formulation constrains the drift of the action trajectory, whereas the implementation penalizes the difference between the deterministic actor output and the previously executed action. During stochastic training, this does not directly constrain the displacement of sampled actions, which also depends on policy variance. Moreover, the persistence–estimation analysis concerns longer action-holding intervals, while SOPER retains fixed-frequency decisions and one-step value updates. The paper does not establish how the implemented algorithm obtains the estimation-error benefit used to motivate it. These gaps weaken the central interpretation of SOPER as learning drift-constrained continuous-time action dynamics.
  2. The experiments do not sufficiently isolate the proposed mechanism from conventional action smoothing. Figure 4 examines previous-action inputs and displacement-budget sensitivity, but does not separate the contributions of a fixed displacement penalty, adaptive multiplier learning, and the modified critic target. The action-timescale distributions demonstrate altered temporal behavior, but do not establish that heterogeneous timescales explain the performance gains. Baseline validation also needs attention: ASAP exhibits little learning on most tasks, without a diagnostic explaining this failure. Finally, the hardware demonstration lacks a matched baseline and already incorporates reference-action feedforward and action-rate penalties, making SOPER’s additional contribution difficult to assess.
Questions for authors
  1. What relationship can be established between the implemented deterministic-output penalty and the displacement of actions actually sampled during training? Does the continuous-time drift interpretation require additional assumptions or constraints on policy variance?
  2. Can you separate the contributions of fixed-coefficient displacement regularization, adaptive multiplier learning, and the modified critic target under matched training budgets? These comparisons would help determine whether the proposed mechanism offers benefits beyond standard action smoothing.
  3. How was the ASAP adaptation validated, and what explains its near-zero performance on most evaluated tasks? Does this behavior persist after reasonable method-specific tuning?
6 comments
AuthorsOpenReview ↗

Rebuttal (1/4)

Thanks for your insightful feedback. SoPer makes continuous-time control practical for high-dimensional dynamcial systems. Motivated by our analysis, SoPer employs a simple and general regularizer to learn heterogeneous action dynamics based on task reward and a single drift budget. Our experiments show consistent gains in sample efficiency across various high-dimensional embodiments with different actuator types, ranging from humanoid to 700-actuator musculoskeletal model. The learned actuator timescales of SoPer provides a quantitative way to inspect the intrinsic temporal structure of tasks during embodied control.

Below we clarify your concerns. Please refer to the revised pdf to see our additional experimental results.

W1.1: The connection between the continuous-time formulation and the implemented stochastic algorithm is incomplete. The theoretical formulation constrains the drift of the action trajectory, whereas the implementation penalizes the difference between the deterministic actor output and the previously executed action. During stochastic training, this does not directly constrain the displacement of sampled actions, which also depends on policy variance.

Q1: What relationship can be established between the implemented deterministic-output penalty and the displacement of actions actually sampled during training? Does the continuous-time drift interpretation require additional assumptions or constraints on policy variance?

Reply to W1.1 & Q1

SoPer combines a direct discretization of the continuous-time objective with entropy-regularized exploration. The constraint governs nominal action revisions without additional variance assumptions, while sampled action displacements also depend on policy variance.

The connection between the continuous-time formulation and the implemented stochastic algorithm

Our implementation of SoPer directly discretizes the drift cost in our continuous-time objective. The continuous-time formulation uses a˙t=uθ(xt,at)\dot a_t=u_\theta(x_t,a_t) with an average squared-drift budget BB. Let z=(x,a−)z=(x,a^{-}), where a−a^{-} is the previously executed action. Discretization with control interval hh gives

aˉθ(z)=a−+h uθ(z),cθ(z)=∥aˉθ(z)−a−∥2=h2∥uθ(z)∥2.\bar a_\theta(z)=a^{-}+h\,u_\theta(z),\qquad c_\theta(z)=\|\bar a_\theta(z)-a^{-}\|^2=h^2\|u_\theta(z)\|^2.

Thus, learning the nominal next-action map implicitly learns the action drift, with displacement budget b=h2Bb=h^2B. Our implementation uses conventional entropy-regularized stochastic policy optimization during training, allowing exploration while learning the action dynamics. During evaluation, we execute the learned policy deterministically.

Why not constraining sampled action displacement directly?

Constraining sampled displacement would make exploration variance count toward the drift budget. For a sampled action a∼πθ(⋅∣z)a\sim\pi_\theta(\cdot\mid z) and the previously executed action a−a^{-}, the expected squared displacement is

E[∥a−a−∥2∣z]=∥E[a∣z]−a−∥2+tr⁡Cov⁡(a∣z).\mathbb E[\|a-a^{-}\|^2\mid z]=\|\mathbb E[a\mid z]-a^{-}\|^2+\operatorname{tr}\operatorname{Cov}(a\mid z).

Even when the conditional mean equals a−a^{-}, the policy incurs a cost equal to its total conditional variance. Meeting a tight budget can therefore compete with exploration encouraged by SAC's entropy objective. In our experiment with a tight drift budget (Figure 6 (b), Appendix F), the average sampled displacement cost remains above the budget, and the corresponding Lagrange multiplier explodes.

SoPer applies the budget to the squared nominal displacement ∥aˉθ(z)−a−∥2=h2∥uθ(z)∥2\|\bar a_\theta(z)-a^{-}\|^2=h^2\|u_\theta(z)\|^2. This directly discretizes the continuous-time drift cost while allowing stochastic exploration during training.

Additional assumptions or constraints of the continuous-time drift interpretation

No additional variance assumption is needed for the continuous-time interpretation of the action dynamics, because cθ(z)=h2∥uθ(z)∥2c_\theta(z)=h^2\|u_\theta(z)\|^2 holds regardless of exploration variance.

AuthorsOpenReview ↗

Rebuttal (2/4)

W1.2: Moreover, the persistence–estimation analysis concerns longer action-holding intervals, while SOPER retains fixed-frequency decisions and one-step value updates. The paper does not establish how the implemented algorithm obtains the estimation-error benefit used to motivate it. These gaps weaken the central interpretation of SOPER as learning drift-constrained continuous-time action dynamics.

Reply to W1.2

SoPer learns drift-constrained action dynamics while retaining feedback at every control step and one-step value updates. The policy tracks its drift budget early in training. The penalty on large revisions from drift constraints can temper the influence of critic errors on action selection.

Fixed-frequency decisions

Frequent feedback is compatible with persistent action dynamics. The continuous-time controller a˙=uθ(x,a)\dot a=u_\theta(x,a) itself uses state-action feedback. Its discretized implementation therefore evaluates the controller at each step, while the drift budget regulates the magnitude of revisions relative to the previously executed action. The frequency of feedback and the magnitude of action revision are distinct aspects of the controller.

One-step value updates

Our continuous-time formulation leaves the value-learning method open. After discretization, the regularized objective admits the Bellman recursion, which supports standard one-step TD learning. Our modified critic target includes future displacement costs consistently with this objective. Therefore, one-step value updates are compatible with learning the value of the proposed action dynamics.

How the implemented algorithm learns drift-constrained continuous-time action dynamics and obtains the estimation-error benefit

SoPer learns drift-constrained action dynamics by optimizing a discretization of the continuous-time objective. Figure 6 (a) in Appendix F shows that the constraint gap approaches zero within the first 10$%$ of training and remains near zero thereafter. The policy thus learns early to track the prescribed drift budget while continuing to optimize task return, consistent with the action dynamics in our analysis. By making large revisions costly, soft persistence can temper the influence of value-estimation errors on action selection.

AuthorsOpenReview ↗

Rebuttal (3/4)

W2.1: The experiments do not sufficiently isolate the proposed mechanism from conventional action smoothing. Figure 4 examines previous-action inputs and displacement-budget sensitivity, but does not separate the contributions of a fixed displacement penalty, adaptive multiplier learning, and the modified critic target.

Q2: Can you separate the contributions of fixed-coefficient displacement regularization, adaptive multiplier learning, and the modified critic target under matched training budgets? These comparisons would help determine whether the proposed mechanism offers benefits beyond standard action smoothing.

Reply to W2.1 & Q2

We first clarify the difference of SoPer and conventional action smoothing, then report the ablation results of SoPer implementation design.

SoPer v.s. conventional action smoothing

SoPer constrains how far the current policy revises the previously executed action. This choice of reference is independent of whether the penalty weight is fixed or learned. Let ata_t denote the action executed at step tt and aˉθ,t\bar a_{\theta,t} the current policy’s nominal output at the corresponding input.

  • SoPer places a budget on the expected squared displacement from the previously executed action to the current nominal output, ct=∥aˉθ,t−at−1∥22c_t=\|\bar a_{\theta,t}-a_{t-1}\|_2^2.
  • Policy-output smoothing regularizes variation in policy outputs across inputs [1][2][3][4]: e.g. ct=∥aˉθ,t−aˉθ,t−1∥22c_t=\|\bar a_{\theta,t}-\bar a_{\theta,t-1}\|_2^2. Both outputs are computed with the current policy, so its reference can differ from the action actually executed at the previous step.
  • Reward-side smoothing subtracts a penalty for changes between consecutive executed actions from the task reward, rt=rttask−β∥at−at−1∥22r_t=r_t^{\mathrm{task}}-\beta\|a_t-a_{t-1}\|_2^2 [5][6].

Figure 7 in Appendix F compares SoPer with conventional action smoothing strategies. We sweep four fixed weights for each smoothing baseline and three budgets for adaptive policy-output smoothing. SoPer achieves the highest evaluation return across the tested settings. The gap persists even when the smoothing weight is learned. The performance gap highlight the difference of SoPer and conventional action smoothing.

Ablation of multiplier learning and the critic target

SoPer’s critic target and multiplier update follow from applying primal–dual optimization to the discretized continuous-time objective. Removing the cost from the critic target omits the future displacement costs required by the regularized Bellman equation. A fixed multiplier skips the dual update that adapts this weight to the prescribed drift budget. Below we compare these two variants to original SoPer implementation.

  • Compared to fixed lambda: We compare SoPer against variant using same constraint with fixed multiplier. Figure 8 (b) in Appendix F shows that the best tuned weight matches SoPer’s final return after a slower start, while other weights yields degraded or near-zero return throughout training.
  • Compared to removing the penalty from critic: We compare SoPer against variant removing the critic side cost. We observe substantial performance degradation compared to full critic target (Figure 8 (a) in Appendix F).

[1] S. Mysore et al. Regularizing Action Policies for Smooth Control with Reinforcement Learning. ICRA, 2021.

[2] T. Kobayashi. L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learning. IROS, 2022.

[3] I. Lee et al. Gradient-based Regularization for Action Smoothness in Robotic Control with Reinforcement Learning. IROS, 2024.

[4] K. Kwak et al. Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States. AAAI, 2026.

[5] Y. Xue et al. A Unified and General Humanoid Whole-Body Controller for Versatile Locomotion. Robotics: Science and Systems, 2025.

[6] N. Rudin et al. Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Conference on Robot Learning, PMLR 164, pp. 91–100, 2022.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.