acceptodds
Under review as a conference paper at ICLR 2027

DAMPER: RETURN-PRIORITIZED GRADIENT CONTROL FOR SMOOTH POLICIES

Abstract

Actor–critic methods achieve strong performance in continuous control, but their policies can produce highly oscillatory actions. A common remedy is to add auxiliary smoothness losses. However, their contribution can be negligible when their gradients are small relative to the native actor gradient. Moreover, existing methods often combine multiple auxiliary losses, complicating loss balancing without necessarily improving the return–smoothness trade-off. We introduce DAMPER (Direction-Aware Magnitude-Controlled Projection with Explicit Return Priority), which combines the native actor gradient with a temporal-consistency gradient through conflict-conditioned projection and adaptive magnitude control. It removes the auxiliary component opposing the actor gradient and scales the retained temporal direction relative to the actor gradient norm, preserving positive alignment with the native actor gradient. Experiments with TD3 and SAC on six continuous-control tasks show reduced action oscillation relative to the native agents in all 12 task–backbone pairs and the best oscillation score among the compared methods in eight, with task-dependent return trade-offs.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Continuous Control, Policy Smoothness, Gradient Conflict, Multi-Objective Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_snWkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Actor-critic methods can produce highly oscillatory actions. Therefore, in this paper, the authors propose DAMPER, combining actor gradient and a modified temporal-consistency gradient. Experiments show that action oscillation is indeed reduced, and DAMPER performs better than existing baselines.

Critical strengths
  1. The presentation is very clear and easy to follow. I enjoy reading Figure 1, because it makes me fully understand the method of this paper within minutes. Method and experiment sections are also well written.

  2. The authors provide open source code, and it appears well organized, clean, and useful.

Critical weaknesses
  1. The novelty is incremental. Combining native actor loss with temporal consistency loss is well studied. Also, conflict-conditioned projection is also from existing work, such as PEGrad. Therefore, this paper mainly propose an engineering method to combine existing strategies. Also, I am not sure why the authors don't include PEGrad in the main results but in a separate table.

  2. I have one addressable concern about the experimental design. While I am not an expert in this exact subfield, I understand that there is a trade-off between return and smoothness. As one can see in Table 1 and 2, it is often the case that DAMPER is only better on one of smoothness / reward. Therefore, the authors should plot a smooth-return curve for each method (by altering hyperparameters) and compare which method curve is better than others. This would be much better than comparing at only one single point. I understand plotting such a curve is extremely costly, so the authors may feel free to only pick one algorithm and task.

Questions for authors

I will increase my score to 3 if the authors add the smooth-return curve, and from the curve, the strength of DAMPER is evident.

Reviewer_XGxGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DAMPER adds a consistency gradient between actions at consecutive states to the actor's normal loss gradient. The key part is how the two are combined: if the auxiliary gradient points against the main one, the opposing component gets dropped, and then the magnitude is interpolated between a capped version and a norm-matched version, using the original auxiliary norm as the knob. The theory is for fixed surrogate losses with plain Euclidean steps. Experiments are on TD3 and SAC across six tasks, with ablations on the direction rule, a scalarization sweep, an interpolation sweep, and some wall-clock numbers (pp. 3-9, Appendices A-F).

Critical strengths
  1. The smoothing effect shows up pretty consistently. In Tables 1-2 the mean oscillation is lower than the native agent in all 12 settings, and lowest among the compared methods in 8 of them. The Hopper ablation (return 3078 vs 317 when you just sum the gradients unconditionally) is a nice illustration of why the directional part actually matters (pp. 7-8, Table 3). I should say I did not rerun anything, these are the reported numbers.
  2. The rule itself is simple and you can reason about it on paper. Appendices A-B walk through the projection and explain the attenuation that comes from normalizing by the original temporal norm. I also appreciate that the authors are upfront that the theory only covers fixed surrogates and not episodic return, which is the right thing to do.
Critical weaknesses
  1. The claimed return vs smoothness advantage is not really isolated under a fair tuning budget. The scalarization baseline only sweeps lambda in [0.05, 1], but DAMPER is allowed to scale a weak auxiliary gradient up beyond 1, so the comparison is a bit lopsided. The eta values are different per task and per backbone and I could not find how they were chosen. Also the eta sweep in Tables 5-6 only reports oscillation and not the return that goes with it. To give one example from Table 1, TD3 Hopper gets a 92.2% reduction in oscillation but also loses 14.6% return. That is not a contradiction of the theory, but it does mean a matched-return comparison is what I would actually need to see.
  2. There are a few gaps that make it hard to reproduce exactly what was run. Algorithm 1 hands things off to whatever native optimizer the backbone uses, but Theorem 1 is about Euclidean steps. Appendix B.1 mentions a numerical guard that is not in the pseudocode. And the FFT metric in Section 6.1 is not fully specified (window, aggregation across dims and seeds, etc.) (pp. 5-6, 14). To be clear, I did not find an actual error in the derivation for the nonzero-gradient case, this is about reproducibility.
Questions for authors
  1. How were eta and the checkpoints picked? Could you add return alongside oscillation for Figure 4, and some seed-level spread?
  2. What is the exact optimizer used, how is the zero-norm case handled, how exactly is the PEGrad comparator updated, and how is the FFT score computed?
  3. Does the advantage hold up if lambda is allowed above 1 with proper scaling, and under matched return? Real robot deployment can stay as future work, I am not asking for that.
1 comment

Thank you for the detailed rebuttal for my concerns. The rebuttal clarifies the reproducibility details and provides encouraging matched-return results on Hopper. However, DAMPER still cannot match the returns of the smoothing baselines on Walker. I maintain my original rating of 2.

Reviewer_XdjWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DAMPER, a primary–auxiliary gradient combination method for TD3 and SAC that aims to reduce action oscillations while prioritizing the native actor objective. It uses an established temporal-consistency loss between policy outputs at consecutive states and computes the native actor and smoothness gradients separately. The auxiliary gradient is projected only when it conflicts with the native actor gradient, and an interpolation parameter η\eta controls its magnitude between cap-only and norm-matched updates. The paper provides a single-step descent analysis for a fixed actor surrogate and a local comparison with fixed-weight scalarization in the weak-auxiliary-gradient regime. Experiments cover six continuous-control tasks with two actor–critic backbones.

Critical strengths
  • The problem is practically relevant, and the method is straightforward to integrate. Action smoothness is an important consideration in continuous control. DAMPER modifies only the training-time actor-gradient combination and introduces no additional networks or filtering overhead during policy execution.

  • The algorithmic components correspond clearly to the stated motivations. Conditional projection addresses directional conflict, while magnitude control addresses auxiliary gradients that are disproportionately weak or strong. Using the pre-projection norm also preserves the attenuation introduced by projection, yielding a clearly specified update rule.

Critical weaknesses
  • W1. The fixed-weight baseline search is too restricted to establish the benefit of adaptive magnitude control.

Section 6.4 searches only λ∈[0.05,1.0]\lambda\in[0.05,1.0], whereas DAMPER’s effective auxiliary coefficient, (nR/nT)η(n_R/n_T)^\eta when nT<nRn_T<n_R, can substantially exceed 11. The comparison therefore does not adequately distinguish the benefit of adaptive scaling from simply allowing stronger regularization. Theorem 2 establishes a local asymptotic advantage over a fixed weight, but does not resolve this empirical concern. A broader logarithmic search, comparable tuning budgets, and return–smoothness Pareto comparisons are needed.

  • W2. The incremental contribution of magnitude control is not sufficiently isolated.

Conditional projection is established in PCGrad [1], and the PCGrad+ baseline in the PEGrad paper already preserves the primary gradient while projecting only conflicting auxiliary gradients—the same directional operation used by DAMPER. The main methodological additions are therefore the norm interpolation and pre-projection normalization. However, the existing comparisons do not fully separate their effects from changes in projection and scaling. Controlled comparisons against PCGrad+ with cap-only scaling and fixed norm matching, together with a pre- versus post-projection normalization ablation, would better establish the value of these additions.

[1] Yu, T., et al. Gradient Surgery for Multi-Task Learning.

Questions for authors

If the search over λ\lambda is extended beyond 11, does DAMPER still improve the return–smoothness Pareto frontier? Are all methods evaluated with comparable hyperparameter-search budgets and the same selection criteria?

1 comment

Thank you for the detailed response and extensive additional experiments. The broader fixed-weight search, controlled comparisons with PCGrad+ variants, and additional hyperparameter sweeps substantially address my concerns and strengthen the empirical support for magnitude control.

Some limitations remain: the methodological novelty is incremental, and the new controlled experiments are limited to TD3 on Hopper and Walker. Nevertheless, the additional evidence is sufficient to support the practical value of the method. I will therefore raise my score to 3, in support of acceptance. I encourage the authors to incorporate these results and clarify the remaining limitations in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.