acceptodds
Under review as a conference paper at ICLR 2027

Regularised RL improves sim-to-real transfer

Abstract

Regularised reinforcement learning (RL) has provable robustness guarantees that should ease the transfer of policies from simulation to robots, known as sim-to-real. Yet, weakly regularised algorithms like Proximal Policy Optimisation (PPO) remain the de facto choice in sim-to-real pipelines, while the practical benefits of regularisation for transfer remain poorly characterised. In this work, we study the sim-to-real performance of commonly used RL algorithms, with the aim of characterising the effects of regularisation on robotic performance. To that end, we derive Soft MellowMax Actor-Critic (\ourMethod), a novel regularised RL algorithm based on the soft mellowmax value function, which carries strong robustness guarantees. On MuJoCo continuous control benchmarks, \ourMethod is competitive with baselines like Soft Actor-Critic (SAC). On real robots, \ourMethod achieves higher return, while exhibiting greater robustness to perturbations in observations, the environment, and dynamics than baselines. Together, these results demonstrate that regularised RL improves real-world policy deployment through better robustness to the sim-to-real gap, and changing real-world conditions.

Primary area:
reinforcement learning
Keywords:
Soft mellowmax, actor-critic algorithm, regularised RL, sim-to-real transfer, robotics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_sWojOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents an extension of soft mellowmax policies from discrete to continuous action spaces, and studies the sim-to-real benefits of the inherent regularization properties of soft mellowmax. The paper includes a mathematical derivation of their proposed policy which is contrasted with mathematical properties of SAC. Similar to DDPG, the proposed method uses a target policy network in addition to target Q network to approximate the intractable reference policy involved in soft mellowmax policy updates. On several Mujoco benchmarks, the proposed method performs mostly comparably to SAC as well as PPO and TD3. More interestingly, on sim-to-real experiments in two domains (block manipulation and quadruped locomotion), the proposed method outperforms these baselines, including in the face of adversarial perturbations such as object mass changes and rough terrain. Based on this, the paper concludes that the inherent regularization of soft mellowmax improves policy robustness.

Critical strengths
  • The paper is mostly well-written and easy to follow, including the technical parts. It could be an accessible reference for researchers interested in RL regularization techniques.

  • The experimental results are rather compelling, showing clear advantages of the method for sim-to-real transfer and robustness. The experiments include multiple repetitions and report standard deviations to quantify statistical significance, which is appreciated.

Critical weaknesses
  • The approach may not be particularly novel. For example, there is an existing approach from early 2025 (reference below) which was not cited by this paper. However, it also extends soft mellowmax to continuous action spaces, and uses ideas from DDPG which overlap with this paper (in particular the use of target policy networks).

Z. Shang, R. Li, C. Zheng, H. Li and Y. Cui, "Relative Entropy Regularized Sample-Efficient Reinforcement Learning With Continuous Actions," in IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 1, pp. 475-485, Jan. 2025, doi: 10.1109/TNNLS.2023.3329513.

  • The paper does not include any anonymous code links or supplementary material, which undermines confidence in reproducibility and veracity of the claims.

  • Certain issues with the presentation: First, the title seems overly broad. This paper demonstrates sim-to-real benefits in the very specific context of the proposed method, as opposed to some general principles about regularized RL more broadly. Second, the descriptions of the results verge on LLM slop. The authors acknowledged LLM use, but this does not change the fact that the LLM's writing is poor. Perhaps in an attempt to save space, there are too many brief/broad/vague statements like "X loses nothing" (line 457, 461, etc.). I am starting to see very similar and frustrating phrasing like this in many papers, reflecting a sad collapse of our field towards a stylistic LLM monoculture that is mentally taxing on the reader (at least, on this reader). I encourage the authors to be more precise and devote more energy towards cultivating their own unique voice and aptitude for communicating results.

Questions for authors

Can the authors elaborate on what differentiates their work from the existing literature I cited above?

4 comments

We thank the reviewer for their helpful review and their interest in our submission.

[W1, Q1] Relation to Shang et al. Thank you for bringing this prior work to our attention, we did not come across it before. You are correct that it is conceptually similar to our work in a number of ways: 1) using the mellowmax operator for its stability, 2) using an actor-critic algorithm for its sample efficiency, and 3) estimating intractable softmax policies. However, there are significant differences between this work and what we propose. These are listed below: Base algorithm: we make use of the SAC actor-critic algorithm, which differs fundamentally from DDPG-style algorithms. First, DDPG uses deterministic policies, meaning that the policy function selects an action, i.e., at=π(a∣s)a_t = \pi(a|s). SAC-style algorithms sample each action stochastically from the policy function, i.e., at∼pi(⋅∣s)a_t \sim pi(\cdot|s). This means that we must learn the entire intractable softmax policy, not just generate samples from it, as in DDPG-style algorithms. Second, DDPG algorithms compute policy gradients on a reward (or Q-value) maximisation loss. SAC-style algorithms instead update policies computing the gradient on a KL divergence loss between a target policy and the current policy. Differences in regularisation leading to different values and policies: we regularise the actor against the policy softmax(αQ)softmax(\alpha Q), using the current Q-value network. This work regularises against a previous policy iterate, essentially doing the same thing as TRPO and PPO, in a sense. Thus, the resulting value function (soft mellowmax vs. mellowmax) and the optimal policy are different. Experimental scope and tested claims: Our paper focuses primarily on sim-to-real transfer and how it can be improved with regularisation. This work focuses primarily on proposing a new DDPG-style algorithm for improved learning stability. Experimentally, both papers test on MuJoCo, but we test our policies on real robots. Handling intractable integrals: this work estimates intractable integrals by computing Monte Carlo estimates of the normalisation terms in the policy. While we did test a similar approach while developing our work, we ultimately decided to learn both the actor and the reference policies because this yielded more stable learning, and better performance.

Our contribution is the particular SMM actor-critic formulation and its evaluation on physical manipulation and locomotion systems under deployment perturbations. We will acknowledge the overlap with CDPP and explain these differences explicitly in the related works section.

[W2] Code and supplementary material. Our anonymised code is available at https://anonymous.4open.science/r/SMM-AC-08E8/

We will release it publicly after the review process.

[W3] Title and presentation. We agree that the original title was too broad and have changed it to “Soft Mellowmax Actor-Critic for Sim-to-Real Transfer”. The abstract, introduction and conclusion now match this scope: claims about regularised methods in general are limited to the locomotion tasks, where both SAC and SMM-AC outperform PPO and FastTD3.

We also agree that phrases such as “loses nothing” were imprecise. Sec. 5.3 now states that the cube perturbations change SMM-AC's mean return by less than 4%, and compares its return under lower friction on rough terrain with its nominal mean. We have also replaced the remaining qualitative summary of retention with counts from Table 2.

We kindly request that you increase your score if our responses have answered your questions and concerns. Thank you again for your questions!

The rebuttal does mostly address my concerns:

  • The description of differences from CDPP is reasonable and indicates some level of novelty in this work.
  • While I haven't carefully checked the anonymous code link, since it is posted here and these discussions will be made public, readers are in a better position to try reproducing the results and check the veracity of this paper's claims for themselves.
  • The wording in 5.3 is overall more palatable to me now. There are still some lines that sound very LLM-like, such as line 463 ("Every method has its largest loss under observation noise, and this is also where the methods differ the most"). But since this is a stylistic issue and disclosed by the authors since the beginning, I won't harp on it any longer.

As such I'm leaning towards increasing my rating. I still hesitate for two reasons: First, I'm the only reviewer on the positive side - although given that another review was flagged as AI-generated, I am just going to focus on my own review and its rebuttal. Second, the rebuttal's comparison with CDPP is still not integrated into the latest PDF version of the manuscript. If the authors integrate it in another posted revision, I'll consider increasing my rating by one point.

Also, a couple more minor things I'd like the authors to address:

  • Line 337: Is "fails fewer than 80%" supposed to be "fails fewer than 20%"? 80% seems like a rather liberal failure rate for safety considerations.
  • Line 487: Typo, missing "T"

Thanks for following up. We added the comparison with CDPP in Section 2, covering the actor, reference policy, value operator, and experimental scope.

The reviewer is right about the safety threshold. The criterion used was fewer than 20% failures. "80%" was simply a typo. We corrected this in Section 5.1, fixed the missing "T", and removed the sentence you pointed out in Section 5.3 as the rest of the paragraph already covers that point. The revised PDF reflects all of these changes.

Based on this final revision I increased my rating.

Reviewer_1bBTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether regularization-induced robustness in RL translates into improved sim-to-real transfer. Motivated by the theoretical connection between regularized and robust RL, the paper proposes Soft MellowMax Actor-Critic (SMM-AC), extending Soft MellowMax to continuous control. SMM-AC approximates the reference policy using a slowly updated reference network, enabling stable actor-critic optimization. Experiments compare SMM-AC against PPO, SAC, and TD3 on MuJoCo tasks and real-world robotic systems under deployment-time perturbations to observations, environment properties, and dynamics. The results suggest that regularization can improve robustness and real-world transfer.

Critical strengths
  • The paper addresses an important and practically relevant problem: robustness and sim-to-real transfer in RL.
  • SMM-AC is an interesting and well-motivated extension of Soft MellowMax to continuous control, with a clear derivation and practical actor-critic formulation.
  • The paper is well written and technically rigorous.
  • The experimental setup is clear, and the inclusion of real-world robotic experiments under deployment perturbations significantly strengthens the empirical contribution.
Critical weaknesses

Missing robustness-aware baselines. The evaluation would be stronger with comparisons against methods explicitly designed for robust RL, such as:

  • Pinto et al. Robust adversarial reinforcement learning. ICML, 2017.
  • Tessler et al. Action robust reinforcement learning and applications in continuous control. ICML, 2019.
  • Kamalaruban et al. Robust Reinforcement Learning via Adversarial training with Langevin Dynamics. NeurIPS, 2020.

These approaches also appear relevant to the related-work discussion.

Questions for authors
  • Why are explicit robustness-aware RL methods not included as baselines? How would SMM-AC compare against adversarial or action-robust approaches such as Pinto et al. (2017), Tessler et al. (2019), or Kamalaruban et al. (2020)?
  • In Fig. 2, are the MuJoCo policies evaluated only on the nominal training environments? If so, do the authors have results under controlled perturbations to environment parameters or dynamics?
3 comments

We thank the reviewer for their constructive review and for suggesting robustness-aware baselines.

[W1, Q1] Robustness-aware baselines.

Our paper asks whether the regularisation already used in standard RL pipelines provides robustness without adversarial training or a model of the perturbation. Robust RL methods take a different route. They train against an adversary whose type and strength must be chosen, at the cost of two-player training. Comparing the two was not the paper's aim, but it is a fair question, so we ran it on Franka Lift, including deployment on the physical robot.

Our original baselines show how SMM-AC compares with commonly used RL methods, but not how it compares with methods trained explicitly for robustness. We have therefore added Action Robust RL [1] and MixedNE-LD [2]. MixedNE-LD trains a similar action adversary but uses Langevin dynamics to approximate a mixed Nash equilibrium of the two-player game. Both were trained on Franka Lift with the same domain randomisation and training setup as SMM-AC, using five seeds each. We deployed them on the physical robot under the same perturbations as in Table 2, with 20 episodes per seed and condition.

Hardware return, mean ±\pm SD across seeds. SMM-AC values are from Table 2 (ω\omega = 800):

PerturbationSMM-ACAction Robust RLMixedNE-LD
None152.9 ±\pm 2.5145.4 ±\pm 4.2149.8 ±\pm 5.6
Cube mass 2×151.6 ±\pm 3.1139.8 ±\pm 4.8145.3 ±\pm 5.9
CoM offset, 15 mm153.0 ±\pm 2.8145.2 ±\pm 5.4141.8 ±\pm 6.1
Observation noise, σ\sigma = 0.001150.2 ±\pm 2.3132.6 ±\pm 7.5140.3 ±\pm 6.8
Observation noise, σ\sigma = 0.005120.9 ±\pm 4.0110.3 ±\pm 9.2107.7 ±\pm 9.9

Both robust methods are strong baselines. Without perturbation they come within about 3 points of SMM-AC. Under the strongest observation noise they keep a high return against lower returns for PPO, FastTD3 and SAC in Table 2. The main reason for these methods to underperform compared to SMM-AC could be that both robust methods are designed to defend against action perturbations, which none of these test conditions uses. Nevertheless, SMM-AC still has the higher mean under every perturbation, and it loses less return in each case. Its spread across seeds is also about half theirs or less.

We will include this result in the main body of the paper soon, we feel that it adds a useful and interesting contribution to our work. Thank you so much for bringing these works to our attention!

[1] Tessler et al. Action robust reinforcement learning and applications in continuous control. ICML, 2019.

[2] Kamalaruban et al. Robust Reinforcement Learning via Adversarial training with Langevin Dynamics. NeurIPS, 2020.

[Q2] Perturbed MuJoCo evaluation. Yes. Fig. 2 evaluates the nominal MuJoCo environments, to establish that SMM-AC is competitive with standard baselines in simulation. We did not perturb the MuJoCo benchmarks. Our robustness evidence comes from the Franka and ANYmal deployments in Table 2, where each trained policy is held fixed while observations or physical conditions change.

App. E provides a controlled shift in simulation: policies trained with narrower domain randomisation are evaluated under the default ranges, which for ANYmal include friction, added mass and pushes (Tables 7, 9, 11 and 12). App. F.1 deploys the same checkpoints on hardware. The revision states the role of each evaluation in Sec. 4 and at the start of Sec. 5.3.

We kindly request that you increase your score if our responses have answered your questions and concerns. Thank you again for your questions!

I thank the authors for the clarifications, additional experiments, and addressing my concerns. I will increase my score.

We sincerely thank the reviewer for acknowledging our efforts and increasing their score.

Reviewer_FNUcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces Soft MellowMax Actor-Critic (SMM-AC), extending soft mellowmax to continuous control through a learned reference policy. Experiments on MuJoCo, a Franka manipulator, and an ANYmal-D quadruped show competitive simulation performance and promising robustness to real-world perturbations.

Critical strengths

Strengths:

Addresses an important question: whether policy regularisation improves sim-to-real transfer and deployment robustness.

Provides substantial hardware evaluation across manipulation, locomotion, and several perturbation types.

Presents a practical algorithm with particularly encouraging results under observation noise.

Critical weaknesses

Theoretical connection: The robustness guarantees motivating SMM are not established for the implemented algorithm with continuous actions and an approximate, slowly updated reference policy.

Limited ablations: The experiments do not isolate whether improvements arise from the SMM objective or stabilisation from the reference network. Reference-update and alternative KL-reference ablations would strengthen the contribution.

Evaluation clarity: Hardware episode counts, checkpoint selection, and the aggregation of retention statistics need clarification. Excluding PPO and FastTD3 from perturbed ANYmal deployment is understandable for safety, but limits direct hardware comparisons.

The hardware results are promising, but stronger ablations and clearer evaluation statistics are needed to support the central explanation for the gains.

Questions for authors

Which robustness guarantees survive approximation of the reference policy?

How sensitive are results to the reference learning rate and α?

How many hardware trials were conducted per seed, and how were retention statistics and safety-screening results computed?

4 comments

We thank the reviewer for the thoughtful feedback.

[W1, Q1] What survives reference approximation? We agree with the reviewer that the robustness guarantees for soft mellowmax established directly by [1] and indirectly by [2] are stated in discrete state-action spaces, not in continuous spaces, as is the case in robotics. While this is the case, we also know that the robustness guarantees of entropy-regularised RL survive conversion to continuous state-action spaces, see [3], and that the central mathematical tools of these robustness proofs work in both continuous and discrete settings, namely the Legendre-Fenchel dual. Thus, we would like to refine our reliance on these prior works: these works served as inspiration to investigate the real-world robustness of regularised continuous control algorithms.

Second, to answer your question on which guarantees survive the approximation of the reference policy in SMM-AC: we agree that global robustness guarantees deviate under function approximation. However, as is standard in Deep Reinforcement Learning literature (e.g., SAC, TRPO), theoretical guarantees are established at optimality and under exact representation. It is well understood that function approximation, stochastic optimisation, and delayed target networks cause algorithms to deviate from their idealised mathematical guarantees during the training process. We do not claim that global robustness is strictly preserved throughout joint network optimization. In particular, robustness guarantees are almost always stated with respect to the optimisation problem itself (i.e., the optimal regularised policy will be robust), not with respect to function approximation or iterative optimisation. Thus, a more standard framing would be to say that at optimality, a regularised policy will be robust to a certain set of perturbations. Notably, this statement holds for SMM in (at least) discrete spaces.

We can, however, provide an exact statement for a fixed learned reference: the KL objective retains an interpretation in terms of reward uncertainty. What changes is the uncertainty set associated with that reference.

Fix a state, a normalised reference density μ\mu, bounded QQ, and ω>0\omega>0. For policies π≪μ\pi\ll\mu with finite KL(π∥μ)\mathrm{KL}(\pi\|\mu) and well-defined expectations, define:

\qquad \mathcal D_\mu=\{\Delta:\mathbb E_\mu e^{\omega\Delta}\leq1\}$$ Then $$F_\mu(Q) =\sup_\pi\left[\mathbb E_\pi Q-\frac{1}{\omega}\mathrm{KL}(\pi\|\mu)\right] =\sup_\pi\inf_{\Delta\in\mathcal D_\mu}\mathbb E_\pi[Q-\Delta]$$ To see this, let $\nu_\mu\propto\mu e^{\omega Q}$. Expanding the KL gives: $$\mathbb E_\pi Q-\frac{1}{\omega}\mathrm{KL}(\pi\|\mu) =F_\mu(Q)-\frac{1}{\omega}\mathrm{KL}(\pi\|\nu_\mu)$$ Also, the entropy variational inequality gives $\mathbb E_\pi\Delta\leq\mathrm{KL}(\pi\|\mu)/\omega$ for every $\Delta\in\mathcal D_\mu$. Equality is attained at $\Delta=\log(\pi/\mu)/\omega$, or approached where the policy density vanishes. This proves the identity for continuous actions. It also quantifies actor approximation: its expected regularised value falls below $F_\mu(Q)$ by exactly $\mathrm{KL}(\pi\|\nu_\mu)/\omega$. There is a conditional reference-error bound. On the bounded action domain, if the learned reference $\hat\rho$ and exact reference $\rho\propto e^{\alpha Q}$ satisfy $\|\log(\hat\rho/\rho)\|_\infty\leq\varepsilon$, then: $$|F_{\hat\rho}(Q)-F_\rho(Q)|\leq\frac{\varepsilon}{\omega}$$ This follows by bounding the density ratio inside the integral between $e^{-\varepsilon}$ and $e^\varepsilon$. The reference loss alone does not certify this uniform bound. These statements apply with the networks fixed and a normalised reference density. They do not prove convergence of the jointly trained networks or preservation of the larger uncertainty set associated with exact SMM. Here $\Delta$ represents a signed change to the reward at the fixed state; the identity does not guarantee robustness to arbitrary observation or dynamics changes. The physical experiments provide empirical evidence for those perturbations. We have added these identities and the conditional reference-error bound, together with their assumptions, to App. B. We have also revised Secs. 2 and 3 and the discussion in Sec. 5.3 to distinguish this analysis from the empirical evidence under observation and dynamics perturbations. [1] Jiralerspong, Marco, et al. "Discrete compositional generation via general soft operators and robust reinforcement learning." ICLR 2026 [2] Brekelmans, Rob, et al. "Your policy regularizer is secretly an adversary." TMLR 2022 [3] Eysenbach, Benjamin, and Sergey Levine. "Maximum entropy RL (provably) solves some robust RL problems." ICLR 2022

[W2] Separating the objective from the reference update. The effects of the objective and the slowly updated reference network were difficult to isolate experimentally since we were unable to get soft mellowmax to work with other setups. For instance, we tried to avoid learning the reference network, which caused significant instabilities in training, to the point where we couldn’t train a network to complete a single MuJoCo task. We would almost reverse the question to say that the stability of the reference network was instrumental in getting SMM-AC to work. Ablating the reference network leads to poor training dynamics, while changing the reference policy changes the value function, so we would no longer be doing soft mellowmax, but another type of RL entirely. In addition, in our early experimentation, we discovered that slowly updated reference networks were necessary because if their learning rate was too high, the actor would fail to learn. We hope that these points clarify why we made these design choices, and how each of them was necessary to ensure that SMM-AC could actually be functional.

[Q2] Sensitivity to the reference learning rate and α\alpha Appendix C sweeps the reference learning rate from 10−610^{-6} to 8×10−58\times10^{-5}. Table 4 reports the selected value for each task. These selections vary, but do not establish a broad stable range within each task. The only claim we feel comfortable making here without broader experimentation is what we observed throughout our experiments: if the reference learning rate was too high, a race condition between the actor and the reference policies would ensue, leading to either 1) learning instability, or 2) failure to learn. Thus, we elected to keep the reference learning rate rather small, which provided improved stability.

We fix α=1\alpha=1 throughout. Table 2 varies ω∈{400,800,1200}\omega\in\{400,800,1200\}, which controls the KL coefficient. It does not measure sensitivity to α\alpha, which controls the exact reference target. We elected not to vary α\alpha because in our experiments, it did not make much of a performance difference. Thus, keeping α=1\alpha = 1 was a design choice to reduce the number of hyperparameters without adversely affecting performance.

[W3, Q3] Hardware evaluation protocol. Section 5.1 specifies five independent training seeds per configuration. Each trained policy, its command, and the low-level controller remain unchanged across deployment perturbations. ANYmal evaluation trials run a 20-second episode with commands as joint desired velocity and success means remaining upright for the full duration. Table 13 provides details about task errors for each method. For each seed and each perturbation condition, we ran 20 episodes on the hardware.

Section 5.1 permits deployment under a perturbation only when a configuration fails in fewer than 20% of simulated episodes under that perturbation. The rule was fixed in advance and applied to every configuration. PPO and FastTD3 did not pass for the ANYmal perturbations. PPO passed 45% to 68% of simulated episodes on ANYmal tasks across different perturbations and FastTD3 passed 35% to 65% of simulated episodes. Their missing entries are unobserved hardware outcomes, not measured hardware failures. We set this rule due to safety reasons given how strong and potentially dangerous our robot is.

We kindly request that you increase your score if our responses have answered your questions and concerns. Thank you again for your questions!

With the discussion period ending soon, we would appreciate your feedback on our rebuttal and whether it addresses your concerns. We would be happy to clarify any remaining questions before the deadline.

Reviewer_FNUcOpenReview ↗

Post Rebuttal

I thank the authors for their detailed feedback, and my concerns are largely resolved. I decide to increase my score.

Reviewer_WJcJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Soft Mellowmax Actor-Critic (SMM-AC), an off-policy algorithm that regularizes SAC’s objective with respect to a reference policy alongside utilizing a soft mellowmax value function. The authors evaluate their modified actor-critic algorithm, SMM-AC, on several Mujoco and IsaacLab environments. On most environments, SMM-AC performs comparably to SAC, but when the authors measure robustness by evaluating performance after perturbing observations and environments, SMM-AC retains most of its baseline performance and does not degrade as sharply as other methods.

Critical strengths
  1. The authors provide a well-motivated adaptation of SAC that utilizes Soft Mellowmax value functions to improve stability and robustness during training. Their justification for adapting Soft MellowMax is clearly explained and easy to follow.
  2. The authors evaluate their approach on a wide variety of Mujoco tasks, and demonstrate sim-to-real transfer and policy robustness using four matched IsaacLab environments. Their robustness evaluations are especially thorough, perturbing a variety of environment parameters alongside observations to demonstrate SMM-AC’s relative performance stability.
Critical weaknesses
  1. SMM-AC’s performance remains comparable to SAC in most of the Mujoco experiments, but it exhibits high variance and can occasionally degrade severely. Given that the Mujoco experiments cover a wider range of embodiments and tasks than the IsaacLab sim-to-real experiments, it is unclear whether SMM-AC’s performance can reliably generalize to harder tasks.
  2. On the matched IsaacLab environments, the performance gap between SMM-AC and SAC in the “none” setting is only sizeable on the Franka lift task, where SAC exhibits high variance. On the other three tasks, the performance differences in the “none” setting are often within a few points of performance. Although the authors evaluate robustness across a variety of perturbations and show that SMM-AC performance generally remains stable, SAC performance on the two locomotion tasks consistently degrades by more than 3 points only in the proprioceptive noise setting. In my opinion, the choice of matched environments and perturbations made in this paper may be too easy, since the robustness experiment results do not clearly suggest that SAC is adversely affected by most of the perturbations. This undermines the author’s central claim that SMM-AC is helpful for improving robustness over existing actor-critic algorithms.
Questions for authors
  1. Can the authors provide any evidence that the improvements in robustness provided by SMM-AC generalize to other manipulation tasks? While the strongest results seem to be on the Franka lift task, most of the tasks reported in the paper are locomotion tasks (across both Mujoco and IsaacLab).
  2. While I understand that simulation performance improvement is not a focus of the paper, examining the Mujoco results (and the author’s explanation for them) suggests that InvertedDoublePendulum-v4 degrades due to instability in the reference policy. Have the authors examined any heuristics or alternative choices for the reference policy outside Equation 6, to perhaps mitigate this instability? Can the authors provide any insights into the conditions that may cause instability (e.g. poor initial performance of the Q function)?
  3. SAC seems to exhibit unusually high variance on Franka-Lift’s “none” task performance compared to the perturbed settings. Can the authors provide any insights into why performance stabilized in the perturbed settings?
7 comments

We thank the reviewer for their careful review.

[W1] MuJoCo variance and generalisation. SMM-AC collapses late in training on InvertedDoublePendulum. We report this as a limitation and discuss the cause and a simple mitigation under Q2. On the other seven tasks, including Humanoid, it performs comparably to SAC (Fig. 2).

The robot experiments use a different implementation, built on FlashSAC instead of CleanRL. App. D tests it on five MuJoCo tasks, with five seeds and three million steps each. It performs comparably to FlashSAC under the selected fixed-temperature settings and trains stably without the learning-rate annealing the CleanRL version needed. Since several implementation details change together, this comparison does not establish how much of the earlier variance was due to the implementation. The InvertedDoublePendulum experiment under Q2 tests one specific change to the updates.

We have revised Sec. 4 to distinguish the CleanRL and FlashSAC results and to state that this comparison does not isolate the cause of the earlier variance.

[W2] Perturbation difficulty and the size of the robustness benefit. We agree that several physical perturbations cause only modest losses for SAC. The clearest evidence is under observation noise, where the difference remains after accounting for nominal performance. At ω=800\omega=800, Table 2 gives:

TaskSAC, nominal →\rightarrow noisySMM-AC, nominal →\rightarrow noisy
ANYmal flat25.2 →\rightarrow 20.525.9 →\rightarrow 24.7
ANYmal rough19.3 →\rightarrow 15.320.8 →\rightarrow 19.3

The corresponding losses are 4.7 versus 1.2 on flat terrain and 4.0 versus 1.5 on rough terrain, so the gap under noise is larger than the nominal gap.

The pattern extends across the reported regularisation strengths. SMM-AC has the higher perturbed mean in 35 of the 36 comparisons with SAC at the same ω\omega in Table 2. The exception is Reach at ω=1200\omega=1200, where its nominal mean is also lower. The ratio of perturbed to nominal mean is higher for SMM-AC in all 36 comparisons. These comparisons describe reported means across related conditions, not 36 independent significance tests. They support an advantage within the tested conditions, with the strongest evidence under observation noise.

The revised Sec. 5.3 states that several physical perturbations cause only modest losses for SAC, quantifies the effect of the cube perturbations on SMM-AC, and reports both comparisons above.

[Q1] Evidence beyond the Lift task. Reach provides a second manipulation result. At ω=800\omega=800, nominal returns are close at 0.61±0.030.61\pm0.03 for SMM-AC and 0.58±0.090.58\pm0.09 for SAC. With observation noise, they become 0.56±0.020.56\pm0.02 and 0.40±0.030.40\pm0.03. The benefit therefore appears in reaching as well as in grasping and lifting. These two tasks do not establish robustness across more complex manipulation tasks, and Pusher (Fig. 2) adds only nominal manipulation evidence. The revised discussion in Sec. 5.3 accordingly restricts the robustness conclusion to the tested settings.

[Q2] Alternative references and the cause of instability. The reference network is trained with its own objective (Eq. 12) to approximate a density proportional to exp⁡(αQ)\exp(\alpha Q), and its log-density enters both the critic target (Eq. 9) and the actor loss (Eq. 11). The collapse occurs late in training, after InvertedDoublePendulum has reached near-maximal return. Our diagnostics do not point to a poor initial critic: the action-value estimates remain stable while returns are near maximal for several hundred thousand steps before diverging, and delaying reference-network updates until 100k steps does not prevent the collapse.

Instead, they point to the reference log-density. Late in training the actor becomes nearly deterministic, and in some states its mean drifts far from the reference. Actions sampled by the actor then receive very low reference density. Because log⁡πref\log\pi_{\text{ref}} is unbounded below, this produces very large log-ratios in the critic target. In our runs, these target spikes preceded critic divergence in every collapse.

To test this, we replaced log⁡πref(a∣s)\log \pi_{\text{ref}}(a\mid s) with max⁡{log⁡πref(a∣s),−5}\max\{\log \pi_{\text{ref}}(a\mid s),-5\} wherever it enters the critic target and the actor loss. Everything else matched the paper configuration. Across four seeds the target spikes and critic divergence disappeared, and no run collapsed over 1M steps. This is a stabilisation heuristic that changes the updates. The clipped log-density is not generally the log of a normalised reference policy.

Sec. 4 reports this result, and App. C (Fig. 5) now gives the diagnostics, the bound, the number of seeds and training steps, and identifies the clamp as a modification of the original updates.

[Q3] Lower SAC variance in the perturbed Lift evaluations.

The policies remain fixed across perturbations, with no retraining or adaptation. The smaller standard deviation therefore concerns variation in deployment returns, not improved training stability. It also accompanies lower mean performance: at ω=800\omega=800, SAC changes from 125.8±69.9125.8\pm69.9 nominally to 68.5±7.868.5\pm7.8 under observation noise with σ=0.005\sigma=0.005.

The aggregate results do not establish why the spread decreases, and we cannot conclude that the perturbation stabilises the policy. The absolute return under this noise remains higher for SMM-AC, at 120.9±4.0120.9\pm4.0. This comparison does not depend on a nominal-return denominator. Retention estimates do depend on it and should be interpreted in light of the large nominal variability.

The revised discussion of Table 2 in Sec. 5.3 (see text in red) distinguishes deployment variability from training stability and notes the effect of nominal variability on retention.

We kindly request that you increase your score if our responses have answered your questions and concerns. Thank you again for your questions!

With the discussion period ending soon, we would appreciate your feedback on our rebuttal and whether it addresses your concerns. We would be happy to clarify any remaining questions before the deadline.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Real robot experiment videos in supplementary material

We thank the reviewers and the Area Chair for the discussion.

In response to reviewers' request, we added three Franka Lift hardware videos to the supplementary material. One SAC rollout shows the gripper closing beside the cube and rising empty. The other SAC rollout and the SMM-AC rollout show correct cube lifts. All three use the nominal cube and illustrate the grasp outcomes discussed in our response to reviewer WJcJ.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.