- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents an extension of soft mellowmax policies from discrete to continuous action spaces, and studies the sim-to-real benefits of the inherent regularization properties of soft mellowmax. The paper includes a mathematical derivation of their proposed policy which is contrasted with mathematical properties of SAC. Similar to DDPG, the proposed method uses a target policy network in addition to target Q network to approximate the intractable reference policy involved in soft mellowmax policy updates. On several Mujoco benchmarks, the proposed method performs mostly comparably to SAC as well as PPO and TD3. More interestingly, on sim-to-real experiments in two domains (block manipulation and quadruped locomotion), the proposed method outperforms these baselines, including in the face of adversarial perturbations such as object mass changes and rough terrain. Based on this, the paper concludes that the inherent regularization of soft mellowmax improves policy robustness.
Critical strengths
-
The paper is mostly well-written and easy to follow, including the technical parts. It could be an accessible reference for researchers interested in RL regularization techniques.
-
The experimental results are rather compelling, showing clear advantages of the method for sim-to-real transfer and robustness. The experiments include multiple repetitions and report standard deviations to quantify statistical significance, which is appreciated.
Critical weaknesses
- The approach may not be particularly novel. For example, there is an existing approach from early 2025 (reference below) which was not cited by this paper. However, it also extends soft mellowmax to continuous action spaces, and uses ideas from DDPG which overlap with this paper (in particular the use of target policy networks).
Z. Shang, R. Li, C. Zheng, H. Li and Y. Cui, "Relative Entropy Regularized Sample-Efficient Reinforcement Learning With Continuous Actions," in IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 1, pp. 475-485, Jan. 2025, doi: 10.1109/TNNLS.2023.3329513.
-
The paper does not include any anonymous code links or supplementary material, which undermines confidence in reproducibility and veracity of the claims.
-
Certain issues with the presentation: First, the title seems overly broad. This paper demonstrates sim-to-real benefits in the very specific context of the proposed method, as opposed to some general principles about regularized RL more broadly. Second, the descriptions of the results verge on LLM slop. The authors acknowledged LLM use, but this does not change the fact that the LLM's writing is poor. Perhaps in an attempt to save space, there are too many brief/broad/vague statements like "X loses nothing" (line 457, 461, etc.). I am starting to see very similar and frustrating phrasing like this in many papers, reflecting a sad collapse of our field towards a stylistic LLM monoculture that is mentally taxing on the reader (at least, on this reader). I encourage the authors to be more precise and devote more energy towards cultivating their own unique voice and aptitude for communicating results.
Questions for authors
Can the authors elaborate on what differentiates their work from the existing literature I cited above?
4 comments
The rebuttal does mostly address my concerns:
- The description of differences from CDPP is reasonable and indicates some level of novelty in this work.
- While I haven't carefully checked the anonymous code link, since it is posted here and these discussions will be made public, readers are in a better position to try reproducing the results and check the veracity of this paper's claims for themselves.
- The wording in 5.3 is overall more palatable to me now. There are still some lines that sound very LLM-like, such as line 463 ("Every method has its largest loss under observation noise, and this is also where the methods differ the most"). But since this is a stylistic issue and disclosed by the authors since the beginning, I won't harp on it any longer.
As such I'm leaning towards increasing my rating. I still hesitate for two reasons: First, I'm the only reviewer on the positive side - although given that another review was flagged as AI-generated, I am just going to focus on my own review and its rebuttal. Second, the rebuttal's comparison with CDPP is still not integrated into the latest PDF version of the manuscript. If the authors integrate it in another posted revision, I'll consider increasing my rating by one point.
Also, a couple more minor things I'd like the authors to address:
- Line 337: Is "fails fewer than 80%" supposed to be "fails fewer than 20%"? 80% seems like a rather liberal failure rate for safety considerations.
- Line 487: Typo, missing "T"
Thanks for following up. We added the comparison with CDPP in Section 2, covering the actor, reference policy, value operator, and experimental scope.
The reviewer is right about the safety threshold. The criterion used was fewer than 20% failures. "80%" was simply a typo. We corrected this in Section 5.1, fixed the missing "T", and removed the sentence you pointed out in Section 5.3 as the rest of the paragraph already covers that point. The revised PDF reflects all of these changes.
Based on this final revision I increased my rating.
We thank the reviewer for their helpful review and their interest in our submission.
[W1, Q1] Relation to Shang et al. Thank you for bringing this prior work to our attention, we did not come across it before. You are correct that it is conceptually similar to our work in a number of ways: 1) using the mellowmax operator for its stability, 2) using an actor-critic algorithm for its sample efficiency, and 3) estimating intractable softmax policies. However, there are significant differences between this work and what we propose. These are listed below: Base algorithm: we make use of the SAC actor-critic algorithm, which differs fundamentally from DDPG-style algorithms. First, DDPG uses deterministic policies, meaning that the policy function selects an action, i.e., at=π(a∣s). SAC-style algorithms sample each action stochastically from the policy function, i.e., at∼pi(⋅∣s). This means that we must learn the entire intractable softmax policy, not just generate samples from it, as in DDPG-style algorithms. Second, DDPG algorithms compute policy gradients on a reward (or Q-value) maximisation loss. SAC-style algorithms instead update policies computing the gradient on a KL divergence loss between a target policy and the current policy. Differences in regularisation leading to different values and policies: we regularise the actor against the policy softmax(αQ), using the current Q-value network. This work regularises against a previous policy iterate, essentially doing the same thing as TRPO and PPO, in a sense. Thus, the resulting value function (soft mellowmax vs. mellowmax) and the optimal policy are different. Experimental scope and tested claims: Our paper focuses primarily on sim-to-real transfer and how it can be improved with regularisation. This work focuses primarily on proposing a new DDPG-style algorithm for improved learning stability. Experimentally, both papers test on MuJoCo, but we test our policies on real robots. Handling intractable integrals: this work estimates intractable integrals by computing Monte Carlo estimates of the normalisation terms in the policy. While we did test a similar approach while developing our work, we ultimately decided to learn both the actor and the reference policies because this yielded more stable learning, and better performance.
Our contribution is the particular SMM actor-critic formulation and its evaluation on physical manipulation and locomotion systems under deployment perturbations. We will acknowledge the overlap with CDPP and explain these differences explicitly in the related works section.
[W2] Code and supplementary material. Our anonymised code is available at https://anonymous.4open.science/r/SMM-AC-08E8/
We will release it publicly after the review process.
[W3] Title and presentation. We agree that the original title was too broad and have changed it to “Soft Mellowmax Actor-Critic for Sim-to-Real Transfer”. The abstract, introduction and conclusion now match this scope: claims about regularised methods in general are limited to the locomotion tasks, where both SAC and SMM-AC outperform PPO and FastTD3.
We also agree that phrases such as “loses nothing” were imprecise. Sec. 5.3 now states that the cube perturbations change SMM-AC's mean return by less than 4%, and compares its return under lower friction on rough terrain with its nominal mean. We have also replaced the remaining qualitative summary of retention with counts from Table 2.
We kindly request that you increase your score if our responses have answered your questions and concerns. Thank you again for your questions!