- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Actor-critic methods can produce highly oscillatory actions. Therefore, in this paper, the authors propose DAMPER, combining actor gradient and a modified temporal-consistency gradient. Experiments show that action oscillation is indeed reduced, and DAMPER performs better than existing baselines.
Critical strengths
-
The presentation is very clear and easy to follow. I enjoy reading Figure 1, because it makes me fully understand the method of this paper within minutes. Method and experiment sections are also well written.
-
The authors provide open source code, and it appears well organized, clean, and useful.
Critical weaknesses
-
The novelty is incremental. Combining native actor loss with temporal consistency loss is well studied. Also, conflict-conditioned projection is also from existing work, such as PEGrad. Therefore, this paper mainly propose an engineering method to combine existing strategies. Also, I am not sure why the authors don't include PEGrad in the main results but in a separate table.
-
I have one addressable concern about the experimental design. While I am not an expert in this exact subfield, I understand that there is a trade-off between return and smoothness. As one can see in Table 1 and 2, it is often the case that DAMPER is only better on one of smoothness / reward. Therefore, the authors should plot a smooth-return curve for each method (by altering hyperparameters) and compare which method curve is better than others. This would be much better than comparing at only one single point. I understand plotting such a curve is extremely costly, so the authors may feel free to only pick one algorithm and task.
Questions for authors
I will increase my score to 3 if the authors add the smooth-return curve, and from the curve, the strength of DAMPER is evident.
Thank you for the detailed rebuttal for my concerns. The rebuttal clarifies the reproducibility details and provides encouraging matched-return results on Hopper. However, DAMPER still cannot match the returns of the smoothing baselines on Walker. I maintain my original rating of 2.