- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors introduce action shaping, a mechanism to add a learned offset to actions. This action head is progressively "absorbed" by the main policy network as it's weight is decayed over training. They tests their method on SAC, TD3 and PPO on many continuous action game but do not compare it to the baselines (without action shaping), and the experiments on Atari do not allow me to conclude if their method is worth the extra computational complexity.
Critical strengths
The paper explores action shaping, and tries to understand how to absorb this shaping into the main policy (I don't know why the policy could not stay with the learned bias).
Critical weaknesses
I have absolutely no clue why are the authors doing this in the first place. The paper lacks strong motivation, it reads a lot like "reward shaping exists, why not action shaping?". The authors do not identify a problem in which this method would help. What is the motivation? The motivation behind reward shaping is e.g. to guide the policy in situations where reward is too sparse.
They also do not compare to any baseline, not even the base method (SAC, PPO, TD3) on which they plug in action shaping. The experiments do not show any base performance impact, but only that removing the action head is not too costly. Why should it be removed? For edge devices? Also, your first question should target base performances. Before showing that removing the action-offset does not impact performances:
- Show that the policy with action offset has better performances than the baselines (with clear second order metrics).
- Motivate why the action offset should be dropped.
I don't say that there is no motivation to such a work, it's just not explicitly outlined in the paper.
On clarity: In Figure 1, place that alpha is going to decay over time. Would tremendously help understanding faster this first Figure (and the whole paper).
Questions for authors
- What is the motivation behind action shaping? What existing problem are you solving?
- Why is there no comparison to the baselines TD3, SAC and PPO in terms of reward?