- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper considers the scenario where external information from an oracle is available that can be useful in policy optimization, and proposed a way to perturb the advantage function to improve learning efficiency. The authors show that only action-dependent perturbations can affect policy gradients, and empirically evaluate two ways of designing such perturbations (i.e., manual and local counterfactual). Results show that using these perturbations can speed up learning.
Critical strengths
The paper is overall clearly written, and the empirical evaluations are well presented (i.e., uncertainty quantification, detailed algorithms/hyperparameters). The cartpole example also provides useful intuition for understanding the contribution.
Critical weaknesses
- The theoretical result (contribution 2) that state-dependent functions do not change the expected policy gradient is well-known.
- The motivation is a bit unclear to me, why should we augment the advantage function rather than doing reward shaping? It appears to me that TD-CF can be viewed simply as changing the reward function, and, considering the fact that the policy gradient becomes biased in both cases, I don't see why we should prefer .
- The ability to "teleport" to an arbitrary state is a very strong assumption, and I'm not sure how practical this is beyond simulators. Moreover, it is not clear to me whether CF guidance offers benefits beyond providing additional advantage estimates through rollouts of alternative actions. Finally, the empirical comparison is not entirely fare considering that CF uses more rollouts.
Questions for authors
- When should we prefer perturbing advantages over reward shaping?
- Do the reported training steps include all counterfactual simulator transitions? If not, how do CF-based methods compare with PPO under matched total interaction budgets?