- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper identifies an attribution gap in counterfactual credit assignment which occurs when later actions effect the Shapely values of earlier ones. The issue is demonstrated in a simple task. The authors then propose RECAP, which adds an estimate of this gap to the causal attributions (Shapely values) used for learning updates. Experiments in three domains show RECAP improves over a baseline with no compensation (CSCA), and performs well compared to PPO. In evaluations at the end of training, RECAP has the highest return of a selection of baseline credit assignment methods.
Critical strengths
- This paper exposes and explores an interesting problem that is relevant to an important challenge in RL (credit assignment).
- The experimental comparison to CSCA is robust and strongly supports the method.
Critical weaknesses
- I found the paper extremely to difficult to follow. Much of the notation is undefined at first use (e.g. , ), and referred to inconsistently (e.g. is a core component that should be clearly named and referred to consistently throughout the paper). In my opinion, important preliminary concepts are not discussed in sufficient detail, and doing so could help clarify concepts, notation and definitions used throughout the paper (e.g. MDPs & policy gradient methods, Shapley values, counterfactual Shapely credit assignment).
- The comparison between RECAP and CSCA is 'fair' and provides strong evidence for the method, but I believe all of the other baselines get significantly fewer samples? E.g. the counterfactual rollouts ('counterfactual suffix simulation') used by both RECAP and CSCA provide additional information, based on privileged rollouts, that is unavailable to PPO. If I'm interpreting Table 10 correctly, CSCA & RECAP get an additional 50M samples in RoundTrip for example. This needs to be clearly declared, justified, and accounted for in the main body and experiment results, ideally by ensuring equivalent experience and rollouts for baselines (e.g. PPO) and RECAP.
- Presentation of experiment results: Figure 2 uses different y-axis scales on each subplot, which makes it harder to interpret the results. I think linear scales throughout would be more appropriate, or at least reproduce the figure with linear scales in the appendix. Also, I'd like to see learning curves for all the baselines (QCA, HQCA, H-DICE, RUDDER), in the appendix would be fine. The final performance of some of the baselines is very similar to RECAP in Table 1 (within reported standard deviation in some cases), so including the full learning curve would allow for a more complete comparison with the baselines.
Questions for authors
- The three domains have all been designed/selected to demonstrate the impact the gap between causal attribution and policy optimisation has on learning. What would you expect the effect of RECAP to be on domains that do not have this gap? Is it zero? It would be interesting to see results for a 'null-case' experiment in an environment where you do not expect RECAP to have any impact.
Thank you for the detailed response and the additional experiments.
The simulation budget experiments are a thorough and open response to my concern about unequal budgets. They show the gains over PPO do not hold when the total budget is matched, addressing my fairness concerns, but raising sample efficiency concerns. Thank you for the analysis of what happens when the attribution-optimization gap vanishes. The sufficient conditions and the null experiment are useful to see.
The response and new results do not change my score. Overall, exposing and addressing the issue in how CSCA's credit is used is a useful contribution, but this was already reflected in my original score.