- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes consensus-breaking re-rollout (CBR). It finds consensus anchors (shared states in failed rollouts, where the same action was chosen), to then replay those anchors and sample different actions. The paper reports gains over GiGPO on ALFWorld and WebShop.
Critical strengths
(1) The idea is simple, compelling and well-motivated: Identifying common state-action pairs that led to failure and selecting different actions seems to be a fitting approach to inform exploration.
(2) The posed question is highly relevant: Identifying how to spend extra rollout budget to maximize information gain is a relevant and timely question.
Critical weaknesses
(1) This submission does not report the experimental setup and the origin of their empirical results to a sufficient level of clarity. This submission further does not provide code. See questions for further specifics.
(2) The authors propose a method, that is limited to a very specific setting (anchors are comparable; there is an alternative action that is better than the selected action; terminal rewards are sufficiently meaningful) and targets a very specific failure case (repetition of a single, non-useful action in a given state). While mentioned in the paper, I feel that this aspect may be more limiting than the paper suggests, and that it is not sufficiently discussed (also, there is no dedicated Limitations section). In consequence, I am concerned that the proposed solution may lack the generality or wide applicability required to meaningfully advance the field.
Questions for authors
(1) Could the authors please provide the link to an anonymous GitHub repository that contains their source code to enable the full reproducibility of all results?
(2) Could the authors please clarify which of their reported results were generated vs. reported from other work? Please, also elaborate what "source-reported" means in this context.
(3) Could the authors please elaborate which variant of GiGPO is used, normalization with or without standard deviation, and why?
(4) The formulation of revisits seems to allow revisitation within the same trajectory. How often did that occur in practice? Would it make sense to limit counts to distinct trajectories, or treat revisitations within a single trajectory and across trajectories differently?
(5) Could the authors provide a real example from their experiments that showcases a single instance where a repeated action in a given state stalled progress, and how selection of an alternative method helped?
(6) Could you please elaborate upon the link of consistently choosing an action and the rollout failing? It is not obvious to me that a subsequent failure be attributed reliably to this precise action.
(7) Given the limitations of their method, could the authors please strengthen the argument that their method is sufficiently general and widely applicable?