- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper proposes a new rollout allocation method for exploration under budget constraints, specifically aimed at multi-interaction agentic tasks. The idea is to group actions from a state based on the one-step effect of the immediate next action and then continue only one uniformly sampled trajectory from the group. The unbiased estimator of the REINFORCE-style gradient is obtained simply by reweighing trajectories by the group size. The paper designs a controlled experimental protocol that aims to isolate the impact of the proposed exploration method under exact budget constraints. Under this protocol, the method improves task success over direct sampling on ALFWorld and WebShop. Additional experiments also show gains in an online task-start setting and across different policy-optimization algorithms.
Critical strengths
The critical strength of the paper is the choice of experimental protocol and baselines. The first set of experiments isolates the impact of the proposed roll-out selection method under a budget cap, comparing it against direct sampling and parser grouping. Then the second set of experiments shows that the approach also works across tasks and different training algorithms. These two stages separate the effect of the proposed method from simple direct sampling and other complementary methods of rollout selection.
Critical weaknesses
Writing: As a human reviewer, I find the paper extremely difficult to read, especially the experiments section.
- The writing does not distinguish between minor details and critical points. Moreover, the discussion of results merely reads the table without any added narrative or interpretive commentary. For example, the following discussion should be split into several longer sentences, and there should be an attempt to explain why on ALFWorld, the parse-based grouping works well: "The four primary Holm-corrected 7B tests support ECE over tuned direct sampling in both environments and over the parser on WebShop; the ALFWorld parser-relative interval includes zero, leaving that comparison inconclusive (Table 9, Appendix C.3). These controls fix continuation states and evaluate policies from task starts. Online validation uses current-policy prefixes: ECE gains 2.9/5.2 points over direct sampling and 1.3/2.2 over the parser on ALFWorld unseen/WebShop SR (Appendix C.2)."
Many similar instances appear in the results section.
Empirical advantage reduces in more general setup of Table 2: When compared across base host methods, the advantage of ECE is much smaller compared to other roll-out selection methods. This raises the question of significance: whether the added complexity of state restoration during training is justified when other arguably simpler rollout-selection methods also provide substantial performance gains.
Questions for authors
- My interpretation of the rows named +selection_method in Table 2 is that each row applied the "selection_method" in the base host algorithm. In that case, why were the other collector methods used in Table 2 not included in Table 1 as baselines?
- On a similar note, does ECE add value when combined with adaptive branching or prefix reuse? Specifically, since ECE addresses which candidate continuations to retain, whereas ARPO and Tree-GRPO address branching and prefix sharing, could you use ECE at branching points selected by ARPO? This will help understand if ECE retains its advantage on top of an optimized state selection method.
- How much efficiency is gained simply by consolidating parse-error and no-state-change into a group and retaining all other trajectories that don't fall into these two groups?
The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.