- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a graph-based approach for identifying critical decision points within a dataset of sequential-decision-making trajectories, and proposes to apply “single-step reinforcement learning” to train a policy only at these points, which is more data and compute-efficient than standard RL because the policy does not need to be rolled out for complete trajectories. The paper describes a method for defining “local rewards” based on sampling multiple potential responses from the policy. It evaluates its approach on AlfWorld and AIME, which are appropriate LLM decision-making benchmarks. The main baseline, “ATLaS + Single Step RL” is in essence an ablation of their step selection mechanism; swapping in a different mechanism proposed in a prior work that treats a different problem setup. All approaches perform worse than standard GRPO (GRPO is not single-step RL, so it may be more data/compute intensive).
Critical strengths
The paper reports evaluation results on well-known benchmarks, and includes comparison against GRPO, a well-known approach to training language models.
Critical weaknesses
1.The most important part of the paper is the graph-based method to determine the key decision points, but the graph construction process (e.g. how to determine the set of vertices given a dataset of trajectories) is not described in the main paper. Near the start of section 3.2 the paper says “Abstract states form nodes of G_q=(V_q,E_q)” and the end of section 3.1 says “The selector uses abstract states x_t = phi(h_t)” but this mapping phi, which I assume quotients the trajectories into abstract states, is not described in the paper. Understanding how the graph is constructed is crucial to understanding the method.
-
The utility of the method is not well motivated. It does outperform the so-called baseline “ATLaS + Single Step RL”, but “ATLaS + Single Step RL” is introduced in this paper. GRPO outperforms all approaches. The authors need to motivate their method by either 1. citing a previous “single step RL” paper (if one exists) and showing that their approach is better 2. exceeding the performance of orthodox approaches (like GRPO) or 3. coming close to the performance of orthodox approaches (like GRPO) with much less data or computation.
-
The paper’s presentation is weak. For example, the start of section 3 does not start with a clean succinct problem setup, instead it builds up the method bit by bit, with minimal signposting, making it hard to follow (especially since some parts, like how the graph nodes are determined, are not explained). Also, the “Baselines” section 4.3 mentions ReAct and Reflexion but these are not compared against empirically. The authors should move these to “Related Work” if they are not going to provide an empirical baseline.
Questions for authors
- How are the nodes of the graph determined? What is the mapping phi?
- Why is the “local reward” scheme well-motivated? It seems to give high rewards to syntactically correct generations that are similar to those that would be emitted by the “old” policy. But is there anything that encourages them to lead to desirable behavior in future steps?
- Are there previous works that treat “single-timestep RL” as you have set it up? How do they define their reward functions? If there do not exist any, can you provide evidence that this paradigm is useful?