- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Sequential Equilibrium Policy Optimization (SEPO), a critic-free method for assigning credit to individual turns or chunks in multi-turn LLM reinforcement learning. SEPO shares a common interaction prefix across rollouts, explores alternative actions within a selected chunk, completes the remaining interaction at an exploitation temperature, and applies group-relative advantages only to the selected chunk. It motivates this procedure by treating turns as cooperative agents and analyzing best-response updates toward a subgame-perfect equilibrium. Experiments combine SEPO with GRPO, GiGPO, and DAPO across ALFWorld, WebShop, and seven search-based QA benchmarks, reporting success-rate improvements of up to 11 percentage points.
Critical strengths
SEPO addresses an important problem with an intuitive mechanism: comparing alternatives from the same history makes terminal rewards more informative about the particular decision being trained. Its compatibility with several existing algorithms, without an additional critic or reward model, is practically attractive. The empirical evaluation is reasonably broad, spanning several model sizes, three task families, and an additional vision-language Sokoban experiment; the strongest improvements occur with DAPO. Crucially, the appendix includes meaningful controls beyond the main fixed-update-data comparisons: against a larger-batch DAPO baseline with matched rollout volume, SEPO improves ALFWorld success from 88% to 93% and WebShop success from 80% to 84%. Component ablations, three-seed results for the main interactive environments, and comparisons with AT²PO strengthen the evidence that the rollout design contributes something beyond simply collecting more samples.
Critical weaknesses
The principal weakness is the gap between the theory and the implemented algorithm. Theorem 4.4 assumes independently parameterized decisions, exact best responses, and fair coverage of all histories, whereas SEPO uses a shared neural policy, sampled alternatives, PPO updates, and preferentially sampled histories; therefore, the theorem does not establish convergence or non-interference for actual SEPO training.
Questions for authors
Since HGPO directly addresses historical-context inconsistency and is discussed in your related work, could you compare SEPO against HGPO under matched rollout and wall-clock budgets? Does adding SEPO to HGPO provide further improvements?
7 comments
Q1 About Theorem and SEPO
1.1 Theorem 4.4 does not directly apply to the implemented algorithm
Thank you for pointing out the need to distinguish more clearly between the theory and its implementation. Theorem 4.4 motivates SEPO by providing a theoretical guarantee for finding a subgame-perfect equilibrium (SPE) through best-response updates. It does not directly analyze SEPO itself. Our argument proceeds in three steps. Section 4.1 models multi-turn LLM interactions as a multi-agent game. Section 4.2 presents a best-response method for solving the game within this framework. Section 4.3 then introduces an approximation of these updates for multi-turn LLMs. Thus, Theorem 4.4 establishes the guarantee for the best-response updates in Section 4.2, whereas SEPO approximates this update principle in the multi-turn LLM setting.
The connection between the theory and SEPO lies in the local evaluation objective, how candidates are compared, and where credit is assigned. Given a history and a fixed continuation policy , a local best response compares the conditional values of candidate actions at , given by .
SEPO reflects this principle in three design choices:
- Prefix sharing fixes the starting point for comparison. Candidates in the same group share the same history, so their returns can be compared to assess the current decision at that history.
- Exploration within the target block and a common continuation policy allow candidates to be compared locally. SEPO samples different candidates for the target block and evaluates their continuations using the same parameter snapshot and decoding rule. Suffix validation uses the policy at the validation temperature as the continuation policy, so the comparison reflects the model’s behavior at evaluation time.
- The loss mask directs the update to the target block being compared. Relative returns within the group are used to update the target block, so the training signal acts directly on the decision positions where the candidates differ.
The best-response perspective therefore guides how SEPO organizes rollouts, constructs the local credit signal, and applies the training objective. In this sense, it motivates SEPO and directly informs its design.
We agree that the main difference between Theorem 4.4 and SEPO lies in how they treat parameter sharing. To quantify how far a local update propagates under shared parameters, we conducted an additional experiment measuring how much an update to one turn affects other turns. In WebShop, we started from the same checkpoint of a previously trained model and ran four separate updates, each targeting only one of the four chunks. Holding the histories and response tokens fixed, we measured the KL divergence between the conditional token distributions before and after each update. To isolate the effect of each chunk, we reset the Adam optimizer’s momentum and disabled regularization terms, including the entropy loss, so that each update came solely from the selected chunk. For each update, we simulated the nonzero advantages that can arise with a group size of eight. We averaged the results over 64 repetitions, as shown in the table below.
| Update Chunk | Target KL () | Other KL () | Other Target Ratio |
|---|---|---|---|
| 1 | 3.369 | 1.409 | 0.418 |
| 2 | 3.104 | 1.186 | 0.382 |
| 3 | 2.109 | 1.123 | 0.532 |
| 4 | 2.100 | 1.068 | 0.508 |
For each of the four updates, the target chunk showed the largest mean KL divergence between the policies before and after the update. Mean KL divergence ranged from 2.10–3.37 × 10⁻³ for the target chunk and from 1.07–1.41 × 10⁻³ when averaged across the non-target chunks. Across the four updates, the divergence for the target chunk was 1.88–2.62 times the average for the non-target chunks. These results show that when an LLM shares parameters across turns, an update at a single chunk, history, or turn also affects decisions elsewhere, but the average effect is substantially smaller than the effect on the target chunk. The assumption of independent parameters therefore does not apply directly in this setting. One possible explanation for the smaller effects elsewhere is that the LLM’s representational capacity limits the extent to which updates at different decision points influence one another. Developing techniques to further reduce these effects of parameter sharing is an important direction for future research.
We have revised the wording in Section 4.2 of the main text to clarify the role of Theorem 4.4. We have also added the experimental results in Figure 5(a) and the accompanying analysis in Section 5.5.
1.2 Using sampled alternatives and PPO updates
The best-response problem defines the local optimization objective, and PPO provides a policy-gradient method for optimizing it under a parameterized policy.
For a fixed history and continuation policy , the local objective for the current action distribution is
Over the space of unrestricted action distributions, any maximizer of assigns all probability mass to actions with the highest conditional value. Such a distribution is an exact best response. For a parameterized policy , the same objective has the following policy gradient:
where both the continuation policy and the baseline are fixed during local optimization. An exact best response and a policy-gradient update can thus target the same local value objective. They differ in how they optimize it.
In practice, we use sampled local comparisons to construct PPO’s clipped surrogate objective. At the old policy, the importance ratio is 1 and clipping is inactive, so the clipped and unclipped surrogates have the same gradient. As the policy changes, clipping limits how much certain samples can continue to drive the update. A single PPO update is therefore a step toward optimizing the local value objective, without necessarily attaining its exact maximizer.
1.3 About preferentially sampled histories
We use validation histories to improve sample efficiency. With a limited number of updates, prefix validation lets us devote more updates to histories the policy is likely to encounter during validation.
For example, consider a history sampled from the current policy at the validation temperature. Because the policy has been optimized for the task, typically reflects a reasonable sequence of decisions from which the task can still be completed. Prefix validation therefore concentrates training on histories that are useful for completing the task and that better reflect those encountered during validation.
Without prefix validation, the policy may sample a history in which an earlier error has already made the task impossible to complete. Training on may then provide no useful signal, reducing the proportion of effective training samples. The marked drop in performance when we removed prefix validation in our ablation study supports this explanation.
Thank you for your thoughtful reviews. In response to the feedback, we have made the following revisions:
We appreciate your feedback and hope that our response addresses your concerns.