- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RTPO is a framework that aims to stabilize multi-turn LLM agent training by optimizing interaction turns in reverse order. It samples alternative continuations from the same environment state using the latest policy to assign credit to individual turns. It also keeps rollout and training contexts consistent while doing this. For Qwen3 models on several benchmarks agentic tasks, the authors report improvements over trajectory-level and turn-level RL baselines.
Critical strengths
Thank you so much for submitting your paper. Overall, the concept is clear to me and well explained. Below are my main strengths in no particular order.
- Clever approach: I think your idea is very interesting and clever. Comparing alternative actions from the same environment state is not novel but a great concept for assigning credit to individual turns. What is novel though is how you regenerate their continuations with the latest policy. This makes a lot of sense, since earlier decisions should be evaluated against the downstream behavior of the current model. When you keep rollouts and training context consistent, your approach becomes very coherent and your results reflect that in training stability.
- Great amount of detail in your analysis: you really go the extra mile and report many interesting evaluations for your method. For example, I really liked the rollout-training consistency and the on-/off-policy analysis. Reading the paper and seeing your effort to explain where your method's improvements arise is much appreciated and gives your method credibility.
There are also some other points that I really liked:
- Presentation: Your figures are high-quality and explain the concept very well. I kept coming back to Fig. 2. Also, the links to the appendix were really helpful.
- Implementation details: while I could not re-implement during review time, I think your pseudocode and expanded descriptions of context handling, sibling generation, and reverse-order updates help make the method concrete and easier to assess in the future.
Critical weaknesses
Unfortunately, despite the mentioned strengths, there are some weaknesses that make your assessment flawed in some cases and raise further questions.
- Algorithmic improvements are questionable: despite the great amount of detailed analysis, I think the main comparison does not really show and/or separate the benefit of RTPO’s optimization procedure from correcting inconsistent rollout and training contexts. In Table 1, the baselines generate responses with earlier reasoning traces removed but train on reconstructed full histories. But your RTPO method preserves the actual rollout context. You could do the same for GRPO and SeeUPO as well, correct? This would also not introduce the reverse-order optimization then. You show that in Table 2 but with a different model (Qwen3-4B instead of Qwen3-8B). The average Pass@1 improvement over the strongest baseline is also relatively small (79 vs 78ish for ARPO). With that, your main results do not show how much improvement comes from the proposed algorithm and how much comes from correcting the context mismatch. You can easily convince me by showing the main baselines evaluated with matched rollout and training contexts for the original model.
- Theory is not convincing (yet): your convergence analysis and result assume tabular representations and fixed policies, among other things. But your practical implementation (if I am not mistaken, please correct me if I am!) instead performs a finite number of clipped updates. But restricting the loss to one turn’s tokens does not freeze the model’s behavior at other turns, so an earlier-turn update can change previously optimized downstream behavior. In App. C you acknowledge this but in my opinion this leaves the theoretical justification for the practical reverse-order procedure incomplete. I could not find a comparison against forward-order or random-order updates with the same context handling, sibling sampling, and on-policy continuation. Such an ablation would help establish whether reverse ordering itself provides the claimed stabilization benefit when parameters are shared across turns. A small example could convince me, though!
Both of these points are easy to address and I would be happy to reconsider my score.
There are some other (mostly weaker) points that I would like to rephrase as questions (down below) and are not critical.
Questions for authors
As mentioned above, there are some non-critical but nevertheless important points that I would like to phrase as questions to the authors.
1) Reporting Relative Improvements: Could you clarify how the main headline improvements of 21.50% over GRPO and 10.76% over SeeUPO were calculated? Using the overall scores in Table 1, I obtain relative improvements of approximately 14.80% and 6.90%. Your percentages appear to normalize the differences by the vanilla model’s score. Is that correct? If so, could you make the denominator explicit and distinguish these values from relative improvements over the respective baselines?
2) Compute Comparisons: again, I really appreciate the detailed cost reporting. You show that under equal GPU-hours, RTPO matches GRPO on Pass@1 while improving Pass@4 and other metrics. Does a similar advantage hold under equal compute on the main math and search tasks. Especially wondering about the stronger turn-level or tree-based baselines? Matching rollout counts alone may not capture differences in generated tokens, optimization work, and synchronization overhead. Just some comments here would be helpful.
3) Related work: I think the related work discussion would benefit from including TSR [1] and GiGPO [2]. TSR improves multi-turn agent training through search-guided rollout construction, making it relevant for assessing the benefit of RTPO’s particular combination of branching, credit assignment, and reverse-order updates. GiGPO estimates local advantages by grouping actions from repeated environment states, which is closely related to RTPO’s state-matched sibling comparisons. Including these works would help clarify what RTPO contributes beyond improved rollout generation and existing state-based credit assignment.
[1] TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agents [2] Group-in-Group Policy Optimization for LLM Agent Training
7 comments
Response(2/4)
W2: The theory relies on tabular representations and fixed policies; a comparison with forward / random orders or a small example
Thank you for the careful reading. Theorem 1 focuses on an idealised tabular setting in which each turn's policy is optimised once and then frozen; as stated in Appendix C.1, it is not a convergence guarantee for the shared-parameter implementation. Such a gap between the analysed setting and the practical algorithm is common in many RL policy optimisation: for example, TRPO's monotonic-improvement theorem is stated for exact advantages and a maximum-KL penalty, whereas the practical algorithm uses sampled advantages, an average-KL constraint and neural policies (Schulman et al., 2015); likewise, exact global-convergence results for policy-gradient methods are typically established for tabular policies, while results with function approximation carry approximation-error terms (Agarwal et al., 2021).
To connect the theory more closely to our implementation, we provide below contexts that hold under shared parameters (Lemma A, Prop. B and Prop. C).
(1) Theory that does not rely on tabular representations or frozen downstream policies Notation. is the boundary state at the start of turn k (the full visible history so far), is the output of turn k, and is the terminal reward. is the distribution of under policy and depends only on the upstream ; (continuing with afterwards) depends only on the downstream . For any k, .
Lemma A (exact stage decomposition). If differs from only in its behavior at turn k, then Proof. The upstream is unchanged, so ; the downstream is unchanged, so . Substituting into the expression for above and subtracting gives the result. The proof only uses the fact that the behavior outside turn k is unchanged; it requires neither a tabular representation nor a frozen downstream.
Corollary (turn-by-turn decomposition). Let , i.e., the new policy is used from turn onward and the old policy before it. Adjacent terms differ only at turn , so by Lemma A
where the states come from the old upstream and the continuations come from the updated downstream.
Prop. A (unbiasedness with a single trunk sample). RTPO samples the trunk only once at the start of each sweep (the states follow ), and at each stage regenerates the siblings and continuations with the latest parameters. If each stage changes only the behavior of its own turn, then when the update order is reverse, , the stage-k objective is exactly an unbiased estimate of the k-th term above: when turn k is processed, the upstream has not yet been updated, so the trunk states still follow the current ; the downstream has already been updated, and the continuations are generated by the current parameters. Hence the sum of the stage objectives equals the true improvement. For any other order, some upstream turn is always updated before turn k, so the current in general no longer equals the trunk distribution; unless the trunk is resampled before that stage, the objective of that stage is biased (the example in Prop. D below is one instance).
Response(3/4)
Prop. B (improvement lower bound under shared parameters). Remove the condition that each stage changes only its own turn. Let be the parameters after each stage within one sweep, and the objective of stage . Then
which has the same form as TRPO's "surrogate objective minus a divergence penalty".
Proof sketch. Within each stage, split the parameter change into "changing only turn " and "changing the other turns". The former is handled by Lemma A, and each term of the latter is controlled by . The difference between the trunk distribution and the current is likewise controlled by TV.
The key difference lies in the second drift term. Under reverse order, the upstream turns are affected within the sweep only indirectly through parameter sharing, so this term contains only unintended cross-turn changes. Under forward and random orders, it also contains the intended updates of the upstream turns.
Prop. C (convergence to a stationary point for any order). Each stage is one (clipped) policy-gradient step. At the start of a stage, the gradient of the stage-k objective equals the turn-k term of plus an bias, which comes from the parameter changes made by the earlier stages of the same sweep; this holds for any order. A whole sweep is therefore one step along with an error. When is -smooth, the gradient estimates have bounded variance, and the step sizes satisfy and , the standard nonconvex stochastic-gradient analysis gives .
Prop. D (Theorem 1 is a special case; a small example that can be computed by hand). Under the protocol "each turn is optimized once and then frozen", reverse order is exactly backward induction in dynamic programming and reaches the recursive optimum in one pass (Theorem 1). Other orders cannot always do so in one pass, as the following two-turn problem shows:
- At turn 1, choose or . After , the reward is 0.5 regardless of what turn 2 does. After , choosing at turn 2 yields 1 and choosing yields 0; the initial policy chooses with probability 0.1.
- forward: Optimize turn 1 first. Here , so it chooses and freezes; the final reward is 0.5.
- reverse: Optimize turn 2 first, choosing after ; then optimize turn 1, where now , so it chooses ; the final reward is 1, i.e., optimal.
- random: With probability 1/2, turn 1 is processed first, with the same result as forward.
- When multiple sweeps are allowed, each sweep makes at least one more turn (counting backward from the last turn) optimal, so any order reaches the optimum within at most K sweeps; reverse needs only one.
Therefore, under shared parameters, the role of reverse order is to obtain unbiased stage objectives from a single trunk sample and to confine the error to unintended cross-turn drift (Prop. B), instead of guaranteeing that the downstream policy no longer changes.
(2) Comparison changing only the update order. Setting: Qwen3-4B, math task. We compare three orders, reverse, forward and random, where random re-permutes the turns at every step. Controlled variables: everything except the order is identical, including context handling (rollout and training use the same context), the number of siblings per boundary state, regenerating the continuations with the latest parameters at each stage, as well as the data, number of steps and hyperparameters.
Table R1: Comparison of update orders
| Order | Pass@1 step 10 | step 20 | step 30 | GPU hours (step 30) |
|---|---|---|---|---|
| reverse | 0.75 | 0.82 | 0.85 | 16.56 |
| forward | 0.70 | 0.78 | 0.78 | 15.99 |
| random | 0.71 | 0.79 | 0.79 | 16.55 |
When the trunk is sampled only once per sweep, reverse order avoids the shift in the trunk state distribution caused by upstream turns being explicitly updated first; the remaining shift comes only from indirect cross-turn changes caused by parameter sharing. In the experiment that changes only the update order, the three orders use similar training GPU hours, and reverse order reaches a Pass@1 of 0.85 at step 30, 7 and 6 percentage points higher than forward and random, respectively.
We will add this new result in the revised manuscript.
Response(1/4)
Thank you for the thoughtful review. We appreciate your recognition of our key contribution, regenerating continuations with the current policy so that earlier decisions are evaluated in light of how that policy would continue from them. We are glad that the rollout–training consistency and on-/off-policy analyses helped explain the gains in training stability, and that Figure 2 and the pseudocode made the method and its implementation easier to follow.
Below, we address each of your concerns and questions point by point.
W1: Separating the contribution of the algorithm itself from that of "fixing the context"; evaluating context-consistent baselines on the original model
Thank you for the suggestion. Under the setting of Table 1 with the original model Qwen3-8B, now we implemented and trained "training–inference-consistent" versions (i.e., Context Matching (CM)) of the baselines (GRPO-CM, SeeUPO-CM). For each turn, these two versions compute the IS ratio under the same context that was actually used when that turn was generated. We found that both CM variants lead to longer outputs and more truncation; we therefore used a longer context limit of 16K and an evaluation limit of 3 turns, and sampled 16 times per problem.
Table R4: Context-consistent baselines (Qwen3-8B, each cell is Pass@1 / Pass@16)
After matching the context, the five-benchmark average Pass@1 / Pass@16 of GRPO changes by −0.26 / +2.74 percentage points, and that of SeeUPO changes by +1.05 / +0.35 percentage points. Under the training–inference-consistent condition, RTPO still achieves the highest five-benchmark average Pass@1 / Pass@16 (61.96 / 79.26)
We will add this new result in the revised manuscript.