acceptodds
Withdrawn from ICLR 2027

An End-to-end Offline RL Approach to Multi-turn Agentic Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training multi-turn agents from coarse trajectory-level feedback poses a fundamental credit-assignment problem, particularly when their interactions with users, tools, and environments are stochastic. In this paper, we propose Stochastic Trajectory Bellman Residual Minimization (Stochastic-TBRM), an end-to-end trajectory-based offline reinforcement learning approach to this problem: given static logs of agent trajectories and a single episodic alignment signal, learn a policy over the agent's sequential decision process. Under realizability, regularity, and coverage conditions, we establish a finite-sample policy guarantee for approximate minimization of the reduced trajectory objective. Controlled stochastic MDPs demonstrate value recovery and useful control from trajectory returns. Interactive-agent benchmarks evaluate deployment pipelines, whose warm-start and action-grounding components are distinguished from the effects of the learning objective.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, multi-turn agents, agentic alignment, trajectory-level rewards, Bellman residual minimization, stochastic transitions, credit assignment, outcome-only supervision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gswCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Stochastic-TBRM, an offline RL method that learns from trajectory-level returns under stochastic transitions. It extends TBRM with an auxiliary model for expected continuation values and provides theoretical analysis and evaluations on controlled MDPs and interactive-agent benchmarks.

Critical strengths

The theoretical analysis is interesting. The paper provides conditional policy guarantees for the trajectory objective and analyzes the errors introduced by the proposed updates.

Critical weaknesses

The theoretical results are interesting, but the practical contribution to multi-turn LLM alignment remains unclear. The significance of the proposed extension is not sufficiently established, and stronger empirical evidence is needed to support its claimed benefits.

  • The practical benefit in LLM agents is not clearly established: The interactive-agent experiments compare complete pipelines with different action-selection and execution mechanisms. In particular, the ALFWorld result uses the BC checkpoint without Stochastic-TBRM updates, while ScienceWorld also benefits from additional action-grounding components. A comparison with the original TBRM within the same agent pipeline is important to establish the value of the stochastic extension. More broadly, the current comparisons do not clearly establish its advantage over applicable existing RL methods. It therefore remains unclear which important limitation in multi-turn agentic alignment the proposed method resolves.

  • The controlled experiments need clearer interpretation: The large performance gap between TBRM and Stochastic-TBRM on Acrobot needs further explanation. The paper attributes this difference to stochasticity correction, but the source of transition stochasticity is not clearly specified, and the analysis does not distinguish this explanation from value-estimation or optimization effects. On FrozenLake, the improvement in Q-MAE is useful, but corresponding policy-performance results are not reported. These results therefore do not fully establish why the proposed extension works or how its value-estimation benefits translate into better control.

Questions for authors
  • What evidence isolates the benefit of Stochastic-TBRM from the BC and action-grounding components, particularly relative to the original TBRM under the same agent pipeline?

  • What explains the large Acrobot performance gap, and what evidence supports attributing it specifically to the correction for stochastic transitions?

Reviewer_yeG8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Stochastic-TBRM, an offline RL method for training multi-turn LLm agents from only a single episode-level outcome label. The authors identify the stochastic nature of tool use outputs and user response etc. causes difficulty in finetuning LLMs due to credit assignment. It extends TBRM (Yuan et al., 2025) stochastic environments by adding an auxiliary critic that corrects the double-sampling bias. However, the paper is difficult to read and unclear, as most of the details needed to understand the experiments are only found in the appendix. The interactive-agent results are highly questionable, and the paper itself admits that they do not isolate the contribution of the proposed method (Section 5.2 lines 469-472, Conclusion line 481).

Critical strengths

The paper clearly identifies why the original TBRM method fails under stochastic transitions and proposes a principled, theoretically grounded fix, with full proofs provided in the appendix.

Critical weaknesses

The interactive-agent experiments are highly asymmetric to the baselines and rely on extensive hand engineering that is not part of the proposed method. The details required to understand these experiments appear almost entirely in the appendix, and they substantially undermine the claims in the main text.

  • Asymmetric comparison: Section 3 and Algorithm 1 describe Stochastic-TBRM as a critic-learning objective. The deployed pipeline in Appendix B.2-B.4 is a far larger system. It includes behaviour cloning (BC) warm starts (which are not explained anywhere in the main text) of an 8B LLaMA based critic’s macro-action head over a hand-designed macro-action space (Table 3), and a separate Qwen3.5-4B model that generates the actual commands. It also includes benchmark-specific state parsers with hand-written rules, a deterministic command executor for ALFWorld, and, for ScienceWorld, a 128-command shortlist. Stochastic-TBRM only ever trains the LLaMA based critic’s macro-action head. The Qwen3.5-4B command generator is untrained on WebShop and ALFWorld, but on ScienceWorld it is separately supervised fine-tuned (not trained with Stochastic-TBRM). In contrast, DMPO and Q-Evolve fine-tune the same Qwen3.5-4B to generate commands directly from the prompt, with none of these interfaces. The comparison in Table 2 is therefore not a comparison of learning methods, but of a hand-engineered pipeline against baselines.
  • ALFWorld does not use the proposed method at all. The 1.0 success rate on ALFWorld in Table 2 is reported as Stochastic-TBRM (Ours). However, Table 5 shows that the selected ALFWorld checkpoint uses 0 Stochastic-TBRM iterations. The main text also claims this in section 5.2, line 444, “it retains its behavior-cloning warm start because later Stochastic-TBRM iterates did not improve its development criterion”. Before this there was no mention of warm starts or behavior cloning. The results are hence achieved entirely by BC combined with the hand-crafted pipeline that was never mentioned in the method.
  • For ScienceWorld, the pipeline was repeatedly modified until it outperformed DMPO and the final result does not even isolate the proposed method. Appendix B.4 describes an evaluation process with an author-predeclared “confirmation gate” that requires a +0.10 improvement over DMPO on held-out development tasks before the test set is “opened”. The initial candidate (BC followed by Stochastic-TBRM training) failed this gate (+0.0321), and an automated controller kept the test set closed. After this candidate failure, additional components were added to the method rather than reporting the weaker results.

There are also many out of place or questionable descriptions like

  • Line 444 “ScienceWorld is reported on a reused test because an earlier candidate failed its predeclared confirmation gate.” This appears in Section 5.2 and it is not interpretable from the main text. There is no mention of this predeclared confirmation gate until the Appendix.
  • Line 470 “ALFWorld retains its BC initialization and ScienceWorld’s BC-shield variant also improves over DMPO, so neither result separates the contribution of Stochastic-TBRM updates from the action-grounding interface (Table 6).” Line 483 “ScienceWorld and WebShop results depend on benchmark-specific grounding and their reported evaluation contracts. These results do not isolate the benefit of critic updates.“ These statements directly claim that the author’s experiments do not isolate their proposed method, contradicting the abstract which mentions “Interactive-agent benchmarks evaluate deployment pipelines, whose warm-start and action-grounding components are distinguished from the effects of the learning objective”

Summary: The paper is difficult to follow, as several components essential to the reported results appear only in the appendix. Appendix B.4 describes an automated controller and an iterative process in which components were added after a failed confirmation gate until the pipeline outperforms DMPO, without isolating the contribution of Stochastic-TBRM. Experiments should test whether the proposed method works, rather than optimize a full pipeline against a benchmark. The BC warm start, macro-action interface, state parsers, executors, command shortlist, supervised command policy and the many guard rules are not part of the proposed method, yet they account for much of the reported gains. Where the objective is isolated, it shows no measurable effect, which the paper itself acknowledges (line 470, 483). Presenting these results in Table 2 as evidence for Stochastic-TBRM is misleading, and the interactive-agent experiments provide no support for the paper’s central claim.

Questions for authors

Please explain the large discrepancies mentioned in the weakness.

Reviewer_44JoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies offline learning for multi-turn agents from trajectory-level outcome labels under stochastic environment transitions. It proposes Stochastic Trajectory Bellman Residual Minimization (Stochastic-TBRM), which introduces an auxiliary model to estimate the conditional expected continuation value. The method constructs Bellman-implied reward proxies and fits their discounted trajectory sums to observed returns, without requiring transition-level rewards.

Critical strengths
  1. The paper tackles a practical problem. In many agent tasks, we only know how well the whole interaction went, which makes it hard to learn which decisions helped. Extending TBRM to handle stochastic interactions with tools and environments is therefore well motivated.

  2. The auxiliary model has a clear purpose. It estimates the expected value of what comes next, helping separate transition noise from value prediction errors. When it matches the true conditional expectation, it removes the bias introduced by using a single sampled next state.

  3. The controlled experiments provide useful support. FrozenLake is a helpful test because the exact soft-optimal Q-function is available for comparison. Stochastic-TBRM brings Q-MAE down from 48.62 to 2.10 compared with the deterministic-TBRM ablation, and the controlled results are reported over ten random seeds.

Critical weaknesses
  1. The interactive-agent experiments do not validate the contribution of Stochastic-TBRM updates. The selected ALFWorld checkpoint performs zero post-BC Stochastic-TBRM updates, so its 100% success rate does not demonstrate a benefit from the proposed learning objective. On the 36-episode ScienceWorld development screen, the BC-shield variant already achieves a reward of 0.5325, compared with 0.5442 for the post-TBRM shield. The paper does not establish whether this incremental improvement is statistically reliable. Comparable same-interface BC versus post-TBRM evaluations are needed across all three benchmarks.

  2. the novelty is primarily an extension of existing ideas. the combination is reasonble but the effectiveness for performance improvement is still questionable.

Questions for authors

Can you report BC versus post-TBRM results under identical execution interfaces, with multiple training seeds and paired confidence intervals?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.