- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Stochastic-TBRM, an offline RL method that learns from trajectory-level returns under stochastic transitions. It extends TBRM with an auxiliary model for expected continuation values and provides theoretical analysis and evaluations on controlled MDPs and interactive-agent benchmarks.
Critical strengths
The theoretical analysis is interesting. The paper provides conditional policy guarantees for the trajectory objective and analyzes the errors introduced by the proposed updates.
Critical weaknesses
The theoretical results are interesting, but the practical contribution to multi-turn LLM alignment remains unclear. The significance of the proposed extension is not sufficiently established, and stronger empirical evidence is needed to support its claimed benefits.
-
The practical benefit in LLM agents is not clearly established: The interactive-agent experiments compare complete pipelines with different action-selection and execution mechanisms. In particular, the ALFWorld result uses the BC checkpoint without Stochastic-TBRM updates, while ScienceWorld also benefits from additional action-grounding components. A comparison with the original TBRM within the same agent pipeline is important to establish the value of the stochastic extension. More broadly, the current comparisons do not clearly establish its advantage over applicable existing RL methods. It therefore remains unclear which important limitation in multi-turn agentic alignment the proposed method resolves.
-
The controlled experiments need clearer interpretation: The large performance gap between TBRM and Stochastic-TBRM on Acrobot needs further explanation. The paper attributes this difference to stochasticity correction, but the source of transition stochasticity is not clearly specified, and the analysis does not distinguish this explanation from value-estimation or optimization effects. On FrozenLake, the improvement in Q-MAE is useful, but corresponding policy-performance results are not reported. These results therefore do not fully establish why the proposed extension works or how its value-estimation benefits translate into better control.
Questions for authors
-
What evidence isolates the benefit of Stochastic-TBRM from the BC and action-grounding components, particularly relative to the original TBRM under the same agent pipeline?
-
What explains the large Acrobot performance gap, and what evidence supports attributing it specifically to the correction for stochastic transitions?