- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To address the distribution mismatch between the policy and the simulator in multi-turn dialogue reinforcement learning, this paper proposes a two-stage "Calibrated Interactive Reinforcement Learning" method. It first employs supervised fine-tuning to adjust the simulator, then freezes the policy and trains it using GRPO. Grounded in finite-horizon theory, experimental results on mathematical and document editing tasks demonstrate that this method significantly improves task performance while substantially reducing token overhead compared to static and uncalibrated baselines.
Critical strengths
The authors rigorously elucidate the mathematical mechanism behind the "snowball effect" of error accumulation in multi-turn dialogues and devise a lightweight, practical calibration and sparring-based training scheme; this approach significantly boosts performance on multi-turn tasks while slashing the assistant's token consumption by half.
Critical weaknesses
-
The calibration dataset used in the paper consists entirely of interaction trajectories between Qwen3-235B and Gemma-3-4B-IT; this implies that the fine-tuned Qwen2.5-7B simulator essentially models "user feedback tailored to Gemma's specific response style and error patterns." If the downstream needs to train other base models, such as Qwen3-4B or Llama-3, does it mean that the teacher model must re-interact with the new model and the simulator must be retrained? The paper experiments solely with Gemma-3-4B-IT and does not verify whether the simulator possesses cross-policy generalization capabilities. If the data pipeline must be rebuilt whenever the target model changes, the practical cost of transferring the framework would be prohibitively high, significantly undermining its utility.
-
It appears the authors merely partitioned a small subset of the existing MATH-Hard or MediumDocEdit datasets to serve as the evaluation set, rather than actually training on out-of-distribution (OOD) test sets.
Questions for authors
If the authors could evaluate performance on an OOD test set and demonstrate improvements over static training, and also show the method's effectiveness by training it on other models, I would consider raising the score.
2 comments
Thank you for the response. However, cross-policy generalization has not been demonstrated, as all experiments use only Gemma-3-4B-IT. Generalization to external OOD benchmarks also remains unverified. These concerns remain unresolved, so I will maintain my original rating of 2.
We thank for highlighting the practical importance of cross-policy transfer and out-of-distribution generalization. Below are our responses to your comments and questions:
1. Reusability of the Calibrated Simulator Across Policy Backbones (Weakness 1, Question 1). Our two-stage framework is explicitly designed to decouple user modeling from policy optimization so that the calibrated simulator can be reused across different policy backbones without rebuilding the calibration pipeline (Section 4, Algorithm 1):
Evaluating cross-family policy backbones (such as Qwen or Llama policies) with the same frozen simulator is a natural extension that we will highlight in Section 5.1 and Appendix B.7.
2. Held-Out Evaluation Protocol and Generalization Scope (Weakness 2, Question 1). Our evaluation follows the established multi-turn collaboration protocol of CollabLLM [1] (Section 5.1), using 200 strictly held-out test problems per domain that are separated from both the Phase I calibration traces and Phase II RL task inputs (Table 3, Appendix D.2). Within this benchmark design, our experiments test performance across two distinct dimensions:
We will clarify both these dimensions and the scope for future external OOD benchmark evaluations in Section 5.1 and Appendix B.7.
[1] Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. CollabLLM: From Passive Responders to Active Collaborators. arXiv preprint arXiv:2502.00640, 2025.