acceptodds
Under review as a conference paper at ICLR 2027

From Static Context to Calibrated Interactive RL: Mitigating Distribution Shift in Multi-turn Dialogue with Aligned Simulator

Abstract

A long-standing goal of the research community is to develop highly interactive LLM-based dialogue agents. Recent research focuses on optimizing policies based on fixed offline logs (Static Context RL) or using a prompt-based simulator (Interactive RL). In this work, we theoretically show that both paradigms are fundamentally limited by context distribution shift—a mismatch between dialogue histories observed during training and those encountered in real conversations. This shift compounds quadratically over turns and severely degrades dialogue quality. Specifically, we attribute this shift to two distinct sources: (i) policy-induced shift, arising from training on static histories rather than self-generated trajectories; and (ii) simulator-induced shift, stemming from discrepancies between simulated and real human behaviors. To address these challenges, we propose Calibrated Interactive RL, a unified framework that couples interactive RL with simulator alignment. By aligning the simulator with human interaction patterns, our approach reduces the sim-to-real gap and mitigates compounding distribution shifts. Experiments across multiple dialogue tasks confirm our theoretical analysis: (i) Interactive RL significantly outperforms the Static Context baseline by mitigating policy distribution shift; and (ii) calibrating simulators with our alignment method further bridges the sim-to-real gap, yielding state-of-the-art downstream performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, User Simulator, Interactive RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LPVhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

To address the distribution mismatch between the policy and the simulator in multi-turn dialogue reinforcement learning, this paper proposes a two-stage "Calibrated Interactive Reinforcement Learning" method. It first employs supervised fine-tuning to adjust the simulator, then freezes the policy and trains it using GRPO. Grounded in finite-horizon theory, experimental results on mathematical and document editing tasks demonstrate that this method significantly improves task performance while substantially reducing token overhead compared to static and uncalibrated baselines.

Critical strengths

The authors rigorously elucidate the mathematical mechanism behind the "snowball effect" of error accumulation in multi-turn dialogues and devise a lightweight, practical calibration and sparring-based training scheme; this approach significantly boosts performance on multi-turn tasks while slashing the assistant's token consumption by half.

Critical weaknesses
  1. The calibration dataset used in the paper consists entirely of interaction trajectories between Qwen3-235B and Gemma-3-4B-IT; this implies that the fine-tuned Qwen2.5-7B simulator essentially models "user feedback tailored to Gemma's specific response style and error patterns." If the downstream needs to train other base models, such as Qwen3-4B or Llama-3, does it mean that the teacher model must re-interact with the new model and the simulator must be retrained? The paper experiments solely with Gemma-3-4B-IT and does not verify whether the simulator possesses cross-policy generalization capabilities. If the data pipeline must be rebuilt whenever the target model changes, the practical cost of transferring the framework would be prohibitively high, significantly undermining its utility.

  2. It appears the authors merely partitioned a small subset of the existing MATH-Hard or MediumDocEdit datasets to serve as the evaluation set, rather than actually training on out-of-distribution (OOD) test sets.

Questions for authors

If the authors could evaluate performance on an OOD test set and demonstrate improvements over static training, and also show the method's effectiveness by training it on other models, I would consider raising the score.

2 comments

We thank for highlighting the practical importance of cross-policy transfer and out-of-distribution generalization. Below are our responses to your comments and questions:

1. Reusability of the Calibrated Simulator Across Policy Backbones (Weakness 1, Question 1). Our two-stage framework is explicitly designed to decouple user modeling from policy optimization so that the calibrated simulator can be reused across different policy backbones without rebuilding the calibration pipeline (Section 4, Algorithm 1):

  • Policy-Agnostic Architecture & Objective: In Phase I (Section 4.1, Equation 5), the SFT objective masks all assistant tokens and optimizes exclusively over user turns utu_t, modeling how a user reacts to whatever assistant response ata_t appears in the history. In Phase II, the calibrated simulator is frozen as a standalone environment interface that accepts standard text messages from any policy πθ\pi_\theta.
  • Task-Driven User Dynamics: In collaborative tasks, core user behaviors—such as providing underspecified initial prompts, clarifying missing constraints when asked, and pointing out logical or editing errors—are governed primarily by the user's task goal (y∗y^*) rather than the specific model family of the assistant. While collecting initial traces with Gemma-3-4B-IT exposes the simulator to representative assistant mistakes, the learned user behavior captures general collaborative feedback patterns rather than Gemma-specific token statistics.

Evaluating cross-family policy backbones (such as Qwen or Llama policies) with the same frozen simulator is a natural extension that we will highlight in Section 5.1 and Appendix B.7.

2. Held-Out Evaluation Protocol and Generalization Scope (Weakness 2, Question 1). Our evaluation follows the established multi-turn collaboration protocol of CollabLLM [1] (Section 5.1), using 200 strictly held-out test problems per domain that are separated from both the Phase I calibration traces and Phase II RL task inputs (Table 3, Appendix D.2). Within this benchmark design, our experiments test performance across two distinct dimensions:

  • Simulator-Shift Generalization: The policy is trained solely against a compact 7B simulator (Qwen2.5-7B-Instruct) and then evaluated against a substantially larger 235B proxy user model (Qwen3-235B-A22B-Instruct) with distinct generation dynamics (Section 5.1).
  • Task-Level Generalization: On task-disjoint held-out problems, calibration improves editing BLEU by 8.58.5 points while reducing assistant-token use by 43.3%43.3\%, demonstrating improved quality and efficiency within the benchmark.

We will clarify both these dimensions and the scope for future external OOD benchmark evaluations in Section 5.1 and Appendix B.7.

[1] Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. CollabLLM: From Passive Responders to Active Collaborators. arXiv preprint arXiv:2502.00640, 2025.

Thank you for the response. However, cross-policy generalization has not been demonstrated, as all experiments use only Gemma-3-4B-IT. Generalization to external OOD benchmarks also remains unverified. These concerns remain unresolved, so I will maintain my original rating of 2.

Reviewer_WzVtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the multi-turn RL for LLM and identified two source of distribution shift. It derives finite horizon return bounds and shows that errors accumulate quadratically with respect to the horizon. The author proposes calibrated interactive RL which first fine-tunes a user simulator and then trains a policy with GRPO against the simulator. It improves on MATH-Chat and document-editing BLEU score.

Critical strengths
  • The framing of the problem is pretty novel. Separating policy induced shift from simulator induced shift is a useful way to think about multi-turn RL.
  • The proofs are correct and add additional insights into the paper.
  • The experiment setup is clean and showcase the effectiveness of the proposed method well.
Critical weaknesses
  • Unclear why but the title in the paper does not match the openreview title.
  • Qwen3-235B is used both for simulator and the judge. Calibration therefore trains the simulator to imitate the exact the same policy that will be tested against, so the gain might reflect matching the evaluator's behavior rather than learning more realistic user behavior.
  • I am a bit unfamiliar but it seems a bit odd that the user simulator would have access to reference answer. The simulator might leak the answer during the conversation.
  • I appreciate that the user include statistical range. But the gain on each domain seems to fall within the noise, which makes the results look weak.
  • GRPO step is too short. GRPO runs for only 10 steps on MATH and 20 on editing, so the policy sees a small fraction of the listed task inputs. It’s unclear whether the results reflect converged training or early-stage differences, and there are no learning curves.
  • Theory has limited novelty. Theorems 3.1 and 3.2 are standard simulation-lemma and compounding-error arguments from Ross & Bagnell applied to dialogue histories.
Questions for authors

Please refer to the weaknesses.

3 comments

We thank for the careful reading and detailed questions regarding our evaluation protocol, training budget, and theoretical framing. Below are our responses to each concern:

1. Title Alignment Between OpenReview and the Manuscript (Weakness 1). The PDF manuscript title is Calibrated Interactive RL: A Distribution-Shift Perspective on Multi-Turn Dialogue, whereas the OpenReview portal metadata retained an earlier working title. We will update the OpenReview submission metadata so that both match exactly.

2. Role Separation and Evaluation Protocol of Qwen3-235B (Weakness 2). At evaluation time, Qwen3-235B serves two separated roles under distinct prompts and inputs: (i) acting as the simulated user that poses incomplete requests and reacts to assistant responses (Appendix C.1), and (ii) acting as the answer judge on MATH-Chat that compares the policy's final answer against the ground-truth solution (Appendix C.3). Across all compared methods, the scored assistant outputs are generated exclusively by Gemma-3-4B-IT rather than Qwen3-235B. All methods share the same evaluation protocol, holding the evaluation user model and scoring procedure fixed across comparisons. On MediumDocEdit-Chat, final scoring relies entirely on deterministic word-level sentence BLEU-4 rather than an LLM judge, confirming that the +8.5+8.5 BLEU and −43.3%-43.3\% token improvements hold independently of LLM judging.

3. Reference Conditioning and Non-Leakage Protocol (Weakness 3). The reference target y∗y* provides the user simulator with the task-specific goal needed to evaluate assistant progress and supply realistic corrective feedback, whereas the dialogue policy πθ\pi_\theta receives only the conversation history hth_t and never accesses y∗y* directly (Sections 3.1 and 4.1, Algorithm 1). Our simulator prompt (Appendix C.1) explicitly enforces progressive disclosure—instructing the simulator to provide vague or incomplete demands initially, reveal details only when asked, and avoid copying target queries directly. Across a complete audit of the dialogue transcripts, we observed no instances of the simulator directly disclosing the ground-truth answer; instead, the simulator challenges errors and prompts the assistant to derive the solution itself (Appendix E, Tables 7–9). We will note this behavioral verification in Section 4.1.

4. Consistent Gains Across Metrics and Evaluation Runs (Weakness 4). We do not claim that every single metric improves by multiple standard deviations in isolation; rather, when accounting for both mean gains and run-to-run variability across repeated evaluations, Calibrated Interactive demonstrates a clear, consistent advantage across tasks:

MetricNaive InteractiveCalibrated InteractiveDifference in Reported Means
MediumDocEdit-Chat BLEU26.1±1.226.1 \pm 1.234.6±1.034.6 \pm 1.0+8.5+8.5 BLEU
MediumDocEdit-Chat assistant tokens4.00k±0.07k4.00\text{k} \pm 0.07\text{k}2.27k±0.08k2.27\text{k} \pm 0.08\text{k}−43.3%-43.3\%
MATH-Chat accuracy89.3%±3.4%89.3\% \pm 3.4\%91.5%±1.3%91.5\% \pm 1.3\%+2.2+2.2 percentage points
MATH-Chat assistant tokens1.97k±0.05k1.97\text{k} \pm 0.05\text{k}1.86k±0.09k1.86\text{k} \pm 0.09\text{k}−5.6%-5.6\%

On MediumDocEdit-Chat, the improvements (+8.5+8.5 BLEU and −43.3%-43.3\% tokens) far exceed the evaluation standard deviations. On MATH-Chat, Calibrated Interactive achieves the highest mean accuracy (91.5%91.5\%) of any model in Table 2—surpassing Naive Interactive (89.3%89.3\%), Static Context (85.0%85.0\%), and the Large-model reference (89.7%89.7\%)—while reducing the evaluation standard deviation from 3.43.4 to 1.31.3 percentage points (−61.8%-61.8\%) and lowering token usage. Together, these results raise the macro-average score from 57.757.7 to 63.0563.05 while reducing average assistant tokens from 2.98k2.98\text{k} to 2.06k2.06\text{k}. We will clarify the repeated-evaluation definition in Section 5.1 and Table 2.

5. GRPO Trajectory Budget and Multi-Turn Horizon Considerations (Weakness 5). Although the number of optimizer steps is 10 on MATH-Chat and 20 on MediumDocEdit-Chat, each GRPO step samples 8,1928{,}192 complete multi-turn dialogue trajectories (Appendix D.2, Table 5), resulting in substantial trajectory-level experience:

TaskGRPO stepsTrajectories per stepTotal dialogue trajectoriesPrompt-group draws (G=32G=32)
MATH-Chat108,19281,9202,560
MediumDocEdit-Chat208,192163,8405,120

Each trajectory spans up to 14 messages (7 user-assistant turns), meaning the policy processes hundreds of thousands of interactive turns during training. Moreover, in multi-turn interactive RL with sparse terminal rewards, overly prolonged policy optimization against a frozen simulator risks over-optimizing simulator-specific quirks and reward hacking [3] (Section 2, Appendix B.7). Our step budgets are specifically chosen to achieve strong performance gains while avoiding simulator overfitting. We will report the complete-dialogue unit and total trajectory counts explicitly in Section 5.1 and Table 5.

6. Theoretical Contribution Relative to Ross and Bagnell (Weakness 6). Our theoretical analysis builds directly on the finite-horizon distribution-propagation framework of Ross and Bagnell [1, 2] (Section 3), and our contribution is its targeted formulation for multi-turn interactive RL:

  • Explicit Kernel Decomposition: Unlike single-agent imitation learning, multi-turn dialogue RL involves two coupled sources of shift. Theorems 3.1 and 3.2 decouple policy-induced shift (Δt,ϵt\Delta_t, \epsilon_t) under a fixed user kernel from simulator-induced shift (Δ0env,δt\Delta_0^{\text{env}}, \delta_t) under a fixed policy.
  • Terminal-Reward Specialization: Equations (3) and (4) specialize these bounds to outcome-based dialogue scoring (one terminal score per episode), showing that uniform local divergence bounds yield linear upper-bound horizon dependence O(H)O(H) rather than the quadratic O(H2)O(H^2) scaling of cumulative per-turn rewards.

This theoretical distinction motivates pairing interactive rollouts with simulator calibration rather than relying on on-policy collection alone. We will highlight this specific formulation in Sections 1 and 3.

[1] Stéphane Ross and Drew Bagnell. Efficient Reductions for Imitation Learning. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, 9, 661-668, 2010.

[2] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 627-635, 2011.

[3] Xiaohua Wang, Muzhao Tian, Yuqi Zeng, et al. Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges. arXiv preprint arXiv:2604.13602, 2026.

I thank the authors for the reply. Overall, most of my concerns remain including using the same model for simulator and judge. Even though we have different prompts, the gain might still come from matching the evaluator's behavior rather than learning more realistic user behavior.

I will maintain my current score.

Reviewer_YwGDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigated a problem that multi-turn dialogue RL trains on fixed offline histories or on a prompted user simulator, and both create distribution shift. The authors propose Calibrated Interactive RL, a two-stage method. It first fits a user simulator to multi-turn interaction data with supervised fine-tuning. It then freezes that simulator and trains the dialogue policy with GRPO through interaction. The contributions include: a distribution-shift analysis that separates policy-induced and simulator-induced error, with return bounds for cumulative and terminal rewards. The Calibrated Interactive RL framework that couples simulator calibration with interactive policy optimization. An empirical comparison on MediumDocEdit-Chat and MATH-Chat showing higher scores and fewer assistant tokens than uncalibrated interaction.

Critical strengths
  1. This paper presents an interesting problem with clear framing, and separates policy-induced shift from simulator-induced shift (as shown in Section 3), which makes the contribution easy to follow.

  2. The authors did a theoretical analysis which points at the method. Theorem 3.2 shows simulator error affects returns, which justifies calibrating the user simulator rather than only the policy.

  3. In the experiments, the key ablation is clear. Naive Interactive and Calibrated Interactive differ mainly in the SFT calibration step (B.5), so that comparison isolates the effect of calibration.

  4. The last part of experiments show notable token savings. On MediumDocEdit-Chat, assistant tokens drop from 4.00k to 2.27k while BLEU rises from 26.1 to 34.6 (Table 2).

Critical weaknesses
  1. As the authors state their limits, Sec. 3.4, A.7, and B.7 admit the theoretical bounds are loose, untested, and subject to evaluator bias.

  2. There is an interval overlap between Calibrated scores (91.5±1.3) and Naive scores (89.3±3.4) over only 3 runs, so it is confusing on which one is better.

  3. Within the proposed method, the same model family plays many roles. E.g., Qwen3-235B generates the teacher traces, simulates evaluation users, and judges math answers (B.7, 5.1), so bias is hard to rule out. Also the calibration data is synthetic. No real user data was used (Sec. 4.1), so "calibration to users" is really calibration to a Qwen3 proxy.

  4. The main baseline comparison is confounded. Static Context uses a different user model and history source than Naive Interactive, and the authors also concede this (Sec. 5.1), so it cannot support causal claims.

  5. Perhaps there is an inconsistency. Figure 2 does not match Table 2. The figure reports the base model at 82.0%, but Table 2 gives 82.3%. The case studies are also only 29 of 200 problems, selected for just showing improvement.

Questions for authors

n/a

2 comments

We thank for the thorough review and constructive feedback on our theoretical framing and experimental controls. Below are our responses to each point:

1. Structural Role and Tightness of the Theoretical Bounds (Weakness 1). The primary role of Theorems 3.1 and 3.2 (Sections 3.2–3.3) and their terminal-reward specialization (Section 3.4, Equations 3–4) is to provide an explicit, principled structural decomposition that separates two conflated sources of shift in multi-turn RL: policy-induced history shift (Δt\Delta_t, driven by conditional policy discrepancies ϵt\epsilon_t) under fixed user dynamics, and simulator-induced kernel mismatch (Δ0env,δt\Delta_0^{\text{env}}, \delta_t) under a fixed policy. Rather than serving as a tight numerical predictor of exact empirical gains, this decomposition identifies why on-policy rollouts alone cannot eliminate simulator mismatch and explains the linear versus quadratic upper-bound horizon dependence under uniform local divergence bounds between terminal and cumulative rewards. While message-level ℓ1\ell_1 bounds are naturally conservative and require additional near-optimality conditions for end-to-end optimization guarantees (Sections 3.4, A.7–A.8), the analysis directly motivates our two-stage design—first fitting simulator responses via SFT and then optimizing the policy interactively—whose empirical effectiveness is validated in Section 5.2. We will sharpen this structural framing in Section 3.4.

2. MATH-Chat Performance Advantage and Evaluation Consistency (Weakness 2). Table 2 summarizes three repeated evaluations of each policy. On MATH-Chat, Calibrated Interactive achieves a higher mean accuracy (91.5%91.5\% vs. 89.3%89.3\%, a +2.2+2.2 percentage point gain), reduces the evaluation standard deviation from 3.43.4 to 1.31.3 percentage points (−61.8%-61.8\%), and simultaneously lowers assistant token consumption (1.86k1.86\text{k} vs. 1.97k1.97\text{k}, −5.6%-5.6\%). Crucially, Calibrated Interactive outperforms every other method in Table 2 on MATH-Chat—including Static Context (85.0%85.0\%), CollabLLM (82.3%82.3\%), and even the Large-model reference (89.7%89.7\%)—demonstrating that simulator calibration produces policies that are both more accurate and more consistent across repeated evaluations than uncalibrated interactive training. We will explicitly note the repeated-evaluation definition in Section 5.1 and the Table 2 caption.

3. Synthetic Calibration Data and Shared Reference Protocol (Weakness 3). Because live human interaction logs carry strict privacy constraints and high operational latency, adopting a capable reference LLM as a proxy human user is an established, reproducible methodology in simulated feedback and multi-turn collaboration research [1, 2] (Section 4.1, Figure 1). All methods share the same evaluation protocol, holding the evaluation user model and scoring procedure fixed across comparisons (Section 5.1). Furthermore, on MediumDocEdit-Chat, final scoring uses deterministic word-level sentence BLEU-4 against ground-truth documents without any LLM judge, yet Calibrated Interactive still achieves a +8.5+8.5 BLEU gain and a 43.3%43.3\% token reduction over Naive Interactive. We will highlight this unified evaluation protocol and cite prior proxy-user frameworks in Section 4.1.

4. Experimental Design of the Static Context Comparison (Weakness 4). Static Context represents training on fixed interaction histories collected with the Qwen3-235B proxy user, whereas Naive Interactive represents online training with an uncalibrated Qwen2.5-7B simulator. Their comparison evaluates these complete training configurations (Section 5.1, Appendix B.5). The effect of simulator calibration is assessed by comparing Naive Interactive with Calibrated Interactive, which shares the policy backbone, simulator backbone, and interactive GRPO setup, with simulator SFT as the distinguishing stage. We will make this two-level experimental structure explicit in Section 5.1 and Table 2.

5. Consistency Between Figure 2(c) and Table 2, and Qualitative Case Selection (Weakness 5). Figure 2(c) reports the outcome of a single representative evaluation run (82.0%82.0\% vs. 91.5%91.5\%, a +9.5+9.5 percentage point gain), whereas Table 2 reports the aggregate mean and standard deviation across three repeated evaluations (82.3%±0.6%82.3\% \pm 0.6\% vs. 91.5%±1.3%91.5\% \pm 1.3\%). Importantly, the 29 qualitative cases analyzed in Appendix E are extracted directly from this paired single-run evaluation—specifically comprising the exact subset of the 200 evaluation problems where the base policy fails (00) and Calibrated Interactive succeeds (11) (Section E.1). This keeps the visual comparison in Figure 2(c) and the case-level behavioral taxonomy in Appendix E strictly aligned on the same run, while Table 2 captures aggregate multi-run performance and token efficiency. In the revised manuscript, we will explicitly label Figure 2(c) as a single representative evaluation run (and note the three-run aggregate means of 82.3%82.3\% vs. 91.5%91.5\% alongside it) in Section 5.2 and the Figure 2 caption, and clarify this connection in Appendix E.1.

[1] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. Advances in Neural Information Processing Systems, 36, 30039-30069, 2023.

[2] Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. CollabLLM: From Passive Responders to Active Collaborators. arXiv preprint arXiv:2502.00640, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.