acceptodds
Under review as a conference paper at ICLR 2027

Robust LLM-based User Simulation with Trained Latent-state Dynamics

Abstract

User simulators for training and evaluating LLM-based conversational agents must perform well off-policy (i.e., when facing novel agents). Recent work has argued that explicitly modeling user _latent state_ can improve simulator fidelity and off-policy robustness, but treating that state as a latent variable in LLM-based user simulation faces challenges such as intractability and unidentifiability given observational data. Consequently, latent-state simulation has generally been approached heuristically. We develop __RUSTLeD__, a principled probabilistic framework for latent-state learning that formalizes fine-tuning on latent-state labels as posterior-regularized maximization of the marginal likelihood of observed conversational data, smoothly interpolating between teacher distillation and autoencoding. Our _self-normalized importance sampling_ EM algorithm (SNIS-EM) addresses the problems facing current heuristic approaches. Beyond offering sound probabilistic foundations for latent-state simulation, __RUSTLeD__ yields simulators that generalize better to unseen agents than natural baselines, both in the likelihood of held-out human conversations and in reproducing human behavior.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
user simulation, latent variable models, causal model learning, LLM-based representations, variational methods

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_MQbAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RUSTLeD, a framework for training LLM-based user simulators with natural-language latent states. It adopts an EM-style approach in which an LLM labeler proposes latent-state trajectories conditioned on complete observed conversations. The labeler’s distribution serves as a semantic anchor for posterior-regularized marginal-likelihood maximization. A self-normalized importance-sampling procedure reweights the proposed trajectories, and the simulator is updated through weighted supervised fine-tuning. Experiments on ConvApparel-V2 evaluate generalization to held-out shopping-assistant personas under two policy splits. The results show benefits over prompted simulators and improvements over direct SFT without latent states on several behavioral metrics.

Critical strengths
  1. The paper studies an interesting and useful problem. Reliable user simulators could reduce the cost of collecting human interactions and help develop and evaluate conversational agents. Making these simulators work well with previously unseen agent policies is an important research direction.

  2. The method is well motivated and produces promising results. The EM-style framework uses LLM-generated latent states as proposals rather than fixed ground-truth labels. It reweights these proposals based on the simulator’s likelihood over the dialogue and latent states, then updates the simulator via weighted SFT. On ConvApparel-V2, the method improves several aspects of simulated behavior compared with prompting and direct SFT without latent states, including repetition and goal-satisfaction calibration.

Critical weaknesses
  1. The labeler, simulator, and judge come from the same model family. The labeler uses Gemini 2.5 Pro, while the simulator and judge use Gemini 2.5 Flash. These models may share biases about what realistic user behavior looks like. This raises concerns about whether the reported gains would hold under an independent evaluation.

  2. The main behavioral metrics lack human validation. The paper uses real human conversations as references and reports some deterministic metrics. However, dialogue acts, goal satisfaction, and human-likeness are assessed by an LLM judge without checking its agreement with human annotators. Since the goal is to simulate human users, a human-annotated subset would help establish whether the judge’s assessments and method rankings are reliable.

  3. Downstream evaluation would strengthen the practical value. It would be useful to test whether agents trained or optimized with this simulator perform better with real users.

Questions for authors
  1. How do you assess whether the learned latent states are reasonable? I understand that users’ true internal states are not observed. Have you checked whether human annotators find the generated states plausible and consistent with the conversations, or whether they change substantially with a different labeler? This would help establish whether the states offer meaningful explanations of user behavior.

  2. Do the judge’s assessments agree with human judgments? Have you evaluated this on a subset of conversations? In particular, do humans reach similar conclusions about the relative quality of the simulators?

6 comments
AuthorsOpenReview ↗

Response to Reviewer MQbA (1/2)

Thank you for the constructive review and clear questions. We address the weakness points raised and questions below.

1. Shared model family across labeler, simulator, and judge (W1)

We address this above in the common response (Table C4).

2. Reliability of behavioral metrics and agreement with human / independent judgments (W2 & Q2)

Our primary behavioral metrics (L1act=∑a∣rm(a)−rhum(a)∣L_1^{\text{act}} = \sum_a \vert r_m(a) - r _ {\text{hum}}(a)\vert and ∣ΔSat∣=∣Satm−Sathum∣\vert\Delta\text{Sat}\vert = \vert\text{Sat} _ m - \text{Sat} _ {\text{hum}}\vert) are much less sensitive to judge leniency than direct pairwise LLM preferences: both real human dialogues and simulated dialogues are labeled with precisely the same temperature-00 categorical rubrics (Appendix F.2), so any constant bias cancels unless the judge shifts its rubric boundary between human and simulated text.

Across 3,072 (5→65 \to 6) and 3,327 (3→83 \to 8) paired dialogues, Gemini 2.5 Flash and OpenAI GPT-5.4 Mini show high direct rubric agreement across all three LLM-judged constructs in Table 1 (plus Appendix F Viol.%) on both the Real Human reference conversations and simulator rollouts:

Table 1 Rubric Construct5→65 \to 6 (3,072 convs)3→83 \to 8 (3,327 convs)Simulator Rank Corr. (Spearman ρ\rho, 5→65\to 6 / 3→83\to 8)
1. Per-Turn 10-Act LabelsJaccard 0.683 (Real Humans 0.711)Jaccard 0.684 (Real Humans 0.711)L1actL_1^{\text{act}} +0.81 / +0.56; L1t≥3L_1^{t\ge 3} +0.84 / +0.46
2. Binary Goal Satisfaction87.6% (Cohen's κ=0.53\kappa=0.53; Humans 84.4%)89.0% (Cohen's κ=0.48\kappa=0.48; Humans 85.5%)Sat.% +0.97 / +0.93
3. Human-Likeness (1–5)89.9% within ±1\pm 1 pt (Humans 89.7%)90.0% within ±1\pm 1 pt (Humans 91.1%)Hum. +0.68 / +0.62

(For Appendix F rubrics, rare Viol.% achieves 96.3% / 96.7% exact agreement [96.5% / 96.1% on Real Humans]; Push. has a constant ~0.9-pt stricter scale offset on GPT-5.4 Mini [3.18 vs. 4.06], giving 67.8% / 65.7% uncentered within-±1\pm 1-pt agreement; 3→83\to 8 ρ\rho spans the 13 fine-tuned models.)

Running a new human-annotator labeling campaign over thousands of multi-turn dialogues was not feasible within the rebuttal window. We will include the cross-family inter-judge agreement analysis and an explicit discussion of human-annotation validation in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer MQbA (2/2)

3. Assessing whether learned latent states are reasonable & sensitivity to the labeler (Q1)

  • Instrumental role in HMLLMs. Because shoppers' true internal states are unobserved, RUSTLeD treats ztz_t instrumentally — as a latent state variable whose role is to carry predictive causal structure from the history HtaugH_t^{\text{aug}} to subsequent user utterances, applicable to any stochastic labeler QlabQ _ {\text{lab}}. SNIS-EM enforces this directly: Appendix D.2 shows that the log importance weight decomposes into emission predictive power log⁡Eθ(U;Z)\log E _ \theta(U; Z) minus causal implausibility, penalizing plausible-sounding annotations that fail to explain what the user actually says.
  • Design and empirical sensitivity to the labeler. Appendix C details our two-stage annotation schema: a 99-field structured state (intent, constraints, affect, trust, patience reserve, task outlook, etc.) paired with a strictly prefix-causal first-person reaction (Impact), while hindsight Evidence citations are stripped before training so future text never enters PθP _ \theta's conditioning set. Sensitivity to the labeler is directly tested by our Smoother vs. Filtering ablation (Section 4.2.4): using the exact same Gemini 2.5 Pro model and 99-field schema, changing only prefix vs. full-conversation conditioning leaves teacher-forced utterance likelihood unchanged (−1.519-1.519 vs. −1.518-1.518) while cutting ∣ΔSat∣\vert\Delta\text{Sat}\vert 4×4\times (8.28.2 to 2.02.0 pp) and verbatim repetition 4×4\times (0.170.17 to 0.040.04). Thus closed-loop rollout behavior genuinely depends on the quality of the learned ztz_t dynamics.
  • Inspecting generated latent states. In Appendix G (Qualitative Examples), we omitted the simulator's generated ztz_t blocks for space; in the revision, we will display the generated latent states alongside the utterances so readers can inspect their plausibility directly.

4. Downstream evaluation with real users (W3)

We agree that training or selecting assistant policies against the simulator and validating them in live human A/B studies is an important next step, though not one that is feasible for us at this time. We note two points:

  • Closed-loop evaluation across 6–86\text{--}8 unseen assistant policies with real human interaction logs from engagement with these policies (ConvApparel-V2) is already a direct multi-policy off-policy evaluation (OPE) test: each unseen persona represents a distinct candidate agent policy absent from training, and measuring ∣ΔSat∣\vert\Delta\text{Sat}\vert, W1W_1, and L1actL_1^{\text{act}} against real humans interacting with those exact policies tests whether the simulator accurately predicts policy-induced changes in human outcomes: this is where SFT (No LS) overshoots human satisfaction by +16.4+16.4 pp and Filtering overshoots it by +8.2+8.2 pp. By contrast, we see a difference of only 0.8–2.00.8\text{--}2.0 pp for RUSTLeD / Smoother.
  • End-to-end RL policy training across multiple simulator baselines followed by live human-subject evaluation constitutes a standalone human-study project beyond the scope of this paper. We will highlight this opportunity clearly in Future Work.

Thank you for the response. I appreciate the effort, but I believe my main concerns remain.

  1. When changing the judge model, the results in Table C4 are mixed. On 5→6, SFT (No LS) has the lowest L1_act (0.410), lower than all RUSTLeD settings (0.465–0.754). On 3→8, RUSTLeD (M=15) has a |ΔSat| of 9.8–10.9 pp, compared with 4.3 pp for SFT (No LS). The two judges also disagree on the human reference itself (Sat. 20.7% vs. 10.5%). From my understanding, this suggests that the method ranking depends on the judge. In addition, the labeler and simulator are still from the same family.
  2. Agreement between two LLM judges does not replace agreement with human annotators, and this agreement is not strong (Cohen’s κ = 0.48–0.53 for goal satisfaction). A small human-annotated subset would be enough to check the judge.
  3. My question about the latent states is not answered. The Smoother vs. Filtering comparison uses the same labeler model with a different input, so it does not show how the states change with a different labeler, or whether humans find them plausible.
  4. I understand that a live human study is not feasible during the rebuttal. However, the existing settings already allow a easy check: whether the simulator ranks the unseen assistants in the same order as real users. This would give some evidence of downstream value.

Since these points are not resolved, I am inclined to maintain my current score.

Reviewer_fEt3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies training LLM-based user simulators that generate a natural-language latent user state before each utterance, with the goal of off-policy (counterfactual) robustness to unseen agents.

Critical strengths
  1. The paper gives a principled account of training on LLM-generated latent states. The formulation leads to a practical procedure based on cached labeler samples and weighted SFT, and makes the tradeoff between following the labeler and fitting observed behavior explicit. I also appreciate the discussion of how thinking traces change the sampling target. This provides useful guidance for designing latent-state simulators beyond the particular algorithm evaluated here.

  2. The distinction between smoothing during annotation and causal conditioning during simulation is a valuable takeaway. The paper explains why later user responses can help infer earlier states while the deployed simulator still uses only the available history. The comparison supports this choice: smoothing improves deeper-turn behavior and satisfaction calibration over filtering, despite nearly identical utterance likelihood. This is a useful result independent of the proposed reweighting.

Critical weaknesses
  1. The experiments support latent-state supervision and smoothing more clearly than they support SNIS-EM reweighting. Compared with the α=0 smoother, reweighting leaves held-out likelihood almost unchanged. The smaller generalization gap mainly comes from lower training likelihood. Behavioral gains are also mixed: α=0.008 improves turn-count distance and overall act divergence, but worsens satisfaction calibration and does not improve deeper-turn act divergence. With evaluation limited to one shopping domain and one model family, the evidence does not yet justify the broad robustness claims or clearly establish what reweighting adds.

  2. It is unclear how α should be selected before testing on unseen agents. The paper reports four values, with different settings performing best on different metrics, but does not describe a validation procedure for choosing among them. This leaves open whether the reported gains would hold for a setting chosen independently of the evaluation results. A validation-based choice of α, followed by reporting its performance across all metrics, would make the empirical case substantially stronger.

Questions for authors
  1. How do you choose α without access to the unseen-agent evaluation data? Was a separate validation set used, and what results do you obtain when α is selected on held-out conversations with seen agents and then fixed for evaluation?

  2. For M=15 and the reported α values, what is the distribution of effective sample size across conversations? This would help establish how much the importance weights differ from uniform weighting and explain the behavioral changes relative to α=0.

  3. Can you report paired-bootstrap confidence intervals for the differences between α=0.008/0.012 and the α=0 smoother in turn-count W₁, dialogue-act divergence, and satisfaction calibration? These comparisons are central to assessing whether reweighting gives a reliable improvement.

  4. Were the judge’s dialogue-act and satisfaction labels checked against human annotations? Since the judge, simulators, and labeler all use Gemini models, evidence that the judge agrees with humans and does not systematically favor particular simulator outputs would strengthen the evaluation.

3 comments
AuthorsOpenReview ↗

Response to Reviewer fEt3 (1/2)

Thank you for the thoughtful review and constructive questions. We address the weakness points raised and your questions below.

1. Value of SNIS-EM reweighting vs. latent-state supervision / smoothing, flat LLeval\text{LL} _ {\text{eval}}, and scope (W1)

SNIS-EM vs. latent-state supervision. We address this above in the common response.

Why LLeval\text{LL} _ {\text{eval}} is flat while LLtrain\text{LL} _ {\text{train}} decreases. As detailed in Appendix F.2, LLeval\text{LL} _ {\text{eval}} scores human utterances teacher-forced on fixed posterior-smoother states ztz_t. Conditioned on a detailed 99-field state right before utu_t, utterance cross-entropy saturates around −1.51-1.51 nats/token across all latent-state models (Filtering and Smoother score −1.519-1.519 and −1.518-1.518). In closed-loop evaluation, however, the simulator must generate its own ztz_t from the conversation history; when α>0\alpha \gt 0, this prevents PθP _ \theta from memorizing prefix-unpredictable seen-agent annotations (monotonically shrinking ∣Δlog⁡Pθ∣\vert\Delta\log P _ \theta\vert from 0.4370.437 to 0.3090.309), which translates into better closed-loop rollout calibration.

To address this teacher-forcing caveat directly, we also re-evaluated all simulators with a K=5K = 5 causal bootstrap particle filter that rolls out zt∼Pθ(⋅∣Htaug)z_t \sim P _ \theta(\cdot \mid H_t^{\text{aug}}) strictly from prefix history (Table C3 in the common response above): on 3→83 \to 8, every (M=5,α>0)(M = 5, \alpha \gt 0) run and (M=15,α=0.012)(M = 15, \alpha = 0.012) achieves higher held-out causal marginal likelihood than α=0\alpha = 0, and on 5→65 \to 6, α>0\alpha \gt 0 matches α=0\alpha = 0 on held-out causal likelihood while significantly shrinking causal Δlog⁡Pθ\Delta\log P _ \theta (p≤0.001p \le 0.001).

We will add these discussions as well as the causal particle-filter evaluations in the revised paper.

Marginal contribution of reweighting vs. sampling. We address this above in the common response (Table C1).

Domain and model scope. Testing counterfactual robustness requires human multi-turn dialogues collected against multiple controlled assistant policies — this is why we use ConvApparel-V2, the only dataset we are aware of that has such distinct strategies (1313 of them). Standard single-policy or unlabelled chat logs (e.g., WildChat) lack this. To address single-family evaluation bias, we report an independent OpenAI GPT-5.4 Mini judge re-evaluation in the common response above (Table C4).

2. Selecting α\alpha without unseen-agent test leakage & seen-agent validation (W2 & Q1)

  • Why four α\alpha values were shown. Simulator fidelity spans several behavioral dimensions (turn count, dialogue acts, goal satisfaction, repetition); unfortunately, there is no single agreed-upon scalar metric prior to fixing a downstream task. Rather than tuning α\alpha on unseen-agent results, we chose {0.005,0.008,0.012,0.020}\lbrace 0.005, 0.008, 0.012, 0.020\rbrace a priori from training-set ESS alone (ESS/15≈\text{ESS}/15 \approx {73%, 51%, 33%, 19%}, zero unseen-agent data) to expose the trade-off curve transparently.
  • Why held-out seen-agent dialogues cannot select α\alpha. All 1,5951{,}595 seen-agent dialogues on 5→65 \to 6 were used for training (the 128128 dialogues in LLtrain\text{LL} _ {\text{train}} are an in-training sample, Appendix F.2). More fundamentally, held-out dialogues with seen agents have zero assistant policy shift, so α=0\alpha = 0 always fits the seen-agent distribution most closely (LLtrain\text{LL} _ {\text{train}} is highest at α=0\alpha = 0, −1.081-1.081 vs. −1.209-1.209 at α=0.020\alpha = 0.020). Because α>0\alpha \gt 0 regularizes against seen-agent overfitting—so behavior transfers under policy shift—validation requires either an ESS rule or a policy-shift validation split.
  • Two concrete zero-leakage selection procedures:
    • (i) Zero-validation default (50%-ESS rule): Fixing ESS/M≈50\text{ESS}/M \approx 50% (α≈1/σℓ\alpha \approx 1/\sigma _ \ell from training log-ratios alone, before touching any evaluation data) uniquely picks α=0.008\alpha = 0.008 for M=15M = 15. Reported across every metric in Table 1, this single pre-selected setting beats unweighted smoothing (α=0\alpha = 0) on turn W1W_1 (0.1880.188 vs. 0.2700.270), mean turns (3.553.55 vs. 3.623.62, human 3.503.50), all-turn L1actL_1^{\text{act}} (0.2700.270 vs. 0.3520.352), 1010-act L1act,t≥3L_1^{\text{act}, t \ge 3} (0.2810.281 vs. 0.3630.363; 0.2300.230 vs. 0.2270.227 on 77-act), TTR (0.7440.744 vs. 0.7260.726), repetition (0.030.03 vs. 0.040.04), human-likeness (3.843.84 vs. 3.803.80), and Δlog⁡Pθ\Delta\log P _ \theta (−0.388-0.388 vs. −0.437-0.437), trading off only ∣ΔSat∣\vert\Delta\text{Sat}\vert (6.66.6 vs. 2.02.0 pp).
    • (ii) Pseudo-unseen policy validation: When a practitioner prioritizes a specific target metric (such as ∣ΔSat∣\vert\Delta\text{Sat}\vert) and has multiple seen policies, α\alpha should be selected by holding out 1–21\text{--}2 seen policies during training as a validation policy-shift split (e.g., 4→14 \to 1 inside the 55 seen agents).
AuthorsOpenReview ↗

Response to Reviewer fEt3 (2/2)

3. ESS distribution across conversations for M=15M = 15 (Q2)

On training conversations with the Smoother (M=15M = 15), the within-conversation standard deviation of the raw log-ratio ℓj=log⁡Pθ(U,Z(j)∣A)−log⁡Qlab(τ(j),Z(j)∣Y)\ell_j = \log P _ \theta(U, Z^{(j)} \mid A) - \log Q _ {\text{lab}}(\tau^{(j)}, Z^{(j)} \mid Y) is σℓ≈108.7\sigma _ \ell \approx 108.7 nats. Across {0,0.005,0.008,0.012,0.020}\lbrace 0, 0.005, 0.008, 0.012, 0.020\rbrace, target effective sample sizes span ESS/15≈\text{ESS}/15 \approx {100%, 73%, 51%, 33%, 19%} (i.e., 15.0,10.9,7.7,5.0,2.815.0, 10.9, 7.7, 5.0, 2.8 effective proposals out of 1515); live logged trainer mean ESS/15\text{ESS}/15 at α=0.008\alpha = 0.008 is 47.347.3% (7.09/157.09/15) on 5→65 \to 6 and 48.848.8% (7.3/157.3/15) on 3→83 \to 8. See Table C2 in the common response above for the logged ESS quantiles (mean, median, [p10, p90]) and scaled log-weight spreads across all four nonzero α\alpha values on 5→65 \to 6; we will add the ESS distribution plot to Appendix D.3.

4. Paired-bootstrap confidence intervals (Q3)

We report the B=10,000B = 10{,}000 paired-bootstrap 95% CIs (Method A − Method B) across the n=256n = 256 shared evaluation seeds (Table F1; negative values improve ↓\downarrow metrics, * = 95% CI excludes 00):

Table F1 — Paired-Bootstrap 95% CIs (Method A − Method B, B=10,000B = 10{,}000, n=256n = 256)

(a) Requested M = 15 comparisons on 5→65 \to 6 (W_1, L_1^act, L_1^(act, t>=3), |ΔSat|):

Comparison (Method A vs. Method B)W1↓W_1 \downarrowL1act↓L_1^{\text{act}} \downarrowL1act,t≥3↓L_1^{\text{act}, t\ge 3} \downarrow∣ΔSat∣\vert\Delta\text{Sat}\vert pp ↓\downarrow
RUSTLeD (15,0.008)(15, 0.008) vs. Smoother (15,0)(15, 0)−0.082 [−0.174,+0.130]−0.082 [−0.229,+0.161]+0.004 [−0.179,+0.181]+4.7 [−1.2,+8.9]
RUSTLeD (15,0.012)(15, 0.012) vs. Smoother (15,0)(15, 0)−0.070 [−0.215,+0.188]+0.059 [−0.104,+0.247]+0.023 [−0.120,+0.178]−1.2 [−4.7,+3.7]

(b) Statistically significant 95% CI improvements from α>0\alpha \gt 0 (across M=15M=15, fixed-M=5M=5, and ESS=5\text{ESS}=5 matched controls):

Comparison (Method A vs. Method B)MetricDifference [95% CI]pp-value
5→65\to 6: RUSTLeD (15,0.008)(15, 0.008) vs. Smoother (15,0)(15, 0)Gem Push. ↑\uparrow+0.18 [+0.04,+0.32]*0.016
5→65\to 6: RUSTLeD (15,0.012)(15, 0.012) vs. Smoother (15,0)(15, 0)TTR err ↓\downarrow−0.031 [−0.052,−0.009]*0.005
5→65\to 6: RUSTLeD (15,0.012)(15, 0.012) vs. Smoother (15,0)(15, 0)Causal Δlog⁡Pθ↑\Delta\log P _ \theta \uparrow+0.086 [+0.059,+0.114]*<10−4\lt 10^{-4}
5→65\to 6: RUSTLeD (5,0.030)(5, 0.030) vs. Smoother (5,0)(5, 0)Turns err ↓\downarrow−0.40 [−0.65,−0.08]*0.016
5→65\to 6: RUSTLeD (5,0.030)(5, 0.030) vs. Smoother (5,0)(5, 0)Rep.% ↓\downarrow−0.21 [−0.44,−0.05]*0.002
5→65\to 6: RUSTLeD (5,0.030)(5, 0.030) vs. Smoother (5,0)(5, 0)GPT Hum. ↑\uparrow+0.13 [+0.02,+0.25]*0.028
3→83\to 8: RUSTLeD (5,0.012)(5, 0.012) vs. Smoother (5,0)(5, 0)Gem Hum. ↑\uparrow+0.38 [+0.23,+0.54]*<10−4\lt 10^{-4}
3→83\to 8: RUSTLeD (5,0.012)(5, 0.012) vs. Smoother (5,0)(5, 0)GPT Hum. ↑\uparrow+0.23 [+0.10,+0.36]*0.0004
3→83\to 8: RUSTLeD (5,0.012)(5, 0.012) vs. Smoother (5,0)(5, 0)GPT ∣ΔSat∣\vert\Delta\text{Sat}\vert pp ↓\downarrow−10.2 [−14.1,−1.9]*0.021
5→65\to 6: RUSTLeD (15,0.012)(15, 0.012) vs. Smoother (5,0)(5, 0) [ESS=5]GPT Hum. ↑\uparrow+0.29 [+0.17,+0.41]*<10−4\lt 10^{-4}
5→65\to 6: RUSTLeD (15,0.012)(15, 0.012) vs. Smoother (5,0)(5, 0) [ESS=5]GPT Viol.% ↓\downarrow−3.91 [−7.39,−0.75]*0.027

With M=15M = 15 smoothing on 5→65 \to 6, point estimates improve on W1W_1 (−0.082-0.082), L1actL_1^{\text{act}} (−0.082-0.082), and ∣ΔSat∣\vert\Delta\text{Sat}\vert (−1.2-1.2 pp at α=0.012\alpha = 0.012), while their 95% CIs span 00 at n=256n = 256—consistent with diminishing marginal returns with 1515 smoother proposals alongside significant gains in Push., TTR calibration, and causal generalization gap (Δlog⁡Pθ\Delta\log P _ \theta). Isolating α>0\alpha \gt 0 at M=5M = 5 (and at matched ESS=5\text{ESS} = 5) yields significant 95% CI improvements on both splits (Table F1(b)).

5. Judge validation and Gemini model-family independence (Q4)

We address this above in the common response (Table C4).

Once again, thanks for the review. We would be happy to answer any further questions before the review period ends today.

Reviewer_1nzyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RUSTLED trains LLM user simulators that generate an explicit latent state before each utterance. The paper reframes the common practice of prompting an external LLM to label latent states and then fine-tuning on those labels as posterior-regularized marginal-likelihood maximization, with the labeler acting as a tunable-strength regularizer rather than ground truth. A single parameter α interpolates between labeler distillation ( α → 0) and marginal-likelihood estimation ( α → 1); an exact variational identity yields a self-normalized importance-sampling EM algorithm. The authors also argue the labeler should use the full conversation (smoothing) rather than the prefix (filtering), and prove that filtering-trained simulators become state-blind and incur irreducible off-policy error under state shift. Experiments on a conversational recommendation benchmark with a counterfactual assistant split compare these variants on likelihood, turn statistics, LLM-judged dialogue acts, and goal satisfaction.

Critical strengths

The reframing is clarifying. Treating label-based SFT as the α → 0 end of a family containing marginal-likelihood estimation is a clean observation, and the first-order expansion (distillation at zeroth order, log-density-ratio policy-gradient correction at first order) explains why current practice works and what it leaves unused.

The theory is load-bearing. The filtering proposition is short and correct, and the accompanying theorem and corollary quantify the resulting off-policy error. The two-user-type worked example, where a filter-trained simulator predicts no improvement for an agent that substantially improves engagement, is the paper's most persuasive artifact.

The hindsight argument is correct. Allowing the labeler to use the full conversation while keeping the simulator causal is a proper application of Baum-Welch logic, stated compactly.

The experimental design targets the right question. Unseen assistants differ in interaction strategy rather than phrasing, so success requires responding to new conversational moves.

Qualitative examples match quantified failures. The three case studies each illustrate a failure mode the paper measures (over-satisfying goals, verbatim repetition, accepting requirement-violating items).

Critical weaknesses

The positive-α gain is confounded with proposal count. The strongest baseline (α=0, smoothing) already achieves turn distance 0.270, deeper-turn act divergence 0.227, repetition 0.04, and a satisfaction gap of 2.0 pp; RUSTLED at α=0.008 reports 0.188, 0.230, and within 3.1 pp. The act-divergence gain is essentially nil, and the paper concedes that the strongest configurations sit within one standard error. Moreover, at α=0, going from five to fifteen proposals already improves turn distance from 0.461 to 0.270 and act divergence from 0.355 to 0.227, so much of the attributed gain may come from having more proposals rather than from reweighting. The missing control fixes fifteen proposals, sets α=0, and subsamples k proposals to match the effective number surviving reweighting.

The finite-sample objective of Algorithm 1 is underspecified. The main text says self-normalization yields a bound tightening with sample count, but the IWAE analogy does not carry over directly: weights are raised to α and the denominator is a joint over traces and states, not a marginal. The loss line's stop-gradient annotation does not say whether the log-weight factor is normalized, nor whether the normalization term is differentiated. These choices yield different finite-sample objectives with different bias directions relative to the target.

The scale of α is constrained theoretically but unvalidated empirically. The main text requires α to shrink as the inverse square root of turn count to retain effective sample size, but the constant is unspecified and no diagnostic of the scaled log-ratio is reported. Since the log-ratio accumulates over turns and tokens, the first-order expansion may not hold at the values used. The paper should report the empirical distribution of the scaled log-ratio and an effective-sample-size diagnostic per α.

The theory–experiment bridge for smoothing versus filtering is not built. The theorem is single-step and the experiments are closed-loop multi-turn. The observed degradation of the filter-trained simulator is not connected to a concrete state-distribution shift in the data. Estimating the shift induced by a specific unseen persona and comparing the predicted error to the observed satisfaction bias would close this gap.

No matched-budget control. The method costs fifteen stronger-labeler calls per conversation against a smaller simulator. Under the same labeling budget, would more conversations with α=0 labeling outperform RUSTLED? Given that five versus fifteen proposals already moves turn distance by 0.19 at α=0, this is a real concern.

Questions for authors

What is the finite-sample objective of Algorithm 1, and its bias direction and rate relative to the target? Is the log-weight factor normalized, and is the normalization term differentiated?

What is the empirical distribution of the scaled log-ratio, and is the sub-Gaussian assumption on per-turn increments plausible? Could α α be selected adaptively by targeting a fixed effective-sample-size fraction?

Can the worked example be connected to the data by estimating the state-distribution shift induced by an upselling persona?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1nzy (1/3)

Thank you for the thorough technical review and detailed, pertinent questions. We address the weakness points raised and questions below.

1. Multi-proposal sampling (MM) vs. reweighting (α\alpha), ESS-matched control, and CIs (W1)

Marginal gains of reweighting. We address this above in the common response.

ESS-matched uniform subsample-kk control. Because the MM proposals per conversation are i.i.d. draws from QlabQ _ {\text{lab}}, uniformly subsampling k=5k = 5 of 1515 proposals at α=0\alpha = 0 is identically distributed to SFT + LS (Smoother, M=5,α=0M = 5, \alpha = 0) (ESS=5.0\text{ESS} = 5.0, Appendix F.3). At matched effective sample size, RUSTLeD (M=15,α=0.012,ESS≈5.0M = 15, \alpha = 0.012, \text{ESS} \approx 5.0) — and even α=0.020\alpha = 0.020 (ESS≈2.8\text{ESS} \approx 2.8) — strictly outperforms (M=5,α=0)(M = 5, \alpha = 0) on every behavioral metric on 5→65 \to 6 in Table 1: turn W1W_1 (0.1990.199 vs. 0.4610.461), all-turn L1actL_1^{\text{act}} (0.4100.410 vs. 0.4960.496), L1act,t≥3L_1^{\text{act}, t \ge 3} (0.2500.250 vs. 0.3550.355), ∣ΔSat∣\vert\Delta\text{Sat}\vert (0.80.8 vs. 5.15.1 pp), repetition (0.090.09 vs. 0.250.25, human 0.020.02), TTR (0.7570.757 vs. 0.6970.697, human 0.7760.776), and human-likeness (3.893.89 vs. 3.783.78). This demonstrates that reweighting adds benefits beyond those that can be explained by the variance reduction of the increased latent sample size.

Pareto trade-off at M=15M = 15 and paired-bootstrap CIs. At M=15M = 15 on 5→65 \to 6, α∈[0.008,0.012]\alpha \in [0.008, 0.012] traces a Pareto frontier rather than a single dominating point: α=0.008\alpha = 0.008 achieves the best turn W1W_1 (0.1880.188 vs. 0.2700.270 at α=0\alpha = 0), best all-turn L1actL_1^{\text{act}} (0.2700.270 vs. 0.3520.352, and 0.2810.281 vs. 0.3630.363 across all 1010 dialogue acts at t≥3t \ge 3), and best repetition (0.030.03), while α=0.012\alpha = 0.012 achieves the best ∣ΔSat∣\vert\Delta\text{Sat}\vert (0.80.8 vs. 2.02.0 pp) and second-best W1W_1 (0.1990.199).

In Table C1 above (and Table F1 in our response to Reviewer fEt3), we report B=10,000B = 10{,}000 paired-bootstrap 95% CIs across n=256n = 256 shared evaluation seeds. With M=15M = 15 smoothing, point estimates improve on W1W_1 (−0.082-0.082), L1actL_1^{\text{act}} (−0.082-0.082), and ∣ΔSat∣\vert\Delta\text{Sat}\vert (−1.2-1.2 pp) with 95% CIs crossing 00, while Push. (+0.18*, p=0.016p = 0.016 at α=0.008\alpha = 0.008), TTR error (−0.031*, p=0.005p = 0.005 at α=0.012\alpha = 0.012), and causal Δlog⁡Pθ\Delta\log P _ \theta (+0.086*, p<10−4p \lt 10^{-4} at α=0.012\alpha = 0.012, Table C3) improve significantly. Isolating α>0\alpha \gt 0 at M=5M = 5 (and at matched ESS=5\text{ESS} = 5) yields significant 95% CI improvements on both splits (Tables C1, F1).

We will refine the presentation of empirical results in the paper along these lines to remove potential misunderstandings or misinterpretations.

2. Finite-sample objective of Algorithm 1, normalization, stop-gradient, and bias (W2 & Q1)

Thank you for catching this underspecification and the imprecise IWAE sentence in Section 3.3. In Algorithm 1:

  • Si(j)=log⁡Pθ(Ui,Zi(j)∣Ai)S_i^{(j)} = \log P _ \theta(U_i, Z_i^{(j)} \mid A_i) and Li(j)=log⁡Qlab(τi(j),Zi(j)∣Yi)L_i^{(j)} = \log Q _ {\text{lab}}(\tau_i^{(j)}, Z_i^{(j)} \mid Y_i) are unnormalized sequence log-probabilities summed over all target tokens (not length-normalized).
  • w~i(j)=exp⁡(log⁡wi(j))/∑k=1Mexp⁡(log⁡wi(k))\tilde{w} _ i^{(j)} = \exp(\log w_i^{(j)}) / \sum _ {k=1}^M \exp(\log w_i^{(k)}) is self-normalized across the MM proposals of conversation ii.
  • sg(w~i(j))\text{sg}(\tilde{w} _ i^{(j)}) stops gradients through both the numerator and the normalization denominator.

Because QlabQ _ {\text{lab}} does not depend on θ\theta, −∇θL(θ)-\nabla _ \theta \mathcal{L}(\theta) is exactly the mini-batch gradient of

F^α,M(θ;Y)=1αlog⁡ ⁣(1M∑j=1M[Pθ(U,Z(j)∣A)Qlab(τ(j),Z(j)∣Y)]α),\hat{\mathcal{F}} _ {\alpha, M}(\theta; Y) = \frac{1}{\alpha} \log\!\left( \frac{1}{M} \sum _ {j=1}^M \left[ \frac{P _ \theta(U, Z^{(j)} \mid A)}{Q _ {\text{lab}}(\tau^{(j)}, Z^{(j)} \mid Y)} \right]^\alpha \right),

since ∇θWjα=αWjα∇θlog⁡Pθ(U,Z(j)∣A)\nabla _ \theta W_j^\alpha = \alpha W_j^\alpha \nabla _ \theta \log P _ \theta(U, Z^{(j)} \mid A), canceling both α\alpha and 1/M1/M.

AuthorsOpenReview ↗

Response to Reviewer 1nzy (2/3)

This gives an exact characterization of both bias sources:

  • Joint vs. marginal proposal (M→∞M \to \infty limit). For α∈(0,1)\alpha \in (0, 1), as M→∞M \to \infty, F^α,M(θ;Y)\hat{\mathcal{F}} _ {\alpha, M}(\theta; Y) converges almost surely to

F~α(θ;Y)=log⁡Pθ(U∣A)−1−ααDα(Pθ(Z∣U,A)∥Q~α(Z∣Y))+Cα(Y),\tilde{\mathcal{F}} _ \alpha(\theta; Y) = \log P _ \theta(U \mid A) - \frac{1-\alpha}{\alpha} D _ \alpha\left( P _ \theta(Z \mid U, A) \Vert \tilde{Q} _ \alpha(Z \mid Y) \right) + C _ \alpha(Y),

where Q~α(Z∣Y)∝Qlab(Z∣Y)exp⁡(α1−αH1−α(τ∣Z,Y))\tilde{Q} _ \alpha(Z \mid Y) \propto Q _ {\text{lab}}(Z \mid Y) \exp\left( \frac{\alpha}{1-\alpha} H _ {1-\alpha}(\tau \mid Z, Y) \right) is the trace-entropy-tilted labeler and Cα(Y)=1−ααlog⁡∑ZQlab(Z∣Y)exp⁡(α1−αH1−α(τ∣Z,Y))≥0C _ \alpha(Y) = \frac{1-\alpha}{\alpha} \log \sum_Z Q _ {\text{lab}}(Z \mid Y) \exp\left( \frac{\alpha}{1-\alpha} H _ {1-\alpha}(\tau \mid Z, Y) \right) \ge 0 is independent of θ\theta. Because Cα(Y)≥0C _ \alpha(Y) \ge 0, F^α,M\hat{\mathcal{F}} _ {\alpha, M} is not a direct lower bound on log⁡Pθ(U∣A)\log P _ \theta(U \mid A) — we will revise that sentence in Section 3.3 — but its population limit is the exact Rényi-regularized objective anchored to Q~α\tilde{Q} _ \alpha, up to a θ\theta-free constant.

  • Finite-MM bias direction and rate. For α>0\alpha \gt 0, by concavity of (1/α)log⁡(⋅)(1/\alpha) \log(\cdot) on the i.i.d. sample mean of WjαW_j^\alpha, E[F^α,M(θ;Y)]\mathbb{E}[\hat{\mathcal{F}} _ {\alpha, M}(\theta; Y)] is monotonically non-decreasing in MM and downward-biased relative to F~α(θ;Y)\tilde{\mathcal{F}} _ \alpha(\theta; Y):

E[F^α,M(θ;Y)]−F~α(θ;Y)=−12αM(ρ∞−1−1)+O(M−2),\mathbb{E}\left[\hat{\mathcal{F}} _ {\alpha, M}(\theta; Y)\right] - \tilde{\mathcal{F}} _ \alpha(\theta; Y) = -\frac{1}{2 \alpha M}\left(\rho _ \infty^{-1} - 1\right) + \mathcal{O}(M^{-2}),

where ρ∞=lim⁡M→∞ESS/M=E[Wα]2/E[W2α]∈(0,1]\rho _ \infty = \lim _ {M \to \infty} \text{ESS}/M = \mathbb{E}[W^\alpha]^2 / \mathbb{E}[W^{2\alpha}] \in (0, 1] (which reduces to −ασℓ22M+O(α2/M)-\frac{\alpha \sigma _ \ell^2}{2M} + \mathcal{O}(\alpha^2 / M) for small α\alpha).

We will write out F^α,M\hat{\mathcal{F}} _ {\alpha, M}, F~α\tilde{\mathcal{F}} _ \alpha, and this finite-MM expansion explicitly in Section 3.3 and Appendix D to be more precise!

3. Empirical scale of α\alpha, log-ratio spread, ESS diagnostics, and adaptive α\alpha (W3 & Q2)

We selected the four nonzero α\alpha values {0.005,0.008,0.012,0.020}\lbrace 0.005, 0.008, 0.012, 0.020\rbrace directly from ESS calibration on the training set. We should have documented this explicitly, and will do so in the revised paper.

On training dialogues with the Smoother (M=15M = 15), the within-conversation standard deviation of the unscaled log-ratio ℓj=log⁡Pθ(U,Z(j)∣A)−log⁡Qlab(τ(j),Z(j)∣Y)\ell_j = \log P _ \theta(U, Z^{(j)} \mid A) - \log Q _ {\text{lab}}(\tau^{(j)}, Z^{(j)} \mid Y) is σℓ≈108.7\sigma _ \ell \approx 108.7 nats, driven largely by sequence-level variation in labeler thinking-trace length. By comparison, per-turn Filtering accumulates ∼4.55\sim 4.55 labeler calls per conversation and has σℓ≈303.1\sigma _ \ell \approx 303.1 nats. Using the Variance Control bound in Appendix D.3, we chose {0,0.005,0.008,0.012,0.020}\lbrace 0, 0.005, 0.008, 0.012, 0.020\rbrace to span ESS/15≈\text{ESS}/15 \approx {100%, 73%, 51%, 33%, 19%} (i.e., 15.0,10.9,7.7,5.0,2.815.0, 10.9, 7.7, 5.0, 2.8 effective proposals; live logged trainer mean ESS/15\text{ESS}/15 at α=0.008\alpha = 0.008 is 47.347.3% on 5→65 \to 6 and 48.848.8% on 3→83 \to 8). For M=5M = 5 on 3→83 \to 8, because ESS=1\text{ESS} = 1 already equals 2020% of MM, reaching the near-top-11 regime shifts the upper grid to α∈{0.03,0.05}\alpha \in \lbrace 0.03, 0.05\rbrace. See Table C2 in the common response above for the exact logged ESS quantiles (median, [p10, p90], mean) and within-conversation scaled log-weight spreads (2.172.17 to 9.359.35 nats) across all four nonzero α\alpha values on 5→65 \to 6; we will add ESS and weight-distribution plots to Appendix D.3 in the revision.

First-order expansion and sub-Gaussianity. At α∈{0.005,0.008}\alpha \in \lbrace 0.005, 0.008\rbrace, ασℓ≈0.54–0.87\alpha \sigma _ \ell \approx 0.54\text{--}0.87, so the first-order covariance expansion in Section 3.3 provides an accurate qualitative picture. However, at α∈{0.012,0.020}\alpha \in \lbrace 0.012, 0.020\rbrace (ασℓ≈1.3–2.2\alpha \sigma _ \ell \approx 1.3\text{--}2.2), higher-order softmax concentration matters. We will clarify that the covariance expansion provides local intuition near α→0\alpha \to 0, whereas Lemma 1 and F^α,M\hat{\mathcal{F}} _ {\alpha, M} hold without Taylor approximation for all α∈(0,1]\alpha \in (0, 1]. Empirically, per-turn log-ratio increments show moderate skew from variable thinking-trace lengths rather than exact Gaussian tails—which is why Appendix D.3 proves both the sub-Gaussian bound in part (i) and the finite-CGF heavy-tailed threshold in part (ii)—and the predicted ESS retention closely tracks logged trainer ESS.

AuthorsOpenReview ↗

Response to Reviewer 1nzy (3/3)

Adaptive ESS-targeted α\alpha. Solving ESSi(αi)=ρM\text{ESS} _ i(\alpha_i) = \rho M per conversation by 1D bisection on the MM cached log-ratios is computationally negligible. One theoretical trade-off is that a per-conversation αi\alpha_i assigns different Rényi penalty weights (1−αi)/αi(1 - \alpha_i)/\alpha_i across conversations (e.g., stronger regularization on longer dialogues), altering the global objective unless α\alpha is solved globally across a batch or corpus. We will add this discussion to Section 3.3 and Appendix D.3.

4. Theory–experiment bridge for smoothing vs. filtering (W4 & Q3)

Theory and experiments play complementary roles here. Appendix E proves in a tractable single-step binary-outcome setting why prefix-only filtering leaves state-outcome coupling unidentified and incurs worst-case off-policy error Δ⋅cb(1−cb)⋅ρ\sqrt{\Delta \cdot c_b(1 - c_b) \cdot \rho} even with infinite on-policy data (with the single-step and known-dynamics caveats stated in Appendix E.5). In ConvApparel-V2, latent states ztz_t are 99-field natural-language descriptions rather than discrete indices k∈{1,…,K}k \in \lbrace 1, \dots, K\rbrace. Estimating discrete qb,qeq_b, q_e and χ2(qe∥qb)\chi^2(q_e \Vert q_b) for a specific unseen persona such as mild_upseller would require fitting an auxiliary clustering or discretization model whose specification choices would dominate the estimate. Instead, Section 4.2.4 tests the direct multi-turn empirical prediction of Appendix E: Filtering matches Smoother on teacher-forced likelihood (−1.519-1.519 vs. −1.518-1.518 on 5→65 \to 6) yet degrades sharply under assistant policy shift (4×4\times ∣ΔSat∣\vert\Delta\text{Sat}\vert, 4×4\times repetition, and higher L1act,t≥3L_1^{\text{act}, t \ge 3}: 0.3240.324 vs. 0.2270.227). Moreover, in Table C3 of the common response above, scoring causal marginal log-likelihood log⁡Pθ(U∣A)\log P _ \theta(U \mid A) via a K=5K = 5 causal bootstrap particle filter directly exposes Filtering's multi-turn transition degradation: Smoother significantly outperforms Filtering on 5→65 \to 6 (+0.031* at M=5M = 5, p=0.0002p = 0.0002; +0.018* at M=15M = 15, p=0.033p = 0.033), and RUSTLeD (α=0.012\alpha = 0.012) beats Filtering on 3→83 \to 8 by +0.018* / +0.024*. We will add this causal likelihood comparison in the revision.

5. Matched-budget control (W5)

  • Exact matched-budget control at M=5M = 5 (5N5N labeler calls, zero extra cost): Because SNIS reweighting (α>0\alpha \gt 0) reuses the exact same cached proposals and training steps as α=0\alpha = 0, comparing RUSTLeD (M=5,α>0)(M = 5, \alpha \gt 0) against Smoother (M=5,α=0)(M = 5, \alpha = 0) is a strict 11-to-11 budget-matched control (see Table C1 in the common response above). At identical cost, α>0\alpha \gt 0 yields significant wins on both splits: on 5→65 \to 6 (α=0.030\alpha = 0.030), turn error (−0.40*, p=0.016p = 0.016), repetition (0.25% → 0.04%, p=0.002p = 0.002), GPT-5.4 Mini Hum. (+0.13*, p=0.028p = 0.028, Table C4), and causal Δlog⁡Pθ\Delta\log P _ \theta (+0.042*, p=0.001p = 0.001, Table C3); on 3→83 \to 8 (α=0.012\alpha = 0.012), Hum. (+0.38* Gem / +0.23* GPT-5.4 Mini, p=0.0004p = 0.0004), Gem Push. (+0.41*), and GPT-5.4 Mini ∣ΔSat∣\vert\Delta\text{Sat}\vert (11.3 → 1.2 pp, p=0.021p = 0.021).
  • Beating 3×3\times unweighted labeling budget at 1/3×1/3\times the cost: Even against Smoother (M=15,α=0)(M = 15, \alpha = 0) (15N15N labeler calls and >2×\gt 2\times training steps), RUSTLeD at M=5M = 5 (5N5N calls) achieves better W1W_1 (0.176 vs. 0.270), L1actL_1^{\text{act}} (0.219 vs. 0.352), and L1act,t≥3L_1^{\text{act}, t \ge 3} (0.207 vs. 0.227) on 5→65 \to 6 (α=0.030\alpha = 0.030), and better W1W_1 (0.559 vs. 0.652) and Hum. (3.95 / 3.74 vs. 3.77 / 3.69) on 3→83 \to 8 (α=0.012\alpha = 0.012).

Thanks for clarifying my concerns, I will keep my rating.

Reviewer_rLw3OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies behavioral fidelity when user simulators interact with unseen conversational-agent policies. It proposes RUSTLeD, which treats natural-language state annotations as latent-variable inference rather than trusted ground truth.

  • A standout design feature is the α-regularized objective: it balances observed-dialogue fit with latent-state posterior alignment and, through the logarithm of an order-α power mean of a density ratio, elegantly unifies label distillation as α→0 with marginal log-likelihood optimization at α=1. Lemma 1’s exact variational form further provides a theoretical basis for SNIS-EM weighted fine-tuning.
  • A second emphasis is the separation between full-dialogue posterior smoothing during training and history-only generation at inference. Across two unseen-agent splits in apparel/footwear recommendation, latent-state fine-tuning improves several behavioral metrics, while reweighting offers additional but mixed benefits over unweighted smoothing.
Critical strengths

1. An impressive unification of label distillation and marginal-likelihood optimization.

  • The α-regularized likelihood design in Section 3.2 a particularly impressive strength of this paper. It brings together the model’s ability to explain observed dialogue and the alignment of its posterior with Q_lab’s latent-state annotation distribution, with α providing a clear, adjustable trade-off. Particularly elegant is the unified formulation as the logarithm of the order-α power mean of a density ratio: α→0 yields label distillation, while α=1 recovers marginal log-likelihood optimization. Beyond its mathematical elegance, this design explicitly connects conventional annotation-based fine-tuning with latent-variable learning, giving α a clear probabilistic interpretation rather than merely treating it as a heuristic loss weight.
  • Lemma 1’s exact variational identity further connects the theoretical objective to EM-style training. It identifies the optimal auxiliary state distribution and expresses the gradient as an expectation of complete-data log-likelihood gradients, grounding the subsequent importance reweighting, weighted supervised fine-tuning, and stop-gradient treatment of weights. This turns the abstract probabilistic objective into a training procedure implementable through sampling approximations;

2. Evaluation goes beyond surface resemblance.

  • Beyond turn counts and lexical statistics, evaluation covers deeper-turn dialogue acts, repetition, and goal-satisfaction calibration. On the 5→6 split, α=.008 reduces all-turn act distance from .352 for unweighted smoothing to .270; α=.012 reduces satisfaction bias from 2.0 to 0.8 percentage points. The appendix also reports regressions on the harder split, enabling a more balanced assessment.

3. Detailed theoretical and empirical support strengthens the paper.

  • The main text and appendices provide detailed theoretical derivations and experimental explanations that make the methodological rationale and interpretation of the results more convincing. Appendices D–E develop the variational objective, estimator analysis, and smoothing/filtering arguments, while discussing their assumptions and scope. Appendix F further examines the main observations through evaluation over all ten dialogue acts, the harder 3→8 split, and candidate-count analysis.
Critical weaknesses

None

Questions for authors

Clarify the scope and interpretation of the best-value highlighting for ΔlogP_θ ​ in Table 1.

  • This metric is defined as LL_eval ​− LL_train. If a smaller absolute gap is preferable, why is −0.309 for RUSTLeD ( α = 0.020 ) highlighted rather than the smaller absolute gaps of the Prompted baselines? Is the highlighting restricted to fine-tuned models?

Figure 2(b) appears to lack a legend distinguishing the green and blue curves.

  • In Figure 2(b), it is difficult to identify which metric each curve represents. Adding a legend or direct curve labels, along with clarifying which vertical axis each curve corresponds to, would improve readability.
1 comment
AuthorsOpenReview ↗

Response to Reviewer rLw3

Thank you for the careful review and constructive feedback. We answer both questions below.

1. Scope and interpretation of Δlog⁡Pθ\Delta\log P _ \theta highlighting in Table 1

The reviewer’s reading is correct: bold and underline in the Δlog⁡Pθ=LLeval−LLtrain\Delta\log P _ \theta = \text{LL} _ {\text{eval}} - \text{LL} _ {\text{train}} column compare fine-tuned models only (this is noted briefly in the Table 1 caption and explained in Appendix F.2).

We exclude Prompted (No LS) and Prompted (With LS) from this ranking because zero-shot models never see Dtrain\mathcal{D} _ {\text{train}}, and they fit human utterances poorly on both splits (LLeval=−2.943\text{LL} _ {\text{eval}} = -2.943 and −2.592-2.592 nats/token, compared with −1.513-1.513 to −1.519-1.519 for the fine-tuned latent-state models). Their small differences (−0.074-0.074 and −0.063-0.063) reflect baseline corpus differences between the two conversation sets rather than resistance to training-set overfitting. Among fine-tuned models on 5→65 \to 6, all Δlog⁡Pθ\Delta\log P _ \theta values are negative, so higher values (↑\uparrow, closer to 00) correspond to a smaller train-to-eval generalization gap, with RUSTLeD at α=0.020\alpha = 0.020 (−0.309-0.309) being best and α=0.012\alpha = 0.012 (−0.349-0.349) second best (vs. −0.437-0.437 at α=0\alpha = 0 and −2.058-2.058 for SFT (No LS)).

In the revision, we will state this scope explicitly in Section 4.1 (Evaluation Metrics) and visually mark the zero-shot rows in Table 1 and the Appendix F full-result tables so the comparison scope is clear at a glance.

2. Legend and vertical-scale mapping in Figure 2(b)

Thank you for flagging this. Figure 2(b) currently relies on colored vertical-scale titles (blue on the left for L1act,t≥3L_1^{\text{act}, t \ge 3} and green on the right for turn W1W_1) and caption symbols (▲ for M=15M = 15 and ⧫ for M=5M = 5 at α=0\alpha = 0). We agree that this makes the curves harder than necessary to parse. In the revision, we will add an explicit in-panel legend and direct curve and scale labels to both Figure 2(b) (5→65 \to 6) and its 3→83 \to 8 counterpart in Appendix F.4.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common Response (1/4)

We thank all the reviewers for their careful reviews, constructive feedback and helpful suggestions. We discuss points specific to individual reviews separately, but begin with points that arise in more than one review.

Multi-proposal sampling (MM) vs. reweighting (α\alpha), Marginal Gains of Reweighting, Value of Reweighting vs. Latent-state Supervision (Revs 1nzy and fEt3)

We agree that each of dynamic latent-state supervision, posterior smoothing, and multi-proposal sampling (M>1M > 1) drive large gains in simulation quality. Indeed, all three of these techniques are directly motivated by our framing of RUSTLeD using an explicit HMLLM & EM formulation, rather than standing outside it.

We want to emphasize that a primary contribution of our work is this probabilistic latent-variable formulation of natural-language latent-state annotation. Prior dynamic latent-state simulators fine-tune on single deterministic (M=1M = 1), prefix-only annotations treated as fixed ground truth. Our treatment of natural-language state annotation as posterior inference is precisely what dictates (i) full-conversation smoothing (Qlab(Z∣Y)Q_{\text{lab}}(Z \mid Y)) during labeling, (ii) prefix-only conditioning (Pθ(zt∣Htaug)P_\theta(z_t \mid H_t^{\text{aug}})) at rollout time, (iii) M>1M > 1 posterior draws, and (iv) likelihood-ratio reweighting (α>0\alpha > 0). As obvious as it may be in hindsight, our multi-sampling approach is itself a vital contribution of the paper. We will make modifications of emphasis in the paper to ensure this point does not get overlooked.

Marginal gains of reweighting. If we take (M=5,α=0)(M = 5, \alpha = 0) as a baseline and compare the impact of (M=15,α=0)(M = 15, \alpha = 0) to that of (M=15,α>0)(M = 15, \alpha > 0) as in Table 1, this does create the impression that the impacts of reweighting are marginal. This is an unintended consequence of the omission of the numbers for (M=5,α>0)(M = 5, \alpha > 0), an omission we will fix. What we actually find is diminishing returns when applying reweighting and multi-sampling jointly, so we need to evaluate both intervention paths: baseline →\to reweight →\to reweight & multi-sample and baseline →\to multi-sample →\to reweight and multi-sample.

Starting from SFT + LS (Smoother, M=5,α=0M = 5, \alpha = 0) on the 3→83 \to 8 split (Appendix F.4):

  • reweighting alone (RUSTLeD, M=5,α=0.03M = 5, \alpha = 0.03, zero extra labeler calls) gives a larger improvement than 3×3\times multi-sampling (M=15,α=0M = 15, \alpha = 0) on turn W1W_1 (0.852→0.6130.852 \to 0.613 vs. 0.6520.652) and all-turn L1actL_1^{\text{act}} (0.637→0.5270.637 \to 0.527 vs. 0.5550.555),
  • whereas 3×3\times multi-sampling gives a larger improvement on deeper-turn L1act,t≥3L_1^{\text{act}, t \ge 3} (0.488→0.3400.488 \to 0.340 vs. 0.3980.398), ∣ΔSat∣\vert\Delta\text{Sat}\vert (5.5→1.25.5 \to 1.2 pp vs. 3.53.5 pp), and repetition (0.19→0.140.19 \to 0.14 vs. 0.290.29).
  • Combining both (M=15,α>0M = 15, \alpha > 0) exhibits diminishing returns on individual metrics while reaching the best overall turn W1W_1 (0.4570.457 at α=0.012\alpha = 0.012), repetition (0.040.04 at α=0.008\alpha = 0.008), TTR (0.7780.778), and generalization gap (−0.118-0.118 at α=0.020\alpha = 0.020).
AuthorsOpenReview ↗

Common Response (2/4)

Furthermore, we have trained M=5M = 5 sweeps across α∈{0.005,0.008,0.012,0.020,0.030,0.050}\alpha \in \{0.005, 0.008, 0.012, 0.020, 0.030, 0.050\} on 5→65 \to 6 and α∈{0.008,0.012,0.020,0.030}\alpha \in \{0.008, 0.012, 0.020, 0.030\} on 3→83 \to 8, reporting the representative 2×22\times 2 operating points in Table C1 (bold = best per column in each split; paired-bootstrap pp-values and 95% CIs are in the bullets below and Table F1):

Table C1 — Two Intervention Paths from Smoother (5, 0): Reweighting Alone (M=5,α>0M=5, \alpha>0) vs. Multi-Sampling (M=15,α=0M=15, \alpha=0) and Both (M=15,α>0M=15, \alpha>0)

SplitMethod (M,αM, \alpha)TurnsW1↓W_1 \downarrowRep.% ↓\downarrowL1act/L1act,t≥3↓L_1^{\text{act}} / L_1^{\text{act}, t\ge 3} \downarrow∣ΔSat∣\vert\Delta\text{Sat}\vert pp ↓\downarrowHum. ↑\uparrow
5→65\to 6 ReferenceReal Humans3.50—0.02——4.08
5→65\to 6 BaselineSmoother (5,05, 0)3.960.4610.250.496 / 0.3555.13.78
5→65\to 6 Reweight onlyRUSTLeD (5,0.0305, 0.030)3.560.1760.040.219 / 0.2078.23.87
5→65\to 6 Reweight onlyRUSTLeD (5,0.0505, 0.050)3.550.2340.050.242 / 0.1727.03.70
5→65\to 6 Multi-sample onlySmoother (15,015, 0)3.620.2700.040.352 / 0.2272.03.80
5→65\to 6 Both (Reweight+MM)RUSTLeD (15,0.00815, 0.008)3.550.1880.030.270 / 0.2306.63.84
5→65\to 6 Both (Reweight+MM)RUSTLeD (15,0.01215, 0.012)3.330.1990.090.410 / 0.2500.83.89
3→83\to 8 ReferenceReal Humans3.33—0.02——4.00
3→83\to 8 BaselineSmoother (5,05, 0)4.140.8520.190.637 / 0.4885.53.57
3→83\to 8 Reweight onlyRUSTLeD (5,0.0125, 0.012)3.880.5590.140.574 / 0.54710.23.95
3→83\to 8 Multi-sample onlySmoother (15,015, 0)3.460.6520.140.555 / 0.3401.23.77
3→83\to 8 Both (Reweight+MM)RUSTLeD (15,0.01215, 0.012)3.340.4570.090.641 / 0.4572.03.76
  • On 5→65 \to 6: Reweighting alone at M=5M = 5 (α=0.030\alpha = 0.030) significantly cuts turn-count error (−0.40*, p=0.016p = 0.016) and verbatim repetition (0.25% → 0.04%, p=0.002p = 0.002), and outperforms 3×3\times multi-sampling (M=15,α=0)(M=15, \alpha=0) on W1W_1, L1actL_1^{\text{act}}, L1act,t≥3L_1^{\text{act}, t\ge 3}, and Hum., while (M=15,α=0)(M=15, \alpha=0) yields lower ∣ΔSat∣\vert\Delta\text{Sat}\vert (2.0 vs. 8.2 pp). Increasing α\alpha further to 0.0500.050 achieves the lowest L1act,t≥3L_1^{\text{act}, t\ge 3} (0.172) at the cost of lower Hum. (3.70 vs. 3.87), showing a trade-off past α=0.030\alpha = 0.030.
  • On 3→83 \to 8: Across the four M=5M=5 runs, all improve causal likelihood (−1.865 to −1.871 vs. −1.880, Table C3), with α=0.012\alpha = 0.012 achieving the best behavioral trade-off: reducing W1W_1 to 0.559 (below M=15,α=0M=15, \alpha=0 at 0.652) and significantly improving Hum. (+0.38*, p<10−4p < 10^{-4} under Gemini; +0.23*, p=0.0004p = 0.0004 under GPT-5.4 Mini, Table C4) and Push. (+0.41*), while adding M=15M = 15 reaches W1=0.457W_1 = 0.457 (−0.395*, p=0.006p = 0.006).

We will fix this omission, add these results, and clarify these points explicitly in the revised paper.

Training ESS diagnostics (Revs 1nzy and fEt3)

Reviewers 1nzy and fEt3 also asked how strongly SNIS weights depart from uniform weighting across conversations and whether our analytical ESS scaling matches actual training behavior. In Algorithm 1, conversation ii's MM proposals receive self-normalized weights w~i(j)=softmaxj(αℓi(j))\tilde{w} _ i^{(j)} = \text{softmax} _ j(\alpha \ell_i^{(j)}) where ℓi(j)=log⁡Pθ(Ui,Zi(j)∣Ai)−log⁡Qlab(τi(j),Zi(j)∣Yi)\ell_i^{(j)} = \log P _ \theta(U_i, Z_i^{(j)} \mid A_i) - \log Q _ {\text{lab}}(\tau_i^{(j)}, Z_i^{(j)} \mid Y_i), giving per-conversation ESSi=1/∑j=1M(w~i(j))2∈[1,M]\text{ESS} _ i = 1 / \sum _ {j=1}^M (\tilde{w} _ i^{(j)})^2 \in [1, M]. Table C2 reports the logged batch-mean ESSi\text{ESS} _ i and within-conversation scaled log-weight spread max⁡jαℓi(j)−min⁡jαℓi(j)\max_j \alpha\ell_i^{(j)} - \min_j \alpha\ell_i^{(j)} over the final 50 training steps up to the evaluated checkpoint on 5→65 \to 6, closely matching our analytical targets (we will add ESS and weight-spread distribution plots to Appendix D.3 in the revision):

Table C2 — Logged Training-Time SNIS ESS & Scaled Log-Weight Spread on 5→65 \to 6

ConfigurationTarget ESS/M\text{ESS}/MLogged ESS\text{ESS} Mean (Med [p10, p90])Logged ESS/M\text{ESS}/MScaled Spread max⁡jαℓj−min⁡jαℓj\max_j \alpha\ell_j - \min_j \alpha\ell_j [p10, p90] (nats)
M=15,α=0.005M=15, \alpha=0.00573%10.42 (10.47 [9.37, 11.52])69.5%2.17 [1.86, 2.45]
M=15,α=0.008M=15, \alpha=0.00851%7.09 (6.99 [5.93, 8.42])47.3%3.54 [3.02, 4.01]
M=15,α=0.012M=15, \alpha=0.01233%4.45 (4.32 [3.49, 5.45])29.7%5.45 [4.67, 6.12]
M=15,α=0.020M=15, \alpha=0.02019%2.32 (2.28 [1.70, 2.86])15.5%9.35 [8.02, 10.44]
M=5,α∈{.005,.008,.012,.020,.030,.050}M=5, \alpha \in \{.005,.008,.012,.020,.030,.050\}—3.91, 3.14, 2.45, 1.76, 1.42, 1.1978.3% → 23.8%1.44 → 15.59
AuthorsOpenReview ↗

Common Response (3/4)

Causal marginal log-likelihood via K=5K = 5 particle filtering (Revs fEt3, 1nzy, and rLw3)

Reviewer fEt3 observed that held-out LLeval\text{LL} _ {\text{eval}} in Table 1 was nearly flat across Filtering, Smoother (α=0\alpha = 0), and RUSTLeD (α>0\alpha \gt 0); Reviewer 1nzy asked for a quantitative multi-turn bridge showing prefix Filtering's off-policy transition degradation; and Reviewer rLw3 asked about interpreting the train–eval gap Δlog⁡Pθ=LLeval−LLtrain\Delta\log P _ \theta = \text{LL} _ {\text{eval}} - \text{LL} _ {\text{train}}. In the submitted paper, LLeval\text{LL} _ {\text{eval}} teacher-forced Pθ(ut∣Htaug,zt(0))P _ \theta(u_t \mid H_t^{\text{aug}}, z_t^{(0)}) on a single hindsight smoother draw (at index 0 or zt(0)∼Qlab(⋅∣Y)z_t^{(0)} \sim Q _ {\text{lab}}(\cdot \mid Y)), which saturates utterance cross-entropy near −1.51-1.51 nats/token and omits the simulator's learned causal transition prior Pθ(zt∣Htaug)P _ \theta(z_t \mid H_t^{\text{aug}}).

To evaluate true causal marginal log-likelihood log⁡Pθ(U∣A)\log P _ \theta(U \mid A) strictly from prefix history, we re-scored all 28 simulators with a K=5K = 5 causal bootstrap particle filter that rolls out zt(k)∼Pθ(⋅∣Htaug,(k))z_t^{(k)} \sim P _ \theta(\cdot \mid H_t^{\text{aug}, (k)}) causally from prefix history, weights particles by Pθ(ut∣Htaug,(k),zt(k))P _ \theta(u_t \mid H_t^{\text{aug}, (k)}, z_t^{(k)}), and resamples before turn t+1t + 1 (Table C3):

Table C3 — Causal K=5K = 5 Particle-Filter Marginal Log-Likelihood (nats/user token; bold = best per column in each split)

Split & MethodCausal LLeval↑\text{LL} _ {\text{eval}} \uparrowCausal LLtrain\text{LL} _ {\text{train}}Causal Δlog⁡Pθ↑\Delta\log P _ \theta \uparrow
5→65\to 6: SFT (No LS)−2.955−1.428−1.527
5→65\to 6: Filtering (M=5/M=15,α=0M=5 / M=15, \alpha=0)−1.882 / −1.879−1.684 / −1.495−0.198 / −0.384
5→65\to 6: Smoother (M=5,α=0M=5, \alpha=0)−1.851−1.640−0.211
5→65\to 6: RUSTLeD (M=5,α=0.030M=5, \alpha=0.030)−1.867−1.698−0.169
5→65\to 6: RUSTLeD (M=5,α=0.050M=5, \alpha=0.050)−1.879−1.713−0.166
5→65\to 6: Smoother (M=15,α=0M=15, \alpha=0)−1.861−1.452−0.409
5→65\to 6: RUSTLeD (M=15,α=0.012M=15, \alpha=0.012)−1.869−1.547−0.323
5→65\to 6: RUSTLeD (M=15,α=0.020M=15, \alpha=0.020)−1.877−1.583−0.293
3→83\to 8: SFT (No LS)−1.905−1.424−0.482
3→83\to 8: Filtering (M=5/M=15,α=0M=5 / M=15, \alpha=0)−1.889 / −1.873−1.940 / −1.688+0.052 / −0.185
3→83\to 8: Smoother (M=5,α=0M=5, \alpha=0)−1.880−1.926+0.046
3→83\to 8: RUSTLeD (M=5,α=0.012M=5, \alpha=0.012)−1.870−1.916+0.046
3→83\to 8: RUSTLeD (M=5,α=0.020M=5, \alpha=0.020)−1.865−1.934+0.069
3→83\to 8: Smoother (M=15,α=0M=15, \alpha=0)−1.858−1.677−0.181
3→83\to 8: RUSTLeD (M=15,α=0.012M=15, \alpha=0.012)−1.849−1.739−0.110
  • Key findings (B=10,000B = 10{,}000 paired bootstrap, * marks p<0.05p \lt 0.05):
    • (1) With causal rollouts, Smoother significantly beats prefix Filtering on held-out log⁡Pθ(U∣A)\log P _ \theta(U \mid A) on 5→65 \to 6 (+0.031*, p=0.0002p = 0.0002 at M=5M = 5; +0.018*, p=0.033p = 0.033 at M=15M = 15, and +1.09* vs. SFT No LS), exposing Filtering's multi-turn transition degradation.
    • (2) On the harder 3→83 \to 8 split, every (M=5,α>0)(M = 5, \alpha \gt 0) run (−1.865 to −1.871 vs. −1.880) and (M=15,α=0.012)(M = 15, \alpha = 0.012) (−1.849, best overall; +0.024*, p=0.004p = 0.004 vs. Filtering and +0.032* vs. Smoother (5, 0)) achieves higher held-out causal likelihood than α=0\alpha = 0, while RUSTLeD (15, 0.012) also shrinks causal Δlog⁡Pθ\Delta\log P _ \theta vs. (15,0)(15, 0) (+0.071*, p<10−4p \lt 10^{-4}).
    • (3) On 5→65 \to 6, α>0\alpha \gt 0 preserves held-out causal likelihood while monotonically and significantly shrinking causal Δlog⁡Pθ\Delta\log P _ \theta relative to each matched-MM α=0\alpha = 0 baseline (+0.042*, p=0.001p = 0.001 and +0.045*, p=0.0004p = 0.0004 at M=5M = 5; +0.086* and +0.116*, p<10−4p \lt 10^{-4} at M=15M = 15). We will add this K=5K = 5 causal particle-filter evaluation in the revised paper.
AuthorsOpenReview ↗

Common Response (4/4)

Shared model family (Gemini) across labeler, simulator, and judge (Revs MQbA and fEt3)

Thank you for this important observation. We can use several controls, together with a new cross-family judge evaluation, to test whether our conclusions depend on a Gemini judge:

  • Judge-free deterministic metrics: Turn W1W_1, mean turns, TTR, and verbatim repetition use zero LLM calls and show the same gains (RUSTLeD at α=0.008\alpha = 0.008: W1=0.188W_1 = 0.188, Rep. =0.03= 0.03 vs. human 0.020.02 and SFT (No LS) 0.170.17).
  • Same-family negative control: Prompted (No LS) and Prompted (With LS) are also Gemini 2.5 Flash, yet the Gemini 2.5 Flash judge scores them worst of all methods (Hum. =3.62= 3.62 and 3.383.38 vs. real humans at 4.084.08; L1act=0.918L_1^{\text{act}} = 0.918 and 1.7851.785), ruling out naive self-family preference. On the training side, Gemini 2.5 Pro proposals are shared across Filtering, Smoother (α=0\alpha = 0), and RUSTLeD (α>0\alpha \gt 0) — and increasing α>0\alpha \gt 0 actively down-weights Gemini 2.5 Pro priors in favor of fitting the real human utterances UU.
  • Symmetric distributional calibration: L1actL_1^{\text{act}} and ∣ΔSat∣\vert\Delta\text{Sat}\vert apply fixed temperature-00 categorical rubrics identically to human and simulated conversations and measure distance ∣rm−rhum∣\vert r_m - r _ {\text{hum}}\vert rather than pairwise preference.
  • Independent cross-family GPT judge. To test whether our conclusions hold under a non-Gemini evaluator, we re-scored all human reference dialogues and simulator rollouts across both 5→65 \to 6 and 3→83 \to 8 using OpenAI GPT-5.4 Mini (Table C4). Across metrics, simulator rankings under GPT-5.4 Mini closely match those under Gemini 2.5 Flash (Spearman rank correlation ρ=+0.97/+0.93\rho = +0.97 / +0.93 on Sat.%, +0.84+0.84 on 5→65\to 6 L1act,t≥3L_1^{\text{act}, t\ge 3}, +0.81+0.81 on L1actL_1^{\text{act}}, and +0.68/+0.62+0.68 / +0.62 on Hum.):

Table C4 — Independent Cross-Family Evaluation with OpenAI GPT-5.4 Mini (n=256n = 256 per row)

Split & MethodL1act↓L_1^{\text{act}} \downarrowL1act,t≥3↓L_1^{\text{act}, t\ge 3} \downarrowSat.%∣ΔSat∣\vert\Delta\text{Sat}\vert pp ↓\downarrowViol.% ↓\downarrowHum. ↑\uparrow
5→65\to 6: Real Humans——20.7%—3.91%3.77
5→65\to 6: Prompted (No LS)1.3671.2503.1%17.60.39%3.71
5→65\to 6: Prompted (With LS)2.0942.0435.5%15.20.78%3.59
5→65\to 6: SFT (No LS)0.4100.37932.0%11.34.30%3.59
5→65\to 6: Filtering (M=15,α=0M=15, \alpha=0)0.4570.44923.4%2.75.47%3.72
5→65\to 6: Smoother (M=5,α=0M=5, \alpha=0)0.5900.48021.1%0.46.25%3.48
5→65\to 6: RUSTLeD (M=5,α=0.030M=5, \alpha=0.030)0.4770.32819.5%1.25.86%3.61
5→65\to 6: Smoother (M=15,α=0M=15, \alpha=0)0.4380.33217.6%3.15.08%3.66
5→65\to 6: RUSTLeD (M=15,α=0.008M=15, \alpha=0.008)0.4650.28522.7%2.04.30%3.65
5→65\to 6: RUSTLeD (M=15,α=0.012M=15, \alpha=0.012)0.7540.48014.8%5.92.34%3.77
5→65\to 6: RUSTLeD (M=15,α=0.020M=15, \alpha=0.020)0.5700.39517.6%3.11.95%3.76
3→83\to 8: Real Humans——18.4%—3.91%3.75
3→83\to 8: SFT (No LS)0.6800.65622.7%4.35.08%3.64
3→83\to 8: Smoother (M=5,α=0M=5, \alpha=0)0.7190.6487.0%11.31.95%3.51
3→83\to 8: RUSTLeD (M=5,α=0.012M=5, \alpha=0.012)0.4650.52717.2%1.23.12%3.74
3→83\to 8: Smoother (M=15,α=0M=15, \alpha=0)0.6210.39813.3%5.13.52%3.69
3→83\to 8: RUSTLeD (M=15,α=0.008M=15, \alpha=0.008)1.0120.6097.4%10.91.56%3.90
3→83\to 8: RUSTLeD (M=15,α=0.012M=15, \alpha=0.012)0.7770.6178.6%9.82.34%3.77
  • Baseline failure modes replicate independently of Gemini: Prompted baselines still severely under-satisfy, while SFT (No LS) still over-satisfies real humans (+11.3+11.3 pp on 5→65\to 6).
  • Consistent RUSTLeD gains on 5→65\to 6: RUSTLeD (M=15,α=0.012/0.020M=15, \alpha=0.012 / 0.020) ranks #1 and #2 among all 11 Table 1 models on Hum. (3.77 / 3.76, matching Real Humans at 3.77; +0.18*, p=0.002p = 0.002 vs. SFT No LS; +0.29*, p<10−4p \lt 10^{-4} vs. Smoother M=5,α=0M=5, \alpha=0). Meanwhile, α=0.008\alpha = 0.008 achieves the lowest L1act,t≥3L_1^{\text{act}, t\ge 3} (0.285), requirement violations (Viol.%) drop monotonically with α\alpha (5.08% → 1.95%), and the M=5M=5 reweighting (α=0.030\alpha=0.030) significantly improves Hum. (+0.13*, p=0.028p = 0.028).
  • Consistent gains on 3→83\to 8 & threshold note: M=5M=5 reweighting (α=0.012\alpha=0.012) significantly improves both Hum. (+0.23*, p=0.0004p = 0.0004) and ∣ΔSat∣\vert\Delta\text{Sat}\vert (11.3 → 1.2 pp, p=0.021p = 0.021), while (M=15,α=0.008)(M=15, \alpha=0.008) reaches the highest Hum. (3.90, +0.21*, p<10−4p \lt 10^{-4} vs. α=0\alpha=0). (Because GPT-5.4 Mini labels human Sat.% higher than Gemini 2.5 Flash [20.7% vs. 10.5% on 5→65\to 6], absolute ∣ΔSat∣\vert\Delta\text{Sat}\vert fold points shift even though simulator Sat.% rank order is preserved at ρ=+0.97\rho = +0.97.)

We will add a discussion of these points and the new results in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.