- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TRACER, a multi-turn user simulator that combines supervised fine-tuning with trajectory-aligned reinforcement learning. The framework models evolving user intent and introduces hierarchical rewards based on terminal outcomes, trajectory alignment, and output format. A deviation-aware advantage modulation mechanism further adjusts turn-level updates according to trajectory deviations. Experiments on real customer-service dialogues demonstrate substantial improvements in behavioral fidelity, conversion prediction, and semantic trajectory consistency. The authors also introduce DM-Bench to evaluate conversational agents based on both response quality and simulated conversion outcomes.
Critical strengths
- A well-motivated framework for multi-turn behavioral alignment.
The paper addresses an important limitation of existing user simulators, which often focus on individual responses rather than complete interaction trajectories. By combining terminal-outcome rewards with DTW-based trajectory alignment, TRACER explicitly models the evolution of user intent across multiple turns. The deviation-aware advantage modulation further provides a reasonable mechanism for adjusting turn-level optimization based on local and future trajectory deviations.
- Strong empirical results with comprehensive evaluations.
TRACER achieves substantial improvements over multiple general-purpose, role-playing, and training-based baselines. The evaluation covers outcome agreement, group-level behavioral fidelity, semantic trajectory consistency, and human-likeness. The ablation studies across two backbone models support the effectiveness of the proposed components. Furthermore, the out-of-domain evaluation and human role-play experiments provide additional evidence of the framework's generalization and practical utility.
Critical weaknesses
- Outcome-dependent reference-group construction limits the generality of the evaluation.
The reference groups are constructed not only based on similar user conditions but also through filtering and selection based on observed conversion outcomes. In particular, groups with tied conversion votes are excluded, and groups with higher outcome agreement are prioritized. This may underrepresent users with similar profiles but different behavioral decisions, which are particularly challenging for faithful user simulation. Consequently, the reported group-level improvements may not fully reflect performance on more heterogeneous real-world user populations. Additional evaluation on groups constructed without outcome-based selection would strengthen the conclusions.
- The independent contribution of advantage modulation requires stronger validation.
Although the ablation results support the overall framework, the improvements specifically attributable to deviation-aware advantage modulation are relatively modest compared with the gains from reinforcement learning as a whole. For example, removing advantage modulation reduces F1 from 79.4 to 78.8 for TRACER-7B. Moreover, the reported standard deviations are based on repeated evaluation of fixed checkpoints rather than independent training runs. Additional experiments across multiple training seeds and comparisons with simpler trajectory-level weighting strategies would help establish the robustness and necessity of the proposed credit assignment mechanism.
Questions for authors
-
Would the main group-level results remain consistent if reference groups were constructed without using conversion outcomes, including groups with mixed or tied outcomes?
-
Are the improvements from deviation-aware advantage modulation consistent across independently trained checkpoints? How does the proposed method compare with simpler trajectory-level advantage weighting strategies?
-
Since the trajectory rewards depend on model-generated intent scores, could the authors further evaluate whether these scores accurately reflect the expressed user behavior, especially for intermediate intent levels?
2 comments
Follow-up on Our Responses
Dear Reviewer sevW,
As the discussion period draws to a close, we would appreciate any feedback you may have on our responses and the revised manuscript. It would be helpful to know whether our clarifications and additional experiments have addressed your concerns, or whether any questions remain.
We understand that your time is limited and would be grateful for even a brief update. We would be happy to provide any further clarification.
Thank you again for your time and constructive feedback.
Best regards,
The Authors
Re-evaluated on the complete candidate pool (W1/Q1); replicated modulation gains under a new training seed with trajectory-weighting control (W2/Q2)
Thank you for recognizing our motivation, framework, and evaluation. We provide further clarification for each of your three concerns below.
Thank you for your suggestion regarding test-set coverage. We first clarify that the original 1:1 balance was imposed only at the group-selection stage, based on group majority labels. All sessions within each selected group were retained, including minority-outcome ones, which contribute to PCR and are evaluated against their own ground-truth labels in ACC/F1. Group-level balancing therefore neither required similar users to make identical decisions nor prescribed a 50% session-level conversion rate.
To examine the effect of group selection, we included all 53 remaining eligible groups from the candidate pool without further selection based on conversion labels, outcome consistency, or model performance. These comprise 17 previously excluded tied groups and 36 positive-majority groups omitted to achieve the original 1:1 balance. The test set therefore expands from 762 groups and 3,866 sessions to 815 groups and 4,427 sessions, with an observed conversion rate of 53.22%. PCR and ACC/F1 are session-level; Group-ΔPCR compares simulated and observed conversion proportions within each group and averages the absolute differences equally across all 815 groups.
TRACER retains its main comparative advantages on this complete candidate pool. Both TRACER-7B and TRACER-4B lead on F1 (78.53 and 79.63, exceeding the strongest non-TRACER baseline by 13+ points) as well as ACC, Group-ΔPCR, and OT-DTW, while predicting conversion rates close to the observed 53.22%.
The table below reports all 11 baselines and both TRACER sizes; these expanded-test-set results will replace Table 1 in the revised manuscript.
These results support the robustness of TRACER’s main comparative advantages to removing the original group-balancing and tie-exclusion criteria.
Thank you for suggesting that we distinguish training from inference randomness and include a simpler weighting control. Beyond the original 7B run (seed 42), we conducted an independent run with seed 43, evaluating each checkpoint with five inference seeds.
Modulation yields consistent gains across two independent training seeds; the trajectory-level weighting control overpredicts conversion by over 11 points and worsens group- and trajectory-level calibration.
(1) Independent training replication. On F1, Full outperforms Uniform by +0.60 under seed 42 (79.4 vs. 78.8) and by +0.72 under seed 43 (78.20 vs. 77.48). The seed-43 Full checkpoint also improves ACC by 0.41 and reduces OT-DTW from 0.374 to 0.367 relative to Uniform. The two runs use the original and expanded test sets, respectively.
(2) Comparison with trajectory-level weighting. We compare Full, Uniform and Trajectory weighting, the last averaging per-turn weights within each trajectory. All three share the same rewards and multi-turn RL.
Means ± std over five inference seeds on 4,427 sessions / 815 groups; observed conversion rate 53.22%.
Trajectory weighting's marginally higher F1 (78.57 vs. 78.20) comes with substantial over-prediction (PCR 64.60% vs. 53.22%), lower ACC (74.75 vs. 76.82), and much larger Group-ΔPCR (0.217 vs. 0.179) and OT-DTW (0.487 vs. 0.367). Uniform trajectory-level scaling does not reproduce the benefits of allocating updates across turns.