acceptodds
Under review as a conference paper at ICLR 2027

TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulation

Abstract

Faithful user simulation is fundamental to building, evaluating, and improving interactive AI at scale. Yet current simulators often produce plausible individual responses without reproducing the intent evolution and outcomes observed in real interactions. We propose TRACER, a multi-turn user simulator that models evolving user intent and aligns simulated trajectories with real ones. TRACER is trained in two stages: supervised fine-tuning on real user dialogues, followed by multi-turn reinforcement learning. The RL stage combines hierarchical outcome- and trajectory-level rewards with deviation-aware advantage modulation, jointly addressing reward sparsity and credit assignment challenges in long dialogues. On real customer-service sessions organized into reference cohorts, TRACER-7B surpasses the strongest baseline by 11.4 conversion F1 points, while outperforming all baselines on group-level conversion-rate error and semantic trajectory distance and generalizing to out-of-distribution scenarios. In human Turing tests, annotators identified TRACER conversations at near-chance accuracy. Building on this simulator, we further introduce the Dynamic Marketing Benchmark, which jointly evaluates persuasion and response quality via simulated interactions, revealing that higher response quality does not necessarily correspond to higher conversion rates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, User Simulation, Multi-turn Dialogue, Behavioral Alignment, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sevWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRACER, a multi-turn user simulator that combines supervised fine-tuning with trajectory-aligned reinforcement learning. The framework models evolving user intent and introduces hierarchical rewards based on terminal outcomes, trajectory alignment, and output format. A deviation-aware advantage modulation mechanism further adjusts turn-level updates according to trajectory deviations. Experiments on real customer-service dialogues demonstrate substantial improvements in behavioral fidelity, conversion prediction, and semantic trajectory consistency. The authors also introduce DM-Bench to evaluate conversational agents based on both response quality and simulated conversion outcomes.

Critical strengths
  1. A well-motivated framework for multi-turn behavioral alignment.

The paper addresses an important limitation of existing user simulators, which often focus on individual responses rather than complete interaction trajectories. By combining terminal-outcome rewards with DTW-based trajectory alignment, TRACER explicitly models the evolution of user intent across multiple turns. The deviation-aware advantage modulation further provides a reasonable mechanism for adjusting turn-level optimization based on local and future trajectory deviations.

  1. Strong empirical results with comprehensive evaluations.

TRACER achieves substantial improvements over multiple general-purpose, role-playing, and training-based baselines. The evaluation covers outcome agreement, group-level behavioral fidelity, semantic trajectory consistency, and human-likeness. The ablation studies across two backbone models support the effectiveness of the proposed components. Furthermore, the out-of-domain evaluation and human role-play experiments provide additional evidence of the framework's generalization and practical utility.

Critical weaknesses
  1. Outcome-dependent reference-group construction limits the generality of the evaluation.

The reference groups are constructed not only based on similar user conditions but also through filtering and selection based on observed conversion outcomes. In particular, groups with tied conversion votes are excluded, and groups with higher outcome agreement are prioritized. This may underrepresent users with similar profiles but different behavioral decisions, which are particularly challenging for faithful user simulation. Consequently, the reported group-level improvements may not fully reflect performance on more heterogeneous real-world user populations. Additional evaluation on groups constructed without outcome-based selection would strengthen the conclusions.

  1. The independent contribution of advantage modulation requires stronger validation.

Although the ablation results support the overall framework, the improvements specifically attributable to deviation-aware advantage modulation are relatively modest compared with the gains from reinforcement learning as a whole. For example, removing advantage modulation reduces F1 from 79.4 to 78.8 for TRACER-7B. Moreover, the reported standard deviations are based on repeated evaluation of fixed checkpoints rather than independent training runs. Additional experiments across multiple training seeds and comparisons with simpler trajectory-level weighting strategies would help establish the robustness and necessity of the proposed credit assignment mechanism.

Questions for authors
  1. Would the main group-level results remain consistent if reference groups were constructed without using conversion outcomes, including groups with mixed or tied outcomes?

  2. Are the improvements from deviation-aware advantage modulation consistent across independently trained checkpoints? How does the proposed method compare with simpler trajectory-level advantage weighting strategies?

  3. Since the trajectory rewards depend on model-generated intent scores, could the authors further evaluate whether these scores accurately reflect the expressed user behavior, especially for intermediate intent levels?

2 comments
AuthorsOpenReview ↗

Re-evaluated on the complete candidate pool (W1/Q1); replicated modulation gains under a new training seed with trajectory-weighting control (W2/Q2)

Thank you for recognizing our motivation, framework, and evaluation. We provide further clarification for each of your three concerns below.

W1 & Q1: Do the main findings hold without selecting reference groups based on conversion outcomes?

Thank you for your suggestion regarding test-set coverage. We first clarify that the original 1:1 balance was imposed only at the group-selection stage, based on group majority labels. All sessions within each selected group were retained, including minority-outcome ones, which contribute to PCR and are evaluated against their own ground-truth labels in ACC/F1. Group-level balancing therefore neither required similar users to make identical decisions nor prescribed a 50% session-level conversion rate.

To examine the effect of group selection, we included all 53 remaining eligible groups from the candidate pool without further selection based on conversion labels, outcome consistency, or model performance. These comprise 17 previously excluded tied groups and 36 positive-majority groups omitted to achieve the original 1:1 balance. The test set therefore expands from 762 groups and 3,866 sessions to 815 groups and 4,427 sessions, with an observed conversion rate of 53.22%. PCR and ACC/F1 are session-level; Group-ΔPCR compares simulated and observed conversion proportions within each group and averages the absolute differences equally across all 815 groups.

TRACER retains its main comparative advantages on this complete candidate pool. Both TRACER-7B and TRACER-4B lead on F1 (78.53 and 79.63, exceeding the strongest non-TRACER baseline by 13+ points) as well as ACC, Group-ΔPCR, and OT-DTW, while predicting conversion rates close to the observed 53.22%.

The table below reports all 11 baselines and both TRACER sizes; these expanded-test-set results will replace Table 1 in the revised manuscript.

MethodPCRACC↑F1 ↑Group-ΔPCR ↓W₁-Turns ↓OT-DTW ↓
Gemini-3.1-Pro42.0855.6153.420.4101.6260.480
Claude-4.5-Sonnet6.5549.0414.740.4802.3700.481
Deepseek-V3.232.9854.1546.800.4272.7800.486
Qwen2.5-7B-Instruct33.8251.5944.380.4575.4440.501
Qwen3-4B-Instruct29.8855.5246.480.4265.4410.506
HumanLM-8B12.8555.0331.930.4010.9470.442
Qwen-Character10.2851.5523.690.4578.5990.522
CharGLM-416.2450.0128.030.4697.4960.529
Doubao-Seed-Character11.6849.7922.620.4738.7230.534
Doubao-Seed-2.0-Pro72.3556.1165.050.4272.9240.469
TRACER-7B52.9377.2178.530.1690.9850.359
TRACER-4B55.7777.8079.630.1681.3920.368

These results support the robustness of TRACER’s main comparative advantages to removing the original group-balancing and tie-exclusion criteria.

W2 & Q2: Are the modest gains from advantage modulation consistent across training seeds, and how does it compare with simpler trajectory-level weighting?

Thank you for suggesting that we distinguish training from inference randomness and include a simpler weighting control. Beyond the original 7B run (seed 42), we conducted an independent run with seed 43, evaluating each checkpoint with five inference seeds.

Modulation yields consistent gains across two independent training seeds; the trajectory-level weighting control overpredicts conversion by over 11 points and worsens group- and trajectory-level calibration.

(1) Independent training replication. On F1, Full outperforms Uniform by +0.60 under seed 42 (79.4 vs. 78.8) and by +0.72 under seed 43 (78.20 vs. 77.48). The seed-43 Full checkpoint also improves ACC by 0.41 and reduces OT-DTW from 0.374 to 0.367 relative to Uniform. The two runs use the original and expanded test sets, respectively.

(2) Comparison with trajectory-level weighting. We compare Full, Uniform and Trajectory weighting, the last averaging per-turn weights within each trajectory. All three share the same rewards and multi-turn RL.

MethodPCRACC ↑F1↑Group-ΔPCR ↓W₁-Turns ↓OT-DTW ↓
Full53.10 ± 0.2576.82 ± 0.0578.20 ± 0.090.179 ± 0.0021.109 ± 0.0140.367 ± 0.001
Uniform51.52 ± 0.0876.41 ± 0.2977.48 ± 0.280.176 ± 0.0021.087 ± 0.0090.374 ± 0.001
Trajectory weighting64.60 ± 0.2974.75 ± 0.2778.57 ± 0.250.217 ± 0.0031.394 ± 0.0160.487 ± 0.002

Means ± std over five inference seeds on 4,427 sessions / 815 groups; observed conversion rate 53.22%.

Trajectory weighting's marginally higher F1 (78.57 vs. 78.20) comes with substantial over-prediction (PCR 64.60% vs. 53.22%), lower ACC (74.75 vs. 76.82), and much larger Group-ΔPCR (0.217 vs. 0.179) and OT-DTW (0.487 vs. 0.367). Uniform trajectory-level scaling does not reproduce the benefits of allocating updates across turns.

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewer sevW,

As the discussion period draws to a close, we would appreciate any feedback you may have on our responses and the revised manuscript. It would be helpful to know whether our clarifications and additional experiments have addressed your concerns, or whether any questions remain.

We understand that your time is limited and would be grateful for even a brief update. We would be happy to provide any further clarification.

Thank you again for your time and constructive feedback.

Best regards,

The Authors

Reviewer_GxsdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To achieve high-fidelity evaluation and training of dialogue agent, the paper develops a multi-turn interactive user simulator that addresses the behavioral instability and inconsistent decision-making in conversation imitation. The simulator “TRACE” is developed by two-stage training (The first fine-tunes on human annotation data; The second uses GRPO) to aligning to human behavior. The paper emphasizes the advantage design in the second state RL which include the outcome reward, trajectory reward and the format reward, and a mechanism to calculate the behavior trajectory deviation. Additionally, the paper provides the corresponding marketing dialogue dataset and evaluation method. The jointly evaluates persuasion and response quality. The user simulator achieves superior comparing to pure proprietary LLM.

Critical strengths

The paper provides a simulator training framework that can endow the model with behavioral stability and consistent decision-making in conversation imitation.

The user simulator comes with a relevant evaluation dataset and benchmark, forming a complete evaluation environment for plug-and-play use in dialogue agent evaluation and fine-grained RL training.

The experimental evaluation is relatively comprehensive. The trained user simulator not only achieves a high score but also can be generalized to other domains, indicating the training framework is effective and well-designed.

Critical weaknesses

Neither the addressing problem nor the used method is rather novel. Behavior consistency of the simulator is widely researched and addressed [e.g. argus, tau-bench]. The paper does not show its merit in comparison with these previous works in either the description or the experiments.

Two fundamental limitations are not independent, and the paper seems not to address them directly. They are more like the same limitation from two aspects. The first limitation is sound: Alignment is limited to linguistic style, not decision-making. However, the second one, addressing the training method of the user simulator, could not be stated as a limitation; instead, it is some kind of specific methodology that solves the first limitation.

The comparison with existing training-based user simulators is limited. Although the paper discusses several recent user simulation approaches in the related work, including methods involving reasoning and reinforcement learning, the experiments only include HUMANLM as a directly trained baseline. For intent alignment, UserLMs is a suitable comparison. In addition, the experiments primarily focus on customer-service scenarios in which conversion is defined as providing contact information. Such a setting offers relatively explicit terminal states and a structured intent progression, but many real-world interactions involve ambiguous, changing, or competing objectives.

Questions for authors
  1. How can the authors verify that the annotated intent trajectories accurately reflect real user decision-making? Actually, I was not fully convinced by the authors’ method that can use such intent trajectories.

  2. What is the essential contribution of deviation-aware advantage modulation compared with standard multi-turn RL? Which module causes the effects?

  3. Could the authors strengthen the comparisons with existing training-based user simulators?

5 comments
AuthorsOpenReview ↗

Clarified novelty vs. ARGUS/τ-bench (W1); consolidated limitations (W2); added UserLM baseline and scoped the setting (W3)

Thank you for recognizing the value of our training framework, evaluation setting, and out-of-domain experiments. We address your concerns below.

W1: The novelty over existing research on behavioral consistency is unclear.

Thank you for encouraging us to clarify our contribution. TRACER uses observed human behavior as the alignment reference, jointly constrains terminal decisions and intermediate intent evolution, and uses trajectory deviations to modulate turn-level updates.

The first arXiv version of ARGUS was submitted on October 15, after our submission on September 25; we therefore discuss it as concurrent work (ARGUS). Although τ-bench, ARGUS, and TRACER all involve multi-turn interactions, their alignment references and evaluation targets differ:

WorkPrimary referenceAlignment objectiveEvaluation target
τ-benchSpecified tasks, domain policies, expected terminal statesCorrect, reliable task completionService agents; success rate, pass^k
ARGUSUser goals, profiles, constraints, agenda statesGoal progression, factual consistency, constraint adherenceUser simulators; CBS, success rate
TRACERObserved terminal behavior and annotated intent trajectoriesAgreement with observed human decisions and evolutionUser simulators; ACC/F1, Group-ΔPCR, OT-DTW

τ-bench targets service-agent reliability and ARGUS targets adherence to specified goals and constraints, whereas TRACER aligns simulated behavior with real interaction records — in which conversion, refusal, and dropout are all behaviors to be modeled — without requiring a predefined agenda. τ-bench, Sections 3–4; ARGUS, Sections 2–3. Component ablations in the original submission examine the contributions of terminal rewards, DTW-based trajectory rewards, and deviation-aware modulation.

W2: The two limitations are not independent.

Thank you for pointing out their connection. We intended to distinguish the behavioral alignment objective from the interaction scope covered by optimization: the first concerns whether simulated decisions and intent evolution match real users; the second concerns whether optimization covers complete interactions rather than individual responses. TRACER addresses both jointly via terminal and trajectory rewards. We agree that presenting them as independent "fundamental limitations" obscures their relationship, and will consolidate them into two aspects of one core limitation in the introduction, related work, and method motivation.

W3-1 & Q3: Comparisons with training-based user simulators are limited; UserLM should be included.

Thank you for suggesting UserLM. We trained a Qwen3-8B user simulator on our data following UserLM's training approach and evaluated it on the expanded test set (data construction, SFT configuration, and evaluation details will be included in the revision). The observed conversion rate is 53.22% (2,356/4,427), providing the reference for predicted PCR.

MethodPCRACC↑F1↑Group-ΔPCR ↓W1-Turns ↓OT-DTW ↓
HumanLM-8B12.8555.0331.930.4010.9470.442
UserLM-8B15.8155.1835.080.4131.5470.372
TRACER-7B52.9377.2178.530.1690.9850.359
TRACER-4B55.7777.8079.630.1681.3920.368

Although UserLM-8B approaches TRACER-7B in OT-DTW substantial gaps remain in F1 and Group-ΔPCR. SFT following UserLM's approach can produce relatively similar semantic trajectories, but this similarity alone does not ensure agreement in terminal decisions or group-level behavioral distributions, supporting our motivation to align behavioral choices alongside response semantics.

W3-2: How does a setting with explicit contact-sharing outcomes address ambiguous, changing, or competing goals?

Thank you for highlighting these settings. We clarify our scope and the rationale for task selection.

(1) We study task-oriented user simulation with observable behavioral references. Identifiable terminal outcomes allow comparison of simulated and real decisions, while dialogue content evidences intermediate intent trajectories. Contact sharing and issue resolution thus provide operational supervision and evaluation criteria for behavioral consistency.

(2) The original submission demonstrates transfer within this scope. On real dialogues from CSC-Conv, a financial customer-service dataset unseen during training, TRACER extends the terminal criterion from marketing conversion to issue resolution, exceeding the strongest baseline by 8.4, 4.8, and 0.7 points in terminal-state ACC, F1, and dialogue semantic similarity.

(3) Open-ended interactions remain an extension. The "Generalization to Open-Ended Social Dialogues" paragraph in our Limitation section identifies settings without clear objectives or success criteria as requiring further study.

AuthorsOpenReview ↗

Validation evidence for intent trajectories (Q1); independent role of deviation-aware modulation with new ablations (Q2)

Q1: How can the annotated intent trajectories be validated against real user decision-making?

Thank you for raising this foundational question. We clarify the role of intent annotations and the supporting evidence below.

(1) Intent trajectories are behavioral proxies grounded in observable dialogue. User replies and terminal behavior come from real conversations; rationales and intent levels provide structured supervision derived from these observations. We use these proxies to constrain simulated behavior without assuming that users' internal reasoning can be recovered.

(2) Human re-annotation directly evaluates the intent annotations (Appendix B.3, Table 4). We sampled 300 dialogues stratified by intent, dialogue length, and user segment. Three trained annotators independently assigned turn-level intent scores on the 0–5 scale using dialogue history and user profiles, without access to our annotation pipeline's outputs. Table 4 reports inter-annotator agreement of Fleiss' κ = 0.77, annotation–human-majority agreement of Cohen's κ = 0.71, and raw agreement of 87.0%, both values falling in the substantial-to-good range; the raw agreement indicates that residual disagreement is concentrated on adjacent levels rather than gross mislabeling. Separately, 92% of the annotated rationales received plausibility ratings of at least 4 on a five-point scale from the same annotators. Because both annotators and our pipeline operate on the same observable dialogue, this agreement establishes the annotations as a well-defined, human-reproducible construct grounded in observable behavior — which is precisely the property required of a behavioral proxy.

(3) Two human studies evaluate the trained simulator at complementary granularities (Section 6.1; Appendices G.2–G.3).

For local intent changes, three annotators labeled 449 strategy instances from 200 conversations covering six strategy types, with pairwise agreement exceeding 90%. Against majority-vote labels, TRACER achieves class-conditional recalls of 88.7% (decrease), 74.2% (unchanged), and 81.0% (increase) against majority-vote labels. These are class-conditional recalls measuring agreement with human expectations during the interaction, rather than only at its endpoint.

For aggregate behavioral feedback, seven assistants were evaluated on the same 100 contexts, yielding 700 human role-play conversations. Human and TRACER conversion-rate rankings agree strongly: Spearman’s ρ = 0.955 and Kendall’s τb = 0.878 (Table 9; Appendix G.3).

Together, these studies support the behavioral plausibility of the trained simulator at both local and aggregate levels, providing complementary indirect evidence for our supervision design.

Q2: What is the independent contribution of deviation-aware advantage modulation over standard multi-turn RL?

Thank you for asking about its independent role. On the 4B model, we compare removing modulation (Uniform), applying trajectory-level weighting (Trajectory weighting), and shuffling turn weights (Shuffled modulation) to isolate modulation itself and its turn-specific allocation.

MethodPCRACC ↑F1 ↑Group-ΔPCR ↓W₁-Turns ↓OT-DTW ↓
Full55.77 ± 0.0877.80 ± 0.1279.63 ± 0.110.172 ± 0.0031.389 ± 0.0080.369 ± 0.001
Uniform56.19 ± 0.3076.76 ± 0.2678.75 ± 0.240.183 ± 0.0031.138 ± 0.0110.375 ± 0.001
Trajectory weighting64.18 ± 0.2575.64 ± 0.1879.25 ± 0.150.209 ± 0.0041.178 ± 0.0130.374 ± 0.001
Shuffled modulation63.96 ± 0.1576.11 ± 0.1479.61 ± 0.130.204 ± 0.0021.200 ± 0.0210.390 ± 0.002

Means ± std across five evaluation seeds on the expanded test set (815 groups, 4,427 sessions) introduced in response to other reviewers' suggestion; observed PCR is 53.22%. Uniform keeps the same rewards and multi-turn RL; Trajectory weighting averages weights within each trajectory; Shuffled modulation permutes their correspondence with turns.

The independent contribution is to allocate update magnitudes according to turn-level deviations. Relative to Uniform, Full improves ACC/F1 by 1.04/0.88 points and reduces Group-ΔPCR (0.172 vs. 0.183) and OT-DTW (0.369 vs. 0.375). Trajectory-level weighting overpredicts conversion (64.18% vs. observed 53.22%), and shuffling turn weights worsens ACC, Group-ΔPCR, and OT-DTW — both supporting the value of turn-specific allocation. The same ordering holds on the 7B model (Full vs. Uniform: ACC 79.0 vs. 78.7; F1 79.4 vs. 78.8), reported in Section 5.3.2.

AuthorsOpenReview ↗

Ethics: data source, basis for use, and privacy protections clarified

Ethics: The source and protection of private dialogues and user profiles require clarification.

Thank you for raising these concerns. We clarify the data source, basis for use, and protection measures below.

(1) Data source and basis for use. The data were collected from a mobile application in China that provides merchants with AI-powered customer-service functionality, and consist of multi-turn conversations between real users and merchant-side AI customer-service agents. Collection and processing followed review by the internal legal and data-compliance teams. According to their review, the research use falls within the platform’s user agreement and privacy policy and complies with applicable local laws and internal privacy policies. Users’ general authorization at registration for the relevant data-use purposes forms the basis for this use. To preserve double-blind anonymity, we withhold identifying platform and organization names; verifiable policy references and the scope of use will be provided in the de-anonymized version.

(2) Profile processing and de-identification. Profiles include age range, gender, education, marital and parental status, occupation, device, region, and economic circumstances. Group matching constructs coarse categories from the first two sentences of the basic-profile field and normalizes numbers, locations, devices, and nicknames. Supplementary-data export replaces session/group IDs with pseudonymous codes and replaces contact details, emails, URLs, and selected identity-, address-, or organization-related expressions. Exported materials passed the configured automated checks. Contact details in simulated conversations are model-generated conversion indicators, not reported real-user contact information.

(3) Human review and redaction. Five professionals with master’s degrees in law and several years of legal practice reviewed the profiles, customer-service dialogues, and other structured text fields record by record under applicable local laws and company privacy-review guidelines. Flagged sensitive passages were replaced following multiple rounds of review and consultation with the legal team. External release remains subject to the required anonymization, privacy-risk assessment, and approval procedures.

The revised Ethics Statement and appendix will document these measures, including profile categorization, de-identification, and supplementary-data export procedures.

Reviewer_SuMfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing user simulators often struggle to reproduce real users’ evolving intentions and final decisions in multi-turn interactions. To address this limitation, the authors propose TRACER, a multi-turn user simulation framework. Specifically, TRACER combines supervised fine-tuning on real dialogues with multi-turn reinforcement learning, using terminal-intent consistency rewards, dynamic time warping-based trajectory rewards, and deviation-aware turn-level advantage modulation to align behavior throughout an interaction. Experiments use 8,040 training sessions and 3,866 test sessions organized into 762 reference groups. Validation includes baseline comparisons, ablation studies, human discrimination tests, and out-of-domain evaluation. TRACER-7B improves conversion F1 by 11.4 points over the strongest baseline, achieves better group-level behavioral and semantic trajectory alignment, and yields a human identification accuracy of 49.5%, close to chance. The main contributions are a trajectory-aligned training method, a multi-reference group evaluation protocol, and the DM-Bench. The results support the value of full-trajectory optimization for user simulation and show that higher response quality does not necessarily correspond to higher conversion rates within the simulated environment.

Critical strengths
  1. Research Background: This paper clearly identifies the gap between linguistic naturalness and behavioral fidelity in existing user simulators, addressing a well-defined problem with practical value for training and evaluating multi-turn dialogue systems.
  2. Method: TRACER uses DTW-based trajectory alignment for both session-level rewards and turn-level advantage modulation, providing a coherent framework that accounts for final decisions and intermediate intent evolution.
  3. Experiments: Comparisons with multiple categories of baselines, out-of-domain testing, and human evaluation, together with ablations of the main components, provide complementary evidence supporting the method’s effectiveness.
Critical weaknesses
  1. The effect of sign-preserving advantage modulation on local errors remains insufficiently validated (Lines 251–259, Eq. 7). The mechanism preserves the sign of the trajectory-level advantage, so misaligned segments in positive-advantage trajectories still receive positive signals, while well-aligned segments in negative-advantage trajectories cannot be rewarded independently. Targeted experiments on local errors would help assess this limitation.
  2. Outcome-based test-set selection may limit the evaluation of behavioral diversity (Appendix B.4, Lines 950–959). The authors exclude groups with tied outcome votes and prioritize groups with higher outcome agreement, underrepresenting users with divergent decisions under similar conditions. An additional test set selected independently of conversion labels would strengthen the evaluation.

If these concerns are adequately addressed, I would consider raising my score.

Questions for authors

I have no specific questions.

4 comments
AuthorsOpenReview ↗

Added single-step diagnostics and full-training comparison (W1)

Thank you for your constructive feedback. To respond to your concerns, we conducted additional local-update diagnostics and expanded the test-set evaluation, which we hope will help clarify the effects of advantage modulation and test-set selection.

W1: The effect of sign-preserving advantage modulation on locally erroneous turns is not sufficiently verified.

Single-step update diagnostics on fixed turns. We construct an anomaly-enriched diagnostic set of 36 positive-advantage erroneous turns and 40 negative-advantage correct turns from historical 7B Full/Uniform rollouts. Correctness is assessed by a single model evaluator using the user profile, needs, and visible dialogue context; reasonable refusal, clarification, and exit are considered valid behavior. This set probes local update behavior rather than natural error prevalence.

All four methods start from the same 7B SFT checkpoint, using the same historical trajectories, trajectory-level advantages, optimizer, and training configuration. Only the advantage-shaping rule differs, and each method performs one parameter update.

We measure whether the update increases erroneous replies’ mean token log-likelihood (LL; 36 turns) or decreases that of correct replies (40 turns). We also count confirmed errors in 72 freely generated replies, sampled twice per erroneous-turn prefix using paired random seeds and evaluated with method identities hidden.

IndicatorUniformFullShuffledSigned-local
Erroneous-reply LL increase (/36)25172525
Correct-reply LL decrease (/40)16152221
Confirmed errors in free generation (/72)3123

Full reduces unintended error reinforcement (25→17 vs. Uniform) while keeping correct-turn suppression comparable (15 vs. 16), suggesting that modulation selectively dampens erroneous turns without additionally suppressing correct ones. Free-generation confirmed errors are also fewer, though event counts are low. The measurement is the change in reply likelihood after one update, not an advantage-sign flip rate.

(2) Behavioral alignment after full RL training. Beyond the per-turn diagnostics in (1), we verify that the design holds after complete RL training. We compare Full, Signed-local, and Shuffled modulation on 4B models, each evaluated on all 4,427 sessions with five inference seeds. Signed-local uses A~i,t=Ai+Ai,tlocal\widetilde A_{i,t}=A_i+A^{\mathrm{local}}_{i,t}, where the local term standardizes negative cumulative deviations within each trajectory (coefficient 1, future-deviation discount 0.3). Shuffled permutes the weight–turn correspondence within each trajectory.

MethodACC ↑F1 ↑Group-ΔPCR ↓W₁-Turns ↓OT-DTW ↓
Full77.80 ± 0.1279.63 ± 0.110.172 ± 0.0031.389 ± 0.0080.369 ± 0.001
Signed-local76.92 ± 0.2578.98 ± 0.230.177 ± 0.0061.423 ± 0.0270.425 ± 0.002
Shuffled76.11 ± 0.1479.61 ± 0.130.204 ± 0.0021.200 ± 0.0210.390 ± 0.002

Means ± std over five inference seeds under a fixed training seed. Signed-local is not better than Full on any metric; OT-DTW rises from 0.369 to 0.425, indicating that an independent local term weakens rather than strengthens semantic trajectory alignment. Full also beats Shuffled on ACC, Group-ΔPCR, and OT-DTW (with comparable F1; W₁-Turns reflects session-length distribution rather than within-trajectory consistency). Together with (1), these results support preserving trajectory-level advantage signs while allocating per-turn magnitudes by deviations, rather than splitting off an independent local signal.

AuthorsOpenReview ↗

Re-evaluated on the complete candidate pool (W2)

W2: Outcome-based test-set selection may limit the evaluation of behavioral diversity.

Thank you for your suggestion regarding test-set coverage. We first clarify that the original 1:1 balance was imposed only at the group-selection stage, based on group majority labels. All sessions within each selected group were retained and evaluated against their own ground-truth labels, so minority-outcome sessions were never excluded from PCR or ACC/F1; the session-level conversion rate was 51.24%, not a prescribed 50%.

To directly examine the effect of group selection, we included all 53 remaining eligible groups from the candidate pool without additional filtering — 17 previously excluded tied groups and 36 positive-majority groups omitted to achieve the original 1:1 balance. This expands the test set from 762 groups / 3,866 sessions to 815 groups / 4,427 sessions, with an observed conversion rate of 53.22%. PCR is computed over all simulated sessions; ACC/F1 use each session's own ground-truth label; Group-ΔPCR is the absolute difference between simulated and observed conversion rates within each group, averaged equally across all 815 groups.

TRACER retains its main comparative advantages on this complete candidate pool. TRACER-7B achieves an F1 of 78.53, exceeding the strongest non-TRACER baseline (Doubao-Seed-2.0-Pro) by 13.48 points; its PCR of 52.93% is close to the observed rate (53.22%), and its Group-ΔPCR (0.169) and OT-DTW (0.359) are the lowest among all non-TRACER baselines. TRACER-4B achieves an F1 of 79.63. The table below reports all 11 baselines and both TRACER sizes; Table 1 in the revised manuscript will be replaced accordingly.

MethodPCR (%)ACC (%) ↑F1 (%) ↑Group-ΔPCR ↓W1-Turns ↓OT-DTW ↓
Gemini-3.1-Pro42.0855.6153.420.4101.6260.480
Claude-4.5-Sonnet6.5549.0414.740.4802.3700.481
Deepseek-V3.232.9854.1546.800.4272.7800.486
Qwen2.5-7B-Instruct33.8251.5944.380.4575.4440.501
Qwen3-4B-Instruct29.8855.5246.480.4265.4410.506
HumanLM-8B12.8555.0331.930.4010.9470.442
UserLM-8B15.8155.1835.080.4131.5470.372
Qwen-Character10.2851.5523.690.4578.5990.522
CharGLM-416.2450.0128.030.4697.4960.529
Doubao-Seed-Character11.6849.7922.620.4738.7230.534
Doubao-Seed-2.0-Pro72.3556.1165.050.4272.9240.469
TRACER-7B52.9377.2178.530.1690.9850.359
TRACER-4B55.7777.8079.630.1681.3920.368

These results support the robustness of TRACER’s main comparative advantages to removing the original group-balancing and tie-exclusion criteria.

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewer SuMf,

As the discussion period draws to a close, we would appreciate any feedback you may have on our responses and the revised manuscript. It would be helpful to know whether our clarifications and additional experiments have addressed your concerns, or whether any questions remain.

We understand that your time is limited and would be grateful for even a brief update. We would be happy to provide any further clarification.

Thank you again for your time and constructive feedback.

Best regards, The Authors

Thank you for the thoughtful response and additional experiments. The single-step diagnostics provide evidence that sign-preserving modulation reduces the risk of reinforcing locally erroneous replies. The small diagnostic sample still warrants some caution. Evaluating all 815 eligible groups, including those with tied outcomes, directly addresses my concern about test-set selection. It is reassuring that TRACER retains its main gains on this expanded set.

Overall, the response has largely addressed my concerns, and I have no further questions. I continue to view 3 as the appropriate rating for this paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of manuscript revisions and additional evidence

We sincerely thank reviewers for their constructive comments and valuable suggestions. Their feedback has helped us clarify the scope of TRACER, strengthen the empirical evaluation, and improve the presentation of our supervision and data construction procedures.

Below, we summarize the main manuscript revisions and provide a reviewer-specific guide to these changes. Revisions are marked in blue. Our detailed responses under each review provide the supporting experimental results and explanations.

1. Evaluation on the complete eligible test pool.
We have expanded the evaluation from 762 groups and 3,866 sessions to all 815 eligible groups and 4,427 sessions, restoring the 17 tied groups and 36 positive-majority groups previously excluded during group selection. The observed conversion rate is 53.22%. Both TRACER model sizes retain their advantages over the evaluated baselines in ACC, F1, Group-ΔPCR, and OT-DTW, with a conversion F1 improvement of 13.48 and 14.58 points for TRACER-7B and TRACER-4B over the strongest non-TRACER baseline. The revised manuscript reports the expanded evaluation in Table 1 and Sections 5.1–5.2, with the construction and selection procedures clarified in Appendix B.4.

2. Additional evidence on advantage modulation.
We have extended the ablation analysis with an additional 7B training run using seed 43 and 4B controls comparing Full with Uniform, Trajectory weighting, Shuffled modulation, and Signed-local. On the expanded test set of 815 groups and 4,427 sessions, Full improves mean F1 over Uniform by 1.20 percentage points under training seed 42 and by 0.72 points under seed 43, showing gains in the same direction across both runs. We report the training runs separately, with standard deviations computed over five inference seeds per fixed checkpoint. Appendix C also includes single-step diagnostics examining local error reinforcement and correct-reply suppression. These additions appear in Appendix C, Extended Ablation, complementing the component ablations in Section 5.3.2.

3. Clearer presentation of human validation.
We have clarified the existing human validation of the intent annotations in Appendix B.3 and added explicit references to the annotation instructions in Appendix H. Three annotators independently assign turn-level intent scores to 300 stratified dialogues without access to the model-produced annotations. We distinguish agreement on these 0–5 intent scores—Fleiss’ κ = 0.77, Cohen’s κ = 0.71, and 87.0% exact agreement—from the separate assessment of rationale plausibility, for which 92% receive ratings of at least 4 on a five-point scale. We also distinguish this direct annotation validation from the downstream behavioral evaluations in Section 6.1 and Appendices G.2–G.3.

4. Clearer positioning and data documentation.
We have revised the introduction and related work to present the behavioral alignment objective and the scope of optimization as connected aspects of the same problem. The revised evaluation includes the additional UserLM-style training-based baseline reported in our responses. We have also expanded the Ethics Statement and Appendix B.1 to describe the data source, the basis for research use, de-identification, manual review, and the approval requirements for external release.

Reviewer-specific revisions

Reviewer SuMf (W1–W2). We thank the reviewer for highlighting local update behavior and test-set selection. In response, Appendix C includes single-step diagnostics and Shuffled/Signed-local controls examining local error reinforcement and the limits of sign-preserving modulation. Table 1, Sections 5.1–5.2, and Appendix B.4 report evaluation on the complete eligible test pool, including previously excluded tied groups.

Reviewer sevW (W1–W2, Q1–Q3). We appreciate the reviewer’s suggestions on evaluation coverage, training variability, and intent supervision. Table 1 and Appendix B.4 address test-set selection, while Appendix C adds an independent training run and weighting controls, with inference-seed variability explicitly identified. Appendices B.3 and H clarify direct annotation validation; Section 6.1 and Appendices G.2–G.3 provide complementary behavioral evidence.

Reviewer Gxsd. We thank the reviewer for encouraging clearer positioning, stronger comparisons, and fuller documentation. Sections 1–2 clarify the contribution and positioning relative to existing user simulators, including ARGUS. Section 5.1 and Table 1 include the UserLM-style comparator alongside HUMANLM. Appendices B.3, C, and H address annotation validation and modulation controls. The Limitations clarify the task-oriented scope, while the Ethics Statement and Appendix B.1 document data use, privacy protections, and release conditions.

We welcome further feedback and would be happy to address any remaining concerns and improve the manuscript accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.