- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes OASIS, a framework that uses latent user thoughts as training-time privileged information to improve assistants that observe only dialogue at deployment. User simulators generate multi-turn interactions with explicit latent thoughts, which inform assistant-generated responses for supervised fine-tuning. A subsequent on-policy stage trains the assistant using token-level log-probability differences between a frozen reference model’s privileged and observable views, without requiring a stronger response-generating teacher. The central contribution is using dynamically generated user states as supervision for assistant self-improvement. Experiments with Qwen3-4B and Qwen3-8B on WildBench and DianJin-CSC report improvements over the base models and SFT initialization, across two WildBench judges and both reference-context and generated-context customer-service evaluations. The paper also includes ablations and human assessments of simulated thought quality.
Critical strengths
-
A well-motivated use of training-time user information. OASIS uses simulator-generated latent user thoughts to supervise an assistant that operates solely on observable dialogue at deployment. This addresses a meaningful challenge: benefiting from richer user information without requiring users to provide it during actual interactions.
-
Consistent reported improvements across complementary settings. The main results show gains across both general assistance and commercial customer service, two model sizes, and two independent WildBench judges. The full method also consistently improves upon its SFT initialization. This breadth provides encouraging evidence of the approach’s empirical utility, even though the precise contribution of latent-thought supervision requires stronger controlled comparisons.
Critical weaknesses
-
OASIS uses assistant-generated training trajectories filtered for satisfactory final outcomes, whereas the Standard-SFT baseline uses ordinary WildChat dialogues. This simultaneously changes the data source, response style, trajectory quality, and filtering procedure. Consequently, the substantial advantage of OASIS-SFT may partly stem from higher-quality synthetic data and successful-trajectory filtering; the current comparison cannot isolate the contribution of latent thoughts. The authors should consider a controlled comparison that keeps the initial tasks, user simulator, generation model, filtering criteria, and training budget fixed, varying only whether latent thoughts are provided when generating the training responses.
-
A major concern is whether the synthetic user data are sufficiently reliable and faithfully reflect real-world user intentions. Since the paper emphasizes understanding implicit needs, the evaluation metrics should assess whether the model correctly identifies the constraints that genuinely matter to the user and whether its final responses better satisfy that user’s actual needs. The paper should also address how to handle unsupported inferences about user preferences and hallucinated assumptions.
Questions for authors
- [1] also concerns latent user information and train model for better interactive assistants with synthetic users, can author identify their main difference from this work?
[1] Bo Peng, Zhiheng Wang, Heyang Gong, and Chaochao Lu. 2025. IP-Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Data. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 17007–17040, Suzhou, China. Association for Computational Linguistics.
2 comments
(2/2) Official Comment by Authors
Q1: Differences Between OASIS and IP-Dialog
Thank you for raising this question about IP-Dialog. Both works explore how implicit user information can improve dialogue assistants, but they differ primarily in research objectives, the source and representation of implicit information, and how such information is utilized during training. Specifically:
-
Research objectives and implicit information. IP-Dialog focuses on evaluating and improving implicit personalization using synthetic data constructed from pre-defined user attributes. In contrast, OASIS aims to improve interactive assistants by leveraging turn-level latent thoughts generated by a trained user simulator, capture users' evolving intentions and constraints during interactions, rather than relying on pre-defined user attributes.
-
Training method. IP-Dialog uses SFT to train the assistant to infer relevant attributes and generate analyses and responses. OASIS uses latent thoughts as training-time privileged information to guide response generation during SFT. Subsequently, OASIS provides dense supervision at interaction states visited by the student through OPSD, using two views of the same model with and without latent thoughts. This enables the assistant to internalize the benefits of latent thoughts while relying only on observable dialogue at deployment.
We have added a citation to IP-Dialog in Section 5 (Related Work, L474–L476) and clarified the research objectives and key differences.
Thank you again for the thoughtful questions and positive evaluation of our work, which serve as great encouragement for us. We hope these clarifications resolve your concerns, and we sincerely appreciate your willingness to further evaluate our work.
(1/2) Official Comment by Authors
Thank you for acknowledging the value of well-motivated use of training-time user information and consistent performance improvements. We have carefully prepared responses to address your concerns.
Thank you for raising this important concern. We agree that a controlled comparison would provide a fairer evaluation and better demonstrate the importance of latent thoughts.
We conducted an experiment with the key difference from OASIS-SFT being whether latent thoughts are accessible. Specifically, SFT w/o Privileged uses the same initial tasks as OASIS-SFT, with Qwen3-8B and Qwen3-4B serving as assistant models that engage in multi-turn interactions with the user simulator without access to its latent thoughts. The final SFT dataset contains about 9,467 dialogues and 28,401 turns, matching the scale of OASIS-SFT.
The experimental results below show that OASIS-SFT achieves WB Scores of 24.71 and 15.55 on Qwen3-8B and Qwen3-4B, outperforming SFT w/o Privileged (−17.50 and −20.41) by 42 and 36 points. These results show the benefits of users' latent thoughts as privileged supervision.
Thank you for raising this important concern. We agree that the faithfulness of generated user thoughts, as well as the model's ability to accurately understand and satisfy real users' needs, is essential for validating the effectiveness of OASIS. We highlight our human evaluation of Faithfulness of Simulated Thoughts and add it with evaluations of Real-User Need Understanding. Specifically,
Faithfulness of Simulated Thoughts. OASIS's general-domain user simulator is trained on ThoughtTrace, which contains latent thoughts reported by real users rather than relying on synthetic user preferences. As reported in Table 2, we evaluate its faithfulness on 500 held-out real-user turns with three independent human annotators. The trained simulator improves Human-Thought Alignment from 3.41 to 3.96 (out of 5), and numbers of ≥4 increase from 69.2% to 86.8%. These results support the reliability of simulated thoughts as proxies for real user states, while acknowledging that they cannot perfectly reproduce real user intentions.
Real-Users Need Understanding. To directly evaluate whether OASIS better understands and satisfies real users' needs, we conduct an evaluation on 200 held-out real-user conversations from ThoughtTrace, using users' self-reported thoughts as evaluation references while providing the assistant with only observable dialogue. Three independent human annotators, each holding a master's degree in engineering, evaluate the assistant responses along four dimensions: (1) Constraint Alignment, whether responses correctly identify and address constraints that genuinely matter to users; (2) Need Satisfaction, whether responses effectively satisfy users' actual needs; (3) Assumption Grounding, whether responses avoid unsupported assumptions about user preferences; and (4) Clarification Appropriateness, whether assistants appropriately seek clarification when important constraints remain uncertain. These results suggest that OASIS better identifies and addresses real users' needs while reducing unsupported preference assumptions. Detailed evaluation criteria are provided in Appendix A.6 (L810-L838).