- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work focuses on the task of LLM-simulated human partners, where the authors aims to investigate simulations fidelity. Specifically, the authors propose new metrics to quantify the fidelity of LLM simulations and identify persistent performance gaps in frontier models that cannot be fully mitigated through in-domain adaptation.
Critical strengths
- The research topic, investigating simulation fidelity, is interesting and holds practical value.
- The proposed fidelity metric is interesting and meaningful, providing a quantitative method to measure the simulation fidelity.
- The experiments are relative extensive, examining simulation fidelity from various perspectives.
- Overall, this work is easy to follow.
Critical weaknesses
- Although investigating simulation fidelity is an interesting research direction, this work does not provide an effective solution or a promising framework for improving simulation fidelity. Instead, it primarily focuses on measuring fidelity and identifying existing performance gaps. The authors should clarify the practical implications of their findings and discuss how the proposed metrics could help improve LLM-simulated human partners.
- The selection and construction of the features described in Section 2.2 need further clarification. Since these features play a critical role in measuring simulation fidelity, the authors should provide more details about how they are selected, defined, and constructed. They should also justify why these features are appropriate for capturing the key aspects of simulation fidelity and discuss any potential limitations of the chosen feature set.
- The proposed simulation fidelity metric relies on ground-truth human feature distributions, which are generally difficult to obtain in many real-world scenarios. This requirement may limit the practical applicability of the proposed method. The authors should clarify the data requirements of the metric and discuss how it could be applied when ground-truth human feature distributions are unavailable or only partially accessible.
Questions for authors
Please read the above sections.
2 comments
Thanks authors for their response. Althought the authors present some extra clarification, the Practical implications still appear a little limited. Moreover, I'm not sure whether it is in accordance with regulations to add extra link in the rebattle stage. In current stage, I will retain my positive scores.
Response to Reviewer Hw6D
Thank you for your recognition of this paper’s strengths. We address your concerns below.
Weakness 1: Practical implications. Our experiments provide three practical insights and recommendations. We will expand on this discussion with the extra page of the camera-ready paper.
Weakness 2: Feature selection. Our features were selected from established conversational and psychotherapy frameworks to capture behaviors relevant to each simulation setting. For user simulation, the seven features cover dialogue act, feedback, politeness, topic shift, emotion, emotional disclosure, and information disclosure. For patient simulation, we draw on psychotherapy-process instruments and constructs such as the Experiencing Scale (Klein et al., 1969), Insight (Hill et al., 1992), and Innovative Moments (Gonçalves et al., 2011), informed by the Generic Model of Psychotherapy, RDoC, and MITI 4.2. Together, these capture how clients present problems, engage emotionally, express expectations, and show signs of change. Their relevance is supported by our frontier-model analyses, which show systematic differences in these dimensions (Appendices F and H). Definitions and codebooks are provided in Appendices C and G, and reliability-based selection is described in Appendix E. We will clarify the feature construction and limitations, including that reliability screening may exclude clinically important behaviors and that our features do not capture all aspects of fidelity, such as longitudinal consistency.
Weakness 3: Ground-truth data requirements. Our fidelity metric is intended for settings where representative human interaction data can be collected or already exists, such as user-interaction datasets and psychotherapy transcripts. Once annotated, these reference distributions can be reused to evaluate multiple simulators. Smaller reference sets can still be used, although uncertainty increases. Our split-half procedure provides one way to estimate finite-sample variation. Without any human reference data, the JSD metric cannot be computed directly. We will clarify these requirements and distinguish direct fidelity measurement from proxy-based comparisons using language-model loss.