- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether improvements from physician-history conditioning depend on the history belonging to the target physician. It introduces a four-source evaluation protocol that holds the prediction event fixed while supplying no profile, pooled same-department history excluding the target, a matched other physician’s history, or the target physician’s own history. The benchmark evaluates multilabel prediction of 11 functional dialogue actions on 164,702 test reply events from 244 physicians. Seven task-adapted predictors cover statistical, retrieval-based, representation-based, and parameter-adaptation approaches. The results show that some target-profile gains largely remain with other same-department histories, while some Target-versus-Other differences arise primarily from deterioration under the other physician’s profile.
Critical strengths
- A meaningful evaluation framework for understanding personalization.
The four-source evaluation provides a useful perspective beyond the conventional Target-versus-No comparison. By introducing department-pooled and matched other-physician profiles, the paper distinguishes gains from shared historical information and those associated with individual-specific histories. The contrasting results for PPlug and OPPU demonstrate why these additional comparisons are important for interpreting personalization performance.
- Comprehensive empirical comparisons and statistical analysis.
The experiments cover seven representative personalization methods across different modeling paradigms and evaluate them on the same prediction events. The benchmark includes 244 physicians and a substantial number of consultation events. Furthermore, the physician-level paired comparisons, bootstrap confidence intervals, and stability analysis across consultation subsets strengthen the empirical findings.
Critical weaknesses
- Insufficient validation of the automatically generated dialogue-action labels.
The authors first use an LLM to annotate 5,975 events and then train a TF-IDF-based classifier to label the remaining data. However, the classifier achieves only 0.676 Micro-F1 and 0.648 Macro-F1 against the teacher annotations. Although additional inter-model agreement analysis is provided, the benchmark lacks independent human validation. Since the main conclusions rely on these automatically generated labels, potential systematic annotation errors could affect the reliability of the reported personalization gains. A human-annotated evaluation subset would substantially strengthen the paper.
- Insufficient controls to fully explain the observed personalization gains.
Although the four-source evaluation is well motivated, the experiments do not fully isolate the effects of meaningful historical information. For example, PPlug achieves almost identical performance using target, department-pooled, and other-physician profiles, while showing substantially worse performance without a profile. This raises the question of whether the improvement comes from useful historical content or simply from providing a profile representation. Furthermore, probability calibration for language-model-based methods is performed using only target-profile development data, which may influence the BCE comparisons across different profile sources. Additional controls with irrelevant profiles and alternative calibration strategies would help clarify these effects.
Questions for authors
-
Could the authors provide an independently human-annotated evaluation subset to validate the dialogue-action labels? Do the main findings remain consistent when evaluated against human-validated annotations?
-
How sensitive are the reported BCE improvements to probability calibration? Would the main conclusions remain unchanged if calibration were performed using a balanced mixture of the four profile sources?
-
For PPlug, could the authors investigate whether random or irrelevant profiles produce similar improvements over No Profile? This would help clarify whether the observed gains depend on meaningful historical information.