acceptodds
Under review as a conference paper at ICLR 2027

Can Individual Physician History Improve Dialogue-Action Prediction? A Personalization Benchmark with Four Profile Sources

Abstract

Profile-based personalization uses individual statistical priors, history retrieval, dense profile representations, and user-specific parameter adaptation. These methods use history differently, but they are usually evaluated in the same way: predictions made with the target individual's profile are compared with predictions made without a profile. An improvement in this comparison does not by itself show that the model uses history specific to that individual. Relevant histories from similar individuals may already help, while a profile from another individual may harm prediction through mismatch. We study this issue in physician dialogue-action prediction. For each target physician reply, we hold the current dialogue fixed and vary only the profile source, comparing no profile, a department-pooled profile, another physician's profile, and the target physician's profile. The pooled profile excludes the target physician, and the other physician comes from the same department and has a similar number of profile consultations. We construct a benchmark with 244 evaluation physicians and 164,702 physician-reply events in the test set, labeled with 11 functional dialogue actions. We evaluate seven representative methods spanning statistical priors, history retrieval, dense profile representations, and user-specific parameter adaptation. Some gains remain after the target physician is excluded, while some Target-versus-Other differences arise because the other physician's profile increases the loss. Same-department history retains part of the profile gain for several methods, whereas the additional improvement from the target physician's history varies by method. Comparing all four sources shows whether an apparent gain remains with same-department history, depends on the target physician's records, or is enlarged by a mismatched profile.

Primary area:
datasets and benchmarks
Keywords:
evaluation and benchmarking, personalization, profile-conditioned prediction, user modeling, medical dialogue, dialogue-action prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NoUrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether improvements from physician-history conditioning depend on the history belonging to the target physician. It introduces a four-source evaluation protocol that holds the prediction event fixed while supplying no profile, pooled same-department history excluding the target, a matched other physician’s history, or the target physician’s own history. The benchmark evaluates multilabel prediction of 11 functional dialogue actions on 164,702 test reply events from 244 physicians. Seven task-adapted predictors cover statistical, retrieval-based, representation-based, and parameter-adaptation approaches. The results show that some target-profile gains largely remain with other same-department histories, while some Target-versus-Other differences arise primarily from deterioration under the other physician’s profile.

Critical strengths
  1. A meaningful evaluation framework for understanding personalization.

The four-source evaluation provides a useful perspective beyond the conventional Target-versus-No comparison. By introducing department-pooled and matched other-physician profiles, the paper distinguishes gains from shared historical information and those associated with individual-specific histories. The contrasting results for PPlug and OPPU demonstrate why these additional comparisons are important for interpreting personalization performance.

  1. Comprehensive empirical comparisons and statistical analysis.

The experiments cover seven representative personalization methods across different modeling paradigms and evaluate them on the same prediction events. The benchmark includes 244 physicians and a substantial number of consultation events. Furthermore, the physician-level paired comparisons, bootstrap confidence intervals, and stability analysis across consultation subsets strengthen the empirical findings.

Critical weaknesses
  1. Insufficient validation of the automatically generated dialogue-action labels.

The authors first use an LLM to annotate 5,975 events and then train a TF-IDF-based classifier to label the remaining data. However, the classifier achieves only 0.676 Micro-F1 and 0.648 Macro-F1 against the teacher annotations. Although additional inter-model agreement analysis is provided, the benchmark lacks independent human validation. Since the main conclusions rely on these automatically generated labels, potential systematic annotation errors could affect the reliability of the reported personalization gains. A human-annotated evaluation subset would substantially strengthen the paper.

  1. Insufficient controls to fully explain the observed personalization gains.

Although the four-source evaluation is well motivated, the experiments do not fully isolate the effects of meaningful historical information. For example, PPlug achieves almost identical performance using target, department-pooled, and other-physician profiles, while showing substantially worse performance without a profile. This raises the question of whether the improvement comes from useful historical content or simply from providing a profile representation. Furthermore, probability calibration for language-model-based methods is performed using only target-profile development data, which may influence the BCE comparisons across different profile sources. Additional controls with irrelevant profiles and alternative calibration strategies would help clarify these effects.

Questions for authors
  1. Could the authors provide an independently human-annotated evaluation subset to validate the dialogue-action labels? Do the main findings remain consistent when evaluated against human-validated annotations?

  2. How sensitive are the reported BCE improvements to probability calibration? Would the main conclusions remain unchanged if calibration were performed using a balanced mixture of the four profile sources?

  3. For PPlug, could the authors investigate whether random or irrelevant profiles produce similar improvements over No Profile? This would help clarify whether the observed gains depend on meaningful historical information.

Reviewer_1kteOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper evaluates two cases for a patient whether history from the same doctor helps more or from other doctor. The results say that another person's history doesnt help. Overall it is an evaluation paper with limited experiments.

Critical strengths

The problem address is important however not a good fit for the audience of this conference.

Critical weaknesses
  1. This is primarily a benchmark paper, not a new-model paper.

  2. The paper does not conclude that history from the same person is unimportant. Rather, it argues model performing better when given a doctor’s history does not necessarily mean that the model is learning something specific to that doctor.

  3. All experiments use one Chinese online consultation platform, one domain, and 244 evaluation physicians.

Questions for authors
  1. Report accuracy for trained teacher model and trained classifier model
  2. Ablation exp : what happens if all the consultations used to build a doctor’s profile happened before the test consultations
  3. compare the target doctor with several different doctors, rather than just one?
Reviewer_KH11OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The way I understand it the paper studies whether personalization gains really come from an individual’s own history. Instead of comparing only target profile vs. no profile, the authors add two controls. One is a pooled profile from other physicians in the same department, and the profile of one other physician from the same department. They test this on physician dialogue-action prediction, using a Chinese online consultation dataset (with action labels produced by an LLM), and run seven personalization methods. For several methods, much of the gain is already there with same-department history. Some target-vs-other gaps exist mainly because the other physician’s profile makes predictions worse.

Critical strengths
  • The paper asks a good question. Comparing target profile vs. no profile doesn’t show that a model uses that individual’s history, and adding a same-department pooled profile and another physician’s profile is a simple cheap way to check. The controls aren’t tied to medicine and could be added to other personalization benchmarks.

  • The four-way comparison shows things that a plain target-vs-no comparison would hide. The provided examples make the paper’s argument concrete, and the trends stay the same across the development, holdout and test sets.

Critical weaknesses
  • It’s not clear that the gains come from physician behavior at all. Over 99% of the labels come from a TF-IDF classifier that agrees with the LLM teacher at only about 0.68 micro-F1, and no labels were checked by humans. The classifier’s mistakes likely follow each physician’s writing style, and the same classifier labels both the profile and the test data. So a model that sees the target’s history could be learning that physician’s labeling errors, which would look exactly like a target-specific gain. Also, the splits are not in time order, and nothing prevents the same patient from appearing in both the profile and the test data (or maybe I didn't understand this correctly). Any of these could produce the main effects the paper reports, so its central claims are not supported yet.

  • The baselines are too weak to say much about the methods they represent in the paper. LaMP, PPlug and PRISP do worse than a simple action frequency baseline under every profile source, and LaMP and PRISP have near zero F1. The strong model, OPPU, gets no gain from the target profile, and the paper doesn’t discuss this. Together with a single dataset and no clear release of the labels, the paper is hard to use as a benchmark.

Questions for authors

See Weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.