- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CPTE-EIG, an active learning criterion for preference elicitation that targets a causal estimand: the Conditional Preference Treatment Effect (CPTE). The CPTE is the probability that a treated outcome is preferred to a control outcome for a patient with covariate x, where preferences are elicited from clinicians through pairwise comparisons of multivariate outcome profiles. The authors model the latent utility with a Gaussian process, linearise the CPTE around the posterior mean utility to obtain a tractable Gaussian approximation with an error bound, and derive a closed-form expected information gain (up to one-dimensional Gauss–Hermite quadrature). Experiments on a semi-synthetic Parkinson’s disease cohort with 13 outcomes show that CPTE-EIG reduces CPTE estimation error and policy regret compared to BALD, EPIG, and random acquisition, especially under mild and strong treatment effects.
Critical strengths
1.Important and well-motivated problem. Clinical decisions often require trading off multiple outcomes, and eliciting preferences is costly. Targeting the causal estimand directly, rather than the preference function or individual predictions, is a principled use of limited clinician time.
2.Novel methodological contribution. To my knowledge, this is the first active learning method for preference elicitation that targets a causal estimand. The formulation of the CPTE as a posterior random variable and the linearisation with an explicit error bound are technically sound.
3.Computational efficiency. The closed-form CPTE-EIG expression, requiring only Gauss–Hermite quadrature, avoids posterior sampling and model refitting per candidate. The ablation in Appendix F.7 confirms that few quadrature nodes match Monte Carlo with thousands of samples.
Critical weaknesses
1.Candidate pool construction. Queries are restricted to pairs of real PPMI profiles sharing the same query covariate. This is realistic but may limit the diversity of comparisons. The impact of this restriction on the learned preference rule is not analyzed.
2.Outcome dimensionality. The experiments use 13 outcomes. The authors note that profiles with too many dimensions may be hard to compare in a single judgment. Cognitive load and clinician reliability with 13-dimensional profiles are not studied.
3.Baseline scope. BALD, EPIG, and random are reasonable baselines, but the paper does not compare to a method that directly models the CPTE or to a non-active elicitation strategy (e.g., balanced sampling). The advantage of targeting the estimand is clear, but the practical gain over simpler heuristics could be more thoroughly explored.
4.Real-world validation absent. The method is evaluated only on semi-synthetic data with a simulated clinician. While this allows oracle evaluation, it does not demonstrate that the approach works with real clinicians and real preference noise.
Questions for authors
Please see weakness
2 comments
We have updated our replies above so that the appendix references match the latest version of the paper. We would be happy to discuss any point that remains unclear.
We sincerely thank the reviewer for the careful reading and constructive comments, which we answer below.
This choice comes from the clinicians we work with: they can only judge reliably profiles that look like real patients (Section 3). We first tried generating queries by continuous optimisation over [0,4]13, but this produced non-integer ordinal scores and clinically unrealistic profiles. A shared query covariate also matches the estimand, since the CPTE only involves contrasts between treated and control profiles at the same x~ (Eq. 7, Appendix F.2). The pool remains diverse, with 1,369,284 comparisons over all ages and both sexes (Appendix F.1). We will add this discussion to Section 3.
The 13 items were chosen with clinicians as the dimensions they weigh in practice when choosing a treatment (Table 1), and removing some would make comparisons less faithful. We agree that cognitive load and rater reliability must be evaluated (Section 6). Several elements help:
We added this to Section 6 and build these checks into our elicitation software.
We added two baselines that directly target the CPTE: CPTE variance reduction and decision EIG (which targets 1(Ψx>1/2)), with the same Gaussian approximation as CPTE-EIG (Appendix F.3). They are close to CPTE-EIG (Figures 2–3, Table S1), so most of the gain comes from targeting the CPTE. CPTE-EIG is still significantly better in some small-budget settings. For example, against decision EIG on the strong effect at 40 comparisons, its regret is 26.1 vs. 36.2×10⁻³ (Appendix F.5). It is significantly worse only with the tiny effect at large budgets, where all methods are already accurate (Appendix F.5).
We also tested a non-active balanced sampling strategy, which queries all patient groups (sex × age × disease duration) equally often. It was significantly worse than CPTE-EIG in every setting, and often no better than random. Since it brought no information beyond the random baseline, we did not include it in the paper. Simple heuristics of this kind spread the queries over the whole population, while the gain comes from targeting the CPTE.
We fully agree that a study with real clinicians is the natural next step, and it motivated this work: clinician time is scarce, so real elicitation needs a strategy that learns the effect from few questions. Real clinicians cannot provide an oracle CPTE, so controlled experiments are needed first to validate the method.
To make these experiments more realistic, we added LLM clinicians (Section 5.2, Figure 3). Each LLM sees the patients' sex, age group, disease duration and 13 outcomes in words, but never the treatments. We use BioDecision-v2-4B [1], and Qwen3-32B [2] with a neuropsychiatric and a motor persona (prompts in Appendix F.2). Their answers are not generated by our model and are noisy. The Qwen3 answers depend on the presentation order in 31–36% of comparisons, and the two personas disagree on 7.4% of them (Appendix F.2, Figure S1).
CPTE-EIG has the lowest mean CPTE error from 40 comparisons on with BioDecision and the neuropsychiatric persona, and at 40–60 comparisons with the motor persona (Figure 3). Over 18 threshold and budget settings, it also has the lowest regret among CPTE-EIG, EPIG, BALD and random in 17 (Appendix F.6). We plan to evaluate it with real clinicians next.
We thank the reviewer again for these helpful suggestions and would be glad to clarify any remaining point.
[1] https://huggingface.co/lighteternal/biodecision-v2-4b [2] https://huggingface.co/Qwen/Qwen3-32B