- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Patient Records with Integrated Signals across Modalities (PRISM), a dataset built from standardized-patient (SP) videos for modeling and evaluating the consistency between simulated patients and human SPs. This paper also proposes a new patient simulation method, Behavior-Aware Patient Simulator (BAPS), which uses an external skill library that is continuously updated during optimization and fixed during inference to align verbal, nonverbal, and information disclosure behaviors with those of human SPs. The experimental results show that BAPS outperforms the other simulation methods compared in this paper on the PRISM dataset.
Critical strengths
- The PRISM dataset provides aligned multimodal data and behavior annotations from human standardized patients, which is useful for modeling and evaluating patient behavior beyond verbal responses.
- The BAPS method is effective without updating model parameters. Its skill library improves patient simulation and can be reused across different backbone models.
Critical weaknesses
- Although the authors claim that their dataset and method cover both verbal and nonverbal dimensions, the model still represents nonverbal behaviors through textual descriptions. The evaluation does not include actual animation, speech, or their synchronization. The results support more appropriate behavioral decisions, but the evidence for the interaction realism of a complete virtual patient and its practical educational value is still limited.
- The experiments show that BAPS outperforms the compared patient simulation methods, but there is no comparison with general prompt or context optimization methods such as ACE and GEPA. Therefore, it is still unclear how much of the improvement comes from the proposed skill library structure and update mechanism. I suggest adding these comparisons using the same training data to further clarify the methodological contribution.
Questions for authors
- Lines 854–857: Why are the four disclosure states assigned scores of −1.0, 0.3, 1.0, and −0.2? Does BAPS still outperform the baselines under other reasonable scoring weights?
- The human evaluation questions are adapted from MaSP into pairwise comparisons. What evidence supports the validity and reliability of this adapted evaluation protocol? Were the questions reviewed by experts, and was agreement among the three annotators measured?
- Line 493: placeholder [LLM name] ; line 501: placeholder [N].
- The authors could add case studies and error analysis to show where BAPS improves patient simulation and where it still fails.
I would be willing to raise my rating if the authors adequately address these concerns.
5 comments
W2
W2: The experiments show that BAPS outperforms the compared patient simulation methods, but there is no comparison with general prompt or context optimization methods such as ACE and GEPA. Therefore, it is still unclear how much of the improvement comes from the proposed skill library structure and update mechanism. I suggest adding these comparisons using the same training data to further clarify the methodological contribution.
A2: Thank you for this valuable suggestion. We address this concern through an additional baseline comparison and a clarification of BAPS's methodological contributions.
Additional baseline results show that BAPS has higher mean scores than ACE on all reported verbal, nonverbal, and disclosure metrics. Since our method is a context evolution method, we would prioritize ACE as a more related baseline compared to the prompt optimization method GEPA. Hence, we have added ACE as a baseline using the same PRISM training set, backbone LLM, and evaluation protocol as BAPS. As shown in the table below, BAPS consistently outperforms ACE across all dimensions:
| Method (runs) | SBERT↑ | BLEU1↑ | BLEU4↑ | ROUGEL↑ |
|---|---|---|---|---|
| BAPS | 0.4231 ± 0.0035 | 0.1322 ± 0.0034 | 0.0306 ± 0.0014 | 0.2525 ± 0.0053 |
| ACE | 0.4111 ± 0.0029 | 0.1257 ± 0.0012 | 0.0297 ± 0.0004 | 0.2492 ± 0.0014 |
| Method | Emotion micro-F1↑ | Emotion Macro-F1↑ | Facial micro-F1↑ | Facial Macro-F1↑ | Gaze↑ | Head↑ | Body↑ | Hand↑ | Voice↑ | Speech↑ | Traj.↑ | Int. MAE↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BAPS | 0.6564 ± 0.0074 | 0.3489 ± 0.0269 | 0.4491 ± 0.0092 | 0.3662 ± 0.0095 | 0.8392 ± 0.0013 | 0.4193 ± 0.0186 | 0.8182 ± 0.0084 | 0.4476 ± 0.0188 | 0.7584 ± 0.0025 | 0.7955 ± 0.0165 | 0.9199 ± 0.0075 | 0.7076 ± 0.0111 |
| ACE | 0.5944 ± 0.0031 | 0.2998 ± 0.0041 | 0.4007 ± 0.0067 | 0.3119 ± 0.0075 | 0.7212 ± 0.0052 | 0.4106 ± 0.0084 | 0.7143 ± 0.0053 | 0.4119 ± 0.0098 | 0.6341 ± 0.0050 | 0.6505 ± 0.0107 | 0.9000 ± 0.0047 | 0.7336 ± 0.0072 |
| Method | Leak rate↓ | Disclosure score↑ |
|---|---|---|
| BAPS | 0.0384 ± 0.0033 | 0.2757 ± 0.0068 |
| ACE | 0.0392 ± 0.0018 | 0.2703 ± 0.0026 |
We have incorporated the ACE comparison results in Table 2 of the revised manuscript.
Moreover, we would also like to clarify our methodological contribution:
- Interactive Patient World Modeling (vs. Static Task-Solving Memory): ACE is designed for single-outcome, static reasoning or tool-use tasks, where evolution consists of accumulating factual heuristics to maximize task completion. In contrast, BAPS builds an interactive patient world model to serve as a high-fidelity training simulator for medical learners in realistic clinical consultations. Rather than optimizing a single scalar answer, a patient world model must govern multi-dimensional behavioral state transitions. Specifically, it regulates how a patient dynamically discloses clinical facts while maintaining coherent verbal tone and nonverbal language across nuanced doctor–patient interactions. This paradigm shift requires a fundamentally different learning architecture rather than surface-level prompt tuning.
- Modality-Decoupled Credit Assignment (vs. Global Scalar Feedback): Because a clinical patient world model must coordinate simultaneous verbal, disclosure, and nonverbal behaviors, relying on a single holistic score (as in ACE) causes credit assignment collapse and cross-modal interference (e.g., penalizing an accurate factual disclosure because a nonverbal cue was slightly off). Instead, BAPS separates evaluation and rule updates across the three behavioral dimensions, enabling dimension-specific error diagnosis and targeted refinement without cross-channel interference.
- Inductive Abstraction & Active Pruning (vs. Additive Accumulation): While ACE performs embedding-based merging, its updates remain accumulative: it merely deduplicates near-identical text while continually appending fragmented rules, expanding the library to 424 entries and introducing retrieval noise. In contrast, BAPS performs inductive abstraction to synthesize shared patterns across instances into compact, transferable principles that replace redundant entries, alongside active pruning to purge rules that trigger downstream errors. This compresses the library by ~4x (converging to 122 rules vs. ACE's 424 on the same data) while achieving superior performance across all metrics, demonstrating that BAPS effectively eliminates redundant information to distill higher-quality rules.
Q1
Q1: Lines 854–857: Why are the four disclosure states assigned scores of −1.0, 0.3, 1.0, and −0.2? Does BAPS still outperform the baselines under other reasonable scoring weights?
A1: Thank you for this question. The four weights in the disclosure score were grounded directly in clinical-pedagogical principles for standardized patient training rather than tuned on data. To verify that our findings do not depend on these exact values, we parameterized the metric and conducted a sensitivity analysis across 23 weight configurations, confirming that BAPS outperforms all baselines across all reasonable settings.
1. State Definitions, Parameterized Formulation, and Value Ranges At each turn, every tracked clinical fact falls into one of four mutually exclusive states depending on whether the clinician has queried the fact and whether the model mentions it:
| Clinical Query Status | Mentioned by Model | Not Mentioned by Model |
|---|---|---|
| Not yet queried | Leak (L, weight -c, default -1.0) | Withheld (W, weight +b, default +0.3) |
| Queried this turn | Revealed (R, weight +a, default +1.0) | Missed (M, weight -d, default -0.2) |
Using non-negative weight magnitudes a, b, c, d \in [0, 1], the parameterized Disclosure Score is defined as:
where S(a,b,c,d)∈[−1.0,+1.0]. Our default setting (a,b,c,d)=(1.0,0.3,1.0,0.2)enforces the pedagogical ordering −c<−d<0<+b<+a: • Equal Unit Boundaries (a=1.0,c=1.0): Because scaling all weights by a positive constant preserves model rankings, we anchor the primary reward and penalty at the unit boundary 1.0within [0,1]. We assign equal magnitude to Revealed (a=1.0) and Leak (c=1.0) because accurately providing queried information and preventing unqueried spoilers are equally critical to standardized patient training. • Modest Restraint Reward (0<b=0.3<a): In multi-turn history taking, clinicians query only one or two facts per turn while most case facts remain unqueried; assigning a modest +0.3to Withheld rewards restraint without allowing an always-silent simulator to achieve a high score. • Milder Omission Penalty (0<d=0.2<c): Unlike an irreversible leak, a Missed fact receives a milder −0.2penalty because clinicians can still elicit omitted details via follow-up questioning, avoiding incentives for premature over-disclosure. 2. Sensitivity Analysis Across 23 Weight Configurations Holding model predictions fixed on the human-annotated test set, we varied one weight at a time across {0,0.25,0.5,0.75,1.0}while holding the other three at their defaults (for example, testing (a,0.3,1.0,0.2)for a∈{0,0.25,0.5,0.75}). This yields 4+5+4+5=18non-default variants plus the default setting itself, totaling 19 unique one-factor configurations. We also included 4 single-term degenerate endpoints where three weights are set to zero (such as (1,0,0,0)), giving 23 configurations in total. Below, we report the mean score and Rank-1 Win Rate (the proportion of configurations in which a method achieves the highest mean score):
| Evaluation Metric | BAPS | PWP | PatientSim | EasyMED |
|---|---|---|---|---|
| Default Score (1.0, 0.3, 1.0, 0.2) | 0.2757 ± 0.0068 | 0.2382 ± 0.0089 | 0.2382 ± 0.0038 | 0.2649 ± 0.0040 |
| Mean Score (19 One-Factor Sweeps) | 0.3200 | 0.2849 | 0.2847 | 0.3099 |
| Rank-1 Win Rate (19 One-Factor Sweeps) | 19 / 19 (100%) | 0 / 19 (0%) | 0 / 19 (0%) | 0 / 19 (0%) |
| Rank-1 Win Rate (All 23 Configurations) | 21 / 23 (91.3%) | 2 / 23 (8.7%) | 0 / 23 (0%) | 0 / 23 (0%) |
BAPS ranks strictly first across all 19/19 one-factor sweeps (by an average margin of +0.0109 over the strongest baseline) as well as on Withheld-only (0,1,0,0) and Leak-only (0,0,1,0), trailing PWP by 0.0019 only on Revealed-only (1,0,0,0) and Missed-only (0,0,0,1). Full per-configuration results are provided in Appendix N (Table 28).
W1
We sincerely thank Reviewer cLFe for the careful evaluation of our manuscript, the encouraging assessment, and the constructive comments. We particularly appreciate the reviewer’s recognition of PRISM’s aligned multimodal behavioral annotations and BAPS’s parameter-free, cross-backbone transferable skill library. Your insightful suggestions regarding audiovisual interaction realism, fair prompt-optimization baselines, evaluation protocol validation, and qualitative error analysis have guided extensive new experiments and revisions that substantially strengthened our manuscript. We address each comment in detail below.
W1: Although the authors claim that their dataset and method cover both verbal and nonverbal dimensions, the model still represents nonverbal behaviors through textual descriptions. The evaluation does not include actual animation, speech, or their synchronization. The results support more appropriate behavioral decisions, but the evidence for the interaction realism of a complete virtual patient and its practical educational value is still limited.
A1: Thank you for this comment. To complement our existing evaluation of dialogue and structured behavioral decisions, we added a blinded pilot in which raters evaluated short audiovisual clips generated with Kling 4.0, extending our assessment to perceived patient portrayal quality in rendered video.
In this paired comparison, patient utterances were rendered either with or without seven BAPS-predicted fields: emotional state, facial expression, gaze direction, head movement, body posture, hand gesture, and intensity. Although BAPS predicts ten dimensions overall, voice emotion, speech rate, and emotion trajectory were not separately specified in the rendering prompts. The comparison condition without BAPS nonverbal fields still allowed spontaneous, natural expressions and movements. We sampled two turns from each of the 13 test clinical cases, resulting in 26 turns. Experimental Results: Four raters evaluated the rendered stimuli across five dimensions adapted from the MaSP instrument: Role Consistency, Interaction Complexity, Nonverbal Naturalness, Real-Patient Likeness, and Overall Quality (selected because they directly capture turn-level behavioral authenticity, whereas other global MaSP items require full-encounter context).
Across all five criteria, clips generated with the seven BAPS fields received more rater preferences than clips without them, including on Real-Patient Likeness and Overall Quality. This turn-level pilot did not separately evaluate field-by-field execution accuracy, audio or cross-frame synchronization, educational outcomes, or real-time interaction. Complete settings, generation templates, and numerical results are provided in Appendix M (Table 27).