acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Behaviour Modeling for LLM-based Virtual Standardized Patients

Abstract

Effective physician–patient communication is central to clinical decision-making and patient-centered care. While Standardized Patients (SPs) are widely regarded as the gold standard for communication training and assessment, they are costly and difficult to scale. Recent LLM-based SP simulators offer a scalable alterna- tive, but largely focus on verbal dialogue, overlooking nonverbal behaviors such as discomfort and hesitation that are essential to realistic patient interaction and clin- ical judgment. To bridge this gap, we introduce Patient Records with Integrated Signals across Modalities (PRISM), a multimodal SP behavior dataset covering 14 medical specialties and more than 8.6k dialogue turns, annotated along 18 be- havioral dimensions. Building on PRISM, we propose Behavior-Aware Patient Simulator (BAPS), an LLM-based simulator that generates realistic multimodal patient behaviors. BAPS is optimized by our proposed self-improving context- evolution framework that iteratively develops a structured skill library covering verbal behavior, information disclosure, and nonverbal expression. Specifically, the context evolution proceeds through iterative simulation-diagnosis-refinement cycles. Firstly, at the simulation stage, BAPS predicts multimodal patient re- sponses using the current skill library. Secondly, the diagnosis stage compares this prediction against real-SP response annotations to identify behavioral discrepan- cies and distill actionable insights. Finally, at the refinement stage, these insights are consolidated into the skill library to improve the subsequent simulations. Af- ter context evolution, the established skill library is used by BAPS at inference time to guide more behaviorally faithful patient simulation. Extensive experi- ments demonstrate that BAPS substantially outperforms existing methods across verbal quality, disclosure control, and multimodal behavioral fidelity, improving the overall multimodal behavioral fidelity score by 0.208 (a 48.6% relative gain) over the strongest baseline. Blinded human evaluation further confirms BAPS’s superiority using MaSP, a validated instrument for assessing the quality of stan- dardized patients. Together, PRISM and BAPS provide a scalable foundation for building virtual SPs that more faithfully reproduce the multimodal behaviors of real patients.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Multimodal Learning, LLM Agents, Context Evolution, Medical AI, Patient Simulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cLFeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Patient Records with Integrated Signals across Modalities (PRISM), a dataset built from standardized-patient (SP) videos for modeling and evaluating the consistency between simulated patients and human SPs. This paper also proposes a new patient simulation method, Behavior-Aware Patient Simulator (BAPS), which uses an external skill library that is continuously updated during optimization and fixed during inference to align verbal, nonverbal, and information disclosure behaviors with those of human SPs. The experimental results show that BAPS outperforms the other simulation methods compared in this paper on the PRISM dataset.

Critical strengths
  1. The PRISM dataset provides aligned multimodal data and behavior annotations from human standardized patients, which is useful for modeling and evaluating patient behavior beyond verbal responses.
  2. The BAPS method is effective without updating model parameters. Its skill library improves patient simulation and can be reused across different backbone models.
Critical weaknesses
  1. Although the authors claim that their dataset and method cover both verbal and nonverbal dimensions, the model still represents nonverbal behaviors through textual descriptions. The evaluation does not include actual animation, speech, or their synchronization. The results support more appropriate behavioral decisions, but the evidence for the interaction realism of a complete virtual patient and its practical educational value is still limited.
  2. The experiments show that BAPS outperforms the compared patient simulation methods, but there is no comparison with general prompt or context optimization methods such as ACE and GEPA. Therefore, it is still unclear how much of the improvement comes from the proposed skill library structure and update mechanism. I suggest adding these comparisons using the same training data to further clarify the methodological contribution.
Questions for authors
  1. Lines 854–857: Why are the four disclosure states assigned scores of −1.0, 0.3, 1.0, and −0.2? Does BAPS still outperform the baselines under other reasonable scoring weights?
  2. The human evaluation questions are adapted from MaSP into pairwise comparisons. What evidence supports the validity and reliability of this adapted evaluation protocol? Were the questions reviewed by experts, and was agreement among the three annotators measured?
  3. Line 493: placeholder [LLM name] ; line 501: placeholder [N].
  4. The authors could add case studies and error analysis to show where BAPS improves patient simulation and where it still fails.

I would be willing to raise my rating if the authors adequately address these concerns.

5 comments

We sincerely thank Reviewer cLFe for the careful evaluation of our manuscript, the encouraging assessment, and the constructive comments. We particularly appreciate the reviewer’s recognition of PRISM’s aligned multimodal behavioral annotations and BAPS’s parameter-free, cross-backbone transferable skill library. Your insightful suggestions regarding audiovisual interaction realism, fair prompt-optimization baselines, evaluation protocol validation, and qualitative error analysis have guided extensive new experiments and revisions that substantially strengthened our manuscript. We address each comment in detail below.

W1: Although the authors claim that their dataset and method cover both verbal and nonverbal dimensions, the model still represents nonverbal behaviors through textual descriptions. The evaluation does not include actual animation, speech, or their synchronization. The results support more appropriate behavioral decisions, but the evidence for the interaction realism of a complete virtual patient and its practical educational value is still limited.

A1: Thank you for this comment. To complement our existing evaluation of dialogue and structured behavioral decisions, we added a blinded pilot in which raters evaluated short audiovisual clips generated with Kling 4.0, extending our assessment to perceived patient portrayal quality in rendered video.

In this paired comparison, patient utterances were rendered either with or without seven BAPS-predicted fields: emotional state, facial expression, gaze direction, head movement, body posture, hand gesture, and intensity. Although BAPS predicts ten dimensions overall, voice emotion, speech rate, and emotion trajectory were not separately specified in the rendering prompts. The comparison condition without BAPS nonverbal fields still allowed spontaneous, natural expressions and movements. We sampled two turns from each of the 13 test clinical cases, resulting in 26 turns. Experimental Results: Four raters evaluated the rendered stimuli across five dimensions adapted from the MaSP instrument: Role Consistency, Interaction Complexity, Nonverbal Naturalness, Real-Patient Likeness, and Overall Quality (selected because they directly capture turn-level behavioral authenticity, whereas other global MaSP items require full-encounter context).

Adapted MaSP Evaluation CriteriaVerbal + NonverbalVerbal-only BaselineTie
Role Consistency26.0% (27)16.3% (17)57.7% (60)
Interaction Complexity46.2% (48)29.8% (31)24.0% (25)
Nonverbal Naturalness40.4% (42)33.7% (35)26.0% (27)
Real-Patient Likeness43.3% (45)26.9% (28)29.8% (31)
Overall Quality46.2% (48)30.8% (32)23.1% (24)

Across all five criteria, clips generated with the seven BAPS fields received more rater preferences than clips without them, including on Real-Patient Likeness and Overall Quality. This turn-level pilot did not separately evaluate field-by-field execution accuracy, audio or cross-frame synchronization, educational outcomes, or real-time interaction. Complete settings, generation templates, and numerical results are provided in Appendix M (Table 27).

W2: The experiments show that BAPS outperforms the compared patient simulation methods, but there is no comparison with general prompt or context optimization methods such as ACE and GEPA. Therefore, it is still unclear how much of the improvement comes from the proposed skill library structure and update mechanism. I suggest adding these comparisons using the same training data to further clarify the methodological contribution.

A2: Thank you for this valuable suggestion. We address this concern through an additional baseline comparison and a clarification of BAPS's methodological contributions.

Additional baseline results show that BAPS has higher mean scores than ACE on all reported verbal, nonverbal, and disclosure metrics. Since our method is a context evolution method, we would prioritize ACE as a more related baseline compared to the prompt optimization method GEPA. Hence, we have added ACE as a baseline using the same PRISM training set, backbone LLM, and evaluation protocol as BAPS. As shown in the table below, BAPS consistently outperforms ACE across all dimensions:

Method (runs)SBERT↑BLEU1↑BLEU4↑ROUGEL↑
BAPS0.4231 ± 0.00350.1322 ± 0.00340.0306 ± 0.00140.2525 ± 0.0053
ACE0.4111 ± 0.00290.1257 ± 0.00120.0297 ± 0.00040.2492 ± 0.0014
MethodEmotion micro-F1↑Emotion Macro-F1↑Facial micro-F1↑Facial Macro-F1↑Gaze↑Head↑Body↑Hand↑Voice↑Speech↑Traj.↑Int. MAE↓
BAPS0.6564 ± 0.00740.3489 ± 0.02690.4491 ± 0.00920.3662 ± 0.00950.8392 ± 0.00130.4193 ± 0.01860.8182 ± 0.00840.4476 ± 0.01880.7584 ± 0.00250.7955 ± 0.01650.9199 ± 0.00750.7076 ± 0.0111
ACE0.5944 ± 0.00310.2998 ± 0.00410.4007 ± 0.00670.3119 ± 0.00750.7212 ± 0.00520.4106 ± 0.00840.7143 ± 0.00530.4119 ± 0.00980.6341 ± 0.00500.6505 ± 0.01070.9000 ± 0.00470.7336 ± 0.0072
MethodLeak rate↓Disclosure score↑
BAPS0.0384 ± 0.00330.2757 ± 0.0068
ACE0.0392 ± 0.00180.2703 ± 0.0026

We have incorporated the ACE comparison results in Table 2 of the revised manuscript.

Moreover, we would also like to clarify our methodological contribution:

  1. Interactive Patient World Modeling (vs. Static Task-Solving Memory): ACE is designed for single-outcome, static reasoning or tool-use tasks, where evolution consists of accumulating factual heuristics to maximize task completion. In contrast, BAPS builds an interactive patient world model to serve as a high-fidelity training simulator for medical learners in realistic clinical consultations. Rather than optimizing a single scalar answer, a patient world model must govern multi-dimensional behavioral state transitions. Specifically, it regulates how a patient dynamically discloses clinical facts while maintaining coherent verbal tone and nonverbal language across nuanced doctor–patient interactions. This paradigm shift requires a fundamentally different learning architecture rather than surface-level prompt tuning.
  2. Modality-Decoupled Credit Assignment (vs. Global Scalar Feedback): Because a clinical patient world model must coordinate simultaneous verbal, disclosure, and nonverbal behaviors, relying on a single holistic score (as in ACE) causes credit assignment collapse and cross-modal interference (e.g., penalizing an accurate factual disclosure because a nonverbal cue was slightly off). Instead, BAPS separates evaluation and rule updates across the three behavioral dimensions, enabling dimension-specific error diagnosis and targeted refinement without cross-channel interference.
  3. Inductive Abstraction & Active Pruning (vs. Additive Accumulation): While ACE performs embedding-based merging, its updates remain accumulative: it merely deduplicates near-identical text while continually appending fragmented rules, expanding the library to 424 entries and introducing retrieval noise. In contrast, BAPS performs inductive abstraction to synthesize shared patterns across instances into compact, transferable principles that replace redundant entries, alongside active pruning to purge rules that trigger downstream errors. This compresses the library by ~4x (converging to 122 rules vs. ACE's 424 on the same data) while achieving superior performance across all metrics, demonstrating that BAPS effectively eliminates redundant information to distill higher-quality rules.

Q1: Lines 854–857: Why are the four disclosure states assigned scores of −1.0, 0.3, 1.0, and −0.2? Does BAPS still outperform the baselines under other reasonable scoring weights?

A1: Thank you for this question. The four weights in the disclosure score were grounded directly in clinical-pedagogical principles for standardized patient training rather than tuned on data. To verify that our findings do not depend on these exact values, we parameterized the metric and conducted a sensitivity analysis across 23 weight configurations, confirming that BAPS outperforms all baselines across all reasonable settings.

1. State Definitions, Parameterized Formulation, and Value Ranges At each turn, every tracked clinical fact falls into one of four mutually exclusive states depending on whether the clinician has queried the fact and whether the model mentions it:

Clinical Query StatusMentioned by ModelNot Mentioned by Model
Not yet queriedLeak (L, weight -c, default -1.0)Withheld (W, weight +b, default +0.3)
Queried this turnRevealed (R, weight +a, default +1.0)Missed (M, weight -d, default -0.2)

Using non-negative weight magnitudes a, b, c, d \in [0, 1], the parameterized Disclosure Score is defined as:

S(a,b,c,d)=aR+bW−cL−dMR+W+L+MS(a,b,c,d) = \frac{aR + bW - cL - dM}{R + W + L + M}

where S(a,b,c,d)∈[−1.0,+1.0]. Our default setting (a,b,c,d)=(1.0,0.3,1.0,0.2)enforces the pedagogical ordering −c<−d<0<+b<+a: • Equal Unit Boundaries (a=1.0,c=1.0): Because scaling all weights by a positive constant preserves model rankings, we anchor the primary reward and penalty at the unit boundary 1.0within [0,1]. We assign equal magnitude to Revealed (a=1.0) and Leak (c=1.0) because accurately providing queried information and preventing unqueried spoilers are equally critical to standardized patient training. • Modest Restraint Reward (0<b=0.3<a): In multi-turn history taking, clinicians query only one or two facts per turn while most case facts remain unqueried; assigning a modest +0.3to Withheld rewards restraint without allowing an always-silent simulator to achieve a high score. • Milder Omission Penalty (0<d=0.2<c): Unlike an irreversible leak, a Missed fact receives a milder −0.2penalty because clinicians can still elicit omitted details via follow-up questioning, avoiding incentives for premature over-disclosure. 2. Sensitivity Analysis Across 23 Weight Configurations Holding model predictions fixed on the human-annotated test set, we varied one weight at a time across {0,0.25,0.5,0.75,1.0}while holding the other three at their defaults (for example, testing (a,0.3,1.0,0.2)for a∈{0,0.25,0.5,0.75}). This yields 4+5+4+5=18non-default variants plus the default setting itself, totaling 19 unique one-factor configurations. We also included 4 single-term degenerate endpoints where three weights are set to zero (such as (1,0,0,0)), giving 23 configurations in total. Below, we report the mean score and Rank-1 Win Rate (the proportion of configurations in which a method achieves the highest mean score):

Evaluation MetricBAPSPWPPatientSimEasyMED
Default Score (1.0, 0.3, 1.0, 0.2)0.2757 ± 0.00680.2382 ± 0.00890.2382 ± 0.00380.2649 ± 0.0040
Mean Score (19 One-Factor Sweeps)0.32000.28490.28470.3099
Rank-1 Win Rate (19 One-Factor Sweeps)19 / 19 (100%)0 / 19 (0%)0 / 19 (0%)0 / 19 (0%)
Rank-1 Win Rate (All 23 Configurations)21 / 23 (91.3%)2 / 23 (8.7%)0 / 23 (0%)0 / 23 (0%)

BAPS ranks strictly first across all 19/19 one-factor sweeps (by an average margin of +0.0109 over the strongest baseline) as well as on Withheld-only (0,1,0,0) and Leak-only (0,0,1,0), trailing PWP by 0.0019 only on Revealed-only (1,0,0,0) and Missed-only (0,0,0,1). Full per-configuration results are provided in Appendix N (Table 28).

Reviewer_ogUGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the limitations of recent LLM-based Standardized Patient (SP) simulators that primarily focus on verbal dialogue, the authors introduce Patient Records with Integrated Signals across Modalities (PRISM), a multimodal SP behavior dataset. PRISM was constructed by using Qwen3-Omni to annotate verbal and nonverbal behaviors from videos, audio, and transcripts of trained SP interactions, followed by human validation on a subset of the annotations. Building on PRISM, the authors propose the Behavior-Aware Patient Simulator (BAPS), an LLM-based simulator designed to generate more realistic multimodal patient behaviors. The key feature of BAPS is its context-evolution framework, which iteratively refines a Skill Library consisting of three components: verbal behavior, information disclosure, and nonverbal expression. The authors evaluate BAPS against three existing LLM-based patient simulators, using real SP responses and behavioral annotations as references. The results show that BAPS achieves higher behavioral fidelity than the baselines, improves through context evolution, and benefits from each skill set primarily in its corresponding behavioral dimension. Human evaluation also shows a preference for BAPS over the strongest baseline, EasyMED.

Critical strengths
  1. The paper addresses an important gap in LLM-based standardized patient simulation. Whereas many existing simulators primarily generate verbal dialogue, communication training also requires attention to how patients express themselves through facial, gestural, postural, and vocal behavior. PRISM and BAPS bring these dimensions into dataset construction and patient-response generation, providing a potentially useful foundation for future embodied SP systems. Although end-to-end realism remains to be demonstrated, the focus on behavioral expression beyond dialogue content is relevant to the practical goals of clinical communication training.

  2. The evaluation examines BAPS from several complementary perspectives. Beyond comparisons with existing simulators, the skill-set ablations provide evidence that verbal, disclosure, and nonverbal guidance contribute to their corresponding performance domains. The authors also examine performance over context evolution, test transfer of the evolved library to another backbone, and complement automatic reference-based metrics with blinded human preference evaluation. This breadth of evaluation helps characterize the contribution and transferability of the learned guidance and provides converging evidence for improvements within the evaluated setting.

Critical weaknesses
  1. The validity of the reference-based evaluation is insufficiently established. PRISM’s automated behavioral annotations serve as both supervision for context evolution and references for evaluation, yet human agreement is only 26.5% for emotion and 45.5% for facial expression. Model agreement is calculated only on human-consensus cases, leaving annotation reliability on disputed cases unresolved. Similarly, the disclosure evaluation treats the recorded SP’s disclosure timing as the standard of appropriateness without establishing whether it reflects explicit scenario instructions, educational objectives, or discretionary actor behavior. A different but plausible disclosure pattern may therefore be penalized as an error. Together, these issues make it difficult to determine whether the reported gains reflect more realistic and appropriate patient behavior or closer imitation of uncertain reference labels and scenario-specific trajectories. This uncertainty directly affects the interpretation of the paper’s central behavioral-fidelity claims.

  2. The methodological novelty of the proposed context-evolution framework is insufficiently established. The paper presents this framework as a standalone contribution, yet its simulation–diagnosis–refinement cycle closely parallels ACE’s generation–reflection–curation process. The structured rule library, helpful/harmful counters, and embedding-based consolidation also have clear precedents in ACE. BAPS introduces patient-specific supervision and a verbal/disclosure/nonverbal decomposition, but the manuscript does not establish which of these adaptations constitutes a substantive methodological advance. The library-removal ablations and evolution curve demonstrate the utility of accumulated guidance, but do not distinguish the proposed mechanism from existing context-engineering approaches or simpler guidance extracted from the same training data. Without a clearer technical distinction and a controlled comparison, the evidence supports a task-specific adaptation more strongly than a new context-evolution method, limiting the paper’s claimed methodological contribution.

Questions for authors
  1. Clarification—annotation coverage and reliability. The manuscript repeatedly reports 18 behavioral annotation dimensions, whereas Appendix D lists 10. Please enumerate all 18 dimensions and explain their relationship to the published schema. For the human validation, please also report the consensus-subset size for each dimension and clarify whether disputed cases were adjudicated or otherwise evaluated. How does the reported model agreement support the reliability of annotations outside the consensus subset?

  2. Clarification—disclosure ground truth and scoring. How are information items and their appropriate disclosure points determined? Are the judgments grounded in explicit SP scenario instructions and educational objectives, or inferred from the recorded disclosure sequence? How are clinically plausible spontaneous disclosures distinguished from leaks? Please justify the reward weights, particularly −1.0 for leaks and −0.2 for missed disclosures. A sensitivity analysis showing whether method rankings remain stable under alternative weights would help establish the robustness of the reported advantage.

  3. Clarification—baseline adaptation and inputs. Please explain how the same underlying PRISM case information was mapped into PWP’s disclosure grid, PatientSim’s clinical profile and persona axes, and EasyMED’s intent-conditioned fact selection. How were persona and disclosure settings initialized? Did all methods receive the same physician queries and follow the same history protocol, including reference versus generated histories and truncation? Identical final prompts are unnecessary, since information filtering is part of the methods, but equivalent source information and faithful adaptations need to be documented.

  4. Clarification—context and dependence in human evaluation. Were the 49 excerpts evaluated independently or in consultation order, and what preceding dialogue or case context accompanied each excerpt? In particular, how could evaluators assess disclosure appropriateness when relevant information might have appeared before the excerpt? Please also clarify whether uncertainty estimates account for excerpts being nested within 13 consultations and for repeated ratings by the same annotators. Providing the evaluation instructions and a sample evaluation screen would help resolve these concerns.

  5. Clarification—scope of the realism claim. Section 7 appropriately acknowledges that the human evaluation concerns text and structured behavioral descriptions rather than rendered audiovisual behavior. Should the main claim therefore be interpreted as improved behavioral decision quality rather than demonstrated embodied multimodal realism? Please align the abstract and conclusions with the evaluated scope. An embodied evaluation would extend the evidence, but the immediate issue is whether the current claims accurately reflect the existing evaluation.

11 comments

We sincerely thank Reviewer ogUG for the careful evaluation of our manuscript and for providing such thoughtful, constructive comments. We particularly appreciate the reviewer’s recognition of our effort to incorporate multimodal behavioral expression into SP simulation and of our multi-perspective evaluation design, while your insightful guidance on benchmark reliability, methodological novelty, protocol transparency, and claim scope alignment has led to comprehensive new experiments and revisions that greatly strengthened the paper. We address each comment in detail below.


W1.1: The validity of the reference-based evaluation is insufficiently established. PRISM’s automated behavioral annotations serve as both supervision for context evolution and references for evaluation, yet human agreement is only 26.5% for emotion and 45.5% for facial expression. Model agreement is calculated only on human-consensus cases, leaving annotation reliability on disputed cases unresolved.

A1.1: Thank you for raising this point. To properly address it, evaluation no longer uses the automated annotations: we newly annotated the full test set (479 turns) with two independent human annotators. Any disagreements were resolved through joint discussion among the two annotators and a third reviewer until a consensus was reached. The revised result table is shown below (and shown in Table 2 in the revised manuscript):

MethodSBERT↑BLEU1↑BLEU4↑ROUGEL↑
BAPS0.4231 ± 0.00350.1322 ± 0.00340.0306 ± 0.00140.2525 ± 0.0053
PWP0.3716 ± 0.00260.0922 ± 0.00230.0180 ± 0.00150.1779 ± 0.0060
PatientSim0.3421 ± 0.00340.0901 ± 0.00190.0180 ± 0.00090.1747 ± 0.0019
Easy MED0.3820 ± 0.00230.1123 ± 0.00100.0225 ± 0.00040.2050 ± 0.0015
MethodEmotion Micro-F1Emotion Macro-F1Facial Micro-F1Facial Macro-F1Gaze accuracyHead accuracyBody accuracyHand accuracyVoice accuracySpeech-rate accuracyTrajectory accuracyIntensity MAE↓
BAPS0.6564 ± 0.00740.3489 ± 0.02690.4491 ± 0.00920.3662 ± 0.00950.8392 ± 0.00130.4193 ± 0.01860.8182 ± 0.00840.4476 ± 0.01880.7584 ± 0.00250.7955 ± 0.01650.9199 ± 0.00750.7076 ± 0.0111
PWP0.3801 ± 0.01080.2548 ± 0.01040.2258 ± 0.01940.2428 ± 0.01420.5553 ± 0.01420.2649 ± 0.00620.1550 ± 0.01200.2000 ± 0.00880.2873 ± 0.02840.4523 ± 0.04570.8210 ± 0.01071.6125 ± 0.0427
PatientSim0.3292 ± 0.00570.2357 ± 0.00910.2352 ± 0.01160.2395 ± 0.01320.6336 ± 0.01960.3164 ± 0.01530.2026 ± 0.00790.2296 ± 0.01270.2541 ± 0.01620.3883 ± 0.01520.7807 ± 0.01671.9504 ± 0.0208
Easy MED0.4209 ± 0.00470.2634 ± 0.00330.2442 ± 0.01030.2620 ± 0.00810.6168 ± 0.00860.3013 ± 0.01180.2287 ± 0.00740.2234 ± 0.00520.3393 ± 0.01290.5974 ± 0.01010.8086 ± 0.01251.4918 ± 0.0131
MethodLeak rate↓Disclosure score↑
BAPS0.0384 ± 0.00330.2757 ± 0.0068
PWP0.0711 ± 0.00650.2382 ± 0.0089
PatientSim0.0666 ± 0.00300.2382 ± 0.0038
Easy MED0.0445 ± 0.00250.2649 ± 0.0040

Details of the annotation procedure and results are in Appendix G (lines 1180–1187 and 1210–1235) and Table 2, respectively.

AuthorsOpenReview ↗

W1.2, W1.3

W1.2: Similarly, the disclosure evaluation treats the recorded SP’s disclosure timing as the standard of appropriateness without establishing whether it reflects explicit scenario instructions, educational objectives, or discretionary actor behavior.

(+Q2.1 Clarification—disclosure ground truth and scoring. How are information items and their appropriate disclosure points determined? Are the judgments grounded in explicit SP scenario instructions and educational objectives, or inferred from the recorded disclosure sequence? )

A1.2: Thank you for this question. We infer the reference disclosure points from the recorded dialogues because the original production scripts are unavailable. For each scored clinical item, we use its first appearance in the dialogue as the operational reference point. We explain below how we identify these items and how source provenance and clinical disclosure conventions inform this evaluation.

1. Clarification on Ground-Truth Extraction (Q2.1): Because our dataset is curated from publicly available medical education videos where production scripts are not published, we construct the structured case profiles with ground-truth disclosure points directly from the recorded dialogues using a standardized clinical intake template(Jefferson Center for Connected Care, n.d.). Specifically:

  • Determining Information Items: We distinguish between general conversational background (excluded from disclosure scoring) and clinically significant findings (e.g., drug allergies, specific symptom triggers, and key medical history) guided by prior SP studies (Baig et al., 2014; Casabianca et al., 2015).
  • Determining Disclosure Points: For each targeted clinical item, its first appearance turn in the dialogue is recorded as the ground-truth disclosure point.

2. Institutional Source Provenance and Disclosure Conventions (W1.2): We document institutional source provenance and explain the clinical rationale for the disclosure convention below:

  • Institutional Source Verification: We classify videos as authorized if they were published directly by institutions meeting our source-qualification criteria or reposted by third-party channels with traceable original sources from such institutions. This term indicates institutional source provenance rather than copyright or redistribution permission. Under this definition, we classify 102/115 videos (88.70%) as authorized, including uploads from Ali Elkharashi, medRebel, and Simulation Training Videos—which remain classified as reposting channels—based on their confirmed institutional origins. This source information describes the production context of the recordings. The disclosure metric uses the recorded sequence as an operational reference, as explained above.
  • Trigger-Driven Clinical Boundaries: In standardized SP demonstrations, key clinically significant findings (such as drug allergies) follow a conditional disclosure rule that they are revealed only when the clinician initiates an appropriate diagnostic inquiry (Jefferson Center for Connected Care, n.d.).

Please refer to Table 12 in Appendix H for more detailed video source information.


W1.3: A different but plausible disclosure pattern may therefore be penalized as an error.

A1.3: Thank you for raising this concern. Distinguishing clinically plausible, spontaneous disclosures from premature leaks is an inherently challenging issue, as there is no universally established boundary between them (Fussell et al., 2009; Siminoff et al., 2011).

To mitigate this ambiguity, beyond the automatic scoring reported in Table 2 of the manuscript, we also conducted human evaluations using a Disclosure Appropriateness criterion informed by the established Maastricht Assessment of Simulated Patients (MaSP) (shown in Figure 3b in the manuscript). This assesses whether information is disclosed appropriately in context, rather than requiring an exact turn-level match with the recorded SP's timing. Human evaluators preferred BAPS over Easy MED in 63.4% of comparisons, compared with 24.1% for Easy MED (12.4% ties).

This complementary evaluation provides evidence of BAPS's disclosure quality beyond adherence to a single recorded disclosure trajectory.

W1.4: Together, these issues make it difficult to determine whether the reported gains reflect more realistic and appropriate patient behavior or closer imitation of uncertain reference labels and scenario-specific trajectories. This uncertainty directly affects the interpretation of the paper’s central behavioral-fidelity claims.

A1.4: Thank you for raising this overarching concern. To address it, we have strengthened our evaluation and calibrated our framing across three steps:

  1. Eliminating Reliance on Uncertain Qwen3-omni annotated Labels (W1.1): By re-annotating the entire test set (N=479 turns) with independent human annotators and consensus adjudication, we demonstrate that BAPS maintains consistent improvements over all baselines on verified human ground truth.
  2. Clarifying the Disclosure Reference (W1.2): We use each scored clinical item's first appearance in the recorded dialogue as the operational reference for comparison. Other disclosure timings may also be clinically reasonable, as discussed in Section 7 (lines 517–519). Institutional provenance describes the videos' production context; the disclosure evaluation measures agreement with the recorded reference.
  3. Validating Behavioral Appropriateness Beyond Exact Trajectory Matching (W1.3): Through our MaSP-informed human evaluation, where raters preferred BAPS over EasyMED in 63.4% of comparisons for Disclosure Appropriateness (vs. 24.1%), these ratings favor BAPS over EasyMED in perceived disclosure appropriateness. Together, these analyses support improved agreement with the human-annotated reference and favorable perceived disclosure appropriateness within this benchmark. Section 7 (lines 517–519) explains the remaining limitation: the recorded SP's first disclosure is an operational reference, and other clinically reasonable or educationally useful disclosure timings may receive a leak penalty.
Reviewer_qdB2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses LLM-based virtual standardized patients (SPs). Its central argument is that existing patient simulators primarily model what a patient says, while paying much less attention to how the patient behaves across multiple modalities, such as facial expression, gaze, gesture, posture, and vocal affect. The paper makes two main contributions: PRISM, a multimodal dataset of standardized-patient consultations, and BAPS, an LLM-based patient simulator that models both verbal and nonverbal behavior.

  1. PRISM: A Multimodal Dataset of Standardized-Patient Consultations

PRISM is constructed from publicly available recordings of standardized-patient consultations, in which SPs are typically trained actors portraying patients. The sources are primarily YouTube videos, including OSCE teaching and training recordings.

The dataset is constructed through the following pipeline:

  • More than 1,200 candidate videos are collected.
  • An LLM pre-screens the videos for complete doctor–SP consultations, after which human reviewers verify consultation completeness and audiovisual quality. This process yields 115 consultations across 14 specialties, containing 8,623 turns and 17.4 hours of recordings.
  • Video, audio, and transcripts are time-aligned, and the consultations are segmented into doctor and patient turns.
  • A multimodal model, Qwen3-Omni, annotates each patient turn with nonverbal behavioral attributes, including emotion, facial expression, gaze, head movement, body posture, hand gesture, vocal emotion, speech rate, emotion trajectory, and emotion intensity. These correspond to ten dimensions explicitly listed in the annotation description, although the paper states that 18 dimensions are used.
  • An LLM extracts a structured clinical case profile from the full transcript of each consultation.

To assess annotation quality, the authors manually evaluate 411 clips with two human annotators. Agreement between the model-generated labels and human annotations exceeds 70% on seven of the ten listed dimensions, while agreement on the remaining three dimensions is approximately 51–56%.

  1. BAPS: An LLM-Based Multimodal Patient Simulator

The second contribution is BAPS, an LLM-based standardized-patient simulator. Given a structured case profile, the dialogue history, and the doctor’s current question, BAPS generates the patient’s verbal response, determines what clinical information should be disclosed, and predicts the patient’s nonverbal behavior.

BAPS is guided by a skill library consisting of short natural-language rules governing three aspects of patient behavior: verbal response generation, information disclosure, and nonverbal behavior.

The skill library is learned from the 90 PRISM training consultations through an iterative procedure that the authors call context evolution: 1 BAPS generates a simulated patient turn using the current skill library. 2 An LLM compares the simulated response with the corresponding real SP turn and extracts generalizable lessons from their differences. 3 Each lesson is either added as a new rule or merged with a similar existing rule. Rules that repeatedly reduce performance are removed. Here, Only the natural-language skill library evolves. Once the evolution process is complete, the library is frozen for evaluation.

The model is evaluated on 13 held-out consultations, with predictions compared turn by turn against the recorded standardized patients. The evaluation covers three aspects:1 Verbal similarity: SBERT similarity, BLEU, and ROUGE-L are used to compare generated and recorded patient responses. 2 Information disclosure: leak rate and disclosure score measure whether the simulated patient reveals appropriate information at the appropriate time. 3 Nonverbal behavior: accuracy, F1 score, and mean absolute error are computed against the Qwen3-Omni-generated behavioral labels.

BAPS is compared with three recent patient simulators: PWP, PatientSim, and EasyMED. To make the comparison more controlled, all baselines are adapted to the same nonverbal output format and run using the same backbone model, DeepSeek-V4-Flash. Under this setup, BAPS achieves the best performance on every reported metric.

The authors also conduct several additional evaluations: 1 Ablation studies that remove individual groups of rules or the entire skill library. 2 Evolution analysis showing validation-set performance across successive context-evolution steps. 3 Transfer evaluation, in which the learned skill library is applied to another model, GPT-6 Astra, without additional evolution. 4 Blinded human evaluation against EasyMED, involving three annotators, 49 excerpts, and eight criteria adapted from MaSP. Across all judgments, BAPS is preferred in 68% of cases, compared with EasyMED.

Overall, the paper’s main contribution is to extend virtual standardized-patient simulation beyond dialogue generation. Rather than modeling only the semantic content of patient responses, PRISM and BAPS attempt to capture the joint dynamics of what a patient says, what information the patient chooses to reveal, and how the patient expresses it nonverbally.

Critical strengths
  1. An important, under-studied problem, addressed with a clear and scalable data pipeline. Most LLM-based SP simulators are judged only on what the patient responds to the doctor’s queries. This paper also asks what information the patient reveals and how the patient behaves nonverbally, and checks these against recorded SP. The PRISM pipeline is clearly described and largely automated, so it could scale to more videos and specialties. Turn-aligned SP video with behavior labels across specialties could potentially be an effective way to get data for building the virtual patients simulator. I also appreciate that the authors openly report human agreement for each dimension.

  2. A practical and inspectable method. The skill library is built without retraining the model, is written in readable natural language, and transferred to a second backbone without further evolution. In principle, it can be re-evolved on new data and inspected or edited by educators, which fits medical-education use well. Under the authors' protocol, the gains are consistent, and human raters preferred BAPS over the strongest baseline on all criteria.

Critical weaknesses
  1. What is measured is two steps removed from what is claimed. The abstract states that BAPS "generates realistic multimodal patient behaviors" (l.20–21) and that PRISM and BAPS support "virtual SPs that more faithfully reproduce the multimodal behaviors of real patients" (l.36–38). The motivation also builds on studies of real patient encounters (l.52–53). However, the data are SP portrayals in public teaching videos, and the ethics statement itself says PRISM contains "simulated clinical encounters … rather than real patient consultations" (l.511–512). Aligning to SPs is a reasonable target for a virtual SP, but SP portrayals are not the same as real patient behavior. They may be scripted or exaggerated for teaching, and their authenticity was not assessed; instruments such as MaSP exist precisely to assess how authentically SPs portray patients. This gap is not discussed as a limitation.

The second step is that the nonverbal "ground truth" is not the SPs' behavior itself but Qwen3-Omni's labels of it, and labels from the same annotator are used both to evolve the skill library and to score it. The validation of these labels is limited:

  • two annotators on about 10% of clips;
  • raw percent agreement, without chance correction or label distributions;
  • only 51–56% agreement on emotion, posture and head movement (Appendix C also notes a bias toward "Still" for head movement).

I also noticed signs that the gains come largely from learning how often each label occurs, rather than behaviors:

  • Gaze (0.692) and hand-gesture (0.794) accuracies are identical across two different backbones.
  • Micro-F1 gains are much larger than macro-F1 gains (facial expression: +0.209 vs. +0.024; with GPT-6 Astra, the emotion macro-F1 even decreases). Because micro-F1 is dominated by frequent labels while macro-F1 weights every label equally, a large micro-F1 gain with little macro-F1 gain means the improvement is concentrated on frequent labels, with little evidence of better prediction of rare but clinically important cues such as Tearful or Grimacing.
  • All baselines score near chance on body posture (0.12–0.14), while BAPS reaches 0.69. No majority-label baseline is reported. As it stands, the evidence supports closer agreement with Qwen3-Omni's labels of recorded SP turns. The author should connect the claim to realistic patient behavior.
  1. The experimental design does not isolate the benefit of BAPS or test it with a fair baseline BAPS is evolved on 90 PRISM training consultations, while PWP, PatientSim and EasyMED never see PRISM data. They only receive a nonverbal output format added to prompts they were not designed for. So it is not clear to me whether BAPS has a real advantage, or whether any simulator given the same data would improve in the same way. Table 3 even suggests that part of the gain comes from BAPS's own base prompt: with an empty skill library, it already exceeds EasyMED. This matters because the context-evolution loop closely follows ACE (Zhang et al., 2026b), including per-rule helpful/harmful counters and incremental merging of similar rules. The paper's specific contribution therefore depends on showing a benefit over alternatives that use the same data.

The evaluation also does not reflect how a virtual SP is used:

  • Nonverbal behavior is only shown as text labels, with no rendering and no evidence that it helps learners. More fundamentally, the paper assumes rather than shows that this form of nonverbal output makes a better training simulator. I agree that patients' nonverbal cues matter in clinical communication. However, BAPS outputs one categorical label per channel per turn, and these are judged by exact agreement with a single SP's machine-derived label. Its practical meaning for virtual patient training is unclear.
  • The human study does not report the raters' background or agreement, compares only against EasyMED, and has no real-SP reference. It therefore shows a preference between two systems, not closeness to real SP behavior.
Questions for authors

Clarifications and analyses that could change my assessment:

  1. Label frequencies. Please report the label distribution of each nonverbal dimension in the test set, a baseline that always predicts the most frequent label, macro-F1 or balanced accuracy for every dimension, and per-label F1 for emotion and facial expression. Why are the gaze and hand-gesture accuracies identical for DeepSeek-V4-Flash and GPT-6 Astra? Please also include the final skill library.
  2. Human labels. Appendix C's 411 human-checked clips were sampled from every video, so some come from the 13 test consultations. If all methods are re-scored on those test clips using the human-consensus labels instead of Qwen3-Omni's, does BAPS keep its advantage? This matters most for head movement, body posture and emotion, where human and model labels disagree most and where BAPS's gains are largest. Ideally, a human-labelled test set would provide a more reliable reference.
  3. Fair comparison. If the baselines get comparable access to PRISM's training data (for example, label frequencies or retrieved training examples in their prompts, or ACE run on the same data), does BAPS still lead? Please also provide each baseline's extended prompt, and report the baselines on GPT-6 Astra.
  4. Turing-style test. Can raters tell BAPS apart from the real SP? Adding the real SP turn as a reference condition would test the faithfulness claim directly. Please also report the raters' background and clinical expertise, inter-rater agreement, and a significance test.
  5. Scope and data. Will you scope the claims to SP behavior, or justify using SP portrayals as a proxy for real patients? How was it verified that the people in the videos are trained SPs? Are training and test consultations separated by YouTube channel and actor?
  6. Why do the "BAPS (full)" values in Table 3 differ from Table 2?

Suggestions for future work (not affecting my score):

  1. Evaluate full simulated conversations in which BAPS builds on its own earlier answers, ideally with rendered avatars and real learners. It would also be valuable to test whether disclosure responds to the doctor's communication (for example, empathic vs. curt versions of the same doctor turn), and whether the rules adapt to patient traits such as personality or health literacy rather than producing one "average" patient.
17 comments

We sincerely thank Reviewer qdB2 for the careful evaluation of our manuscript and for providing so many thoughtful, invaluable comments. We particularly appreciate the reviewer’s recognition of our effort to move virtual SP evaluation beyond verbal responses to encompass information disclosure and nonverbal behavior through a scalable data pipeline and a transferable skill library, while also highlighting the need for a more cautious scoping of our claims and more rigorous ground-truth, baseline, and human evaluations.

Below, we provide detailed responses to each the comments:


W1.1: What is measured is two steps removed from what is claimed. The abstract states that BAPS "generates realistic multimodal patient behaviors" (l.20–21) and that PRISM and BAPS support "virtual SPs that more faithfully reproduce the multimodal behaviors of real patients" (l.36–38). The motivation also builds on studies of real patient encounters (l.52–53). However, the data are SP portrayals in public teaching videos, and the ethics statement itself says PRISM contains "simulated clinical encounters … rather than real patient consultations" (l.511–512). Aligning to SPs is a reasonable target for a virtual SP, but SP portrayals are not the same as real patient behavior. They may be scripted or exaggerated for teaching, and their authenticity was not assessed; instruments such as MaSP exist precisely to assess how authentically SPs portray patients. This gap is not discussed as a limitation.

A1.1: Thank you for this careful reading. We apologize that our original wording was misleading. We have recalibrated our claims accordingly. Specifically, we revised three sections:

  1. Abstract (l.20–21 in the revised manuscript**).** "an LLM-based simulator that generates realistic multimodal patient behaviors" → "an LLM-based simulator that generates multimodal behaviors modeled on human SP portrayals."
  2. Abstract (revised manuscript, lines 38–39). "virtual SPs that more faithfully reproduce the multimodal behaviors of real patients" → "virtual SPs that more faithfully reproduce the multimodal behavioral decisions of human SPs”.
  3. Introduction (revised manuscript, lines 53–54 and 58–60). "Real patient encounters are inherently multimodal" → "in real clinical encounters, patients communicate through more than words", followed by the added sentence after explaining why behavioral cues are important for real patient encounters: "These behavioral cues are hence also central to training: recognizing and responding to them is a core communication skill that SP-based training aims to develop, yet a verbal-only simulator gives learners no cues to practice on."

Regarding the authenticity of the SP portrayals in PRISM, we verified the institutional provenance of all recordings to ensure they reflect professional medical-education standards. Defining authorized videos as those published directly by qualified medical or educational institutions (or reposted with a traceable original source from such institutions), we confirm 102 authorized videos out of our 115-video inventory (88.70%). All 13 remaining videos with unclear institutional authorization were placed exclusively in the training split so that our evaluation set consists entirely of verified institutional recordings, with full source records detailed in Table 12 in Appendix H.

AuthorsOpenReview ↗

W1.2, W1.3


W1.2:
The second step is that the nonverbal "ground truth" is not the SPs' behavior itself but Qwen3-Omni's labels of it, and labels from the same annotator are used both to evolve the skill library and to score it. The validation of these labels is limited:

  • two annotators on about 10% of clips;
  • raw percent agreement, without chance correction or label distributions;
  • only 51–56% agreement on emotion, posture and head movement (Appendix C also notes a bias toward "Still" for head movement).

A1.2: We thank the reviewer for the question and concern. To properly address this problem, following established multi-turn dialogue annotation practices (e.g., Li et al., 2017), we re-annotated the entire test set (479 turns) using two independent human annotators. Any disagreements were resolved through joint discussion among the two annotators and a third reviewer until a consensus was reached.

The evaluation results shown below (also shown in the revised Table 2 in the manuscript) indicate that BAPS maintains its leading performance.

MethodSBERT↑BLEU1↑BLEU4↑ROUGEL↑Leak rate↓Disclosure score↑
BAPS0.4231 ± 0.00350.1322 ± 0.00340.0306 ± 0.00140.2525 ± 0.00530.0384 ± 0.00330.2757 ± 0.0068
PWP0.3716 ± 0.00260.0922 ± 0.00230.0180 ± 0.00150.1779 ± 0.00600.0711 ± 0.00650.2382 ± 0.0089
PatientSim0.3421 ± 0.00340.0901 ± 0.00190.0180 ± 0.00090.1747 ± 0.00190.0666 ± 0.00300.2382 ± 0.0038
Easy MED0.3820 ± 0.00230.1123 ± 0.00100.0225 ± 0.00040.2050 ± 0.00150.0445 ± 0.00250.2649 ± 0.0040
MethodEmotion Micro-F1 ↑Emotion Macro-F1 ↑Facial Micro-F1 ↑Facial Macro-F1 ↑Gaze accuracy ↑Head accuracy ↑Body accuracy ↑Hand accuracy ↑Voice accuracy ↑Speech-rate accuracy ↑Trajectory accuracy ↑Intensity MAE↓
BAPS0.6564 ± 0.00740.3489 ± 0.02690.4491 ± 0.00920.3662 ± 0.00950.8392 ± 0.00130.4193 ± 0.01860.8182 ± 0.00840.4476 ± 0.01880.7584 ± 0.00250.7955 ± 0.01650.9199 ± 0.00750.7076 ± 0.0111
PWP0.3801 ± 0.01080.2548 ± 0.01040.2258 ± 0.01940.2428 ± 0.01420.5553 ± 0.01420.2649 ± 0.00620.1550 ± 0.01200.2000 ± 0.00880.2873 ± 0.02840.4523 ± 0.04570.8210 ± 0.01071.6125 ± 0.0427
PatientSim0.3292 ± 0.00570.2357 ± 0.00910.2352 ± 0.01160.2395 ± 0.01320.6336 ± 0.01960.3164 ± 0.01530.2026 ± 0.00790.2296 ± 0.01270.2541 ± 0.01620.3883 ± 0.01520.7807 ± 0.01671.9504 ± 0.0208
Easy MED0.4209 ± 0.00470.2634 ± 0.00330.2442 ± 0.01030.2620 ± 0.00810.6168 ± 0.00860.3013 ± 0.01180.2287 ± 0.00740.2234 ± 0.00520.3393 ± 0.01290.5974 ± 0.01010.8086 ± 0.01251.4918 ± 0.0131

Details of the annotation procedure are in Appendix G.

Unless otherwise specified, all the experiments below are based on the human re-annotated test set.


W1.3: I also noticed signs that the gains come largely from learning how often each label occurs, rather than behaviors:

  • Gaze (0.692) and hand-gesture (0.794) accuracies are identical across two different backbones.

A1.3: Thank you for this question. The reported identical accuracies (in the original manuscript Table 2) arise from two distinct reasons, neither of which indicates that the gains come largely from just learning how often each label occurs.

  1. Regarding gaze, the reported accuracies across two different backbones (DeepSeek-V4-Flash and GPT-6 Astra) appear identical due to averaging and rounding to three decimal places (DeepSeek-V4-Flash: 0.6917 ± 0.0036, GPT-6 Astra: 0.6923 ± 0.0029), where the underlying predictions differ between models. Furthermore, both of them outperform the majority-class baseline (0.6528), which further indicates that their performance is not solely driven by predicting the most frequent label.
  2. For hand gesture, a label-processing error caused the valid "None" category to be mistakenly excluded from evaluation, leading to misleading accuracy values. We have corrected this issue and recomputed the results. The updated accuracies are 0.4476±0.0188 on the DeepSeek-V4-Flash backbone and 0.4874±0.0020 on the GPT-6 Astra backbone, respectively. We appreciate the reviewer for pointing this out so that we had a chance to correct it.

W1.4:

  • Micro-F1 gains are much larger than macro-F1 gains (facial expression: +0.209 vs. +0.024; with GPT-6 Astra, the emotion macro-F1 even decreases). Because micro-F1 is dominated by frequent labels while macro-F1 weights every label equally, a large micro-F1 gain with little macro-F1 gain means the improvement is concentrated on frequent labels, with little evidence of better prediction of rare but clinically important cues such as Tearful or Grimacing.

A1.4: Thank you for highlighting this concern. To provide a more comprehensive evaluation, we report macro-F1 for all behavioral dimensions and per-label F1 for emotion and facial expression, also as suggested in part of Q1.1 (”Please report …, macro-F1 or balanced accuracy for every dimension, and per-label F1 for emotion and facial expression. “).

In the result tables shown below, BAPS achieves the highest Macro-F1 across all ten dimensions among the three existing methods. Regarding per-label F1, BAPS achieves the highest mean F1 for all eight emotion labels and six out of seven facial-expression labels. While PWP performs slightly higher on the facial label "Confused" (0.0259 vs. 0.0000) compared to the other three methods, this category contains only two positive samples (GT n=2), limiting the reliability of that specific comparison. To further address this long-tail challenge in future work, we will expand our dataset to enrich rare clinical cues, and we have added this in Section 7, Limitations and Future Work (lines 508–510), of the revised manuscript.

Table 16. Macro-F1 for nonverbal dimensions

Methodemotions (GT n=461)facial_expressions (GT n=452)gaze_direction (GT n=452)head_movement (GT n=450)body_posture (GT n=453)hand_gesture (GT n=385)voice_emotion (GT n=458)speech_rate (GT n=463)emotion_trajectory (GT n=466)intensity_score (GT n=464)
BAPS0.3489 ± 0.02690.3662 ± 0.00950.3237 ± 0.00850.2889 ± 0.01470.3436 ± 0.02030.2358 ± 0.03120.3280 ± 0.04280.5376 ± 0.02330.4824 ± 0.03900.1634 ± 0.0121
PWP0.2548 ± 0.01040.2428 ± 0.01420.2443 ± 0.00470.1647 ± 0.01430.1602 ± 0.01240.1986 ± 0.01500.2436 ± 0.02330.3088 ± 0.03100.4270 ± 0.00460.0141 ± 0.0047
PatientSim0.2357 ± 0.00910.2395 ± 0.01320.2575 ± 0.01060.1814 ± 0.00600.1750 ± 0.01020.1920 ± 0.00380.1982 ± 0.01100.2907 ± 0.00960.3912 ± 0.01690.0108 ± 0.0022
EasyMED0.2634 ± 0.00330.2620 ± 0.00810.2598 ± 0.00460.2237 ± 0.01160.1987 ± 0.00420.1998 ± 0.00330.2647 ± 0.01250.3908 ± 0.00890.4221 ± 0.01850.0215 ± 0.0055

Table 17. Per-label F1 for emotion

MethodNeutral (GT n=387)Anxious (GT n=54)Fearful (GT n=9)Painful (GT n=73)Hesitant (GT n=27)Confused (GT n=11)Relieved (GT n=8)Trusting (GT n=15)
BAPS0.9005 ± 0.00540.2521 ± 0.03720.2451 ± 0.00850.6360 ± 0.01020.1462 ± 0.01490.0673 ± 0.05920.2826 ± 0.06680.2609 ± 0.1291
PWP0.7079 ± 0.01720.2477 ± 0.01150.0996 ± 0.02230.5646 ± 0.03260.1137 ± 0.01150.0641 ± 0.04440.1157 ± 0.01690.1255 ± 0.0267
PatientSim0.6137 ± 0.01170.2224 ± 0.01180.0625 ± 0.01220.6198 ± 0.04730.1106 ± 0.00360.0472 ± 0.02230.0745 ± 0.02830.1348 ± 0.0302
Easy MED0.7706 ± 0.00580.2516 ± 0.00460.1327 ± 0.01540.5505 ± 0.01030.1227 ± 0.00790.0365 ± 0.02080.1104 ± 0.01450.1322 ± 0.0073

Table 18. Per-label F1 for facial expression.

MethodFrowning_Pain (GT n=103)Grimacing (GT n=34)Guarded_Eyes (GT n=3)Tearful (GT n=19)Relaxed (GT n=181)Confused (GT n=2)Blank (GT n=196)
BAPS0.4355 ± 0.00810.4042 ± 0.04670.0290 ± 0.05020.7962 ± 0.01680.5146 ± 0.01580.0000 ± 0.00000.3842 ± 0.0183
PWP0.3722 ± 0.02090.3334 ± 0.04500.0112 ± 0.00510.5166 ± 0.02460.2932 ± 0.04360.0259 ± 0.03650.1469 ± 0.0381
PatientSim0.4077 ± 0.01950.3861 ± 0.05970.0170 ± 0.00280.4576 ± 0.02170.2906 ± 0.02050.0000 ± 0.00000.1174 ± 0.0254
Easy MED0.3666 ± 0.01110.3812 ± 0.02830.0142 ± 0.00040.6134 ± 0.04100.3514 ± 0.01570.0000 ± 0.00000.1070 ± 0.0132

We have added the per-dimension Macro-F1 results (Table 16) and the per-label F1 breakdowns for emotion and facial expressions (Tables 17–18) to Appendix I of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.