Multimodal Behaviour Modeling for LLM-based Virtual Standardized Patients

Effective physician–patient communication is central to clinical decision-making and patient-centered care. While Standardized Patients (SPs) are widely regarded as the gold standard for communication training and assessment, they are costly and difficult to scale. Recent LLM-based SP simulators offer a scalable alterna- tive, but largely focus on verbal dialogue, overlooking nonverbal behaviors such as discomfort and hesitation that are essential to realistic patient interaction and clin- ical judgment. To bridge this gap, we introduce Patient Records with Integrated Signals across Modalities (PRISM), a multimodal SP behavior dataset covering 14 medical specialties and more than 8.6k dialogue turns, annotated along 18 be- havioral dimensions. Building on PRISM, we propose Behavior-Aware Patient Simulator (BAPS), an LLM-based simulator that generates realistic multimodal patient behaviors. BAPS is optimized by our proposed self-improving context- evolution framework that iteratively develops a structured skill library covering verbal behavior, information disclosure, and nonverbal expression. Specifically, the context evolution proceeds through iterative simulation-diagnosis-refinement cycles. Firstly, at the simulation stage, BAPS predicts multimodal patient re- sponses using the current skill library. Secondly, the diagnosis stage compares this prediction against real-SP response annotations to identify behavioral discrepan- cies and distill actionable insights. Finally, at the refinement stage, these insights are consolidated into the skill library to improve the subsequent simulations. Af- ter context evolution, the established skill library is used by BAPS at inference time to guide more behaviorally faithful patient simulation. Extensive experi- ments demonstrate that BAPS substantially outperforms existing methods across verbal quality, disclosure control, and multimodal behavioral fidelity, improving the overall multimodal behavioral fidelity score by 0.208 (a 48.6% relative gain) over the strongest baseline. Blinded human evaluation further confirms BAPS’s superiority using MaSP, a validated instrument for assessing the quality of stan- dardized patients. Together, PRISM and BAPS provide a scalable foundation for building virtual SPs that more faithfully reproduce the multimodal behaviors of real patients.