Align-Taper: Aligning VLMs with Test-Time Personalization for Fine-Grained Multimodal Emotion Recognition

Video-based emotion recognition (ER) is inherently multimodal, as subtle affective states are expressed through subject-specific combinations of facial, vocal, and language cues. Test-time adaptation (TTA) methods have been successfully employed to personalize vision–language models (VLMs) during inference for facial ER. However, state-of-the-art VLMs mainly rely on visual–textual alignment and often overlook complementary information from other modalities. To address this limitation, we propose Align-Taper-FT, an efficient multimodal approach that augments visual semantics based on action units (AUs) with an acoustic cluster dictionary. Audio is incorporated as an additional modality, providing complementary acoustic cues for fine-grained ER. Specifically, source-domain acoustic representations are clustered into an audio dictionary, and a Similarity-to-Probability Dictionary (SPD) module maps cluster similarities directly to class probabilities without requiring a trainable audio classifier. These probabilities are fused with AU-guided visual predictions during source pretraining. A generic source-trained model is often insufficient for unseen target subjects because expressive behavior and modality reliability vary across individuals. To address this, we introduce Align-Taper-TTA, a test-time personalization method that selects the visual window whose prediction is both confident and consistent with the audio class probabilities. The synchronized audio segment is then employed with the selected visual window to adapt AU prompts and a lightweight subject-specific correction parameter. Experiments on StressID and BAH datasets show that Align-Taper achieves state-of-the-art performance on CLIP and TTA methods for ER. Results further show stronger subject-wise personalization across target subjects. Our code is included in the supplementary materials and will be released upon publication.