acceptodds
Withdrawn from ICLR 2027

Align-Taper: Aligning VLMs with Test-Time Personalization for Fine-Grained Multimodal Emotion Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video-based emotion recognition (ER) is inherently multimodal, as subtle affective states are expressed through subject-specific combinations of facial, vocal, and language cues. Test-time adaptation (TTA) methods have been successfully employed to personalize vision–language models (VLMs) during inference for facial ER. However, state-of-the-art VLMs mainly rely on visual–textual alignment and often overlook complementary information from other modalities. To address this limitation, we propose Align-Taper-FT, an efficient multimodal approach that augments visual semantics based on action units (AUs) with an acoustic cluster dictionary. Audio is incorporated as an additional modality, providing complementary acoustic cues for fine-grained ER. Specifically, source-domain acoustic representations are clustered into an audio dictionary, and a Similarity-to-Probability Dictionary (SPD) module maps cluster similarities directly to class probabilities without requiring a trainable audio classifier. These probabilities are fused with AU-guided visual predictions during source pretraining. A generic source-trained model is often insufficient for unseen target subjects because expressive behavior and modality reliability vary across individuals. To address this, we introduce Align-Taper-TTA, a test-time personalization method that selects the visual window whose prediction is both confident and consistent with the audio class probabilities. The synchronized audio segment is then employed with the selected visual window to adapt AU prompts and a lightweight subject-specific correction parameter. Experiments on StressID and BAH datasets show that Align-Taper achieves state-of-the-art performance on CLIP and TTA methods for ER. Results further show stronger subject-wise personalization across target subjects. Our code is included in the supplementary materials and will be released upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fine-Grained Emotion Recognition, Test-Time Adaptation, Vision-Language Models, Multimodal Learning, Affective Computing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RbL7OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Action Unit Prompting for Fine-Grained Video and audio Emotion Recognition which is an interesting topic and has good impact in real time applications. Authors evaluated their proposed methodology on two benchmark datasets and showed that adding audio data along with visual entropy and audio-guided JS divergence improved the model accuracy.

Critical strengths

None

Critical weaknesses

In the literature, spectrogram-based features are useful for emotion recognition as compared to manually extracted features like OpenSMILE. Authors should include spectrogram as opposed to OpenSMILE for comparison.

I think, this study need s systematic experimental evaluations and as such it is at preliminary stage.

It would be really useful if authors adopt this method for emotion recognition on benchmark datasets like IEMOCAP, MELD, CMU-MOSEI datasets.

Coming to creating a audio dictionary using a clustering method can’t be generalized due to fixed dictionary. During training, dictionary may have different audio segments with different words. But during test time, this can be changed completely.

authors should include ablation study to show that adding audio dictionary clustering along with other modules actually improved the performance.

Questions for authors

In the abstract: What does CLIP means? Authors should define when they use the term for the first time. Authors are selective in citing the papers in section 2. There are many papers on multimodal emotion recognition? Why did authors cited very few papers?

Figure 1 needs more information. For example, how many clusters? How authors choose the number of clusters?

T3AL Liberatori et al. (2024) (CVPR’24) reported results on THUMOS14 and the ActivityNet-v1.3. How authors compared these results with different datasets StressID and BAH in the paper.

Work reported in this paper takes the core idea from “ Test-Time Zero-Shot Temporal Action Localization” and did incremental analysis on “CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition”. However, there is no ablation study to show that adding audio cluster dictionary actually improved the accuracy of the proposed model.

Also, authors should include all the datasets when they compare with the SOTA methods.

In the literature, spectrogram-based features are useful for emotion recognition as compared to manually extracted features like OpenSMILE. Authors should include spectrogram as opposed to OpenSMILE for comparison.

I think, this study need s systematic experimental evaluations and as such it is at preliminary stage.

It would be really useful if authors adopt this method for emotion recognition on benchmark datasets like IEMOCAP, MELD, CMU-MOSEI datasets.

Coming to creating a audio dictionary using a clustering method can’t be generalized due to fixed dictionary. During training, dictionary may have different audio segments with different words. But during test time, this can be changed completely.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We address each comment below, where “W” denotes a weakness, “Q” a question, and “A” our response.

"Q1: In the abstract: What does CLIP mean and selective in citing the papers in section 2."

"A-1": We have defined CLIP as Contrastive Language–Image Pre-training at its first occurrence in the abstract.

Regarding the related work, we would like to clarify that our research focuses on fine-grained video-based emotion recognition, particularly subtle affective states such as stress and ambivalence, rather than conventional multimodal emotion recognition involving basic emotional categories. Accordingly, Section 2 prioritizes studies most closely related to fine-grained video-based ER, personalization, and test-time adaptation. While the broader multimodal emotion recognition literature is extensive, not all of these studies directly address the specific challenges considered in our work. We would appreciate it if the reviewer could identify specific relevant papers that they believe are missing, so that we can assess their relevance and incorporate them where appropriate.

"Q2: Figure 1 needs more information and Clusters."

"A-2": We would like to clarify that the number of acoustic clusters is not predefined. We employ HDBSCAN, a density-based clustering algorithm that automatically discovers clusters based on the distribution of acoustic features, without requiring a fixed number of clusters. We have revised Figure 1 (main manuscript) to make the clustering process and the automatic determination of the number of clusters clearer. Furthermore, HDBSCAN depends on two parameters: min_cluster_size and min_samples. To select the best values for both parameters, we did an ablation study mentioned in Supplementary B.3 and Figure 5.

"Q3: T3AL, How authors compared these results with different datasets"

"A-3": To the best of our knowledge, relatively few methods address video-based test-time personalization using vision–language models such as CLIP, particularly for subtle expression recognition. We therefore included T3AL [1] as a relevant video-based test-time adaptation baseline. We did not directly compare its originally reported results on THUMOS14 or ActivityNet-v1.3 with our results. Instead, we applied the T3AL approach to StressID and BAH and evaluated it on the same datasets as our proposed method to provide a consistent comparison. We will clarify this in the revised manuscript to avoid any misunderstanding.

"Q4 and W5: Core idea from T3AL, incremental analysis on CLIP-AUTT, and no ablation study"

"A-4": We clarify that T3AL [1] is included as a video-based TTA baseline, rather than serving as the foundation of our proposed method. While ALIGN-TAPER extends CLIP-AUTT [2], it introduces three methodological contributions: (i) an HDBSCAN-based acoustic dictionary to capture fine-grained vocal patterns, (ii) audiovisual consistency-guided informative-window selection, and (iii) subject-specific adaptive fusion during test-time personalization.

To provide stronger experimental evidence, we have extended our comparisons to include additional visual-audio TTA methods, using feature concatenation to combine audio with visual into existing approaches using the same OpenSMILE backbone. As shown in Table R1, these results highlight that the improvement comes not merely from incorporating audio, but from our source-constructed acoustic dictionary, which captures fine-grained acoustic patterns to guide informative-window selection and subject-specific audiovisual adaptation. Importantly, the improvements in F1-score suggest more balanced recognition across emotion classes, particularly in challenging scenarios where subtle affective cues and class imbalance make reliable predictions difficult.

StressIDBAH
Method (Visual+Audio)WARF1WARF1
T3AL [1]77.461.965.345.7
CRG [3]79.567.170.044.3
CLIP-AUTT [2]80.978.270.242.0
ALIGN-TAPER-TTA81.979.472.770.2

Table R1. Comparison of audiovisual TTA methods on StressID and BAH.

We added comparisons (Sec. 4.1, Tables 1–2; Supp. Sec. C, Tables 9–10) and component ablations (Supp. Sec. D, Tables 6–7) to highlight ALIGN-TAPER's contributions beyond CLIP-AUTT.

References:

[1] Liberatori, Benedetta, et al. "Test-time zero-shot temporal action localization." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2] Zeeshan, Muhammad Osama, et al. "Clip-autt: Test-time personalization with action unit prompting for fine-grained video emotion recognition." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[3] Zhai, Haotian, et al. "Mitigating cache noise in test-time adaptation for large vision-language models." 2025 IEEE International Conference on Multimedia and Expo. IEEE, 2025.

AuthorsOpenReview ↗

Next Response

"Q5: Should include all the datasets when they compare with the SOTA methods."

"A-5": We would like to clarify that the problem addressed in this work is video-based test-time personalization for fine-grained expression recognition, particularly subtle affective states such as stress and ambivalence. This setting differs from conventional emotion recognition, which primarily focuses on basic emotional categories. To the best of our knowledge, relatively few existing methods specifically address this problem. We therefore compare our approach with relevant state-of-the-art expression recognition and test-time adaptation methods, adapting them to StressID and BAH under the same evaluation protocol. These datasets are directly aligned with the problem considered in our work, whereas the original benchmarks of some competing methods address different tasks. Thus, our comparisons focus on the relevance of the methods and consistency of the evaluation setting rather than including datasets unrelated to fine-grained expression recognition.

"Q6 and W1: Comparison with spectrogram-based features."

"A-6": OpenSMILE extracts interpretable acoustic descriptors directly from the audio signal, including pitch variations, loudness, jitter, shimmer, spectral characteristics, and temporal patterns such as pauses and voiced/unvoiced durations. While spectrogram-based representations also preserve rich acoustic information, these subtle characteristics may require additional feature learning to be explicitly represented. Our approach instead uses compact, predefined acoustic descriptors to construct a lightweight dictionary without training a separate audio encoder.

To further investigate the effectiveness of these representations, we conducted additional experiments on StressID using spectrogram-based features and EMO-TTA [1], an audio-language-based test-time adaptation method built on a CLIP-like architecture. To focus on the contribution of acoustic information, these comparisons consider only audio and text modalities, excluding visual information.

As shown in Table R2, our approach achieves higher performance, highlighting the benefit of clustering OpenSMILE descriptors into representative acoustic patterns rather than relying solely on spectrogram-derived or pretrained audio-language embeddings. These patterns provide complementary information for recognizing subtle emotional variations across subjects. Since the compared approaches also differ in architecture, the results support the effectiveness of our overall acoustic representation strategy rather than isolating the feature extractor alone.

MethodWARF1
Spectrogram-based42.235.9
EMO-TTA [2]47.942.1
Ours (Audio-only)71.566.2

Table R2. Comparison of audio-only recognition methods on StressID, averaged across 10 target subjects.

We have also incorporated these experiments into Section 4.2 and Table 4 of the main manuscript.

References:

[1] Shi, Jiacheng, et al. "Emo-tta: Improving test-time adaptation of audio-language models for speech emotion recognition." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

AuthorsOpenReview ↗

Next Response

"Q7 & W2: Need s systematic experimental evaluations."

"A-7": We would like to clarify that our study includes systematic evaluations of the proposed ALIGN-TAPER method across multiple datasets that account for individual variability and different emotion recognition tasks, comparisons with existing baseline methods, and ablation studies of individual components. In the revised manuscript, we have further expanded the experimental analysis to provide a more comprehensive assessment. Specifically:

  • Comparison with existing methods: We have expanded our evaluation by including additional visual-audio baselines, extending CLIP-AU [1] and Exp-CLIP [2] for source training, and CLIP-AUTT [1], T3AL [3], and CRG [4] for test-time adaptation. For a fair comparison, we incorporate acoustic features into these methods using the same feature-concatenation strategy. We have also included MCP [5], a recent TTA method, as an additional baseline. The updated results are presented in Tables 1 and 2, including subject-wise comparisons.

  • Acoustic dictionary evaluation: We conducted standalone audio-based comparisons against spectrogram-based representations and EMO-TTA [6], reported in Table 4, to examine the effectiveness of our acoustic dictionary.

  • Component-level ablations: Beyond the existing analysis of fusion strategies, AU adaptation, and window selection, we have added experiments evaluating audiovisual consistency-guided window selection (visual entropy vs. entropy + JS divergence), subject-specific adaptive fusion (fixed vs. adaptive weights), and acoustic dictionary construction (single class-level prototypes vs. HDBSCAN clustering).

  • Robustness analysis: We further evaluated both source fine-tuning and test-time adaptation across five random seeds, reporting mean performance, standard deviation, and subject-wise results. These results are included in Section D of the supplementary material.

Together, these experiments evaluate the individual contributions of our proposed components, their effectiveness against alternative approaches, and the consistency of the results across target subjects and random initializations. We believe these additional analyses substantially strengthen the experimental validation of ALIGN-TAPER. We would also appreciate it if the reviewer could identify any specific experiments or evaluation aspects they consider missing, so that we can address those concerns more directly.

References:

[1] Zeeshan, Muhammad Osama, et al. "Clip-autt: Test-time personalization with action unit prompting for fine-grained video emotion recognition." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[2] Zhao, Zengqun, et al. “Enhancing zero-shot facial expression recognition by LLM knowledge transfer.” 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) . IEEE, 2025.

[3] Liberatori, Benedetta, et al. "Test-time zero-shot temporal action localization." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[4] Zhai, Haotian, et al. "Mitigating cache noise in test-time adaptation for large vision-language models." 2025 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2025.

[5] Chen, Xinyu, et al. "Multi-cache enhanced prototype learning for test-time generalization of vision-language models." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[6] Shi, Jiacheng, et al. "Emo-tta: Improving test-time adaptation of audio-language models for speech emotion recognition." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

AuthorsOpenReview ↗

Next Response

"Q8 & W3: Adopt for other datasets."

"A-8": Our main focus is video-based test-time personalization for fine-grained expression recognition, particularly subtle affective state. Therefore, we selected StressID and BAH, which are directly aligned with our research objectives. Nevertheless, we acknowledge the value of evaluating the proposed method on more general multimodal emotion recognition benchmarks, such as IEMOCAP, MELD, and CMU-MOSEI, to further investigate its generalization across different emotion recognition tasks. We will consider extending our evaluation to these datasets in future work.

"Q9 & W3: Audio dictionary using a clustering method can’t be generalized"

"A-9":: We would like to clarify that the objective of our acoustic dictionary is not to recognize spoken words or their semantic content, but rather to capture variations in how individuals speak that may be associated with subtle affective states. Specifically, we extract 88-dimensional OpenSMILE eGeMAPS features that characterize acoustic patterns, including pitch (F0), loudness, voice instability (jitter and shimmer), spectral characteristics (spectral slope, MFCCs, and spectral flux), and temporal patterns such as voiced/unvoiced durations and pauses (as shown in Table R3). These descriptors capture vocal delivery and acoustic variation rather than explicitly representing word identities.

Feature GroupAcoustic Descriptors
PitchF0, jitter, pitch percentiles, slopes
Voice qualityShimmer, harmonics-to-noise ratio, harmonic differences
SpectralAlpha ratio, Hammarberg index, spectral slopes, MFCCs, spectral flux
FormantsFormant frequencies, bandwidths, relative energies
LoudnessLoudness, variation, peaks, sound level
TemporalVoiced/unvoiced durations, voiced segments per second

Table R3. OpenSMILE eGeMAPS acoustic features (88-D)

The complete breakdown of the 88 acoustic parameters is provided in the supplementary material.

The acoustic dictionary is constructed from these speech-related descriptors rather than lexical information. Therefore, changes in the words spoken at test time do not inherently invalidate the learned acoustic representations. While variations in speech content and recording conditions may influence acoustic features, our approach focuses on capturing acoustic patterns potentially transferable across different utterances, rather than relying on a fixed vocabulary.

Reviewer_Vf3VOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript presents Align-TAPER, a multimodal framework for fine-grained emotion recognition with test-time personalization. The method combines AU-guided visual representations with an acoustic dictionary constructed using OpenSMILE features and HDBSCAN clustering. Audiovisual consistency guides informative-window selection, while AU prompts and subject-specific fusion weights are adapted using entropy minimization. Experiments on StressID and BAH demonstrate improvements over existing CLIP-based and test-time adaptation methods.

Critical strengths

The paper addresses a relevant challenge in multimodal emotion recognition by incorporating complementary acoustic information into subject-specific adaptation. The acoustic dictionary offers an efficient alternative to training additional audio classifiers. The method achieves strong results on StressID and BAH, with particularly substantial F1 improvements on BAH. The fusion comparisons, component ablations, and subject-wise evaluations provide useful empirical evidence.

Critical weaknesses
  1. The framework builds upon CLIP-AUTT (Zeeshan et al., ECCV 2026), extending AU-guided test-time adaptation with an acoustic dictionary, audiovisual consistency guidance, and subject-specific fusion. Although these extensions are relevant, their individual contributions are not sufficiently demonstrated. Comparisons with simpler class-level acoustic prototypes and more targeted evaluations of the added multimodal components would help clarify their methodological value.
  2. Table 3 evaluates AU prompt tuning and window selection but does not independently isolate audiovisual consistency guidance and subject-specific fusion-weight adaptation. In particular, comparisons between visual-entropy-only and audiovisual JS-divergence-based selection, as well as fixed versus adaptive fusion weights, are missing. These experiments are important for determining which mechanisms drive the reported improvements.
  3. In Section 3.3, the modality fusion weight is optimized through prediction-entropy minimization. For fixed modality predictions, entropy is concave with respect to the fusion coefficient, potentially favoring extreme modality weights rather than reliable audiovisual balancing. The paper does not analyze the learned weights or their behavior under conflicting modality predictions. Such analysis would better substantiate the effectiveness of subject-specific multimodal adaptation.
  4. The improvements over CLIP-AUTT on StressID are relatively small (1.1 WAR and 1.5 F1 points), while Appendix C shows subject-level performance regressions. Experiments are reported using a single fixed seed without uncertainty estimates. Furthermore, Appendix B.3 evaluates HDBSCAN configurations on target subjects, but it is unclear whether the final hyperparameters were selected exclusively using source-domain validation. Clarifying this protocol and reporting performance variability would strengthen the experimental conclusions
Questions for authors
  1. What is the methodological novelty over CLIP-AUTT, and how much does each new component contribute?
  2. How do audiovisual consistency guidance and adaptive fusion weights independently affect performance?
  3. Does entropy minimization favor extreme fusion weights, particularly under conflicting modality predictions?
  4. Are the improvements statistically significant, and were hyperparameters selected using only source-domain data?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank Reviewer Vf3V for the careful review and for recognizing the relevance of our research problem and the methodological contributions of ALIGN-TAPER. We also appreciate the reviewer acknowledging the strong experimental results on StressID and BAH, as well as the comprehensive fusion comparisons, component ablations, and subject-wise evaluations. We address each comment below, where “W” denotes a weakness, “Q” a question, and “A” our response.

"W-1 & Q1: Methodological novelty over CLIP-AUTT"

"A-1": The key contribution of this work is to effectively incorporate audio information into video-based test-time personalization for fine-grained expression recognition. Audio provides important complementary cues for recognizing subtle affective states, which may not be clearly distinguishable from facial expressions alone. However, effectively leveraging acoustic information while accounting for subject-specific variability remains challenging. To address this, our method introduces three main methodological contributions:

  • We construct an acoustic dictionary using OpenSMILE descriptors and HDBSCAN clustering to capture fine-grained acoustic patterns associated with subtle affective states. To further evaluate its effectiveness, we conducted standalone audio-based experiments, excluding visual information, and compared our approach with alternative acoustic representation methods, as shown in Table R1. The results demonstrate the benefits of our proposed acoustic dictionary. We have also incorporated these experiments into Section 4.2 and Table 4 of the main manuscript.
  • Audiovisual consistency-guided informative-window selection. We leverage visual entropy and Jensen–Shannon (JS) divergence to identify informative temporal windows based on prediction uncertainty and audiovisual consistency.
  • Subject-specific adaptive fusion. We dynamically adjust the contributions of visual and acoustic modalities during test-time personalization to account for individual differences in emotional expression.
MethodWARF1
Spectrogram-based42.235.9
EMO-TTA [1]47.942.1
Ours (Audio + Text)71.566.2
Table R1. Comparison of audio-only recognition methods on StressID, averaged across 10 target subjects.

To further demonstrate the contribution of audio clustering, we conducted an additional experiment using class-level acoustic prototypes. Instead of HDBSCAN clustering, we construct a single prototype for each emotion class by averaging the corresponding source-domain OpenSMILE feature vectors using the same preprocessing and normalization procedure. All other components and experimental settings remain unchanged.

Audio DictionaryWARF1
ALIGN-TAPER-FT
Class-level prototypes73.367.8
HDBSCAN dictionary (Ours)76.070.4
Improvement+2.7+2.6
ALIGN-TAPER-TTA
Class-level prototypes76.275.0
HDBSCAN dictionary (Ours)81.979.4
Improvement+5.7+4.4

Table R2. Comparison of class-level single acoustic prototypes and the proposed HDBSCAN dictionary on StressID, averaged across 10 target subjects.

In Table R2, the results demonstrate that the proposed HDBSCAN-based dictionary consistently outperforms simpler class-level prototypes in both Align-TAPER-FT and Align-TAPER-TTA settings. These findings support the importance of capturing multiple acoustic patterns within each emotion class rather than representing an entire class with a single prototype.

Furthermore, the Component Analysis of TTA (main paper, Sec. 4.2 and Tab. 3 (Right)) evaluates the individual contributions of AU-prompt adaptation and informative-window selection. Additional analyses are provided in the supplementary material, including HDBSCAN hyperparameter sensitivity (Sec. B.3 and Fig. 5) and full-video audio guidance versus synchronized local audio (Sec. C.3 and Fig. 6). Details on the 88-dimensional OpenSMILE acoustic descriptors used to construct the dictionary (Sec. B.2 and Tab. 5). The additional ablations on audiovisual consistency guidance and fixed versus subject-adaptive fusion weights, reported in our rebuttal, and included in the supplementary material, Sec. D.3 and Tab. 7.

References:

[1] Shi, Jiacheng, et al. "Emo-tta: Improving test-time adaptation of audio-language models for speech emotion recognition." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

AuthorsOpenReview ↗

Next Response

"W-2 & Q2: How do audiovisual consistency guidance and adaptive fusion weights independently affect performance?"

"A-2": We conducted additional ablation experiments to evaluate the independent contributions of audiovisual consistency guidance and subject-specific adaptive fusion.

Window selection: In the visual-entropy-only configuration shown in Table R3, informative windows are selected solely based on the prediction entropy of the visual modality, without incorporating audiovisual JS-divergence guidance. This achieves 78.4% WAR and 72.4% F1. Incorporating audiovisual consistency guidance improves performance to 81.9% WAR and 79.4% F1, suggesting that cross-modal agreement provides complementary information for selecting informative visual windows.

Window SelectionWARF1
Visual entropy only78.472.3
Visual entropy + audiovisual JS divergence (Ours)81.979.4

Table R3. Ablation of informative-window selection on StressID, averaged across 10 target subjects.

Fusion weights: We further compare fixed equal-weight and audio-dominant fusion against our subject-specific adaptive fusion (Table R4). Equal weighting achieves 65.9% WAR and 56.3% F1, while increasing the audio contribution improves performance to 77.9% WAR and 73.5% F1. Our adaptive fusion achieves 81.9% WAR and 79.4% F1, indicating that adapting modality contributions to individual target subjects is more effective than the evaluated fixed-weight configurations.

Fusion WeightsWARF1
Fixed equal weights65.956.3
Fixed audio-dominant weights (V: 0.28, A: 0.72)77.973.5
Subject-adaptive weights (Ours)81.979.4

Table R4. Ablation of audiovisual fusion strategies.

All experiments were conducted on the StressID dataset, and the reported results are averaged across 10 target subjects. We kept all other parameters unchanged. For the window-selection experiments, we used subject-adaptive weighting in both settings, while for the three fusion-weighting experiments, we consistently used audiovisual JS divergence for window selection.

"W-3 & Q3: Does entropy minimization favor extreme fusion weights, particularly under conflicting modality predictions?”

"A-3": For fixed modality predictions, we acknowledge that prediction entropy is concave with respect to the fusion coefficient, and its minimization may favor a dominant modality, particularly when audiovisual predictions conflict. However, in our method, the subject-specific fusion weight is optimized jointly with the AU prompts during TTA, allowing the visual predictions to evolve rather than remaining fixed. Moreover, our objective is not necessarily to assign equal importance to both modalities, but to adapt their relative contributions according to the characteristics of each target subject.

To further evaluate the effectiveness of subject-specific adaptive fusion, we conducted additional experiments on StressID, with results averaged across 10 target subjects. As shown in Table R4, we compared three fusion strategies while keeping all other settings unchanged. Our subject-adaptive fusion outperforms both fixed-weight configurations, highlighting the benefit of dynamically adjusting visual and acoustic contributions to individual target subjects during test-time personalization.

The experiments were added to the supplementary material Section D.1-2 and Table 6.

AuthorsOpenReview ↗

Next Response

"W-4 & Q4: The improvements over CLIP-AUTT on StressID, fixed seed, and hyperparameters”

"A-4": We acknowledge that the improvements over CLIP-AUTT on StressID are relatively modest, reflecting the difficulty of personalized fine-grained emotion recognition, where subtle affective cues and individual variability make adaptation challenging. However, the benefits are more pronounced on the BAH dataset, which presents a more challenging, longer videos, imbalanced, in-the-wild scenario involving subtle ambivalence and hesitancy. In particular, our method achieves greater improvements in F1-score, highlighting the benefit of incorporating acoustic information when visual cues alone are insufficient. We have incorporated these additional comparisons and discussions into the revised manuscript. Section 4.1 and Table 1 (main comparison) and Table 2 (subject-wise results) in the main manuscript, along with Section C and Tables 9 & 10 in the supplementary material.

Hyperparameter selection: We clarify that all HDBSCAN configurations were evaluated exclusively on labeled source-domain data, with the final hyperparameters selected using a held-out source validation set. These parameters remain fixed during adaptation to unlabeled target subjects, and no target labels are used for hyperparameter selection. We have clarified this protocol in Supplementary Section B.3.

Performance variability: To assess robustness, we repeated both source fine-tuning and test-time adaptation using five random seeds (the original seed and four additional seeds). In Table R5, we report the mean and standard deviation across these runs, with each run evaluated over the same 10 target subjects on StressID.

The results show that ALIGN-TAPER-TTA maintains consistent performance across five random seeds, with low standard deviations in both WAR and F1. Furthermore, comparing ALIGN-TAPER-FT and ALIGN-TAPER-TTA using the same seeds confirms that test-time adaptation consistently improves both metrics, demonstrating the robustness and effectiveness of our approach.

Detailed results are included in the supplementary materials Section D.4 and Table 8.

MethodWAR ↑F1 ↑
ALIGN-TAPER-FT74.6 ± 1.269.1 ± 1.8
ALIGN-TAPER-TTA81.1 ± 0.578.9 ± 0.5

Table R5. Performance comparison of ALIGN-TAPER-FT and ALIGN-TAPER-TTA across five random seeds.

Reviewer_zzXuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ALIGN-TAPER, a multimodal test-time personalization (TTA) framework for video emotion recognition (ER). ALIGN-TAPER leverages the acoustic cues in a lightweight manner. Experiments on the StressID and BAH datasets demonstrate that ALIGN-TAPER outperforms other ER and TTA baselines.

Critical strengths
  1. This paper is easy to read.

  2. The proposed ALIGN-TAPER method introduces a lightweight acoustic cluster dictionary and a multimodal-guided informative window selection mechanism. This design incorporates the audio modality into VLM-based emotion recognition without relying on computationally heavy audio encoders, while enabling subject-specific modality weighting during test-time personalization.

  3. The method is evaluated on the StressID and the challenging in-the-wild BAH datasets. The reported experimental performance consistently outperforms existing SOTA baselines.

Critical weaknesses
  1. Incremental Novelty: The proposed ALIGN-TAPER exhibits limited novelty compared to the existing CLIP-AUTT method. The contribution is primarily incremental, building upon CLIP-AUTT by introducing an audio modality via a lightweight training scheme, incorporating audio consistency guidance into the Test-Time Adaptation (TTA) window selection mechanism, and adding a subject-specific fusion correction parameter for TTA optimization.

  2. Unfair Comparison in Main Experiments: The experimental setup does not ensure a fair comparison regarding input modalities. In Table 1 and Table 2, all state-of-the-art (SOTA) methods rely exclusively on visual and textual modalities. In contrast, ALIGN-TAPER leverages visual, textual, and audio modalities. The reported performance improvements might largely stem from the introduction of additional information (the audio modality) rather than the superiority of the proposed TTA mechanism itself.

  3. Marginal Improvement Over Naive Fusion: The authors only construct audio-inclusive baselines in the ablation study (Table 3) to justify the superiority of their dictionary-based architecture. However, the results indicate that a simple feature concatenation strategy performs only marginally worse than the proposed ALIGN-TAPER-FT (achieving 74.4% vs. 76.0% in WAR, and 69.0% vs. 70.4% in F1). This minimal performance gap suggests that the specifically designed fusion mechanism for integrating the three modalities (visual, textual, and audio) is not significantly more effective than a trivial baseline.

Questions for authors

Fair Baseline Augmentation: To ensure a fair comparison, what would be the performance if the other SOTA methods in Table 1 and Table 2 were augmented with the audio modality using a trivial or naive fusion approach (such as simply concatenating audio features)?

Generalizability of the Proposed Mechanism: Can the proposed TTA framework (particularly the dictionary-based alignment and multimodal consistency guidance) be generalized or adapted to other TTA scenarios involving different modalities or downstream tasks beyond emotion recognition?

Methodological Comparison with Recent TTA Works: How does the proposed method theoretically and methodologically compare to other recent TTA methods for emotion recognition, such as BiM-TTA (NeurIPS 2025) and EMO-TTA (ICASSP 2026)? Setting aside the differences in input modalities, what are the specific advantages and disadvantages of your TTA optimization strategy compared to these contemporary approaches?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer zzXu for the careful review and for recognizing the clarity of our paper and the methodological contributions of ALIGN-TAPER. We also appreciate the reviewer acknowledging the strong experimental performance on StressID and the challenging in-the-wild BAH dataset. We address each comment below, where “W” denotes a weakness, “Q” a question, and “A” our response.

"W-1: Incremental Novelty"

"A-1": While ALIGN-TAPER builds upon CLIP-AUTT [1], its contributions go beyond simply incorporating an audio modality. Unlike CLIP-AUTT, which relies exclusively on visual features and entropy-based window selection, our method introduces three key components that distinguish it from existing approaches:

  • Fine-grained acoustic dictionary: Rather than relying on conventional audio embeddings or a single class-level acoustic prototype, we construct an HDBSCAN-based dictionary that captures diverse vocal patterns within each emotion class. This also differs from audio-language adaptation methods such as EMO-TTA [2], which update class-conditional distributions of CLAP features.

  • Audiovisual consistency-guided window selection: Unlike CLIP-AUTT's visual entropy-based selection, we jointly consider visual uncertainty and audiovisual consistency through Jensen–Shannon divergence to identify informative windows for test-time adaptation.

  • Subject-specific adaptive fusion: Instead of using fixed modality contributions or conventional feature concatenation, our approach dynamically adjusts the relative importance of visual and acoustic evidence for each target subject.

Together, these components address complementary challenges in personalized fine-grained emotion recognition: capturing subtle acoustic variations, identifying informative audiovisual segments, and accounting for individual differences in modality relevance. Unlike conventional visual-only TTA or multimodal fusion approaches, ALIGN-TAPER integrates these mechanisms into a unified framework for subject-specific test-time personalization.

"Q-1 & W-2: Unfair Comparison in Main Experiments"

"A-2": To ensure a fair comparison, we conducted additional experiments by incorporating audio into strong and recent TTA methods, including T3AL[3], CRG[4], and CLIP-AUTT[1]. For all baselines, we use the same OpenSMILE acoustic features as our method and apply simple concatenation with visual representations, while retaining their respective TTA strategies. The choice of concatenation is supported by our fusion ablation in Table 3 (Left) in the main paper.

StressIDBAH
Method (Visual+Audio)WARF1WARF1
T3AL [3]77.461.965.345.7
CRG [4]79.567.170.044.3
CLIP-AUTT [1]80.978.270.242.0
ALIGN-TAPER-TTA81.979.472.770.2

Table R1. Comparison of audiovisual TTA methods on StressID and BAH.

As shown in Table R1, ALIGN-TAPER-TTA consistently outperforms existing audiovisual TTA methods on both datasets. Compared with audiovisual CLIP-AUTT, our method improves F1 by 1.2 points on StressID and a substantial 28.2 points on BAH. These gains highlight that the benefit comes not merely from incorporating audio, but from our source-constructed acoustic dictionary, which captures fine-grained vocal patterns and guides informative-window selection and subject-specific audiovisual adaptation. Notably, on BAH, where vocal cues are particularly relevant for recognizing subtle ambivalence and hesitancy, the substantial F1 improvement suggests better recognition of challenging affective classes. While WAR can remain relatively high when predictions favor frequent classes, the larger F1 gain suggests more balanced recognition rather than simply higher overall accuracy.

We have incorporated these additional comparisons and discussions into the revised manuscript. Section 4.1 and Table 1 (main comparison) and Table 2 (subject-wise results) in the main manuscript, along with Section C and Tables 9 & 10 in the supplementary material.

References:

[1] Zeeshan, Muhammad Osama, et al. "Clip-autt: Test-time personalization with action unit prompting for fine-grained video emotion recognition." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[2] Shi, Jiacheng, et al. "Emo-tta: Improving test-time adaptation of audio-language models for speech emotion recognition." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3] Liberatori, Benedetta, et al. "Test-time zero-shot temporal action localization." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[4] Zhai, Haotian, et al. "Mitigating cache noise in test-time adaptation for large vision-language models." 2025 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2025.

AuthorsOpenReview ↗

Next Response

"Q-2: Generalizability of the Proposed Mechanism"

"A-3": Our method primarily focuses on fine-grained expression recognition, particularly capturing subtle and nuanced changes in human affective states through complementary visual and acoustic cues. These subtle variations are often difficult to recognize from facial expressions alone, motivating the integration of multiple modalities and subject-specific personalization.

Nevertheless, the proposed approach could be extended to broader human behavior analysis and behavioral change applications, particularly in e-health. For example, analyzing conversational behaviors, such as turn-taking, interruptions, response delays, and speaking patterns, can provide valuable insights into how individuals interact, participate, and engage in social or collaborative settings. By leveraging complementary cues from facial expressions and speech, the method could support a more comprehensive analysis of human behavior in such settings.

Beyond these applications, the proposed dictionary-based representations, multimodal consistency guidance, and adaptive fusion could potentially be extended to tasks such as human activity recognition. However, such extensions would require task-specific feature representations, appropriate modality alignment, and adaptation of the prediction method. Evaluating these possibilities beyond fine-grained expression recognition remains an interesting direction for future work.

"W-3: Marginal Improvement Over Naive Fusion"

"A-4": We would like to clarify that Table 3 (Left) in the main manuscript evaluates different fusion strategies applied to the CLIP-AU backbone, rather than standalone naive fusion baselines. The objective was to investigate how different fusion techniques integrate acoustic information with the existing visual-textual representations. We recognize that this distinction was not sufficiently clear and have updated Section 4.2 and Table 3 (Left) to explicitly identify the underlying method and fusion strategy.

Furthermore, to ensure a fairer comparison, we have added results for CLIP-AU [1] and extended CLIP-AUTT [1] with audio information using the same simple feature-concatenation strategy. We have also included additional visual-audio methods T3AL [2] and CRG[3], applying the same concatenation operation to combine visual and acoustic features for a consistent comparison. These additional results are now reported in Section 4.1 Table 1 (main comparison) and Table 2 (subject-wise results), allowing us to better assess the benefits of our proposed approach beyond simply incorporating audio features.

References:

[1] Zeeshan, Muhammad Osama, et al. "Clip-autt: Test-time personalization with action unit prompting for fine-grained video emotion recognition." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[2] Liberatori, Benedetta, et al. "Test-time zero-shot temporal action localization." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[3] Zhai, Haotian, et al. "Mitigating cache noise in test-time adaptation for large vision-language models." 2025 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2025.

AuthorsOpenReview ↗

Next Response

"Q-3: Methodological Comparison with Recent TTA Works"

"A-4": The BiM-TTA [1] addresses distribution shifts and missing modalities in physiological emotion recognition through entropy-based filtering, cross-modal consistency, and selective network updates. Instead, our method focuses on fine-grained audiovisual personalization by adapting AU prompts and subject-specific fusion weights while keeping the pretrained CLIP encoders frozen. EMO-TTA [2] performs gradient-free adaptation by updating class statistics in pretrained CLAP embeddings, offering computational efficiency but relying on predefined distributional assumptions. In contrast, our method leverages a source-constructed acoustic dictionary to capture fine-grained acoustic patterns and guide audiovisual personalization through AU-prompt and fusion-weight adaptation, at the cost of additional test-time optimization.

To further investigate the effectiveness of our acoustic dictionary, we conducted additional experiments on StressID using only audio and text modalities, excluding visual information, to better examine the differences between existing audio-based approaches and our proposed acoustic representation. We compared against EMO-TTA and a spectrogram-based baseline.

MethodWARF1
Spectrogram-based42.235.9
EMO-TTA [2]47.942.1
Ours (Audio-only)71.566.2

Table R2. Comparison of audio-only recognition methods on StressID, averaged across 10 target subjects.

As shown in Table R2, our method achieves higher performance, highlighting the benefits of the source-constructed acoustic dictionary. Unlike spectrogram-based representations or EMO-TTA's adaptation of class statistics in pretrained CLAP embeddings, our dictionary uses clustering to capture diverse, fine-grained acoustic patterns from labeled source subjects. These representations provide more informative acoustic evidence for subject-specific recognition, rather than relying solely on global audio representations.

This ablation is also included in Section 4.2 and Table 4 of the revised main manuscript.

References:

[1] Jia, Ziyu, et al. "A multimodal BiMamba network with test-time adaptation for emotion recognition based on physiological signals." Advances in Neural Information Processing Systems 38 (2026).

[2] Shi, Jiacheng, et al. "Emo-tta: Improving test-time adaptation of audio-language models for speech emotion recognition." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.