- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Action Unit Prompting for Fine-Grained Video and audio Emotion Recognition which is an interesting topic and has good impact in real time applications. Authors evaluated their proposed methodology on two benchmark datasets and showed that adding audio data along with visual entropy and audio-guided JS divergence improved the model accuracy.
Critical strengths
None
Critical weaknesses
In the literature, spectrogram-based features are useful for emotion recognition as compared to manually extracted features like OpenSMILE. Authors should include spectrogram as opposed to OpenSMILE for comparison.
I think, this study need s systematic experimental evaluations and as such it is at preliminary stage.
It would be really useful if authors adopt this method for emotion recognition on benchmark datasets like IEMOCAP, MELD, CMU-MOSEI datasets.
Coming to creating a audio dictionary using a clustering method can’t be generalized due to fixed dictionary. During training, dictionary may have different audio segments with different words. But during test time, this can be changed completely.
authors should include ablation study to show that adding audio dictionary clustering along with other modules actually improved the performance.
Questions for authors
In the abstract: What does CLIP means? Authors should define when they use the term for the first time. Authors are selective in citing the papers in section 2. There are many papers on multimodal emotion recognition? Why did authors cited very few papers?
Figure 1 needs more information. For example, how many clusters? How authors choose the number of clusters?
T3AL Liberatori et al. (2024) (CVPR’24) reported results on THUMOS14 and the ActivityNet-v1.3. How authors compared these results with different datasets StressID and BAH in the paper.
Work reported in this paper takes the core idea from “ Test-Time Zero-Shot Temporal Action Localization” and did incremental analysis on “CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition”. However, there is no ablation study to show that adding audio cluster dictionary actually improved the accuracy of the proposed model.
Also, authors should include all the datasets when they compare with the SOTA methods.
In the literature, spectrogram-based features are useful for emotion recognition as compared to manually extracted features like OpenSMILE. Authors should include spectrogram as opposed to OpenSMILE for comparison.
I think, this study need s systematic experimental evaluations and as such it is at preliminary stage.
It would be really useful if authors adopt this method for emotion recognition on benchmark datasets like IEMOCAP, MELD, CMU-MOSEI datasets.
Coming to creating a audio dictionary using a clustering method can’t be generalized due to fixed dictionary. During training, dictionary may have different audio segments with different words. But during test time, this can be changed completely.
4 comments
Next Response
"Q5: Should include all the datasets when they compare with the SOTA methods."
"A-5": We would like to clarify that the problem addressed in this work is video-based test-time personalization for fine-grained expression recognition, particularly subtle affective states such as stress and ambivalence. This setting differs from conventional emotion recognition, which primarily focuses on basic emotional categories. To the best of our knowledge, relatively few existing methods specifically address this problem. We therefore compare our approach with relevant state-of-the-art expression recognition and test-time adaptation methods, adapting them to StressID and BAH under the same evaluation protocol. These datasets are directly aligned with the problem considered in our work, whereas the original benchmarks of some competing methods address different tasks. Thus, our comparisons focus on the relevance of the methods and consistency of the evaluation setting rather than including datasets unrelated to fine-grained expression recognition.
"Q6 and W1: Comparison with spectrogram-based features."
"A-6": OpenSMILE extracts interpretable acoustic descriptors directly from the audio signal, including pitch variations, loudness, jitter, shimmer, spectral characteristics, and temporal patterns such as pauses and voiced/unvoiced durations. While spectrogram-based representations also preserve rich acoustic information, these subtle characteristics may require additional feature learning to be explicitly represented. Our approach instead uses compact, predefined acoustic descriptors to construct a lightweight dictionary without training a separate audio encoder.
To further investigate the effectiveness of these representations, we conducted additional experiments on StressID using spectrogram-based features and EMO-TTA [1], an audio-language-based test-time adaptation method built on a CLIP-like architecture. To focus on the contribution of acoustic information, these comparisons consider only audio and text modalities, excluding visual information.
As shown in Table R2, our approach achieves higher performance, highlighting the benefit of clustering OpenSMILE descriptors into representative acoustic patterns rather than relying solely on spectrogram-derived or pretrained audio-language embeddings. These patterns provide complementary information for recognizing subtle emotional variations across subjects. Since the compared approaches also differ in architecture, the results support the effectiveness of our overall acoustic representation strategy rather than isolating the feature extractor alone.
| Method | WAR | F1 |
|---|---|---|
| Spectrogram-based | 42.2 | 35.9 |
| EMO-TTA [2] | 47.9 | 42.1 |
| Ours (Audio-only) | 71.5 | 66.2 |
Table R2. Comparison of audio-only recognition methods on StressID, averaged across 10 target subjects.
We have also incorporated these experiments into Section 4.2 and Table 4 of the main manuscript.
References:
[1] Shi, Jiacheng, et al. "Emo-tta: Improving test-time adaptation of audio-language models for speech emotion recognition." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
Next Response
"Q7 & W2: Need s systematic experimental evaluations."
"A-7": We would like to clarify that our study includes systematic evaluations of the proposed ALIGN-TAPER method across multiple datasets that account for individual variability and different emotion recognition tasks, comparisons with existing baseline methods, and ablation studies of individual components. In the revised manuscript, we have further expanded the experimental analysis to provide a more comprehensive assessment. Specifically:
-
Comparison with existing methods: We have expanded our evaluation by including additional visual-audio baselines, extending CLIP-AU [1] and Exp-CLIP [2] for source training, and CLIP-AUTT [1], T3AL [3], and CRG [4] for test-time adaptation. For a fair comparison, we incorporate acoustic features into these methods using the same feature-concatenation strategy. We have also included MCP [5], a recent TTA method, as an additional baseline. The updated results are presented in Tables 1 and 2, including subject-wise comparisons.
-
Acoustic dictionary evaluation: We conducted standalone audio-based comparisons against spectrogram-based representations and EMO-TTA [6], reported in Table 4, to examine the effectiveness of our acoustic dictionary.
-
Component-level ablations: Beyond the existing analysis of fusion strategies, AU adaptation, and window selection, we have added experiments evaluating audiovisual consistency-guided window selection (visual entropy vs. entropy + JS divergence), subject-specific adaptive fusion (fixed vs. adaptive weights), and acoustic dictionary construction (single class-level prototypes vs. HDBSCAN clustering).
-
Robustness analysis: We further evaluated both source fine-tuning and test-time adaptation across five random seeds, reporting mean performance, standard deviation, and subject-wise results. These results are included in Section D of the supplementary material.
Together, these experiments evaluate the individual contributions of our proposed components, their effectiveness against alternative approaches, and the consistency of the results across target subjects and random initializations. We believe these additional analyses substantially strengthen the experimental validation of ALIGN-TAPER. We would also appreciate it if the reviewer could identify any specific experiments or evaluation aspects they consider missing, so that we can address those concerns more directly.
References:
[1] Zeeshan, Muhammad Osama, et al. "Clip-autt: Test-time personalization with action unit prompting for fine-grained video emotion recognition." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.
[2] Zhao, Zengqun, et al. “Enhancing zero-shot facial expression recognition by LLM knowledge transfer.” 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) . IEEE, 2025.
[3] Liberatori, Benedetta, et al. "Test-time zero-shot temporal action localization." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.
[4] Zhai, Haotian, et al. "Mitigating cache noise in test-time adaptation for large vision-language models." 2025 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2025.
[5] Chen, Xinyu, et al. "Multi-cache enhanced prototype learning for test-time generalization of vision-language models." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
[6] Shi, Jiacheng, et al. "Emo-tta: Improving test-time adaptation of audio-language models for speech emotion recognition." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
Next Response
"Q8 & W3: Adopt for other datasets."
"A-8": Our main focus is video-based test-time personalization for fine-grained expression recognition, particularly subtle affective state. Therefore, we selected StressID and BAH, which are directly aligned with our research objectives. Nevertheless, we acknowledge the value of evaluating the proposed method on more general multimodal emotion recognition benchmarks, such as IEMOCAP, MELD, and CMU-MOSEI, to further investigate its generalization across different emotion recognition tasks. We will consider extending our evaluation to these datasets in future work.
"Q9 & W3: Audio dictionary using a clustering method can’t be generalized"
"A-9":: We would like to clarify that the objective of our acoustic dictionary is not to recognize spoken words or their semantic content, but rather to capture variations in how individuals speak that may be associated with subtle affective states. Specifically, we extract 88-dimensional OpenSMILE eGeMAPS features that characterize acoustic patterns, including pitch (F0), loudness, voice instability (jitter and shimmer), spectral characteristics (spectral slope, MFCCs, and spectral flux), and temporal patterns such as voiced/unvoiced durations and pauses (as shown in Table R3). These descriptors capture vocal delivery and acoustic variation rather than explicitly representing word identities.
| Feature Group | Acoustic Descriptors |
|---|---|
| Pitch | F0, jitter, pitch percentiles, slopes |
| Voice quality | Shimmer, harmonics-to-noise ratio, harmonic differences |
| Spectral | Alpha ratio, Hammarberg index, spectral slopes, MFCCs, spectral flux |
| Formants | Formant frequencies, bandwidths, relative energies |
| Loudness | Loudness, variation, peaks, sound level |
| Temporal | Voiced/unvoiced durations, voiced segments per second |
Table R3. OpenSMILE eGeMAPS acoustic features (88-D)
The complete breakdown of the 88 acoustic parameters is provided in the supplementary material.
The acoustic dictionary is constructed from these speech-related descriptors rather than lexical information. Therefore, changes in the words spoken at test time do not inherently invalidate the learned acoustic representations. While variations in speech content and recording conditions may influence acoustic features, our approach focuses on capturing acoustic patterns potentially transferable across different utterances, rather than relying on a fixed vocabulary.
Rebuttal by Authors
We address each comment below, where “W” denotes a weakness, “Q” a question, and “A” our response.
"Q1: In the abstract: What does CLIP mean and selective in citing the papers in section 2."
"A-1": We have defined CLIP as Contrastive Language–Image Pre-training at its first occurrence in the abstract.
Regarding the related work, we would like to clarify that our research focuses on fine-grained video-based emotion recognition, particularly subtle affective states such as stress and ambivalence, rather than conventional multimodal emotion recognition involving basic emotional categories. Accordingly, Section 2 prioritizes studies most closely related to fine-grained video-based ER, personalization, and test-time adaptation. While the broader multimodal emotion recognition literature is extensive, not all of these studies directly address the specific challenges considered in our work. We would appreciate it if the reviewer could identify specific relevant papers that they believe are missing, so that we can assess their relevance and incorporate them where appropriate.
"Q2: Figure 1 needs more information and Clusters."
"A-2": We would like to clarify that the number of acoustic clusters is not predefined. We employ HDBSCAN, a density-based clustering algorithm that automatically discovers clusters based on the distribution of acoustic features, without requiring a fixed number of clusters. We have revised Figure 1 (main manuscript) to make the clustering process and the automatic determination of the number of clusters clearer. Furthermore, HDBSCAN depends on two parameters: min_cluster_size and min_samples. To select the best values for both parameters, we did an ablation study mentioned in Supplementary B.3 and Figure 5.
"Q3: T3AL, How authors compared these results with different datasets"
"A-3": To the best of our knowledge, relatively few methods address video-based test-time personalization using vision–language models such as CLIP, particularly for subtle expression recognition. We therefore included T3AL [1] as a relevant video-based test-time adaptation baseline. We did not directly compare its originally reported results on THUMOS14 or ActivityNet-v1.3 with our results. Instead, we applied the T3AL approach to StressID and BAH and evaluated it on the same datasets as our proposed method to provide a consistent comparison. We will clarify this in the revised manuscript to avoid any misunderstanding.
"Q4 and W5: Core idea from T3AL, incremental analysis on CLIP-AUTT, and no ablation study"
"A-4": We clarify that T3AL [1] is included as a video-based TTA baseline, rather than serving as the foundation of our proposed method. While ALIGN-TAPER extends CLIP-AUTT [2], it introduces three methodological contributions: (i) an HDBSCAN-based acoustic dictionary to capture fine-grained vocal patterns, (ii) audiovisual consistency-guided informative-window selection, and (iii) subject-specific adaptive fusion during test-time personalization.
To provide stronger experimental evidence, we have extended our comparisons to include additional visual-audio TTA methods, using feature concatenation to combine audio with visual into existing approaches using the same OpenSMILE backbone. As shown in Table R1, these results highlight that the improvement comes not merely from incorporating audio, but from our source-constructed acoustic dictionary, which captures fine-grained acoustic patterns to guide informative-window selection and subject-specific audiovisual adaptation. Importantly, the improvements in F1-score suggest more balanced recognition across emotion classes, particularly in challenging scenarios where subtle affective cues and class imbalance make reliable predictions difficult.
Table R1. Comparison of audiovisual TTA methods on StressID and BAH.
We added comparisons (Sec. 4.1, Tables 1–2; Supp. Sec. C, Tables 9–10) and component ablations (Supp. Sec. D, Tables 6–7) to highlight ALIGN-TAPER's contributions beyond CLIP-AUTT.
References:
[1] Liberatori, Benedetta, et al. "Test-time zero-shot temporal action localization." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.
[2] Zeeshan, Muhammad Osama, et al. "Clip-autt: Test-time personalization with action unit prompting for fine-grained video emotion recognition." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.
[3] Zhai, Haotian, et al. "Mitigating cache noise in test-time adaptation for large vision-language models." 2025 IEEE International Conference on Multimedia and Expo. IEEE, 2025.