acceptodds
Under review as a conference paper at ICLR 2027

From Cues to Segments: Learning Latent Affective Units for Multimodal Temporal Emotion Localization

Abstract

Multimodal temporal sentiment localization aims to identify and localize emotion-bearing segments in untrimmed multimodal sequences. However, cross-modal temporal misalignment and sparse affective cues are challenging for existing fusion methods, which often introduce redundant or unreliable information. To address these challenges, we propose LLAU, a multimodal fusion method that learns temporally structured latent affective units to construct dense and boundary-sensitive representations for emotion localization. Specifically, LLAU learns shared and modality-specific units to capture complementary affective cues. By jointly considering cross-modal support, semantic affinity and temporal affinity, it iteratively refines soft token-to-unit assignments and unit representations, thereby organizing sparse multimodal cues into compact affective representations. The learned assignments serve as affective priors for reconstructing dense features along the original timeline and guide modality-specific experts in weighting residual features to mitigate the loss of emotion boundary details during aggregation and reconstruction. We further introduce EmoTEL-8, a target-centric temporal emotion localization benchmark integrating visual, acoustic, and textual modalities. It provides identity-consistent visual tracks of target individuals, complete acoustic and textual context, and precise temporal boundaries for eight emotion categories. Experimental results demonstrate that LLAU achieves state-of-the-art performance on EmoTEL-8 and the public audio-visual benchmark TSL-300.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Emotion Localization, Multimodal Dataset

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qQmBOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The study proposes LLAU, a multimodal fusion method that organizes asynchronous emotion cues into temporally structured latent units for emotion localization. It also introduces EmoTEL-8 and evaluates the method on this dataset and TSL-300.

Critical strengths
  1. This study addresses temporal misalignment while preserving modality-specific cues and local boundary details.

  2. A larger-scale dataset is proposed. Extensive experiments across multiple detectors, supported by extensive ablations, provide evidence for the proposed design.

Critical weaknesses

Comparisons with key fusion baselines, including EMOE, are restricted to the authors’ proposed dataset. The evaluation on the public benchmark TSL-300 is limited to some prior methods especially in the video-level supervision category. Why multimodal fusion methods (e.g., [TSL-Net + EMOE]) is not evaluated on TSL-300?

Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Fusion Comparisons on TSL-300 under Video-Level and Point-Level Supervision

We thank the reviewer for recognizing and supporting our contributions, and for suggesting a more comprehensive comparison on the public benchmark. In response, we have added TSL-Net + EMOE experiments on TSL-300. To further strengthen the comparison, we also include FINE [1], a multimodal fusion method published at AAAI 2026. Both baselines are compared with LLAU under video-level and point-level supervision. Within each supervision setting, all methods use the same data splits, audio-visual input features, TSL-Net localization framework, and evaluation protocol, allowing a controlled comparison of their fusion mechanisms.

The results are presented below. All metrics are reported as percentages, and bold denotes the best result within each supervision setting.

SupervisionMethod[email protected]@0.15@0.20@0.25@0.30Avg. mAPAvg. RecallAvg. F2
Video-levelTSL-Net15.9012.079.506.915.119.90 ± 2.4041.13 ± 5.4826.99 ± 3.04
Video-levelTSL-Net + EMOE15.5112.729.847.244.9810.06 ± 1.0149.94 ± 3.3525.80 ± 1.84
Video-levelTSL-Net + FINE17.5615.1811.698.215.3411.60 ± 0.9055.61 ± 2.1828.70 ± 1.11
Video-levelLLAU (ours)21.2016.4312.519.947.5613.53 ± 1.0659.04 ± 1.7530.47 ± 1.23
Point-levelTSL-Net + EMOE25.4422.2619.1416.1813.0819.22 ± 1.2577.63 ± 1.0330.30 ± 1.18
Point-levelTSL-Net + FINE27.5824.5120.4717.1113.0520.54 ± 0.5976.66 ± 1.6728.00 ± 0.33
Point-levelLLAU (ours)30.8826.9022.5617.2213.2722.17 ± 0.6478.55 ± 1.5734.32 ± 0.85

LLAU achieves the highest average mAP, Recall, and F2 under both supervision settings. Specifically, its average mAP reaches 13.53% and 22.17% under video-level and point-level supervision, respectively, exceeding EMOE by 3.47 and 2.95 percentage points and FINE by 1.93 and 1.63 percentage points. These additional results show that LLAU’s advantages over the compared fusion baselines also extend to the public TSL-300 benchmark, providing further evidence of its effectiveness across datasets and supervision settings. We will include the complete results, experimental settings, and baseline adaptation details in the revised manuscript.

[1] Y. Liu and S. Wang, “FINE: Factorized Multimodal Sentiment Analysis via Mutual INformation Estimation,” AAAI, 2026, pp. 1955–1963.

Dear Reviewer qQmB,

Thank you again for your helpful suggestion to extend the fusion comparisons on TSL-300. We have now uploaded the revised manuscript with a 10-page main text.

As requested, we have added the corresponding experiment to the revised manuscript. We hope this additional result addresses your suggestion.

We would be very happy to hear your thoughts and discuss any remaining questions. We hope that our responses and revisions strengthen your confidence in our work, and we would greatly appreciate your feedback and support.

Best regards,

The Authors

Reviewer_8JUnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the task of temporal emotion localization in multimodal videos by proposing the Learning Latent Affective Units (LLAU) framework. LLAU learns both shared and modality-specific latent affective units from visual, audio, and textual modalities, progressively organizing low-level emotional cues into emotionally coherent segments with a temporal structure. By leveraging cross-modal semantic relationships, temporal continuity constraints, "sink" units for discarding irrelevant information, and a residual reconstruction mechanism, the framework enhances the modeling of emotional boundaries and fine-grained affective transitions. Furthermore, the authors introduce EmoTEL-8, a dataset comprising 325 videos annotated with 8 emotion categories, and conduct experiments on both EmoTEL-8 and TSL-300.

Critical strengths
  • A new dataset for multimodal temporal emotion localization has been constructed in this paper. The EmoTEL-8 dataset provides annotations for eight emotion categories along with their corresponding temporal intervals, establishing a new experimental foundation for the task and enhancing the paper’s contribution to the field. This is the most critical strength.
  • A comprehensive modeling framework for Latent Emotional Units is proposed. The LLAU framework organizes continuous visual, audio, and textual cues into latent emotional units, which are then structured into emotion segments with defined temporal boundaries. This is a highly systematic framework.
Critical weaknesses
  • Emotional boundaries and category labels are inherently somewhat subjective. In fact, clear-cut boundaries may not even exist; rather, the process is likely dynamic and characterized by gradual transitions.
  • The proposed EmoTEL-8 dataset exhibits a pronounced long-tail distribution of emotions.
  • The innovation of the proposed method is not sufficiently prominent. The core components of LLAU are standard techniques within the multimodal domain; concepts such as shared units and modality-specific units are well-established, recurring topics in this field.
  • The semantic interpretation of the "latent emotional units" is inadequate. While the paper characterizes these learned units as fundamental components possessing emotional significance, they lack explicit, human-annotated semantic labels. The provided visualizations merely demonstrate a correlation between these units and certain emotional segments; they do not sufficiently prove that these units correspond to stable, interpretable, and fundamental emotional building blocks.
  • The paper lacks experimental results broken down by specific emotions; Tables 2 and 3 only provide an assessment of global performance. Furthermore, the set of baseline methods used for comparison fails to include the latest approaches from 2026.
  • The author has not made the code public, which limits their contribution to the community significantly.
Questions for authors

I am not particularly familiar with the specific field of Multimodal Temporal Emotion Localization. Based on my general knowledge of the multimodal domain, I have offered some personal insights, as outlined in the "Weaknesses" section. Should any of these points be inaccurate, I invite the authors to clarify and provide a reasoned explanation.

4 comments
AuthorsOpenReview ↗

Part 1/3: Annotation Reliability, Class Imbalance, and Methodological Contribution (Q1–Q3)

We thank the reviewer for the constructive feedback and for recognizing EmoTEL-8 as an important contribution, as well as the systematic design of LLAU. Below, we clarify the annotation principles and methodological contribution, provide additional quantitative evidence and address the suggestions on evaluation.

Q1. Subjectivity of emotion categories and temporal boundaries

EmoTEL-8 annotates the designated person’s emotion expressions using observable facial, bodily, and vocal cues together with the surrounding audiovisual and conversational context. We recognize that emotional transitions can be gradual and that categories and boundaries may admit multiple reasonable interpretations. To address this, eight annotators first followed common guidelines and completed pilot annotations, then independently watched each video in full and annotated the target person’s emotion categories and temporal boundaries.

We computed Fleiss’ κ for each video from these eight independent annotations and excluded videos with κ < 0.2 before joint discussion. After content screening and agreement-based filtering, 325 videos were retained, with a mean pre-adjudication κ of 0.47. This value measures agreement before subsequent review and adjustment. Senior annotators then reviewed the retained annotations, and remaining disagreements over categories or boundaries were jointly examined using multimodal cues and context to determine the final annotations.

This procedure balances annotation reliability with the complexity of real-world emotion. Agreement-based filtering may underrepresent subtle or ambiguous expressions, so generalization to such cases requires further validation. We will clarify the distinction between independent agreement assessment and final adjudication, together with the benchmark’s scope, in the revised manuscript.

Q2. The long-tailed emotion distribution

EmoTEL-8 preserves the diversity and uneven occurrence of emotion expressions in the collected continuous videos. Emotion frequencies depend on the situations, individuals, and narrative content represented in these videos. Comparable class imbalance also occurs in MELD[1], DFEW[2], and Aff-Wild2[3] Retaining this distribution makes localization of both frequent and rare emotions part of the benchmark’s challenge. We report complementary metrics: mAP weights emotion categories equally, whereas wMAP weights them by instance frequency. The additional per-class results in Q6 further expose performance on less frequent emotions. We will clarify this dataset design consideration in the revision.

Q3. The methodological contribution beyond standard components

Our contribution is the connection between temporally structured evidence organization and dense temporal representation recovery for emotion localization. LLAU jointly uses cross-modal support, semantic affinity, and temporal affinity to iteratively refine soft token-to-unit assignments. It then reuses these assignments to guide dense reconstruction along the original timeline and condition modality-specific residual refinement. Thus, the units aggregate asynchronous affective evidence, while their assignments also shape the representations used for localization.

The targeted controls examine this design:

ConfigurationAverage-mAP (%)Drop from Full (pp)
Full LLAU17.47—
w/o Temporal Structure15.761.71
Both Conditioning Paths Off16.021.45
Parameter-matched Temporal Network16.181.29

Removing temporal structure reduces performance despite retaining shared and modality-specific units. Disabling assignment conditioning in both reconstruction and residual refinement also lowers performance while retaining the units, decoder attention, and experts. The parameter-matched temporal network further tests whether additional capacity alone explains the improvement.

Together, these comparisons support the specific contribution of organizing evidence through temporally structured units and using the learned assignments to construct localization representations. We will make this distinction from established shared/specific representation designs more explicit.

AuthorsOpenReview ↗

Part 2/3: Latent-Unit Interpretation and Per-Emotion Performance (Q4–Q6)

Q4. Semantic interpretation of the latent affective units

We use latent affective units to describe temporally structured, input-dependent representations that organize asynchronous affective evidence for localization. Their content depends on the current video and the learned soft assignments. Interval-level emotion supervision acts on their combined reconstruction, encouraging the units to retain emotion-discriminative information. We examine their role through emotion discrimination, allocation behavior, and localization utility.

First, Appendix D.3, Table 11 evaluates emotion classification on the same ground-truth intervals using a common readout protocol and matched auxiliary classification supervision. Results are means and standard deviations over five random seeds:

MethodMacro-F1 (%)
Prior-aware Row-softmax31.10 ± 0.93
EMOE32.80 ± 1.12
LLAU35.61 ± 0.75

LLAU exceeds Prior-aware Row-softmax and EMOE by 4.51 and 2.81 percentage points, respectively. This provides quantitative evidence that the interval representations constructed through the latent units contain useful emotion-discriminative information.

Second, we added allocation statistics on clean inputs across all 65 test videos. We group predicted intervals by correctness using class and temporal matching with ground-truth annotations, and separately group the timeline into ground-truth emotion and background intervals. Within each video and group, we aggregate assignment mass from all three modalities into the five destinations below, normalize using a common denominator, and then average equally across videos containing that group. Values are percentages.

Interval groupSharedVisual-specificAcoustic-specificTextual-specificSink
Correct predictions24.18430.70418.07426.5720.466
Incorrect predictions25.05032.48215.81625.7720.881
Ground-truth emotion intervals24.11231.45217.17726.3480.910
Ground-truth background intervals26.14025.41616.36526.7175.362

Emotion intervals exhibit higher visual-specific allocation and lower sink allocation than background intervals. Correct predictions show higher acoustic-specific and textual-specific allocation than incorrect predictions. These results extend the qualitative examples into dataset-level evidence of associations between allocation behavior, emotion intervals, and localization outcomes.

Third, we tested the contribution of the modality-specific path by freezing the model and redirecting its modality-specific assignment mass to the allocation sink. The random control removes mass from shared and modality-specific assignments, matching the amount removed for each token.

ConditionAverage-mAP (%)
Full model17.4700
Modality-specific mass redirected to the sink10.1486
Matched-mass random removal, mean11.8138

Removing modality-specific mass produces an Average-mAP 1.6652 points lower than matched-mass random removal, supporting the task value of information organized through this path.

These findings support an operational interpretation: the latent units organize emotion-discriminative temporal representations that contribute to localization.

Q5. Research-resource availability

We recognize the importance of accessible research resources for reproducibility and further research. If the paper is accepted, we are willing to release the relevant code in stages after completing the necessary preparation, together with usage instructions to support further research by the community.

Q6. Per-emotion evaluation

We added per-class AP results for LLAU and EMOE, together with the number of test instances in each category. AP is averaged over the same tIoU thresholds used in the main evaluation: {0.1,0.2,0.3,0.4,0.5}\{0.1,0.2,0.3,0.4,0.5\}.

EmotionTest instancesEMOE AP (%)LLAU AP (%)Change (pp)
Disgust185.277.94+2.67
Confusion378.6210.12+1.50
Sadness4316.6418.08+1.44
Fear2216.3916.98+0.59
Contempt166.106.37+0.27
Anger4729.0529.15+0.10
Happiness23633.5239.24+5.72
Surprise9212.3511.90−0.45
Overall Average-mAP51115.9917.47+1.48

LLAU improves AP for seven of the eight emotions. Happiness exhibits the largest improvement, while less frequent categories, including Disgust, Confusion, and Contempt, also benefit. The gains therefore extend across categories with different sample frequencies, although their magnitude varies. We will add these results and the accompanying analysis to complement the aggregate metrics.

AuthorsOpenReview ↗

Part 3/3: Comparisons with Recent Methods and References (Q7)

Q7. Comparisons with recent methods from 2026

We added comparisons on EmoTEL-8 with two recent methods: EBMC [4] (CVPR 2026) and FINE [5] (AAAI 2026). Each threshold column reports mAP / wMAP; all values are percentages.

MethodYear@0.10@0.20@0.30@0.40@0.50Average-mAPAverage-wMAP
EBMC202619.05 / 28.5116.22 / 24.8913.92 / 20.6110.59 / 16.127.52 / 12.0113.46 ± 1.1020.43 ± 0.93
FINE202622.92 / 29.8218.00 / 24.8515.53 / 20.6110.96 / 14.987.97 / 10.9815.08 ± 0.7820.25 ± 1.14
LLAU—25.07 / 36.3521.73 / 32.5117.90 / 27.7013.58 / 20.819.08 / 14.6817.47 ± 0.5326.41 ± 0.36

LLAU maintains the highest mAP and wMAP at every reported tIoU threshold among these methods. Its Average-mAP exceeds FINE, the stronger of the two added baselines on this metric, by 2.39 percentage points. We will include these comparisons in the revised manuscript.

References

[1] S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations,” ACL, 2019, pp. 527–536.

[2] X. Jiang, Y. Zong, W. Zheng, C. Tang, W. Xia, C. Lu, and J. Liu, “DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild,” ACM MM, 2020, pp. 2881–2889.

[3] D. Kollias and S. Zafeiriou, “Analysing Affective Behavior in the Second ABAW2 Competition,” ICCV Workshops, 2021, pp. 3652–3660.

[4] K. He, Y. Ding, X. Wang, F. Li, C. Teng, and D. Ji, “Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis,” CVPR, 2026, pp. 30183–30193.

[5] Y. Liu and S. Wang, “FINE: Factorized Multimodal Sentiment Analysis via Mutual INformation Estimation,” AAAI, 2026, pp. 1955–1963.

Dear Reviewer 8JUn,

Thank you again for recognizing the value of EmoTEL-8 and for your constructive suggestions. We have now uploaded the revised manuscript with a 10-page main text.

Our responses clarify the annotation protocol and the methodological contribution of LLAU, and provide further quantitative evidence on the role of the latent affective units. We also added per-emotion results and comparisons with EBMC and FINE in response to your suggestions on evaluation.

We would be very happy to continue the discussion and clarify any remaining questions. We hope that these clarifications and additional results strengthen your confidence in our work, and we would greatly appreciate your feedback and support.

Best regards,

The Authors

Reviewer_W5CCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new method (LLAU) for multimodal temporal emotion localization, which introduces temporally structured latent affective units that consist of shared units and modality-specific units, to address the issues that the affective cues from different modalities may be temporally misaligned, the truly informative emotional evidence is often sparse, and direct fixed-time alignment or unrestricted cross-temporal interaction may introduce redundant or unreliable information. The proposed method first constructs token-to-unit assignments by jointly considering cross-modal support, semantic affinity, and temporal affinity, and then iteratively refines both the assignments and the latent units. The final assignments are further used for dense feature reconstruction and to condition modality-specific residual experts, with the aim of recovering local boundary information that may be lost during aggregation. Additionally, this work constructs a new dataset (EmoTEL-8) to validate the performance of the proposed method. Experimental results also illustrate the proposed method outperforms existing fusion methods.

Critical strengths

This paper presents a complete framework for addressing temporal misalignment and sparse affective cues in multimodal temporal emotion localisation. Compared with simple cross-modal fusion, the idea of organising asynchronous affective evidence using temporally structured latent units and then reconstructing a dense temporal representation is interesting and progressive. Moreover, the experiment is also comprehensive, as the authors compare LLAU with multiple multimodal fusion baselines and evaluate across four temporal localisation architectures. Compared with EMOE, LLAU improves performance by approximately 1.48~2.48% across ActionFormer, TriDet, TemporalMaxer, and DyFADet.

Critical weaknesses
  1. The current experiments do not sufficiently disentangle the benefit of the latent affective unit mechanism from that of the additional auxiliary supervision. In the fully supervised EmoTEL-8 setting, in addition to the detector localization loss, LLAU introduces an interval-level emotion classification loss, LunitL_{unit}, which directly supervises the unit reconstruction. When this auxiliary supervision is removed, the Average-mAP drops from 17.47 to 16.19, while the EMOE baseline achieves 15.99. This suggests that a large portion of the 1.48 percentage-point improvement of Full LLAU over EMOE may be attributable to the additional supervision rather than to the latent-unit architecture itself. The current “w/o Unit Auxiliary Supervision” ablation demonstrates that this loss is effective, but it does not answer whether LLAU would still retain a clear advantage if a parameter-matched baseline or EMOE were equipped with a similar interval-level auxiliary classification objective. Therefore, I do not think the current experiments sufficiently support attributing the headline performance gain to the core fusion mechanism proposed in the paper.

  2. The interpretation of the shared and modality-specific affective units lacks direct validation. LLAU uses cross-modal cosine support to determine shared and specific assignments: tokens with higher cross-modal support are more likely to be assigned to shared units, while those with lower support are more likely to be assigned to modality-specific units. However, low cross-modal support does not necessarily indicate that a token represents a “reliable modality-specific affective cue.” It may also result from modality noise, representation-space mismatch, temporal misalignment, or information unrelated to emotion. The paper introduces an allocation sink to handle invalid information and provides a small number of qualitative unit visualizations, but this evidence is still insufficient to demonstrate that the specific units systematically preserve useful modality-specific affective information rather than simply absorbing tokens that lack support from other modalities.

Questions for authors
  1. Full LLAU uses additional interval-level unit auxiliary supervision on EmoTEL-8. After removing LunitL_{unit}, the Average-mAP drops from 17.47 to 16.19, which is already very close to EMOE at 15.99. Could the authors equip EMOE or the parameter-matched temporal baseline with an analogous auxiliary classification objective using the same ground-truth interval labels? This would help determine whether the performance gain mainly comes from the latent-unit design itself or from the additional supervision pathway.

  2. Why is low cross-modal support interpreted as “modality-specific affective evidence”? Low support may also arise from noise, temporal misalignment, or representation mismatch between different encoders. Could the authors provide a controlled corruption experiment, for example by deliberately corrupting one modality and analyzing how the corresponding tokens are distributed in terms of assignment mass across shared units, modality-specific units, and the sink?

  3. Could the authors quantitatively report the average assignment mass allocated to shared, visual-specific, audio-specific, text-specific units, and the sink, and further analyze their distributions for correct versus incorrect predictions and for emotion versus background intervals? Such quantitative analysis would provide more direct evidence for the semantic interpretation of the latent units than the limited qualitative cases currently presented.

  4. The initial candidate videos for EmoTEL-8 are screened based on “emotional variation” and “contextual completeness,” and videos with Fleiss’ κ<0.2\kappa < 0.2 are excluded, resulting in an average κ\kappa of 0.47 for the retained dataset. Could this filtering procedure remove ambiguous or difficult-to-interpret emotional segments, thereby biasing the benchmark toward samples on which annotators can more easily reach agreement? I suggest that the authors discuss this potential selection bias and its implications for generalization to more ambiguous real-world scenarios.

  5. The authors use Fleiss’ κ\kappa to measure agreement for segment annotations with continuous temporal boundaries. Could the authors clarify how κ\kappa is computed in this setting? The current main text and appendix do not sufficiently explain how temporal interval annotations are converted into the fixed evaluation units required for calculating Fleiss’ κ\kappa.

  6. EmoTEL-8 exhibits substantial class imbalance, with 1,151 Happiness instances but only 78 Contempt instances. Although the paper reports both mAP and wMAP, I suggest providing per-class AP results, especially for rare emotion categories, to determine whether the observed improvements are mainly driven by the more frequent classes.

  7. The external evaluation on TSL-300 is valuable, but this benchmark only contains audio-visual modalities and uses video-level or point-level weak supervision, whereas the main EmoTEL-8 setting is fully supervised, trimodal, and target-centric. Could the authors further validate LLAU on another fully supervised trimodal benchmark?

5 comments
AuthorsOpenReview ↗

Part 1/4: Auxiliary Supervision and Controlled Corruption (Q1–Q2)

We thank the reviewer for the constructive feedback and for recognizing the value of organizing asynchronous affective evidence through temporally structured latent units, as well as our evaluation across multiple temporal localization architectures. We address the concerns below with additional controlled experiments, quantitative analyses, and clarifications.

Q1. Distinguishing the latent-unit design from auxiliary supervision

We added an EMOE comparison with analogous interval-level auxiliary classification supervision using the same ground-truth interval labels. The five-seed results on EmoTEL-8 are:

MethodAverage-mAP (%)
EMOE15.99 ± 0.85
EMOE + interval-level auxiliary supervision16.08 ± 0.67
Full LLAU17.47 ± 0.53

Auxiliary supervision improves EMOE by 0.09 percentage points, while Full LLAU retains a 1.39-point advantage under matched interval-level supervision. Together with the original ablation, which shows a 1.28-point improvement from auxiliary supervision within LLAU, these results suggest that the benefit depends on the representation being supervised.

In LLAU, Lunit\mathcal{L}_{\mathrm{unit}} directly supervises the unit reconstruction before the decoder and residual experts contribute, encouraging the latent units to preserve emotion-discriminative information. The comparison supports the contribution of the latent-unit design in conjunction with interval-level auxiliary supervision, rather than attributing the improvement simply to access to additional supervision. We will include this comparison in the revised manuscript.

Q2. Interpreting low cross-modal support and testing controlled corruption

Low cross-modal support indicates weak local corroboration from other modalities; it does not by itself establish that a token contains reliable modality-specific affective evidence. In LLAU, support biases the relative assignment costs of shared and modality-specific units. The final assignments also depend on semantic affinity, temporal affinity, and competition with the allocation sink. Modality-specific units therefore represent modality-conditioned information learned for emotion localization, rather than tokens deemed reliable solely because their cross-modal support is low.

To examine how corrupted information is allocated, we froze the model and conducted controlled single-modality corruption on all 65 test videos. For each video, we separately corrupted the visual, acoustic, or textual modality while keeping the other inputs unchanged. We randomly selected approximately 30% of valid tokens in blocks of eight temporal positions and replaced their features with Gaussian noise generated using the training-set mean and standard deviation of each feature dimension. Each video was evaluated once with clean inputs and once for each of the three corrupted modalities.

We compared the final soft assignments at the same selected positions before and after corruption. For each token, we summed the assignment mass allocated to shared units, the corresponding modality-specific units, and the allocation sink, and normalized by its total assignment mass. We then averaged these fractions within each video using valid-duration weights, followed by an equal-weight average across videos.

Corrupted modalitySharedModality-specificSinkAverage-mAP after corruption (%)
Visual12.17% → 10.31%84.94% → 74.07%2.89% → 15.62%15.829
Acoustic52.57% → 51.81%44.43% → 43.49%3.00% → 4.70%16.065
Textual4.74% → 0.95%92.58% → 60.53%2.67% → 38.52%16.809

Arrows denote clean-to-corrupted changes at the selected positions. Sink allocation increases by 12.73, 1.70, and 35.85 percentage points for visual, acoustic, and textual tokens, respectively, while modality-specific allocation decreases in all three cases. Thus, the model redirects more corrupted mass to the sink under the tested Gaussian perturbations, with stronger responses for visual and textual features.

AuthorsOpenReview ↗

Part 2/4: Allocation Patterns and Modality-Specific Contributions (Q3)

Q3. Quantitative allocation analysis and the contribution of modality-specific units

We added an allocation analysis on clean inputs across all 65 test videos. Predicted intervals were grouped as correct or incorrect according to their class and temporal matching with ground-truth annotations. Separately, ground-truth annotations were used to distinguish emotion from background intervals.

For each video and interval group, we aggregated assignment mass from all three modalities into five destinations: shared units, visual-specific units, acoustic-specific units, textual-specific units, and the allocation sink. We normalized these quantities using a common denominator and then averaged equally across videos containing the relevant group. Each row sums to approximately 100%.

Interval groupSharedVisual-specificAcoustic-specificTextual-specificSink
Correct predictions24.18430.70418.07426.5720.466
Incorrect predictions25.05032.48215.81625.7720.881
Ground-truth emotion intervals24.11231.45217.17726.3480.910
Ground-truth background intervals26.14025.41616.36526.7175.362

Correct predictions exhibit higher acoustic-specific and textual-specific allocation and lower sink allocation than incorrect predictions, indicating an association between allocation patterns and localization outcomes. Emotion and background intervals differ most clearly in visual-specific and sink allocation: visual-specific allocation is higher in emotion intervals, whereas sink allocation is higher in background intervals. These statistics show different retention and rejection patterns across the groups.

To further assess the task contribution of the modality-specific path, we froze the model and redirected its modality-specific assignment mass to the sink. As a control, we randomly removed the same amount of mass for each token from its retained shared and modality-specific assignments and redirected it to the sink.

ConditionAverage-mAP (%)
Full model17.4700
Modality-specific mass redirected to the sink10.1486
Matched-mass random removal, mean11.8138

Removing modality-specific mass yields an Average-mAP 1.6652 points lower than matched-mass random removal. This supports the task contribution of the modality-specific path beyond the effect of reducing retained mass alone. Together, the group-level allocation statistics and path intervention complement the qualitative visualizations with quantitative evidence about allocation behavior and localization utility. We will include these results and their statistical procedures in the revision.

AuthorsOpenReview ↗

Part 3/4: Dataset Selection, Annotation Agreement, and Per-Class Performance (Q4–Q6)

Q4. Potential selection bias in EmoTEL-8

The screening procedure aims to ensure that emotion intervals have observable multimodal cues and sufficient contextual support, providing a reproducible basis for localization evaluation. This involves a trade-off between annotation reliability and the complexity of real-world emotion: agreement-based filtering may reduce the representation of subtle, implicit, or multiply interpretable emotional episodes.

Accordingly, EmoTEL-8 primarily evaluates localization of emotion intervals supported by discernible cues. Generalization to highly ambiguous situations, emotions requiring substantial person-specific context, or applications demanding sensitivity to subtle emotional changes requires further validation. We will add this potential selection bias and its implications for the benchmark’s scope to the limitations discussion.

Q5. Computing Fleiss’ κ\kappa from continuous interval annotations

We use video frames at 24 fps as the common evaluation units and convert the eight annotators’ independent, pre-adjudication intervals into framewise categorical labels. Frame ff corresponds to tf=f/24t_f=f/24 seconds and receives the label of interval [s,e)[s,e) when s≤tf<es\leq t_f<e. Times not annotated by a given annotator receive the background label. The calculation covers all frames and ten categories: the eight target emotions, Other, and background.

For video vv containing NvN_v frames, let nfcn_{fc} denote the number of annotators assigning category cc to frame ff, with ∑c=110nfc=8\sum_{c=1}^{10}n_{fc}=8. The framewise pairwise agreement and video-level observed agreement are

\qquad \bar P_v=\frac{1}{N_v}\sum_{f=0}^{N_v-1}P_f.$$ Let $p_{vc}$ denote the proportion of all annotation decisions assigned to category $c$. The expected chance agreement and video-level Fleiss’ $\kappa$ are $$p_{vc}=\frac{\sum_{f=0}^{N_v-1}n_{fc}}{8N_v}, \qquad P_{e,v}=\sum_{c=1}^{10}p_{vc}^{2}, \qquad \kappa_v=\frac{\bar P_v-P_{e,v}}{1-P_{e,v}}.$$ Dataset-level agreement is the unweighted arithmetic mean of $\kappa_v$ over retained videos. Framewise conversion is used for agreement assessment; the final localization annotations retain their original temporal intervals. We will add these mapping rules, symbol definitions, and equations to Appendix A.2. ### Q6. Per-class performance under class imbalance We added a per-class AP comparison between LLAU and EMOE. AP is averaged over the EmoTEL-8 tIoU thresholds $\{0.1,0.2,0.3,0.4,0.5\}$. | Emotion | Test instances | EMOE AP (%) | LLAU AP (%) | Change (pp) | |---|---:|---:|---:|---:| | Disgust | 18 | 5.27 | 7.94 | +2.67 | | Confusion | 37 | 8.62 | 10.12 | +1.50 | | Sadness | 43 | 16.64 | 18.08 | +1.44 | | Fear | 22 | 16.39 | 16.98 | +0.59 | | Contempt | 16 | 6.10 | 6.37 | +0.27 | | Anger | 47 | 29.05 | 29.15 | +0.10 | | Happiness | 236 | 33.52 | 39.24 | +5.72 | | Surprise | 92 | 12.35 | 11.90 | −0.45 | The overall Average-mAP increases from **15.99% to 17.47%**, and Average-wMAP increases from **23.48% to 26.41%**. LLAU improves AP for seven of the eight emotions. Happiness has the largest gain, but improvements also occur for less frequent categories, including Disgust, Confusion, and Contempt. Thus, the gains are not confined to the most frequent category, although their magnitude varies across emotions. We will add this table and analysis to the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.