- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies a potential limitation of current semantic recognition: representations optimized for the current label may fail to preserve information relevant to future semantic transitions, a phenomenon termed Semantic Transition Aliasing (STA). To address this, the authors propose PSWS, which combines transition-aware representation learning, evidence-conditioned state evolution, and future-state distillation. Experiments show improved transition-signature separation (Table 4) and modest gains in future semantic prediction (Table 2). However, the paper does not convincingly establish that STA is a major limitation of existing methods or that reducing STA is responsible for the observed prediction improvements.
Critical strengths
-
Meaningful problem formulation. The paper highlights a relevant distinction between recognizing current semantics and learning representations suitable for future prediction. The transition-signature objective provides a concrete way to encourage transition-sensitive representations.
-
Reasonably extensive experiments. The paper includes controlled future-prediction baselines, component ablations, transition-separation analyses, routing analyses, and computational cost reports. These experiments provide evidence that the proposed architecture improves prediction performance.
Critical weaknesses
- The proposed mechanism has an insufficiently examined conceptual limitation.
Different realized next labels do not necessarily imply different predictive information in the observed histories. Under stochastic transitions or annotation ambiguity, two histories may have similar conditional future distributions but different observed next labels. The transition-signature objective may therefore encourage unnecessary representation separation. While this is not a demonstrated failure mode, the paper does not sufficiently discuss this limitation of its underlying assumption.
- Several evaluation definitions and protocols remain unclear.
Transition-F1 and Shift-MF1 are both described in relation to samples satisfying , yet their reported values differ substantially. For example, PSWS obtains 59.00 in Table 2 but 39.80 in Table 4 on MELD. Their precise definitions and evaluation scopes need clarification. Furthermore, Figure 3(c) reports predictions at and , but the method does not clearly explain how subsequent transitions are predicted without additional multimodal observations. This is particularly important because closed-loop multi-step prediction is explicitly left for future work in the Conclusion.
- The performance gains are relatively modest, and their reliability and computational trade-offs are insufficiently established.
Compared with GraphSmile+Transition+KD, PSWS improves Future Macro-F1 by only 1.23 and 1.30 percentage points on IEMOCAP-4 and MELD, respectively. Although the paper reports five random seeds, uncertainty estimates for the strongest controlled baseline are not provided. Moreover, Table 8 shows substantially increased training costs relative to GraphSmile, but does not directly compare PSWS with GraphSmile+Transition+KD in terms of model size, computational cost, or inference latency. These omissions make it difficult to assess whether the improvements justify the additional architectural complexity.
Questions for authors
-
Evidence for the STA mechanism. Can the authors provide more direct evidence that STA is a major source of prediction errors in existing methods and that reducing transition aliasing leads to improved future prediction? In particular, how does transition-signature contrastive learning compare with future-label contrastive learning under the same architecture? Which representation is used to compute Margin and Ret@1 in Table 4?
-
Evaluation definitions and multi-step prediction. How exactly are Transition-F1 and Shift-MF1 defined, and why do their reported values differ substantially? How are predictions at and in Figure 3(c) generated without using future observations as predictor inputs?
-
Statistical reliability and efficiency. Can the authors provide seed-level variance or paired comparisons for the strongest controlled baseline and directly compare its parameter count, training cost, and inference latency with PSWS?
Response to Reviewer jjAY (1/3)_I
We sincerely thank the reviewer for recognizing the significance of our research problem and the comprehensive experimental evaluation. We appreciate the careful comments on the theoretical foundation of Semantic Transition Aliasing (STA), evaluation protocols, statistical reliability, and computational efficiency. We address these concerns in three parts.
\
W1 & Q1. Conceptual Validity of STA and Its Relationship to Prediction Errors
STA is motivated by a fundamental distinction between two established tasks in conversational affective computing. Emotion Recognition in Conversation (ERC) identifies the emotion expressed by an observed utterance, whereas Emotion Prediction in Conversation (EPC) estimates the emotion of a forthcoming utterance from preceding dialogue history. Previous EPC studies have investigated interlocutor interactions, multi-utterance contextual dependencies, and multimodal emotional cues (Shi et al., Interspeech 2020, 2023, 2024). These studies highlight the importance of retaining information relevant to future emotional changes rather than optimizing representations exclusively for current-label recognition.
We agree with the reviewer that different realized next labels do not necessarily indicate different predictive distributions. Emotional transitions may involve intrinsic stochasticity, unobserved contextual factors, and annotation ambiguity. Our motivation is therefore not to separate every pair of histories producing different outcomes, but to preserve transition-relevant information that is predictable from the observed history. This distinction can be formalized through conditional predictive sufficiency. Let the semantic representation be a deterministic function of the observed multimodal history: Zt=fθ(Ht). We characterize the predictive information retained in the history but not preserved by the representation, conditional on the current label, as I(Yt+1;Ht∣Zt,Yt).
Since the representation is deterministically computed from the history, the following identity holds: I(Yt+1;Zt∣Yt). The first conditional entropy measures the uncertainty remaining after representation learning, whereas the second represents uncertainty given the observed history. Their difference characterizes predictive information lost through representation compression. Conditioning on the current label serves to examine distinctions within the same semantic class; it does not imply that ground-truth labels are supplied to the predictor at inference. This formulation distinguishes representation insufficiency from irreducible transition uncertainty. Even a predictively sufficient representation cannot eliminate stochasticity inherent in future emotional outcomes.
Accordingly, the transition signatures in Eq. (4), σt=(yt,yt+1), serve as empirical supervision for organizing transition-related representations. The contrastive objective encourages the preservation of distinctions associated with observed transitions; it is not an exact estimator of conditional predictive information. Under highly stochastic or ambiguous outcomes, different realized labels may provide imperfect contrastive supervision. This distinction clarifies the intended scope of STA and the inductive bias introduced by transition-identifiable learning.
The three components of PSWS address complementary requirements arising from this formulation. TISA separates current-semantic evidence from transition-relevant evidence through dedicated query banks and integrates them through cross-attention. This allows the state representation to retain distinctions useful for future prediction without discarding information needed for current recognition. ECTR models state evolution through textual, acoustic, visual, and interaction-specific experts. Its evidence-conditioned routing adapts the state increment to the observed transition cues rather than imposing a homogeneous transition function across dialogue histories. LFSD constrains the predicted future-state destination and increment through EMA-encoded targets available during training. The student predictor operates exclusively on observed history at inference.
These mechanisms jointly address predictive information preservation, heterogeneous state evolution, and future-oriented state supervision.