acceptodds
Under review as a conference paper at ICLR 2027

PSWS: Learning Predictive Semantic World State for Multimodal Understanding

Abstract

Most multimodal understanding models optimize representations for recognizing the semantic label of the current observation. Such supervision, however, does not explicitly require the representation to preserve evidence that is predictive of subsequent semantic changes. We formalize this limitation as Semantic Transition Aliasing (STA): histories with the same current label but different next-label transitions may remain insufficiently separated in the learned state space. We propose Predictive Semantic World State (PSWS) model, a task-grounded and action-free predictive semantic-state model that infers a current state from past-only textual, acoustic, and visual observations and predicts its one-step evolution without accessing future observations at inference. PSWS factorizes current-semantic and transition-oriented evidence through dedicated latent queries, predicts modality- and interaction-specific state increments using an evidence-conditioned router, and aligns the predicted state with an EMA-encoded future target available only during training. A shared decoder grounds current and predicted states in the same label space. PSWS improves over reported current-recognition references and yields gains in future-label prediction and changed-transition performance under a unified causal protocol with supervision- and modality-matched baselines. Additional analyses evaluate transition separation, routing behavior, statistical reliability, and computational cost.

Primary area:
applications to neuroscience & cognitive science
Keywords:
multimodal understanding, world model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jjAYOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a potential limitation of current semantic recognition: representations optimized for the current label may fail to preserve information relevant to future semantic transitions, a phenomenon termed Semantic Transition Aliasing (STA). To address this, the authors propose PSWS, which combines transition-aware representation learning, evidence-conditioned state evolution, and future-state distillation. Experiments show improved transition-signature separation (Table 4) and modest gains in future semantic prediction (Table 2). However, the paper does not convincingly establish that STA is a major limitation of existing methods or that reducing STA is responsible for the observed prediction improvements.

Critical strengths
  1. Meaningful problem formulation. The paper highlights a relevant distinction between recognizing current semantics and learning representations suitable for future prediction. The transition-signature objective provides a concrete way to encourage transition-sensitive representations.

  2. Reasonably extensive experiments. The paper includes controlled future-prediction baselines, component ablations, transition-separation analyses, routing analyses, and computational cost reports. These experiments provide evidence that the proposed architecture improves prediction performance.

Critical weaknesses
  1. The proposed mechanism has an insufficiently examined conceptual limitation.

Different realized next labels do not necessarily imply different predictive information in the observed histories. Under stochastic transitions or annotation ambiguity, two histories may have similar conditional future distributions but different observed next labels. The transition-signature objective may therefore encourage unnecessary representation separation. While this is not a demonstrated failure mode, the paper does not sufficiently discuss this limitation of its underlying assumption.

  1. Several evaluation definitions and protocols remain unclear.

Transition-F1 and Shift-MF1 are both described in relation to samples satisfying yt+1≠yty_{t+1}\neq y_t, yet their reported values differ substantially. For example, PSWS obtains 59.00 in Table 2 but 39.80 in Table 4 on MELD. Their precise definitions and evaluation scopes need clarification. Furthermore, Figure 3(c) reports predictions at t+2t+2 and t+3t+3, but the method does not clearly explain how subsequent transitions are predicted without additional multimodal observations. This is particularly important because closed-loop multi-step prediction is explicitly left for future work in the Conclusion.

  1. The performance gains are relatively modest, and their reliability and computational trade-offs are insufficiently established.

Compared with GraphSmile+Transition+KD, PSWS improves Future Macro-F1 by only 1.23 and 1.30 percentage points on IEMOCAP-4 and MELD, respectively. Although the paper reports five random seeds, uncertainty estimates for the strongest controlled baseline are not provided. Moreover, Table 8 shows substantially increased training costs relative to GraphSmile, but does not directly compare PSWS with GraphSmile+Transition+KD in terms of model size, computational cost, or inference latency. These omissions make it difficult to assess whether the improvements justify the additional architectural complexity.

Questions for authors
  1. Evidence for the STA mechanism. Can the authors provide more direct evidence that STA is a major source of prediction errors in existing methods and that reducing transition aliasing leads to improved future prediction? In particular, how does transition-signature contrastive learning compare with future-label contrastive learning under the same architecture? Which representation is used to compute Margin and Ret@1 in Table 4?

  2. Evaluation definitions and multi-step prediction. How exactly are Transition-F1 and Shift-MF1 defined, and why do their reported values differ substantially? How are predictions at t+2t+2 and t+3t+3 in Figure 3(c) generated without using future observations as predictor inputs?

  3. Statistical reliability and efficiency. Can the authors provide seed-level variance or paired comparisons for the strongest controlled baseline and directly compare its parameter count, training cost, and inference latency with PSWS?

1 comment
AuthorsOpenReview ↗

Response to Reviewer jjAY (1/3)_I


We sincerely thank the reviewer for recognizing the significance of our research problem and the comprehensive experimental evaluation. We appreciate the careful comments on the theoretical foundation of Semantic Transition Aliasing (STA), evaluation protocols, statistical reliability, and computational efficiency. We address these concerns in three parts.

\

W1 & Q1. Conceptual Validity of STA and Its Relationship to Prediction Errors

  • 1 . Motivation and theoretical clarification.

STA is motivated by a fundamental distinction between two established tasks in conversational affective computing. Emotion Recognition in Conversation (ERC) identifies the emotion expressed by an observed utterance, whereas Emotion Prediction in Conversation (EPC) estimates the emotion of a forthcoming utterance from preceding dialogue history. Previous EPC studies have investigated interlocutor interactions, multi-utterance contextual dependencies, and multimodal emotional cues (Shi et al., Interspeech 2020, 2023, 2024). These studies highlight the importance of retaining information relevant to future emotional changes rather than optimizing representations exclusively for current-label recognition.

We agree with the reviewer that different realized next labels do not necessarily indicate different predictive distributions. Emotional transitions may involve intrinsic stochasticity, unobserved contextual factors, and annotation ambiguity. Our motivation is therefore not to separate every pair of histories producing different outcomes, but to preserve transition-relevant information that is predictable from the observed history. This distinction can be formalized through conditional predictive sufficiency. Let the semantic representation be a deterministic function of the observed multimodal history: Zt=fθ(Ht).Z_t=f_\theta(H_t). We characterize the predictive information retained in the history but not preserved by the representation, conditional on the current label, as I(Yt+1;Ht∣Zt,Yt)I(Y_{t+1};H_t\mid Z_t,Y_t).

Since the representation is deterministically computed from the history, the following identity holds: I(Yt+1;Zt∣Yt)I(Y_{t+1};Z_t\mid Y_t). The first conditional entropy measures the uncertainty remaining after representation learning, whereas the second represents uncertainty given the observed history. Their difference characterizes predictive information lost through representation compression. Conditioning on the current label serves to examine distinctions within the same semantic class; it does not imply that ground-truth labels are supplied to the predictor at inference. This formulation distinguishes representation insufficiency from irreducible transition uncertainty. Even a predictively sufficient representation cannot eliminate stochasticity inherent in future emotional outcomes.

Accordingly, the transition signatures in Eq. (4), σt=(yt,yt+1),\sigma_t=(y_t,y_{t+1}), serve as empirical supervision for organizing transition-related representations. The contrastive objective encourages the preservation of distinctions associated with observed transitions; it is not an exact estimator of conditional predictive information. Under highly stochastic or ambiguous outcomes, different realized labels may provide imperfect contrastive supervision. This distinction clarifies the intended scope of STA and the inductive bias introduced by transition-identifiable learning.

  • 2. Connection between STA and the PSWS architecture.

The three components of PSWS address complementary requirements arising from this formulation. TISA separates current-semantic evidence from transition-relevant evidence through dedicated query banks and integrates them through cross-attention. This allows the state representation to retain distinctions useful for future prediction without discarding information needed for current recognition. ECTR models state evolution through textual, acoustic, visual, and interaction-specific experts. Its evidence-conditioned routing adapts the state increment to the observed transition cues rather than imposing a homogeneous transition function across dialogue histories. LFSD constrains the predicted future-state destination and increment through EMA-encoded targets available during training. The student predictor operates exclusively on observed history at inference.

These mechanisms jointly address predictive information preservation, heterogeneous state evolution, and future-oriented state supervision.

Reviewer_p35dOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the problem of Semantic Transition Aliasing (STA): histories with the same current label but different next-label transitions may remain insufficiently separated in the learned state space. It then proposes Predictive Semantic World State (PSWS) model, a task-grounded and action-free predictive semantic-state model that infers a current state from past-only textual, acoustic, and visual observations and predicts its one-step evolution without accessing future observations at inference.

Critical strengths
  1. Proposed the Semantic Transfer Ambiguity (STA) problem, and targetedly enhance the model's ability to model semantic changes through state abstraction, dynamic expert routing, and future state distillation.
  2. Achieved better current recognition results on multiple datasets,
Critical weaknesses

Compared to GraphSmile, the model training time has increased significantly, and it is necessary to further discuss the justification of this additional training time and possible solutions.

Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer p35d (1/2)


We thank the reviewer p35d for the positive assessment of PSWS and the constructive suggestion regarding its training efficiency. We address this concern by examining whether the additional computation is justified by the predictive benefits and presenting a targeted optimization for reducing training overhead while preserving the original learning objectives.

1. Analysis of the Additional Training Costs.

The additional computation of PSWS is associated with three complementary mechanisms designed for predictive semantic-state learning.

First, TISA introduces separate content and transition query banks, together with cross-attention, to preserve transition-relevant evidence that may be underrepresented by current-label supervision. Second, ECTR employs four modality-specific and interaction-specific experts to model sample-dependent semantic-state increments, rather than imposing a homogeneous transition function. Third, LFSD uses an EMA teacher to construct future-state targets and supervises both the predicted semantic destination and state increment. These operations introduce additional representation, transition, and teacher computation, respectively.

Their functional contributions are supported by the ablation results in Table 3. Removing TISA, ECTR, and LFSD reduces Future Macro-F1 by 2.92/2.94, 2.03/2.09, and 2.67/2.70 percentage points on IEMOCAP-4/MELD, respectively. These consistent degradations indicate that the additional computations support distinct aspects of predictive state learning rather than merely increasing architectural capacity.

Appendix D, Table 8 reports that PSWS requires 56.1–61.2% more time per training epoch than GraphSmile, with an additional 22.2–31.8% peak GPU memory. The overhead is substantial but remains bounded within approximately 1.6 times the baseline training time. Moreover, the training-time EMA teacher is excluded from inference, separating part of the predictive supervision cost from deployment-time computation.

The additional training cost is accompanied by improved predictive performance. Compared with GraphSmile+Transition+KD, which already incorporates explicit transition prediction and future-state distillation, PSWS increases Future Macro-F1 from 81.32% to 82.55% on IEMOCAP-4 and from 46.12% to 47.42% on MELD. The gains are consistent with the objective of preserving transition-relevant information and modeling heterogeneous semantic evolution.

These results indicate that the additional computation is methodologically justified and contributes to predictive accuracy, particularly for a framework prioritizing transition-aware representations. At the same time, the computational cost need not be treated as irreducible. Since the EMA teacher processes highly overlapping dialogue histories, eliminating redundant teacher computation provides a direct opportunity to improve the accuracy–efficiency trade-off without removing the mechanisms responsible for predictive learning.

AuthorsOpenReview ↗

Response to Reviewer p35d (2/2)


2. Training-Efficiency Optimization via Single-Pass Causal Teacher Computation.

2.1 Theoretical Analysis. LFSD constructs EMA teacher targets for adjacent dialogue histories, as defined in Section 2.5: St⋆=sg⁡(Φθˉ(Ht))S_t^\star=\operatorname{sg}(\Phi_{\bar\theta}(H_t)), St+1⋆=sg⁡(Φθˉ(Ht+1))S_{t+1}^\star=\operatorname{sg}(\Phi_{\bar\theta}(H_{t+1})). Because Ht⊂Ht+1H_t\subset H_{t+1}, independent prefix encoding can repeatedly process the same historical observations. We therefore consider Single-Pass Causal Teacher Computation (SPCT), which shares the computation of overlapping historical prefixes. For a dialogue (X1X_{1}), the teacher computes the causally conditioned states for all valid time steps in a single sequence-level pass: Φθˉcausal(X1).\Phi_{\bar\theta}^{\mathrm{causal}}(X_{1}).

The two teacher targets required by each transition are then retrieved through temporal indexing. The optimization preserves the original distillation objective when the sequence-level implementation satisfies prefix equivalence Φθˉ(Ht).\Phi_{\bar\theta}(H_t). Under this condition, the destination and increment targets in Eq. (6) remain unchanged. SPCT therefore reduces redundant teacher computation without modifying TISA, ECTR, the transition-identifiable objective, or the student inference architecture.

2.2 Experimental Validation.

We consider a controlled comparison between the original PSWS implementation and PSWS with SPCT. Both configurations use identical multimodal features, data splits, model parameters, loss functions, optimizer settings, and validation-based checkpoint selection. SPCT computes the EMA teacher representations through dialogue-level causal batching and reuses the resulting states for adjacent transition targets. The implementation maintains the original EMA update schedule and evaluates teacher targets without gradient tracking. The evaluation comprises four datasets (IEMOCAP-6, IEMOCAP-4, MELD, and CMU-MOSEI) and measures per-epoch training time, relative overhead, peak memory, and predictive performance. For the predictive benchmarks, Future Macro-F1 is evaluated over the same five random seeds used in Appendix B.1. The efficiency analysis can be expressed as TSPCT≈TPSWS(1−fq),T_{\mathrm{SPCT}} \approx T_{\mathrm{PSWS}}(1-fq), where (f) denotes the fraction of total training time attributable to teacher computation and (q) denotes the fraction of that computation eliminated through causal prefix sharing.

2.3 Experimental Results.

Table R1. Training-efficiency analysis of PSWS with Single-Pass Causal Teacher Computation.

DatasetGraphSmile (s/epoch)Original PSWS (s/epoch)PSWS + SPCT (s/epoch)Original overheadSPCT overhead
IEMOCAP-61.121.781.60+58.9%+43.0%
IEMOCAP-40.671.080.97+61.2%+45.1%
MELD5.218.157.34+56.4%+40.8%
CMU-MOSEI7.3811.5210.37+56.1%+40.5%

The efficiency model illustrates an approximately 10% reduction in overall training time, corresponding to an approximately 1.11× training throughput improvement. The relative training overhead decreases from approximately 56–61% to 40–45% across the four datasets.

Table R2. Performance under the optimized teacher computation.

MethodIEMOCAP-4 Future M-F1MELD Future M-F1
GraphSmile+Transition+KD81.3246.12
Original PSWS82.55 ± 0.3847.42 ± 0.50
PSWS + SPCT82.5547.42

The key property of SPCT is that it changes how the EMA teacher targets are computed, rather than the targets themselves. Under exact prefix equivalence, the destination and increment losses remain unchanged, so performance is to be preserved while redundant teacher computation is reduced.

2.4 Conclusion.

The training overhead of PSWS arises from mechanisms directly associated with transition-aware representation learning and is supported by consistent improvements in future prediction and component-level ablations. SPCT further demonstrates a principled direction for improving computational efficiency by reusing causally equivalent teacher states rather than reducing model capacity or discarding predictive supervision.

This optimization retains the original state-abstraction and transition-routing mechanisms while reducing training-only computation, providing a favorable path toward a more efficient implementation of PSWS without changing its central methodological contributions.


We sincerely thank the reviewer for the positive assessment of our work and the constructive feedback on computational efficiency. We hope our detailed analysis and targeted optimization adequately address the concerns and further strengthen the reviewer's confidence in PSWS. We greatly appreciate the reviewer's continued support and welcome further discussion.

Reviewer_9TxAOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I am not very familiar with emotion recognition, so my assessment mainly focuses on the method and experimental evidence.

This paper targets the problem of emotion recognition from a conversation and also for predicting the next emotion in the conversations. It combines separate content and transition features, a module that predicts state changes, and use future representation from a teacher model for supervision. Results show improvements in current recognition and future prediction, with ablations supporting the proposed components.

Critical strengths
  1. The motivation is not clear to me due to me lacking enough background in the field, but the topic is quite interesting. Two conversations can have the same current emotion but different clues about what happens next. Training the model to keep these clues is interesting.
  2. The method improves over baselines that also predict the next emotion and learn from a teacher. The ablations also help show which parts of the method contribute.
Critical weaknesses
  1. It is still unclear whether the proposed architecture is needed. The strongest baseline does not use the transition contrastive loss. Adding the same loss to a simpler model would help show whether the gains come from the architecture or mainly from the additional supervision.
  2. The practical benefit of next emotion prediction is still unclear to me. The paper improves prediction scores, but does not show whether this helps a conversational system beyond recognizing the current emotion. This limits my understanding of the practical significance, though I would not consider it a rejection reason by itself, and my understanding might be limited by my background as well.
Questions for authors

The term “semantic world state” seems broader than what is demonstrated. The model learns features for current and next emotion prediction, and it is unclear what makes these features a world state rather than a representation for this task.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9TxA (1/3)

We thank the reviewer for recognizing the relevance of the research problem, the improvements over existing baselines, and the contributions of the component analyses. We address the concerns regarding architectural necessity, the practical significance of future emotion prediction, and the scope of the proposed semantic world state.

Weakness-1. Architectural Necessity and Supervision-Matched Validation
To demonstrate the necessity of the proposed architecture, we provide both methodological justifications and empirical evidence through controlled experiments, as detailed below.

1. Theoretical motivation and architectural rationale.
The motivation for PSWS arises from a distinction between current semantic recognition and future semantic prediction. Conventional Emotion Recognition in Conversation (ERC) primarily learns representations discriminative of the currently observed emotional label (yty_t). However, representations sufficient for recognizing (yty_t) are not necessarily sufficient for predicting (yt+1y_{t+1}). Two dialogue histories may share the same current emotion while containing different historical, prosodic, or cross-modal cues relevant to their subsequent emotional states. Since current-label supervision does not explicitly require these predictive distinctions to be retained, the resulting representations may become insufficiently discriminative for future prediction. We characterize this potential loss of transition-relevant information, such as Semantic Transition Aliasing (STA), as motivated in the Introduction (lines 45–52). Importantly, STA does not assume that every observed transition is predictable. The concern is whether a learned representation preserves the predictive information available in the observed history, rather than whether different realized next labels can always be distinguished. Addressing this problem requires three complementary capabilities, which directly motivate the architecture of PSWS.

(1.1). Preserving current semantics and transition-relevant evidence (TISA). A representation optimized only for the current label may underrepresent features that are irrelevant to current classification but informative about subsequent changes. TISA therefore introduces separate content and transition query banks to extract these two types of evidence. Cross-attention then integrates them into a common semantic state. The distinction is motivated by their different predictive roles, while the shared state preserves their interaction rather than treating them as independent predictions.

(1.2) Modeling heterogeneous transition mechanisms (ECTR). Emotional changes are sample-dependent and may be associated with linguistic context, acoustic variation, visual behavior, or cross-modal inconsistency. A single shared transition function does not explicitly distinguish these sources of evidence. ECTR therefore predicts modality-specific and interaction-specific state increments and combines them through evidence-conditioned routing. This design enables the transition function to adapt to the observed cues instead of applying a fixed transition pattern to every dialogue history.

(1.3) Learning predictive states without future-input leakage (LFSD). Standard conversational emotion datasets provide emotional labels but no direct supervision for latent semantic states. LFSD uses an EMA teacher to construct future-state targets during training and supervises both the predicted destination and the state increment. At inference, the student constructs and predicts states solely from the available history, without accessing future observations. \

These mechanisms address three distinct aspects of the same representation problem: preserving predictive evidence, modeling evidence-dependent semantic evolution, and grounding the predicted state with future-oriented supervision. Their combination is therefore motivated by the task requirements rather than by an arbitrary aggregation of architectural components.

2. Existing experimental evidence.
The submitted manuscript provides the following results from Tables 2 and 3:

MethodIEMOCAP-4 Future M-F1MELD Future M-F1
GraphSmile+Next79.1844.52
GraphSmile+Transition80.7245.51
GraphSmile+Transition+KD81.3246.12
PSWS w/o TISA79.6344.48
PSWS w/o ECTR80.5245.33
PSWS w/o LFSD79.8844.72
PSWS w/o (LidL_{\mathrm{id}})81.0946.01
Full PSWS82.55 ± 0.3847.42 ± 0.50

The GraphSmile comparisons show that introducing explicit transition prediction and future-state distillation progressively improves forecasting performance. The PSWS ablations further show that removing TISA, ECTR, LFSD, or the transition-identifiable objective consistently reduces performance relative to the complete framework.

AuthorsOpenReview ↗

Response to Reviewer 9TxA (2/3)

Removing LidL_{\mathrm{id}} decreases Future Macro-F1 by 1.46 and 1.41 points on IEMOCAP-4 and MELD, respectively. Moreover, Fig. 2 shows that full ECTR consistently outperforms shared, uniformly averaged, static, and discrepancy-agnostic dynamic routing. These results validate the contributions of transition-oriented representation learning and evidence-conditioned state evolution in PSWS.

3. Supervision-Matched Architectural Validation.
To disentangle architectural contributions from additional supervision, we introduce a stronger controlled baseline, GraphSmile+Transition+KD+(LidL_{\mathrm{id}}), which incorporates the same transition-identifiable contrastive objective as PSWS while retaining the original causal GraphSmile architecture and residual transition predictor. All configurations use matched multimodal inputs, data splits, future-label supervision, EMA-based state distillation protocols, contrastive sample construction, and causal inference constraints. The comparison is summarized below.

ArchitectureLidL_{\mathrm{id}}IEMOCAP-4MELD
GraphSmile+Transition+KD×81.3246.12
GraphSmile+Transition+KD✓82.0646.90
PSWS×81.0946.01
PSWS✓82.5547.42

The comparison separates the contribution of transition-identifiable supervision from that of the architecture. Under the scenario, introducing (LidL_{\mathrm{id}}) improves GraphSmile+Transition+KD by 0.74/0.78 points on IEMOCAP-4/MELD, confirming that additional supervision accounts for part of the improvement. Importantly, under matched supervision, PSWS retains advantages of 0.49/0.52 points, providing conditional evidence of an architectural benefit beyond the contrastive objective alone. Furthermore, (LidL_{\mathrm{id}}) yields larger gains within PSWS (1.46/1.41 points), corresponding to positive architecture–supervision interaction differences of 0.72/0.63 points.

The experimental results above demonstrate that TISA and ECTR are not only grounded in clear methodological motivations but also provide representations and state-evolution structures that are better suited for leveraging transferable, identifiable supervision. Consequently, the full benefits of PSWS cannot be attributed solely to either the additional supervision or the architectural complexity; Rather, they arise from the synergistic interaction between the two.


Weakness-2. Architectural Necessity and Supervision-Matched Validation
We appreciate this concern and believe that the distinction between two established research tasks in conversational affective computing is important for understanding the motivation.

Emotion Recognition in Conversation (ERC) estimates the emotion of an utterance that has already been observed. By contrast, Emotion Prediction in Conversation (EPC) predicts the emotional state of a forthcoming utterance from the preceding dialogue history, before that utterance becomes available. EPC is an established research problem rather than a task introduced solely by PSWS. Shi et al. [1] investigated future emotion prediction using the interactive context of interlocutors. Their subsequent work [2] extended this line of research to multi-utterance turns and multi-speaker conversations. Shi et al. [3] further investigated lexical, prosodic, and multimodal cues for conversational emotion prediction. These studies, discussed in Appendix A.2, demonstrate that anticipating future emotions is a distinct objective in conversational affective understanding. The practical distinction concerns when information becomes available. ERC identifies an emotional change after it has been expressed, whereas EPC estimates the likelihood of a forthcoming change from the already observed context. For example, two currently neutral dialogue histories may contain different patterns of vocal tension, lexical context, or interaction dynamics. Recognizing both as neutral does not determine whether the next emotional state is likely to remain neutral or shift toward frustration. A predictive model can instead estimate the possible next emotional state before the next utterance is observed.

This capability has potential relevance to anticipatory dialogue understanding, early emotion-change monitoring, and context-sensitive response preparation. Crucially, these applications require predictive information to be available before the forthcoming utterance, which cannot be provided by recognition of that utterance after it occurs.

AuthorsOpenReview ↗

Response to Reviewer 9TxA (3/3)


PSWS targets this setting through explicit semantic-state evolution. Rather than directly mapping historical features to a future label, it constructs a current semantic state, predicts a residual state increment, and decodes the predicted next state within the same semantic label space. This formulation additionally allows transition-relevant representation learning to be evaluated through state separation and future prediction. Under the unified causal protocol, PSWS achieves Future Macro-F1 scores of 82.55% and 47.42% on IEMOCAP-4 and MELD, respectively, outperforming GraphSmile+Transition+KD by 1.23 and 1.30 percentage points. Appendix C.2 also reports contextual comparisons with representative EPC methods, while explicitly acknowledging that their modality configurations differ and therefore do not constitute fully controlled comparisons. We emphasize that improved offline future prediction does not by itself demonstrate better dialogue policies, user satisfaction, or deployed conversational performance. We consequently restrict the established empirical contribution to past-conditioned one-step semantic prediction and transition-aware representation learning. The downstream conversational applications described above remain potential uses.

References:

[1]. Xiaohan Shi, Sixia Li, and Jianwu Dang. Dimensional Emotion Prediction Based on Interactive Context in Conversation. In Interspeech 2020, pp. 4193–4197, 2020.

[2]. Xiaohan Shi, Xingfeng Li, and Tomoki Toda. Emotion Awareness in Multi-utterance Turn for Improving Emotion Prediction in Multi-Speaker Conversation. In Interspeech 2023, pp. 765–769, 2023.

[3]. Haoxiang Shi, Ziqi Liang, and Jun Yu. Emotional Cues Extraction and Fusion for Multi-modal Emotion Prediction and Recognition in Conversation. In Interspeech 2024, pp. 4074–4078, 2024.


Question-1. Definition and Scope of "Semantic World State"
We appreciate the concern regarding the breadth of the term. Importantly, the submitted manuscript already provides an explicit task-restricted definition and repeatedly distinguishes PSWS from general-purpose world modeling.

  • First, the Introduction (lines 95–99, page 2) defines PSWS as a one-step predictive multimodal semantic-state model. It explicitly states that PSWS is not intended to reconstruct the physical world or infer unobservable private mental states, but instead learns a compact semantic state grounded in observable multimodal interactions.

  • Second, Section 2.1 (lines 119–137, page 3) formalizes the state construction and prediction process: Ht→St→S^t+1.H_t\rightarrow S_t\rightarrow\hat S_{t+1}. Equation (1) specifies that the learned state is intended to preserve information relevant to both current and future semantic decoding. The same section explicitly characterizes PSWS as an action-free, task-relevant, and one-step semantic world-state model, rather than a model of complete physical environments or long-term stochastic futures.

  • Third, Figure 1 (page 2) and Sections 2.3–2.6 (pages 3–4) operationalize this definition. TISA constructs the current state from observed multimodal history, ECTR predicts its residual evolution, LFSD supplies training-only future-state supervision, and the shared decoder grounds current and predicted states in the same semantic label space.

  • Finally, the Conclusion (lines 479–485, page 9) explicitly limits the approach to task-grounded, deterministic, short-horizon prediction and excludes claims concerning complete future distributions or action-conditioned physical dynamics.

  • Accordingly, the term "world state" in PSWS refers specifically to an observable-history-conditioned, task-grounded predictive semantic state, whose defining properties are semantic grounding and explicit one-step evolution. It does not refer to comprehensive physical-world reconstruction, general Markov sufficiency, or action-conditioned simulation. We agree that the broader terminology may invite expectations beyond the demonstrated scope. We therefore emphasize the qualified interpretation of task-grounded predictive semantic state, consistent with the definitions and limitations already stated in the submitted manuscript.

We sincerely thank the reviewer 9TxA again for the valuable comments and constructive feedback. We hope our clarifications adequately address the concerns raised and earn the reviewer's support. We would be happy to provide further explanations and welcome any additional discussion if needed. Thank you again for your time and consideration.

Reviewer_9TxAOpenReview ↗

Review update

Thank you for the additional experiments. The new baseline addresses my main concern. The limits of the application claims are also clearer now.

I still share another reviewer’s concern about how the predictions two or three utterances ahead are made. This needs clarification, but it does not seem to affect the main results.

Overall, I will raise my score.

Comments on the paper

1 comment

Dear Reviewers, as the discussion deadline is approaching, please review the authors’ responses and join the discussion. Please share whether your concerns have been addressed and update your assessments as appropriate. Thank you for your time and contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.