- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PSYCHE, a multimodal conversational emotion recognizer that models each speaker's emotion as a continuous latent trajectory. Its Causal Source Decomposition separates emotional inertia from influence by the interlocutor, and a Neural Jump ODE uses these drivers to update the state between utterances while speech acts create discrete jumps. The model also outputs a self/other balance and trigger attention. On IEMOCAP and MELD, it reports weighted F1 scores of 72.51 and 67.55, with additional ablations and attribution analyses.
Critical strengths
- The core idea is easy to grasp: emotion can carry over between turns and also be affected by the other speaker.
- CSD makes those two sources visible in the model instead of hiding them in one blended context vector.
- The causal mask and pre-speech state design are thoughtful choices for avoiding future-turn leakage.
- The paper includes useful ablations and dynamics analyses, not just benchmark scores.
Critical weaknesses
1.The main comparison gives PSYCHE extra offline commonsense features. Without both cues, its IEMOCAP W-F1 is 70.74, below GraphSmile's 71.32, so the headline gain is hard to attribute to the new architecture alone. 2.The paper calls the outputs causal attributions, but the evidence is mostly correlations with annotations. Attention weights and a learned gate do not by themselves show that a particular utterance caused the emotion change. 3.The gate is trained with an LLM-derived prior and losses that encourage lower self-ratios at shifts. That makes the later shift analyses less independent as evidence for the gate's meaning. 4.It is still unclear what the ODE adds beyond CSD and its losses. Removing the ODE costs 2.30 W-F1 points, while removing CSD costs 4.97; a matched discrete-time version would help isolate the contribution. 5.The main table uses the best validation checkpoint, while the reported three-seed mean is lower. Paired significance tests or consistent multi-seed results for the strongest baselines would make the small margins easier to judge. 6.The RECCON trigger results need more context in the main paper, especially the number of evaluated cases, subset construction, and variation across seeds. 7.The offline LLM features and attribution prior could affect results in ways that are hard to reproduce or audit. The paper should make the exact prompts, visible context, filtering, and human checks easy to verify.
Questions for authors
- Can you provide an apples-to-apples comparison in which the strongest baselines receive the same entity- and scene-level commonsense features, as well as a clear comparison using no commonsense features? Please state exactly what dialogue context was visible when the offline LLM generated each feature.
- Can you add a discrete-time baseline that preserves the self/other split, CSD-style drivers, and training losses but removes the Neural Jump ODE? This would isolate whether the continuous-time formulation contributes beyond the source decomposition and its auxiliary objectives.
- For the RECCON trigger evaluation, how many transitions and candidates are used, how is the evaluated subset selected, and what are the mean and standard deviation across seeds? Please also clarify whether any trigger labels or RECCON-derived information influence training, including the offline attribution prior.
4 comments
Response to Reviewer cNzX [2/2]
We appreciate this comment. We agree that the gate is trained with an LLM-derived prior and losses that encourage lower self-ratios at shifts. To check whether this training makes the shift analysis less independent, we remove these losses and test whether the shift-vs-stay ordering survives. It survives: a variant trained without both and still gives of at shifts versus at stays, a gap of with , nearly matching the full model's gap of (Figure 6, see the revised manuscript, around L.504). So the independence is not reduced: the shift analysis remains independent of these losses. The trigger attention makes this independence even clearer: it receives no attribution supervision whatsoever, the sole attribution anchor regressing only and never , yet it localizes the trigger with NDCG and MAP against RECCON. The localization is thus emergent and unsupervised, independent of any training objective.
We appreciate this comment. The row you cite, removing the ODE costs 2.30, is in fact the discrete-time baseline you ask for: it keeps the self/other split, the CSD drivers, and all training losses, and only replaces the Neural Jump ODE with a direct state update. We apologize that the earlier label "-w/o ODE" did not make this clear. We have relabeled it "-w/o ODE (discrete-time)" in Table 7 and report the construction of every ablation in Appendix 3.6. It scores 70.21 W-F1 on IEMOCAP, 2.30 below the full model, so the continuous-time formulation contributes 2.30 points on top of the decomposition and its auxiliary objectives.
We appreciate this comment. The three-seed results are reported in Appendix 3.3:
| Method | seed 22 | seed 1246 | seed 3407 | mean ± std |
|---|---|---|---|---|
| PSYCHE | 71.77 | 71.90 | 72.51 | 72.06 ± 0.40 |
| GraphSmile | 69.63 | 71.01 | 71.32 | 70.65 ± 0.90 |
| ECERC | 70.13 | 69.12 | 70.24 | 69.83 ± 0.62 |
| EMART | 68.52 | 67.37 | 68.02 | 67.97 ± 0.58 |
PSYCHE still leads the strongest baseline GraphSmile by 1.41 points on the three-seed mean, and its standard deviation (0.40) is the smallest, so the lead is stable across seeds.
These background details are now reported in the manuscript and the Appendix.
1. Evaluation subset and scale. The human-verified subset contains transitions, built from RECCON's 665 annotated emotion-cause transitions by keeping the inter-personal and hybrid ones with at least one candidate utterance and a gold cause inside the candidate window, and then the 41 with candidates. No random sampling is involved: the 41 transitions are the exhaustive set satisfying the criteria above. Because is small, we supplement it with a large-scale offline-annotated proxy evaluation, covering IEMOCAP and MELD valid samples.
2. Stability and significance. Across the three training seeds, the results are STable 5, Appendix, Sec. 3.7:
| Seed | 22 | 1246 | 3407 | mean ± std |
|---|---|---|---|---|
| NDCG | 0.8113 | 0.8372 | 0.8290 | 0.826 ± 0.013 |
| MAP | 0.7677 | 0.7749 | 0.7683 | 0.770 ± 0.004 |
On the seed-3407 checkpoint we bootstrap the transitions and obtain 95% confidence intervals of NDCG and MAP . A paired bootstrap against the Uniform baseline gives (NDCG) and (MAP).
3. Independence from training. No trigger label or RECCON-derived information influences training: the trigger attention receives no attribution supervision, and the offline attribution prior only anchors the self-cause ratio , never .
We would be grateful if the reviewer would revisit the rating in light of these.
Thank you for the author's response. I am keeping my original rating.
Response to Reviewer cNzX
We sincerely thank the reviewer for taking the time to carefully consider our responses. We are delighted that our clarifications and additional experimental results have addressed your concerns and strengthened your confidence in our work. We greatly appreciate your recognition and support.
Response to Reviewer cNzX [1/2]
We thank the reviewer for the careful reading and the positive assessment of our core idea. We are glad it came across as easy to grasp, because that is exactly how we arrived at it: we asked how a person would explain an emotion change, and the continuous ODE was simply the natural way to carry that explanation, keeping the model small. We have added the requested controlled comparisons and disclosures, as detailed below.
1. Fair comparison.
To examine whether the headline gain comes from the architecture itself or from the extra commonsense cues, we add two controlled comparisons.
(i). Granting the strongest baselines the same cues. We retrain GraphSmile and ECERC with the same commonsense cues, as reported in STable 5, Appendix, Sec. 3.7:
(STable 5: IEMOCAP W-F1 %. com. = commonsense cues.) The commonsense cues help GraphSmile by +0.96 and leave ECERC nearly unchanged, at +0.21, while PSYCHE still leads at 72.51.
(ii). The CSD gain is independent of the cues. A 2$\times$2 cross-ablation that varies the commonsense cues and CSD isolates the CSD contribution, as reported in Table 6, Sec. 4.3, around L.544:
(Table 6: IEMOCAP W-F1.) The CSD gain is +4.97 with the cues and +4.41 without them, i.e., CSD contributes to accuracy independently of the cues. This does not make the cues redundant: they serve a complementary role, the situational grounding that separates, e.g., composure at a funeral from composure that masks disappointment. This is why knowledge enhancement is a common practice in MERC, such as MKE-IGN [1] and KAMIN$^{[2]}$.
Together, the two experiments show that the headline gain comes from the architecture, not the added cues. To be transparent, we also state in the manuscript that without any cue, PSYCHE reaches 70.74, slightly below GraphSmile's 71.32. This gap does not change the core contribution. The cues provide what the conversation is about, whereas CSD provides the source-level interpretability the cues cannot substitute. Its self-cause ratio and trigger attention, validated against RECCON in our reply to Q3, decompose how and why an emotion changes and are read directly off the forward pass without any language model. This interpretability, not a raw classification margin, is the core contribution.
2. Audibility of the LLM-generated cues.
Regarding the LLM context used to generate the cues, the entity-level cue is generated from the full dialogue transcript, and the scene-level cue from only the dialogue-onset background, namely the speaker identities, the setting, and the power dynamics, with no future utterance outcome visible. The exact prompts, JSON schemas, and extraction scripts are released in the Appendix 2.3.
We appreciate this comment on the evidence for the attribution. We agree that the attention weights and the gate are correlational rather than causal: they localize the utterance most associated with an emotion change, and we validate this localization against human annotations, but we do not claim that they prove, in the intervention sense, that this utterance caused the change. In the paper, "causal" modifies the source decomposition, and the attribution is source-level; we did not intend "causal attribution" as a claim of intervention-level causation. Sec. 3.2 states the scope explicitly as time-admissibility of causal effect rather than causal discovery. Φtself and Ψtother are drivers, the two source-conditioned inputs to the Neural Jump ODE, and the only causal assumption we rely on is time-admissibility, that a cause precedes its effect in time and is attributable to a source. We validate the localization against independent human annotations of the emotion cause, RECCON, the evidence appropriate for a recognition model whose aim is interpretability.