acceptodds
Under review as a conference paper at ICLR 2027

TD-AFGW: TONE-AWARE MULTIMODAL EMOTION TRANSFER WITHOUT TARGET EMOTION LABELS

Abstract

Transferring emotion recognition to tonal languages with scarce annotations requires addressing cross-lingual representation shift and the overlap between lexical tone and emotional prosody. We introduce , a multimodal framework trained with labeled source conversations and target conversations without emotion labels. A text-conditioned pitch predictor, ToneNet, estimates a lexical-tone baseline and supplies residual pitch features and feature-wise acoustic conditioning. Anchored fused Gromov–Wasserstein alignment combines within-domain relations with fixed facial and paralinguistic cues. The transport plan propagates source labels, while a tone-swap consistency gate selects target soft labels for distillation. A visual-to-audio-to-text curriculum progressively enables adaptation. We study English-to-Mandarin transfer from MELD to EmotionTalk, using Mandarin as an established tonal-language benchmark for task-specific annotation scarcity. Under shared source-prior logit calibration, the full model achieves weighted F1 and macro F1 across three seeds, versus and for source-only transfer. Ablations support the contributions of anchored geometry, ToneNet, and consistency filtering. Per-class results show particularly strong gains for anger and surprise, alongside persistent minority-class weaknesses. The findings support tone-aware transfer on this benchmark; broader low-resource generalization and fully target-label-blind model selection remain to be established.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal Emotion Recognition, Cross-Lingual Transfer Learning, Unsupervised Domain Adaptation, Tonal Language Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_qmYsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper try to simulate transferring emotional classification model to seen tonal languages. To do that, this model utilizes source model as trained with English dataset (MELD dataset), and transfering it into Mandarin (Emotiontalk dataset).

Critical strengths

None.

Critical weaknesses
  1. While the paper mentioned that "it does not imply that Mandarin is generally a low-resource language or reproduce the absence of transcripts and pretrained models in many low-resource settings.", but I am not sure it is adoptable to other languages. From the beginning, ToneNet get "syllable" as an income. However, for low-resource languages, there is no meaningful syllable extractor. Also, after syllable extraction, it calculates surface tone by sandhi processing, but it also might not eligible to low-resource languages. In addition, they mentioned about low-resource language, but their "unsupervised dataset" is larger than English dataset.

  2. Why ANCHORED FUSED GROMOV–WASSERSTEIN ALIGNMENT is utilized? In section 4.3, it just mentioned AFGW, and did not mention why it is meaningful target. Also few variables, such as zz, are not defined.

  3. The figures appears to heavily AI-generated and does not provide any additional visual insights. Figures 1, 2, and 3 all appear to be heavily AI-generated.

  4. Results are too weak to demonstrate. All results shows low accuracy than "Always neutral". Meanwhile, all other models' accuracy is lower than 40% for 7-class classification, I am concerning that showing weighted F1 and macro F1 is meaningful. In addition, the proposed method does not have meaningful differences with DANN or MMD (citation is needed).

  5. Missing confusion matrix. While many previous studies on emotion classification/generation present confusion matrix, this paper does not include one.

  6. Writing issue The method section is largely diffult to follow, as the authors describe the methods without sufficiently explaining the motivation behind their design choices.

Questions for authors

Please refer to the Weakness section.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer qmYs

Question 1:

Answer: Thank you for raising this. We agree that it is a real limitation, and you are right that our “low-resource” framing was too strong. • Syllables and tone sandhi. ToneNet does need syllable-level text processing. For Mandarin we use pinyin with rule-based sandhi, which matches the human-annotated surface tones of AISHELL-3 on 96.8% of syllables (94.1% without the sandhi rules) (Appendix A). Many low-resource languages do not have such tools, so the method in its current form is not directly applicable to them. Section 3 now states that the method still relies on transcripts, linguistic tone processing, speech alignment and neutral target-language speech, and that a setting without these resources is outside what we evaluated. • A second tonal language. To check whether the idea carries over at all, we repeated the full pipeline on Vietnamese, where syllables are written separately and tones are marked in the spelling, so no syllable extractor or sandhi rules are needed (Section 5.4; Appendix Tables 16 and 17). The method did not help there either: it is 1.44 W-F1 below a fairly selected source-only model. • Dataset size. You are right that the unlabeled EmotionTalk pool (12,551 clips after removing the holdout) is larger than the MELD training set (9,966 utterances), and Mandarin is not a low-resource language. We added a label-budget experiment: fine-tuning with only 1% of the target labels (126 clips) already reaches 53.22 W-F1 on test, far above every zero-label adaptation method, and zero-shot emotion2vec+ reaches 46.85 (Section 5.5; Appendix Table 18). Section 6 now states clearly that, on this benchmark, a small labeled budget or an off-the-shelf model is the better practical choice.

Question 2:

Answer: Thank you, this is a fair point. We added a short motivation at the start of Section 4.3: Gromov–Wasserstein matching can compare the structure inside each domain without a shared feature space, but it is ambiguous up to symmetries, while a few fixed cues that are assumed comparable across languages can break those symmetries. We also now define α (the weight of the relational term against the anchor term) and ε (the entropic regularization) right after Eq. (6), and β1–β3 right after Eq. (12). The variable name was cut off in the review text, so if there is another symbol we missed, we would be glad to fix it. We also want to be honest that the new experiments do not support this motivation in practice. The coupling carries almost no class information (its soft labels are correct for 35.3% of development clips, against 35.4% for always predicting neutral), and the anchors are only weakly comparable between MELD and EmotionTalk (Section 5.3, “Coupling soft labels carry no class information”; Appendix Tables 4 and 15).

Question 3:

Answer: Thank you for the honest feedback. Figures 1–3 are schematic diagrams of the method, drawn as TikZ code; as stated in our AI use statement, generative AI helped with the figure code and layout. We agree that they mostly restate the text. In this revision we kept them as method overviews, but we added figures that carry evidence and are drawn directly from the run outputs: Figure 4 (development W-F1 during training, showing the decline in Stage 2 and the collapse in Stage 3) and a re-drawn Figure 5 (ablation results with reference lines), both in Appendix C. If you feel that Figures 1–3 do not earn their space, we are happy to simplify or remove them in the final version.

Question 4:

Answer: Thank you. You were right, and the re-run shows it even more clearly. Under the corrected protocols the method does not beat simple baselines. • Accuracy against always-neutral. Always-neutral reaches 41.52% accuracy on the test set. Without target labels, TD-AFGW reaches 41.47% only because it predicts neutral for more than 99% of clips (Appendix Table 6). With development-based selection it reaches 45.01%, against 42.37% for a source-only model selected the same way and 45.34% for MMD. • Metrics. Because accuracy is dominated by the neutral class (41.5% of the test set), we report weighted and macro F1 next to two simple references, an always-neutral predictor and a class-prior random predictor (Table 1). We also give per-class F1 (Appendix Table 5), confusion matrices (Appendix Table 6) and paired bootstrap intervals over clips and over actors (Appendix Table 7). • DANN and MMD. All baselines were re-trained under the new protocols with three seeds. TD-AFGW is not better than them: under Protocol B, MMD reaches 46.09 W-F1 on test against 43.76 for TD-AFGW (difference −2.33, clip-bootstrap interval [−3.86, −0.85]), and DANN is on par (43.13). We added the citations for DANN, MMD and OT-based adaptation where the baselines are introduced (Section 5.1, “Baselines and metrics”). We removed all claims of improvement from the Abstract, Section 1 and Section 7.

AuthorsOpenReview ↗

Author Response to Reviewer qmYs

Question 5:

Answer: Thank you, this is a good suggestion. We added confusion matrices (row-normalized, pooled over three seeds) in Appendix Table 6: for Mandarin under Protocol A (source-only, MMD and TD-AFGW) and for Vietnamese (source-only and TD-AFGW). They make the failure easy to see: the Mandarin TD-AFGW model predicts neutral for 98.8–99.8% of the clips of every class, while MMD spreads its predictions much more evenly. They are referenced in Section 5.2 (“Classes”).

Question 6:

Answer: Thank you, we accept this criticism. Within the page limit we made targeted changes: a motivation for anchored FGW (Section 4.3), definitions of α, ε and β1–β3 (Sections 4.3 and 4.5), a clearer description of how the tone-swap contour is built (Section 4.4), and a rewritten paragraph on the decision rule (Section 4.5, “Decision rule”). Most of the revision effort went into the evaluation, because the new results changed what the paper can claim. Section 5.3 now explains, component by component, what each part was meant to do and what we actually measured. We will keep improving the clarity of the method section for the final version. Thank you again for your time and for the detailed review.

Reviewer_qmYsOpenReview ↗

Response to the author's comments

Dear Authors, Thank you for your revisions and responses to my comments. However, I still have several concerns that have not been adequately addressed

  1. Thank you for including additional tonal languages, but the results on the Vietnamese data do not appear to demonstrate meaningful improvements from the proposed method.
  2. I am still struggling to understand the motivation for applying AFGW.
  3. Whether the figures were created using Tikz or AI-generated tools was not the main point of my comment. My primary concern is that these figures do not "provide additional visual insights" with the pattern of heavily AI generated figures. I recommend to revise all figures from ground for next submission. 4/5. As shown in table 6, 99% of the proposed model's predictions are biased to the neutral. Given this severe class imbalance in the predictions , I do not believe the reported results demonstrate a meaningful improvement from the proposed method.

Therefore, I will maintain my current score.

Reviewer_D3vCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TD-AFGW for transferring multimodal emotion recognition from English to the tonal language Mandarin without target-domain emotion labels. The method uses only text to predict a baseline pitch for lexical tones, producing residual pitch features and FiLM conditioning. Then uses fixed facial and paralinguistic features as cross-domain anchors and combines them with within-domain relational structure for optimal transport alignment. And then propagates source-domain soft labels through the transport plan, filters pseudo-labels using a tone-swap consistency gate, and gradually adapts with a visual-to-audio-to-text modality curriculum.

Critical strengths
  • Linking lexical tone and emotional prosody is an interesting topic, which has rarely been explicitly modeled.

  • The inference does not require source samples; the deployment cost is low.

  • Good evaluation consists of different metrics and ablation studies.

Critical weaknesses
  • Baselines are outdated. DANN and MMD are very old unsupervised domain adaptation methods. Why not use multimodal large language models? Also, recent audio language models have already shown capabilities in the area on which this paper focuses.

  • The framework consists of existing methods. FGW, OT, and FiLM are all existing techniques. This paper reads more like an engineering effort rather than a fundamentally new paper with real novelties.

  • There is only one language mapping between English and Mandarin. Also, Mandarin is not a resource-limited language. Thus, the motivation for "without target emotion labels" becomes weak.

Questions for authors
  • Could you add a supervised upper bound on the target domain, as well as stronger, more recent baselines (such as zero-shot multilingual LLMs)?

  • How is F_swap selected in tone-swap, is it randomly or for a specific tone?

  • Since changing a word’s lexical tone may also changes its meaning, this could make it inconsistent with the transcription. How does this affect the validity of the gating mechanism? Have you evaluated the effect of artifacts introduced by WORLD resynthesis?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer D3vC

Question 1:

Answer: Thank you, these comparisons turned out to be very informative. We added them in Section 5.5 and Appendix Table 18, on the same test and holdout splits. • Supervised upper bound. Training on all adaptation labels reaches 60.41 W-F1 on test (63.33 on the holdout), and fine-tuning the source model with 100% of the labels reaches 61.36 (63.56). • Label budget. With 1%, 5% and 20% of the target labels, fine-tuning reaches 53.22, 55.49 and 58.38 on test. Even 1% (126 clips) is far above every zero-label adaptation method (the best is MMD with 46.09; TD-AFGW reaches 43.76). • Zero-shot models. emotion2vec+ large (audio only) reaches 46.85 (48.23 on the holdout), Qwen2.5-7B-Instruct on the transcript reaches 47.89 (45.87), and Qwen2-Audio-7B-Instruct with audio and transcript reaches 35.40 (35.10). • Vietnamese. A supervised model reaches 58.11 W-F1, 17 points above the fairly selected source-only model (Appendix Table 16). These results also support your point about motivation: on Mandarin, a small labeled budget or an off-the-shelf model is the better practical choice. We now say this in the Abstract and in Section 6.

Question 2:

Answer: Thank you for asking; our original text was not clear. The replacement tone is chosen at random, not for a specific tone. Each syllable with tone 1–4 receives a different tone drawn at random (with a fixed seed), and neutral-tone syllables are kept. ToneNet then predicts the contour for this new tone sequence, and the contour is placed on the syllable boundaries from CTC forced alignment, while the residual pitch is kept. This is now written out in Section 4.4, and the implementation details are in Appendix A (“Counterfactual controls”).

Question 3:

Answer: Thank you, this question was spot on. We tested it directly with two control interventions: a vocoder-only resynthesis with no pitch change, and a random pitch perturbation with the same magnitude as the tone swap (Appendix A). What we found: • The gate’s divergence correlates 0.76 with the vocoder-only divergence and 0.74 with the random-pitch divergence, so it mostly measures sensitivity to resynthesis rather than to lexical tone. • At every admission rate, soft labels admitted by the tone-swap gate are no more accurate than randomly admitted ones (AUROC 0.50, against 0.47 for random pitch and 0.47 for vocoder-only) (Appendix Tables 4 and 11). • Training with a random-pitch gate, a gate that removes vocoder artifacts, or random admission at the same rate gives the same results as the tone-swap gate, within 0.12 W-F1 (Appendix Table 11). • In practice, at τ = 0.20 the gate admits 98–100% of clips, so it hardly filters anything (Section 5.3; Appendix Table 10). So, as you suspected, stability under this intervention does not show that a label is correct, and a possible mismatch with the transcript only adds to the problem. We removed the claim that the gate selects better pseudo-labels (Sections 5.3 and 6). Thank you again for the constructive review.

Thank you for the detailed response and the new experiments, which answer my questions. However, the revised results show the method does not beat simple baselines or a small labeled budget, and the novelty concern remains, so I keep my original rating.

Reviewer_ogpXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TD-AFGW, a framework for cross-lingual multimodal emotion recognition without target emotion labels. The method combines ToneNet for text-conditioned lexical-pitch modeling, anchored Fused Gromov–Wasserstein alignment for cross-domain representation matching, and tone-swap consistency filtering for pseudo-label distillation. The authors evaluate English-to-Mandarin transfer from MELD to EmotionTalk, reporting improvements from 27.59 to 31.42 weighted F1 and from 15.29 to 18.01 macro F1 over source-only transfer. Component ablations suggest that the anchor, ToneNet, consistency filtering, and modality curriculum contribute to the overall performance.

Critical strengths
  1. Well-motivated treatment of lexical tone in cross-lingual emotion recognition. The paper addresses an important challenge in tonal-language adaptation, where lexical tone and emotional prosody share F0 information. Combining text-conditioned pitch prediction with multimodal alignment is a reasonable and potentially useful approach.

  2. Relatively comprehensive component analysis. The paper evaluates multiple domain adaptation baselines, reports results across three seeds, and includes component ablations, class-level performance, and source-domain retention. These analyses help identify the contributions and limitations of the proposed framework.

Critical weaknesses
  1. The target-label-free evaluation protocol is not sufficiently established. Although target emotion labels are excluded from training losses, Section 5.1 and Section 6 acknowledge that target test trajectories and threshold comparisons were inspected. In Appendix C.2, the selected tone-swap threshold τ=0.20\tau=0.20 also achieves the highest target test W-F1 among the evaluated settings. The paper does not establish that checkpoint selection, threshold tuning, and calibration were determined independently of target labels. This is particularly important because the main contribution is evaluated under a zero-target-emotion-label setting. A fully specified label-free selection procedure, ideally verified on an untouched test set, is needed.

  2. The proposed tone-disentanglement mechanism lacks direct validation. ToneNet is intended to separate predictable lexical-tone variation from affective prosody, but the experiments only establish a system-level benefit from including ToneNet. Its held-out pitch prediction achieves Pearson r=0.357r=0.357 and MAE of 2.685 semitones (Appendix C), suggesting substantial unexplained pitch variation. There is no direct evaluation showing that residualization reduces lexical-tone information while preserving emotion-related information. Furthermore, the ToneNet ablation removes both FiLM conditioning and residual-based anchor features, making their individual contributions unclear. Controlled tone-predictability and emotion-preservation experiments would strengthen the central methodological claim.

Additional concerns: Evaluation is limited to one language pair and a single target corpus. Absolute macro F1 remains low (18.01%), with particularly poor performance on sadness, fear, and disgust. Source-domain W-F1 also decreases from 49.80 to 46.26 after adaptation. These results limit claims about robust emotion transfer and generalization to other tonal languages.

Questions for authors
  1. How exactly were the final checkpoint, tone-swap threshold, and calibration parameters selected without target emotion labels? Were any decisions influenced by the inspected target test results?

  2. Can the authors measure lexical-tone predictability and emotion predictability before and after ToneNet residualization? Does the residual representation retain emotion information while suppressing lexical tone?

  3. Can the authors separately ablate FiLM conditioning and residual-pitch anchor features to establish which component is responsible for the reported improvement?

  4. How robust are the gains to a different tonal-language dataset or cross-corpus transfer setting? Given the low absolute macro F1, can the authors clarify which emotion classes benefit reliably from adaptation?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer ogpX

Question 1:

Answer: Thank you for asking this so directly; it was the most important question for us. We have to admit that, in the submitted version, decisions were influenced by the test set: the Stage 2 and Stage 3 checkpoints were kept by their test weighted F1, and the calibration coefficient and the two class offsets were picked by a grid search on test labels. τ = 0.20 was the default value, but the threshold comparison was inspected on the test set, so we cannot show that it was fixed beforehand. The unlabeled adaptation set was always disjoint from the test set, so the problem was selection, not data overlap. We withdraw all of those numbers and replace them with the re-run results below. In the revision (Section 5.1, “Protocols”; Appendix A, “Pre-registration and unsealing”): • All selection rules were written and hashed before training. Protocol A uses no target labels at all: Stage 1 is chosen by MELD development cross-entropy, Stages 2 and 3 use their last epoch, τ = 0.20, and predictions are raw argmax (no calibration, no class offsets). Protocol B uses only the labeled EmotionTalk development split: epochs are chosen on it, one λ_cal (0.7) is fitted on it with the source-only models, δ = 0, and the source-only model gets the same development-based selection so that the comparison is fair. • Test and holdout labels were locked until all predictions were frozen and hashed, and then opened once. The holdout (groups G13 and G14, with 4 actors who appear nowhere else) is new: in the submitted version these clips were only part of the unlabeled adaptation pool, and their labels had never been used for evaluation or for any selection. They are now also removed from the adaptation pool. • Results: under Protocol A, TD-AFGW collapses to 24.68 / 24.29 W-F1 (test / holdout) against 39.12 / 32.61 for source-only; under Protocol B it reaches 43.76 / 41.51 against 43.16 / 38.89 for the fairly selected source-only model, and it stays below MMD (Table 1). • We also checked the label-free criteria described in the paper (Appendix Table 9). The FGW cost is a useful label-free criterion for Stage 2 (43.47 on test, against 33.08 for the last epoch), but no criterion, even one that uses labels, rescues Stage 3. The claim of a gain under label-blind selection is no longer supported, and the Abstract, Section 1, Section 5 and Section 7 now say so.

Question 2:

Answer: Thank you, this was a key missing analysis, and the answer is not what we had claimed. We added probes in Section 5.3 (“Residualization on conversational speech”) and Appendix Table 14. • Read speech. On held-out AISHELL-3 speakers the residual does remove much of the lexical tone: tone-probe accuracy drops from 65.6% (measured contour) to 39.7% (residual), against a majority rate of 29.2%. Vietnamese read speech behaves the same way (76.8% to 38.8%). • Conversational target speech. Here ToneNet fits poorly (Pearson r drops from 0.525 to 0.208; Appendix Table 13), and the residual predicts tone better than raw pitch: 37.3% to 47.1% (majority 34.7%). In Vietnamese the residual still carries much of the tone signal (53.9% to 43.6%, majority 27.2%). • Emotion information. It stays roughly the same and close to chance: pitch-statistic probes give 18.6 macro F1 for raw pitch and 19.5 for the residual (chance 14.3); in Vietnamese, 14.8 and 17.1. • We also checked how much of the predicted contour leaks into the residual statistics (R² 0.145 from the residual against 0.053 from raw pitch) and how much the pitch tracker matters (Harvest and DIO residuals correlate 0.88). So, on the speech we actually adapt to, the residual does not suppress lexical tone while keeping emotion. We removed that claim (Sections 5.3 and 6).

Question 3:

Answer: Thank you, we ran exactly this. Table 2 now has two extra rows that remove only FiLM or only the residual from the anchor (also shown in Appendix Figure 5). On Mandarin (Protocol A), removing ToneNet changes test W-F1 by +2.00, and almost all of that comes from removing FiLM (+1.89), while removing only the residual from the anchor changes almost nothing (+0.08). On Vietnamese, the ToneNet ablations are within seed variation (−0.35 and 0.00). Because the full model has collapsed under Protocol A, these numbers describe how the components change a failing Stage 3, not what helps transfer. In any case, neither pathway is responsible for an improvement, so we removed the claim that ToneNet contributes to transfer (Section 5.3, “Ablations”).

AuthorsOpenReview ↗

Author Response to Reviewer ogpX

Question 4:

Answer: Thank you for both questions. • Robustness. We evaluated on an untouched EmotionTalk holdout (4 unseen actors) and on a second tonal language, Vietnamese (23,347 clips from drama series and talk shows; Section 5.4; Appendix Tables 16 and 17). The gains do not hold up. On Vietnamese, TD-AFGW is 1.44 W-F1 below a fairly selected source-only model and higher on only one of three seeds, and on the talk-show part it is close to always predicting neutral. MMD is the only method with a small but consistent gain (+0.57). • Classes. Under Protocol B, the full model gains on neutral and joy on every seed and on both evaluation sets (joy +2.18 on test and +8.51 on the holdout) and loses on surprise and disgust on every seed (disgust −7.62 and −5.69); fear, sadness and anger are mixed (Section 5.2, “Classes”; Appendix Tables 5 and 6). On Vietnamese it gains only on joy (39.61 against 29.37) and loses on every other class, most on neutral (54.94 against 60.72), disgust (8.53 against 11.96) and sadness (27.29 against 29.96) (Appendix Table 17). • Low macro F1 and weak classes. Macro F1 is 33.34 on test under Protocol B (8.81 under Protocol A). Disgust (11.07) and fear (18.44) remain the weakest classes, and disgust is worse than the reference on every seed (Appendix Table 5). So we cannot point to an emotion class that benefits reliably, beyond a shift toward the frequent neutral and positive classes, and the paper now says this. Thank you again for your careful and constructive review.

Reviewer_C59gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Recognition of emotion and other paralinguistic features whose signatures lie in fundamental frequency contours is very different in tonal languages such as Chinese, where the fundamental frequency trajectory itself is a feature of the language, and in non-tonal languages such as English. Thus, transferring emotion recognition from an English model to Chinese/Mandarin is a non-trivial task.

The authors propose "Tone-Disentangled Anchored Fused Gromov–Wasserstein" (TD-AFGW) to address this problem. TD-AFGW attempts to perform English-to-Mandarin multimodal emotion transfer without target emotion labels in training. To do so, it computes the residual between the measured f0s and f0s predicted (only) from the next by 'tonenet'. The statistics of these residual are used as vector representations for the recording. Matched English and Mandarin recordings are then topologically arranged using Gromov–Wasserstein alignment. An alternating update rule then learns the mapping of emotions between the languages, and the representations and their topological arrangement. The transferred emotion is then used to train a multi-modal emotion recognition model.

The model was validated using MELD as source and EmotionTalk as target. Compared to using the English system directly, the adapted model improves the weighted F1 slightly, while also degrading the accuracy.

Overall the idea is interesting, but there are many confounding factors, e.g. the residual includes estimation error from the pitch tracker and doesn't actually disentangle emotion, the results are iffy, and the model has been evaluated on only one benchmark. The writing is also largely AI and quite opaque.

Critical strengths
  1. The problem is well motivated. Lexical tone and emotional prosody both share pitch as a signature in tonal languages such as mandarin, making it hard to transfer emotion recognition from, say, English to Mandarim.

  2. The idea of using a residual pitch to detect emotion is a novel idea

  3. The proposed transfer mechanism, with facial/vocal anchors guiding cross-language matching and optimal transport of soft labels is novel.

  4. The tone swap filter and stability gate provide safeguards against transfer error.

  5. The paper tries hard not to overclaim. It explicitly distinguishes residual pitch from pure emotion, prediction stability from correctness, and label-free training from label-blind model selection.

Critical weaknesses
  1. The evaluation is compromised. The test trajectories and threshold comparisons were inspected when reporting gains, so this has an aspect of tuning on the test data. The gains therefore do not establish performance under target-label-blind model selection.

  2. The central tone-separation mechanism is entangled. The residual between tonenet predictions and measured pitch is confounded by several factors besides the emotion. Also, the ablation also bundles residual features and acoustic conditioning, leaving their individual contributions unclear.

  3. The tone-swap gate has uncertain interpretation. Swapping lexical pitch can alter the linguistic meaning or conflict with the unchanged transcript, which is very possible in a tonal language. How are the two temporally aligned? The results from arbitrary alignment are not different from random noisification. Also, it can introduce synthesis artifacts. So prediction stability under this intervention establishes neither emotion preservation nor transferred-label correctness.

  4. This is likely minor -- facial and vocal anchors need not be comparable across cultures and recording conditions. So, relational similarity may not guarantee correct emotion correspondence.

  5. The experimental evidence is limite. The evaluation covers one language pair with simultaneous corpus changes. The macro F1 remains only 18.01, while accuracy slightly declines. Among the emotions, disgust recognition substantially worsens. These results support a narrow aggregate improvement rather than broadly reliable transfer.

  6. It is unclear that the work is reproducible. Exact calibration coefficients, class offsets, divergence convention, and transport-solver settings needed to reproduce the reported operating points have not been reported.

Questions for authors
  1. Test contamination: Were checkpoints, the tone-swap threshold, calibration coefficients, or class offsets chosen or revised after inspecting target test results? Please specify how each was selected and whether an untouched evaluation set exists.

  2. Information in residual: Does the “No ToneNet” ablation remove both residual-pitch anchor features and FiLM conditioning? Are there existing results isolating these pathways, or directly testing whether residualization reduces lexical-tone information while preserving emotion information?

  3. Tone-swap alignment: How is the replacement contour in the tone-swap intervention selected and aligned? What is the evidence that the gate selects more accurate transferred labels, rather than merely predictions insensitive to pitch or synthesis artifacts?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer C59g

Question 1:

Answer: Thank you very much for this question; you were right to be concerned. When we re-checked our code and logs, we found that in the submitted version (1) the Stage 2 and Stage 3 checkpoints were kept by their weighted F1 on the EmotionTalk test set, evaluated after every epoch; (2) the calibration coefficient λ_cal and the class offsets for fear and disgust were chosen by a grid search on test labels (0.6, 0.8, 0.8); (3) τ = 0.20 was the default value, but the threshold comparison was inspected on the test set, so we cannot show that it was fixed beforehand; and (4) there was no untouched evaluation set. The adaptation pool was always disjoint from the test set (different recording groups and actors). We withdraw all the submitted numbers and replace them with the re-run results below. How each choice is made now (Section 5.1, “Protocols”; Appendix A): • Checkpoints. Protocol A: the last epoch of Stages 2 and 3 (Stage 1 by MELD development cross-entropy). Protocol B: the best calibrated W-F1 on the labeled EmotionTalk development split. • τ. Fixed at 0.20 before training. The sweep in Appendix Table 10 is a sensitivity check only, and the gate admits 98–100% of clips at every fixed τ anyway. • λ_cal. Protocol A: none (raw argmax). Protocol B: one value (0.7), fitted on the development split with the source-only models and shared by all rows. • Class offsets. δ = 0 everywhere. • Untouched set. A new holdout (groups G13 and G14: 2,862 clips from 4 actors who appear in no other split; in the submitted version these clips were only in the unlabeled adaptation pool, and they are now removed from it). Its labels were locked and opened only once, after all rules and predictions had been frozen and hashed (Appendix A, “Pre-registration and unsealing”). With these rules, TD-AFGW does not beat a fairly selected source-only model (Table 1; Appendix Table 7), and we changed the conclusions accordingly. The exact calibration coefficients, divergence convention and solver settings, which you also asked for under reproducibility, are now listed in Appendix A.

Question 2:

Answer: Thank you. Yes, in the submitted version the “No ToneNet” ablation removed both pathways at once. We now separate them (Table 2: “No FiLM only” and “No residual in anchor only”) and test the residual directly with probes (Appendix Table 14). • Ablations. On Mandarin, removing FiLM changes test W-F1 by +1.89 and removing only the residual by +0.08; on Vietnamese both are within noise. Neither pathway helps. • Probes. On read speech the residual removes much of the lexical tone (tone accuracy 65.6% to 39.7%), but on the conversational target speech it does the opposite (37.3% to 47.1%), because ToneNet fits that speech poorly (r = 0.208). Emotion information stays near chance (18.6 to 19.5 macro F1, chance 14.3). • Pitch-tracker error. As you pointed out, the residual also contains tracker error: Harvest and DIO residuals correlate 0.88, and their difference is about a quarter of the residual variance. So the residual is entangled exactly as you suggested, and we removed the claim that ToneNet separates tone from emotion (Sections 5.3 and 6).

Question 3:

Answer: Thank you, this question led to some of the clearest findings in the revision. • Selection and alignment. Each syllable with tone 1–4 gets a different, randomly chosen tone (fixed seed), and neutral-tone syllables are kept. ToneNet predicts the contour of the new tone sequence, which is placed on the syllable boundaries from CTC forced alignment, keeping the residual pitch (Section 4.4; Appendix A). • Evidence. We compared the gate with matched controls: a vocoder-only resynthesis and a random pitch change of the same size. The tone-swap divergence correlates 0.76 and 0.74 with these controls, admitted soft labels are no more accurate than randomly admitted ones (AUROC 0.50), and training with the control gates gives the same scores (Appendix Tables 4 and 11). At τ = 0.20 the gate admits 98–100% of clips. So, as you suspected, the gate selects predictions that are insensitive to resynthesis rather than more accurate labels, and with randomly chosen contours the intervention behaves much like random pitch noise. We removed the claim that the gate improves pseudo-label quality (Sections 5.3 and 6). Thank you again for the thorough review; your comments shaped most of this revision.

Reviewer_C59gOpenReview ↗

Thank you for your response. I will keep my score

I thank the authors for their honest responses to my comments.

Overall, they have not actually been able to show significant gains, when appropriate controls are applied.

I do believe the idea they propose is novel and the entire methodology is interesting. The work is simply not ready for primetime just yet. I encourage them to fix the issues -- and I believe given more time than is available in an ICLR response window, they will be able to do so.

For now I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.