- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses LLM-based multimodal affective computing with a frozen LLM. The authors name a phenomenon, text-prior reinforcement, in which a text-directed preference is reinforced along the path from evidence construction to label decoding, and propose TIE, which organizes audio/visual evidence into text-referenced and text-residual tokens via text-conditioned temporal routing, maps them to the LLM through a role-aware low-rank interface, and adds two training-only objectives: a migration loss computed by re-running the frozen final block, and an EMD-based order-aware loss for sentiment regression. On five MSA/ERC benchmarks with ChatGLM3-6B and Qwen3.5 backbones, TIE outperforms compact-prompt baselines on most settings, and its training configuration also improves MSE-Adapter and PaP.
Critical strengths
- The motivation is timely, and the diagnostic analyses (layer-wise readout, source-path attenuation, evidence replacement) go beyond standard ablations.
- The evaluation is broad (3 MSA + 2 ERC datasets, two languages, three backbones, five seeds), and the two training-only objectives transfer to other constructors at no inference cost (Tables 4, 15).
- The reproducibility details are thorough, including prompts, label serialization, decoding protocol, and timing protocol.
Critical weaknesses
- The central claim is not established. Appendix E states the analyses "do not establish that the former suppresses the latter"; Figure 7 shows both Org and Full develop a positive text-directed margin in middle layers, Table 22 shows Full's transcript contribution is in fact stronger than Org's (+0.2009 vs. +0.1171), and the interventions "rarely flip the predicted sign." The paper should either reframe text-prior reinforcement as an observation or show that TIE mitigates it.
- The gains come mostly from evidence organization (residual source +12.4%, reference loss +10.6%, Tables 17–18), a representation-level design. The "migration" components that justify the title have small, within-std effects (Tables 20–21).
- Main tables report means only. Using appendix stds, the MAE gains over PaP on MOSI (0.549 vs. 0.563) and MOSEI (0.489 vs. 0.495) are within about one std; MELD/CHERMA have no stds, and TIE loses to PaP on CHERMA. Qwen3.5-9B results have no same-backbone baseline except on SIMS-v2.
- Comparisons are uneven: PaP is re-implemented with author-chosen hyperparameters; reproduced and cited numbers are mixed within one table; task-specific baselines use different alignment settings; cited MLLM emotion methods (e.g., AffectGPT) are not compared. With a 6B LLM, TIE ties the BERT-based CICA on MOSEI MAE and is worse on Corr and Acc-7, and the 2B variant falls below many BERT-level models, so backbone scale appears to drive much of the gain.
- The method stacks many components with three loss weights (λ_ord = 5.0) and per-dataset learning rates, with no sensitivity analysis. The terminology (evidence migration, incremental, role-aware) wraps standard operations and makes the paper harder to read than necessary; no code release is mentioned.
While the paper has these limitations, the breadth of evaluation and the transferable training recipe are real contributions. If the authors add stds and significance tests to the main tables, same-backbone baselines for Qwen3.5-9B, and reframe the first point, I would consider giving it a higher score.
Questions for authors
see above
7 comments
Response to Reviewer awqb (Part 2 of 3): Baselines and Method Details (W4–W5)
W4
1. PaP hyperparameter selection.
We used the published trainable parameter count to guide the dimensions in our reimplementation. Appendix C.3 documents the backbone-specific adaptations, and Appendix C.4 describes the shared tuning procedure.
2. Reproduced versus cited results.
We now mark reproduced baseline values in italics in Tables 1, 2, and 3.
3. Alignment protocol.
We consistently cite unaligned results when both aligned and unaligned results are available (see Appendix B.3, Coupled Mamba). (We appreciate how carefully you checked the alignment protocols across baselines. We had overlooked this detail ourselves.)
4. Baseline comparison protocols.
Our controlled compact-prompt comparisons use a shared backbone, inputs, and evaluation protocol. We do not mix published MLLM scores into these comparisons because their evaluation protocols differ (see Appendix B.3, MLLM affective evaluation).
5. Comparison with CICA.
On CMU-MOSEI, TIE with ChatGLM3-6B has higher Has0 F1 and Acc-2 than CICA's reported scores (Table 1). We are glad to see strong progress in affective computing with task-specific models such as CICA. TIE addresses a complementary question: how to organize audio and visual evidence for prediction in a frozen LLM. While our results do not establish that TIE mitigates text-prior reinforcement, we hope our approach and diagnostics offer a useful contribution to the affective computing community.
6. Comparison with BERT-based models.
Task-specific models are a strong choice for affective prediction. However, as LLMs see wider use, enabling them to perform affective prediction is also an important research direction. MAG and CHFN fine-tune BERT for multimodal sentiment analysis (Rahman et al., 2020; Guo et al., 2022). Even with a smaller backbone, task-focused training can be costly: UniSA reports about three days for its first pretraining stage on eight 32 GB V100 GPUs (Li et al., 2023b). Following the plug-and-play motivation of MSE-Adapter (Yang et al., 2025a), TIE uses a detachable adapter. In this plug-and-play paradigm, the same LLM weights can stay loaded while the adapter is enabled for multimodal affective prediction or bypassed for ordinary dialogue. Switching between these modes requires no backbone reload, making this design a promising direction for affect-aware human–AI interaction.
W5
1. Hyperparameter selection and sensitivity.
On CH-SIMS v2.0, the tested loss weights mainly affected convergence speed, with little change in validation metrics. We observed similarly small changes when transferring the tested settings to CMU-MOSI and used the resulting task-specific weights without further dataset-specific tuning. Appendix C.4 also describes validation-based learning-rate selection and memory-based batch sizing. Appendix C.3 documents which baseline settings were retained or adapted.
2. Terminology and presentation.
Thank you for pointing out that these terms could be clearer. Evidence migration (Eq. 1) describes how audio and visual evidence is mapped into input tokens for the frozen LLM and can influence its predictions. Incremental (Eq. 4) means adding a low-rank correction to a shared projection. Role-aware (Section 3.3.1) means using different corrections for text-referenced and text-residual evidence, shared across audio and vision within each role. The pun in TIE is intentional: we want to tie audio and visual evidence to the prediction, not just to the prompt.
3. Code release.
We plan to release our code if the paper is accepted. We would be honored to contribute a resource for future work in the affective computing community.
Response to Reviewer awqb (Part 3 of 3): Revision Summary and Paired Tests (Q)
Q
We are deeply grateful for your willingness to reconsider your rating and for spelling out the changes that would inform that reassessment.
Our response to W1 explains how we reframed text-prior reinforcement as an empirical observation. In W3, we summarize the new same-backbone baselines for Qwen3.5-9B on all five datasets and point to the standard deviations in the appendix. We also report the paired tests in our response to Reviewer j25t's Q2. With five paired seeds, CMU-MOSI, CH-SIMS v2.0, MELD, and CHERMA reached the minimum attainable two-sided -value of 0.0625. None of the comparisons reached significance at the 0.05 level.
We have now addressed these points in the revision and would be grateful if you could review the revised manuscript and consider a higher rating if you find that the changes address your concerns.
Response to Reviewer awqb (Part 1 of 3): Claims, Ablations, and Results (W1–W3)
W1
Reframe text-prior reinforcement as an observation.
Your perceptive observation helped us sharpen the paper's central claim. We are deeply grateful for the rigor with which you distinguished what our diagnostics show from what the evidence supports about TIE. The distinction you drew substantially improved the paper. We revised the abstract and introduction to distinguish the empirical observation of text-prior reinforcement from TIE's design goal: enabling audio and visual evidence to continue influencing predictions through the frozen LLM. We also revised the opening of Section 4.4, Evidence-Migration Analysis, to distinguish the availability of audio and visual evidence from its influence on predictions and to clarify what each diagnostic can establish. We do not claim that TIE mitigates text-prior reinforcement. As you note, attenuating transcript-position contributions had a larger effect in Full than in Org. Text provides context for interpreting audio and visual cues, and text-directed contributions can coexist with evidence supporting the multimodal target. TIE's predictive gains do not establish weaker text influence.
W2
1. Constructor design.
As illustrated in Figure 1, the central idea of TIE is to treat the successive transformations from evidence construction to predictive utilization as a unified process. Our evidence organization (Section 3.2) is designed to preserve task-relevant audio and visual cues when resampling them into a few tokens. We agree that evidence organization yields the larger gains in our ablations. This is unsurprising: effective migration depends on the quality of the evidence it receives.
2. Migration supervision.
Migration supervision targets how those evidence tokens are mapped into the frozen LLM. The individual migration-related gains on CMU-MOSI are modest (Table 24). To test whether the supervision source adds value with the organizer held fixed, we added a matched first-token comparison on CHERMA with Qwen3.5-9B. Frozen terminal improves accuracy and weighted F1 over Full forward by 1.18 and 1.28 percentage points, respectively (Table 23).
W3
1. Run-to-run variability.
We now report means and standard deviations in Appendix D.5 (Supplementary Run-to-Run Variability of Main Results).
2. CHERMA comparison with PaP.
PaP does not specify whether it used dialogue context on CHERMA. With scene information and preceding-clip transcripts, ChatGLM3-6B TIE reaches 75.67% accuracy and 75.51% weighted F1, close to PaP's published 75.33% and 75.28% (Tables 3 and 16). Under our matched protocol, TIE exceeds reproduced PaP-Org in weighted F1 by 1.35 percentage points with context and 0.93 points without it. Separately, with Qwen3.5-9B, TIE uses 21.3% fewer trainable parameters than PaP-Org on CHERMA (Table 25).
3. Complete the Qwen3.5-9B baseline comparisons.
We now report same-backbone MSE-Adapter and PaP baselines for Qwen3.5-9B on all five datasets (Tables 1, 2, and 3).