acceptodds
Under review as a conference paper at ICLR 2027

TIE: Text-Bounded Incremental Evidence Migration for LLM-Based Multimodal Affective Computing

Abstract

LLM-based multimodal affective computing uses a large language model (LLM) to directly predict sentiment or emotion from multimodal inputs. Non-textual affective evidence is constructed, transformed into LLM-compatible representations, and propagated through the language-centered LLM. Along this path, a text-directed preference may be reinforced even when the constructed multimodal evidence supports the correct prediction, a phenomenon we term text-prior reinforcement. To address this problem, we formulate LLM-based multimodal affective prediction as Text-Bounded Incremental Evidence Migration (TIE), treating the full path from evidence construction to its use in prediction as a unified process. Guided by this formulation, TIE uses textual semantics and text-bounded temporal routing to organize audio and visual evidence along text-referenced and text-residual pathways. To preserve the influence of this evidence beyond representation construction, TIE transforms the organized evidence into LLM-compatible representations and applies a migration-specific signal derived from the frozen final prediction block. For sentiment regression, an order-aware objective bridges label generation and ordered sentiment estimation. TIE outperforms same-backbone compact-prompt baselines on most of the five benchmarks. The combined training configuration also improves existing compact-prompt methods. Controlled ablations validate TIE's components, while migration analyses provide empirical support for text-prior reinforcement.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal sentiment analysis, emotion recognition, large language models, multimodal learning, evidence migration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_awqbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses LLM-based multimodal affective computing with a frozen LLM. The authors name a phenomenon, text-prior reinforcement, in which a text-directed preference is reinforced along the path from evidence construction to label decoding, and propose TIE, which organizes audio/visual evidence into text-referenced and text-residual tokens via text-conditioned temporal routing, maps them to the LLM through a role-aware low-rank interface, and adds two training-only objectives: a migration loss computed by re-running the frozen final block, and an EMD-based order-aware loss for sentiment regression. On five MSA/ERC benchmarks with ChatGLM3-6B and Qwen3.5 backbones, TIE outperforms compact-prompt baselines on most settings, and its training configuration also improves MSE-Adapter and PaP.

Critical strengths
  1. The motivation is timely, and the diagnostic analyses (layer-wise readout, source-path attenuation, evidence replacement) go beyond standard ablations.
  2. The evaluation is broad (3 MSA + 2 ERC datasets, two languages, three backbones, five seeds), and the two training-only objectives transfer to other constructors at no inference cost (Tables 4, 15).
  3. The reproducibility details are thorough, including prompts, label serialization, decoding protocol, and timing protocol.
Critical weaknesses
  1. The central claim is not established. Appendix E states the analyses "do not establish that the former suppresses the latter"; Figure 7 shows both Org and Full develop a positive text-directed margin in middle layers, Table 22 shows Full's transcript contribution is in fact stronger than Org's (+0.2009 vs. +0.1171), and the interventions "rarely flip the predicted sign." The paper should either reframe text-prior reinforcement as an observation or show that TIE mitigates it.
  2. The gains come mostly from evidence organization (residual source +12.4%, reference loss +10.6%, Tables 17–18), a representation-level design. The "migration" components that justify the title have small, within-std effects (Tables 20–21).
  3. Main tables report means only. Using appendix stds, the MAE gains over PaP on MOSI (0.549 vs. 0.563) and MOSEI (0.489 vs. 0.495) are within about one std; MELD/CHERMA have no stds, and TIE loses to PaP on CHERMA. Qwen3.5-9B results have no same-backbone baseline except on SIMS-v2.
  4. Comparisons are uneven: PaP is re-implemented with author-chosen hyperparameters; reproduced and cited numbers are mixed within one table; task-specific baselines use different alignment settings; cited MLLM emotion methods (e.g., AffectGPT) are not compared. With a 6B LLM, TIE ties the BERT-based CICA on MOSEI MAE and is worse on Corr and Acc-7, and the 2B variant falls below many BERT-level models, so backbone scale appears to drive much of the gain.
  5. The method stacks many components with three loss weights (λ_ord = 5.0) and per-dataset learning rates, with no sensitivity analysis. The terminology (evidence migration, incremental, role-aware) wraps standard operations and makes the paper harder to read than necessary; no code release is mentioned.

While the paper has these limitations, the breadth of evaluation and the transferable training recipe are real contributions. If the authors add stds and significance tests to the main tables, same-backbone baselines for Qwen3.5-9B, and reframe the first point, I would consider giving it a higher score.

Questions for authors

see above

7 comments
AuthorsOpenReview ↗

Response to Reviewer awqb (Part 1 of 3): Claims, Ablations, and Results (W1–W3)

We’re truly grateful for the time and care you put into reviewing our paper. Your thoughtful feedback helped us improve it.

Please download the Revision 4 PDF of our manuscript from OpenReview and start on page 49, where our responses to your review begin. Our responses are the same in the PDF and on OpenReview. The PDF is easier to navigate: you can use the clickable links to jump straight to the relevant passages, figures, and tables.

W1

Reframe text-prior reinforcement as an observation.

Your perceptive observation helped us sharpen the paper's central claim. We are deeply grateful for the rigor with which you distinguished what our diagnostics show from what the evidence supports about TIE. The distinction you drew substantially improved the paper. We revised the abstract and introduction to distinguish the empirical observation of text-prior reinforcement from TIE's design goal: enabling audio and visual evidence to continue influencing predictions through the frozen LLM. We also revised the opening of Section 4.4, Evidence-Migration Analysis, to distinguish the availability of audio and visual evidence from its influence on predictions and to clarify what each diagnostic can establish. We do not claim that TIE mitigates text-prior reinforcement. As you note, attenuating transcript-position contributions had a larger effect in Full than in Org. Text provides context for interpreting audio and visual cues, and text-directed contributions can coexist with evidence supporting the multimodal target. TIE's predictive gains do not establish weaker text influence.

W2

1. Constructor design.

As illustrated in Figure 1, the central idea of TIE is to treat the successive transformations from evidence construction to predictive utilization as a unified process. Our evidence organization (Section 3.2) is designed to preserve task-relevant audio and visual cues when resampling them into a few tokens. We agree that evidence organization yields the larger gains in our ablations. This is unsurprising: effective migration depends on the quality of the evidence it receives.

2. Migration supervision.

Migration supervision targets how those evidence tokens are mapped into the frozen LLM. The individual migration-related gains on CMU-MOSI are modest (Table 24). To test whether the supervision source adds value with the organizer held fixed, we added a matched first-token comparison on CHERMA with Qwen3.5-9B. Frozen terminal improves accuracy and weighted F1 over Full forward by 1.18 and 1.28 percentage points, respectively (Table 23).

W3

1. Run-to-run variability.

We now report means and standard deviations in Appendix D.5 (Supplementary Run-to-Run Variability of Main Results).

2. CHERMA comparison with PaP.

PaP does not specify whether it used dialogue context on CHERMA. With scene information and preceding-clip transcripts, ChatGLM3-6B TIE reaches 75.67% accuracy and 75.51% weighted F1, close to PaP's published 75.33% and 75.28% (Tables 3 and 16). Under our matched protocol, TIE exceeds reproduced PaP-Org in weighted F1 by 1.35 percentage points with context and 0.93 points without it. Separately, with Qwen3.5-9B, TIE uses 21.3% fewer trainable parameters than PaP-Org on CHERMA (Table 25).

3. Complete the Qwen3.5-9B baseline comparisons.

We now report same-backbone MSE-Adapter and PaP baselines for Qwen3.5-9B on all five datasets (Tables 1, 2, and 3).

AuthorsOpenReview ↗

Response to Reviewer awqb (Part 2 of 3): Baselines and Method Details (W4–W5)

W4

1. PaP hyperparameter selection.

We used the published trainable parameter count to guide the dimensions in our reimplementation. Appendix C.3 documents the backbone-specific adaptations, and Appendix C.4 describes the shared tuning procedure.

2. Reproduced versus cited results.

We now mark reproduced baseline values in italics in Tables 1, 2, and 3.

3. Alignment protocol.

We consistently cite unaligned results when both aligned and unaligned results are available (see Appendix B.3, Coupled Mamba). (We appreciate how carefully you checked the alignment protocols across baselines. We had overlooked this detail ourselves.)

4. Baseline comparison protocols.

Our controlled compact-prompt comparisons use a shared backbone, inputs, and evaluation protocol. We do not mix published MLLM scores into these comparisons because their evaluation protocols differ (see Appendix B.3, MLLM affective evaluation).

5. Comparison with CICA.

On CMU-MOSEI, TIE with ChatGLM3-6B has higher Has0 F1 and Acc-2 than CICA's reported scores (Table 1). We are glad to see strong progress in affective computing with task-specific models such as CICA. TIE addresses a complementary question: how to organize audio and visual evidence for prediction in a frozen LLM. While our results do not establish that TIE mitigates text-prior reinforcement, we hope our approach and diagnostics offer a useful contribution to the affective computing community.

6. Comparison with BERT-based models.

Task-specific models are a strong choice for affective prediction. However, as LLMs see wider use, enabling them to perform affective prediction is also an important research direction. MAG and CHFN fine-tune BERT for multimodal sentiment analysis (Rahman et al., 2020; Guo et al., 2022). Even with a smaller backbone, task-focused training can be costly: UniSA reports about three days for its first pretraining stage on eight 32 GB V100 GPUs (Li et al., 2023b). Following the plug-and-play motivation of MSE-Adapter (Yang et al., 2025a), TIE uses a detachable adapter. In this plug-and-play paradigm, the same LLM weights can stay loaded while the adapter is enabled for multimodal affective prediction or bypassed for ordinary dialogue. Switching between these modes requires no backbone reload, making this design a promising direction for affect-aware human–AI interaction.

W5

1. Hyperparameter selection and sensitivity.

On CH-SIMS v2.0, the tested loss weights mainly affected convergence speed, with little change in validation metrics. We observed similarly small changes when transferring the tested settings to CMU-MOSI and used the resulting task-specific weights without further dataset-specific tuning. Appendix C.4 also describes validation-based learning-rate selection and memory-based batch sizing. Appendix C.3 documents which baseline settings were retained or adapted.

2. Terminology and presentation.

Thank you for pointing out that these terms could be clearer. Evidence migration (Eq. 1) describes how audio and visual evidence is mapped into input tokens for the frozen LLM and can influence its predictions. Incremental (Eq. 4) means adding a low-rank correction to a shared projection. Role-aware (Section 3.3.1) means using different corrections for text-referenced and text-residual evidence, shared across audio and vision within each role. The pun in TIE is intentional: we want to tie audio and visual evidence to the prediction, not just to the prompt.

3. Code release.

We plan to release our code if the paper is accepted. We would be honored to contribute a resource for future work in the affective computing community.

AuthorsOpenReview ↗

Response to Reviewer awqb (Part 3 of 3): Revision Summary and Paired Tests (Q)

Q

We are deeply grateful for your willingness to reconsider your rating and for spelling out the changes that would inform that reassessment.

Our response to W1 explains how we reframed text-prior reinforcement as an empirical observation. In W3, we summarize the new same-backbone baselines for Qwen3.5-9B on all five datasets and point to the standard deviations in the appendix. We also report the paired tests in our response to Reviewer j25t's Q2. With five paired seeds, CMU-MOSI, CH-SIMS v2.0, MELD, and CHERMA reached the minimum attainable two-sided pp-value of 0.0625. None of the comparisons reached significance at the 0.05 level.

We have now addressed these points in the revision and would be grateful if you could review the revised manuscript and consider a higher rating if you find that the changes address your concerns.

Reviewer_Ny1rOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIE for MSA and MER. TIE organizes acoustic and visual features into text-referenced and text-residual representations and projects them into four input tokens through a role-aware interface. It introduces auxiliary supervision using the frozen final prediction block and an order-aware objective for sentiment regression. Experiments show that the method can achieve advanced performance

Critical strengths
  1. The paper try to demonstrate that constructe useful multimodal representations does not necessarily ensure their effective utilization. It is interesting.

  2. Extensive experiments shows the effectiveness of the method.

Critical weaknesses
  1. Removing transcript and prompt states changes the final block’s attention context relative to normal inference. Consequently, I think that improve prediction under this modified context does not directly establish better evidence utilization during inference.

  2. The reliability and practical value of the auxiliary supervision need stronger evidence.

Questions for authors
  1. Why should supervision under the five-state terminal computation improve evidence utilization in the complete inference sequence?

  2. Are the gains over matched ordinary-forward first-token supervision consistent across datasets and backbones?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ny1r: Migration Context and Matched Supervision (W1–W2 and Q1–Q2)

We’re truly grateful for the time and care you put into reviewing our paper. Your thoughtful feedback helped us improve it.

Please download the Revision 6 PDF from OpenReview. Your review and our response begin on p. 49; the answers to W1/Q1 and W2/Q2 are on p. 50. We include the updated tables below. Values are five-seed means ± population standard deviations.

W1 / Q1

Terminal supervision and full-sequence inference.

You are right that removing the transcript and prompt states changes the final block’s attention context. Mig-Short is a training signal, not an inference shortcut: it trains the migration interface using four evidence states and one decision state. At inference, the model processes the complete sequence. Mig-FullCtx tests whether the auxiliary computation benefits from retaining the whole sequence, while still allowing auxiliary gradients to flow only through the evidence states.

Table 18 (p. 35). Migration context. Parentheses show relative performance changes from Mig-Short; negative values indicate degradation.

ChatGLM3-6BContextMAE ↓F1 ↑Acc-2 ↑Corr ↑
CMU-MOSIMig-Short0.5490±.013190.34±0.5390.38±0.540.8831±.0066
CMU-MOSIMig-FullCtx0.5497±.0096 (−0.1%)90.26±1.51 (−0.1%)90.38±1.41 (0.0%)0.8813±.0035 (−0.2%)
CH-SIMS v2.0Mig-Short0.2393±.009884.88±1.3784.82±1.440.8024±.0186
CH-SIMS v2.0Mig-FullCtx0.2405±.0097 (−0.5%)84.01±1.47 (−1.0%)83.98±1.41 (−1.0%)0.7908±.0185 (−1.4%)
Qwen3.5-9BContextAcc ↑Weighted F1 ↑
CHERMAMig-Short75.44±0.1375.42±0.09
CHERMAMig-FullCtx75.10±0.23 (−0.5%)75.07±0.11 (−0.5%)

Retaining the full auxiliary context did not lower mean MAE on either sentiment dataset or improve mean weighted F1 on CHERMA. This supports using the compact computation in these settings. It does not establish that the model uses audio or vision more effectively at inference.

W2 / Q2

Gains over matched first-token supervision.

Revision 6 adds CMU-MOSI with Qwen3.5-9B. In each comparison, we match the auxiliary target and loss weight, restrict auxiliary gradients to the migration interface, and keep the constructor design and other objectives unchanged. Only the source of first-token supervision differs. Below, parentheses show relative improvements over Full forward.

Table 22 (p. 36). CH-SIMS v2.0 with ChatGLM3-6B.

SourceMAE ↓F1 ↑Acc-2 ↑Corr ↑
Full forward0.2465±.011883.74±1.6983.71±1.650.7890±.0154
Frozen terminal0.2393±.0098 (+2.9%)84.88±1.37 (+1.4%)84.82±1.44 (+1.3%)0.8024±.0186 (+1.7%)

Table 23 (p. 36). CHERMA with Qwen3.5-9B.

SourceAcc ↑Weighted F1 ↑
Full forward74.25±0.3174.13±0.42
Frozen terminal75.44±0.13 (+1.6%)75.42±0.09 (+1.7%)

Table 24 (p. 36). CMU-MOSI with Qwen3.5-9B.

SourceMAE ↓F1 ↑Acc-2 ↑Corr ↑
Full forward0.5303±.011389.88±0.6790.09±0.630.8916±.0047
Frozen terminal0.5258±.0145 (+0.8%)90.40±0.55 (+0.6%)90.47±0.61 (+0.4%)0.8954±.0043 (+0.4%)

The five-seed means favor Frozen terminal in all three tested settings, but the gains vary in size: MAE falls by 2.9% on CH-SIMS v2.0 and 0.8% on CMU-MOSI; on CHERMA, weighted F1 rises by 1.29 percentage points. This is consistency in direction across the tested settings, not a claim of statistical significance.

Table 29 (p. 39). What changes in the final predictions? On 2,727 CHERMA test utterances where the text and multimodal labels disagree, a prediction can match only one of them, or neither. Entries are match rates (%); parentheses show relative changes from Mig-Short.

SourceContextText label ↓Multimodal label ↑Neither label ↓
Full forward–18.80±0.86 (+7.7%)64.49±0.89 (−2.4%)16.71±0.49 (+1.6%)
Frozen terminalMig-Short17.46±1.0866.09±0.6316.45±0.54
Frozen terminalMig-FullCtx17.68±1.18 (+1.3%)65.70±1.17 (−0.6%)16.62±0.31 (+1.0%)

Compared with Full forward, Mig-Short matches the multimodal target 1.60 percentage points more often and the text label 1.34 points less often during full-sequence inference. This shows which predictions change, not which modality caused the change.

We’d appreciate hearing whether these comparisons address your concerns.

Thank you for your efforts to solve my concerns and revising the work several times. My concern has been addressed. I read all the responses (including those from the other reviewers). Overall, I will maintain the current rating.

Additionally, will you release the source code?

AuthorsOpenReview ↗

Source Code Release

Yes, we plan to release our code upon acceptance.

Thank you for taking the time to read our revisions and our responses to the other reviewers. Your questions helped us clarify the distinction between training supervision and evidence use at inference. We’re glad the revisions addressed your concern.

Reviewer_j25tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that when a frozen LLM predicts sentiment or emotion from compact multimodal prompts, a text-directed preference can be reinforced as evidence moves through the model, which it calls text-prior reinforcement. TIE organizes audio and visual features into text-referenced and text-residual evidence tokens, maps them through a role-aware low-rank interface, adds an auxiliary loss computed with the frozen final block, and uses an EMD-based loss for sentiment regression. It reports gains over MSE-Adapter and PaP on five MSA and ERC benchmarks with ChatGLM3-6B and Qwen3.5 backbones.

Critical strengths
  1. The paper asks a useful question: does non-textual evidence injected into a frozen LLM actually affect the final decision?
  2. The Org/Full study adds the same training terms to MSE-Adapter and PaP and lowers their MAE on CH-SIMS v2.0 with two backbones, at no inference cost.
  3. The appendix is thorough, with five-seed standard deviations, efficiency measurements, and a check of constrained versus native decoding.
  4. The authors are candid about what their diagnostic analyses do and do not show.
Critical weaknesses
  1. Text-prior reinforcement is never defined as a measurable quantity. Appendix E states that the analyses do not show text suppressing useful evidence, and the margin changes rarely flip predictions, so the main motivation remains a hypothesis.
  2. Figure 3 does not show that TIE reduces text-directed behavior. TIE-Full and TIE-Org have similar text-directed shift, and in the highest-discrepancy bin PaP has both lower shift and lower MAE than TIE.
  3. Outside CH-SIMS v2.0, the gains over PaP are small or absent. On CMU-MOSEI the MAE gain is 0.006, below TIE's own seed standard deviation of 0.0076, and on CHERMA TIE is below PaP (74.34 vs. 75.28 WF1).
  4. The parts tied to the paper's story add little. In Table 21, removing the role-aware interface, the order-aware loss, or migration supervision changes MAE by 0.0045-0.0102, within one standard deviation. Table 20 shows the same pattern.
Questions for authors

1.Can you report a single text-prior metric for MSE-Adapter, PaP, TIE-Org and TIE-Full on the same samples, ideally on more than one dataset? This would show whether TIE actually reduces the problem it is designed for. 2.Can you add mean±std to the main tables and run paired per-seed tests against PaP on all five datasets? Please also state whether the PaP results on CMU-MOSEI, MELD and CHERMA were reproduced under your protocol. 3.How does a text-only model with the same frozen LLM, prompt and trainable budget perform? Please also report TIE without audio and without visual evidence. 4.Why does letting the auxiliary gradient reach the organizer raise MAE from 0.239 to 0.265 (Table 19)? Were the reproduced baselines tuned per dataset in the same way as TIE?

6 comments
AuthorsOpenReview ↗

Clarification on Q3 and Revision 1 Update

Thank you for suggesting these controls in Q3. Which setup would you prefer for the audio and visual ablations? Both options involve retraining TIE.

OptionHow the omitted modality is handled
A. Zero inputSet the omitted modality’s inputs to zero during training and evaluation, keeping TIE’s architecture unchanged.
B. Remove the branchRemove the omitted modality’s branch and its evidence tokens before retraining.

If we use Option B, should each ablation also match TIE-Full’s trainable parameter count, or does that requirement apply only to the text-only baseline?

AuthorsOpenReview ↗

Response to Reviewer j25t (Part 1 of 2): Text-Directed Behavior and Component Effects (W1–W4)

We’re truly grateful for the time and care you put into reviewing our paper. Your thoughtful feedback helped us improve it.

Please download the Revision 4 PDF of our manuscript from OpenReview and start on page 52, where our responses to your review begin. Our responses are the same in the PDF and on OpenReview. The PDF is easier to navigate: you can use the clickable links to jump straight to the relevant passages, figures, and tables.

W1

Scope of text-prior reinforcement.

Text-prior reinforcement describes how a text-directed preference can strengthen along the evidence-migration path. The probability gap p(yT)−p(yM)p(y_T)-p(y_M) under the frozen terminal head is one diagnostic: it tracks relative support for the text versus multimodal label across layers (Figure 4). Separately, source-position interventions measure changes in the final prediction margin that rarely flip the predicted sign (Table 29). The analyses support coexistence of text-directed contributions and useful audio and visual evidence, not suppression of that evidence by text (see Section 4.4, Evidence-Migration Analysis).

W2

Interpreting text-directed shift.

We thank the reviewer for highlighting this comparison. However, lower BTB_T does not by itself establish weaker reliance on text. To clarify what BTB_T measures, we added two input-independent controls to Figure 3: one always predicts zero, and the other always predicts the mean of the multimodal training labels (−0.0579-0.0579). Both yield positive BTB_T in the highest-discrepancy bin (n=34n=34), despite using no input. PaP's lower MAE remains an advantage in that bin, and we have clarified this distinction in the revised manuscript (Appendix E.1, Constant-prediction controls).

W3

1. Magnitude of the MAE gain on CMU-MOSEI.

We agree that the MAE gain over PaP with ChatGLM3-6B is small. With Qwen3.5-9B, TIE and PaP also have similar MAE (0.4882 vs. 0.4871), while TIE uses 54.1% fewer trainable parameters than PaP (2,583,876 vs. 5,624,898; Table 25). In a separate comparison of training scoring schemes, strict constrained scoring reduces the ChatGLM3-6B TIE MAE from 0.4894 to 0.4799, while increasing training time per seed from 3.946 to 26.341 hours (Table 5). These comparisons illustrate two trade-offs: similar MAE with fewer trainable parameters, and lower MAE with higher training cost.

2. CHERMA comparison with PaP.

PaP does not specify whether it used dialogue context on CHERMA. With scene information and preceding-clip transcripts, ChatGLM3-6B TIE reaches 75.67% accuracy and 75.51% weighted F1, close to PaP's published 75.33% and 75.28% (Tables 3 and 16). Under our matched protocol, TIE exceeds reproduced PaP-Org in weighted F1 by 1.35 percentage points with context and 0.93 points without it. Separately, with Qwen3.5-9B, TIE uses 21.3% fewer trainable parameters than PaP-Org on CHERMA (Table 25).

W4

Component effects and uncertainty.

We agree that the individual component gains on CMU-MOSI are modest. We now report a matched first-token comparison on CHERMA with Qwen3.5-9B to provide clearer evidence for the contribution of frozen-terminal supervision: it improves accuracy and weighted F1 by 1.18 and 1.28 percentage points, respectively, over Full forward (Table 23).

AuthorsOpenReview ↗

Response to Reviewer j25t (Part 2 of 2): Diagnostic Comparisons, Paired Tests, and Controls (Q1–Q4)

Q1

Cross-model comparison of text-directed shift.

Figure 3 already compares MSE-Adapter, PaP, TIE-Org, and TIE-Full using the same BTB_T metric on the same CH-SIMS v2.0 test samples with differing text and multimodal labels. Thank you for suggesting this additional comparison, which helped strengthen the empirical evidence in our paper. We now report an analysis of 2,727 CHERMA test samples with differing labels: Frozen terminal reduces the text-label match rate by 1.34 percentage points and increases the multimodal-label match rate by 1.60 points relative to Full forward (Table 28).

Q2

1. Run-to-run variability and paired tests.

To keep the main tables compact, we now report additional means and standard deviations in Appendix D.5 (Supplementary Run-to-Run Variability of Main Results). For Qwen3.5-9B, we matched each TIE-Full run to the PaP-Org run with the same random seed (1111, 2222, 3333, 4444, or 5555). The mean paired difference favored TIE on CMU-MOSI, CH-SIMS v2.0, MELD, and CHERMA, and favored PaP on CMU-MOSEI. The two-sided exact sign-flip test gave pp-values of 0.0625 (CMU-MOSI), 0.3750 (CMU-MOSEI), 0.0625 (CH-SIMS v2.0), 0.0625 (MELD), and 0.0625 (CHERMA). With five paired seeds, the smallest attainable two-sided pp-value under the exact sign-flip test is 0.0625: of the 25=322^5=32 sign assignments, at least two are as extreme as the observed result, namely the observed assignment and its global sign reversal. CMU-MOSI, CH-SIMS v2.0, MELD, and CHERMA reached this bound. Therefore, none of the comparisons reached statistical significance at the 0.05 level.

2. PaP result provenance.

The CMU-MOSEI, MELD, and CHERMA results in the ChatGLM3-6B PaP rows were taken from the PaP paper, not reproduced under our protocol.

Q3

1. Parameter-matched text-only control.

On CMU-MOSI with Qwen3.5-9B, TIE Full reduces MAE from 0.5913 to 0.5258 (11.1% relative) compared with the parameter-matched text-only control (Table 24).

2. Audio and visual ablations.

TBD.

Q4

1. Effect of extending auxiliary gradients to the organizer.

Allowing the auxiliary gradient to update the evidence organizer increased MAE from 0.2393 to 0.2653 on CH-SIMS v2.0 (Table 21). We did not determine the cause of this degradation. Based on this result, we restrict the auxiliary gradient to the migration interface.

2. Dataset-specific tuning of reproduced baselines.

For adapted baselines, only the learning rate was selected on the validation split, following the same procedure as for TIE. Batch sizes were set for throughput within GPU memory limits (Appendix C.4). We retained the official ChatGLM3-6B MSE-Adapter configuration without further tuning and adjusted backbone-dependent dimensions when switching LLMs (Appendix C.3).

We’d appreciate hearing whether the new results address your main concerns and which questions, if any, remain important to your assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary: What We Revised and What the Controls Establish

Dear Reviewers, Area Chair, and Senior Area Chair,

Thank you for the time and care you put into reviewing our work.

TIE asks how audio and visual cues compressed into a few tokens can continue to inform a frozen language model's predictions. The discussion prompted us to separate the contribution of the evidence constructor from that of its training signal, and to explain more precisely what our diagnostics measure.

1. How we addressed the main review concerns

1.1 Generality and fair comparisons

We completed same-backbone reproductions of MSE-Adapter and PaP across the English and Chinese sentiment and emotion benchmarks. On CHERMA, we compared both PaP and TIE with and without scene information and preceding dialogue. These controls complement a separate transfer result: on the Chinese sentiment benchmark, applying the Full training configuration to either baseline constructor lowered mean error on both tested backbones without lengthening the inference prompt. On CHERMA, Full also improved on Org when neither used the order-aware loss, so the benefit there does not depend on a regression objective.

1.2 What migration supervision adds

We brought the matched sentiment-regression comparisons together with an emotion-recognition test on CHERMA. With the constructor, auxiliary target, loss weight, and gradient scope fixed, Frozen terminal gave better mean predictions than ordinary-forward first-token supervision in each tested setting. We separately compared compact and full-sequence contexts for auxiliary training. The longer context did not improve mean performance on CHERMA; both variants still use the full sequence at inference.

1.3 When audio and vision matter

Alongside the existing parameter-matched text-only control on CMU-MOSI, we added CHERMA retraining controls that omit audio, vision, or both without changing the architecture or parameter count. Removing either hurt performance; removing both hurt most. Modality-specific annotations, used only for analysis, let us ask which input mattered on a given group of utterances. Retaining audio worked better when only the audio label agreed with the target; the ordering reversed when only vision agreed. This pattern persisted with equal weight across emotion classes, whether or not the text label agreed with the target, and on text-conflict cases where only one supervision variant was correct.

1.4 What changed in our interpretation

We revised the abstract, introduction, and analysis to distinguish text-prior reinforcement as a motivating observation from TIE's design goal. Constant predictors showed that a text-directed shift can arise even without a transcript. We therefore distinguish output behavior from layerwise label support, propagation of replacement effects, and the effect of attenuating source positions. We extended the layerwise readout to CHERMA and compared paired predictions under different supervision: on text-conflict utterances, more decisions moved from the text label to the multimodal target than in reverse. This shows which decisions changed, not that TIE weakened text influence. Text can help interpret audio and visual cues.

1.5 Reproducibility and scope

We distinguished reproduced from cited results, clarified alignment and decoding protocols, and documented backbone adaptations, validation-based checkpoint and learning-rate selection, trainable parameter counts, and run-to-run variability. Our response also reports exact tests pairing runs by seed. We specified where the loss weights were tuned. We also clarified the reference loss and auxiliary gradient path. The conclusion now distinguishes a potential use of the detachable adapter in dialogue from an application tested here.

2. Reviewers' perspectives and follow-up

2.1 awqb

awqb valued the diagnostics, breadth of evaluation, and reproducibility details, and asked us to separate evidence organization from migration and observation from mitigation. They said our responses addressed their concerns in detail and helped them understand the method, and maintained their positive rating.

2.2 Ny1r

Ny1r distinguished useful representations from their use at inference and asked for stronger supervision controls. After reading the responses, they said their concern had been addressed, maintained their positive rating, and asked about code availability. We plan to release the code upon acceptance.

2.3 j25t

j25t noted transfer across constructors but questioned the gains and their interpretation. They maintained their weak-reject rating before we posted the modality ablations they requested. As the public discussion period draws to a close, we have not received a further reply. We hope these results merit a higher rating.

Thank you for helping us strengthen our experiments and clarify our claims.

Best wishes,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.