- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies adapter-based frozen LLMs for multimodal sentiment analysis and emotion recognition. The paper points out that an adapter can pass only a few tokens to the LLM, while audio-visual streams are long, so most information must be compressed into a small number of tokens. Under this "bandwidth" limit, it argues that the key question is which audio-visual evidence to keep. It also argues that existing methods mostly compress or prune tokens passively, without dynamic guidance from the text when selecting evidence. To address this, the paper proposes TEB, which uses the transcript of the current utterance to guide the selection of audio-visual evidence. TEB builds a fixed number of text-conditioned queries from the transcript. It organizes the audio-visual features into two complementary memories: one keeps local temporal segments, and the other aggregates global information through learned anchors. Each query reads the two memories separately and mixes the results with a gate that depends on the consistency gap between the audio-visual and text representations. Each query then produces one evidence token for the frozen LLM. The method is evaluated on several sentiment analysis and emotion recognition datasets with multiple LLM backbones, together with equal-budget interface comparisons and component ablations.
Critical strengths
The paper addresses a relevant problem: in adapter-based frozen LLMs, audio-visual input can only be compressed into a few tokens, and the question is how to keep useful information during compression so that these few tokens carry as much task-relevant evidence as possible.
Critical weaknesses
-
Issues in the related work
- The paper is positioned as an adapter-based method, so the related work should give adapters a dedicated section. Instead, the first paragraph mainly covers classic fusion methods such as tensor fusion and cross-modal attention. These have little to do with the bandwidth problem the paper targets.
- The first paragraph mentions the CH-SIMS v2.0 dataset but does not explain how it relates to the method. Dataset descriptions belong in the experimental section.
- The title and introduction present emotion recognition in conversation (ERC) as one of the target tasks, and the introduction stresses that ERC requires modeling inter-utterance interaction and speaker context. However, TEB makes predictions for each utterance on its own and does not use any dialogue context. In that case, the ERC setting reduces to utterance-level multimodal sentiment analysis or emotion recognition. Why, then, is ERC singled out?
- The second paragraph lists related methods without saying what problem each one addresses or what remains unsolved. It is also unclear how TEB differs from them.
- The related work notes that TCR and QaP already use text conditioning, and MSE-Adapter is also text-guided. This conflicts with the claim in the introduction that evidence selection "lacks dynamic text-conditioned guidance." The paper should clarify what is new in TEB compared with these methods.
-
The bandwidth motivation is not supported by experiments
- The paper does not show that existing methods actually suffer when the token budget is reduced. All baselines are evaluated at a fixed B=4, and the only budget sweep tests TEB alone. Without baseline results at different budgets, it is unclear whether limited bandwidth is a real problem for existing adapters, or whether TEB handles it better.
- The budget range is too narrow to support the bandwidth claim. The introduction uses Emotion-LLaMA, with 256 video tokens, to motivate the problem, but the sweep only covers B from 1 to 8. There is no large-budget or uncompressed reference to show how much performance is lost by compressing to a few tokens. Within this range, B=8 even performs worse than B=4.
Questions for authors
- Since TEB does not model dialogue context, why does the paper emphasize ERC throughout? How does the ERC setting differ from utterance-level emotion recognition in this work?
- TCR, QaP and MSE-Adapter already use text to guide audio-visual features. What exactly is new in TEB compared with these methods?
- Why does the related work focus on classic fusion methods instead of adapter-based methods?
- How do the baselines, such as MSE-Adapter and the single-memory resampler, perform at different budgets? Does their performance drop as the budget decreases?
- Can the authors report results at larger budgets (e.g., 64, 128 or 256 tokens) or without compression? This would show whether a small budget actually hurts performance and whether TEB reduces this loss.
3 comments
Q2: Design Distinction from TCR, QaP, and MSE-Adapter
TEB organizes text-conditioned evidence access around individual LLM-facing output positions. Each of the positions is assigned a transcript-conditioned query that retrieves separately from local-segment and learned-latent candidate memories. The query, both retrieved summaries, and the utterance context then jointly determine the mixing weights that update that position's state before token projection, keeping both candidate organizations separately accessible until each output token is formed.
Section 2.2 contrasts this pipeline directly with prior text-guided methods:
- TCR resamples video features into fixed-length query outputs conditioned on task text from a single feature stream;
- QaP integrates querying prompts with text inside the language backbone and injects modality information across internal layers via layer-wise resamplers;
- MSE-Adapter applies text-guided Hadamard mixing and multi-scale fusion to temporally summarized audio-visual representations, then projects the pre-fused representation into pseudo tokens.
Rather than resampling a single stream, injecting layer-wise prompts inside the LLM, or projecting pre-fused representations, TEB operates prior to the frozen LLM through separately normalized reads over local-segment and learned-latent memories followed by a post-retrieval gate. As detailed in Section 3.3, separate reads produce two memory-specific summaries, which the gate then combines using the query, both summaries, and the utterance context. In contrast, joint attention determines candidate weights under a shared softmax before aggregating them into a single readout.
Two controlled comparisons provide empirical support for this design:
- Comparison against single-memory text conditioning (Table 3): At with ChatGLM3-6B, TEB improves five-seed mean MAE on SIMS-V2 from to and mean WF1 on CHERMA from % to % over the text-conditioned single-memory resampler, evaluating the complete dual-memory and access design beyond text conditioning alone.
- Same-candidate access controls (Table 4): With the same local and latent candidate-memory construction retained across access variants on CHERMA (, ChatGLM3-6B), TEB achieves % mean WF1, outperforming both a single concatenated read (%) and a fixed mixture (%), directly linking the structural distinction to the benefit of separate retrieval and conditional mixing as a joint design.
W2 and Q4: Baseline Sensitivity to the Output Budget ()
We evaluated TEB, MSE-Adapter, and the text-conditioned single-memory resampler across common budgets of on CHERMA using ChatGLM3-6B (five training seeds, without structured calibration). Each budget setting was trained independently under our unified comparison protocol. Section 4.3(c) and Appendix Table 7 report the full results, summarized in Table R1.
Table R1: Five-seed Weighted F1 (, %) across output-token budgets on CHERMA (ChatGLM3-6B). Bold indicates the highest mean WF1 at each budget.
| Interface | |||
|---|---|---|---|
| MSE-Adapter | 68.67 ± 1.14 | 72.78 ± 0.37 | 71.35 ± 0.85 |
| Text-conditioned resampler | 72.90 ± 0.57 | 72.80 ± 0.69 | 72.35 ± 0.25 |
| TEB (Ours) | 73.15 ± 0.58 | 74.18 ± 0.27 | 72.28 ± 1.27 |
Sensitivity to budget reduction differs across baseline architectures. Reducing from to lowers MSE-Adapter's mean WF1 by points (from % to %), compared with points for TEB, confirming that tight budget reduction substantially degrades an existing affect adapter while TEB retains much more of its performance. Meanwhile, the text-conditioned resampler remains relatively flat between (%) and (%), whereas TEB effectively utilizes the -token budget to reach the highest overall mean (%). Overall, TEB achieves the highest mean WF1 at both compact budgets ( and ), while at the resampler is higher by points and none of the three interfaces improves over its score.
These comparisons motivate studying candidate organization and evidence access at compact output budgets, without implying that every baseline deteriorates uniformly as shrinks. In Section 3, we explicitly define "bandwidth" as the adapter's LLM-facing output-token budget—an interface design condition distinct from the LLM's context window—and we have removed the 256-token Emotion-LLaMA example from the introduction.
Q5: Larger Budgets () and Direct Memory Forwarding
In our extended budget sweep (Table R1), we scaled the output budget up to tokens—eight times the setting used in our main comparisons. Notably, increasing from to lowers mean WF1 across all three evaluated interfaces (from % to % for MSE-Adapter, % to % for the resampler, and % to % for TEB). Thus, within the tested expansion range, compressing multimodal evidence into tokens does not incur a performance penalty relative to a much larger -token budget.
Given the available single-GPU memory and our fixed-batch-size comparison protocol, we limited the extended cross-interface sweep to ; budgets of – tokens were not evaluated. Direct memory-to-LLM forwarding remains a separate untested comparison: TEB at still constructs tokens through retrieval, mixing, and prompt composition, so it is not an uncompressed reference. Performance relative to direct memory forwarding therefore remains unmeasured, as documented in Appendix E.
Response to Reviewer ufaV
Thank you for highlighting the need to clearly distinguish TEB from existing text-guided interfaces, clarify our task scope, and evaluate the bandwidth motivation comparatively. In the revised manuscript, we have:
Tables prefixed with R belong to this response; all other section, table, and appendix references correspond to the revised manuscript.
W1 and Q3: Organization and Focus of the Related Work
Section 2 now separates general affect-modeling background from the adapter interfaces most directly relevant to TEB:
In addition, Section 1 now explicitly acknowledges that existing text-conditioned interfaces already use language to guide feature extraction, removing the conflicting statement that prior evidence selection lacks dynamic text-conditioned guidance. Instead, the revised introduction frames our research question around how text guidance is coupled to candidate memory organization and post-retrieval summary mixing under a fixed output-token budget.
Q1: Utterance-Level Prediction on Conversational Benchmarks
TEB predicts affect from the transcript, audio, and visual streams of the current utterance alone. We evaluate MELD, a conversational emotion-recognition benchmark, in an utterance-level setting without dialogue history or inter-utterance speaker modeling. CHERMA provides multimodal emotion labels for utterance clips and is likewise evaluated at the utterance level. In both cases, the candidate memories are constructed strictly from within-utterance audio-visual sequences.
To avoid any ambiguity, the revised abstract explicitly identifies utterance-level emotion recognition, and Sections 1, 2.1, and 4.1 distinguish this utterance-level scope from conversational architectures that model multi-turn dialogue context. These evaluations therefore assess utterance-level multimodal emotion recognition; they do not evaluate inter-utterance conversational reasoning.