acceptodds
Under review as a conference paper at ICLR 2027

TEB: A Bandwidth-Constrained Evidence Allocation Framework for Enhancing Multimodal Sentiment Analysis and Emotion Recognition in Large Language Models

11 views

Abstract

Adapter-based LLM frameworks for multimodal sentiment analysis and emotion recognition in AI conversations are attractive for their lightweight flexibility, but the adapter interface provides only tokens. Long audio-visual streams must be compressed into a few evidence tokens while preserving conversation-relevant cues. The key challenge is deciding which evidence to retain under this bandwidth constraint. We propose the Text-Conditioned Evidence Bottleneck (TEB), a bandwidth-constrained evidence allocation framework. TEB uses transcript embeddings to build utterance-relevant query states, organizes audio-visual sequences into local segment and latent global memories, and routes evidence per query through soft gating based on audio-text and visual-text consistency discrepancies. Low-rank coefficients and a shared prompt dictionary map each query to one LLM-receivable evidence token. On three LLMs and four public datasets, TEB improves the five-seed mean of all metrics over MSE-Adapter. At , it reduces ChatGLM3-6B's MAE on SIMS-V2 from 0.279 to 0.265 and increases weighted F1 on CHERMA from 72.80 to 74.18. LLaMA2-7B shows significant improvements in Acc-2, F1, weak sentiment accuracy, MAE, and correlation coefficient on SIMS-V2, while Qwen-1.8B also demonstrates consistent gains on MOSEI and CHERMA. These results establish TEB as a lightweight, flexible, and scalable foundation to advance LLMs' multimodal sentiment analysis and emotion recognition capabilities. Our code is available at https://anonymous.4open.science/r/TEBTEB-111.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Sentiment Analysis, Large Language Models, Emotional Support Conversation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ufaVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies adapter-based frozen LLMs for multimodal sentiment analysis and emotion recognition. The paper points out that an adapter can pass only a few tokens to the LLM, while audio-visual streams are long, so most information must be compressed into a small number of tokens. Under this "bandwidth" limit, it argues that the key question is which audio-visual evidence to keep. It also argues that existing methods mostly compress or prune tokens passively, without dynamic guidance from the text when selecting evidence. To address this, the paper proposes TEB, which uses the transcript of the current utterance to guide the selection of audio-visual evidence. TEB builds a fixed number of text-conditioned queries from the transcript. It organizes the audio-visual features into two complementary memories: one keeps local temporal segments, and the other aggregates global information through learned anchors. Each query reads the two memories separately and mixes the results with a gate that depends on the consistency gap between the audio-visual and text representations. Each query then produces one evidence token for the frozen LLM. The method is evaluated on several sentiment analysis and emotion recognition datasets with multiple LLM backbones, together with equal-budget interface comparisons and component ablations.

Critical strengths

The paper addresses a relevant problem: in adapter-based frozen LLMs, audio-visual input can only be compressed into a few tokens, and the question is how to keep useful information during compression so that these few tokens carry as much task-relevant evidence as possible.

Critical weaknesses
  • Issues in the related work

    • The paper is positioned as an adapter-based method, so the related work should give adapters a dedicated section. Instead, the first paragraph mainly covers classic fusion methods such as tensor fusion and cross-modal attention. These have little to do with the bandwidth problem the paper targets.
    • The first paragraph mentions the CH-SIMS v2.0 dataset but does not explain how it relates to the method. Dataset descriptions belong in the experimental section.
    • The title and introduction present emotion recognition in conversation (ERC) as one of the target tasks, and the introduction stresses that ERC requires modeling inter-utterance interaction and speaker context. However, TEB makes predictions for each utterance on its own and does not use any dialogue context. In that case, the ERC setting reduces to utterance-level multimodal sentiment analysis or emotion recognition. Why, then, is ERC singled out?
    • The second paragraph lists related methods without saying what problem each one addresses or what remains unsolved. It is also unclear how TEB differs from them.
    • The related work notes that TCR and QaP already use text conditioning, and MSE-Adapter is also text-guided. This conflicts with the claim in the introduction that evidence selection "lacks dynamic text-conditioned guidance." The paper should clarify what is new in TEB compared with these methods.
  • The bandwidth motivation is not supported by experiments

    • The paper does not show that existing methods actually suffer when the token budget is reduced. All baselines are evaluated at a fixed B=4, and the only budget sweep tests TEB alone. Without baseline results at different budgets, it is unclear whether limited bandwidth is a real problem for existing adapters, or whether TEB handles it better.
    • The budget range is too narrow to support the bandwidth claim. The introduction uses Emotion-LLaMA, with 256 video tokens, to motivate the problem, but the sweep only covers B from 1 to 8. There is no large-budget or uncompressed reference to show how much performance is lost by compressing to a few tokens. Within this range, B=8 even performs worse than B=4.
Questions for authors
  1. Since TEB does not model dialogue context, why does the paper emphasize ERC throughout? How does the ERC setting differ from utterance-level emotion recognition in this work?
  2. TCR, QaP and MSE-Adapter already use text to guide audio-visual features. What exactly is new in TEB compared with these methods?
  3. Why does the related work focus on classic fusion methods instead of adapter-based methods?
  4. How do the baselines, such as MSE-Adapter and the single-memory resampler, perform at different budgets? Does their performance drop as the budget decreases?
  5. Can the authors report results at larger budgets (e.g., 64, 128 or 256 tokens) or without compression? This would show whether a small budget actually hurts performance and whether TEB reduces this loss.
3 comments
AuthorsOpenReview ↗

Response to Reviewer ufaV

Thank you for highlighting the need to clearly distinguish TEB from existing text-guided interfaces, clarify our task scope, and evaluate the bandwidth motivation comparatively. In the revised manuscript, we have:

  1. Restructured Section 2 (Related Work) to give query interfaces and LLM adapters a dedicated subsection with an explicit mechanism-by-mechanism comparison to TEB, while moving dataset details to Section 4;
  2. Clarified the utterance-level task scope in the abstract, introduction, and experimental setup, and removed conflicting claims regarding prior text-conditioned guidance and the 256-token motivating example;
  3. Completed cross-budget comparisons (B=1,4,32B = 1, 4, 32) across TEB, MSE-Adapter, and the text-conditioned resampler under a unified protocol.

Tables prefixed with R belong to this response; all other section, table, and appendix references correspond to the revised manuscript.

W1 and Q3: Organization and Focus of the Related Work

Section 2 now separates general affect-modeling background from the adapter interfaces most directly relevant to TEB:

  • Section 2.1 (Multimodal Affect Modeling): Retains a concise overview of classic fusion methods to provide task background and discusses the related non-LLM dual-path architecture DPDF-LQ. The description of the CH-SIMS v2.0 dataset has been moved to the experimental setup (Section 4.1).
  • Section 2.2 (Query Interfaces and LLM Adapters): Gives multimodal adapters a dedicated subsection covering learned-query resamplers, text-conditioned extraction, and affect adapters. It outlines the specific mechanism of each representative method and concludes with a focused structural comparison to TEB.

In addition, Section 1 now explicitly acknowledges that existing text-conditioned interfaces already use language to guide feature extraction, removing the conflicting statement that prior evidence selection lacks dynamic text-conditioned guidance. Instead, the revised introduction frames our research question around how text guidance is coupled to candidate memory organization and post-retrieval summary mixing under a fixed output-token budget.

Q1: Utterance-Level Prediction on Conversational Benchmarks

TEB predicts affect from the transcript, audio, and visual streams of the current utterance alone. We evaluate MELD, a conversational emotion-recognition benchmark, in an utterance-level setting without dialogue history or inter-utterance speaker modeling. CHERMA provides multimodal emotion labels for utterance clips and is likewise evaluated at the utterance level. In both cases, the candidate memories are constructed strictly from within-utterance audio-visual sequences.

To avoid any ambiguity, the revised abstract explicitly identifies utterance-level emotion recognition, and Sections 1, 2.1, and 4.1 distinguish this utterance-level scope from conversational architectures that model multi-turn dialogue context. These evaluations therefore assess utterance-level multimodal emotion recognition; they do not evaluate inter-utterance conversational reasoning.

Q2: Design Distinction from TCR, QaP, and MSE-Adapter

TEB organizes text-conditioned evidence access around individual LLM-facing output positions. Each of the BB positions is assigned a transcript-conditioned query that retrieves separately from local-segment and learned-latent candidate memories. The query, both retrieved summaries, and the utterance context then jointly determine the mixing weights that update that position's state before token projection, keeping both candidate organizations separately accessible until each output token is formed.

Section 2.2 contrasts this pipeline directly with prior text-guided methods:

  • TCR resamples video features into fixed-length query outputs conditioned on task text from a single feature stream;
  • QaP integrates querying prompts with text inside the language backbone and injects modality information across internal layers via layer-wise resamplers;
  • MSE-Adapter applies text-guided Hadamard mixing and multi-scale fusion to temporally summarized audio-visual representations, then projects the pre-fused representation into pseudo tokens.

Rather than resampling a single stream, injecting layer-wise prompts inside the LLM, or projecting pre-fused representations, TEB operates prior to the frozen LLM through separately normalized reads over local-segment and learned-latent memories followed by a post-retrieval gate. As detailed in Section 3.3, separate reads produce two memory-specific summaries, which the gate then combines using the query, both summaries, and the utterance context. In contrast, joint attention determines candidate weights under a shared softmax before aggregating them into a single readout.

Two controlled comparisons provide empirical support for this design:

  1. Comparison against single-memory text conditioning (Table 3): At B=4B = 4 with ChatGLM3-6B, TEB improves five-seed mean MAE on SIMS-V2 from 0.2790.279 to 0.2650.265 and mean WF1 on CHERMA from 72.8072.80% to 74.1874.18% over the text-conditioned single-memory resampler, evaluating the complete dual-memory and access design beyond text conditioning alone.
  2. Same-candidate access controls (Table 4): With the same local and latent candidate-memory construction retained across access variants on CHERMA (B=4B=4, ChatGLM3-6B), TEB achieves 74.1874.18% mean WF1, outperforming both a single concatenated read (73.5373.53%) and a fixed mixture (73.2373.23%), directly linking the structural distinction to the benefit of separate retrieval and conditional mixing as a joint design.

W2 and Q4: Baseline Sensitivity to the Output Budget (B=1,4,32B = 1, 4, 32)

We evaluated TEB, MSE-Adapter, and the text-conditioned single-memory resampler across common budgets of B=1,4,32B = 1, 4, 32 on CHERMA using ChatGLM3-6B (five training seeds, without structured calibration). Each budget setting was trained independently under our unified comparison protocol. Section 4.3(c) and Appendix Table 7 report the full results, summarized in Table R1.

Table R1: Five-seed Weighted F1 (mean±sample std\text{mean} \pm \text{sample std}, %) across output-token budgets BB on CHERMA (ChatGLM3-6B). Bold indicates the highest mean WF1 at each budget.

InterfaceB=1B = 1B=4B = 4B=32B = 32
MSE-Adapter68.67 ± 1.1472.78 ± 0.3771.35 ± 0.85
Text-conditioned resampler72.90 ± 0.5772.80 ± 0.6972.35 ± 0.25
TEB (Ours)73.15 ± 0.5874.18 ± 0.2772.28 ± 1.27

Sensitivity to budget reduction differs across baseline architectures. Reducing BB from 44 to 11 lowers MSE-Adapter's mean WF1 by 4.114.11 points (from 72.7872.78% to 68.6768.67%), compared with 1.031.03 points for TEB, confirming that tight budget reduction substantially degrades an existing affect adapter while TEB retains much more of its performance. Meanwhile, the text-conditioned resampler remains relatively flat between B=1B=1 (72.9072.90%) and B=4B=4 (72.8072.80%), whereas TEB effectively utilizes the 44-token budget to reach the highest overall mean (74.18±0.2774.18 \pm 0.27%). Overall, TEB achieves the highest mean WF1 at both compact budgets (B=1B=1 and B=4B=4), while at B=32B=32 the resampler is higher by 0.070.07 points and none of the three interfaces improves over its B=4B=4 score.

These comparisons motivate studying candidate organization and evidence access at compact output budgets, without implying that every baseline deteriorates uniformly as BB shrinks. In Section 3, we explicitly define "bandwidth" as the adapter's LLM-facing output-token budget—an interface design condition distinct from the LLM's context window—and we have removed the 256-token Emotion-LLaMA example from the introduction.

Q5: Larger Budgets (B≥32B \ge 32) and Direct Memory Forwarding

In our extended budget sweep (Table R1), we scaled the output budget up to B=32B = 32 tokens—eight times the B=4B = 4 setting used in our main comparisons. Notably, increasing BB from 44 to 3232 lowers mean WF1 across all three evaluated interfaces (from 72.7872.78% to 71.3571.35% for MSE-Adapter, 72.8072.80% to 72.3572.35% for the resampler, and 74.1874.18% to 72.2872.28% for TEB). Thus, within the tested 8×8\times expansion range, compressing multimodal evidence into B=4B = 4 tokens does not incur a performance penalty relative to a much larger 3232-token budget.

Given the available single-GPU memory and our fixed-batch-size comparison protocol, we limited the extended cross-interface sweep to B=32B=32; budgets of 6464–256256 tokens were not evaluated. Direct memory-to-LLM forwarding remains a separate untested comparison: TEB at B=32B=32 still constructs tokens through retrieval, mixing, and prompt composition, so it is not an uncompressed reference. Performance relative to direct memory forwarding therefore remains unmeasured, as documented in Appendix E.

Reviewer_9FZtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes TEB, a lightweight adapter that helps a frozen LLM do multimodal sentiment analysis and emotion recognition. The LLM only gets a handful of soft tokens to carry the audio and visual information, and the authors frame this limit as a "bandwidth" constraint. The main idea is to use the transcript to decide which parts of the audio and video should go into those few tokens. It does this with text-guided queries that look at both fine-grained segments and a coarser summary of each stream. Only the adapter is trained, and the LLM stays frozen. The paper compares TEB with MSE-Adapter, the closest prior adapter, on several sentiment and emotion benchmarks and LLM backbones.

Critical strengths
  1. The paper studies adapter design in a controlled way, which is the right approach. It compares several ways of building the prompt tokens under the same token budget, with the same downstream components and the same frozen LLM, and it adds a retrained text-only control. It keeps published and in-house results separate, and openly reports that ATGFB-MFF scores higher in many settings. Matched comparisons like this are useful.
Critical weaknesses
  1. Reproducibility of the method and main results

I ran the released code as provided, with the released settings, on the one backbone I could fit (Qwen-1.8B) for SIMS-V2 and MELD. On SIMS-V2, over the same five seeds, I get MAE 0.333 +/- 0.025 versus 0.304 +/- 0.007 in the paper. On MELD, over three seeds, I get weighted F1 52.1 +/- 5.6 versus 61.05 +/- 0.36. In both cases my numbers are below the in-house MSE-Adapter results reported in the paper and the published MSE-Adapter results. On the same machine and data, I could reproduce MSE-Adapter's published SIMS-V2 result from their official code, so I don't think my setup explains the gap. The spread across seeds is also several times larger than the paper reports. The released code doesn't force deterministic training, and in practice retraining the same SIMS-V2 seed changed MAE by more than the reported five-seed standard deviation. So the reported deviations are much smaller than what I observe. Unless the reported runs were deterministic, the paper's per-seed claims (for example, that TEB beats the single-memory variant on CHERMA in all five matched seeds) can't be read as clean paired comparisons.

The released code also doesn't include the three alternative token generators in the controlled comparison, the in-house MSE-Adapter runs, or the calibration add-on. Yet the controlled comparison is the paper's core evidence, and the main improvement claim rests on the in-house MSE-Adapter rows. Those in-house baselines are sometimes weaker than the published ones. On ChatGLM3 for MELD, TEB ties the published MSE-Adapter on accuracy and is below it on weighted F1.

  1. Incremental novelty, statistical rigor, and the "bandwidth-constrained evidence allocation" claim is not fully supported

The individual parts of TEB are already well known. Text- or instruction-conditioned learned queries appear in InstructBLIP's instruction-aware Q-Former and LLaMA-VID's text-guided context token (neither cited), as well as TCR and QaP (which are cited). The latent memory is essentially Perceiver or Set Transformer style pooling with learned seeds. Segment pooling, a sigmoid gate between two attention reads, and a low-rank prompt dictionary are all standard. Within affect adapters, there are already MSE-Adapter's text-guided mixer, ATGFB-MFF, and DPDF-LQ. DPDF-LQ already combines a global and a local path with learnable query attention for sentiment analysis, just without an LLM. TEB is a reasonable combination of existing pieces, but it is incremental. The second listed contribution "theoretical innovation" is also an architecture description rather than a theoretical contribution or justification.

The bandwidth and allocation framing, which is the paper's central claim, is never really tested. The token budget is a design choice, not a real constraint. Frozen LLMs can take many more tokens, and, as noted in the paper, Emotion-LLaMA uses hundreds of video tokens. The token-budget sweep shows that a single token gets within about a point of four tokens, and eight tokens are no better. If that's the case, it's hard to argue that carefully allocating evidence across tokens is what drives the results. There is no reference that skips the bottleneck and passes the memory cells straight to the LLM, and no larger budgets are tried, so we can't tell whether compression loses anything. The allocation itself is also never analyzed. There is no look at what each query attends to, whether queries specialize, how the gate behaves, or which modality drives which token, as the paper also notes. In a quick check on one of my MELD models, zeroing all audio and visual input at test time barely changed the scores. Zeroing at test time adds some distribution shift, and the same check hurt a lot on SIMS-V2. Still, it suggests the "evidence tokens" may carry little audio-visual evidence in some settings. The "consistency discrepancy" signal is also weak. It compares pooled audio or visual features with an average of word embeddings, through separately learned projections that nothing pushes into a shared space. It is not clear that this measures audio-text consistency, and removing it barely changes the results.

The gains reported for the individual design choices are small and confounded. Against its closest variants (a single concatenated read, a fixed mix, and no discrepancy signal), TEB is mostly ahead by a fraction of a point. Running Welch tests on the reported means and standard deviations, none of these gaps are significant on SIMS-V2, and on CHERMA they are borderline at best. Across all the ablations, nothing survives a multiple-comparison correction. The full model also has more trainable parameters than most of the ablated variants, so even these small gaps are mixed up with capacity. With the larger seed variance I saw in my runs, they would likely disappear. The paper reports no significance tests or confidence intervals. ATGFB-MFF scores higher on MOSEI, MELD and CHERMA for every backbone, even allowing for protocol differences. For ICLR I'd expect either a clearly new idea or a convincing analysis of why the design works. Here the pieces are known, the central framing is untested, and the margins over close variants are small and confounded with capacity.

Questions for authors
  1. What are the exact configs used per backbone and dataset? With the released code and config I get different results. What is different from the runs in the paper?
  2. The main premise is that a small token budget forces the adapter to allocate evidence carefully. But one token gets within about a point of four, and frozen LLMs can easily take many more tokens. What happens if the memory cells are passed directly to the LLM without the bottleneck, or with much larger budgets? Without this comparison, how do we know the bottleneck matters?
  3. Is there any direct evidence that allocation happens? For example, do different queries attend to different segments or modalities, and do the gate values differ across queries and samples, or do they end up nearly the same?
  4. The local memory weights are computed from the average of all queries, so every query sees the same weighted local memory. In what sense is this weighting query-conditioned? Did you try per-query weights?
  5. The discrepancy signal compares pooled audio or visual features with averaged word embeddings through separately learned projections, and no objective aligns them. Why should this measure audio-text or visual-text consistency? SIMS-V2 provides unimodal labels, so does the signal correlate with samples where the text and audio or visual labels disagree?
  6. The queries are built from the transcript's token embeddings which are looked up directly from the LLM's input embedding table (not contextualized). Why not use contextual text features, for example the frozen LLM's hidden states? This seems like a more natural way to let the transcript guide evidence selection.
  7. How much of the gain over MSE-Adapter remains at matched trainable parameters? TEB has about twice as many trainable parameters as MSE-Adapter on Qwen.
  8. Nothing in the bandwidth and allocation idea is specific to affect. Have you tried TEB on any other audio-visual task, or with stronger learned audio and visual encoders instead of the low-dimensional handcrafted features used on MOSEI and SIMS-V2?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 9FZt

We thank Reviewer 9FZt for the detailed reproduction attempt and technical review. Below, we first address the discrepancy in our initial public code release (Response 1.1), then detail the manuscript revisions positioning TEB and its statistical scope, and finally present additional controlled experiments across token budgets, modality restrictions, and joint-read capacity. Table R1 summarizes the manuscript revisions.

Tables prefixed with R belong to this response; all other section, equation, table, and appendix references correspond to the revised manuscript.

Table R1: Summary of key manuscript revisions addressing Reviewer 9FZt's comments.

Review PointRevision in the ManuscriptLocation
Per-seed statementsDefined σ\sigma as five-seed sample SD (not a CI); framed seed win counts strictly as descriptive observations§4.1, §4.3(a), App. C.5
Published vs. in-house baselinesMarked published rows as cross-protocol references; explicitly noted where published scores exceed TEB§4.1, §4.2, Tables 1–2
"Theoretical innovation" framingRenamed contribution to reflect two-stage retrieval and conditional mixing design§1
Missing prior work citationsCited and compared InstructBLIP, LLaMA-VID, Perceiver, and DPDF-LQ§2.1, §2.2
Bandwidth & budget sweepsDefined BB as adapter output positions; added cross-interface comparisons at B=1,4,32B = 1, 4, 32§3, §4.3(c), App. Table 7
Modality & capacity controlsAdded retrained T+A / T+V controls on Qwen-1.8B (SIMS-V2), parameter table, and count-normalized residual joint read§4.3(b,d), Table 5, App. Tables 9, 13

W1 and Q1: Reproducibility, Code Release Scope, and Baseline Comparisons

1.1 Discrepancy between the initial release and paper runs (W1.1, Q1). Thank you for investing the time to run the released code and document the discrepancies. Upon auditing our initial public repository, we found that it contained a simplified implementation prepared for sharing rather than the complete adapter implementation used in our reported experiments. This simplification altered parts of the adapter computation without disclosing the resulting differences from the paper formulation. The mismatch is entirely on our side rather than your setup, and we sincerely apologize for this release error and the time it cost you.

We are preparing a corrected anonymous release that preserves the complete TEB computation described in the revised manuscript. We also distinguish between our original reported runs and a fresh verification of the corrected public release: this response does not claim that updating the code alone has re-verified every benchmark row from scratch. Section 4.1 and Appendix A describe the evaluation conventions and implementation in the manuscript, whereas full run-level reproducibility also requires pairing the corrected code release with the exact run configurations.

1.2 Determinism and descriptive per-seed reporting (W1.1). The current training script sets random seeds and enables cudnn.deterministic = True; this setting alone does not establish end-to-end deterministic training. We do not attribute the observed reproduction gaps or run-to-run variation to a particular GPU operation without verification. Section 4.1 defines the reported ±σ\pm\sigma as the sample standard deviation of the five training runs, not a confidence interval or a guarantee about repeated execution of a fixed seed. Section 4.3(a) and Appendix C.5 report seed-level win counts as descriptive observations, not as paired significance tests. These reporting clarifications are separate from resolving the implementation and configuration mismatch discussed above.

1.3 Scope of the public code release and baseline tables (W1.2). The public release is intended to provide a standalone implementation of full TEB. The alternative token generators, internal access ablations, structured-calibration extension, and in-house MSE-Adapter training pipeline are not bundled into that release, which limits direct re-execution of the controlled rows from the main repository alone. Section 3 and Appendices A–C specify the interface computations, control definitions, and evaluation conventions, while Appendix B concentrates the definitions of the alternative interfaces. These descriptions document the comparisons in detail, though they do not substitute for executable control implementations.

Tables 1 and 2 distinguish MSE-Adapter runs under our unified protocol from published reference values, with the comparison conventions stated in Section 4.1. Published results reflect their original source protocols, and we do not assign their differences from our unified-protocol runs to an unverified cause. The tables explicitly retain the higher published scores: for ChatGLM3-6B on MELD, published MSE-Adapter reports 66.4566.45% Acc / 65.3365.33% WF1 compared with TEB's 66.2366.23% / 64.9964.99%, and published ATGFB-MFF reports 68.9368.93% WF1. Section 4.2 discusses these published references alongside our protocol-matched comparisons.

W2: Architectural Positioning, Bandwidth Evaluation, and Statistical Scope

2.1 Positioning relative to prior work and contribution framing (W2.1). We have revised Section 1 and Section 2 to place TEB accurately within existing literature. Section 2.1 discusses DPDF-LQ's non-LLM global learnable-query and local cross-attention fusion, and Section 2.2 cites Perceiver-style latent pooling, InstructBLIP's instruction-aware queries, and LLaMA-VID's text-guided context tokens alongside TCR, QaP, MSE-Adapter, and ATGFB-MFF. We have also replaced the "Theoretical Innovation" contribution heading in Section 1 with a direct architectural statement of our two-stage retrieval and conditional mixing design.

TEB's contribution lies in the joint design of candidate organization and output-specific evidence access. Each of the BB LLM-facing output positions retrieves one summary from local-segment memory and one from learned-latent memory under separate attention normalizations. A gate then combines the summaries using the query, both summaries, and utterance context (Section 3.3), whereas joint attention determines candidate weights under a shared softmax before aggregation. While DPDF-LQ fuses global and local representations for direct sentiment prediction, TEB retains two candidate organizations for separate retrieval and conditional mixing at each LLM-facing output position. This places the contribution at the interface-design level rather than claiming text conditioning or latent pooling as new primitives.

2.2 Cross-budget evaluation and scope of the bandwidth formulation (W2.2, Q2). In Section 3, we define "bandwidth" strictly as the number of LLM-facing output positions BB exposed by the adapter—an interface design condition rather than a context-length limit of the frozen LLM. To test how TEB and existing baselines behave across different budgets, we completed a five-seed comparison of TEB, MSE-Adapter, and the text-conditioned single-memory resampler at B=1,4,32B = 1, 4, 32 on CHERMA with ChatGLM3-6B (Section 4.3(c) and Appendix Table 7), summarized in Table R2.

Table R2: Five-seed Weighted F1 (mean±sample std\text{mean} \pm \text{sample std}, %) across output-token budgets BB on CHERMA (ChatGLM3-6B). Bold indicates the highest mean WF1 at each budget.

InterfaceB=1B = 1B=4B = 4B=32B = 32
MSE-Adapter68.67 ± 1.1472.78 ± 0.3771.35 ± 0.85
Text-conditioned resampler72.90 ± 0.5772.80 ± 0.6972.35 ± 0.25
TEB (Ours)73.15 ± 0.5874.18 ± 0.2772.28 ± 1.27

Reducing BB from 44 to 11 lowers MSE-Adapter's mean WF1 by 4.114.11 points, compared with 1.031.03 points for TEB, while the text-conditioned resampler remains nearly flat between B=1B=1 (72.9072.90%) and B=4B=4 (72.8072.80%). TEB achieves the highest mean at B=1B=1 (73.15±0.5873.15 \pm 0.58%) and B=4B=4 (74.18±0.2774.18 \pm 0.27%). At B=32B=32, the resampler's mean is 0.070.07 points higher than TEB's, and increasing the budget from 44 to 3232 tokens improves none of the three interfaces. These results support TEB's effectiveness at compact output budgets and show that budget sensitivity differs across interfaces. TEB at B=32B=32 still uses retrieval, mixing, and prompt composition, so it is not an uncompressed pass-through reference (Appendix E).

2.3 Modality contributions vs. test-time zeroing (W2.2). We added separately retrained T+A (visual branch removed) and T+V (audio branch removed) controls on SIMS-V2 using Qwen-1.8B (B=4B=4, five seeds; Appendix Table 9). These controls evaluate the predictive contribution of each modality under retraining:

Table R3: Five-seed MAE (↓\downarrow, mean±sample std\text{mean} \pm \text{sample std}) for separately retrained modality controls on SIMS-V2 (Qwen-1.8B, B=4B=4). Bold indicates the lowest mean MAE.

Retrained Input ConditionMAE (↓\downarrow)
T+A (Transcript + Audio only, visual branch removed)0.425 ± 0.034
T+V (Transcript + Visual only, audio branch removed)0.370 ± 0.016
T+A+V (Full TEB)0.304 ± 0.007

Full TEB achieves lower MAE than either retrained modality-restricted variant and better means across all five reported metrics, supporting predictive contributions from both audio and visual inputs in this setting. The ChatGLM3-6B text-only comparisons provide additional evidence: full TEB reduces SIMS-V2 MAE from 0.3340.334 to 0.2650.265 and raises CHERMA WF1 from 52.9252.92% to 74.1874.18% (Section 4.3(d), Appendix Table 8). Because retraining without a modality addresses a different question from zeroing inputs to an already-trained model at test time, these controls do not directly resolve the test-time zeroing observation on your Qwen/MELD checkpoint, for which no separate intervention is reported in the revised manuscript.

Reviewer_RagwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a bandwidth-constrained evidence allocation framework (Text-Conditioned Evidence Bottleneck, TEB) to enhance LLM multimodal sentiment analysis and emotion recognition. Based on the given B continuous multimodal prompt token bandwidth, TEB establishes B transcript-conditioned query states, and organize the audio-visual representation into the local segment memory and learned latent memory. Experimental results on three LLMs and four public datasets illustrate that the proposed method is more effective than the compared existing methods.

Critical strengths

This work introduces a new method (TEB), a bandwidth-constrained evidence allocation framework, which offers a reference adapter design for LLM multimodal sentiment analysis and emotion recognition. The motivation is clear and easily followed, and the description is also readable. The proposed method seems interesting as well. A lot experiments are conducted to validate the performance of the proposed method.

Critical weaknesses
  1. For this work, the motivation is clear and easily followed in the part of introduction, but the part of ‘Method’ is hard to understand. In fact, I think that the introduction of the proposed method is chaotic in this part, and the writing quality is worse than in other parts, so that the method’s novelty is difficult to follow. These introduced sub-parts in TEB do not correspond to Figure 1, and it is unclear which part corresponds to Figure 2 (even the description of Figure 2 is missing in the main text; only the figure is shown). The keys for each part are unclear, merely indicating that they are given in the supplementary materials. I think Supplementary materials should be complementary, and the core of the proposed method should be explained as clearly as possible in the main text.

  2. Although many experiments are conducted to validate the effectiveness of the proposed method, the core ‘bandwidth-constrained’ claim is not directly examined by the cross-bandwidth comparative experiment. In this paper, it is mentioned that the more adapter interface limited, the more important it is to rationally divide the evidence. Whereas, the current sweep only reports the TEB, but doesn't have the curve of MSE-Adapter or single-memory resampler under different B. Thus, it may not directly judge whether the advantages of TEB improved with bandwidth shortening. It admits that the current sweep could not prove a comparative low-budget advantage.

Questions for authors
  1. Could the author add a parameter-matched or computed-matched control for the strongest single-memory baseline? Except for the fixed output token number, the current TEB still adds memory construction, retrieval path, gating and trainable capacity, which could not exclude the extra factors' influence.

  2. How about the change of TEB’s advantages compared with the baseline when B is smaller? Be stronger?
    Compare and draw the performance-vs-bandwidth curves of MSE-Adapter and the text-conditioned single-memory resampler?

  3. Could the author rerun the ATGFB-MFF under the exact same situation, with the same feature input, the decoding rule, random seed, and checkpoint-selection protocol? Because the current result of ATGFB-MFF is higher than TEB under multiple settings, this comparison may influence the judgment of its empirical competitiveness.

  4. The current text-only experiment could only prove that the audio-visual input helps, but could not show that it has learned the truly meaningful audio-vs-visual evidence allocation. Could it add audio-only, visual-only, audio-shuffled, visual-shuffled, or other modality-specific controls?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ragw

We sincerely thank you for your constructive feedback, particularly regarding the clarity of Section 3 and the need for comparative evidence across output token budgets and modality controls. In response, we have:

  1. Reorganized Section 3 around the step-by-step construction of each evidence token, moving all core mathematical definitions into the main text and aligning the subsections directly with Figures 1 and 2;
  2. Completed cross-budget comparisons (B=1,4,32B = 1, 4, 32) across TEB, MSE-Adapter, and the text-conditioned resampler under a unified protocol;
  3. Added retrained modality-restricted controls (T+A\text{T+A} and T+V\text{T+V}) and an expanded joint-read control with count normalization and a residual mixer.

Tables prefixed with R belong to this response; all other section, equation, table, and figure references correspond to the revised manuscript.

W1: Method Clarity and Correspondence with Figures 1 and 2

We have restructured Section 3 so that the narrative tracks each of the BB output positions sequentially through query construction, candidate memory creation, retrieval, conditional mixing, and final token generation. Specifically, each transcript-conditioned query retrieves one summary from the local-segment memory and one from the learned-latent memory. The query, both retrieved summaries, and the utterance context then jointly determine the mixing weights used to update the query before projecting it into an LLM-facing evidence token.

The four subsections of Section 3 now mirror the four stages shown in Figure 2:

  • Stage I / Section 3.1 (Transcript-Conditioned Queries & Context Modulation): Defined in Eqs. 2–3.
  • Stage II / Section 3.2 (Local-Segment & Learned-Latent Candidate Memories): Defined in Eqs. 4–6.
  • Stage III / Section 3.3 (Separate Retrieval & Adaptive Evidence Mixing): Defined in Eqs. 7–8.
  • Stage IV / Section 3.4 (Token Projection & Task-Supervised Learning): Defined in Eqs. 9–10.

All core tensor definitions and operations now reside in the main text, while implementation details are placed in Appendix A. Furthermore, the caption of Figure 1 explicitly links the high-level pipeline blocks to Sections 3.1–3.4, and Figure 2 is introduced in the method overview with explicit pointers to Eqs. 3 and 5 for context conditioning and memory construction.

Section 3.3 also explains why retrieval precedes mixing: each memory applies its own attention normalization, and the gate combines the retrieved summaries using their content, the query, and the utterance context. Joint attention also produces input-dependent weights, but normalizes over both candidate sets before aggregation. This distinction connects Eqs. 7–8 to the concatenated-read and fixed-mixture controls in Table 4.

W2 and Q2: Comparative Performance Across Output Budgets

We completed the cross-bandwidth comparison at B=1,4,32B = 1, 4, 32 for TEB, MSE-Adapter, and the text-conditioned single-memory resampler on CHERMA using ChatGLM3-6B (five training seeds, without structured calibration). Each budget setting was trained independently under our unified comparison protocol. The results (reported in Section 4.3(c) and Appendix Table 7) are summarized in Table R1.

Table R1: Five-seed Weighted F1 (mean±sample std\text{mean} \pm \text{sample std}, %) across output-token budgets BB on CHERMA (ChatGLM3-6B). Bold indicates the highest mean WF1 at each budget.

InterfaceB=1B = 1B=4B = 4B=32B = 32
MSE-Adapter68.67 ± 1.1472.78 ± 0.3771.35 ± 0.85
Text-conditioned resampler72.90 ± 0.5772.80 ± 0.6972.35 ± 0.25
TEB (Ours)73.15 ± 0.5874.18 ± 0.2772.28 ± 1.27

Key Observations:

  1. Behavior at the smallest budget (B=4→1B = 4 \to 1): How TEB's margin evolves depends on the baseline architecture. Reducing BB from 44 to 11 lowers MSE-Adapter's mean WF1 by 4.114.11 points, compared with 1.031.03 points for TEB. TEB's lead over MSE-Adapter therefore widens from +1.40+1.40 to +4.48+4.48 points. Against the text-conditioned resampler, TEB's lead is +0.25+0.25 points at B=1B=1 and +1.38+1.38 points at B=4B=4. TEB benefits from increasing the budget from 11 to 44 tokens, whereas the resampler's mean remains similar (72.9072.90% versus 72.8072.80%). This difference highlights that the value of additional output positions depends on the interface design. TEB at B=4B=4 achieves the highest mean WF1 among these evaluated settings (74.18±0.2774.18 \pm 0.27%).
  2. Performance at the larger budget (B=32B = 32): Increasing the budget from B=4B=4 to B=32B=32 improves none of the three interfaces. The resampler's mean is 0.070.07 points higher than TEB's at B=32B=32, and both are below their B=4B=4 scores.

Overall, TEB achieves the highest mean performance at compact budgets (B=1B=1 and B=4B=4), though its advantage over the single-memory resampler decreases as BB shrinks from 44 to 11. Together, the compact-budget comparisons and the lack of improvement from 44 to 3232 tokens support designing evidence access for the intended output budget. Here, "bandwidth" refers to the adapter's output-token budget rather than the LLM's context window.

Q1: Parameter/Compute Matching and Expanded Joint-Read Control

To examine a joint read with additional mixing capacity and cell-count balancing, we introduced a count-normalized concatenated read with a residual mixer. It retains the same local and latent candidate-memory construction (Section 4.3(b), Appendix C.1, and Table 13). Specifically, this variant corrects the joint-attention logits for each memory's valid cell count and appends an extra residual MLP mixer after retrieval. On CHERMA with ChatGLM3-6B (B=4B=4), this expanded variant achieves a five-seed WF1 of 73.47±0.7573.47 \pm 0.75%, compared with 73.53±0.7173.53 \pm 0.71% for the original concatenated read and 74.18±0.2774.18 \pm 0.27% for TEB.

The augmented joint-read control remains below TEB in mean WF1, despite retaining both candidate organizations and adding a residual mixer. This provides an additional comparison of complete access designs at the same output budget. Because this control modifies count correction and mixing capacity together, and a standalone parameter- or compute-matched text-conditioned resampler was not evaluated, our comparisons match the output-token budget and shared training protocol while trainable capacity and computation vary across interfaces. Section 4.1 and Appendix C.1 state this scope; Table 5 reports TEB's trainable parameter counts across backbones and datasets.

Q3: Evaluation of ATGFB-MFF Under a Shared Protocol

An official implementation of ATGFB-MFF was not available to us when we attempted reproduction. We reconstructed the method from the original paper and made several adjustments under our pipeline, but performance remained substantially below the published values. Because we could not verify that our paper-based reconstruction faithfully recovered the original method, we refrained from reporting those lower scores as a unified-protocol baseline.

Instead, the revised manuscript retains ATGFB-MFF's originally published scores in the main benchmark tables and explicitly documents the protocol differences in Section 4.1 (noting that their reported pipeline uses different feature extractors and, for regression tasks, interval-based decoding). Section 4.2 also explicitly highlights settings where published ATGFB-MFF scores exceed TEB (e.g., 68.9368.93% vs. 64.9964.99% WF1 on MELD with ChatGLM3-6B). Accordingly, while ATGFB-MFF's published results serve as an important cross-protocol reference for empirical competitiveness, our controlled mechanistic claims are scoped strictly to the interfaces evaluated under our unified protocol, and we do not claim superiority over ATGFB-MFF under matched conditions.

Q4: Modality-Specific Controls and Scope of Evidence Allocation

To assess each non-text modality's contribution, we separately retrained two restricted-input variants on SIMS-V2 using Qwen-1.8B (B=4B=4, five seeds). T+A\text{T+A} excludes the visual branch, and T+V\text{T+V} excludes the audio branch; both retain the transcript and follow the full model's comparison protocol. Section 4.3(d) and Appendix Table 9 report the full five-metric evaluation; Table R2 summarizes the MAE results.

Table R2: Five-seed MAE (↓\downarrow, mean±sample std\text{mean} \pm \text{sample std}) for separately retrained modality controls on SIMS-V2 (Qwen-1.8B, B=4B=4). Bold indicates the lowest mean MAE.

Retrained Input ConditionMAE (↓\downarrow)
T+A\text{T+A} (Transcript + Audio only, visual branch removed)0.425 ± 0.034
T+V\text{T+V} (Transcript + Visual only, audio branch removed)0.370 ± 0.016
T+A+V\text{T+A+V} (Full TEB)0.304 ± 0.007

With the visual branch excluded (T+A\text{T+A}), mean MAE rises from 0.3040.304 to 0.4250.425; excluding the audio branch (T+V\text{T+V}) raises it to 0.3700.370. Full TEB outperforms both restricted-input variants across all five reported metrics, supporting complementary predictive contributions from both audio and visual inputs in this setting. Moreover, T+V\text{T+V} outperforms T+A\text{T+A}, showing unequal effects of the two input restrictions.

Finally, Section 4.3(d) and Appendix C.3 distinguish modality-level contribution from query-level evidence allocation. The retrained controls support the predictive contribution of retaining each non-text modality in this setting, but do not identify which modality drives an individual output token. Test-time cross-sample shuffling would instead probe a trained model's sensitivity to disrupting the correspondence between an audio or visual input and the current utterance; this intervention was not evaluated here.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.