acceptodds
Under review as a conference paper at ICLR 2027

LayerRefit: Final-Relative Refinement for Affective Multimodal LLMs

Abstract

Encoder hierarchies contain affective cues whose utility varies across modalities and conditions. We introduce LayerRefit to adapt pretrained multimodal large language models (MLLMs) by converting this evidence into residual corrections of their native modality tokens. Depth-specific evidence projections transform differences from the final encoder state; a shared correction projection maps them into token updates, combined with input-dependent weights. Backbone weights and input sequence length remain fixed. Across three affective backbones, source-trained branches improve 22 of 27 cross-benchmark evaluations. Models trained with audio, vision, and text improve 11 of 12 inference-time input configurations on the two primary backbones without retraining, using fewer than 0.6M added parameters. Ablations favor final-relative evidence with shared token correction, and norm-matched interventions show that learned update directions improve recognition. Code: https://anonymous.4open.science/r/LayerRefit-7263/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Affective Multimodal Large Language Models, Encoder-Depth Utilization, Representation Refinement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ND3WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LayerRefit, a lightweight adaptation method that uses intermediate encoder features to generate low-rank residual updates to post-connector modality tokens while keeping the backbone frozen and token count unchanged. Experiments across three affective multimodal models show gains on most benchmarks and input configurations, with ablations examining the contribution of its main design choices.

Critical strengths
  1. Each backbone’s refinement branch is selected using source validation and then held fixed across nine target benchmarks. Gains in 22 of 27 evaluations, with average improvements on all three backbones, give the method a broader empirical basis than a single-model or single-dataset study. This is the strongest evidence supporting its usefulness.
  2. A branch trained with audio, vision, and text improves 11 of 12 inference-time input configurations across the two primary backbones without retraining. The gains therefore persist when some training modalities are absent, which adds practical value when the available inputs vary.
Critical weaknesses
  1. The novelty over existing multi-layer fusion methods is not sufficiently established. Prior work such as Dense Connector, MMFuser, and instruction-guided fusion already incorporates intermediate encoder features into multimodal representations. LayerRefit introduces a particular combination of final-relative evidence, shared low-rank projection, and residual token correction, but the paper does not convincingly establish the benefit of this combination over closely related alternatives. Although Table 2 includes fusion and adaptation baselines, the DF baseline lacks a precise architectural description, and comparisons with identifiable fusion methods are limited. This makes it difficult to assess whether the gains reflect a substantive methodological advance or choices in parameter allocation and training.
  2. The edit-location ablation does not isolate the effect of where the correction is applied. In Table 3, post-connector editing achieves 61.87 Macro-F1 with 0.592M added parameters, whereas pre-connector editing achieves 59.68 with only 0.161M. Since both location and capacity change, the performance gap cannot be attributed to location alone. A parameter-matched comparison, or a capacity sweep at both locations, would provide stronger evidence for the choice of post-connector editing.
  3. The benefit of the final-relative formulation is not clearly isolated. Table 3 compares intermediate-only and final-layer-only inputs, but lacks a simple baseline that jointly takes HiH_i and HLH_L, for example by concatenating their normalized features before projection. Such a comparison would help determine whether the gain comes from subtraction itself or from giving the correction branch access to both representations. Layer normalization alone does not explain why direct subtraction is preferable, and the geometric analyses do not directly address this choice. A joint-input baseline with a comparable parameter budget would make this design claim more convincing.
Questions for authors

See weakness, please.

Reviewer_f4k6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The study proposes LayerRefit, a parameter-efficient method for adapting pretrained affective multimodal LLMs by calculating the normalized differences across encoder states plus input-dependent corrections with a shared low-rank output projection. Experiments over three backbones and nine benchmarks show that LayerRefit improves on most of the backbone–benchmark combinations with little added compute.

Critical strengths
  1. LayerRefit improves performance across affective backbones and benchmarks while preserving pretrained weights and token counts. Also the latency increase is modest, supporting its practical appeal.

  2. LayerRefit also shows flexibility with incomplete modalities, and when modalities available at inference differ from what are used during adaptation.

  3. The experiments also ablate alternative feature choices, correction structures and directions, supporting each components design contributes to the gains.

Critical weaknesses
  1. From Table 1 the gains from LayerRefit are modest, which improves Avg-9 by 1.2–2.1 points, and some results show small declines or no change: Emotion-LLaMAv2 + MOSEI (−0.16), AffectGPT + OV-MERD+ (−0.12), and Light-MER + MOSI (0.00).
  2. Some comparison experiments change multiple factors. In Table 3, replacing input-dependent weighting with uniform aggregation also reduces the parameter count (0.592M -> 0.403M). In Table 2, H-LoReFT and LayerRefit edit representations at different locations. The benchmark also changes across comparison experiments: Table 2 evaluates alternative adapters only on MER2024, Table 3 tests structural ablations only on AffectGPT. These choices should be clarified as it makes it harder to identify which parts of the method drive the gains and whether the advantages generalize across tasks and backbones.
  3. The paper states that configurations were selected using source-validation NLL. In all three grids of Table 6, source-validation and target rankings agree perfectly. Given how sensitive results are to the configuration, authors can report source-validation NLL and clarify selection procedure to help verify that the choices were made independently of target performance.
Questions for authors
  1. To validate the significance of LayerRefit, can you report standard deviations or confidence intervals for Tables 1–3?
  2. Can you explain the evaluation scope like benchmark selection in Tables 2–3 and provide comparisons that change only one factor?
  3. Can you check source-validation NLL for configuration selection and clarify how the experiment configurations were selected before target evaluation?
Reviewer_ku3HOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LayerRefit proposes a parameter-efficient adaptation method for multimodal emotion understanding models. It extracts task-relevant evidence from intermediate audio and visual encoder layers, combines this evidence through input-conditioned projections, and applies a low-rank residual correction to native modality tokens while keeping the pretrained encoders, connector, and LLM frozen. Experiments on three backbones and nine benchmarks report improved recognition performance with fewer than 0.6M additional parameters and modest inference overhead. The paper also investigates the contributions of intermediate-layer evidence, correction structure, and correction direction.

Critical strengths
  1. LayerRefit achieves improvements across multiple backbones with fewer than 0.6M additional parameters. Comparisons against parameter-matched alternatives further consolidate its strategy of integrating intermediate-layer information.
  2. The ablations examine evidence conditioning, correction structure, depth selection, and edit location. These experiments provide useful evidence and insights than benchmark numbers alone.
Critical weaknesses
  1. The lack of uncertainty estimates weaken confidence in the reported benchmark gains. Three-seed averages alone do not establish that the observed differences are robust.
  2. The method combines established adaptation techniques, while evaluation is concentrated on three affective MLLMs and a single 864-example source pool. The incomplete overlap audit further limits confidence in the breadth of the cross-benchmark generalization claims.
  3. Many architectural ablations use a fixed six-class candidate-label readout, whereas training uses full-vocabulary autoregressive supervision. Improvements under this restricted evaluation may not necessarily translate to better open-ended emotion understanding or generation.
Questions for authors
  1. Does the overlap audit cover all nine target benchmarks, including near-duplicate recordings and shared original media? Is the source pool represented in AffectGPT's original training data?
  2. Which specific component of LayerRefit accounts for its advantage over parameter-matched representation-editing and depth-fusion baselines? Does this advantage persist consistently across all three backbones?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Key Findings

We thank the reviewers for their thoughtful feedback and recognition of LayerRefit’s practical value. Our detailed responses provide additional evidence addressing the main concerns:

  • Reliability and practical value. Ten-seed evaluations yield positive paired 95% confidence intervals for Avg-9 gains on all three original backbones. These results complement the original improvements in 22/27 benchmark settings and 11/12 inference-time input configurations, using compact source-trained branches without target-specific retraining.

  • Evidence for the design. Budget-, rank-, and initialization-controlled comparisons support final-relative refinement beyond simply accessing intermediate and final states. On AffectGPT/MER2024, advantages of 2.14 Macro-F1 and 2.09 generation-HIT points remain over joint-input refinement with matched ranks and initial reader outputs. A four-budget sweep also supports post-connector editing at comparable adaptation cost.

  • Broader applicability and generated outputs. The advantage over budget-matched joint-input refinement extends across four backbones, including Qwen2.5-Omni. Native-generation and free-form description evaluations show improvements in emotion prediction and relevant descriptive content. We also report the nine-target overlap audit and original source-validation selection records.

Together, these findings support LayerRefit’s contribution: a compact, reusable adaptation design that turns encoder-depth evidence into effective corrections while preserving pretrained weights and the native token interface.

We hope these clarifications and additional results address the reviewers’ concerns and provide a stronger basis for considering the contribution and practical value of our work. We would sincerely appreciate hearing whether any concerns remain, and welcome the opportunity to address them during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.