- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes LayerRefit, a lightweight adaptation method that uses intermediate encoder features to generate low-rank residual updates to post-connector modality tokens while keeping the backbone frozen and token count unchanged. Experiments across three affective multimodal models show gains on most benchmarks and input configurations, with ablations examining the contribution of its main design choices.
Critical strengths
- Each backbone’s refinement branch is selected using source validation and then held fixed across nine target benchmarks. Gains in 22 of 27 evaluations, with average improvements on all three backbones, give the method a broader empirical basis than a single-model or single-dataset study. This is the strongest evidence supporting its usefulness.
- A branch trained with audio, vision, and text improves 11 of 12 inference-time input configurations across the two primary backbones without retraining. The gains therefore persist when some training modalities are absent, which adds practical value when the available inputs vary.
Critical weaknesses
- The novelty over existing multi-layer fusion methods is not sufficiently established. Prior work such as Dense Connector, MMFuser, and instruction-guided fusion already incorporates intermediate encoder features into multimodal representations. LayerRefit introduces a particular combination of final-relative evidence, shared low-rank projection, and residual token correction, but the paper does not convincingly establish the benefit of this combination over closely related alternatives. Although Table 2 includes fusion and adaptation baselines, the DF baseline lacks a precise architectural description, and comparisons with identifiable fusion methods are limited. This makes it difficult to assess whether the gains reflect a substantive methodological advance or choices in parameter allocation and training.
- The edit-location ablation does not isolate the effect of where the correction is applied. In Table 3, post-connector editing achieves 61.87 Macro-F1 with 0.592M added parameters, whereas pre-connector editing achieves 59.68 with only 0.161M. Since both location and capacity change, the performance gap cannot be attributed to location alone. A parameter-matched comparison, or a capacity sweep at both locations, would provide stronger evidence for the choice of post-connector editing.
- The benefit of the final-relative formulation is not clearly isolated. Table 3 compares intermediate-only and final-layer-only inputs, but lacks a simple baseline that jointly takes and , for example by concatenating their normalized features before projection. Such a comparison would help determine whether the gain comes from subtraction itself or from giving the correction branch access to both representations. Layer normalization alone does not explain why direct subtraction is preferable, and the geometric analyses do not directly address this choice. A joint-input baseline with a comparable parameter budget would make this design claim more convincing.
Questions for authors
See weakness, please.
Summary of Key Findings
We thank the reviewers for their thoughtful feedback and recognition of LayerRefit’s practical value. Our detailed responses provide additional evidence addressing the main concerns:
Reliability and practical value. Ten-seed evaluations yield positive paired 95% confidence intervals for Avg-9 gains on all three original backbones. These results complement the original improvements in 22/27 benchmark settings and 11/12 inference-time input configurations, using compact source-trained branches without target-specific retraining.
Evidence for the design. Budget-, rank-, and initialization-controlled comparisons support final-relative refinement beyond simply accessing intermediate and final states. On AffectGPT/MER2024, advantages of 2.14 Macro-F1 and 2.09 generation-HIT points remain over joint-input refinement with matched ranks and initial reader outputs. A four-budget sweep also supports post-connector editing at comparable adaptation cost.
Broader applicability and generated outputs. The advantage over budget-matched joint-input refinement extends across four backbones, including Qwen2.5-Omni. Native-generation and free-form description evaluations show improvements in emotion prediction and relevant descriptive content. We also report the nine-target overlap audit and original source-validation selection records.
Together, these findings support LayerRefit’s contribution: a compact, reusable adaptation design that turns encoder-depth evidence into effective corrections while preserving pretrained weights and the native token interface.
We hope these clarifications and additional results address the reviewers’ concerns and provide a stronger basis for considering the contribution and practical value of our work. We would sincerely appreciate hearing whether any concerns remain, and welcome the opportunity to address them during the discussion.