- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how affective fine-tuning changes the functional organization of multimodal foundation models. Through parameter analysis across 13 model instances and controlled experiments on representative models, the authors find that FFN-only adaptation outperforms attention-only adaptation under approximately matched trainable-parameter budgets. They further distinguish standalone adaptation capacity from contributions learned during joint optimization: although the gate, up, and down projections perform similarly when trained independently, recovery and intervention experiments identify the gate projection as a particularly influential component after joint adaptation. Checkpoint analyses track the development of this asymmetry during training. Motivated by these findings, the authors propose Gate-Focused Efficient Tuning (GET), which trains LoRA adapters only on gate projections and retains 96.2–98.0% of the All-Proj scores using 19.3–24.5% of its trainable parameters.
Critical strengths
-
The paper separates what a projection can learn independently from the role its updates assume when multiple projections are optimized together. This distinction is more informative than simply identifying the largest parameter updates or the most damaging ablations.
-
The study combines weight-change measurements, parameter-matched adaptation, selective recovery, removal/randomization/reversal interventions, compositional recovery, and checkpoint analysis. The intervention results are particularly noteworthy: removing the gate updates reduces AffectGPT performance by 10.64 points and OneEmo performance by 5.47 points, providing evidence that the observed asymmetry has functional consequences.
Critical weaknesses
- The interventions establish unequal dependence on learned updates, but do not sufficiently distinguish specialization from perturbation sensitivity.
Removing or reversing a projection’s updates changes both their learned information and the magnitude of the perturbation applied to the network. Figure 5 also shows that the gate projection accumulates larger updates in the analyzed trajectory. Consequently, its greater intervention sensitivity could partly reflect larger perturbations, its multiplicative position in the FFN, or stronger interactions with other adapted components.
The recovery experiments provide valuable complementary evidence, but recovery also depends on compatibility with the otherwise unadapted model. Similar standalone performance does not rule out architectural or optimization biases during joint training. Norm-matched and activation-matched perturbations, explicitly specified randomization procedures, and continuous intervention-strength sweeps would help distinguish these explanations. Evaluating general capabilities after the same interventions would further clarify whether the observed damage is disproportionately affective.
- The evidence for affect-related specificity and generality remains limited.
The non-affective controls differ in tasks, backbone configurations, evaluation metrics, and adaptation protocols. Their lower GET-to-All-Proj ratios therefore cannot isolate affective content as the reason gate-focused adaptation works well. The authors acknowledge this limitation, but a stronger comparison would hold the backbone, data scale, training procedure, and output format fixed while varying the task.
Moreover, the 13-model breadth primarily concerns parameter-change statistics. The targeted intervention analysis covers AffectGPT and OneEmo, while the main functional recovery trajectory is substantially narrower. The scope of the stronger functional conclusions should be distinguished from the broader descriptive survey.
- GET’s advantage over other equally small adaptation strategies is not established.
Table 9 already reports strong alternatives: gate-only adaptation achieves 73.68, while down-only adaptation achieves 73.43 under approximately matched parameter budgets. The 0.25-point difference is small and is not accompanied by uncertainty estimates. The paper’s own finding of comparable standalone FFN capacity therefore raises an important practical question: does selecting the gate based on post-training specialization improve adaptation beyond simply selecting an FFN projection?
Comparisons against lower-rank All-Proj LoRA, lower-rank FFN-only LoRA, and adaptive rank allocation under the same total trainable-parameter budget are missing. Gate-, up-, and down-only comparisons should also be extended beyond the principal AffectGPT experiment.
Questions for authors
- Does the gate projection remain the most influential pathway when interventions are matched by update norm or induced activation change? How exactly are randomized updates generated, and are their norms and ranks preserved?
- Can you construct affective and non-affective comparisons using the same backbone and matched training conditions? Do gate interventions selectively harm affective performance, or do they cause similarly large degradation on general capabilities?
4 comments
Response to Reviewer Tsqu (Part 2/3)
R3-W1 & Q2. General-Capability Effects of the Same Interventions
Setup. We evaluate affective and general performance on the same affectively adapted AffectGPT All-Proj checkpoint under the norm- and rank-preserving randomization described above. Each perturbed model is evaluated on both task groups without resampling the perturbation; all non-target weights remain unchanged.
Results. Gate randomization reduces affective performance from 76.30 to 67.93, while the general score increases from 22.50 to 22.85 (Table R3.4). Thus, the affective decline is not accompanied by a corresponding general-score decline in this evaluation. Other projections show a similar contrast, so this pattern is not exclusive to gate_proj. It characterizes different responses of the evaluated affective and general tasks to the learned updates.
Table R3.4. Affective and general performance under identical projection interventions. Parentheses show changes from each evaluation’s own baseline; score differences across the two evaluation scales are not treated as equivalent.
| Perturbed projection | Affective evaluation | General evaluation |
|---|---|---|
| Unperturbed All-Proj | 76.30 | 22.50 |
| q | 68.34 (↓7.96) | 23.12(↑0.61) |
| k | 68.83 (↓7.47) | 23.20(↑0.70) |
| v | 68.93 (↓7.37) | 22.59(↑0.09) |
| o | 69.76 (↓6.54) | 23.12(↑0.62) |
| gate | 67.93 (↓8.37) | 22.85(↑0.35) |
| up | 68.79 (↓7.51) | 23.03(↑0.53) |
| down | 69.24 (↓7.06) | 22.85(↑0.35) |
R3-W2 & Q2. Same-Backbone Affective/Non-Affective Comparison
Matched Training Conditions.
Setup. Using the same AffectGPT framework and pretrained base checkpoint, we match training data scale (maintain consistency with the data scale of MER25), training procedure, and output format while adapting on either MER25 affective data or VALOR-32K non-affective multimodal data. We compare All-Proj and GET in both settings. Affective evaluation follows the original eight-benchmark protocol; non-affective evaluation uses OmniBench accuracy. Both use zero-shot evaluation.
Results. GET retains 96.6% of All-Proj performance in the affective setting and 86.5% in the non-affective setting (Table R3.5). This comparison controls the backbone and training conditions that differed in the original public-model comparisons. Evaluation datasets and metrics remain different, and the OmniBench reference score is close to its 25% chance level, limiting interpretation of that retention ratio. We therefore report the contrast as a description of these two settings, rather than attributing it solely to affective content.
Table R3.5. Affective/non-affective adaptation with the same backbone and matched training conditions. Retention is GET’s score divided by the corresponding All-Proj score within each setting. The affective score averages eight benchmarks; the non-affective score is OmniBench accuracy.
| Task setting | Training data | All-Proj | GET | Retention |
|---|---|---|---|---|
| Affective understanding | MER25 | 76.30 | 73.68 | 96.6% |
| General multimodal understanding | VALOR-32K | 25.48 | 22.05 | 86.5% |
R3-W2. Coverage of the Survey and Functional Analyses
Our study combines a 13-instance parameter survey spanning 9 model designs, controlled Attention/FFN adaptation and GET comparisons on three affective models, and projection-level recovery analysis on AffectGPT and targeted intervention analyses on AffectGPT and OneEmo. During rebuttal, we added further intervention controls, alongside the task comparisons reported above. Within these functional analyses, AffectGPT’s checkpoint recovery (§4.4, Figure 5(b)) examines how contribution differences develop during training. Parameter-update dynamics are additionally analyzed on EAI-LLM (Appendix A.5), separately from functional recovery.
We will make these complementary scopes explicit throughout the manuscript: the broad survey characterizes parameter-change patterns, controlled adaptation comparisons evaluate learning capacity, and recovery/intervention analyses examine jointly learned contributions. Conclusions about their development during training will be tied to the analyzed AffectGPT trajectory.
Response to Reviewer Tsqu (Part 3/3)
R3-W3. GET versus Equally Small Adaptation Strategies
(1) Cross-Model Single-Projection Comparisons.
Setup. We compare gate-only, up-only, and down-only tuning across all three affective models used to evaluate GET, with mathematical reasoning as a non-affective control. Within each model, configurations start from the same corresponding base checkpoint and use identical training data, matched trainable-parameter counts, and matched training budgets.
Results. Gate-only achieves the highest mean on the three affective models, exceeding the strongest alternative by 0.97, 0.09, and 0.17 points, respectively (Table R3.6). The small latter margins align with the comparable standalone FFN capacity reported in §4.1 and Appendix A.6.2.
Table R3.6. Single-projection tuning under matched budgets. Results are means of two independent runs, following each model’s evaluation and dataset-aggregation protocols. Bold indicates the highest mean per row.
| Task | Model | Trainable params. per configuration | Gate-only | Up-only | Down-only |
|---|---|---|---|---|---|
| Affective | AffectGPT | 10.09M | 74.40 | 73.43 | 73.33 |
| Affective | Emotion-LLaMA | 30.93M | 87.66 | 86.85 | 87.57 |
| Affective | EAI-LLM | 30.93M | 70.01 | 69.84 | 69.53 |
| Mathematical reasoning | LLaMA (Gaussian) | 30.93M | 19.26 | 19.21 | 17.56 |
(2) Budget-Matched Allocation Strategies.
Setup. We compare GET with lower-rank FFN-LoRA across all three FFN projections, lower-rank ALL-LoRA across the major Attention and FFN projections, and AdaLoRA with adaptive allocation. Methods use the same corresponding base checkpoint, training data, and training budget. We adjust fixed ranks or AdaLoRA’s target budget to approximately match GET’s final effective adaptation parameter count.
Results. GET achieves the highest mean on the three affective models, exceeding the strongest alternative by 0.55, 1.61, and 0.07 points, respectively (Table R3.7). AdaLoRA performs best on mathematical reasoning. Thus, broader or adaptive allocation does not consistently improve affective performance at these budgets, while relative performance varies across settings.
Table R3.7. Allocation strategies under matched final effective parameter budgets. Results are means of two independent runs. AdaLoRA’s count reflects its final retained ranks, with a dynamic budget during training. ALL-LoRA is the budget-matched configuration, distinct from the original larger-budget All-Proj reference. Bold indicates the highest mean per row.
| Task | Model | Final effective params. (approx.) | GET | FFN-LoRA | ALL-LoRA | AdaLoRA |
|---|---|---|---|---|---|---|
| Affective | AffectGPT | 10.09M | 74.40 | 73.70 | 73.85 | 73.36 |
| Affective | Emotion-LLaMA | 30.93M | 87.66 | 86.02 | 86.05 | 85.84 |
| Affective | EAI-LLM | 30.93M | 70.01 | 69.89 | 69.94 | 59.65 |
| Mathematical reasoning | LLaMA (Gaussian) | 30.93M | 19.26 | 19.14 | 18.91 | 25.81 |
(3) Uncertainty and GET’s Interpretation.
The comparisons report two-run means. This provides limited information about variability, particularly for small margins, so we interpret the results as competitive performance rather than statistically established superiority.
The comparisons identify multiple viable FFN targets. GET uses joint-adaptation analyses to motivate a fixed target, then evaluates that choice through standalone tuning and budget-matched comparisons. We therefore position it as a simple, competitive strategy informed by functional analysis, rather than a universally optimal selection rule.
Follow-up on Our Response
Dear Reviewer,
Thank you again for your detailed review. Following your suggestions, we added:
- Perturbation controls: randomization at matched relative strength, and interventions matched for local FFN output changes.
- Task controls: same-backbone affective/non-affective adaptation under matched training conditions, and cross-capability evaluation under identical interventions.
- GET comparisons: cross-model gate/up/down tuning and budget-matched fixed/adaptive LoRA baselines.
We also clarified the coverage of the parameter survey and functional analyses. We would appreciate any remaining questions and would be happy to clarify the added experiments.
Response to Reviewer Tsqu (Part 1/3)
Thank you for your constructive review. We appreciate your recognition of the distinction between independent learning capacity and jointly optimized functional roles, and the particularly noteworthy intervention results. Following your suggestions, we added perturbation controls, affective/non-affective comparisons, and budget-matched GET baselines, while clarifying the scope of each analysis.
(1) Norm- and Rank-Preserving Randomization.
Setup. We randomize the learned LoRA factors A and B while keeping pretrained weights unchanged. We sample and orthogonalize random components, then adjust their mixture with the original factors to preserve the Frobenius norms of A, B, and the effective update ΔW=sBA, while retaining the effective update’s rank. Different projections use independent random seeds but the same relative update perturbation strength: ρ=∥ΔW∥F∥ΔW′−ΔW∥F. Each projection retains its own update norm and rank; matching concerns relative change, not identical absolute perturbation norms. Only the target projection’s update is replaced.
Results. Gate randomization produces the largest degradation in both models, with drops of 8.37 points on AffectGPT and 2.88 points on OneEmo (Table R3.1).
Table R3.1. Norm- and rank-preserving randomization at matched relative update perturbation strength. Parentheses show changes from each unperturbed All-Proj model.
(2) Deterministic Interventions at Matched Relative Weight Change.
Setup. As a deterministic complement, we reverse and rescale each learned LoRA update, selecting coefficients separately for each projection to match:
η=∥Wp∗∥F∥Wp′−Wp∗∥F.
where Wp∗ and Wp′ are the fully adapted and intervened weights. All other weights remain unchanged. Unlike the randomization above, this control normalizes against the fully adapted weight, rather than the learned update.
Results. Gate intervention again produces the largest degradation in both models: 26.22 points on AffectGPT and 5.88 points on OneEmo (Table R3.2). The ranking therefore persists across randomized and deterministic interventions using different normalization references.
Table R3.2. Deterministic reverse-direction interventions at matched relative weight-change magnitude. Parentheses show changes from each unperturbed All-Proj model.
(3) Local Interventions Matched for FFN Output Changes.
Setup. Weight-space matching does not guarantee comparable activation changes. We therefore separately attenuate gate_proj and up_proj updates at a selected layer:
\qquad 0\leq\lambda_p\leq1.$$ **$\lambda_p=0$ retains the full update; $\lambda_p=1$ removes it.** Pretrained weights and all other updates remain unchanged. Using **128 fixed unlabeled calibration samples**, we separately set gate and up coefficients to approximately match the FFN output-change magnitude **after down_proj and before residual addition**, without tuning to test performance. We test layers **4, 14, and 23** of AffectGPT’s **28-layer** language backbone and **4, 14, and 24** of OneEmo. **Results.** Gate attenuation produces larger MER23 drops than up attenuation at **all three tested layers in both models** (Table R3.3). The gate–up difference remains observable under this local output-matching procedure. **Table R3.3. MER23 performance under calibration-matched FFN output perturbations.** The unperturbed baseline scores on MER23 are **78.54 for AffectGPT** and **76.03 for OneEmo**. Values in parentheses indicate performance drops relative to the corresponding baseline. | Model | Layer | Gate attenuation | Up attenuation | |:-:|:-:|:-:|:-:| | AffectGPT | 4 | 75.18 (↓3.36) | 76.46 (↓2.08) | | AffectGPT | 14 | 75.70 (↓2.84) | 76.14 (↓2.40) | | AffectGPT | 23 | 75.83 (↓2.71) | 76.84 (↓1.70) | | OneEmo | 4 | 75.29 (↓0.74) | 75.41 (↓0.62) | | OneEmo | 14 | 75.09 (↓0.94) | 75.98 (↓0.05) | | OneEmo | 24 | 74.90 (↓1.13) | 75.32 (↓0.71) |