acceptodds
Under review as a conference paper at ICLR 2027

Understanding Affective Adaptation in Multimodal Foundation Models: Emergent Functional Specialization

Abstract

Despite rapid progress in multimodal affective foundation models, how affective capabilities emerge within their internal architectures remains poorly understood. A critical open question is whether affective fine-tuning induces diffuse changes across the model or organizes computation into functionally specialized pathways. We systematically investigate this question through a broad module-level analysis of 13 affective model instances spanning nine model designs, multiple scales, tasks, and training paradigms, complemented by controlled functional analyses on representative models. We find that affective adaptation exhibits a consistent yet non-exclusive functional organization. Under matched trainable-parameter budgets, adapting only the feed-forward network (FFN) consistently outperforms adapting only the attention modules across all evaluated settings and, on average, nearly matches the performance obtained by tuning all major Transformer projections, identifying the FFN as a particularly efficient adaptation substrate. More strikingly, although the gate, up, and down projections exhibit comparable standalone adaptation capacity, their learned functional contributions become differentiated after joint optimization. Module recovery and targeted interventions identify gate_proj as a particularly prominent pathway, while checkpoint analysis shows that this differentiation develops over the course of training. We characterize this phenomenon as emergent functional specialization: distinct pathway-level roles are not fully explained by standalone adaptation capacity, but arise through joint affective adaptation. Building on this finding, Gate-Focused Efficient Tuning (GET) retains 96.2-98.0% of the performance obtained by tuning all major Transformer projections while using only 19.3-24.5% as many trainable parameters.

Primary area:
interpretability and explainable AI
Keywords:
Affective Adaptation, Multimodal Foundation Models, Functional Specialization, Gated Feed-Forward Networks

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_TsquOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how affective fine-tuning changes the functional organization of multimodal foundation models. Through parameter analysis across 13 model instances and controlled experiments on representative models, the authors find that FFN-only adaptation outperforms attention-only adaptation under approximately matched trainable-parameter budgets. They further distinguish standalone adaptation capacity from contributions learned during joint optimization: although the gate, up, and down projections perform similarly when trained independently, recovery and intervention experiments identify the gate projection as a particularly influential component after joint adaptation. Checkpoint analyses track the development of this asymmetry during training. Motivated by these findings, the authors propose Gate-Focused Efficient Tuning (GET), which trains LoRA adapters only on gate projections and retains 96.2–98.0% of the All-Proj scores using 19.3–24.5% of its trainable parameters.

Critical strengths
  1. The paper separates what a projection can learn independently from the role its updates assume when multiple projections are optimized together. This distinction is more informative than simply identifying the largest parameter updates or the most damaging ablations.

  2. The study combines weight-change measurements, parameter-matched adaptation, selective recovery, removal/randomization/reversal interventions, compositional recovery, and checkpoint analysis. The intervention results are particularly noteworthy: removing the gate updates reduces AffectGPT performance by 10.64 points and OneEmo performance by 5.47 points, providing evidence that the observed asymmetry has functional consequences.

Critical weaknesses
  1. The interventions establish unequal dependence on learned updates, but do not sufficiently distinguish specialization from perturbation sensitivity.

Removing or reversing a projection’s updates changes both their learned information and the magnitude of the perturbation applied to the network. Figure 5 also shows that the gate projection accumulates larger updates in the analyzed trajectory. Consequently, its greater intervention sensitivity could partly reflect larger perturbations, its multiplicative position in the FFN, or stronger interactions with other adapted components.

The recovery experiments provide valuable complementary evidence, but recovery also depends on compatibility with the otherwise unadapted model. Similar standalone performance does not rule out architectural or optimization biases during joint training. Norm-matched and activation-matched perturbations, explicitly specified randomization procedures, and continuous intervention-strength sweeps would help distinguish these explanations. Evaluating general capabilities after the same interventions would further clarify whether the observed damage is disproportionately affective.

  1. The evidence for affect-related specificity and generality remains limited.

The non-affective controls differ in tasks, backbone configurations, evaluation metrics, and adaptation protocols. Their lower GET-to-All-Proj ratios therefore cannot isolate affective content as the reason gate-focused adaptation works well. The authors acknowledge this limitation, but a stronger comparison would hold the backbone, data scale, training procedure, and output format fixed while varying the task.

Moreover, the 13-model breadth primarily concerns parameter-change statistics. The targeted intervention analysis covers AffectGPT and OneEmo, while the main functional recovery trajectory is substantially narrower. The scope of the stronger functional conclusions should be distinguished from the broader descriptive survey.

  1. GET’s advantage over other equally small adaptation strategies is not established.

Table 9 already reports strong alternatives: gate-only adaptation achieves 73.68, while down-only adaptation achieves 73.43 under approximately matched parameter budgets. The 0.25-point difference is small and is not accompanied by uncertainty estimates. The paper’s own finding of comparable standalone FFN capacity therefore raises an important practical question: does selecting the gate based on post-training specialization improve adaptation beyond simply selecting an FFN projection?

Comparisons against lower-rank All-Proj LoRA, lower-rank FFN-only LoRA, and adaptive rank allocation under the same total trainable-parameter budget are missing. Gate-, up-, and down-only comparisons should also be extended beyond the principal AffectGPT experiment.

Questions for authors
  1. Does the gate projection remain the most influential pathway when interventions are matched by update norm or induced activation change? How exactly are randomized updates generated, and are their norms and ranks preserved?
  2. Can you construct affective and non-affective comparisons using the same backbone and matched training conditions? Do gate interventions selectively harm affective performance, or do they cause similarly large degradation on general capabilities?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Tsqu (Part 1/3)

Thank you for your constructive review. We appreciate your recognition of the distinction between independent learning capacity and jointly optimized functional roles, and the particularly noteworthy intervention results. Following your suggestions, we added perturbation controls, affective/non-affective comparisons, and budget-matched GET baselines, while clarifying the scope of each analysis.

 

R3-W1 & Q1. Perturbation Controls and Randomization Procedure

 

(1) Norm- and Rank-Preserving Randomization.

Setup. We randomize the learned LoRA factors AA and BB while keeping pretrained weights unchanged. We sample and orthogonalize random components, then adjust their mixture with the original factors to preserve the Frobenius norms of AA, BB, and the effective update ΔW=sBA\Delta W=sBA, while retaining the effective update’s rank. Different projections use independent random seeds but the same relative update perturbation strength: ρ=∥ΔW′−ΔW∥F∥ΔW∥F.\rho=\frac{\|\Delta W'-\Delta W\|_F}{\|\Delta W\|_F}. Each projection retains its own update norm and rank; matching concerns relative change, not identical absolute perturbation norms. Only the target projection’s update is replaced.

Results. Gate randomization produces the largest degradation in both models, with drops of 8.37 points on AffectGPT and 2.88 points on OneEmo (Table R3.1).

Table R3.1. Norm- and rank-preserving randomization at matched relative update perturbation strength. Parentheses show changes from each unperturbed All-Proj model.

Perturbed projectionAffectGPTOneEmo
Unperturbed All-Proj76.3071.80
q68.34 (↓7.96)72.41 (↑0.61)
k68.83 (↓7.47)71.55 (↓0.25)
v68.93 (↓7.37)71.61 (↓0.19)
o69.76 (↓6.54)72.22 (↑0.42)
gate67.93 (↓8.37)68.92 (↓2.88)
up68.79 (↓7.51)72.27 (↑0.47)
down69.24 (↓7.06)72.50 (↑0.70)

 

(2) Deterministic Interventions at Matched Relative Weight Change.

Setup. As a deterministic complement, we reverse and rescale each learned LoRA update, selecting coefficients separately for each projection to match:

η=∥Wp′−Wp∗∥F∥Wp∗∥F.\eta = \frac{\lVert W'_p-W_p^* \rVert_F}{\lVert W_p^* \rVert_F}.

where Wp∗W_p^* and Wp′W'_p are the fully adapted and intervened weights. All other weights remain unchanged. Unlike the randomization above, this control normalizes against the fully adapted weight, rather than the learned update.

Results. Gate intervention again produces the largest degradation in both models: 26.22 points on AffectGPT and 5.88 points on OneEmo (Table R3.2). The ranking therefore persists across randomized and deterministic interventions using different normalization references.

Table R3.2. Deterministic reverse-direction interventions at matched relative weight-change magnitude. Parentheses show changes from each unperturbed All-Proj model.

Perturbed projectionAffectGPTOneEmo
Unperturbed All-Proj76.3071.80
q56.12 (↓20.18)69.07 (↓2.73)
k63.64 (↓12.66)71.27 (↓0.53)
v68.36 (↓7.94)71.49 (↓0.31)
o54.78 (↓21.52)72.22 (↑0.42)
gate50.08 (↓26.22)65.92 (↓5.88)
up63.55 (↓12.75)70.26 (↓1.54)
down65.76 (↓10.54)70.13 (↓1.67)

 

(3) Local Interventions Matched for FFN Output Changes.

Setup. Weight-space matching does not guarantee comparable activation changes. We therefore separately attenuate gate_proj and up_proj updates at a selected layer:

\qquad 0\leq\lambda_p\leq1.$$ **$\lambda_p=0$ retains the full update; $\lambda_p=1$ removes it.** Pretrained weights and all other updates remain unchanged. Using **128 fixed unlabeled calibration samples**, we separately set gate and up coefficients to approximately match the FFN output-change magnitude **after down_proj and before residual addition**, without tuning to test performance. We test layers **4, 14, and 23** of AffectGPT’s **28-layer** language backbone and **4, 14, and 24** of OneEmo. **Results.** Gate attenuation produces larger MER23 drops than up attenuation at **all three tested layers in both models** (Table R3.3). The gate–up difference remains observable under this local output-matching procedure. **Table R3.3. MER23 performance under calibration-matched FFN output perturbations.** The unperturbed baseline scores on MER23 are **78.54 for AffectGPT** and **76.03 for OneEmo**. Values in parentheses indicate performance drops relative to the corresponding baseline. | Model | Layer | Gate attenuation | Up attenuation | |:-:|:-:|:-:|:-:| | AffectGPT | 4 | 75.18 (↓3.36) | 76.46 (↓2.08) | | AffectGPT | 14 | 75.70 (↓2.84) | 76.14 (↓2.40) | | AffectGPT | 23 | 75.83 (↓2.71) | 76.84 (↓1.70) | | OneEmo | 4 | 75.29 (↓0.74) | 75.41 (↓0.62) | | OneEmo | 14 | 75.09 (↓0.94) | 75.98 (↓0.05) | | OneEmo | 24 | 74.90 (↓1.13) | 75.32 (↓0.71) |
AuthorsOpenReview ↗

Response to Reviewer Tsqu (Part 2/3)

 

R3-W1 & Q2. General-Capability Effects of the Same Interventions

Setup. We evaluate affective and general performance on the same affectively adapted AffectGPT All-Proj checkpoint under the norm- and rank-preserving randomization described above. Each perturbed model is evaluated on both task groups without resampling the perturbation; all non-target weights remain unchanged.

Results. Gate randomization reduces affective performance from 76.30 to 67.93, while the general score increases from 22.50 to 22.85 (Table R3.4). Thus, the affective decline is not accompanied by a corresponding general-score decline in this evaluation. Other projections show a similar contrast, so this pattern is not exclusive to gate_proj. It characterizes different responses of the evaluated affective and general tasks to the learned updates.

Table R3.4. Affective and general performance under identical projection interventions. Parentheses show changes from each evaluation’s own baseline; score differences across the two evaluation scales are not treated as equivalent.

Perturbed projectionAffective evaluationGeneral evaluation
Unperturbed All-Proj76.3022.50
q68.34 (↓7.96)23.12(↑0.61)
k68.83 (↓7.47)23.20(↑0.70)
v68.93 (↓7.37)22.59(↑0.09)
o69.76 (↓6.54)23.12(↑0.62)
gate67.93 (↓8.37)22.85(↑0.35)
up68.79 (↓7.51)23.03(↑0.53)
down69.24 (↓7.06)22.85(↑0.35)

 

R3-W2 & Q2. Same-Backbone Affective/Non-Affective Comparison

Matched Training Conditions.

Setup. Using the same AffectGPT framework and pretrained base checkpoint, we match training data scale (maintain consistency with the data scale of MER25), training procedure, and output format while adapting on either MER25 affective data or VALOR-32K non-affective multimodal data. We compare All-Proj and GET in both settings. Affective evaluation follows the original eight-benchmark protocol; non-affective evaluation uses OmniBench accuracy. Both use zero-shot evaluation.

Results. GET retains 96.6% of All-Proj performance in the affective setting and 86.5% in the non-affective setting (Table R3.5). This comparison controls the backbone and training conditions that differed in the original public-model comparisons. Evaluation datasets and metrics remain different, and the OmniBench reference score is close to its 25% chance level, limiting interpretation of that retention ratio. We therefore report the contrast as a description of these two settings, rather than attributing it solely to affective content.

Table R3.5. Affective/non-affective adaptation with the same backbone and matched training conditions. Retention is GET’s score divided by the corresponding All-Proj score within each setting. The affective score averages eight benchmarks; the non-affective score is OmniBench accuracy.

Task settingTraining dataAll-ProjGETRetention
Affective understandingMER2576.3073.6896.6%
General multimodal understandingVALOR-32K25.4822.0586.5%

 

R3-W2. Coverage of the Survey and Functional Analyses

Our study combines a 13-instance parameter survey spanning 9 model designs, controlled Attention/FFN adaptation and GET comparisons on three affective models, and projection-level recovery analysis on AffectGPT and targeted intervention analyses on AffectGPT and OneEmo. During rebuttal, we added further intervention controls, alongside the task comparisons reported above. Within these functional analyses, AffectGPT’s checkpoint recovery (§4.4, Figure 5(b)) examines how contribution differences develop during training. Parameter-update dynamics are additionally analyzed on EAI-LLM (Appendix A.5), separately from functional recovery.

We will make these complementary scopes explicit throughout the manuscript: the broad survey characterizes parameter-change patterns, controlled adaptation comparisons evaluate learning capacity, and recovery/intervention analyses examine jointly learned contributions. Conclusions about their development during training will be tied to the analyzed AffectGPT trajectory.

AuthorsOpenReview ↗

Response to Reviewer Tsqu (Part 3/3)

 

R3-W3. GET versus Equally Small Adaptation Strategies

 

(1) Cross-Model Single-Projection Comparisons.

Setup. We compare gate-only, up-only, and down-only tuning across all three affective models used to evaluate GET, with mathematical reasoning as a non-affective control. Within each model, configurations start from the same corresponding base checkpoint and use identical training data, matched trainable-parameter counts, and matched training budgets.

Results. Gate-only achieves the highest mean on the three affective models, exceeding the strongest alternative by 0.97, 0.09, and 0.17 points, respectively (Table R3.6). The small latter margins align with the comparable standalone FFN capacity reported in §4.1 and Appendix A.6.2.

Table R3.6. Single-projection tuning under matched budgets. Results are means of two independent runs, following each model’s evaluation and dataset-aggregation protocols. Bold indicates the highest mean per row.

TaskModelTrainable params. per configurationGate-onlyUp-onlyDown-only
AffectiveAffectGPT10.09M74.4073.4373.33
AffectiveEmotion-LLaMA30.93M87.6686.8587.57
AffectiveEAI-LLM30.93M70.0169.8469.53
Mathematical reasoningLLaMA (Gaussian)30.93M19.2619.2117.56

 

(2) Budget-Matched Allocation Strategies.

Setup. We compare GET with lower-rank FFN-LoRA across all three FFN projections, lower-rank ALL-LoRA across the major Attention and FFN projections, and AdaLoRA with adaptive allocation. Methods use the same corresponding base checkpoint, training data, and training budget. We adjust fixed ranks or AdaLoRA’s target budget to approximately match GET’s final effective adaptation parameter count.

Results. GET achieves the highest mean on the three affective models, exceeding the strongest alternative by 0.55, 1.61, and 0.07 points, respectively (Table R3.7). AdaLoRA performs best on mathematical reasoning. Thus, broader or adaptive allocation does not consistently improve affective performance at these budgets, while relative performance varies across settings.

Table R3.7. Allocation strategies under matched final effective parameter budgets. Results are means of two independent runs. AdaLoRA’s count reflects its final retained ranks, with a dynamic budget during training. ALL-LoRA is the budget-matched configuration, distinct from the original larger-budget All-Proj reference. Bold indicates the highest mean per row.

TaskModelFinal effective params. (approx.)GETFFN-LoRAALL-LoRAAdaLoRA
AffectiveAffectGPT10.09M74.4073.7073.8573.36
AffectiveEmotion-LLaMA30.93M87.6686.0286.0585.84
AffectiveEAI-LLM30.93M70.0169.8969.9459.65
Mathematical reasoningLLaMA (Gaussian)30.93M19.2619.1418.9125.81

 

(3) Uncertainty and GET’s Interpretation.

The comparisons report two-run means. This provides limited information about variability, particularly for small margins, so we interpret the results as competitive performance rather than statistically established superiority.

The comparisons identify multiple viable FFN targets. GET uses joint-adaptation analyses to motivate a fixed target, then evaluates that choice through standalone tuning and budget-matched comparisons. We therefore position it as a simple, competitive strategy informed by functional analysis, rather than a universally optimal selection rule.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your detailed review. Following your suggestions, we added:

  • Perturbation controls: randomization at matched relative strength, and interventions matched for local FFN output changes.
  • Task controls: same-backbone affective/non-affective adaptation under matched training conditions, and cross-capability evaluation under identical interventions.
  • GET comparisons: cross-model gate/up/down tuning and budget-matched fixed/adaptive LoRA baselines.

We also clarified the coverage of the parameter survey and functional analyses. We would appreciate any remaining questions and would be happy to clarify the added experiments.

Reviewer_aNjtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how affective fine-tuning distributes functionality across Transformer projections. FFN-only adaptation outperforms attention-only adaptation under comparable parameter budgets. Although gate, up, and down projections perform similarly when trained independently, recovery, intervention, and checkpoint analyses indicate a stronger contribution from gate_proj after joint training. This motivates GET, which preserves most of the evaluated affective performance with substantially fewer trainable parameters.

Critical strengths
  1. The experiments are extensive and follow a clear logic. In particular, distinguishing standalone adaptation capacity from contribution after joint training is useful, and the recovery, intervention, and training-dynamics analyses provide complementary support for the observed differentiation.

  2. Similar adaptation patterns across models handling different modalities make the findings appealing beyond a single task or input setting. GET also gives the analysis a practical outcome by retaining most of the reference performance with a much smaller trainable-parameter budget.

Critical weaknesses
  1. The analysis establishes where adaptation has a stronger effect, but offers limited insight into why. Gate and up projections have the same dimensions and parallel inputs, with the activation function providing a key structural asymmetry. The paper does not explain how this asymmetry, pretrained representations, or optimization dynamics lead to the observed difference. As a result, GET feels more like a useful empirical tuning rule than a source of deeper understanding of affective adaptation.

  2. The scope is tied to backbones with gated FFNs. It remains unclear whether comparable affective capabilities can be learned by models with standard ungated FFNs, and how functional contributions would be distributed in that setting. This matters for both GET's applicability and the broader interpretation of gate-centered specialization. The current results support the role of gating in the examined architectures, but do not establish that gating is necessary for strong affective performance.

Questions for authors
  1. Beyond the modulation hypothesis in Section 6, is there evidence explaining why gate_proj contributes more than up_proj after joint training? An analysis of activations, gradients, or interventions matched for their effect on FFN outputs could help distinguish functional specialization from differences in sensitivity.

  2. How should the conclusions be scoped for architectures without gate_proj? As an extension, a controlled comparison with an ungated FFN backbone would help assess whether similar affective performance can be achieved and whether adaptation concentrates elsewhere.

3 comments
AuthorsOpenReview ↗

Response to Reviewer aNjt (Part 1/2)

Thank you for your constructive review. We appreciate your recognition of the extensive experiments and clear logic, complementary recovery, intervention, and training-dynamics analyses, and similar adaptation patterns across modalities and GET's practical value. Following your suggestions, we added FFN-output-matched interventions and an ungated-backbone analysis to further examine gate–up differences and architectural scope.

 

R2-W1 & Q1. Additional Analysis of Gate–Up Differences

 

(1) Interventions Matched for FFN Output Changes.

Setup. AffectGPT’s language backbone has 28 Transformer layers. Starting from the jointly adapted All-Proj model, we separately attenuate gate_proj and up_proj updates at layers 4, 14, and 23, covering different depths.

For the target projection p∈{g,u}p\in\{g,u\}, we partially withdraw its learned LoRA update:

=W_p^{0}+(1-\lambda_p)\Delta W_p, \qquad 0\leq\lambda_p\leq1.$$ Here, $W_p^{0}$ is the pretrained weight and $\Delta W_p$ is the learned update. **$\lambda_p=0$ retains the full update; $\lambda_p=1$ removes it entirely.** All pretrained weights and other adaptation updates remain unchanged. Using **128 fixed, unlabeled calibration samples**, we separately select $\lambda_g$ and $\lambda_u$ to approximately match the target layer’s FFN output-change magnitude, measured **after down_proj and before residual addition**. Coefficients are fixed during calibration without tuning against test performance. **Results.** **Gate attenuation produces a larger performance drop at all three tested layers**, exceeding the corresponding up drop by 0.44–1.28 points (Table R2.1). The gate–up difference therefore remains observable under this calibration-matched setting. **Table R2.1. MER23 performance after calibration-matched FFN output perturbations.** The unperturbed All-Proj score is **78.54**. Parentheses show drops from this baseline; the final column compares gate and up drops. | Layer | Gate attenuation: score (drop) | Up attenuation: score (drop) | Additional gate drop | |---|:-:|:-:|:-:| | 4 | 75.18 (↓3.36) | 76.46 (↓2.08) | 1.28 | | 14 | 75.70 (↓2.84) | 76.14 (↓2.40) | 0.44 | | 23 | 75.83 (↓2.71) | 76.84 (↓1.70) | 1.01 |   **(2) Interpretation and Remaining Mechanistic Questions.** The checkpoint recovery analysis (§4.4, Figure 5(b)) shows similar early recovery performance across FFN projections, followed by differentiation during joint adaptation. The added experiment examines this difference while controlling local output-change magnitude. Together, these analyses explore **when functional differences develop and how they respond to this additional control**. Matching magnitude leaves perturbation direction uncontrolled, so downstream directional sensitivity remains a possible contributor. The respective roles of activation asymmetry, pretrained representations, and optimization dynamics also remain open.
AuthorsOpenReview ↗

Response to Reviewer aNjt (Part 2/2)

 

R2-W2 & Q2. Ungated FFNs and GET’s Architectural Scope

 

(1) Module Sensitivity in an Ungated Backbone.

Setup. We added EmoOPT [1], based on OPT-13B [2], a decoder-only Transformer with standard ungated FFNs: fc1 expands features, ReLU applies the nonlinearity, and fc2 projects them back. It has no separate gate_proj or multiplicative gating branch. Starting from the full-parameter SFT checkpoint, we separately randomize Attention and FFN projections, matching relative weight-change magnitude across modules. Only the target projection is perturbed; all other weights remain unchanged.

Results. Perturbing fc1 produces the largest degradation, exceeding that from fc2 or any Attention projection (Table R2.2). In this ungated model, module sensitivity is therefore also nonuniform, with the strongest response at the FFN’s first projection.

Table R2.2. EmoOPT performance under randomization matched by relative weight-change magnitude. Scores are the arithmetic mean of E-c accuracy and SST5 macro-F1. Drops are measured against the same unperturbed full-parameter SFT checkpoint.

Perturbed projectionMean scoreDrop
Unperturbed full-parameter SFT model0.393—
q0.3590.034
k0.3700.023
v0.3460.047
o0.3660.027
fc10.3310.062
fc20.3710.022

 

(2) Architectural Scope.

GET’s gate-only rule applies to architectures containing gate_proj. Ungated architectures require their own module analyses rather than assuming that fc1 is an equivalent target. The EmoOPT experiment extends the module-sensitivity analysis to an ungated backbone. It does not establish that gate-related functionality transfers to fc1. This single-backbone extension also leaves comparable performance across architectures and gating’s necessity for strong affective performance unresolved. We will include this experiment and explicitly scope the gate-centered conclusions to the examined gated architectures.

References

[1] Liu et al. EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysis. KDD, 2024.

[2] Zhang et al. OPT: Open Pre-trained Transformer Language Models. arXiv:2205.01068, 2022.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your constructive review. Following your suggestions, we added FFN-output-matched gate/up interventions and an ungated OPT-backbone analysis, and clarified the mechanistic interpretation and architectural scope.

We would welcome any remaining questions on these additions and would be happy to provide further clarification.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7qGQ (Part 1/2)

Thank you for your constructive review. We appreciate your recognition of the complementary evidence from our controlled analyses and the combination of broad module-level coverage with a practical outcome. Following your suggestions, we added gate/up/down comparisons across all three GET evaluation models and matched-budget comparisons with fixed and adaptive LoRA allocation strategies.

 

R1-W1 & Q1. Rationale for Gate Selection

Our study investigates how affective adaptation is distributed across Transformer components (§§3–4). A key finding is that FFN projections have comparable standalone adaptation capacity but develop differentiated contributions after joint training, with gate_proj becoming particularly prominent. We added direct comparisons to evaluate its target selection.

 

(1) Cross-Model Comparisons under Matched Budgets.

Setup. We compare gate-only, up-only, and down-only tuning on all three GET evaluation models, with mathematical reasoning as a non-affective control. Within each model, configurations start from the same corresponding base checkpoint and use identical training data, matched trainable-parameter counts, and matched training budgets.

Results. Gate-only achieves the highest mean on all three affective models, although some margins are small and other FFN projections also retain substantial performance (Table R1.1). In mathematical reasoning, retention is lower across configurations, with gate-only close to up-only.

Table R1.1. Standalone projection tuning: mean performance retention (%). Results are means of two independent runs, following §5.2 and Appendix A.8. Retention equals the mean score divided by the corresponding original All-Proj reference score. Bold indicates the highest mean per row.

TaskModelDatasetsTrainable params. per config.All-Proj reference scoreGate-onlyUp-onlyDown-only
AffectiveAffectGPT810.09M76.3097.596.296.1
AffectiveEmotion-LLaMA130.93M88.8198.797.898.6
AffectiveEAI-LLM330.93M73.0195.995.795.2
Mathematical reasoningLLaMA (Gaussian)230.93M24.1879.779.472.6

 

(2) Implications for GET’s Selection Rationale.

Close standalone results indicate multiple viable adaptation targets, consistent with §4.1, Figure 4(a), and Appendix A.6.2, Table 9. Recovery and interventions (§§4.2–4.3, Figure 4(b), Table 2) instead examine unequal contributions after joint adaptation. Comparable standalone capacity can coexist with differentiated jointly learned roles. GET translates this functional analysis into a fixed-target strategy. Across two independent runs, Gate-only retains 95.9%–98.7% of All-Proj's affective performance while using only 19.3%–24.5% of its trainable parameters. The results support gate_proj as a competitive, functionally motivated choice, while the small margins limit claims of a consistent selection advantage.

AuthorsOpenReview ↗

Response to Reviewer 7qGQ (Part 2/2)

 

R1-W2 & Q2. Budget-Matched LoRA Allocation Comparisons

 

(1) Fixed and Adaptive Allocation Baselines.

Setup. We compare GET with FFN-LoRA across all three FFN projections, ALL-LoRA across the major Attention and FFN projections, and AdaLoRA with adaptive low-rank allocation. All methods use the same base checkpoint, training data, and matched training budgets. We adjust the fixed LoRA ranks or AdaLoRA's target budget to approximately match GET's final effective trainable parameter count. Notably, the FFN-LoRA rank configuration differs from that in manuscript Table 1: the Table 1 matches the trainable parameter counts of FFN and Attention, this experiment adjusts the rank to match GET's parameter count.

Results. GET achieves the highest mean on all three affective models, with close results on EAI-LLM (Table R1.2). AdaLoRA performs best on mathematical reasoning and exceeds the original All-Proj reference.

Table R1.2. Allocation strategies: mean performance retention (%) under matched final effective parameter budgets. Results are means of two independent runs. Retention uses the original All-Proj references in Table R1.1, not the budget-matched ALL-LoRA configurations below. AdaLoRA’s count reflects its final retained ranks; its training-time budget follows a dynamic schedule. Bold indicates the highest mean per row.

TaskModelFinal effective params. per method (approx.)GETFFN-LoRAALL-LoRAAdaLoRA
AffectiveAffectGPT10.09M97.596.696.896.1
AffectiveEmotion-LLaMA30.93M98.796.996.996.7
AffectiveEAI-LLM30.93M95.995.795.881.7
Mathematical reasoningLLaMA (Gaussian)30.93M79.779.278.2106.7

 

(2) Implications for the Allocation Rule.

These comparisons show that concentrating capacity on gate_proj is competitive with broader or adaptive allocation in the evaluated affective settings. Other low-budget strategies also retain substantial performance, so high retention is not unique to gate selection. The mathematical control illustrates variation across settings rather than isolating a causal effect of task type.

We therefore position GET as a simple, competitive strategy informed by functional analysis. With two runs and some small margins, the results establish practical competitiveness rather than statistically significant superiority. We will add these comparisons and report target modules, ranks, parameter-counting conventions, and training settings, including AdaLoRA’s initial/final budgets and allocation schedule.

AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

Thank you for your thoughtful and constructive feedback. Your suggestions have helped us strengthen the controlled analyses, adaptation comparisons, and explanation of the study’s scope.

We would like to clarify the broader motivation and contribution of this work. Strong affective performance alone does not explain how a pretrained model acquires that capability or how its components participate in it. Studying this organization can help the community interpret affective adaptation and make more informed adaptation choices.

 

(1) Making the acquisition of affective capabilities experimentally examinable.

Our study connects a survey of 13 affective model instances spanning 9 designs with controlled adaptation, recovery, intervention, and training-dynamics analyses. These analyses examine distinct questions: where parameters change, what individual components can learn, and how learned updates contribute within an adapted model. The added perturbation controls further examine how the observed responses depend on intervention magnitude.

 

(2) Distinguishing what a component can learn from the role it assumes during joint learning.

Different FFN projections achieve comparable standalone performance, yet their jointly learned contributions differentiate in the examined models. Learning capacity and realized contribution are therefore distinct properties. This distinction is useful beyond choosing gate_proj: it helps interpret module importance and motivates further investigation of how architecture, pretrained representations, and optimization shape functional organization.

 

(3) Connecting functional analysis with practical adaptation design.

GET tests whether this analysis can guide a concrete adaptation choice. Our added comparisons show competitive mean performance against alternative projection targets and fixed/adaptive LoRA allocations. Gate-only tuning retains 96.2-98.0% of All-Proj affective performance using 19.3%–24.5% of its trainable parameters. Its practical role is to test the usefulness of analysis-informed selection, with the comparisons also clarifying where alternative choices perform similarly.

We hope this work encourages studying affective adaptation through learning capacity, jointly learned roles, and their development during training, alongside task performance. The underlying mechanisms remain open, and the study provides concrete observations and experimental controls for investigating them.

We sincerely appreciate your time and consideration and welcome any further questions during the remaining discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.