- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MOMA (Masked Orthogonal Matrix Alignment), a post-merging framework that addresses the misalignment between a merged encoder and task-specific classifier heads. Motivated by empirical analyses suggesting that discrepancies between merged and fine-tuned representations can be approximated by orthogonal transformations, the method jointly learns task-specific alignment matrices and a shared mask over the aggregated task vector. The optimization minimizes representation-level discrepancies with the task-specific fine-tuned models. After training, the mask is incorporated into the encoder weights, and the alignment matrices are absorbed into the classifier heads, introducing no additional parameters or computation at inference time. The authors evaluate MOMA on image and text classification benchmarks and report improved average accuracy over existing post-merging approaches across several merging baselines. The experiments are complemented by analyses of transformation constraints, component contributions, and sensitivity to calibration data availability.
Critical strengths
- The paper offers a novel perspective by focusing on the misalignment between merged representations and task-specific classifier heads.
- The empirical results provide convincing evidence of the proposed method’s effectiveness.
Critical weaknesses
-
The motivation for preserving representation geometry through a strongly constrained transformation is insufficiently justified. The authors compare transformation families with varying degrees of freedom and favor orthogonal transformations because they achieve performance comparable to more flexible alternatives while preserving representation geometry. However, the paper does not adequately explain why geometry preservation should be an objective in addition to predictive performance, or what concrete benefit this preservation provides. Without establishing the importance of this additional objective, the characterization of orthogonal alignment as the best trade-off between performance recovery and geometric preservation is not fully convincing.
-
The rationale for introducing mask learning is not sufficiently consistent with the earlier motivation for orthogonal alignment. The authors introduce mask learning alongside orthogonal alignment to increase the degrees of freedom in the joint optimization. However, Sections 3.3 and 3.4 argue that a rigid orthogonal transformation is sufficient to correct the representation misalignment while preserving its geometry. This creates an apparent tension: if orthogonal alignment is already sufficient, why is it necessary to modify the encoder itself through a learned mask? Conversely, if mask learning is needed to address discrepancies that a rigid transformation cannot correct, this should be reconciled with the earlier claims. The paper needs a more coherent explanation of why mask learning is necessary and how its role relates to the proposed geometric interpretation.
-
The advantages of MOMA over Linear + Masking are unclear. Section 5.4 shows that Linear + Masking outperforms MOMA, but the authors describe it as a “relaxed empirical upper bound” rather than treating it as a competing method. Why not adopt this variant instead? The two approaches appear likely to have comparable computational costs, although this requires empirical verification, leaving the practical advantage of choosing MOMA unclear. The results are also consistent with an alternative interpretation: greater optimization freedom and fitting capacity, rather than preservation of representation geometry, may be responsible for the stronger performance. Moreover, the regularization analysis in Appendix D.1, Table 8, reports better performance as the orthogonality regularization is weakened. This observation does not substantiate the benefit of enforcing orthogonality and instead reinforces the need to explain why the orthogonal constraint is preferable.
-
The claim about prioritizing logit consistency does not clearly reflect the actual optimization objective. Around line 064, the paper states that “MOMA prioritizes output logit consistency by rectifying the inherent orthogonal misalignment.” However, the proposed objective minimizes discrepancies between hidden representations , rather than directly optimizing consistency between output logits. The authors should clarify the relationship between the stated goal and the implemented objective, and revise the wording accordingly.
-
The empirical validation is largely restricted to classification tasks. Validation on natural language generation tasks remains insufficient. Appendix D.3 compares different transformation families and shows that orthogonal alignment achieves representation-level Euclidean distances comparable to those obtained with affine and linear transformations. However, this does not establish the effectiveness of the full MOMA framework for generation. As the authors acknowledge, it remains unclear how directly their conclusions extend to settings with a shared language-model head rather than separate task-specific classifier heads. Furthermore, the parameter-absorption argument is presented for linear heads; its applicability to arbitrary nonlinear heads is not demonstrated. The scope of the claims should therefore be clarified, and stronger validation is needed to support applicability beyond classification.
Questions for authors
-
Robustness under practical failure modes. Prior work [1] discusses failure modes in more practical model-merging settings. How robust is MOMA under such conditions, particularly when the norms of individual task vectors differ substantially or prediction confidence is low?
-
Robustness to surrogate calibration data. In practical applications, inputs from each target task may not be available. How robust is MOMA when its mask and alignment transformations are optimized using surrogate data rather than task-specific inputs? How much of the reported performance is retained under this data mismatch?
[1] Yoshida, Kotaro, et al. "Distac: Conditioning task vectors via distillation for robust model merging.", International Conference on Learning Representations, 2026.
5 comments
Thank you for your response. However, critical concerns still remain.
First, while the authors now frame the primary goal of the paper as investigating how much of the mismatch can be explained by a transformation that preserves distances and angles, why is exploring this meaningful in the first place? The underlying motivation remains unclear. Furthermore, despite this reframing, the current manuscript still positions MOMA as a practical post-merging framework designed to achieve SOTA performance with zero additional parameters, creating a clear contradiction with the authors' rebuttal.
Second, regarding why mask learning was additionally introduced, the authors only respond to my questions about causality and logical consistency with empirical ablation numbers. Rather, the fact that masking alone yields a +14.4 percentage point improvement, substantially outperforming orthogonal alignment alone (76.4%), undermines the paper's core hypothesis that performance degradation after merging is primarily caused by an orthogonal rotational misalignment. Instead, these results indicate that modifying the feature space itself via encoder mask learning is the dominant factor.
Thank you very much for your careful response and valuable comments. Our reply is as follows.
First, while the authors now frame the
We believe that studying what the mismatch between the merged embedding and the classifier head actually is, and where its limits lie, is meaningful. In particular, when this mismatch is an angle- and norm-preserving orthogonal transformation, we consider this finding itself meaningful. Therefore, we believe the novelty of the orthogonal-transformation finding is greater than that of the SOTA results.
We fully understand your concern about the SOTA results. At the same time, the current best result, MOMA+Linear, is also proposed in our paper and adds no extra parameters. It can be understood as a variant of MOMA. In the next revision, we will revise the wording regarding SOTA and add MOMA-L (the linear version of MOMA) to the main experimental results.
Second, regarding why mask
-
To strictly guarantee that the transformation is orthogonal, we constrained the orthogonal transformation using
torch.nn.utils.parametrizations.orthogonal. This hard constraint is very strict and difficult to converge, resulting in an accuracy of only 76.4. If we instead use an orthogonality regularizer as a soft constraint, the accuracy reaches about 83%, comparable to using masking alone. Given that the orthogonal transformation has far fewer parameters than the mask matrix while achieving comparable performance, we consider the orthogonal transformation to have better parameter efficiency. -
We are not claiming that adjusting the feature space is unimportant. In fact, if it can be shown that an orthogonal mismatch between the merged embedding and the classifier head causes the performance degradation, then adjusting the merged embedding, training a transformation, or training the classifier head can all alleviate this orthogonal mismatch. If masking were the sole dominant factor, adding orthogonal alignment after masking should bring no notable improvement. In practice, under the soft constraint, the gain of orthogonality alone (83%) is already comparable to that of masking alone (83.5%), and adding orthogonal alignment on top of masking still improves accuracy by 4.7% (hard constraint, 83.5→88.2) or 5.9% (soft constraint, 83.5→89.4). This indicates that the orthogonal transformation is independently effective.
Thank you again for your suggestions. In the revised version, we will clarify the paper’s contributions in light of our discussion during the rebuttal period, and make the corresponding revisions to the manuscript accordingly.
Thank you for your valuable comments. Our responses are as follows.
Our main contribution is to show that merged representations retain substantial discriminative information and that a restricted, geometry-preserving transformation can correct their mismatch with the classifiers. We agree that a general linear transformation can also be absorbed into a linear classifier head and yields higher accuracy in our experiments. We chose orthogonal alignment to investigate how much of the mismatch can be explained by a simple transformation that preserves distances and angles. Achieving SOTA accuracy was secondary to this goal. Further, orthogonal alignment achieves 88.2% CV accuracy, and relaxing the constraint to allow a general linear transformation raises accuracy to 89.9%. This comparison shows that substantial performance recovery is already possible under the stricter geometric constraint.
The two components are complementary: masking adjusts the shared encoder, while orthogonal alignment improves compatibility with task-specific heads. Adding orthogonal alignment to masking improves CV accuracy from 83.5% to 88.2%. This 4.7 percentage point improvement demonstrates a meaningful contribution beyond mask learning and shows that the gains cannot be attributed to masking alone.
Our point is that representation drift does not necessarily impair classification by itself. For example, KNN classification on merged representations can outperform some existing merging methods. In the setting studied here, performance degrades when this drift disrupts compatibility between the merged representations and the fixed classifier heads.
Feature-level alignment serves as an effective surrogate objective for logit-level alignment. We have also evaluated direct logit-level alignment using KL divergence, which achieves slightly better performance than our feature-level objective. However, since our main comparison methods, Surgery and ProbSurgery, both perform feature-level alignment, we retain this objective to keep the calibration signal consistent with these baselines.
Our end-to-end experiments cover classification with task-specific linear heads. Appendix D.3 examines representation alignment in generative models, while the effectiveness of the full framework with a shared language-model head and its applicability to nonlinear heads remain open questions. We hope to explore this in future work.
Thank you for highlighting the failure modes studied by DisTaC. We conducted additional experiments comparing Surgery and MOMA under the Norm Mismatch and Low Confidence settings. The results below show that MOMA is effective in both settings.
Thank you for raising this question. We conducted additional experiments using out-of-distribution calibration data. In the eight-task merging setting, for example, we held out one task at a time for testing, used data from the other seven tasks for calibration, and averaged the results across the held-out tasks.
We observed a substantial drop in accuracy on the CV tasks, from 88.2% to 67.2%, and a smaller drop on the NLP tasks, from 76.3% to 74.4%. We believe one possible reason is that the visual features differ considerably across the CV datasets. For example, some datasets contain images of cars, while others contain images of digits, making transfer across tasks difficult. Although the NLP tasks have different label sets, they share vocabulary and grammatical structures, which may facilitate transfer. We will include this observation in the revised manuscript.