acceptodds
Under review as a conference paper at ICLR 2027

Transformers Align in Subspaces, Not Units: Rotation-Aware Model Merging

Abstract

Model merging combines trained networks in weight space, and every alignment method implicitly picks a symmetry group for comparing them. We show that group is mis-specified for Transformers. Transport-based alignment searches the Birkhoff polytope ; self-attention is also invariant under continuous rotations of its query-key and value-output subspaces, and is exactly the permutations, so no transport plan - however soft - expresses a non-trivial rotation. The LayerNorm residual stream, in turn, admits among isometries exactly the rotations fixing the mean axis, where a soft plan is inexact, not merely suboptimal. Rotation-aware Optimal Transport (RaOT) aligns in that quotient. Building on interface-wise rotation fusion (Zhang et al., 2025), it differs in four ways: activation-estimated alignments from one forward pass; head correspondence and within-head rotation solved jointly as one exact assignment; a mean-preserving Procrustes map on the residual stream; and identity competing with every estimate, which supplies a provenance switch and a test of whether alignment should act at all. This is a mechanism paper, not a leaderboard entry. No method - ours included - merges independently trained ImageNet ViTs to usable accuracy, so we read the barrier through matched contrasts: across nine public pairs and a four-width series inside one model family, what the isometries' estimator adds beyond the permutation's is larger at larger width. The polytope's own share shrinks only on the original three-pair ladder - across pairs it straddles zero, and a pre-registered same-recipe study reverses it - so we claim the isometries half alone, on recipe- or corpus-different parents, and not a width law. Alignment buys accuracy in frozen-feature transfer and, for parents differing in task and initialization, in trunk-head compatibility rather than in the backbone representation. On fine-tuned siblings of one initialization the gate rejects every transform: the role there is detection, not repair.

Primary area:
optimization
Keywords:
model merging, weight-space alignment, parameter symmetries, optimal transport, Procrustes, Transformers, vision transformers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UEboOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Rotation-aware Optimal Transport (RaOT), a training-free framework for aligning Transformer models before parameter fusion. Building on prior rotation-based transformer fusion, RaOT compares the models’ activations on the same unlabeled inputs to estimate their alignment. The author jointly determines how to match attention heads and rotate their internal representations to each token’s mean across feature dimensions, making it compatible with LayerNorm. The theoretical analysis characterizes the relevant symmetries and establishes a non-increase in aligned weight distance for RaOT-safe. Experiments on independently trained vision Transformers show reductions in interpolation loss barriers and improvements in frozen-feature transfer. In the multitask setting with different initializations, the observed gains primarily reflect improved compatibility between backbones and retained task heads.

Critical strengths
  1. The paper presents a straightforward motivation for rotation-aware alignment. The storyline from Transformer symmetries to architecture-compatible alignment rules and identity-based gating is intuitive and provides a clear rationale for the main design choices.
  2. The evaluation includes comprehensive scenarios and analysis.
Critical weaknesses
  1. Multi-model alignment depends on the choice of anchor. In Appendix C.1, the other models are individually aligned to a designated parent model. When models cannot be aligned perfectly, changing the anchor selection may affect both the estimated alignments and the merged model performance. The existing multi-model experiments do not establish how substantial such dependence is.
  2. Limited improvement of gating. The experiments support the gate as a safeguard, but its ability to select beneficial transformations within a regime remains unclear. On independently trained models, the weight-distance gate produces higher barriers than ungated alignment at all three scales. Meanwhile, the held-out activation gate accepts all candidates when the fitting budget is matched.
Questions for authors

Table 4 shows that weight-based residual alignment combined with activation-based head outperforms the fully activation-based method. Does this advantage persist across other model pairs? Meanwhile, under what kind of conditions would the authors recommend the hybrid estimator?

Reviewer_DbQTOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper investigates symmetry-aware alignment for merging independently trained Transformers. The authors argue that permutation-based optimal transport cannot capture the continuous rotational symmetries of attention layers. They propose Rotation-aware Optimal Transport (RaOT), which combines activation-based alignment, joint head matching and rotation estimation, LayerNorm-compatible residual-stream rotations, and a gating mechanism to determine whether alignment should be applied. Experiments investigate alignment geometry, loss barriers, and downstream performance across different Transformer architectures and initialization regimes.

Critical strengths

The paper addresses an interesting problem in model merging, highlighting the limitations of permutation-based alignment for Transformers.

The characterization of LayerNorm-compatible rotations and the joint optimization of head correspondence and rotations are technically interesting.

The experimental analysis is extensive, including controlled experiments, ablations, and comparisons with existing approaches. The authors also acknowledge important limitations of their method.

Critical weaknesses
  • The paper is extremely difficult to follow. The writing is unnecessarily dense, with excessive technical jargon, lengthy explanations, and repeated qualifications. The abstract and introduction fail to communicate the main contributions clearly, while the numerous methodological variants further complicate the presentation.
  • The novelty relative to existing rotation-based alignment methods is somewhat limited. Several theoretical observations and methodological components build directly on previous work, particularly Zhang et al. (2025). The paper does not sufficiently emphasize which contributions represent substantial advances over this baseline.
  • The practical significance of the experimental improvements is questionable. Despite improvements in alignment objectives and loss barriers, independently trained ImageNet ViTs remain far from usable accuracy after merging. Moreover, the improvements in the multi-task experiments appear primarily attributable to head compatibility rather than successful backbone merging.
  • The theoretical guarantees concern weight-space distances and upper bounds on loss barriers, but do not establish improvements in actual merging performance. Indeed, the proposed gating mechanism can lead to worse results than ungated alignment.
Questions for authors

What is the most significant contribution beyond Zhang et al. (2025)? Could the authors clarify which proposed components provide the largest improvements and why they are necessary?

Why does weight-based estimation of the residual-stream rotation outperform activation-based estimation? This seems particularly relevant given the emphasis on activation-driven alignment.

Can the authors provide stronger evidence that the proposed method improves actual backbone merging? The current results show improvements in alignment metrics and head compatibility, but limited success in obtaining functional merged models.

Could the authors substantially simplify the presentation? A clearer separation between the central method, its theoretical motivation, and the numerous experimental variants would considerably improve readability and make the contributions easier to assess.

Reviewer_6qCHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper studies symmetry-aware alignment for merging independently trained Transformers. RaOT combines activation-based estimation, joint head assignment and intra-head rotations, LayerNorm-compatible residual-stream alignment, and an identity-based weight-distance gate.

  • Its main technical components are the characterization of LayerNorm-compatible transformations, the exact solution of the coupled activation-alignment objective through Procrustes optimization and linear assignment, and the conditional connection between weight distance and a smoothness-based loss-barrier bound.

  • The experiments investigate alignment geometry, interpolation barriers, frozen-feature transfer, retained-head compatibility, and the shared-initialization regime in which alignment should often be rejected.

Critical strengths
  • Sound mathematical formulation. The characterization of LayerNorm-compatible transformations (Proposition 2) and the exact joint head-matching solver (Eqs. (7)–(9)) are technically well motivated. In particular, the latter accounts for the shared Q/K rotation constraint and avoids alternating optimization.

  • Careful experimental analysis. The paper includes meaningful permutation-versus-rotation comparisons, component ablations, endpoint-equivalence checks, and comparisons with released implementations (Tables 1, 3–5; Appendices E.9 and E.14).

Critical weaknesses
  • While RaOT introduces several extensions to rotation-based alignment, closely related approaches have already explored Transformer rotations and residual-stream alignment [1, 2]. The paper does not sufficiently establish which of its additional components constitutes a substantial advance over these methods. This is a central concern for my assessment, as the significance of the contribution depends on whether RaOT offers a meaningful methodological improvement beyond existing alignment techniques.

  • The theoretical discussion in Section 3.6 does not fully justify how the centered activation objective controls functional deviation or interpolation quality. Empirically, independently merged models retain very low midpoint accuracy (Table 1), while the retained-head experiments show almost no additional benefit from merging the backbone over transporting task heads alone (Table 2). Although the alignment improvements are meaningful, these limitations make it difficult to determine how much the proposed method advances model merging itself, which substantially affects my assessment of its practical significance.

I open to increasing my score when all my concerns are addressed.

References

[1] Zhang, B., Zheng, Z., Chen, Z., & Li, J. (2025). Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion. Proceedings of the 42nd International Conference on Machine Learning (ICML), PMLR 267, 77090–77106.

[2] Theus, A., Cabodi, A., Anagnostidis, S., Orvieto, A., Singh, S. P., & Boeva, V. (2025). Generalized Linear Mode Connectivity for Transformers. Advances in Neural Information Processing Systems (NeurIPS 2025).

[3] Carlini, N., et al. (2024). Stealing Part of a Production Language Model. Proceedings of the 41st International Conference on Machine Learning (ICML), PMLR 235, 5680–5705. See Appendix C.5–C.6 in the extended version.

Questions for authors
  • Zhang et al. [1] already established rotation-based alignment for Transformer fusion, while Theus et al. [2] explored global residual-stream alignment and generalized linear mode connectivity. Related LayerNorm-compatible orthogonal symmetries were also discussed by Carlini et al. [3, Appendix C.5–C.6]. Although RaOT differs in its activation-based estimation, coupled head matching, and treatment of LayerNorm, it is not entirely clear which component constitutes the main advance. How does RaOT differ from these approaches in terms of its transformation space, optimization objective, and interpolation procedure, and which of these differences are essential to its empirical improvements?

  • Section 3.6 (p. 7, lines 367–370) suggests that layerwise Lipschitz continuity connects the activation discrepancy in Eq. (7) to functional deviation. However, the objective only matches centered Q/K/V activations and does not directly account for output readers, activation means, or downstream interactions. Under what assumptions does this objective control functional discrepancy or interpolation quality? A more precise explanation would help distinguish the motivation for the alignment objective from the weight-distance guarantee in Theorem 1.

  • Section 3.3 (Eq. (6), p. 5) acknowledges that the Procrustes solution is non-unique when the cross-covariance matrix is rank-deficient, with the implementation using the completion returned by SVD. How sensitive are the resulting rotations and merged models to this ambiguity, particularly when some singular values are close to zero? It would be useful to understand whether different equally optimal rotations can lead to different loss barriers or midpoint accuracies, and whether the identity-based gate addresses this issue.

  • Table 20 (p. 32) presents held-out rotation-alignment measurements, but does not include the corresponding permutation baseline under the same evaluation protocol. Does the advantage of rotation over permutation persist when both transformations are estimated using identical calibration samples and evaluated on the same held-out inputs? Such a comparison would help establish whether the improvement reflects a more generalizable correspondence rather than the additional flexibility of orthogonal alignment.

  • The results in Table 1 (p. 8) show that the midpoint accuracy of independently merged ViT-B models remains low (3.81% for RaOT-align and 7.99% for the hybrid variant), compared with 79.1% and 81.9% for the parent models. Moreover, Table 2 (p. 9) reports nearly identical performance between merging the backbone (23.8 ± 1.8%) and retaining the anchor backbone while transporting task heads (23.7 ± 1.9%). Given these results, what is the main demonstrated advantage of RaOT as a model-merging method beyond improving representation alignment and head compatibility? A clearer interpretation of this distinction would help assess the practical contribution.

References

[1] Zhang, B., Zheng, Z., Chen, Z., & Li, J. (2025). Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion. Proceedings of the 42nd International Conference on Machine Learning (ICML), PMLR 267, 77090–77106.

[2] Theus, A., Cabodi, A., Anagnostidis, S., Orvieto, A., Singh, S. P., & Boeva, V. (2025). Generalized Linear Mode Connectivity for Transformers. Advances in Neural Information Processing Systems (NeurIPS 2025).

[3] Carlini, N., et al. (2024). Stealing Part of a Production Language Model. Proceedings of the 41st International Conference on Machine Learning (ICML), PMLR 235, 5680–5705. See Appendix C.5–C.6 in the extended version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.