- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Rotation-aware Optimal Transport (RaOT), a training-free framework for aligning Transformer models before parameter fusion. Building on prior rotation-based transformer fusion, RaOT compares the models’ activations on the same unlabeled inputs to estimate their alignment. The author jointly determines how to match attention heads and rotate their internal representations to each token’s mean across feature dimensions, making it compatible with LayerNorm. The theoretical analysis characterizes the relevant symmetries and establishes a non-increase in aligned weight distance for RaOT-safe. Experiments on independently trained vision Transformers show reductions in interpolation loss barriers and improvements in frozen-feature transfer. In the multitask setting with different initializations, the observed gains primarily reflect improved compatibility between backbones and retained task heads.
Critical strengths
- The paper presents a straightforward motivation for rotation-aware alignment. The storyline from Transformer symmetries to architecture-compatible alignment rules and identity-based gating is intuitive and provides a clear rationale for the main design choices.
- The evaluation includes comprehensive scenarios and analysis.
Critical weaknesses
- Multi-model alignment depends on the choice of anchor. In Appendix C.1, the other models are individually aligned to a designated parent model. When models cannot be aligned perfectly, changing the anchor selection may affect both the estimated alignments and the merged model performance. The existing multi-model experiments do not establish how substantial such dependence is.
- Limited improvement of gating. The experiments support the gate as a safeguard, but its ability to select beneficial transformations within a regime remains unclear. On independently trained models, the weight-distance gate produces higher barriers than ungated alignment at all three scales. Meanwhile, the held-out activation gate accepts all candidates when the fitting budget is matched.
Questions for authors
Table 4 shows that weight-based residual alignment combined with activation-based head outperforms the fully activation-based method. Does this advantage persist across other model pairs? Meanwhile, under what kind of conditions would the authors recommend the hybrid estimator?