- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes PrefGeo, a training-free method for combining preference-aligned flow-matching models. It represents each expert using left and right rotations relative to a shared base model and merges these rotations in Cayley coordinates. Experiments on SD3.5-M and FLUX.1-Dev evaluate compositional fidelity, text rendering, and aesthetic preference.
Critical strengths
-
The method addresses a useful problem. Combining existing preference experts without additional training could reduce the cost of multi-preference alignment. The proposed approach operates directly on model weights and avoids an additional alignment or distillation stage.
-
The reported results are promising. PrefGeo improves multiple metrics over the base models on both backbones. The expert-combination experiments suggest that it can retain capabilities from different experts. The observation that replacing a preference-aligned model's singular values with those of the base model largely preserves the evaluated task performance is also informative.
Critical weaknesses
-
The contribution beyond OrthoMerge is not sufficiently established. Eq. (11) is algebraically equivalent to the magnitude-corrected aggregation in OrthoMerge [1, Eqs. (7)–(8)]. The two-sided SVD representation is a clear difference, but its benefit over OrthoMerge's orthogonal-residual variant is not evaluated.
-
The weight-space baseline comparison needs clarification. Weight averaging in Table 3 achieves GenEval/OCR scores of 0.85/0.89, compared with 0.41/0.47 for Task Arithmetic in Table 1. The merging coefficients and their selection are not reported. PrefGeo still outperforms weight averaging, but it is unclear how much of the larger gap in Table 1 comes from the baseline configuration.
-
The geometric evidence needs a more careful interpretation. Under Eq. (6) as written, the near-unit singular-vector alignment in Figure 3(b) follows by construction and would also hold for an SFT model. It therefore does not by itself establish a preference-specific property. The spectral findings remain informative, but should be distinguished from this identity.
Questions for authors
-
How does PrefGeo compare with the orthogonal-residual variant of OrthoMerge using the same preference experts and evaluation settings?
-
Equal-weight averaging of the same three experts is algebraically equivalent to Task Arithmetic [2] with , provided the same parameters are merged. What scaling coefficient was used for Task Arithmetic in Table 1, what averaging weights were used for the W-space baseline in Table 3, and how were they selected? Could the authors explain the performance gap and clarify whether both baselines use the same parameter handling?
-
Are the rotations in Figure 3(b) computed exactly as stated in Eq. (6), or are additional constraints imposed?
References:
[1] Sihan Yang, Kexuan Shi, and Weiyang Liu. Orthogonal Model Merging. ICML 2026.
[2] Gabriel Ilharco et al. Editing Models with Task Arithmetic. ICLR 2023.
2 comments
Reply to Reviewer AM2s (part 2)
3. [Q3, W3]: Are the rotations in Figure 3(b) computed exactly as stated in Eq. (6), or are additional constraints imposed? (Interpretation of Geometric Evidence). The geometric evidence needs a more careful interpretation. Under Eq. (6) as written, the near-unit singular-vector alignment in Figure 3(b) follows by construction and would also hold for an SFT model. It therefore does not by itself establish a preference-specific property. The spectral findings remain informative, but should be distinguished from this identity.
Response: We thank the reviewer for the insightful observation. The rotations in Figure 3(b) are computed directly according to Eq. (6), without imposing additional constraints. We agree that, given and the orthonormality of the singular vectors, the relationship follows by construction. Therefore, the near-unit singular-vector alignment in Figure 3(b) verifies the correctness of the rotation extraction procedure and demonstrates that the estimated rotations accurately capture the correspondence between the singular-vector directions of the base and human-preference-aligned models.
We would like to clarify that Figures 3(a) and 3(b) serve complementary purposes within our preference geometric analysis. Specifically, Figure 3(a) establishes the key empirical observation underlying PrefGeo: human-preference-aligned models approximately preserve the singular values of the base model. Figure 3(b) builds upon this observation by verifying the rotation matrices extracted using Eq. (6), illustrating how the singular-vector directions of the base model can be aligned with those of the human-preference-aligned models. Together, these two figures characterize the weight transformations associated with human-preference alignment from the perspectives of singular-value preservation and singular-vector rotation.
Importantly, the empirical foundation of PrefGeo lies in the approximate preservation of singular values under human preference alignment. As shown in Figure 3(a), the normalized spectral distances of the HPA model remain below across layers, whereas those of the SFT model exceed . Furthermore, Figure 2(b) demonstrates that replacing the HPA model's singular values with those of the base model largely preserves its GenEval performance (0.9204 vs. 0.9224), while the same operation reduces the SFT model's performance from 0.7512 to 0.7286. These results provide empirical evidence that the spectral structure of human-preference-aligned models remains remarkably stable, supporting the use of rotation-based weight reconstruction while retaining the singular values of the base model.
Reply to Reviewer AM2s (part 1)
Response to Reviewer AM2s
We sincerely thank reviewer AM2s for the constructive feedback and for recognizing the practical relevance of our training-free multi-preference fusion framework, the promising experimental results, and the informative findings regarding singular-value preservation. We appreciate the concerns regarding the distinction between PrefGeo and OrthoMerge, the fairness and consistency of baseline comparisons, and the interpretation of the geometric evidence. To address these concerns, we provide additional experimental comparisons, clarify the baseline configurations, and refine the interpretation of our geometric findings. In the revised manuscript, we have highlighted all revisions and newly added content in blue for ease of reference. Our detailed responses are provided below:
1. [Q1,W1]: How does PrefGeo compare with the orthogonal-residual variant of OrthoMerge using the same preference experts and evaluation settings?
Response : We conducted additional experiments with six variants of OrthoMerge[1] on SD3.5-M, evaluating their performance across GenEval, OCR, and PickScore. The quantitative results are presented below.
As shown in Table C-1, PrefGeo consistently outperforms all six OrthoMerge variants across all three evaluation metrics, achieving the highest GenEval (0.92), OCR (0.93), and PickScore (22.92).
2. [Q2,W2]: Equal-weight averaging of the same three experts is algebraically equivalent to Task Arithmetic [2] with λ=1/3, provided the same parameters are merged. What scaling coefficient was used for Task Arithmetic in Table 1, what averaging weights were used for the W-space baseline in Table 3, and how were they selected? Could the authors explain the performance gap and clarify whether both baselines use the same parameter handling?
Response: The scaling coefficient λ used for Task Arithmetic in Table 1 was set to 1.0, following the default hyperparameter setting provided in its official implementation. We conducted additional experiments on SD3.5-M comparing Task Arithmetic with different merging coefficients λ and weight-space averaging . Specifically, Task Arithmetic constructs the merged weights as WTA=W0+λ∑i=13(Wi−W0), where W0 denotes the base model weights, Wi represents the weights of the i-th preference-specific expert, and λ controls the contribution of the expert updates. We evaluate Task Arithmetic with λ∈0.0,0.3,0.5,0.7,1.0 and additionally consider uniform weight-space averaging with λ≈1/3. The results are summarized below.
As shown in Table C-2, Task Arithmetic is sensitive to the merging coefficient λ, with its performance deteriorating substantially as λ increases beyond the optimal range. Although weight-space averaging achieves competitive results, it remains inferior to PrefGeo across all three evaluation metrics. Overall, PrefGeo consistently outperforms all compared variants, demonstrating the effectiveness of our method in integrating multiple preference-specific experts.
References
[1] "Orthogonal Model Merging." ICML 2026.
[2] "Editing Models with Task Arithmetic." ICLR 2023