acceptodds
Under review as a conference paper at ICLR 2027

PrefGeo: Preference Geometry for Multi-Preference Fusion of Flow Matching Models

Abstract

Human preference alignment has emerged as an effective approach for improving flow-matching-based text-to-image generation. However, existing methods primarily focus on aligning models with a single preference, while extending alignment to multiple preferences, such as compositional fidelity, text rendering, and aesthetics, remains challenging. Recent training-based multi-preference alignment methods, including DiffusionOPD, DiffusionNFT, and MapReduce LoRA, require at least 80GB of GPU memory and 170 GPU hours of training, making personalized multi-preference alignment computationally expensive and difficult to scale. In this paper, we introduce PrefGeo, a training-free framework that exploits the preference geometry between preference-aligned models and their shared base model. PrefGeo fuses models aligned with different preferences into a unified model without additional training. The fusion involves only model-weight operations and can be completed on a single NVIDIA RTX 4090 24GB GPU in 18 minutes, using only 0.5GB of GPU memory. Extensive experiments demonstrate that models produced by PrefGeo retain more than of the performance of their corresponding single-preference models, while eliminating the substantial training costs of existing multi-preference alignment methods.

Primary area:
generative models
Keywords:
Multi-preference fusion, Flow-matching model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AM2sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PrefGeo, a training-free method for combining preference-aligned flow-matching models. It represents each expert using left and right rotations relative to a shared base model and merges these rotations in Cayley coordinates. Experiments on SD3.5-M and FLUX.1-Dev evaluate compositional fidelity, text rendering, and aesthetic preference.

Critical strengths
  • The method addresses a useful problem. Combining existing preference experts without additional training could reduce the cost of multi-preference alignment. The proposed approach operates directly on model weights and avoids an additional alignment or distillation stage.

  • The reported results are promising. PrefGeo improves multiple metrics over the base models on both backbones. The expert-combination experiments suggest that it can retain capabilities from different experts. The observation that replacing a preference-aligned model's singular values with those of the base model largely preserves the evaluated task performance is also informative.

Critical weaknesses
  • The contribution beyond OrthoMerge is not sufficiently established. Eq. (11) is algebraically equivalent to the magnitude-corrected aggregation in OrthoMerge [1, Eqs. (7)–(8)]. The two-sided SVD representation is a clear difference, but its benefit over OrthoMerge's orthogonal-residual variant is not evaluated.

  • The weight-space baseline comparison needs clarification. Weight averaging in Table 3 achieves GenEval/OCR scores of 0.85/0.89, compared with 0.41/0.47 for Task Arithmetic in Table 1. The merging coefficients and their selection are not reported. PrefGeo still outperforms weight averaging, but it is unclear how much of the larger gap in Table 1 comes from the baseline configuration.

  • The geometric evidence needs a more careful interpretation. Under Eq. (6) as written, the near-unit singular-vector alignment in Figure 3(b) follows by construction and would also hold for an SFT model. It therefore does not by itself establish a preference-specific property. The spectral findings remain informative, but should be distinguished from this identity.

Questions for authors
  1. How does PrefGeo compare with the orthogonal-residual variant of OrthoMerge using the same preference experts and evaluation settings?

  2. Equal-weight averaging of the same three experts is algebraically equivalent to Task Arithmetic [2] with λ=1/3\lambda=1/3, provided the same parameters are merged. What scaling coefficient was used for Task Arithmetic in Table 1, what averaging weights were used for the W-space baseline in Table 3, and how were they selected? Could the authors explain the performance gap and clarify whether both baselines use the same parameter handling?

  3. Are the rotations in Figure 3(b) computed exactly as stated in Eq. (6), or are additional constraints imposed?

References:

[1] Sihan Yang, Kexuan Shi, and Weiyang Liu. Orthogonal Model Merging. ICML 2026.

[2] Gabriel Ilharco et al. Editing Models with Task Arithmetic. ICLR 2023.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer AM2s (part 1)

Response to Reviewer AM2s

We sincerely thank reviewer AM2s for the constructive feedback and for recognizing the practical relevance of our training-free multi-preference fusion framework, the promising experimental results, and the informative findings regarding singular-value preservation. We appreciate the concerns regarding the distinction between PrefGeo and OrthoMerge, the fairness and consistency of baseline comparisons, and the interpretation of the geometric evidence. To address these concerns, we provide additional experimental comparisons, clarify the baseline configurations, and refine the interpretation of our geometric findings. In the revised manuscript, we have highlighted all revisions and newly added content in blue for ease of reference. Our detailed responses are provided below:

1. [Q1,W1]: How does PrefGeo compare with the orthogonal-residual variant of OrthoMerge using the same preference experts and evaluation settings?

Response : We conducted additional experiments with six variants of OrthoMerge[1] on SD3.5-M, evaluating their performance across GenEval, OCR, and PickScore. The quantitative results are presented below.

Table C-1. Comparison of PrefGeo with six OrthoMerge variants on SD3.5-M across GenEval, OCR, and PickScore. Bold indicates the best performance, ↑\uparrow denotes that higher values are better.

ModelGenEval$\uparrow$OCR$\uparrow$PickScore$\uparrow$
SD3.5-M (Baseline)0.650.6921.69
OrthoMerge-G-TA0.860.8322.49
OrthoMerge-G-TIES0.880.8222.78
OrthoMerge-G-TSVM0.830.8822.73
OrthoMerge-C-TA0.800.7822.26
OrthoMerge-C-TIES0.900.8122.84
OrthoMerge-C-TSVM0.890.9022.46
PrefGeo(Ours)0.920.9322.92

As shown in Table C-1, PrefGeo consistently outperforms all six OrthoMerge variants across all three evaluation metrics, achieving the highest GenEval (0.92), OCR (0.93), and PickScore (22.92).

2. [Q2,W2]: Equal-weight averaging of the same three experts is algebraically equivalent to Task Arithmetic [2] with λ=1/3\lambda=1/3, provided the same parameters are merged. What scaling coefficient was used for Task Arithmetic in Table 1, what averaging weights were used for the W-space baseline in Table 3, and how were they selected? Could the authors explain the performance gap and clarify whether both baselines use the same parameter handling?

Response: The scaling coefficient λ\lambda used for Task Arithmetic in Table 1 was set to 1.0, following the default hyperparameter setting provided in its official implementation. We conducted additional experiments on SD3.5-M comparing Task Arithmetic with different merging coefficients λ\lambda and weight-space averaging . Specifically, Task Arithmetic constructs the merged weights as WTA=W0+λ∑i=13(Wi−W0),W_{\mathrm{TA}} = W_0 + \lambda \sum_{i=1}^{3}(W_i-W_0), where W0W_0 denotes the base model weights, WiW_i represents the weights of the ii-th preference-specific expert, and λ\lambda controls the contribution of the expert updates. We evaluate Task Arithmetic with λ∈0.0,0.3,0.5,0.7,1.0\lambda \in \\{0.0, 0.3, 0.5, 0.7, 1.0\\} and additionally consider uniform weight-space averaging with λ≈1/3\lambda \approx 1/3. The results are summarized below.

Table C-2. Comparison of Task Arithmetic with different merging coefficients λ\lambda, uniform weight-space averaging and PrefGeo on SD3.5-M. The best results are highlighted in bold. ↑\uparrow denotes that higher values are better.

Methodλ\lambdaGenEval ↑OCR ↑PickScore ↑
Task Arithmetic (Baseline)0.000.650.6921.69
Task Arithmetic0.300.840.8822.41
Task Arithmetic0.500.790.8322.18
Task Arithmetic0.700.610.6521.32
Task Arithmetic1.000.410.4720.20
Task Arithmetic (Weight Average)0.330.850.8922.39
PrefGeo (Ours)—0.920.9322.92

As shown in Table C-2, Task Arithmetic is sensitive to the merging coefficient λ\lambda, with its performance deteriorating substantially as λ\lambda increases beyond the optimal range. Although weight-space averaging achieves competitive results, it remains inferior to PrefGeo across all three evaluation metrics. Overall, PrefGeo consistently outperforms all compared variants, demonstrating the effectiveness of our method in integrating multiple preference-specific experts.

References

[1] "Orthogonal Model Merging." ICML 2026.

[2] "Editing Models with Task Arithmetic." ICLR 2023

AuthorsOpenReview ↗

Reply to Reviewer AM2s (part 2)

3. [Q3, W3]: Are the rotations in Figure 3(b) computed exactly as stated in Eq. (6), or are additional constraints imposed? (Interpretation of Geometric Evidence). The geometric evidence needs a more careful interpretation. Under Eq. (6) as written, the near-unit singular-vector alignment in Figure 3(b) follows by construction and would also hold for an SFT model. It therefore does not by itself establish a preference-specific property. The spectral findings remain informative, but should be distinguished from this identity.

Response: We thank the reviewer for the insightful observation. The rotations in Figure 3(b) are computed directly according to Eq. (6), without imposing additional constraints. We agree that, given RL=UiU0⊤R_L=U_iU_0^\top and the orthonormality of the singular vectors, the relationship RLU0=UiR_LU_0=U_i follows by construction. Therefore, the near-unit singular-vector alignment in Figure 3(b) verifies the correctness of the rotation extraction procedure and demonstrates that the estimated rotations accurately capture the correspondence between the singular-vector directions of the base and human-preference-aligned models.

We would like to clarify that Figures 3(a) and 3(b) serve complementary purposes within our preference geometric analysis. Specifically, Figure 3(a) establishes the key empirical observation underlying PrefGeo: human-preference-aligned models approximately preserve the singular values of the base model. Figure 3(b) builds upon this observation by verifying the rotation matrices extracted using Eq. (6), illustrating how the singular-vector directions of the base model can be aligned with those of the human-preference-aligned models. Together, these two figures characterize the weight transformations associated with human-preference alignment from the perspectives of singular-value preservation and singular-vector rotation.

Importantly, the empirical foundation of PrefGeo lies in the approximate preservation of singular values under human preference alignment. As shown in Figure 3(a), the normalized spectral distances of the HPA model remain below 2×10−32\times10^{-3} across layers, whereas those of the SFT model exceed 0.050.05. Furthermore, Figure 2(b) demonstrates that replacing the HPA model's singular values with those of the base model largely preserves its GenEval performance (0.9204 vs. 0.9224), while the same operation reduces the SFT model's performance from 0.7512 to 0.7286. These results provide empirical evidence that the spectral structure of human-preference-aligned models remains remarkably stable, supporting the use of rotation-based weight reconstruction while retaining the singular values of the base model.

Reviewer_bVfhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PrefGeo, a training-free model fusion framework for multi-preference alignment in flow-matching text-to-image models. The method is motivated by an empirical observation termed preference geometry, which reveals that preference alignment largely preserves the singular values of the base model's weights while primarily inducing orthogonal rotations in the left and right singular vector spaces. Leveraging this insight, PrefGeo estimates preference-specific rotation matrices via Orthogonal Procrustes analysis, fuses them in the skew-symmetric space using the Cayley transform with a norm-preserving constraint, and applies the aggregated rotations back to the base model. Experiments demonstrate the effectiveness and efficiency of PrefGeo.

Critical strengths
  1. The paper identifies a compelling structural property of preference-aligned flow-matching models: RLHF preserves the singular value spectrum of the base model's weights while primarily inducing orthogonal rotations in the left and right singular vector spaces (unlike SFT). This insight is consequential because it explains why standard linear model-merging methods fail on flow-matching models and directly motivates a principled, closed-form fusion pipeline using Orthogonal Procrustes analysis and the Cayley transform.

  2. PrefGeo achieves an exceptional efficiency-performance trade-off by reformulating multi-preference alignment as a training-free weight fusion problem. It completes model merging in just 18 minutes using 0.5 GB of VRAM on a single consumer GPU, while retaining over 95% of single-expert performance across both SD3.5-M and FLUX.1-Dev, making scalable multi-preference alignment highly practical.

Critical weaknesses
  1. The formulation of rotation estimation and fusion (Equations 5–8) lacks rigor for rectangular weight matrices. When m≠nm \neq n, RL,i=UiU0⊤R_{L,i} = U_i U_0^\top under thin SVD is not orthogonal (U0U0⊤≠IU_0 U_0^\top \neq I), while full SVD leaves the singular vectors of zero singular values non-unique. Moreover, Equation 6 enforces only O(d)O(d) with det⁡(R)=±1\det(R) = \pm 1, not the det⁡(R)=1\det(R) = 1 constraint required by Equation 5, and an eigenvalue of −1-1 renders the inverse Cayley transform singular (R+IR + I not invertible). These issues cast doubt on the validity and exact implementation of the core algorithm for general layers, and should be clarified.
  2. Two points concerning Equations 10–11. First, in Equation 11, if experts induce strongly conflicting rotations such that ∣∑iQL,iℓ∣F|\sum_i Q_{L,i}^\ell|_F approaches zero, does the norm-preserving rescaling amplify noise and lead to instability? Second, can Equation 10 be extended with non-uniform weights wiw_i to support controllable trade-offs among competing preferences? A brief analysis (or empirical test) addressing both would strengthen the paper.
Questions for authors
  1. Does the observed preference geometry (singular value invariance) generalize across alignment algorithms (e.g., DPO, ReFL) and LoRA fine-tuning? In the FLUX.1-Dev experiments, LoRA experts are merged as W0+BAW_0 + BA, yet low-rank additive updates typically alter the top singular values. It would be helpful to quantify how well singular value invariance holds in this setting, compared with full-parameter RLHF.
  2. The baseline results (Table 1) and ablation (Table 3) appear inconsistent for linear weight merging: Task Arithmetic scores only 0.41 on GenEval and 0.47 on OCR in Table 1, yet "W Space" averaging achieves 0.85 and 0.89 in Table 3—even though TA with scaling 1/N1/N is equivalent to uniform weight averaging. This suggests the Table 1 baselines may be suboptimally tuned, and weakens the evidence that the proposed geometric framework outperforms simple weight averaging. The authors should reconcile these results and report properly tuned baselines.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer bVfh (part 1)

Response to Reviewer bVfh

We sincerely thank reviewer bVfh for the thoughtful and constructive feedback and for recognizing the significance of our preference geometry observation, the clarity of our presentation, and the strong efficiency-performance trade-off achieved by PrefGeo. We appreciate the concerns regarding the mathematical rigor of rotation estimation and fusion for rectangular matrices, the numerical stability and flexibility of Cayley-space fusion, the generalizability of singular-value preservation across different alignment methods and LoRA fine-tuning, and the fairness and consistency of baseline comparisons. To address these concerns, we provide additional theoretical clarifications, numerical stability analyses, and experimental evaluations, while refining the baseline configurations to ensure fair comparisons. In the resubmitted manuscript, we have highlighted all revisions and newly added content in blue for ease of reference. Our detailed responses are provided below:

1. [W1]: The formulation of rotation estimation and fusion (Equations 5–8) lacks rigor for rectangular weight matrices. When m≠nm \neq n, RL,i=UiU0⊤R_{L,i} = U_i U_0^\top under thin SVD is not orthogonal (U0U0⊤≠IU_0 U_0^\top \neq I), while full SVD leaves the singular vectors of zero singular values non-unique. Moreover, Equation 6 enforces only O(d)O(d) with det⁡(R)=±1\det(R) = \pm 1, not the det⁡(R)=1\det(R) = 1 constraint required by Equation 5, and an eigenvalue of −1-1 renders the inverse Cayley transform singular (R+IR + I not invertible). These issues cast doubt on the validity and exact implementation of the core algorithm for general layers, and should be clarified.

Response:

(1) Orthogonality of the weight matrices considered in our experiments. We would like to clarify that our SVD-based rotation estimation and Cayley-space fusion are applied to selected linear projection layers within the Transformer blocks of SD3.5-M and FLUX.1-Dev. Importantly, all weight matrices involved in our reported rotation-based fusion experiments are square matrices, rather than general rectangular matrices. Specifically, for each selected weight matrix Wiℓ∈Rdℓ×dℓW_i^\ell\in\mathbb{R}^{d_\ell\times d_\ell}, we compute its full SVD: Wiℓ=UiℓΣiℓ(Viℓ)⊤,W_i^\ell=U_i^\ell\Sigma_i^\ell(V_i^\ell)^\top, where Uiℓ,Viℓ∈Rdℓ×dℓU_i^\ell,V_i^\ell\in\mathbb{R}^{d_\ell\times d_\ell} are square orthogonal matrices. Accordingly, the left and right transformations in Equation (6) are computed as RL,iℓ=Uiℓ(U0ℓ)⊤,RR,iℓ=Viℓ(V0ℓ)⊤.R_{L,i}^\ell=U_i^\ell(U_0^\ell)^\top, R_{R,i}^\ell=V_i^\ell(V_0^\ell)^\top. Since the singular-vector matrices are square and orthogonal, we have (RL,iℓ)⊤RL,iℓ=Idℓ,(RR,iℓ)⊤RR,iℓ=Idℓ.(R_{L,i}^\ell)^\top R_{L,i}^\ell=I_{d_\ell}, (R_{R,i}^\ell)^\top R_{R,i}^\ell=I_{d_\ell}. Furthermore, applying these transformations to the base-model weights yields RL,iℓW0ℓ(RR,iℓ)⊤=UiℓΣ0ℓ(Viℓ)⊤,R_{L,i}^\ell W_0^\ell(R_{R,i}^\ell)^\top=U_i^\ell\Sigma_0^\ell(V_i^\ell)^\top, which preserves the singular values of the base model while incorporating the expert's singular-vector directions.

Therefore, the orthogonality issue associated with thin SVD for rectangular matrices does not arise in our experimental setting.

(2) Orthogonal and special-orthogonal constraints. We clarify that our implementation incorporates a determinant correction during rotation estimation to ensure that the resulting rotation matrices belong to the special orthogonal group (SO(d)). Specifically, given the SVD of the cross-covariance matrix UiU0⊤=PΛQ⊤U_iU_0^\top=P\Lambda Q^\top, when the resulting rotation matrix has a determinant of −1-1, we apply a determinant correction as follows: R=Pdiag⁡(1,…,1,det⁡(PQ⊤))Q⊤,R=P\operatorname{diag}(1,\ldots,1,\det(PQ^\top))Q^\top, which ensures that the corrected rotation matrix satisfies both R⊤R=IR^\top R=I and det⁡(R)=1\det(R)=1.

(3)Singular inverse Cayley transform. The inverse Cayley transform in Eq. (7) requires R+IR+I to be invertible. When this matrix is singular or numerically ill-conditioned, we bypass the corresponding Cayley-space fusion and perform the merging directly in weight space for the affected matrix, thereby avoiding unstable matrix inversion.

In practice, we did not encounter a non-invertible R+IR+I matrix in any of our merging experiments. Nevertheless, we have incorporated the aforementioned safeguards as precautionary measures to ensure numerical stability.

AuthorsOpenReview ↗

Reply to Reviewer bVfh (part 2)

2. [W2]: Two points concerning Equations 10–11. First, in Equation 11, if experts induce strongly conflicting rotations such that ∥∑iQL,iℓ∥F\left\|\sum_i Q_{L,i}^{\ell}\right\|_F approaches zero, does the norm-preserving rescaling amplify noise and lead to instability? Second, can Equation 10 be extended with non-uniform weights wiw_i to support controllable trade-offs among competing preferences? A brief analysis (or empirical test) addressing both would strengthen the paper.

Response:

(1) Vanishing sum of Cayley representations. When the Frobenius norm of the sum of preference-specific Cayley representations approaches zero, the normalization in Eq. (11) becomes numerically unstable. To address this issue, we fall back to the unnormalized average of the Cayley representations in this degenerate case to ensure numerical stability. In addition, we conducted an additional experiment comparing Q-space averaging with and without the proposed norm constraint on SD3.5-M. The results are presented in Table B-1.

Table B-1. Comparison of Q-space averaging without the norm constraint and PrefGeo on SD3.5-M. Bold indicates the best performance, and ↑\uparrow denotes that higher values are better.

MethodGenEval$\uparrow$OCR$\uparrow$PickScore$\uparrow$
Q-space averaging without norm constraint0.890.9022.69
PrefGeo (Ours)0.920.9322.92

The results in Table B-1 demonstrate the effectiveness of the proposed norm-preserving rescaling strategy.

(2) Controllable Preference Trade-offs. Eq. (10) can be naturally extended to support non-uniform preference weights, allowing controllable trade-offs among competing preferences. Specifically, we introduce preference-specific coefficients αi≥0\alpha_i\geq 0 and generalize Eq. (10) as Qavg=∑i=1NαiQi∑i=1NαiQ_{\mathrm{avg}}=\frac{\sum_{i=1}^{N}\alpha_i Q_i}{\sum_{i=1}^{N}\alpha_i} where αi\alpha_i controls the contribution of the ii-th preference expert. The resulting weighted average is subsequently processed using the same norm-preserving rescaling as in Eq. (11).

To empirically validate this flexibility, we conducted additional experiments on SD3.5-M with different preference-weight configurations for GenEval, OCR, and aesthetics.

Table B-2. Performance of PrefGeo under different preference-weight configurations on SD3.5-M. The weight ratios correspond to GenEval:OCR:AES. ↑\uparrow denotes that higher values are better.

Preference weights (GenEval:OCR:AES)GenEval ↑OCR ↑PickScore ↑Average ↑
PrefGeo (1:1:1)0.92350.928422.92170.9112
PrefGeo (2:1:1)0.93600.918222.12840.9018
PrefGeo (1:2:1)0.89310.948422.45460.9017
PrefGeo (1:1:2)0.88090.910623.18680.8944

As shown in Table B-2, increasing the weight of a particular preference expert improves its corresponding evaluation metric. Specifically, assigning a higher weight to GenEval improves its score from 0.9235 to 0.9360, prioritizing OCR increases its score from 0.9284 to 0.9484, and emphasizing aesthetics improves PickScore from 22.9217 to 23.1868. Meanwhile, uniform weighting achieves the highest average score, suggesting a more balanced trade-off across the three preferences.

AuthorsOpenReview ↗

Reply to Reviewer bVfh (part 3)

3. [Q1]: Does the observed preference geometry (singular value invariance) generalize across alignment algorithms (e.g., DPO, ReFL) and LoRA fine-tuning? In the FLUX.1-Dev experiments, LoRA experts are merged as W0+BAW_0+BA , yet low-rank additive updates typically alter the top singular values. It would be helpful to quantify how well singular value invariance holds in this setting, compared with full-parameter RLHF.

Response:

(1) Generalization to DPO-based preference alignment. We additionally examine RealAlign-SD-3.5-M [1], a publicly available preference-aligned model trained with Diffusion-DPO. Specifically, we incorporate the released LoRA updates into the SD3.5-M base model and analyze the singular values of the resulting weight matrices. The results are presented in the table below:

Table B-3. Singular-value preservation under DPO-based finetuning (RealAlign) on SD3.5-M. We also present the results for SFT and random weight perturbations (Rand.) for reference.

RankSD3.5-MRealAlignSFTRand.
17.76277.7611 (−0.02%)7.3191 (−5.71%)8.3135 (+7.10%)
26.66726.6782 (+0.17%)6.3293 (−5.07%)7.1484 (+7.22%)
46.27116.2792 (+0.13%)5.9782 (−4.67%)6.7154 (+7.09%)
85.70095.7160 (+0.27%)5.5045 (−3.44%)6.0908 (+6.84%)
165.00285.0259 (+0.46%)4.8971 (−2.11%)5.3301 (+6.54%)
324.38314.3847 (+0.04%)4.3571 (−0.59%)4.6769 (+6.70%)
643.68223.7010 (+0.51%)3.7285 (+1.26%)3.8890 (+5.62%)
1283.07823.0955 (+0.56%)3.2190 (+4.57%)3.2063 (+4.16%)
2562.39462.4085 (+0.58%)2.6077 (+8.90%)2.3949 (+0.01%)

Table B-3 demonstrates that the approximate singular-value invariance observed under preference alignment also holds for DPO-based fine-tuning.

(2) Generalization to LoRA fine-tuning on FLUX.1-Dev. We would also like to clarify that our FLUX.1-Dev experiments already employ LoRA-fine-tuned preference experts, rather than full-parameter fine-tuned models. Specifically, we use three publicly available LoRA experts from MapReduce LoRA [2], corresponding to GenEval, OCR, and aesthetics. We incorporate their LoRA updates into the FLUX.1-Dev base weights and subsequently apply PrefGeo for multi-preference fusion. We further report the singular values and their relative changes for each human-preference-aligned expert on FLUX.1-Dev, as presented in Table B-4.

Table B-4. Comparison of singular values and their relative changes (%) across the FLUX.1-Dev base model and the GenEval, OCR, and AES experts.

RankFLUX.1-DevGenEvalOCRAES
115.642815.6440 (+0.0080%)15.6430 (+0.0011%)15.6437 (+0.0055%)
212.379012.3835 (+0.0365%)12.3799 (+0.0074%)12.3795 (+0.0038%)
410.898710.9057 (+0.0637%)10.9058 (+0.0652%)10.9008 (+0.0189%)
89.80639.8067 (+0.0045%)9.8078 (+0.0160%)9.8052 (−0.0106%)
168.20898.2124 (+0.0422%)8.2119 (+0.0369%)8.2088 (−0.0018%)
327.26617.2657 (−0.0054%)7.2657 (−0.0058%)7.2661 (+0.0004%)
646.11896.1201 (+0.0195%)6.1224 (+0.0568%)6.1199 (+0.0162%)
1285.04515.0491 (+0.0787%)5.0459 (+0.0164%)5.0458 (+0.0145%)
2563.93313.9353 (+0.0551%)3.9355 (+0.0612%)3.9337 (+0.0158%)

The results in Table B-4 demonstrate that approximate singular-value invariance holds for LoRA-based preference fine-tuning on FLUX.1-Dev, supporting the generalizability of the observed preference geometry across different fine-tuning paradigms and model backbones.

While a direct comparison with full-parameter RLHF on FLUX.1-Dev would provide additional insights, the substantial computational resources required for full-parameter RLHF fine-tuning are beyond our current computational budget. Consequently, we are unable to conduct full-parameter RLHF experiments on FLUX.1-Dev and instead rely on publicly available LoRA-fine-tuned preference experts to investigate the generalizability of preference geometry. We consider extending this analysis to full-parameter RLHF on larger models an important direction for future work.

References:

[1] "When Preference Labels Fall Short: Aligning Diffusion Models from Real Data." ICML 2026.

[2] "MapReduce LoRA: Advancing The Pareto Front in Multi-Preference Optimization for Generative Models." ICCV, 2026.

AuthorsOpenReview ↗

Reply to Reviewer bVfh (part 4)

4. [Q2]: The baseline results (Table 1) and ablation (Table 3) appear inconsistent for linear weight merging: Task Arithmetic scores only 0.41 on GenEval and 0.47 on OCR in Table 1, yet "W Space" averaging achieves 0.85 and 0.89 in Table 3—even though TA with scaling is equivalent to uniform weight averaging. This suggests the Table 1 baselines may be suboptimally tuned, and weakens the evidence that the proposed geometric framework outperforms simple weight averaging. The authors should reconcile these results and report properly tuned baselines.

Response: The scaling coefficient λ\lambda used for Task Arithmetic in Table 1 was set to 1.0, following the default hyperparameter setting provided in its official implementation. We conducted additional experiments on SD3.5-M comparing Task Arithmetic with different merging coefficients λ\lambda and weight-space averaging . Specifically, Task Arithmetic constructs the merged weights as WTA=W0+λ∑i=13(Wi−W0),W_{\mathrm{TA}} = W_0 + \lambda \sum_{i=1}^{3}(W_i-W_0), where W0W_0 denotes the base model weights, WiW_i represents the weights of the ii-th preference-specific expert, and λ\lambda controls the contribution of the expert updates. We evaluate Task Arithmetic with λ∈0.0,0.3,0.5,0.7,1.0\lambda \in \\{0.0, 0.3, 0.5, 0.7, 1.0\\} and additionally consider uniform weight-space averaging with λ≈1/3\lambda \approx 1/3. The results are summarized below.

Table B-5. Comparison of Task Arithmetic with different merging coefficients λ\lambda, uniform weight-space averaging and PrefGeo on SD3.5-M. The best results are highlighted in bold. ↑\uparrow denotes that higher values are better.

Methodλ\lambdaGenEval ↑OCR ↑PickScore ↑
Task Arithmetic (Baseline)0.000.650.6921.69
Task Arithmetic0.300.840.8822.41
Task Arithmetic0.500.790.8322.18
Task Arithmetic0.700.610.6521.32
Task Arithmetic1.000.410.4720.20
Task Arithmetic (Weight Average)0.330.850.8922.39
PrefGeo (Ours)—0.920.9322.92

As shown in Table B-5, Task Arithmetic is sensitive to the merging coefficient λ\lambda, with its performance deteriorating substantially as λ\lambda increases beyond the optimal range. Although weight-space averaging achieves competitive results, it remains inferior to PrefGeo across all three evaluation metrics. Overall, PrefGeo consistently outperforms all compared variants, demonstrating the effectiveness of our method in integrating multiple preference-specific experts.

Reviewer_ZMcLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to merge several flow-matching experts, each aligned to a different preference, into one model without training. It reports a "preference geometry" phenomenon: preference alignment leaves the singular values of the weights essentially unchanged and only rotates the singular vectors. PrefGeo therefore estimates each expert's left and right rotations relative to the base with a closed-form Procrustes solution, averages them in Cayley space under a norm constraint, and applies the result to the base weights, which takes 18 minutes on a single RTX 4090. Experiments merge GenEval, OCR and aesthetics experts on SD3.5-M, with an extension to FLUX.1-Dev.

Critical strengths

The problem is of practical relevance: given existing single-preference experts, a training-free merge into a multi-preference model is much cheaper than rerunning multi-reward RL or distillation. The method involves only SVD and closed-form operations, is simple to implement, and Appendix A gives a complete derivation of the fusion problem. The comparison is fairly comprehensive, covering single-preference experts, four training-based methods and three model-merging methods, together with in-domain and out-of-domain metrics, a user study, and results on FLUX.1-Dev. On SD3.5-M the Average is close to DiffusionOPD (0.871 vs. 0.894), and on FLUX.1-Dev it is on par with MapReduce LoRA (0.881 vs. 0.883). The fusion-space ablation in Table 3 is informative: directly averaging rotation matrices degrades badly, showing that the choice of fusion space does matter.

Critical weaknesses

The evidence does not establish the claimed preference geometry. The reported “rotation relationship” follows algebraically from the construction: the identical reconstruction scores in Figure 2(b) and constant similarity of 1 in Figure 3(b) therefore do not independently demonstrate that preference alignment induces rotations. The substantive observation is singular-value preservation, but this lacks controls matched for update magnitude; small updates alone can produce small spectral changes. Moreover, the analysis mainly covers one weight type and one expert, despite applying rotation fusion to all weights. These gaps leave the method’s central geometric premise insufficiently supported.  The comparisons do not establish an advantage over simpler merging methods. Table 1 reports poor Task Arithmetic results, yet the uniform weight average in Table 3—equivalent to Task Arithmetic with an appropriate coefficient—performs substantially better. Baseline hyperparameters are also unspecified. This is particularly consequential because, near the identity, the proposed fusion reduces to a scaled average of task vectors to first order. A coefficient sweep for Task Arithmetic and comparisons against averaging with matched scaling and fusion without the norm constraint are needed to isolate the proposed method’s benefit. Without these controls, the reported gains could reflect baseline tuning or update scaling rather than the proposed geometry.

Questions for authors

What is the per-layer relative update size of the HPA expert and of the SFT model? Are the singular values equally preserved for an SFT model with matched update magnitude and for random perturbations? Does the phenomenon hold for other weight types, for the OCR and aesthetics experts, and on FLUX.1-Dev? Which hyperparameters were used for the three merging baselines in Table 1? What are the results of Task Arithmetic with a λ sweep, a weight-space average with the same coefficient , and a Q-space average without the norm constraint? Can the method be compared with OrthoMerge? How are the cases and a singular handled? How are non-square matrices and non-matrix parameters fused? How were the three experts on FLUX.1-Dev obtained, and what was their training cost?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer ZMcL (part 1)

Response to Reviewer ZMcL

We sincerely thank reviewer ZMcL for the constructive feedback and for recognizing the practical relevance, computational efficiency, and comprehensive evaluation of PrefGeo. We appreciate the concerns regarding the empirical evidence for preference geometry, baseline fairness, and mathematical details. In the resubmitted manuscript, we have highlighted all revisions and newly added content in blue for ease of reference. Our detailed responses are provided below:

1. [Q1,Q2]: What is the per-layer relative update size of the HPA expert and of the SFT model? Are the singular values equally preserved for an SFT model with matched update magnitude and for random perturbations? (Update Magnitude and Spectral Preservation)

Response: To clarify, we quantify the layer-wise relative update magnitude as ρvℓ=∥Wvℓ−W0ℓ∥F∥W0ℓ∥F,\rho_v^\ell = \frac{\lVert W_v^\ell - W_0^\ell \rVert_F}{\lVert W_0^\ell \rVert_F}, where ℓ\ell denotes the layer index, vv denotes the model variant, and WvℓW_v^\ell and W0ℓW_0^\ell represent the weights of the corresponding model variant and the base model (SD3.5-M) at layer ℓ\ell, respectively. We report the layer-wise relative update magnitudes of the human-preference-aligned (HPA) model for GenEval, the supervised fine-tuning (SFT) model for GenEval, and the randomly perturbed (Rand.) model, denoted by ρHPAℓ\rho_{\mathrm{HPA}}^\ell, ρSFTℓ\rho_{\mathrm{SFT}}^\ell, and ρRand.ℓ\rho_{\mathrm{Rand.}}^\ell, respectively. The results are presented in the table below:

Table A-1: Layer-wise relative weight update magnitudes (%) of the HPA, SFT and Rand. models.

ℓ\ellρHPAℓ\rho^\ell_\mathrm{HPA}ρSFTℓ\rho^\ell_\mathrm{SFT}ρRand.ℓ\rho^\ell_\mathrm{Rand.}│ℓ\ellρHPAℓ\rho^\ell_\mathrm{HPA}ρSFTℓ\rho^\ell_\mathrm{SFT}ρRand.ℓ\rho^\ell_\mathrm{Rand.}
04.77%5.17%4.76%│124.53%4.26%4.53%
14.86%4.92%4.83%│134.26%4.58%4.21%
24.37%4.57%4.32%│144.55%4.31%4.55%
34.63%4.38%4.66%│154.43%4.68%4.43%
44.53%4.81%4.50%│164.37%4.14%4.38%
54.55%4.33%4.55%│174.27%4.53%4.26%
64.58%4.75%4.56%│185.17%5.43%5.12%
74.49%4.19%4.42%│195.87%5.64%5.87%
84.42%4.69%4.42%│204.35%4.62%4.30%
94.40%4.25%4.40%│215.14%4.89%5.11%
104.56%4.84%4.56%│225.35%5.62%5.32%
114.63%4.39%4.61%│239.98%9.72%9.95%

As shown in the Table A-1, the SFT and HPA models exhibit very similar layer-wise relative weight update magnitudes, with only minor differences across layers. However, as shown in Figure 2(a) of our manuscript, the SFT model does not exhibit the same phenomenon of singular-value preservation as the HPA model. This comparison suggests the preference geometry of the HPA model cannot be attributed merely to the small magnitude of weight updates but instead reflects a distinctive property of preference alignment.

Furthermore, we construct a Rand. model with layer-wise relative weight update magnitudes matched to those of the HPA model. To quantify spectral changes, we compute the relative change in the ii-th singular value of each model with respect to the base SD3.5-M model as $\frac{\sigma_i^{\mathrm{expert}}-\sigma_i^{\mathrm{SD3.5-M}}} {\sigma_i^{\mathrm{SD3.5-M}}}\times 100\%$, where σiexpert\sigma_i^{\mathrm{expert}} and σiSD3.5−M\sigma_i^{\mathrm{SD3.5-M}} denote the ii-th singular values of the corresponding expert model and the base SD3.5-M model, respectively. The corresponding relative changes in singular values are presented below.

Table A-2. Comparison of singular values and their relative changes (%) with respect to the base SD3.5-M model.

RankSD3.5-MHPASFTRand.
17.76277.7707 (+0.10%)7.3191 (−5.71%)8.3135 (+7.10%)
26.66716.6643 (−0.04%)6.3293 (−5.07%)7.1484 (+7.22%)
46.27106.2703 (−0.01%)5.9782 (−4.67%)6.7154 (+7.09%)
85.70085.7017 (+0.02%)5.5045 (−3.44%)6.0908 (+6.84%)
165.00285.0060 (+0.06%)4.8971 (−2.11%)5.3301 (+6.54%)
324.38314.3862 (+0.07%)4.3571 (−0.59%)4.6769 (+6.70%)
643.68213.6831 (+0.03%)3.7285 (+1.26%)3.8890 (+5.62%)
1283.07813.0771 (−0.03%)3.2190 (+4.57%)3.2063 (+4.16%)
2562.39462.3982 (+0.15%)2.6077 (+8.90%)2.3949 (+0.01%)

As shown in the Table A-2 above and Figure 7 in the appendix of the revised manuscript, SFT and Rand. models do not inherently preserve singular values, further demonstrating that the preference geometry cannot be explained solely by the magnitude of weight updates, but rather reflects an intrinsic property of preference alignment.

AuthorsOpenReview ↗

Reply to Reviewer ZMcL (part 2)

2. [Q3]: Does the phenomenon hold for other weight types, for the OCR and aesthetics experts, and on FLUX.1-Dev?

Response: To further investigate the generalizability of preference geometry, we extend our spectral analysis to multiple human-preference-aligned (HPA) experts and model backbones. Specifically, we examine the GenEval, OCR and Aesthetics (AES) experts based on SD3.5-M, as well as human-preference-aligned experts based on FLUX.1-Dev. The corresponding singular values and their relative changes are presented below.

Table A-3. Comparison of singular values and their relative changes (%) across the SD3.5-M base model and the GenEval, OCR, and AES experts.

RankSD3.5-MGenEvalOCRAES
17.76277.7707 (+0.1038%)7.7603 (−0.0308%)7.7672 (+0.0578%)
26.66716.6643 (−0.0420%)6.6689 (+0.0266%)6.6698 (+0.0402%)
46.27106.2703 (−0.0125%)6.2725 (+0.0236%)6.2698 (−0.0199%)
85.70085.7017 (+0.0153%)5.7015 (+0.0118%)5.7018 (+0.0166%)
165.00285.0060 (+0.0645%)5.0021 (−0.0131%)5.0027 (−0.0012%)
324.38314.3862 (+0.0718%)4.3831 (+0.0014%)4.3848 (+0.0398%)
643.68213.6831 (+0.0261%)3.6827 (+0.0160%)3.6821 (+0.0005%)
1283.07813.0771 (−0.0334%)3.0776 (−0.0163%)3.0790 (+0.0293%)
2562.39462.3982 (+0.1533%)2.3941 (−0.0192%)2.3945 (−0.0009%)

Table A-4. Comparison of singular values and their relative changes (%) across the FLUX.1-Dev base model and the GenEval, OCR, and AES experts.

RankFLUX.1-DevGenEvalOCRAES
115.642815.6440 (+0.0080%)15.6430 (+0.0011%)15.6437 (+0.0055%)
212.379012.3835 (+0.0365%)12.3799 (+0.0074%)12.3795 (+0.0038%)
410.898710.9057 (+0.0637%)10.9058 (+0.0652%)10.9008 (+0.0189%)
89.80639.8067 (+0.0045%)9.8078 (+0.0160%)9.8052 (−0.0106%)
168.20898.2124 (+0.0422%)8.2119 (+0.0369%)8.2088 (−0.0018%)
327.26617.2657 (−0.0054%)7.2657 (−0.0058%)7.2661 (+0.0004%)
646.11896.1201 (+0.0195%)6.1224 (+0.0568%)6.1199 (+0.0162%)
1285.04515.0491 (+0.0787%)5.0459 (+0.0164%)5.0458 (+0.0145%)
2563.93313.9353 (+0.0551%)3.9355 (+0.0612%)3.9337 (+0.0158%)

Tables A-3 and A-4 show that these models exhibit the same phenomenon observed in the GenEval expert: preference alignment largely preserves the singular values of the base model while modifying the singular-vector directions. Furthermore, Figures 8 and 9 in the appendix of the revised manuscript illustrate the singular-value spectra across different ranks. These consistent observations across different preference objectives and model architectures provide further empirical evidence suggesting that preference geometry is not specific to a particular preference expert or model backbone, but represents a more general property of human-preference-aligned models.

AuthorsOpenReview ↗

Reply to Reviewer ZMcL (part 3)

3. [Q4]: Which hyperparameters were used for the three merging baselines in Table 1?

Response: We used the same three human-preference-aligned experts (GenEval, OCR, AES) in Table 1 of the manuscript. Let W0W_0 denote the SD3.5-M base weight and Δi=Wi−W0\Delta_i=W_i-W_0 the dense weight update induced by expert ii. For all baselines, we adopted the default hyperparameters provided in their respective official implementations. The baseline-specific settings were as follows:

  • Task Arithmetic (TA). We usedWTA=W0+λ∑i=13Δi,λ=1.0.W_{\mathrm{TA}} =W_0+\lambda\sum_{i=1}^{3}\Delta_i, \qquad \lambda=\mathbf{1.0}. Here, λ\lambda is a global coefficient controlling the overall strength of the summed expert updates.

  • Task Singular Vector Merging (TSV-M). For each expert update, we first computed$\Delta_i=U_i\Sigma_iV_i^\top.$ The singular components retained from all experts were concatenated, and their left and right singular-vector frames were orthogonalized using the SVD-based polar decomposition. The resulting TSV-M update was added to the W0W_0 with a merge scale of 1.0\mathbf{1.0}, meaning that the reconstructed update was applied without additional rescaling. We did not impose a manually selected rank cap. Following TSV-M, the maximum number of components available to each expert for an m×nm\times n weight matrix was determined automatically as rbudget=⌊min⁡(m,n)N⌋r_{\mathrm{budget}} =\left\lfloor\frac{\min(m,n)}{N}\right\rfloor, where N=3\mathbf{N=3} is the number of experts. The effective number retained from expert ii was therefore$\min\!\left(r_i,r_{\mathrm{budget}}\right).$ Because every LoRA expert had rank ri=32\mathbf{r_i=32} and the automatic budget exceeded 32\mathbf{32} for all merged matrices, all 32\mathbf{32} components from every expert were retained. Consequently, each merged update contained 3×32=96\mathbf{3\times32=96} singular components and had rank at most 96\mathbf{96}.

  • TIES-Merging. We used the standard three-stage Trim–Elect–Merge procedure:

    1. Trim density =0.2=0.2: for each expert and each target weight matrix, only the 20% largest-magnitude entries of Δi\Delta_i were retained; the remaining entries were set to zero.
    2. Mass-based sign election: at each parameter coordinate, the elected sign was sj=sign⁡!(∑iΔ~i,j)s_j=\operatorname{sign}\\!\left(\sum_i\widetilde{\Delta}_{i,j}\right), where Δ~i\widetilde{\Delta}_i denotes the trimmed update. Hence, the sign was determined by the total signed magnitude rather than simply by counting positive and negative experts.
    3. Disjoint-mean aggregation: after electing sjs_j, updates whose signs disagreed with sjs_j were discarded, and the remaining nonzero, sign-consistent values were averaged at that coordinate.

    Finally, the TIES update was added to the base model with a merge scale of 1.0\mathbf{1.0}.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Reply to All Reviewers

We sincerely thank the reviewers for their positive feedback and constructive suggestions. We have carefully addressed each comment by providing detailed responses and conducting additional experiments where necessary. Furthermore, we have resubmitted the revised manuscript, with all modifications and newly added content highlighted in blue for ease of reference. We welcome any further questions or comments from the reviewers and would be happy to engage in additional discussions.

Authors of paper 17054

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Responses

Dear Reviewer ZMcL, bVfh, AM2s,

We sincerely thank you for your valuable time and constructive feedback. We have carefully addressed all the concerns raised in your reviews, provided detailed point-by-point responses, and submitted a revised manuscript with additional experimental results and clarifications.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could take a moment to review our responses and provide any follow-up feedback before the discussion closes. In particular, we would be grateful to know whether our responses and additional experiments have adequately addressed your concerns.

If any questions or concerns remain, we would be more than happy to provide further clarifications promptly.

Thank you again for your time and consideration.

Authors of paper 17054

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.