acceptodds
Under review as a conference paper at ICLR 2027

LEMMA: Learned Model Merging for Text-to-Image Generation via Multi-Objective Optimization

Abstract

Merging specialist text-to-image models into a unified generator can consolidate diverse generative capabilities while minimizing deployment overhead. However, existing methods rely on static, global heuristics that ignore layer-specific functionality and fail to balance multiple objectives. To address this, we propose LEMMA, a novel framework that formulates text-to-image model merging as a multi-objective optimization problem. LEMMA adaptively combines specialist task vectors through layer-tailored merging coefficients that are learned directly on the target objectives, while the base and specialist models remain frozen. We demonstrate LEMMA on two foundation architectures (SDXL and SD3) across two multi-objective setups, by merging reward specialists under multi-objective Direct Preference Optimization (DPO), and by merging style specialists under multi-objective Supervised Fine-Tuning (SFT). Extensive experiments show that LEMMA consistently outperforms prior merging baselines across both setups. Importantly, LEMMA often surpasses dedicated single-task specialists on their own target metrics, improving all objectives simultaneously rather than trading one capability for another. These gains are generalized to unobserved evaluation metrics and are corroborated by a user study with human raters, all while introducing no additional inference overhead.

Primary area:
generative models
Keywords:
model merging, text-to-image generation, diffusion models, multi-objective, direct preference optimization, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HFhHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LEMMA for consolidating specialist text-to-image models into a single generator. Rather than applying a global merging weight, it learns layer-specific coefficients for combining specialist task vectors while keeping the base and specialist weights frozen. These coefficients are optimized through multi-objective DPO for reward specialists or pooled-data SFT for style specialists, without adding inference overhead after merging. Experiments with SDXL and SD3 on PartiPrompts and WikiArt report improvements over merging baselines and some individual specialists, complemented by merge-granularity ablations and human preference evaluations.

Critical strengths
  1. LEMMA offers a simple, parameter-efficient approach to consolidating specialist text-to-image models. By optimizing only layer-specific merging coefficients, it accommodates both reward alignment through DPO and multi-style generation through SFT. The coefficients can then be folded into a single checkpoint without additional inference overhead, making the framework practically useful for reusing existing specialists.
  2. The empirical gains are supported by complementary forms of evaluation. On SDXL, improvements extend beyond the training rewards to held-out metrics, while the blind user study favors LEMMA over TIES in both the DPO and SFT settings tested. The merge-granularity ablations further support the benefit of finer-grained coefficients over a single global weight per specialist.
Critical weaknesses
  1. The methodological contribution appears incremental relative to existing learnable merging methods. Equation (9) uses the same layer-wise task-vector combination as AdaMerging, while ZipLoRA already learns merging coefficients for text-to-image diffusion models with frozen source weights. Although ZipLoRA addresses subject–style LoRAs rather than full-checkpoint reward specialists, it contradicts the paper’s broader positioning of learnable merging as confined to discriminative or autoregressive models. LEMMA’s main distinction is applying DPO and pooled-style SFT to optimize these coefficients, with reward calibration and frontier-based pair selection inherited from CaPO. This is a potentially useful extension, but the paper does not sufficiently distinguish a new merging mechanism from applying established coefficient optimization to different training objectives. The first-of-its-kind framing therefore overstates the methodological advance.
  2. The evaluation does not sufficiently establish the claimed multi-objective advantage. The main merging baselines are static methods, whereas LEMMA receives additional DPO or SFT supervision. The Mixture baseline and merge-granularity ablations are useful, but they do not establish an advantage over competitive multi-reward optimization such as CaPO under comparable data and training budgets. The claim that LEMMA resolves objective conflicts is also stronger than the supporting analysis: Section 4.2 infers that Pareto-dominant training pairs make the gradient improve all rewards jointly, although optimizing their relative likelihood does not ensure improvement in every expected reward. The results likewise require a clear distinction between improving over the pretrained model and retaining specialist performance: LEMMA’s SDXL VILA score is 6.13 versus the specialist’s 6.21, while its SD3 PickScore is 22.67 versus TIES’s 22.70. These exceptions do not negate the reported gains, but stronger comparisons and a more precise treatment of the remaining trade-offs are needed to support the central multi-objective claim.
Questions for authors
  1. Novelty beyond existing learnable merging. Could the authors clarify what is new in LEMMA’s merging mechanism beyond AdaMerging’s layer-wise task-vector parameterization and ZipLoRA’s learned diffusion-model merging? In particular, is the main contribution the use of DPO/SFT supervision for full-checkpoint specialists, or is there an additional methodological distinction that is not clear from the manuscript?
  2. Comparisons under matched supervision. How does LEMMA compare with CaPO using the same backbone, preference data, and reward models under comparable optimization budgets? Beyond the existing granularity ablations, what evidence distinguishes the benefit of LEMMA’s merging design from the additional target-objective training unavailable to the static merging baselines? Please distinguish the resources required for merge optimization from those used to train the specialists.
  3. Justification of joint reward improvement. What justifies the statement in Section 4.2 that Pareto-dominant training pairs cause the gradient to reinforce all rewards jointly? Dominance between two sampled images does not by itself determine how every expected reward changes after updating the generator. Is this statement intended as a formal property under additional assumptions, or as an empirical motivation for the preference objective?
  4. Interpretation of the remaining trade-offs. Does improvement “without trading one capability for another” refer to the pretrained model, the individual specialists, or competing merged models? How should the lower SDXL VILA score relative to its specialist (6.13 versus 6.21) and lower SD3 PickScore relative to TIES (22.67 versus 22.70) be interpreted? Clarifying which differences are statistically meaningful, and distinguishing target-reward trade-offs from variation in held-out metrics, would help establish the scope of this claim.
8 comments

We sincerely thank the reviewer for the positive evaluation of our paper, including recognizing LEMMA's practicality and the breadth of our evaluation, and for the careful questions. We address each of them below.

AuthorsOpenReview ↗

1. Our main contribution is learning the merge of full-checkpoint specialists on the generator's own DPO or SFT objective in text-to-image models, with signed, unbounded coefficients.

LEMMA learns its merge in a fundamentally different way from AdaMerging (Yang et al., ICLR 2024) and ZipLoRA (Shah et al., ECCV 2024), where LEMMA optimizes the generator's own multi-objective DPO or SFT loss in a text-to-image setting, with signed, unbounded coefficients learned on training data only. A shared component with AdaMerging is Eq. 9, layer-wise merge parameterization, which we will credit explicitly in the final version. We summarize the differences below:

  • vs AdaMerging. AdaMerging is designed for discriminative models, where it learns layer-wise coefficients, initialized at 0.3 and clamped to [0, 1], by minimizing the entropy of class predictions on the unlabeled test data. Beyond the shared parameterization, LEMMA differs in three ways: (i) it optimizes the generator's own multi-objective DPO or SFT loss instead of a classification surrogate, (ii) its coefficients are signed and unbounded, so a layer can subtract a specialist's update to remove conflicting skills, and (iii) it learns the merge once on training data and never sees the test inputs, whereas AdaMerging performs the merge on the unlabeled test samples.
  • vs ZipLoRA. ZipLoRA merges two LoRAs (one subject, one style) with column-wise coefficients, trained to reconstruct each source LoRA's outputs. LEMMA merges any number of full-checkpoint specialists on the target multi-objective, and the merge by LEMMA can exceed the specialists on their own rewards (SDXL: MPS 11.24 vs 10.99; VQAScore 0.8148 vs 0.8050).
  • vs CaPO. CaPO is not a merging method. It fine-tunes the full model on multi-reward preferences, producing a multi-objective specialist. It is therefore a baseline for LEMMA rather than a competing merge method. Our Mixture baseline in the original submission follows its "DPO + FRS" variant, and LEMMA outperforms it (please see below, reported in point 2). LEMMA instead reuses existing specialists and trains only 1,680 scalars per specialist on SDXL (490 on SD3). Reward calibration and frontier-based pair selection come from CaPO, as Sec. 4.2 states ("Following CaPO (Lee et al., 2025)"), which we do not claim.
AuthorsOpenReview ↗

2. LEMMA outperforms CaPO and its gains come from the merge design.

Below, we first compare LEMMA with CaPO directly, then explain that its gains come from the merge design, and finally show budget comparisons.

  • (a) Comparison with CaPO. First, CaPO is not a merging method. It fine-tunes the pretrained model on multi-reward preferences. LEMMA instead starts from the specialists. Each specialist holds knowledge unique to its objective on top of knowledge shared with the other specialists, and LEMMA's merge combines both. Meanwhile, CaPO, starting from the pretrained model, cannot use the specialists unique knowledge. Please note that the Mixture baseline in the original submission follows CaPO's "DPO + FRS" variant [Lee et al., 2025, Tab. 2]: the pretrained model fine-tuned with the DPO loss and frontier-based pair selection on the same backbone, data and rewards. LEMMA outperforms it on every metric of Tables 1-2 (HZ-WS 9.37 vs 2.81 on SDXL and 4.35 vs 0.13 on SD3).
  • (b) Advantage over static merges. Static merge methods (e.g., Task Arithmetic, TIES) combine the specialists with a few global coefficients. LEMMA’s advantage over them does not only come from training on the objective, but also it remains when the alternatives receive the same target supervision. (i) Task Arithmetic and TIES with their coefficients carefully tuned on the validation set: LEMMA still scores higher than both (SDXL HZ-WS 9.37 vs 5.64 and 4.89; SD3 HZ-WS 4.35 vs 2.60 and 3.15; SFT LPIPS 0.6765 vs 0.6812 and 0.6920). (ii) The coarse variant of our granularity ablation (Tables 5-6), i.e., Task Arithmetic with one weight per specialist learned with exactly LEMMA’s objective, data, and steps. Here, full-depth LEMMA beats it on SDXL (+7.99 HZ-WS) and on SFT (LPIPS 0.6765 vs 0.6842; CLIP 0.8116 vs 0.8052), and also beats the intermediate variant with one weight per top-level module (+4.18 HZ-WS; LPIPS 0.6765 vs. 0.6818, CLIP 0.8116 vs. 0.8039). (iii) The adapted AdaMerging (point 1, reported above), learned layer-wise with the same training but with coefficients restricted to [0, 1] as in its paper: LEMMA achieves improved performance compared to it. Therefore, target supervision alone does not explain the gains. LEMMA's merge formulation with layer-wise, signed coefficients with use of unique knowledge from the specialists do.
  • (c) Resources. On SDXL, LEMMA learns its coefficients from 160k preference pairs (5,000 updates of batch 32), a quarter of the 640k pairs used to train the multi-objective model or each specialist (2,500 updates of batch 256); on SD3, it uses 80k for LEMMA vs 320k for multi-objective model or each specialist. Since specialists are trained once and reused, LEMMA is the cheaper option whenever they already exist, which is the setting it targets.
Reviewer_o18ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LEMMA freezes the base and specialist text-to-image models and learns only a small per-specialist, per-layer merge-coefficient matrix via gradient descent under multi-objective DPO or SFT, producing a single merged model that outperforms existing merging methods with no added inference cost.

Critical strengths

The method is simple, well-motivated, and practical. It learns only a tiny per-specialist, per-layer coefficient matrix with all models frozen, yields a standard checkpoint with no inference overhead, and applies to both multi-objective DPO and SFT. This makes it an easily reproducible and deployable alternative to static, depth-agnostic merging heuristics in text-to-image generation.

Critical weaknesses

The central claim is that learned, layer-wise coefficients are what drive the gains, yet the paper does not compare against AdaMerging (or a layer-wise variant), the most directly relevant learnable baseline. All merging baselines are training-free, whereas LEMMA is optimized on target data with validation-tuned hyperparameters (e.g., lr=0.1, β=3000 on SDXL), and it is unclear whether baselines such as Task Arithmetic received comparable tuning. Hence the benefit of layer-wise learning cannot be separated from the benefit of extra data and optimization, which undermines the main contribution.

Questions for authors
  1. How does LEMMA differ from AdaMerging (and a layer-wise variant)? Could you add it as a baseline under the same data and compute budget?
  2. Are the main results from a single run or averaged over seeds? Please report means and standard deviations, and describe the significance tests (unit of analysis, sample size, multiple-comparison correction).
  3. Given the tie on VQAScore and slightly lower PickScore than TIES, can you provide confidence intervals and tone down “best on nearly all metrics”?
  4. What are the wall-clock time, memory, and GPU count for learning A, compared with training a specialist or the Mixture baseline?
  5. Could the authors visualize the learned α across layers and specialists, and discuss whether the patterns (e.g., which specialist dominates where, where negative values appear) match the claim that capabilities are localized in different layers?
8 comments

We sincerely thank the reviewer for the precise and constructive review. Following the reviewer’s questions, we added AdaMerging as a baseline, tuned the static baselines, and addressed each question below.

AuthorsOpenReview ↗

1. LEMMA differs from AdaMerging in its objective, its coefficients and its data, and outperforms AdaMerging under the same data and step budget.

We thank the reviewer for raising this question and suggesting this baseline.

  • Differences. AdaMerging is designed for discriminative models, where it learns layer-wise coefficients, initialized at 0.3 and clamped to [0, 1], by minimizing the entropy of class predictions on the unlabeled test data. LEMMA shares the layer-wise parameterization (Eq. 9) but differs in three ways: (i) it optimizes the generator's own multi-objective DPO or SFT loss instead of a classification surrogate, (ii) its coefficients are signed and unbounded, so a layer can subtract a specialist's update to remove conflicting skills, and (iii) it learns the merge once on training data and never sees the test inputs, whereas AdaMerging performs the merge on the unlabeled test samples.
  • AdaMerging cannot be applied to image generation as is. A diffusion model outputs no class distribution, so AdaMerging's entropy objective is undefined. This is why the original submission had not included it. That said, in this rebuttal, we adapted AdaMerging for text-to-image generation.
  • AdaMerging (adapted). We keep AdaMerging's layer-wise parameterization, 0.3 initialization and [0, 1] clamping, but replace its entropy objective, which is undefined for diffusion models, with LEMMA's own objective, data and step budget. Note that this adapted baseline is no longer prior work. It already includes LEMMA's main contribution, learning the merge from the generator's own DPO or SFT objective. Comparing against it therefore isolates the remaining difference, LEMMA's signed, unbounded coefficients.

DPO:

MethodMPSVILAVQAScoreHPSv2.1PickScoreImageRewardSciScore
AdaMerging (adapted)11.076.110.807528.8322.470.87934.18
LEMMA11.246.130.814829.5422.580.97934.46

SFT, averaged over the four styles:

MethodLPIPS ↓CLIP ↑
AdaMerging (adapted)0.68100.8086
LEMMA0.67650.8116

In both tables, LEMMA outperforms the adapted AdaMerging on every metric, which shows the benefit of signed, unbounded coefficients.

AuthorsOpenReview ↗

2. Task Arithmetic and TIES use the settings recommended by their papers. Tuning them on the validation set helps little, and LEMMA still outperforms them.

We thank the reviewer for this suggestion. The submission used the settings recommended by their papers: top-20% trimming and scale 1.0 for TIES [Yadav et al., 2023] and λ=0.5\lambda = 0.5 for Task Arithmetic [Ilharco et al., 2023]. Following the reviewer's suggestion, we tuned both on the validation set (Task Arithmetic: λ∈[0.2,1.25]\lambda \in [0.2, 1.25], including uniform averaging λ=1/K\lambda = 1/K; TIES: top-k∈{0.1,0.2,0.5}k \in \{0.1, 0.2, 0.5\} ×\times scale ∈{0.5,1.0,1.5}\in \{0.5, 1.0, 1.5\}):

Setting (selection metric)Task Arithmetic (default → tuned)TIES (default → tuned)LEMMA
SDXL DPO (HZ-WS ↑)4.85 → 5.644.86 → 4.899.37
SD3 DPO (HZ-WS ↑)3.85 → 2.604.07 → 3.154.35
SD3 SFT (LPIPS ↓ / CLIP ↑)0.6884 / 0.8061 → 0.6812 / 0.81020.6885 / 0.8063 → 0.6920 / 0.81010.6765 / 0.8116

Validation tuning changes both baselines very little: it leads to only small changes on SDXL DPO and SD3 SFT, and on SD3 DPO the validation-selected settings score lower on the test set than the defaults. Across all three settings, LEMMA outperforms both the default and tuned versions.

Reviewer_XsaQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an optimization scheme for weighted average of image generation models that have been tuned for specific reward such as to benefit from all the post training. The com bination has weights specific to layers and specialized models and is optimized with either multi-objective DPO or SFT. The combination is able to retrain the strong characteristics of each specialized model.

Critical strengths

The proposed method looks simple enough and with a low parameter count. It also gets rid of the storage needed for all the specialized models.

Critical weaknesses

The experiments are underwhelming. None of the specialized models are significantly stronger than the baseline, so there is little hope than a combination can be impressive (with this being not as strong on the SFT dataset). A multi-objective "specialist" is missing from the experiments whereas it is the better comparison. Also, 3 rewards is not much, and I don't really see the point of having held out rewards as the multi-objective should prevent reward hacking anyway. I think having a set of 6 reward (HPSv2, PickScore and ImageReward on top of the proposed ones, or maybe SciScore to have something more orthogonal) would be more convincing.

Questions for authors

Why are the specialized models not better? On DPO with SDXL, the training on MPS gets better scores at VQA than training on VQA. That sounds like a bug to me given the nature of DPO. What is the score of a full model training with the multi-objectve rather than the merging? What is it's a LoRA instead of the full model (which can be too much)?

8 comments

We sincerely thank the reviewer for the careful reading and the concrete suggestions, which we addressed point by point below.

AuthorsOpenReview ↗

1. The specialists improve over the pretrained model, by margins typical for these rewards, as indicated by prior work. LEMMA achieves beyond them.

Summary of the gains on the multi-objective DPO setup:

On DPO, all six specialist gains (three metrics ×\times two backbones) are positive. MPS and VILA are highly significant (p<10−7p \lt 10^{-7}). VQAScore is marginally significant (p=0.05p = 0.05 for SDXL, p=0.09p = 0.09 for SD3), since VQAScore is near its ceiling (VQAScore range is 0-1), where the pretrained models already reach a median of 0.90 on SDXL and 0.94 on SD3.

Target rewardSDXL: pretrained → specialistSDXL: gain [95% Confidence Interval], p-valueSD3: pretrained → specialistSD3: gain [95% Confidence Interval], p-value
MPS10.87 → 10.99+0.12 [0.08, 0.16], p<10−7p \lt 10^{-7}11.49 → 11.70+0.20 [0.16, 0.25], p<10−10p \lt 10^{-10}
VILA5.92 → 6.21+0.29 [0.27, 0.31], p<10−10p \lt 10^{-10}5.84 → 5.98+0.14 [0.12, 0.15], p<10−10p \lt 10^{-10}
VQAScore0.8001 → 0.8050+0.0049 [−0.0001, 0.0097], p=0.05p = 0.050.8406 → 0.8460+0.0054 [−0.0007, 0.0118], p=0.09p = 0.09

Please note that gains of this size are typical for these metrics and models.

  • Diffusion Soup (Biggs et al., ECCV 2024, Table 1): Small specialist gains don't stop the model merging from being the best model. Specialists change TIFA by at most ±0.5 points (85.5 → 85.0-86.0) and CLIP by at most 0.001 (0.259 → 0.260), yet their uniform soup is the best model on every metric.
  • CaPO (Lee et al., CVPR 2025, Table 2), with the same three rewards as ours: DPO specialist (denoted as SUM) improves MPS by +0.18, VILA by +0.24, and VQAScore by +0.003 on SDXL, and by +0.11, +0.039, and +0.002 on SD3.

Our specialist gains are in the same range, so they do not limit the model merge.

On SDXL, LEMMA improves all three rewards over the pretrained model (MPS +0.37, VILA +0.21, VQAScore +0.015). Its MPS and VQAScore gains are about 3×3\times the specialist gains, so it surpasses them on their own rewards, and it retains 74% of the VILA specialist gain. On SD3, it surpasses the MPS specialist and matches the other two. It has the highest HZ-WS on both backbones across all competing methods: 9.37 vs 4.86 for TIES on SDXL, and 4.35 vs 4.07 on SD3.

Summary of the gains on the multi-objective SFT setup:

Meanwhile, on the SFT setup, the specialists improve even more clearly. In every style, the specialist improves both LPIPS and CLIP over the pretrained model, by 4.1% in LPIPS and 2.1% in CLIP on average (up to 5.9% and 3.4%) (Table 3; lower LPIPS and higher CLIP are better).

Target styleLPIPS ↓: pretrained → specialistCLIP ↑: pretrained → specialist
Baroque0.7029 → 0.66150.7770 → 0.8037
Expressionism0.7288 → 0.70530.7780 → 0.7873
Impressionism0.7085 → 0.68440.8102 → 0.8265
Realism0.6962 → 0.66800.7878 → 0.8031
Average0.7091 → 0.67980.7883 → 0.8052

LEMMA again has the best LPIPS and CLIP of all methods in Table 3 (0.6765 / 0.8116), beating TIES and even the four specialists, each evaluated only on its own style. Also, blind raters prefer LEMMA over TIES (60.7±7.760.7 \pm 7.7%) and the pretrained model (83.3±9.283.3 \pm 9.2%).

Specialist gains of this size therefore do not limit the merge: LEMMA combines them into a single model that is stronger overall than every baseline and surpasses the specialists on most of their own targets.

AuthorsOpenReview ↗

2. The MPS and VQAScore specialists tie on VQAScore on SDXL, which is expected, not a bug.

The MPS and VQAScore specialists on SDXL are statistically indistinguishable (0.8053 vs 0.8050; difference +0.0003, CI [−0.005, +0.006], p=0.91p = 0.91), so the MPS specialist is not better on VQAScore. A tie is expected for two reasons: MPS (Zhang et al., CVPR 2024) is trained on human preferences that include semantic alignment with the prompt, so optimizing MPS also improves alignment; and VQAScore is near its ceiling (range 0-1), which keeps both gains small.

Consistent with this, on SD3 the VQAScore specialist does score highest on VQAScore (0.8460 vs 0.8423).

Finally, VQAScore improves most when the specialists are merged by LEMMA: +0.015 over the pretrained model on SDXL, three times the VQAScore specialist's gain and nearly twice that of any other merge in Table 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.