acceptodds
Under review as a conference paper at ICLR 2027

OptIMerge: Optimization-Based Merging of Diffusion Models

Abstract

The arithmetic merging of weights from multiple task-specific models fine-tuned from a shared base model, commonly known as model souping, has emerged as a popular strategy for multi-task learning without the computational overhead of joint retraining. However, naive scalar weight averaging frequently leads to severe performance degradation when the constituent models learn structurally divergent feature representations, leading to destructive parameter interference. We study this problem in a largely unexplored setting: merging image-to-image (I2I) diffusion models fine-tuned for dense, pixel-accurate tasks such as image restoration. Unlike the large language models (LLMs) and text-to-image (T2I) diffusion models for which most merging methods were developed, I2I experts encode conflicting spatial priors (e.g., anisotropic motion-blur vs. isotropic defocus-blur) in their convolutional filters, where both naive soup and LLM-style element-wise merging introduce visible artifacts. We introduce OptIMerge, a gradient-based optimization framework: instead of relying on heuristic scalars, OptIMerge learns highly expressive, yet parameter-efficient merger coefficients by minimizing a multi-task diffusion loss over a small set of 100–500 images. OptIMerge parameterizes the merge with architecture-aware transformations: for attention layers, a structured low-rank feature alignment with explicit channel biases; for convolutional layers, per-kernel merger coefficients, and a low-rank Cayley transform that aligns channels by orthogonal rotation without distorting the filters. By keeping the experts frozen and optimizing only these lightweight merger parameters (equivalent to < 0.4% of the network footprint), OptIMerge achieves simultaneous multi-task capabilities that rival computationally expensive joint training and decisively outperform zero-shot model soups.

Primary area:
generative models
Keywords:
diffusion models, model merge, image generative models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_d8JDOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents OptIMerge, an optimization-based framework for merging task-specific diffusion models. It learns lightweight merging parameters with architecture-aware designs, including biased LoRA-based attention alignment and Cayley-transform-based convolutional feature alignment, to reduce parameter interference between experts. The method is evaluated on image restoration model merging and personalized T2I model merging experiments.

Critical strengths
  • The paper raises an interesting perspective on model merging for image restoration diffusion models.
  • The paper is clearly written and well presented.
  • Proposes architecture-aware merging strategies for attention and convolution modules.
  • Provides extensive experiments and ablations on image restoration and personalized T2I merging tasks.
  • Achieves parameter-efficient merging by optimizing only lightweight merging parameters.
Critical weaknesses
  • The motivation for applying model merging to image restoration is not sufficiently discussed compared with existing merging studies in LLMs and image generation. The necessity of merging independently trained restoration experts instead of directly performing joint multi-task training remains unclear.
  • The generalization and scalability of OptIMerge are not sufficiently demonstrated. The experiments mainly focus on pairwise merging among three restoration tasks, while the performance gap becomes larger in three-expert merging. More evaluations with a broader range of restoration tasks and larger numbers of merged experts are needed.
  • The baseline comparison is not sufficiently convincing. The evaluated baselines mainly consist of fixed 0.5 weight averaging and two zero-shot model merging strategies. However, directly averaging two independently fine-tuned restoration models with a fixed coefficient may be a relatively limited baseline, especially when the models are trained for different restoration tasks with potentially conflicting representations.
  • Some important experimental details are insufficiently reported. In particular, the training schedules and optimization budgets of the joint training baseline and the expert training + merging pipeline are unclear. These details are important for a fair comparison and for understanding whether the improvements come from the proposed merging strategy or from differences in training budgets.
Questions for authors

My main concerns lie in the motivation of model merging for image restoration, its generalization ability, the fairness of baseline comparisons, and the advantages over direct joint multi-task training, especially in terms of training procedures and optimization budgets. If these concerns are properly addressed, I would consider increasing my score.

11 comments
AuthorsOpenReview ↗

Response to Reviewer d8JD

We thank the reviewer for recognizing the novelty, clarity, and parameter efficiency of our architecture-aware merging framework. Below we address your four questions on motivation, generalization/scalability, stronger baselines, and training/optimization budgets.

1. Motivation for model merging for image restoration / I2I

We thank the reviewer for raising this point and will expand this discussion in the revision. The motivation for merging image-to-image (I2I) restoration models is closely aligned with that of merging in the large language models (LLMs) and text-to-image (T2I) generation. In each case, a single large pretrained foundation model is fine-tuned into many task-specific models, and practitioners want to combine the capabilities of these models into one, without pooling all the training data or retraining a multi-task model from scratch. Many such fine-tuned models are shared publicly as checkpoints or LoRAs, often without their training data. The observations that made weight-space merging effective for LLMs therefore carry over to diffusion-based I2I restoration models. We detail this correspondence below.

Single-task training is common practice. In many practical settings, image restoration models are trained for a single degradation type, such as motion deblurring, defocus deblurring, denoising or super-resolution. Each such model is optimized on its own task-specific paired dataset. Supporting several degradations therefore usually requires either maintaining several specialist models or training a single multi-task model from scratch.

Restoration experts share a common pretrained initialization. Modern diffusion-based restoration models are typically obtained by fine-tuning a large pretrained text-to-image diffusion model on the restoration task, instead of training from random initialization. This mirrors the prevailing practice for large language models, where task-specific models are fine-tuned from a common foundation model.

It is precisely this shared-initialization regime that motivates weight-space merging. Models fine-tuned from the same pretrained checkpoint tend to remain in a common region of parameter space, which allows their weights to be combined meaningfully [Model Soups, Task Arithmetic, TIES, DARE, RegMean]. The same property has been exploited for text-to-image personalization [Mix-of-Show, ZipLoRA, Diffusion Soup]. Our restoration experts follow exactly this protocol: each is fine-tuned from the same pretrained text-to-image checkpoint. Merging is therefore a natural way to consolidate them into a single multi-task model, for the same reasons it is used to consolidate fine-tuned LLMs.

Merging avoids the costs of joint training. Joint multi-task training is the principal alternative, but it has three practical drawbacks:

  • It requires simultaneous access to the training data of every task.
  • It must be repeated whenever a task is added or an expert is updated.
  • It must contend with interference between degradations, which all-in-one restoration methods typically mitigate with degradation-aware prompts or conditioning [AirNet, PromptIR, DA-CLIP].

In contrast, OptIMerge requires only the fine-tuned experts and 100-500 calibration pairs per task. It costs approximately 14-16 TPU chip-hours, compared to approximately 159 chip-hours for a single joint retraining.

2. Additional Results on denoising task for Image Restoration

To demonstrate generalization beyond deblurring and super-resolution to a restoration task with an opposing high-frequency prior (noise removal vs. detail sharpening), we trained a 4th restoration expert for denoising task. We follow the exact same training and optimization setup as we followed to train our other experts. We use 1024×10241024\times1024 images from the Smartphone Image Denoising Dataset (SIDD) [Abdelhamed et al., 2018]. We merged this SIDD expert with OID motion deblurring (SIDD + OID) and with DF2K super-resolution (SIDD + SR), comparing against Naive Soup, Joint multi-task training, and parameter-matched LoRA on Soup (3.36M params) under the exact same training setup and calibration data. OptIMerge outperforms souping baselines on most of the metrics.

Abdelhamed, Abdelrahman, Stephen Lin, and Michael S. Brown. "A high-quality denoising dataset for smartphone cameras." 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018.

Results on SIDD + OID

ModelTrainable ParamsSIDD PSNR ↑\uparrowSIDD SSIM ↑\uparrowSIDD LPIPS ↓\downarrowOID PSNR ↑\uparrowOID SSIM ↑\uparrowOID LPIPS ↓\downarrow
Experts and joint training
SIDD Expert—35.580.91080.0861———
OID Expert————22.920.62660.2113
Joint Multi-Task890M35.270.90510.080722.580.61930.2174
Souping methods
Naive Soup032.440.88790.132421.040.58180.4799
LoRA on Soup3.36M33.140.89010.100221.300.56700.2751
OptIMerge (Ours)3.19M34.940.90370.083422.660.61760.2208

Results on SIDD + SR

ModelTrainable ParamsSIDD PSNR ↑\uparrowSIDD SSIM ↑\uparrowSIDD LPIPS ↓\downarrowSR (DF2K) PSNR ↑\uparrowSR (DF2K) SSIM ↑\uparrowSR (DF2K) LPIPS ↓\downarrow
Experts and joint training
SIDD Expert—35.580.91080.0861———
SR Expert————21.260.53230.3131
Joint Multi-Task890M33.500.89950.080821.370.53720.3059
Souping methods
Naive Soup032.400.89730.143920.400.55650.5330
LoRA on Soup3.36M32.890.88660.106520.730.51220.3266
OptIMerge (Ours)3.19M33.590.89640.088621.090.53090.2688

3. Additional optimization-based baselines under the same calibration data

We implemented four additional optimization-based baselines under the same calibration data: a learned global scalar, a parameter-matched LoRA on model soup, AdaMerging, and RegMean. OptIMerge outperforms all four baselines, as shown in the tables below. Please refer to Section A of the Global Response for more details on these baselines.

ModelOID PSNR ↑\uparrowOID SSIM ↑\uparrowOID LPIPS ↓\downarrowDPDD PSNR ↑\uparrowDPDD SSIM ↑\uparrowDPDD LPIPS ↓\downarrow
Experts and joint training
OID Expert22.920.62660.2113---
DPDD Expert---23.390.63920.2023
Joint expert22.590.61730.215423.370.64040.2091
Souping methods
Naive soup20.880.56010.289923.150.63980.2641
Learned global scalar22.070.60320.231023.170.64330.3223
LoRA on soup21.990.59210.249823.090.62680.2534
RegMean21.200.57120.269523.300.64150.2316
Layer-wise AdaMerging22.340.59600.243422.910.61660.3042
OptIMerge22.720.61900.216823.910.65610.2191
ModelOID PSNR ↑\uparrowOID SSIM ↑\uparrowOID LPIPS ↓\downarrowSR PSNR ↑\uparrowSR SSIM ↑\uparrowSR LPIPS ↓\downarrow
Experts and joint training
OID Expert22.920.62660.2113---
SR Expert---21.260.53230.3131
Joint22.030.58670.238621.490.54070.2993
Souping methods
Naive soup20.310.52920.293820.990.49700.3501
Learned global scalar20.560.54000.283220.950.48050.3736
LoRA on soup20.870.5510.29420.870.49790.3619
Layer-wise AdaMerging21.660.57780.249220.870.48060.3860
OptIMerge22.160.59800.229821.450.53380.3207
ModelDPDD PSNR ↑\uparrowDPDD SSIM ↑\uparrowDPDD LPIPS ↓\downarrowSR PSNR ↑\uparrowSR SSIM ↑\uparrowSR LPIPS ↓\downarrow
Experts and joint training
DPDD Expert23.390.63920.2023---
SR Expert---21.260.53230.3131
Joint23.110.62930.218521.240.52550.316
Souping methods
Naive soup22.570.61210.277421.100.52590.3447
Learned global scalar22.730.61730.262321.010.5160.3608
LoRA on soup22.690.60130.299520.350.49270.3739
Layer-wise AdaMerging22.220.59920.279720.660.48990.3783
OptIMerge23.130.62580.236521.120.52310.3210
Reviewer_krnaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper merges I2I diffusion restoration experts by learning merge coefficients on 100–500 calibration images with the experts frozen. Attention layers use a biased-LoRA mask plus an alignment adapter, and conv layers use a low-rank Cayley rotation (about 3.2M trainable parameters). It beats soup, TIES and DARE on three restoration pairs and is roughly on par with joint training.

Critical strengths

The I2I merging setting is under-explored, and the method consistently closes most of the gap between the soup and the experts. Treating conv kernels as atomic units is a sensible design.

Critical weaknesses

The method trains on in-domain data for 10k–15k steps but is compared only with zero-shot merges. A same-size LoRA trained on the soup and AdaMerging/RegMean are missing, and the merged model even beats the DPDD expert (23.91 vs. 23.39 dB), so the gain may just be extra fine-tuning. The Cayley rotation is applied to W_B's output channels without compensating the next layer, so it changes model B rather than aligning it, and the "matches joint training" claim is not supported by Tab. 1(b)(c).

Questions for authors

How does a 3.2M-parameter LoRA trained on the soup with the same data and steps perform? Are the calibration, validation and test sets disjoint, and what are the training time and memory?

2 comments

We thank the reviewer for the constructive review. Below we address each concern with new controlled experiments and clarifications.

1. Parameter-Matched LoRA on Soup, AdaMerging, and RegMean

We implemented four additional optimization-based baselines.

  • Learned Global Scalar: We trained a baseline where we learn a single global model merge parameter.
  • LoRA on model soup: As requested, we implemented a LoRA on model soup with a comparable number of parameters (3.36M parameters) to our method which has 3.19M trainable parameters for all three pairs of datasets on image restoration tasks. We find that OptIMerge outperforms parameter-matched LoRA on Soup on all metrics. This is further evidence that gains are not only from calibration data.
  • AdaMerging and RegMean: As requested, we implemented AdaMerging and RegMean. We note that neither the AdaMerging nor the RegMean was designed for our setting of merging I2I generative U-net models. AdaMerging [Yang et al., 2024] is specifically designed for classification models as it learns merging coefficients by minimizing prediction entropy, which requires a classifier's categorical output. RegMean [Jin et al., 2023] merges only linear layers in closed form and was developed for Transformer language models. Our U-Net contains conv layers in various blocks, and some of the conflict between experts can also be attributed to convolution layers that RegMean does not cover. We nevertheless implement both methods with reasonable adaptations, and run them on our setup using the same calibration data as OptIMerge. For AdaMerging, we replace the entropy loss with the diffusion training loss and run two variants: learning one global coefficient, and learning one coefficient per layer. For RegMean, we follow the original recipe: linear and 1×1 conv layers are merged in closed form, and all other parameters are averaged. Its statistics are averaged over diffusion timesteps, and its regularization strength is chosen on calibration data only. For RegMean, we only show results on OID + DPDD.

OptIMerge outperforms all four baselines on image restoration tasks as shown in the tables below.

ModelOID PSNR ↑\uparrowOID SSIM ↑\uparrowOID LPIPS ↓\downarrowDPDD PSNR ↑\uparrowDPDD SSIM ↑\uparrowDPDD LPIPS ↓\downarrow
Experts and joint training
OID Expert22.920.62660.2113---
DPDD Expert---23.390.63920.2023
Joint expert22.590.61730.215423.370.64040.2091
Souping methods
Naive soup20.880.56010.289923.150.63980.2641
Learned global scalar22.070.60320.231023.170.64330.3223
LoRA on soup21.990.59210.249823.090.62680.2534
RegMean21.200.57120.269523.300.64150.2316
Layer-wise AdaMerging22.340.59600.243422.910.61660.3042
OptIMerge22.720.61900.216823.910.65610.2191
ModelOID PSNR ↑\uparrowOID SSIM ↑\uparrowOID LPIPS ↓\downarrowSR PSNR ↑\uparrowSR SSIM ↑\uparrowSR LPIPS ↓\downarrow
Experts and joint training
OID Expert22.920.62660.2113---
SR Expert---21.260.53230.3131
Joint22.030.58670.238621.490.54070.2993
Souping methods
Naive soup20.310.52920.293820.990.49700.3501
Learned global scalar20.560.54000.283220.950.48050.3736
LoRA on soup20.870.5510.29420.870.49790.3619
Layer-wise AdaMerging21.660.57780.249220.870.48060.3860
OptIMerge22.160.59800.229821.450.53380.3207
ModelDPDD PSNR ↑\uparrowDPDD SSIM ↑\uparrowDPDD LPIPS ↓\downarrowSR PSNR ↑\uparrowSR SSIM ↑\uparrowSR LPIPS ↓\downarrow
Experts and joint training
DPDD Expert23.390.63920.2023---
SR Expert---21.260.53230.3131
Joint23.110.62930.218521.240.52550.316
Souping methods
Naive soup22.570.61210.277421.100.52590.3447
Learned global scalar22.730.61730.262321.010.5160.3608
LoRA on soup22.690.60130.299520.350.49270.3739
Layer-wise AdaMerging22.220.59920.279720.660.48990.3783
OptIMerge23.130.62580.236521.120.52310.3210

Based on these results, we conclude that the gains we observe on model soups are not from additional fine-tuning with a few hundreds of optimization steps.

Clarification on "optimization steps". The step counts we report for OptIMerge count micro-steps, i.e. single forward/backward passes on one mini-batch. As mentioned in the Appendix of our manuscript, gradients are accumulated over 64 micro-steps before each optimizer update, so the number of Adam updates is far smaller, the total training run is about 234 updates which corresponds to 15K micro-steps. OptIMerge therefore converges in a few hundred parameter updates of 3.2M coefficients. We will state both counts in the revision and rename "steps" to "micro-steps" where it refers to OptIMerge.

Why does the merged model beat the DPDD expert? This is a one-off case. We hypothesize that merging OID and DPDD with OptIMerge lets the model soup benefit from some priors in OID model to better solve DPDD task. DPDD is a quite small dataset and might get an advantage from using some sort of cross task adaptation.

2. About Cayley Rotation in output channels

Thank you for this interesting question about compensation of the Cayley rotation in the next layer. In our method, all layers are optimized jointly end-to-end, and downstream mixing weights absorb the per-layer channel change without needing an explicit R⊤R^\top. Regardless, we empirically tested your suggestion by applying RR to WB,conv1W _ {B,\text{conv1}}'s output channels and R⊤R^\top to WB,conv2W _ {B,\text{conv2}}'s input channels inside every ResNet block on OID + DPDD. Here, conv1\text{conv1} and conv2\text{conv2} indicate adjacent conv layers in the UNet

  • Compensated Conv-Cayley + Biased LoRA: 22.41 / 0.6054 / 0.2221 (OID) and 23.63 / 0.6428 / 0.2174 (DPDD) vs. uncompensated 22.66 / 0.6165 / 0.2171 and 23.86 / 0.6540 / 0.2186.
  • Compensated Full OptIMerge: 22.77 / 0.6206 / 0.2189 (OID) and 23.85 / 0.6520 / 0.2134 (DPDD) vs. uncompensated 22.72 / 0.6190 / 0.2168 and 23.91 / 0.6561 / 0.2191.

Above, each triplet is PSNR/SSIM/LPIPS. Thus, explicit next-layer compensation brings no empirical gain, which probably happens because downstream mixing weights and non-linearities absorb the per-layer channel transformation.

About qualifying "Matches Joint Training" We agree and will qualify this wording throughout the abstract and text.

Clarification about training, validation, calibration and test set Yes, these datasets are disjoint.

5. Details of Computational time

Compute and training cost. OptIMerge merges the expert models with ~40× fewer optimizer updates and ~10× fewer chip-hours than training a single expert or a joint model. We report the details in a table below.

Stage / MethodChips (topology)Steps (Adam updates)Chip-hoursPeak HBM / chipStorage
Single-Task Expert (OID, DPDD, SIDD)64 TPU v5p10,000 optimization steps15979.8 GB3.21 GB (for each expert model)
Joint Multi-Task (2-way and 3 way)64 TPU v5p10,000 optimization steps153–15979.8 GB3.21 GB
OptIMerge4 TPU v5p234 optimization steps (15,000 microsteps)13.6–16~89 GB12.8 MB
OptIMerge 3-way4 TPU v5p234 optimization steps (15,000 microsteps)1987.7 GB25.5 MB

We hope that we have adequately answered your questions. We are happy to answer any follow up questions.

Reviewer_pqcSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OptIMerge, an optimization-based framework for merging independently fine-tuned diffusion models into a single multi-task model. It introduces architecture-aware parameterizations, including biased LoRA for attention layers and low-rank Cayley transforms for convolutional layers, to mitigate parameter interference through feature alignment and weight mixing. Experiments on image restoration and text-to-image personalization tasks demonstrate improved performance over conventional model merging methods while optimizing fewer than 0.4% of the model parameters

Critical strengths

1. Architecture-Aware Model Merging for Full-Parameter Diffusion Models

Unlike prior approaches such as Diffusion Soup, AdaMerging, and ZipLoRA, which primarily focus on weight averaging, coarse-grained merging coefficients, or LoRA-based merging, respectively, this paper introduces architecture-aware parameterizations that jointly perform weight mixing and feature alignment for full-parameter diffusion U-Nets. In particular, the distinct treatment of attention and convolutional layers through biased LoRA and low-rank Cayley transforms is a meaningful technical contribution, especially for dense image-to-image restoration tasks where spatial structures are important.

2. Promising Empirical Results in Challenging Model Merging Scenarios

The paper demonstrates substantial improvements over conventional model merging approaches across multiple image restoration tasks, including motion deblurring, defocus deblurring, and super-resolution. The proposed method recovers much of the performance degradation caused by naive weight averaging and achieves competitive results with joint multi-task training while optimizing fewer than 0.4% of the model parameters. These results suggest that architecture-aware merging is a promising direction for consolidating specialized diffusion models.

Critical weaknesses

1. Insufficient Evidence for the Effectiveness of Architecture-Aware Merging and Feature Alignment

The main contribution of this paper is the introduction of architecture-aware parameterizations that jointly perform weight mixing and feature alignment for full-parameter diffusion U-Nets. However, it remains unclear whether the performance improvements primarily arise from the proposed architecture-aware design or simply from additional supervised optimization using calibration data.

Although the paper provides ablation studies on different parameterizations, most existing baselines, such as Naive Soup, TIES, and DARE, are zero-shot merging methods. Comparisons with simpler optimization-based alternatives, such as LoRA fine-tuning of the averaged model or optimized layer-wise interpolation under the same calibration data and optimization budget, would help better establish the necessity of the proposed parameterization.

Furthermore, a central motivation of OptIMerge is to resolve feature misalignment between independently fine-tuned models. However, the experiments mainly demonstrate improvements in downstream performance without directly verifying whether the learned transformations actually align feature representations. In particular, the low-rank alignment adapters introduce additional learnable capacity, while the Cayley transformation guarantees orthogonality but does not necessarily imply meaningful feature alignment. It is therefore difficult to determine whether the improvements result from resolving representation misalignment or simply from increased adaptation capacity.

2. Unclear Computational Efficiency of the Proposed Merging Process

The paper emphasizes parameter efficiency by optimizing fewer than 0.4% of the model parameters. However, parameter efficiency does not necessarily translate into computational efficiency. In particular, Appendix A.1 indicates that OptIMerge requires 15,000 optimization steps, which may still incur substantial computational overhead despite the small number of trainable parameters.

The paper does not provide sufficient information about the actual merging cost, such as GPU hours, peak memory usage, or training time relative to joint multi-task training and simpler parameter-efficient adaptation methods. Without such comparisons, it is difficult to assess the practical computational advantages of the proposed approach. Additionally, the optimization budget appears inconsistent between Section 4.1 (approximately 10K steps) and Appendix A.1 (15K steps), which should be clarified.

Questions for authors

Q1. Clarification on the Prediction Objective Used for Optimization

Equation (2) formulates the merging optimization objective using an epsilon-prediction (noise-prediction) loss, whereas Section 4.1 states that the image restoration experiments employ a v-prediction (velocity-prediction) diffusion model. Could the authors clarify which prediction target is actually used when optimizing the merging parameters? If v-prediction is used in the experiments, does the optimization objective differ from Equation (2)?

2 comments
AuthorsOpenReview ↗

Response to Reviewer pqcS

We thank the reviewer for highlighting our strong empirical results across challenging restoration tasks. Below we address each concern with new controlled experiments and clarifications.

1. Results on additional optimization-based baselines under the same calibration data

We implemented and evaluated the requested optimization-based baselines, specifically LoRA fine-tuning of the models soup and optimized layer-wise interpolation. In addition, we also implemented AdaMerging and RegMean. We note that all four optimization baselines were trained with the exact same calibration data, diffusion loss, and step-matched optimization budgets.

  • Learned Global Scalar: We trained a baseline where we learn a single global model merge parameter.
  • LoRA on model soup: We implemented a LoRA on model soup with a comparable number of parameters (3.36M vs 3.19M parameters) to our method for all three pairs of datasets on image restoration tasks. We find that OptIMerge outperforms parameter-matched LoRA on Soup on all metrics. This is further evidence that gains are not only from calibration data.
  • AdaMerging and RegMean: We also implement AdaMerging and RegMean. We note that neither the AdaMerging nor the RegMean was designed for our setting of merging I2I generative U-net models. AdaMerging [Yang et al., 2024] is specifically designed for classification models as it learns merging coefficients by minimizing prediction entropy, which requires a classifier's categorical output. RegMean [Jin et al., 2023] merges only linear layers in closed form and was developed for Transformer language models. Our U-Net contains conv layers in various blocks, and some of the conflict between experts can also be attributed to convolution layers that RegMean does not cover. We nevertheless implement both methods with reasonable adaptations, and run them on our setup using the same calibration data as OptIMerge. For AdaMerging, we replace the entropy loss with the diffusion training loss and run two variants: learning one global coefficient, and learning one coefficient per layer. For RegMean, we follow the original recipe: linear and 1×1 conv layers are merged in closed form, and all other parameters are averaged. Its statistics are averaged over diffusion timesteps, and its regularization strength is chosen on calibration data only. For RegMean, we only show results on OID + DPDD.

OptIMerge outperforms all four baselines on image restoration tasks as shown in the tables below.

ModelOID PSNR ↑\uparrowOID SSIM ↑\uparrowOID LPIPS ↓\downarrowDPDD PSNR ↑\uparrowDPDD SSIM ↑\uparrowDPDD LPIPS ↓\downarrow
Experts and joint training
OID Expert22.920.62660.2113---
DPDD Expert---23.390.63920.2023
Joint expert22.590.61730.215423.370.64040.2091
Souping methods
Naive soup20.880.56010.289923.150.63980.2641
Learned global scalar22.070.60320.231023.170.64330.3223
LoRA on soup21.990.59210.249823.090.62680.2534
RegMean21.200.57120.269523.300.64150.2316
Layer-wise AdaMerging22.340.59600.243422.910.61660.3042
OptIMerge22.720.61900.216823.910.65610.2191
ModelOID PSNR ↑\uparrowOID SSIM ↑\uparrowOID LPIPS ↓\downarrowSR PSNR ↑\uparrowSR SSIM ↑\uparrowSR LPIPS ↓\downarrow
Experts and joint training
OID Expert22.920.62660.2113---
SR Expert---21.260.53230.3131
Joint22.030.58670.238621.490.54070.2993
Souping methods
Naive soup20.310.52920.293820.990.49700.3501
Learned global scalar20.560.54000.283220.950.48050.3736
LoRA on soup20.870.5510.29420.870.49790.3619
Layer-wise AdaMerging21.660.57780.249220.870.48060.3860
OptIMerge22.160.59800.229821.450.53380.3207
ModelDPDD PSNR ↑\uparrowDPDD SSIM ↑\uparrowDPDD LPIPS ↓\downarrowSR PSNR ↑\uparrowSR SSIM ↑\uparrowSR LPIPS ↓\downarrow
Experts and joint training
DPDD Expert23.390.63920.2023---
SR Expert---21.260.53230.3131
Joint23.110.62930.218521.240.52550.316
Souping methods
Naive soup22.570.61210.277421.100.52590.3447
Learned global scalar22.730.61730.262321.010.5160.3608
LoRA on soup22.690.60130.299520.350.49270.3739
Layer-wise AdaMerging22.220.59920.279720.660.48990.3783
OptIMerge23.130.62580.236521.120.52310.3210

2. Computational Efficiency

As requested, we report details of compute and training cost below.

Stage / MethodChips (topology)Steps (Adam updates)Chip-hoursPeak HBM / chipStorage
Single-Task Expert (OID, DPDD, SIDD)64 TPU v5p10,000 optimization steps15979.8 GB3.21 GB (for each expert model)
Joint Multi-Task (2-way and 3 way)64 TPU v5p10,000 optimization steps153–15979.8 GB3.21 GB
OptIMerge4 TPU v5p234 optimization steps (15,000 microsteps)13.6–16~89 GB12.8 MB
OptIMerge 3-way4 TPU v5p234 optimization steps (15,000 microsteps)1987.7 GB25.5 MB

Clarification about 10K (Sec. 4.1) vs. 15K (App. A.1) Steps. We apologize for this confusion. The experts were trained for 20K-50K optimization steps. We then picked the checkpoint with the best performance. In general, we found that this happens at around 10K steps, and therefore report this as the training budget for the expert models.

The step counts we report for OptIMerge in the Appendix count micro-steps, a single forward/backward pass on one mini-batch. They are not optimizer updates. Because our compute budget was limited, we used gradient accumulation to reach an effective batch size of 256. As mentioned in the Appendix of our manuscript, gradients are accumulated over 64 micro-steps before each optimizer update, so the number of actual parameter updates is far smaller: a full OptIMerge run requires only 234 optimization updates (which corresponds to 15K micro-steps). OptIMerge therefore converges in only a few hundred parameter updates of ∼3.2M coefficients. By comparison, training each expert or the joint model requires 10K updates of the full U-Net. We will state both counts in the revision and rename "steps" to "micro-steps" where it refers to OptIMerge.

Prediction Objective in Equation (2): ϵ\epsilon-prediction vs. vv-prediction

Thank you for catching this notation mismatch. In all image restoration experiments both the experts and OptIMerge optimize the exact same vv-prediction MSE objective, minimizing Ez0,ϵ,t[∥vθmerge(ϕ)(zt,t,cimg)−vt∥22]\mathbb{E} _ {z _ 0, \epsilon, t}\big[\Vert v _ {\theta _ {\text{merge}}(\phi)}(z _ t, t, c _ {\text{img}}) - v _ t\Vert _ 2^2\big] where vt=αtϵ−σtz0v _ t = \alpha _ t \epsilon - \sigma _ t z _ 0 and t∼U{1,…,T}t \sim \mathcal{U}\lbrace 1, \dots, T\rbrace. We used ϵ\epsilon-prediction notation, as it is widely used in diffusion models literature and shown to be equivalent to vv-prediction by up to a scaling factor [Salimans and Ho, 2022]. We will add this clarification about prediction objective to the paper.

Salimans, Tim, and Jonathan Ho. "Progressive distillation for fast sampling of diffusion models." arXiv preprint arXiv:2202.00512 (2022).

We hope that we have adequately answered your questions. We are happy to answer any follow up questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

We thank all three reviewers for their constructive feedback. We identified two primary concerns: 1) Additional fair optimization-based baselines under same calibration data, and 2) Clarification about computational cost. Based on the feedback, we ran the requested experiments and performed requested analyses during the rebuttal period. We provide an overview of the new results below. We also include details in the individual responses and will upload a revised PDF with these results.

1(A). Results on additional optimization-based baselines under the same calibration data

We implemented four additional optimization-based baselines.

  1. Learned Global Scalar: We trained a baseline where we learn a single global model merge parameter.
  2. LoRA on model soup: As requested by Reviewers pqcS, krna and d8JD, we implemented a LoRA on model soup with a comparable number of parameters (3.36M parameters) to our method (3.19M parameters) for all three pairs of datasets on image restoration tasks. We find that OptIMerge outperforms parameter-matched LoRA on Soup on all metrics. This is further evidence that gains are not only from calibration data.
  3. AdaMerging and RegMean: As requested by Reviewer krna, we implemented AdaMerging and RegMean. We note that neither the AdaMerging nor the RegMean was designed for our setting of merging I2I generative U-net models. AdaMerging [Yang et al., 2024] is specifically designed for classification models as it learns merging coefficients by minimizing prediction entropy, which requires a classifier's categorical output. RegMean [Jin et al., 2023] merges only linear layers in closed form and was developed for Transformer language models. Our U-Net contains conv layers in various blocks, and some of the conflict between experts can also be attributed to convolution layers that RegMean does not cover. We nevertheless implement both methods with reasonable adaptations, and run them on our setup using the same calibration data as OptIMerge. For AdaMerging, we replace the entropy loss with the diffusion training loss and learn either one global coefficient or one coefficient per layer. For RegMean, we follow the original recipe: linear and 1×1 conv layers are merged in closed form, and all other parameters are averaged. Its statistics are averaged over diffusion timesteps, and its regularization strength is chosen on calibration data only. For RegMean, we only show results on OID + DPDD.

OptIMerge outperforms all four baselines as shown in the table below:

ModelOID PSNR ↑\uparrowOID SSIM ↑\uparrowOID LPIPS$\downarrow$DPDD PSNR ↑\uparrowDPDD SSIM ↑\uparrowDPDD LPIPS ↓\downarrow
OID Expert22.920.62660.2113---
DPDD Expert---23.390.63920.2023
Joint Model22.590.61730.215423.370.64040.2091
Souping methods
Naive soup20.880.56010.289923.150.63980.2641
Learned global scalar22.070.60320.231023.170.64330.3223
LoRA on soup21.990.59210.249823.090.62680.2534
RegMean21.200.57120.269523.300.64150.2316
Layer-wise AdaMerging22.340.59600.243422.910.61660.3042
OptIMerge22.720.61900.216823.910.65610.2191
ModelOID PSNR$\uparrow$OID SSIM$\uparrow$OID LPIPS$\downarrow$SR PSNR$\uparrow$SR SSIM$\uparrow$SR LPIPS$\downarrow$
OID Expert22.920.62660.2113---
SR Expert---21.260.53230.3131
Joint Model22.030.58670.238621.490.54070.2993
Souping methods
Naive soup20.310.52920.293820.990.49700.3501
Learned global scalar20.560.54000.283220.950.48050.3736
LoRA on soup20.870.5510.29420.870.49790.3619
Layer-wise AdaMerging21.660.57780.249220.870.48060.3860
OptIMerge22.160.59800.229821.450.53380.3207
ModelDPDD PSNR$\uparrow$DPDD SSIM$\uparrow$DPDD LPIPS$\downarrow$SR PSNR$\uparrow$SR SSIM$\uparrow$SR LPIPS$\downarrow$
DPDD Expert23.390.63920.2023---
SR Expert---21.260.53230.3131
Joint Model23.110.62930.218521.240.52550.316
Souping methods
Naive soup22.570.61210.277421.100.52590.3447
Learned global scalar22.730.61730.262321.010.5160.3608
LoRA on soup22.690.60130.299520.350.49270.3739
Layer-wise AdaMerging22.220.59920.279720.660.48990.3783
OptIMerge23.130.62580.236521.120.52310.3210
AuthorsOpenReview ↗

Global Response - Pt 2

1(B). Additional results on denoising task for Image Restoration

To demonstrate generalization beyond deblurring and super-resolution to a restoration task with an opposing high-frequency prior (noise removal vs. detail sharpening), we trained a 4th restoration expert for denoising task. We follow the exact same training and optimization setup as we followed to train our other experts. We use 1024×10241024\times1024 images from the Smartphone Image Denoising Dataset (SIDD) [Abdelhamed et al., 2018]. We merged this SIDD expert with OID motion deblurring (SIDD + OID) and with DF2K super-resolution (SIDD + SR), comparing against Naive Soup, Joint multi-task training, and parameter-matched LoRA on Soup (3.36M params) under the exact same training setup and calibration data. OptIMerge outperforms souping baselines on most of the metrics.

Abdelhamed, Abdelrahman, Stephen Lin, and Michael S. Brown. "A high-quality denoising dataset for smartphone cameras." 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018.

Results on SIDD + OID

ModelTrainable ParamsSIDD PSNR ↑SIDD SSIM ↑SIDD LPIPS ↓OID PSNR ↑OID SSIM ↑OID LPIPS ↓
SIDD Expert—35.580.91080.0861———
OID Expert————22.920.62660.2113
Joint Multi-Task890M35.270.90510.080722.580.61930.2174
Naive Soup032.440.88790.132421.040.58180.4799
LoRA on Soup3.36M33.140.89010.100221.300.56700.2751
OptIMerge (Ours)3.19M34.940.90370.083422.660.61760.2208

Results on SIDD + SR

ModelTrainable ParamsSIDD PSNR ↑SIDD SSIM ↑SIDD LPIPS ↓SR (DF2K) PSNR ↑SR (DF2K) SSIM ↑SR (DF2K) LPIPS ↓
SIDD Expert—35.580.91080.0861———
SR Expert————21.260.53230.3131
Joint Multi-Task890M33.500.89950.080821.370.53720.3059
Naive Soup032.400.89730.143920.400.55650.5330
LoRA on Soup3.36M32.890.88660.106520.730.51220.3266
OptIMerge (Ours)3.19M33.590.89640.088621.090.53090.2688

2. Details of compute and training cost

Compute and training cost. OptIMerge merges the expert models with ~40× fewer optimizer updates and ~10× fewer chip-hours than training a single expert or a joint model. We report the details in a table below.

Clarification on "optimization steps". The step counts we report for OptIMerge count micro-steps, i.e. single forward/backward passes on one mini-batch. They are not optimizer updates. Because our compute budget was limited, we used gradient accumulation to reach an effective batch size of 256. As mentioned in the Appendix of our manuscript, gradients are accumulated over 64 micro-steps before each optimizer update, so the number of Adam updates is far smaller, the total training run is about 234 updates which corresponds to 15K micro-steps. OptIMerge therefore converges in a few hundred parameter updates of ∼3.2M coefficients. By comparison, training each expert or the joint model requires 10K updates of the full U-Net. We will state both counts in the revision and rename "steps" to "micro-steps" where it refers to OptIMerge.

Stage / MethodChips (topology)Steps (Adam updates)Chip-hoursPeak HBM / chipStorage
Single-Task Expert (OID, DPDD, SIDD)64 TPU v5p10,000 optimization steps15979.8 GB3.21 GB (for each expert model)
Joint Multi-Task (2-way and 3 way)64 TPU v5p10,000 optimization steps153–15979.8 GB3.21 GB
OptIMerge4 TPU v5p234 optimization steps (15,000 microsteps)13.6–16~89 GB12.8 MB
OptIMerge 3-way4 TPU v5p234 optimization steps (15,000 microsteps)1987.7 GB25.5 MB

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.