- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents OptIMerge, an optimization-based framework for merging task-specific diffusion models. It learns lightweight merging parameters with architecture-aware designs, including biased LoRA-based attention alignment and Cayley-transform-based convolutional feature alignment, to reduce parameter interference between experts. The method is evaluated on image restoration model merging and personalized T2I model merging experiments.
Critical strengths
- The paper raises an interesting perspective on model merging for image restoration diffusion models.
- The paper is clearly written and well presented.
- Proposes architecture-aware merging strategies for attention and convolution modules.
- Provides extensive experiments and ablations on image restoration and personalized T2I merging tasks.
- Achieves parameter-efficient merging by optimizing only lightweight merging parameters.
Critical weaknesses
- The motivation for applying model merging to image restoration is not sufficiently discussed compared with existing merging studies in LLMs and image generation. The necessity of merging independently trained restoration experts instead of directly performing joint multi-task training remains unclear.
- The generalization and scalability of OptIMerge are not sufficiently demonstrated. The experiments mainly focus on pairwise merging among three restoration tasks, while the performance gap becomes larger in three-expert merging. More evaluations with a broader range of restoration tasks and larger numbers of merged experts are needed.
- The baseline comparison is not sufficiently convincing. The evaluated baselines mainly consist of fixed 0.5 weight averaging and two zero-shot model merging strategies. However, directly averaging two independently fine-tuned restoration models with a fixed coefficient may be a relatively limited baseline, especially when the models are trained for different restoration tasks with potentially conflicting representations.
- Some important experimental details are insufficiently reported. In particular, the training schedules and optimization budgets of the joint training baseline and the expert training + merging pipeline are unclear. These details are important for a fair comparison and for understanding whether the improvements come from the proposed merging strategy or from differences in training budgets.
Questions for authors
My main concerns lie in the motivation of model merging for image restoration, its generalization ability, the fairness of baseline comparisons, and the advantages over direct joint multi-task training, especially in terms of training procedures and optimization budgets. If these concerns are properly addressed, I would consider increasing my score.
11 comments
2. Additional Results on denoising task for Image Restoration
To demonstrate generalization beyond deblurring and super-resolution to a restoration task with an opposing high-frequency prior (noise removal vs. detail sharpening), we trained a 4th restoration expert for denoising task. We follow the exact same training and optimization setup as we followed to train our other experts. We use images from the Smartphone Image Denoising Dataset (SIDD) [Abdelhamed et al., 2018]. We merged this SIDD expert with OID motion deblurring (SIDD + OID) and with DF2K super-resolution (SIDD + SR), comparing against Naive Soup, Joint multi-task training, and parameter-matched LoRA on Soup (3.36M params) under the exact same training setup and calibration data. OptIMerge outperforms souping baselines on most of the metrics.
Abdelhamed, Abdelrahman, Stephen Lin, and Michael S. Brown. "A high-quality denoising dataset for smartphone cameras." 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018.
Results on SIDD + OID
| Model | Trainable Params | SIDD PSNR | SIDD SSIM | SIDD LPIPS | OID PSNR | OID SSIM | OID LPIPS |
|---|---|---|---|---|---|---|---|
| Experts and joint training | |||||||
| SIDD Expert | — | 35.58 | 0.9108 | 0.0861 | — | — | — |
| OID Expert | — | — | — | — | 22.92 | 0.6266 | 0.2113 |
| Joint Multi-Task | 890M | 35.27 | 0.9051 | 0.0807 | 22.58 | 0.6193 | 0.2174 |
| Souping methods | |||||||
| Naive Soup | 0 | 32.44 | 0.8879 | 0.1324 | 21.04 | 0.5818 | 0.4799 |
| LoRA on Soup | 3.36M | 33.14 | 0.8901 | 0.1002 | 21.30 | 0.5670 | 0.2751 |
OptIMerge (Ours) | 3.19M | 34.94 | 0.9037 | 0.0834 | 22.66 | 0.6176 | 0.2208 |
Results on SIDD + SR
| Model | Trainable Params | SIDD PSNR | SIDD SSIM | SIDD LPIPS | SR (DF2K) PSNR | SR (DF2K) SSIM | SR (DF2K) LPIPS |
|---|---|---|---|---|---|---|---|
| Experts and joint training | |||||||
| SIDD Expert | — | 35.58 | 0.9108 | 0.0861 | — | — | — |
| SR Expert | — | — | — | — | 21.26 | 0.5323 | 0.3131 |
| Joint Multi-Task | 890M | 33.50 | 0.8995 | 0.0808 | 21.37 | 0.5372 | 0.3059 |
| Souping methods | |||||||
| Naive Soup | 0 | 32.40 | 0.8973 | 0.1439 | 20.40 | 0.5565 | 0.5330 |
| LoRA on Soup | 3.36M | 32.89 | 0.8866 | 0.1065 | 20.73 | 0.5122 | 0.3266 |
OptIMerge (Ours) | 3.19M | 33.59 | 0.8964 | 0.0886 | 21.09 | 0.5309 | 0.2688 |
3. Additional optimization-based baselines under the same calibration data
We implemented four additional optimization-based baselines under the same calibration data: a learned global scalar, a parameter-matched LoRA on model soup, AdaMerging, and RegMean. OptIMerge outperforms all four baselines, as shown in the tables below. Please refer to Section A of the Global Response for more details on these baselines.
| Model | OID PSNR | OID SSIM | OID LPIPS | DPDD PSNR | DPDD SSIM | DPDD LPIPS |
|---|---|---|---|---|---|---|
| Experts and joint training | ||||||
| OID Expert | 22.92 | 0.6266 | 0.2113 | - | - | - |
| DPDD Expert | - | - | - | 23.39 | 0.6392 | 0.2023 |
| Joint expert | 22.59 | 0.6173 | 0.2154 | 23.37 | 0.6404 | 0.2091 |
| Souping methods | ||||||
| Naive soup | 20.88 | 0.5601 | 0.2899 | 23.15 | 0.6398 | 0.2641 |
| Learned global scalar | 22.07 | 0.6032 | 0.2310 | 23.17 | 0.6433 | 0.3223 |
| LoRA on soup | 21.99 | 0.5921 | 0.2498 | 23.09 | 0.6268 | 0.2534 |
| RegMean | 21.20 | 0.5712 | 0.2695 | 23.30 | 0.6415 | 0.2316 |
| Layer-wise AdaMerging | 22.34 | 0.5960 | 0.2434 | 22.91 | 0.6166 | 0.3042 |
OptIMerge | 22.72 | 0.6190 | 0.2168 | 23.91 | 0.6561 | 0.2191 |
| Model | OID PSNR | OID SSIM | OID LPIPS | SR PSNR | SR SSIM | SR LPIPS |
|---|---|---|---|---|---|---|
| Experts and joint training | ||||||
| OID Expert | 22.92 | 0.6266 | 0.2113 | - | - | - |
| SR Expert | - | - | - | 21.26 | 0.5323 | 0.3131 |
| Joint | 22.03 | 0.5867 | 0.2386 | 21.49 | 0.5407 | 0.2993 |
| Souping methods | ||||||
| Naive soup | 20.31 | 0.5292 | 0.2938 | 20.99 | 0.4970 | 0.3501 |
| Learned global scalar | 20.56 | 0.5400 | 0.2832 | 20.95 | 0.4805 | 0.3736 |
| LoRA on soup | 20.87 | 0.551 | 0.294 | 20.87 | 0.4979 | 0.3619 |
| Layer-wise AdaMerging | 21.66 | 0.5778 | 0.2492 | 20.87 | 0.4806 | 0.3860 |
OptIMerge | 22.16 | 0.5980 | 0.2298 | 21.45 | 0.5338 | 0.3207 |
| Model | DPDD PSNR | DPDD SSIM | DPDD LPIPS | SR PSNR | SR SSIM | SR LPIPS |
|---|---|---|---|---|---|---|
| Experts and joint training | ||||||
| DPDD Expert | 23.39 | 0.6392 | 0.2023 | - | - | - |
| SR Expert | - | - | - | 21.26 | 0.5323 | 0.3131 |
| Joint | 23.11 | 0.6293 | 0.2185 | 21.24 | 0.5255 | 0.316 |
| Souping methods | ||||||
| Naive soup | 22.57 | 0.6121 | 0.2774 | 21.10 | 0.5259 | 0.3447 |
| Learned global scalar | 22.73 | 0.6173 | 0.2623 | 21.01 | 0.516 | 0.3608 |
| LoRA on soup | 22.69 | 0.6013 | 0.2995 | 20.35 | 0.4927 | 0.3739 |
| Layer-wise AdaMerging | 22.22 | 0.5992 | 0.2797 | 20.66 | 0.4899 | 0.3783 |
OptIMerge | 23.13 | 0.6258 | 0.2365 | 21.12 | 0.5231 | 0.3210 |
Response to Reviewer d8JD
We thank the reviewer for recognizing the novelty, clarity, and parameter efficiency of our architecture-aware merging framework. Below we address your four questions on motivation, generalization/scalability, stronger baselines, and training/optimization budgets.
1. Motivation for model merging for image restoration / I2I
We thank the reviewer for raising this point and will expand this discussion in the revision. The motivation for merging image-to-image (I2I) restoration models is closely aligned with that of merging in the large language models (LLMs) and text-to-image (T2I) generation. In each case, a single large pretrained foundation model is fine-tuned into many task-specific models, and practitioners want to combine the capabilities of these models into one, without pooling all the training data or retraining a multi-task model from scratch. Many such fine-tuned models are shared publicly as checkpoints or LoRAs, often without their training data. The observations that made weight-space merging effective for LLMs therefore carry over to diffusion-based I2I restoration models. We detail this correspondence below.
Single-task training is common practice. In many practical settings, image restoration models are trained for a single degradation type, such as motion deblurring, defocus deblurring, denoising or super-resolution. Each such model is optimized on its own task-specific paired dataset. Supporting several degradations therefore usually requires either maintaining several specialist models or training a single multi-task model from scratch.
Restoration experts share a common pretrained initialization. Modern diffusion-based restoration models are typically obtained by fine-tuning a large pretrained text-to-image diffusion model on the restoration task, instead of training from random initialization. This mirrors the prevailing practice for large language models, where task-specific models are fine-tuned from a common foundation model.
It is precisely this shared-initialization regime that motivates weight-space merging. Models fine-tuned from the same pretrained checkpoint tend to remain in a common region of parameter space, which allows their weights to be combined meaningfully [Model Soups, Task Arithmetic, TIES, DARE, RegMean]. The same property has been exploited for text-to-image personalization [Mix-of-Show, ZipLoRA, Diffusion Soup]. Our restoration experts follow exactly this protocol: each is fine-tuned from the same pretrained text-to-image checkpoint. Merging is therefore a natural way to consolidate them into a single multi-task model, for the same reasons it is used to consolidate fine-tuned LLMs.
Merging avoids the costs of joint training. Joint multi-task training is the principal alternative, but it has three practical drawbacks:
In contrast, OptIMerge requires only the fine-tuned experts and 100-500 calibration pairs per task. It costs approximately 14-16 TPU chip-hours, compared to approximately 159 chip-hours for a single joint retraining.