acceptodds
Under review as a conference paper at ICLR 2027

ReMix: Efficient Diffusion Serving with Adaptive Reuse and Model Mixture

Abstract

Serving text-to-image diffusion models online is expensive, because each request runs tens of sequential denoising steps through a large network. Mixture-of-models serving reduces this cost by caching images that a large model has already generated and repainting them with a small model for similar prompts. Existing methods select the repainting strength by fixed hand-crafted rules and repaint the entire image, which miscalibrates the strength and overwrites content that was already correct. We propose **ReMix**, an efficient diffusion serving framework that reuses cached images and generates with a mixture of models. **ReMix** profiles the denoising trajectory of each cached generation to select a per-request repainting start step and repaints under a graduated change map that expands from the high-change regions to the full image, leaving remaining pixels at the fidelity of the cached image. On COCO and DiffusionDB, **ReMix** cut the mean serving latency by 52.3% and 26.3% against full large-model generation, and achieves the state-of-the-art CLIP, FID, IS, and PickScore among all serving methods.

Primary area:
generative models
Keywords:
Diffusion model, Model serving, Image generation, Caching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e32pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ReMix, a mixture-of-models diffusion serving framework that addresses inefficient step selection and quality degradation from uniform full-image repainting. ReMix uses profiling-driven adaptive step selection to determine per-sample repainting points based on individual denoising trajectories, together with Graduated Repainting to focus computation on semantically changed regions while preserving high-quality cached content. Experiments on COCO and DiffusionDB demonstrate that REMIX consistently improves generation quality over existing serving methods and can even outperform the large model on key image-quality metrics.

Critical strengths
  • This paper address an important and timely problem in diffusion model serving to save the computation while reducing the overall serving latency,

  • The overall system design is easy to follow with specialized cache that can reuse the large diffusion model outcome and leveraging small diffusion model for customization based on the current prompt.

  • Evaluation also demonstrate that the proposed methods wins in terms of its output fidelity and runtime latency performance.

Critical weaknesses
  • The generality of the proposed method is a concerns in terms of its caching effectiveness under different requests, image resolution.
  • Another key concern is about its overhead of profiling when running on the large diffusion model initially.
  • How to translate the design advantage towards the real-world benefits is unclear.
Questions for authors
  • How about different resolutions (other than 512x512) would affect its effectiveness and overhead?
  • Other than latency, how does this in terms of its throughput when handling multiple different requests at scale? I think those are more meaningful metrics when considering the cache-based design and real-world serving settings.
  • For your profiling and analysis, besides those algorithmic metrics/scores, what is the potential GPU hardware computing and memory resources utilization? Given the costs of setting up and maintaining of servers are relatively fixed (e.g., per-hour cost of renting those GPU servers on cloud) and better utilization of those computing/memory resources would amortize its individual requests serving costs.
  • how is the cache hit rate and the overlapping of those prompt and generated images from large diffusion model. for requests from different users may be it is relatively low, while requests from the same users’ requests could potentially be higher, is there any worse-case analysis of its efficiency?
  • Does the FIFO is generally work for all these settings or it is more depending on the most recent observations of those online requests?
3 comments

We thank the reviewer for their detailed and valuable feedback.

We respond to your concerns below:

W1&Q1 Generality: This work focuses on improving image quality in mixture-of-models serving. We therefore use the same cache management policy and image resolution across methods to control these factors. Cache capacity and hit rate do affect the results, but optimizing cache management is outside the scope of this work. We also ran repeated experiments at resolutions of 1024×1024 and 768×768. Higher resolution increases both generation and repainting costs by roughly similar proportions, so mixture-of-models serving still offers a clear latency advantage over serving with only the large model. Image quality scores change little across these resolutions. Since the baselines are compared on their repainting procedures at the same resolution, these settings do not change the main conclusions.

coco:

ResolutionMethodOverall CLIP ScoreRepaint Latency (s)
512×512CacheGenius30.7510.760
512×512MoDM31.0590.829
512×512DiffEdit30.9331.255
512×512Nirvana30.7871.015
512×512ReMix31.4421.145
768×768CacheGenius30.6111.534
768×768MoDM30.9821.652
768×768DiffEdit30.9222.504
768×768Nirvana30.2341.858
768×768ReMix31.2712.290
1024×1024CacheGenius30.5162.954
1024×1024MoDM30.8273.185
1024×1024DiffEdit30.8164.839
1024×1024Nirvana29.5983.413
1024×1024ReMix31.1034.418

diffusiondb:

ResolutionMethodOverall CLIP ScoreRepaint Latency (s)
512×512CacheGenius32.2600.767
512×512MoDM32.3880.906
512×512DiffEdit32.0891.244
512×512Nirvana31.3231.
512×512ReMix32.9511.159
768×768CacheGenius32.1881.542
768×768MoDM32.4391.815
768×768DiffEdit32.1352.505
768×768Nirvana31.6321.912
768×768ReMix32.8062.319
1024×1024CacheGenius32.2172.964
1024×1024MoDM32.2683.509
1024×1024DiffEdit32.0704.840
1024×1024Nirvana30.4543.519
1024×1024ReMix32.4074.476

W2 Cold start: During the initial cold-start period, more requests are handled by the large model, leaving more idle time in the small-model worker's queue. Our idle-time scheduler uses these gaps to run profiling on the small-model worker and interrupts it when inference tasks arrive. Profiling can thus use existing idle resources, limiting its effect on user requests and serving capacity. Appendix A.6 now provides the scheduling details. The table below shows that most profiling time can be overlapped, supporting this low-overhead design.

DatasetLoadProfileOverlap(%)CLIP(hits) ↑Tput(req/s) ↑P95(s) ↓
COCOLowOff--28.730.188421.1
On99.529.250.186479.0
HighOff--28.560.1501711.3
On99.628.920.1471762.8
DiffusionDBLowOff--29.960.135861.5
On99.530.430.134881.0
HighOff--29.530.1102631.3
On99.630.410.1102632.1

Q2 Throughput and other system metrics: We have added these measurements. This work focuses on improving image quality in mixture-of-models serving with little additional overhead, rather than fully optimizing the serving system. Batching and scaling model replicas could further improve service-level objectives (SLOs) and system performance. In Section 4.4, we use Gamma-distributed request arrivals and report throughput and P95 latency under high and low loads. ReMix improves image quality with a maximum throughput decrease of 2.0% and a maximum P95 latency increase of about 13.8%. These results show that the method improves generation quality while maintaining overall serving efficiency.

**Q3 **Utilization: We used NVML to sample GPU utilization and memory usage every 0.5 seconds. Diffusion inference is compute-intensive: the A100 hosting the large model has average GPU utilization above 98% and uses 22.9 GB of memory. The GPU hosting the small model has about 40% utilization and uses 8.6 GB of memory. There is room to improve resource use on the small-model side, but this requires joint optimization of cache management, scheduling, and batching. ReMix can serve as a basis for such a system extension, and its current idle-time scheduler already uses some of the available idle resources.

Q4& Q5 Cache settings: Following MoDM, we set the CLIP cache-hit threshold to 0.26 for all methods. With the relatively small cache capacity, the hit rate is about 40%–50%. All baselines use FIFO replacement. On DiffusionDB, we fixed the cache capacity at 10,000 and measured the distance in the request sequence between the request associated with a hit entry and the current request. This distance follows a long-tailed distribution. We have not studied in depth whether this pattern is directly related to user submission behavior. FIFO also helps maintain cache diversity by preventing frequently reused images from remaining indefinitely, which could make generated images increasingly similar.

Request distanceCache hitsShare (%)
<10002,79632.1
1000–20001,26514.5
2000–30001,08712.5
3000–40008349.6
4000–50006417.4
5000–60006457.4
6000–70004805.5
7000–80004124.7
8000–90003634.2
9000–100001812.1
>1000000.0

I thank the authors for the rebuttal and the additional experiments. The resolution and GPU utilization results are helpful, and the idle-time profiling analysis addresses my cold-start concern.

However, several key concerns remain.

ReMix’s quality margin over MoDM narrows as resolution increases (0.56, 0.37, and 0.14 CLIP points on DiffusionDB at 512, 768, and 1024), while its repaint latency stays about 28% higher, so the absolute gap grows from 0.25s to 0.97s. Since latency scales roughly with pixel count, this trend raises concerns beyond 1024×1024: the benefit may vanish while the cost keeps growing, profiling memory for retained intermediate latents will increase, and small models may not natively support such resolutions.

Section 4.4 shows that the reported 2% throughput and 13.7% P95 figures measure profiling overhead only, so a throughput comparison against the baselines is still missing, and the reported P95 latencies suggest an overloaded regime.

Worst-case behavior under low hit rates and the choice of FIFO also remain unexplored.

Therefore, I will keep my current score.

Reviewer_8WruOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies efficient text-to-image diffusion serving through cross-request reuse. Images generated by a larger model are cached and subsequently edited by a smaller model for semantically related requests. ReMix introduces two components: profiling-driven start-step selection and Graduated Repainting. The former profiles each cached generation using changes in model predictions and intermediate CLIP scores, then maps the source–target semantic gap to a repainting start step. The latter derives a change map from source- and target-conditioned noise predictions and progressively expands the editable region while anchoring the remaining latent regions to the cached image.

Critical strengths

Reusing expensive generations across related requests is a useful direction for reducing inference cost. The paper identifies two reasonable limitations of existing reuse pipelines: coarse selection of repainting strength and degradation of reusable content during full-image editing.

Table 1 reports improvements over MoDM in CLIP, PickScore, FID, and IS on both datasets, with relatively small increases in mean latency.

Critical weaknesses

The quantitative claims are internally inconsistent, including claims central to the paper. Section 4.2 reports ReMix latencies of 4.19/6.48 seconds, whereas Table 1 reports 4.42/6.61 seconds. The claim that ReMix is 18.3% faster than MoDM on COCO conflicts with Table 1, where MoDM takes 4.25 seconds and ReMix takes 4.42 seconds. Similarly, the text reports FID values of 102.09/116.19, whereas Table 1 reports 111.41/127.13.

The noise-prediction difference used to obtain the change map follows DiffEdit. More importantly, Differential Diffusion already uses timestep-dependent thresholding of a change map and blends generated latents with noised versions of the original image to control spatial edit strength. This closely related work is neither discussed nor evaluated.

Table 5 reports 9.422 seconds of additional profiling work per cache entry, compared with 8.610 seconds for large-model generation. Of this overhead, 9.006 seconds are excluded from response latency because they execute asynchronously. Asynchronous execution does not eliminate resource consumption or contention with subsequent requests. Without throughput, tail latency, total GPU cost, and cache-reuse statistics under a fixed hardware budget, the claimed serving benefit remains incomplete.

Questions for authors

Which experimental values are authoritative? Please reconcile Table 1, Table 2, Figure 5, and their accompanying descriptions, and provide corrected latency reductions and component-level conclusions.

Where does asynchronous profiling execute, and does it share GPU resources with request serving? Under the same total hardware budget, what are the throughput, p95/p99 latency, and total GPU cost when profiling is included? How often is each profiled entry reused before eviction?

What distinguishes Graduated Repainting from Differential Diffusion beyond the source of the change map and its application to cached requests? Can you compare the two with identical backbones, maps, starting steps, and evaluation pairs?

What are the LPIPS reference pairs and the FID reference images for each dataset? Please report the evaluation sizes, cache splits, and variability across runs, and clarify whether the conclusions remain valid for another model pair or a shorter sampling schedule.

3 comments

We thank the reviewer for their detailed and valuable feedback.

We respond to your concerns below:

W1&Q1 Data correctness: We apologize for the inconsistencies between the reported data and text. After we unified the resolutions, ReMix ranked second and fourth in latency on the two datasets, respectively. The FID inconsistency came from mixing results from different ReMix runs. We corrected the tables but did not update the text; the revision will make them consistent. On COCO and DiffusionDB, ReMix has mean latencies of 4.42 s and 6.61 s and FID scores of 111.41 and 127.13, respectively. Compared with SD3.5, mean latency decreases by 49.7% and 24.8%. Compared with CacheGenius, FID decreases by 4.8% and 3.6%, while IS increases by 23.2% and 34.6%. We will update the abstract, main text, and captions, and check all ablation conclusions.

W2&Q3 Relation to Differential Diffusion: Thank you for pointing out this work. A direct comparison between ReMix and Differential Diffusion would not be fair for three reasons:

  1. Different tasks. Differential Diffusion targets user-guided image editing, while ReMix adapts cached images to new generation requests.
  2. Different inputs. Differential Diffusion relies on user-specified repainting regions. ReMix automatically identifies these regions from semantic differences between the new request and the cached image.
  3. Different repainting strategies. Differential Diffusion expands user-specified regions to form a gradual change map. ReMix starts from semantic conflicts, preserving still-correct content early and allowing broader adjustments later.

Although both methods use change maps, they assume different inputs and use these maps for different purposes. A fair comparison would therefore require matching the task and input conditions; directly comparing the original methods would not isolate their effectiveness.

W3&Q2 System performance: Profiling consists of loop-based tasks that can be divided into small units. We therefore use interruptible scheduling: profiling runs on an idle worker and is interrupted when that worker receives an inference task. Implementation details have been added to Appendix A.6, and system experiments to Section 4.4. The largest throughput decrease is 2.0%, and the largest P95 latency increase is about 13.8%. The evaluation uses only the two A100 GPUs reported in the paper, with no additional devices. With the current cache capacity, 20%–40% of cache entries are used on COCO and DiffusionDB. ReMix focuses on image quality after a cache hit rather than increasing cache-entry usage. All baselines therefore use the same cache policy to control this factor.

DatasetLoadProfileOverlap(%)CLIP(hits) ↑Tput(req/s) ↑P95(s) ↓
COCOLowOff--28.730.188421.1
On99.529.250.186479.0
HighOff--28.560.1501711.3
On99.628.920.1471762.8
DiffusionDBLowOff--29.960.135861.5
On99.530.430.134881.0
HighOff--29.530.1102631.3
On99.630.410.1102632.1

Q4 Metric clarification: FID uses the original images from the corresponding dataset as its reference distribution: COCO 2017 validation images for COCO, and the original DiffusionDB images for DiffusionDB. LPIPS also uses the corresponding original images as references.

Q4 Transferability: For cache splits, this experiment sets the cache capacity to 10% of the total dataset size. Increasing the capacity can improve the hit rate to some extent, allowing more requests to benefit from mixture-of-models serving. The paper reports stable results obtained across repeated experiments.

ReMix caches final images and summaries of generation trajectories, rather than latent variables tied to a specific repainting model. When switching models, repainting can use the new model's own image encoder, noise scheduler, and denoising network. Locating changed regions by comparing model outputs for the old and new prompts from the same noisy state also does not depend on the UNet architecture of SD1.5. To test this, Section 4.2 now includes a model pair using Stable Diffusion 3.5 Large and Medium. ReMix achieves the best or second-best results on several quality metrics, showing that its quality benefits carry over to another model pair.

DatasetMethodCLIP ↑PickScore ↑FID ↓IS ↑LPIPS ↓Mean Lat. (s) ↓
COCOVanilla31.6922.81112.0313.620.25422.51
CacheGenius30.4721.92119.9313.120.25314.77
MoDM<u>31.18</u><u>22.17</u><u>112.96</u>12.340.24813.27
DiffEdit30.9821.84116.2912.610.24915.32
Nirvana31.0221.51134.2712.080.229<u>13.75</u>
ReMix31.5722.42110.86<u>12.68</u><u>0.234</u>14.79
DiffusionDBVanilla32.9121.57133.178.050.25022.53
CacheGenius31.9021.16138.717.380.25217.85
MoDM<u>32.47</u><u>21.33</u>133.85<u>7.75</u>0.243<u>17.25</u>
Nirvana32.1220.95136.977.49<u>0.238</u>17.16
DiffEdit32.0321.04137.397.510.25018.31
ReMix32.6721.37<u>134.49</u>8.230.22817.99

ReMix can also adjust the repainting strength when the number of sampling steps changes. We added the following experiments to test this:

Semantic alignment scores on COCO:

Sampling stepsCacheGeniusMoDMDiffEditNirvanaReMix
3030.58430.92830.96531.25931.403
3530.69331.12331.05431.26831.680
4030.75731.00931.16830.99231.599
4530.60730.85531.16530.82431.801
5030.50630.81330.96331.21431.816

Semantic alignment scores on DiffusionDB:

Sampling stepsCacheGeniusMoDMDiffEditNirvanaReMix
3031.83031.98331.84831.67232.215
3531.91031.75831.68831.35732.311
4032.01432.29031.88131.44232.545
4531.88432.01431.79231.29132.334
5031.93832.14331.82231.52532.433

Across the tested sampling steps, ReMix achieves the highest semantic alignment scores on both datasets. Semantic alignment is a central metric in this work.

Reviewer_8WruOpenReview ↗

Thank you for the reply.

Thank you for the detailed responses and additional experiments, which address several of my concerns. However, the original numerical inconsistencies affected key performance claims, and the corrected results show that ReMix does not consistently outperform existing serving methods in latency. While the additional system-level experiments are helpful, I remain concerned about whether the quality improvements sufficiently justify the additional latency and profiling costs. Given these remaining concerns, I will maintain my rating.

Reviewer_bAghOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ReMix reuses images from similar prompts to make text-to-image generation faster. It uses SD3.5 to generate images on cache misses and SD1.5 to edit cached images on hits. The two main contributions are choosing the repainting start step from a profile of the large model’s denoising trajectory, and gradually expanding an editing mask so that regions needing changes are repainted first. The authors evaluate ReMix on COCO and DiffusionDB against SD3.5, CacheGenius, MoDM, DiffEdit, and Nirvana. They also test each component separately, vary the cache hit threshold, and measure profiling overhead. Table 1 shows better CLIP, PickScore, FID, and IS than the four serving baselines on both datasets, while reducing latency relative to full SD3.5 generation.

Critical strengths

The use of the cached image’s denoising trajectory to choose the repainting steps is the part I find most interesting. It gives the scheduler information about the individual image beyond the prompt similarity used by MoDM’s fixed step table. According to Table 2, adding profiling improves CLIP, PickScore, and FID on both datasets, with and without the mask. This provides some evidence that the proposed step selection is useful, though it does not establish how much of the gain comes from using an image-specific profile.

Critical weaknesses
  1. I had trouble reconciling the experimental discussion with the tables. Section 4.2 says ReMix is fastest on COCO and second fastest on DiffusionDB, but Table 1 ranks it second and fourth among the five serving methods. More importantly, Section 4.4 says the mask alone reduces DiffusionDB IS from 7.62 to 6.57 and explains this decline, while Table 2 shows an increase to 9.81. The conclusion also says ReMix beats SD3.5 on FID on both datasets, but its COCO FID is worse in Table 1 (111.41 versus 110.13). These differences change the interpretation of the results, especially what the ablations tell us about each component.

  2. The cost of background profiling is not fully accounted for in the serving evaluation. Table 5 reports 9.006 seconds of offline analysis per cache entry, slightly longer than an SD3.5 generation at 8.610 seconds. This analysis is excluded from latency measurements, and the paper does not say whether it uses the same GPUs as request processing. The paper says later cache hits amortize this cost, but does not show the effect under load or how much reuse is needed. Since faster serving is the main motivation, this matters to my assessment of the reported gains.

Questions for authors
  1. Could you clarify which results are correct where the text and tables disagree? In particular, does adding the mask alone increase DiffusionDB IS from 7.62 to 9.81, as Table 2 shows, or decrease it to 6.57, as Section 4.4 states? Please also reconcile the latency rankings and the claim that ReMix beats SD3.5 on FID on both datasets.

  2. Where does background profiling run relative to request processing, and is it running during the latency experiments? If profiling uses separate resources, what are those resources, and how many cache hits are needed to amortize its cost? I am asking about the evaluated setup, since this determines what the reported serving gains account for.

2 comments

We thank the reviewer for their detailed and valuable feedback.

We respond to the concerns below:

W1&Q1 Data correctness: After we unified the resolutions, ReMix ranked second and fourth in latency on the two datasets, respectively. The FID and IS inconsistencies came from mixing results from different ReMix runs: we corrected the tables but did not update the text. The revision will consistently use the latest table values. On COCO and DiffusionDB, ReMix has mean latencies of 4.42 s and 6.61 s, respectively, compared with 8.78 s and 8.79 s for SD3.5, a reduction of 49.7% and 24.8%. Its FID scores are 111.41 and 127.13, IS scores are 18.29 and 11.13, and CLIP scores are 31.36 and 32.78. We will check the abstract, main text, figures, tables, and ablation analyses, and remove old values and conclusions that do not match these results.

W2&Q2 Profiling cost: Profiling shares GPUs with inference and requires no additional devices. ReMix schedules profiling asynchronously during idle periods on each worker, overlapping its computation with online serving. Profiling still requires computation, but it adds little overhead to online serving, so it does not rely on cache hits to amortize a substantial foreground cost. Appendix A.6 now describes the scheduling logic: profiling is split into interruptible tasks that run when workers are idle. We also added system-level measurements, submitting COCO and DiffusionDB prompts with Gamma-distributed arrivals under QPS=5 and QPS=1 settings. With profiling enabled, throughput changes from 0.188 to 0.186, from 0.150 to 0.147, from 0.135 to 0.134, and remains at 0.110 req/s across the four settings. The largest decrease is 2.0%. These results show that ReMix improves image quality while largely retaining the efficiency advantage of mixture-of-models serving.

DatasetLoadProfileOverlap(%)CLIP(hits)↑Tput(req/s) ↑P95(s)↓
COCOLowOff--28.730.188421.1
On99.529.250.186479.0
HighOff--28.560.1501711.3
On99.628.920.1471762.8
DiffusionDBLowOff--29.960.135861.5
On99.530.430.134881.0
HighOff--29.530.1102631.3
On99.630.410.1102632.1

Thank you for the response and additional experiments. Some inconsistent claims are still unresolved. One problem I found is in the ablation study in Section 4.4. The authors stated "This gain depends on the start step the profiler provides, since under the heuristic step the mask alone lowers the DiffusionDB Inception Score from 7.62 to 6.57 by suppressing regions that still carry prompt-relevant content." But table 2 shows an improvement to 9.81 which contradicts this claim and there's no explanation.

The results authors provided directly affect the main claims of the paper. The method does not outperform SD3.5 on COCO in terms of FID.

Therefore, I will keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.