- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ReMix, a mixture-of-models diffusion serving framework that addresses inefficient step selection and quality degradation from uniform full-image repainting. ReMix uses profiling-driven adaptive step selection to determine per-sample repainting points based on individual denoising trajectories, together with Graduated Repainting to focus computation on semantically changed regions while preserving high-quality cached content. Experiments on COCO and DiffusionDB demonstrate that REMIX consistently improves generation quality over existing serving methods and can even outperform the large model on key image-quality metrics.
Critical strengths
-
This paper address an important and timely problem in diffusion model serving to save the computation while reducing the overall serving latency,
-
The overall system design is easy to follow with specialized cache that can reuse the large diffusion model outcome and leveraging small diffusion model for customization based on the current prompt.
-
Evaluation also demonstrate that the proposed methods wins in terms of its output fidelity and runtime latency performance.
Critical weaknesses
- The generality of the proposed method is a concerns in terms of its caching effectiveness under different requests, image resolution.
- Another key concern is about its overhead of profiling when running on the large diffusion model initially.
- How to translate the design advantage towards the real-world benefits is unclear.
Questions for authors
- How about different resolutions (other than 512x512) would affect its effectiveness and overhead?
- Other than latency, how does this in terms of its throughput when handling multiple different requests at scale? I think those are more meaningful metrics when considering the cache-based design and real-world serving settings.
- For your profiling and analysis, besides those algorithmic metrics/scores, what is the potential GPU hardware computing and memory resources utilization? Given the costs of setting up and maintaining of servers are relatively fixed (e.g., per-hour cost of renting those GPU servers on cloud) and better utilization of those computing/memory resources would amortize its individual requests serving costs.
- how is the cache hit rate and the overlapping of those prompt and generated images from large diffusion model. for requests from different users may be it is relatively low, while requests from the same users’ requests could potentially be higher, is there any worse-case analysis of its efficiency?
- Does the FIFO is generally work for all these settings or it is more depending on the most recent observations of those online requests?
3 comments
W2 Cold start: During the initial cold-start period, more requests are handled by the large model, leaving more idle time in the small-model worker's queue. Our idle-time scheduler uses these gaps to run profiling on the small-model worker and interrupts it when inference tasks arrive. Profiling can thus use existing idle resources, limiting its effect on user requests and serving capacity. Appendix A.6 now provides the scheduling details. The table below shows that most profiling time can be overlapped, supporting this low-overhead design.
| Dataset | Load | Profile | Overlap(%) | CLIP(hits) ↑ | Tput(req/s) ↑ | P95(s) ↓ |
|---|---|---|---|---|---|---|
| COCO | Low | Off | -- | 28.73 | 0.188 | 421.1 |
| On | 99.5 | 29.25 | 0.186 | 479.0 | ||
| High | Off | -- | 28.56 | 0.150 | 1711.3 | |
| On | 99.6 | 28.92 | 0.147 | 1762.8 | ||
| DiffusionDB | Low | Off | -- | 29.96 | 0.135 | 861.5 |
| On | 99.5 | 30.43 | 0.134 | 881.0 | ||
| High | Off | -- | 29.53 | 0.110 | 2631.3 | |
| On | 99.6 | 30.41 | 0.110 | 2632.1 |
Q2 Throughput and other system metrics: We have added these measurements. This work focuses on improving image quality in mixture-of-models serving with little additional overhead, rather than fully optimizing the serving system. Batching and scaling model replicas could further improve service-level objectives (SLOs) and system performance. In Section 4.4, we use Gamma-distributed request arrivals and report throughput and P95 latency under high and low loads. ReMix improves image quality with a maximum throughput decrease of 2.0% and a maximum P95 latency increase of about 13.8%. These results show that the method improves generation quality while maintaining overall serving efficiency.
**Q3 **Utilization: We used NVML to sample GPU utilization and memory usage every 0.5 seconds. Diffusion inference is compute-intensive: the A100 hosting the large model has average GPU utilization above 98% and uses 22.9 GB of memory. The GPU hosting the small model has about 40% utilization and uses 8.6 GB of memory. There is room to improve resource use on the small-model side, but this requires joint optimization of cache management, scheduling, and batching. ReMix can serve as a basis for such a system extension, and its current idle-time scheduler already uses some of the available idle resources.
Q4& Q5 Cache settings: Following MoDM, we set the CLIP cache-hit threshold to 0.26 for all methods. With the relatively small cache capacity, the hit rate is about 40%–50%. All baselines use FIFO replacement. On DiffusionDB, we fixed the cache capacity at 10,000 and measured the distance in the request sequence between the request associated with a hit entry and the current request. This distance follows a long-tailed distribution. We have not studied in depth whether this pattern is directly related to user submission behavior. FIFO also helps maintain cache diversity by preventing frequently reused images from remaining indefinitely, which could make generated images increasingly similar.
| Request distance | Cache hits | Share (%) |
|---|---|---|
| <1000 | 2,796 | 32.1 |
| 1000–2000 | 1,265 | 14.5 |
| 2000–3000 | 1,087 | 12.5 |
| 3000–4000 | 834 | 9.6 |
| 4000–5000 | 641 | 7.4 |
| 5000–6000 | 645 | 7.4 |
| 6000–7000 | 480 | 5.5 |
| 7000–8000 | 412 | 4.7 |
| 8000–9000 | 363 | 4.2 |
| 9000–10000 | 181 | 2.1 |
| >10000 | 0 | 0.0 |
I thank the authors for the rebuttal and the additional experiments. The resolution and GPU utilization results are helpful, and the idle-time profiling analysis addresses my cold-start concern.
However, several key concerns remain.
ReMix’s quality margin over MoDM narrows as resolution increases (0.56, 0.37, and 0.14 CLIP points on DiffusionDB at 512, 768, and 1024), while its repaint latency stays about 28% higher, so the absolute gap grows from 0.25s to 0.97s. Since latency scales roughly with pixel count, this trend raises concerns beyond 1024×1024: the benefit may vanish while the cost keeps growing, profiling memory for retained intermediate latents will increase, and small models may not natively support such resolutions.
Section 4.4 shows that the reported 2% throughput and 13.7% P95 figures measure profiling overhead only, so a throughput comparison against the baselines is still missing, and the reported P95 latencies suggest an overloaded regime.
Worst-case behavior under low hit rates and the choice of FIFO also remain unexplored.
Therefore, I will keep my current score.
We thank the reviewer for their detailed and valuable feedback.
We respond to your concerns below:
W1&Q1 Generality: This work focuses on improving image quality in mixture-of-models serving. We therefore use the same cache management policy and image resolution across methods to control these factors. Cache capacity and hit rate do affect the results, but optimizing cache management is outside the scope of this work. We also ran repeated experiments at resolutions of 1024×1024 and 768×768. Higher resolution increases both generation and repainting costs by roughly similar proportions, so mixture-of-models serving still offers a clear latency advantage over serving with only the large model. Image quality scores change little across these resolutions. Since the baselines are compared on their repainting procedures at the same resolution, these settings do not change the main conclusions.
coco:
diffusiondb: