- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper observes that quantization tolerance differs across resolution stages and spatial regions. To this end, they propose PyraQuant, which allocates bit-widths to different regions based on the differences between features obtained from the previous stages and from latent frequency mixing. At higher resolutions, the method further reuses cached predictions for regions requiring limited refinement. Experimental results demonstrate the effectiveness of their method.
Critical strengths
(+) The paper presents an efficient training-free bit-allocation framework for progressive diffusion models.
(+) The method is supported with extensive experiments and ablation studies.
(+) The paper presents deployment results in Table 3, demonstrating the effectiveness of their method in a practical setting.
Critical weaknesses
[Validity]
(-) It is unclear whether spatially varying W3/W4 execution can translate into practical hardware speedup. Applying the same weight matrix at different precisions to different token groups may require fragmented or separate GEMM executions, potentially reducing the expected efficiency gains.
(-) The lower-precision configuration reuses the scaling factors of the base quantizer while merging code levels. Since the optimal scale generally depends on bit-width, the resulting W3 configuration may be suboptimal compared with an independently calibrated W3 quantizer.
[Generality]
(-) PyraQuant is presented as a general framework for progressive high-resolution diffusion, but its bit-allocation scheme relies on the ScaleDiff-specific signal , and both FLUX and SDXL are evaluated within the same ScaleDiff pipeline. Its applicability to other progressive high-resolution diffusion frameworks is therefore unclear to me.
(-) The residual magnitude is used to determine weight precision while activation precision remains fixed. I think the paper should better justify this design and discuss whether activation or joint weight/activation precision allocation would be more appropriate.
[Misc]
(-) The paper would be easier to follow with a brief explanation of the latent frequency mixing operation around L217.
Questions for authors
Please see the weaknesses.
6 comments
Author Response to Reviewer EXEk (2/3)
W3. PyraQuant is presented as a general framework for progressive high-resolution diffusion, but its bit-allocation scheme relies on the ScaleDiff-specific signal , and both FLUX and SDXL are evaluated within the same ScaleDiff pipeline. Its applicability to other progressive high-resolution diffusion frameworks is therefore unclear to me.
A3: Thank you for suggesting a direct evaluation beyond ScaleDiff. We have now evaluated PyraQuant on MegaFusion with 500 prompts and RepLDM with 200 prompts. Relative to their uniform W8A8 bases, PyraQuant reduces BOPs by 54.2% and 60.7%, respectively, while retaining comparable generation quality.
| Pipeline | Method | Effective bits ↓ | BOPs ↓ | FID ↓ | KID (×10³) ↓ | IS ↑ | CLIP ↑ | HPSv2 ↑ | IR ↑ |
|---|---|---|---|---|---|---|---|---|---|
| MegaFusion (500 prompts) | FP16 | 16.00 | 100.0% | 88.5 | 7.2 | 9.40 | 27.73 | 0.2651 | 0.231 |
| Uniform W8A8 | 8.40 | 68.6% | 89.4 | 7.8 | 9.20 | 27.72 | 0.2640 | 0.219 | |
| + PyraQuant | 4.76 | 31.4% | 86.0 | 6.0 | 9.39 | 27.74 | 0.2662 | 0.296 | |
| RepLDM (200 prompts) | FP16 | 16.00 | 100.0% | 117.5 | 3.0 | 7.02 | 27.04 | 0.2859 | 0.641 |
| Uniform W8A8 | 8.40 | 69.7% | 115.4 | 2.3 | 6.92 | 27.06 | 0.2846 | 0.638 | |
| + PyraQuant | 3.71 | 27.4% | 118.8 | 3.7 | 6.95 | 26.96 | 0.2845 | 0.637 |
On MegaFusion, PyraQuant improves all six reported quality metrics over uniform W8A8 while more than halving its BOPs. On RepLDM, HPSv2, ImageReward, and CLIP remain close to the quantized base with approximately 61% fewer BOPs. Effective bit-width decreases from 8.40 to 4.76 and 3.71, respectively.
The allocation mechanism does not require the host framework to use LFM for sampling. Both frameworks provide a transition latent through image-space upsampling and VAE encoding. We obtain by interpolating the preceding stage’s latent estimate directly in latent space, then compute
The same regional RMS score drives allocation. This signal is constructed for scoring without replacing the host’s sampling input.
Cache initialization follows each framework’s stage transition. RepLDM directly uses its clean stage-entry latent ; MegaFusion constructs a clean reference by completing the preceding stage at lower resolution, with this additional computation included in the reported costs. Cached predictions then enter the corresponding scheduler through Eq. (8).
These results demonstrate that PyraQuant’s spatial allocation and inheritance mechanisms extend beyond ScaleDiff through adaptations to the transition signal and cache reference. We will incorporate both evaluations and their integration details into the revised manuscript.
Author Response to Reviewer EXEk (3/3)
W4. The residual magnitude is used to determine weight precision while activation precision remains fixed. I think the paper should better justify this design and discuss whether activation or joint weight/activation precision allocation would be more appropriate.
A4: Thank you for this helpful suggestion. We also considered weight-only, activation-only, and joint precision reduction. To compare these choices, we evaluated their quality–computation tradeoffs on FLUX with the SVDQuant W4A4 base. Following the diagnostic protocol of Table 11, we use 16 images, keep the 1K and 2K stages identical, and apply each configuration throughout the 4K stage. Reconstruction metrics are measured against the output with the 4K stage run in bf16.
| 4K configuration | Per-token BOPs | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| W4A4 (base) | 100% | 36.38 | 0.963 | 0.045 |
| W3A4 (weight-only) | 95.0% | 32.53 | 0.921 | 0.105 |
| W4A3 (activation-only) | 95.0% | 32.44 | 0.916 | 0.116 |
| W3A3 (joint) | 91.3% | 31.09 | 0.892 | 0.154 |
At the same BOPs, weight-only reduction gives slightly better reconstruction quality than activation-only reduction, with better SSIM and LPIPS on 14 of the 16 images. Joint reduction further lowers per-token BOPs from 95.0% to 91.3%, but costs another 1.44 dB in PSNR. The additional computational saving is modest because attention remains at 16-bit precision and accounts for a large share of per-token BOPs. These results support weight-only reduction as our default choice. It also allows the lower-precision weights to be derived directly from the stored codes while retaining the base quantizer’s activation quantization scheme.
The residual signal is also informative for activation precision: under activation reduction, the additional error relative to W4A4 in the highest residual quartile is 4.5× that in the lowest quartile, compared with 4.7× under weight reduction. The same trend holds across all 16 images. This supports extending the allocation principle to activation or joint precision choices, while the comparison above motivates our current weight-only configuration.
These results and the rationale for the default configuration have been added to Appendix C.2 (Table 12) of the revised manuscript, highlighted in blue.
W5. The paper would be easier to follow with a brief explanation of the latent frequency mixing operation around L217.
A5: Thank you for this helpful suggestion. We have added a brief explanation of latent frequency mixing around the original L217 in Section 3.2 and highlighted the changes in blue in the revised manuscript. The added text reads:
"At each resolution transition, the progressive pipeline constructs two upsampled latents from the previous-stage output: , obtained by interpolation in latent space, and , obtained by VAE decoding, upsampling in RGB space, and VAE re-encoding. Latent frequency mixing combines the low-frequency component of with the high-frequency component of to construct the starting latent for denoising at the new resolution (Koh et al., 2025):
where denotes low-pass filtering implemented by downsampling followed by upsampling."
This addition clarifies the mixing operation and the construction of before we introduce the transition residual in Eq. (3).
Author Response to Reviewer EXEk (1/3)
A1: Thank you for highlighting this practical execution issue. Our deployment evaluation measures end-to-end latency, including routing, halo processing, and crop-execution overhead. On FLUX, PyraQuant reduces generation time from 41.0 s to 24.4 s, achieving a 1.68× speedup, with comparable generation quality verified using the timed executor on all 2,000 prompts.
This benefit follows from the allocation policy’s control over refinement. Active regions are grouped into local crops, while cached regions bypass backbone evaluation. Across stages, only children of regions retained at high precision remain candidates for further refinement, connecting regional precision assignments to subsequent computational savings. For these deployment measurements, both the full-canvas baseline and PyraQuant use the same INT8 backend; within each crop, both active precision groups use the stored W4 base weights without separate W3/W4 GEMM calls. The measured speedup therefore reflects routed computation and inherited reuse, with their execution overhead included. The same allocation also achieves a 1.67× speedup under bf16 execution.
For regions that still require refinement, the nested lower-precision state additionally reduces effective bit-width and BOPs while maintaining comparable generation quality, as evaluated through QDQ. This quantization benefit complements the measured deployment gains. We will clarify the execution settings and distinguish these sources of efficiency in the revised manuscript.
A2: Thank you for suggesting this comparison. We share the base quantizer’s scales so that W3 can be derived directly from the stored W4 codes, without additional calibration or a separate set of weights. To assess the quality tradeoff, we compared nested W3 with two independently calibrated W3 variants.
Following the diagnostic protocol of Table 11, we use FLUX with an RTN W4A8 base on 16 images, keep the 1K and 2K stages identical, and apply W3 throughout the 4K stage. Both independent variants quantize the original weights directly: (a) sets each group’s scale from its maximum absolute weight; (b) searches seven candidate scales per group and selects the one with the lowest weight error.
Variant (a) improves reconstruction quality, including a 0.27 dB PSNR gain. Variant (b), however, reduces weight error by approximately 5% relative to nested W3 but yields worse PSNR, SSIM, and LPIPS on all 16 images. Thus, optimizing the W3 scale for weight error does not necessarily improve the generated outputs.
In the full PyraQuant pipeline, W3 is applied only to selected regions. Table 4(a) shows that replacing W3 with the higher-precision W4 in these regions does not improve HPSv2, ImageReward, or CLIPScore: the scores are 0.2822 / 0.617 / 26.85, compared with 0.2827 / 0.620 / 26.90 for PyraQuant. Together, these results support nested W3 as a practical choice that preserves generation quality while sharing the existing weight representation and avoiding additional calibration. We will add this comparison to Appendix C.2 and clarify the design rationale in Section 3.1.