acceptodds
Under review as a conference paper at ICLR 2027

PyraQuant: Spatial Mixed-Precision Quantization for High-Resolution Diffusion Models

10 views

Abstract

High-resolution diffusion generation remains expensive because denoising cost grows rapidly with image resolution. Progressive generation reduces this cost by first generating at a low resolution and then progressively increasing the resolution, while existing diffusion quantization methods mainly allocate precision across layers, channels, or denoising steps without exploiting the stage-wise and spatial structure of this process. We find that progressive refinement exhibits two forms of non-uniformity: quantization tolerance varies across resolution stages, and refinement changes differ across spatial regions. Based on these observations, we propose PyraQuant, a training-free precision allocation framework for quantized progressive diffusion models. PyraQuant keeps the base quantizer at the lowest resolution. At each subsequent resolution transition, it assigns each region a precision according to how much it changes, carrying these decisions to the next resolution. The lower-precision configuration is derived directly from the original quantized representation without additional calibration or model copies, and regions requiring limited refinement reuse cached predictions. Experiments show that on FLUX.1-schnell with the SVDQuant W4A4 base quantizer, PyraQuant reduces effective precision from 4.48 to 2.00 bits while maintaining comparable generation quality, and achieves a end-to-end speedup on UltraHR-eval4K.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
2D, Diffusion, AR, Image, quantization, CV

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EXEkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes that quantization tolerance differs across resolution stages and spatial regions. To this end, they propose PyraQuant, which allocates bit-widths to different regions based on the differences between features obtained from the previous stages and from latent frequency mixing. At higher resolutions, the method further reuses cached predictions for regions requiring limited refinement. Experimental results demonstrate the effectiveness of their method.

Critical strengths

(+) The paper presents an efficient training-free bit-allocation framework for progressive diffusion models.

(+) The method is supported with extensive experiments and ablation studies.

(+) The paper presents deployment results in Table 3, demonstrating the effectiveness of their method in a practical setting.

Critical weaknesses

[Validity]
(-) It is unclear whether spatially varying W3/W4 execution can translate into practical hardware speedup. Applying the same weight matrix at different precisions to different token groups may require fragmented or separate GEMM executions, potentially reducing the expected efficiency gains.

(-) The lower-precision configuration reuses the scaling factors of the base quantizer while merging code levels. Since the optimal scale generally depends on bit-width, the resulting W3 configuration may be suboptimal compared with an independently calibrated W3 quantizer.

[Generality]
(-) PyraQuant is presented as a general framework for progressive high-resolution diffusion, but its bit-allocation scheme relies on the ScaleDiff-specific signal LFM−LUL_{FM}-L_U, and both FLUX and SDXL are evaluated within the same ScaleDiff pipeline. Its applicability to other progressive high-resolution diffusion frameworks is therefore unclear to me.

(-) The residual magnitude e(B)e(B) is used to determine weight precision while activation precision remains fixed. I think the paper should better justify this design and discuss whether activation or joint weight/activation precision allocation would be more appropriate.

[Misc]
(-) The paper would be easier to follow with a brief explanation of the latent frequency mixing operation around L217.

Questions for authors

Please see the weaknesses.

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer EXEk (1/3)

W1. It is unclear whether spatially varying W3/W4 execution can translate into practical hardware speedup. Applying the same weight matrix at different precisions to different token groups may require fragmented or separate GEMM executions, potentially reducing the expected efficiency gains.

A1: Thank you for highlighting this practical execution issue. Our deployment evaluation measures end-to-end latency, including routing, halo processing, and crop-execution overhead. On FLUX, PyraQuant reduces generation time from 41.0 s to 24.4 s, achieving a 1.68× speedup, with comparable generation quality verified using the timed executor on all 2,000 prompts.

This benefit follows from the allocation policy’s control over refinement. Active regions are grouped into local crops, while cached regions bypass backbone evaluation. Across stages, only children of regions retained at high precision remain candidates for further refinement, connecting regional precision assignments to subsequent computational savings. For these deployment measurements, both the full-canvas baseline and PyraQuant use the same INT8 backend; within each crop, both active precision groups use the stored W4 base weights without separate W3/W4 GEMM calls. The measured speedup therefore reflects routed computation and inherited reuse, with their execution overhead included. The same allocation also achieves a 1.67× speedup under bf16 execution.

For regions that still require refinement, the nested lower-precision state additionally reduces effective bit-width and BOPs while maintaining comparable generation quality, as evaluated through QDQ. This quantization benefit complements the measured deployment gains. We will clarify the execution settings and distinguish these sources of efficiency in the revised manuscript.


W2. The lower-precision configuration reuses the scaling factors of the base quantizer while merging code levels. Since the optimal scale generally depends on bit-width, the resulting W3 configuration may be suboptimal compared with an independently calibrated W3 quantizer.

A2: Thank you for suggesting this comparison. We share the base quantizer’s scales so that W3 can be derived directly from the stored W4 codes, without additional calibration or a separate set of weights. To assess the quality tradeoff, we compared nested W3 with two independently calibrated W3 variants.

Following the diagnostic protocol of Table 11, we use FLUX with an RTN W4A8 base on 16 images, keep the 1K and 2K stages identical, and apply W3 throughout the 4K stage. Both independent variants quantize the original weights directly: (a) sets each group’s scale from its maximum absolute weight; (b) searches seven candidate scales per group and selects the one with the lowest weight error.

4K weight configurationPSNR ↑SSIM ↑LPIPS ↓
Nested W3 (shared scales)30.480.8860.170
Independent W3 (a): max-absolute scale30.750.8950.141
Independent W3 (b): weight-error search29.700.8680.205

Variant (a) improves reconstruction quality, including a 0.27 dB PSNR gain. Variant (b), however, reduces weight error by approximately 5% relative to nested W3 but yields worse PSNR, SSIM, and LPIPS on all 16 images. Thus, optimizing the W3 scale for weight error does not necessarily improve the generated outputs.

In the full PyraQuant pipeline, W3 is applied only to selected regions. Table 4(a) shows that replacing W3 with the higher-precision W4 in these regions does not improve HPSv2, ImageReward, or CLIPScore: the scores are 0.2822 / 0.617 / 26.85, compared with 0.2827 / 0.620 / 26.90 for PyraQuant. Together, these results support nested W3 as a practical choice that preserves generation quality while sharing the existing weight representation and avoiding additional calibration. We will add this comparison to Appendix C.2 and clarify the design rationale in Section 3.1.

AuthorsOpenReview ↗

Author Response to Reviewer EXEk (2/3)

W3. PyraQuant is presented as a general framework for progressive high-resolution diffusion, but its bit-allocation scheme relies on the ScaleDiff-specific signal LFM−LUL_{FM}-L_U, and both FLUX and SDXL are evaluated within the same ScaleDiff pipeline. Its applicability to other progressive high-resolution diffusion frameworks is therefore unclear to me.

A3: Thank you for suggesting a direct evaluation beyond ScaleDiff. We have now evaluated PyraQuant on MegaFusion with 500 prompts and RepLDM with 200 prompts. Relative to their uniform W8A8 bases, PyraQuant reduces BOPs by 54.2% and 60.7%, respectively, while retaining comparable generation quality.

PipelineMethodEffective bits ↓BOPs ↓FID ↓KID (×10³) ↓IS ↑CLIP ↑HPSv2 ↑IR ↑
MegaFusion (500 prompts)FP1616.00100.0%88.57.29.4027.730.26510.231
Uniform W8A88.4068.6%89.47.89.2027.720.26400.219
+ PyraQuant4.7631.4%86.06.09.3927.740.26620.296
RepLDM (200 prompts)FP1616.00100.0%117.53.07.0227.040.28590.641
Uniform W8A88.4069.7%115.42.36.9227.060.28460.638
+ PyraQuant3.7127.4%118.83.76.9526.960.28450.637

On MegaFusion, PyraQuant improves all six reported quality metrics over uniform W8A8 while more than halving its BOPs. On RepLDM, HPSv2, ImageReward, and CLIP remain close to the quantized base with approximately 61% fewer BOPs. Effective bit-width decreases from 8.40 to 4.76 and 3.71, respectively.

The allocation mechanism does not require the host framework to use LFM for sampling. Both frameworks provide a transition latent LRUL_{RU} through image-space upsampling and VAE encoding. We obtain LUL_U by interpolating the preceding stage’s latent estimate directly in latent space, then compute

R=HP(LRU)−HP(LU).R=\mathrm{HP}(L_{RU})-\mathrm{HP}(L_U).

The same regional RMS score e(B)e(B) drives allocation. This signal is constructed for scoring without replacing the host’s sampling input.

Cache initialization follows each framework’s stage transition. RepLDM directly uses its clean stage-entry latent LRUL_{RU}; MegaFusion constructs a clean reference by completing the preceding stage at lower resolution, with this additional computation included in the reported costs. Cached predictions then enter the corresponding scheduler through Eq. (8).

These results demonstrate that PyraQuant’s spatial allocation and inheritance mechanisms extend beyond ScaleDiff through adaptations to the transition signal and cache reference. We will incorporate both evaluations and their integration details into the revised manuscript.

AuthorsOpenReview ↗

Author Response to Reviewer EXEk (3/3)

W4. The residual magnitude is used to determine weight precision while activation precision remains fixed. I think the paper should better justify this design and discuss whether activation or joint weight/activation precision allocation would be more appropriate.

A4: Thank you for this helpful suggestion. We also considered weight-only, activation-only, and joint precision reduction. To compare these choices, we evaluated their quality–computation tradeoffs on FLUX with the SVDQuant W4A4 base. Following the diagnostic protocol of Table 11, we use 16 images, keep the 1K and 2K stages identical, and apply each configuration throughout the 4K stage. Reconstruction metrics are measured against the output with the 4K stage run in bf16.

4K configurationPer-token BOPsPSNR ↑SSIM ↑LPIPS ↓
W4A4 (base)100%36.380.9630.045
W3A4 (weight-only)95.0%32.530.9210.105
W4A3 (activation-only)95.0%32.440.9160.116
W3A3 (joint)91.3%31.090.8920.154

At the same BOPs, weight-only reduction gives slightly better reconstruction quality than activation-only reduction, with better SSIM and LPIPS on 14 of the 16 images. Joint reduction further lowers per-token BOPs from 95.0% to 91.3%, but costs another 1.44 dB in PSNR. The additional computational saving is modest because attention remains at 16-bit precision and accounts for a large share of per-token BOPs. These results support weight-only reduction as our default choice. It also allows the lower-precision weights to be derived directly from the stored codes while retaining the base quantizer’s activation quantization scheme.

The residual signal is also informative for activation precision: under activation reduction, the additional error relative to W4A4 in the highest residual quartile is 4.5× that in the lowest quartile, compared with 4.7× under weight reduction. The same trend holds across all 16 images. This supports extending the allocation principle to activation or joint precision choices, while the comparison above motivates our current weight-only configuration.

These results and the rationale for the default configuration have been added to Appendix C.2 (Table 12) of the revised manuscript, highlighted in blue.


W5. The paper would be easier to follow with a brief explanation of the latent frequency mixing operation around L217.

A5: Thank you for this helpful suggestion. We have added a brief explanation of latent frequency mixing around the original L217 in Section 3.2 and highlighted the changes in blue in the revised manuscript. The added text reads:

"At each resolution transition, the progressive pipeline constructs two upsampled latents from the previous-stage output: LUL_U, obtained by interpolation in latent space, and LRUL_{RU}, obtained by VAE decoding, upsampling in RGB space, and VAE re-encoding. Latent frequency mixing combines the low-frequency component of LUL_U with the high-frequency component of LRUL_{RU} to construct the starting latent for denoising at the new resolution (Koh et al., 2025):

LFM=LP(LU)+[LRU−LP(LRU)],L_{\mathrm{FM}} = \mathrm{LP}(L_U) + \bigl[L_{RU}-\mathrm{LP}(L_{RU})\bigr],

where LP\mathrm{LP} denotes low-pass filtering implemented by downsampling followed by upsampling."

This addition clarifies the mixing operation and the construction of LFML_{\mathrm{FM}} before we introduce the transition residual in Eq. (3).

Reviewer_VZFxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PyraQuant, a computation-allocation framework for quantized progressive diffusion generation. It assigns spatial regions to high-precision computation, lower-precision computation, or cached reuse using latent residuals available at resolution transitions. High-precision regions are recursively inherited at subsequent resolutions, while unselected regions bypass backbone evaluation. Lower-precision states are derived from the existing quantized representation, and active regions are processed through local routing with halo context.

Critical strengths

1.A natural formulation of cross-resolution computation allocation. Organizing high-precision refinement, low-precision refinement, and reuse within a shared spatial hierarchy directly exploits the structure of progressive generation.

2.An inexpensive allocation signal. The residual is computed from existing transition tensors, requiring neither a learned predictor nor an additional backbone evaluation.

3.Targeted evidence beyond aggregate benchmarks. The residual correlates with the reconstruction error caused by skipping regions, and budget-matched selection experiments support its usefulness relative to random and inverted selection. The paper also reports routed execution, latency, memory, and compatibility with multiple base quantizers.

Critical weaknesses

1.Mixed precision contributes substantially less than caching in the main FLUX setting. Removing the lower-precision state increases BOPs only from 13.6% to 13.8%, whereas removing caching increases them to 24.4%. This does not invalidate the framework, but suggests that its primary contribution is spatial computation allocation rather than a new quantization mechanism. (Table 9)

2.The residual evidence does not fully justify precision assignment or cross-resolution inheritance. Existing diagnostics primarily establish which regions can tolerate skipped refinement. They do not directly establish sensitivity to high-versus-low precision or explain why lower-resolution importance should determine eligibility for subsequent refinement.

3.The inheritance comparison is not compute-matched. The existing experiment compares inheritance and independent reselection at the same threshold, resulting in different execution budgets. Matching their budgets would better isolate whether the inherited hierarchy improves allocation quality. (Table 13)

4.Quality and deployment speed have not been jointly validated on the same numerical execution path. Quality evaluation uses QDQ simulation, whereas runtime measurement uses a separate packed-INT8 executor. Without quality results from the timed executor or a numerical correspondence analysis, the reported quality and latency cannot yet be interpreted as one verified deployment operating point. (p. 6; Appendix E.3, p. 24)

Questions for authors

1.Correspondence between quality and runtime evaluation. Quality is evaluated using QDQ, whereas latency is measured with a separate packed-INT8 executor. How do these paths correspond numerically, including their high- and low-precision states? Can you provide quality measurements from the timed executor or quantify its output differences from the QDQ implementation?

2.Independent contribution of mixed precision. In Table 9, removing the low-precision state has a relatively small effect on FLUX BOPs compared with removing caching. Under fixed routing and cache decisions, what quality–cost benefit is specifically attributable to mixed precision? Please also specify the exact nested-code mapping, including representative values and rounding.

3.What does the allocation signal predict? The residual diagnostics support predicting the cost of skipping refinement. Does the same signal predict the incremental error of low- versus high-precision computation? Is there evidence that regions assigned lower precision at 2K remain suitable for cached reuse at 4K?

4.Inheritance versus independent reselection. The inheritance ablation compares methods at the same threshold but different execution budgets. At matched executed BOPs, does inherited allocation still outperform independent 4K reselection? How often would independent reselection reactivate regions excluded by the inherited hierarchy?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer VZFx (1/2)

Q1. Correspondence between quality and runtime evaluation. Quality is evaluated using QDQ, whereas latency is measured with a separate packed-INT8 executor. How do these paths correspond numerically, including their high- and low-precision states? Can you provide quality measurements from the timed executor or quantify its output differences from the QDQ implementation?

A1. Thank you for this helpful suggestion. We evaluated the exact timed executor on all 2,000 UltraHR-eval4K prompts, so its generation quality can be assessed together with its latency and memory usage.

MethodLatency / Peak memoryFID ↓KID ×10³ ↓IS ↑CLIP ↑HPSv2 ↑ImageReward ↑
FLUX INT8 full canvas41.0 s / 28.1 GiB45.711.811.6027.460.28220.713
FLUX PyraQuant executor24.4 s / 17.3 GiB42.99.311.7127.410.28460.731
SDXL INT8 full canvas68.9 s / 9.8 GiB37.73.912.4428.180.29240.747
SDXL PyraQuant executor59.5 s / 8.9 GiB38.64.612.3928.050.29080.769

FID and KID use the UltraHR reference images. Latency and final-stage peak memory follow the original L40S measurement protocol.

With the same INT8 backend, PyraQuant achieves 1.68× speedup and 38% lower peak memory on FLUX, with improvements in five quality metrics and nearly unchanged CLIPScore. On SDXL, it achieves 1.16× speedup and 9% lower peak memory, with comparable quality. These results demonstrate reduced computation and memory usage while preserving generation quality on the actual deployment path.

For numerical correspondence, both paths share the base weight representation, spatial allocation rules, routing, and cache mechanism. QDQ evaluates W4A4/nested W3A4 on FLUX and W8A8/nested W4A8 on SDXL. The deployment executor processes both high- and low-precision active regions using the base quantized weights converted for INT8 execution, with A8 activations; SDXL convolutions remain at 16-bit precision. The quality measurements above directly evaluate this deployment configuration.


Q2. Independent contribution of mixed precision. In Table 9, removing the low-precision state has a relatively small effect on FLUX BOPs compared with removing caching. Under fixed routing and cache decisions, what quality–cost benefit is specifically attributable to mixed precision? Please also specify the exact nested-code mapping, including representative values and rounding.

A2. Thank you for highlighting this distinction. We agree that caching accounts for most of the BOPs reduction in the main FLUX configuration. The lower-precision state is designed to retain refinement in regions that still benefit from computation, while using lower numerical precision.

We evaluated 500 prompts per model with fixed regional assignments. Replacing low-precision computation with high-precision computation preserves routing and cache decisions.

ModelConfigurationEffective bits ↓Executed BOPs ↓HPSv2 ↑ImageReward ↑CLIP ↑PSNR ↑ / LPIPS ↓
FLUXLow → high2.2013.85%0.28230.61826.8726.21 / 0.134
FLUXFull method2.0113.61%0.28270.62026.9025.66 / 0.148
SDXLLow → high6.1326.53%0.28960.74927.9123.69 / 0.162
SDXLFull method5.3324.31%0.28920.75627.8723.67 / 0.165

BOPs are relative to the 16-bit pipeline, as in Table 1. PSNR and LPIPS are measured against the corresponding uniform quantized base.

On FLUX, the lower-precision state reduces effective bit-width by 8.6%, with generation scores comparable to those of the high-precision replacement.

The computational benefit is more pronounced on SDXL, where mixed precision reduces effective bit-width by 13.1% and BOPs from 26.53% to 24.31%, with comparable generation quality. FLUX’s smaller aggregate BOPs reduction reflects its substantial attention computation, which remains at 16-bit precision.

The exact nested mappings are as follows:

FLUX W4→W3. For stored codes q4∈[−8,7]q_4\in[-8,7] and group scale s4s_4,

\qquad \widehat w_{\mathrm{lo}}=s_4\operatorname{sign}(c)\,r_{|c|}.$$ where $(r_0,r_1,r_2,r_3)=(0,1.5,3.5,6)$. The magnitude groups are $0$, $\{1,2\}$, $\{3,4\}$, and $\{5,6,7,8\}$, restricted to the stored code range. This deterministic lookup uses the original scale, with no subsequent rounding or recalibration. **SDXL linear layers, W8→W4.** For stored codes $q_8\in[-127,127]$ and scale $s_8$, $$c=\operatorname{round}\left(q_8\frac{7}{127}\right), \qquad \widehat w_{\mathrm{lo}}=c\frac{127}{7}s_8.$$ Rounding is to the nearest integer, with ties to even. The representative values lie on a uniform W4 grid over the original scaling range, determined from $s_8$ without recalibration.
AuthorsOpenReview ↗

Author Response to Reviewer VZFx (2/2)

Q3. What does the allocation signal predict? The residual diagnostics support predicting the cost of skipping refinement. Does the same signal predict the incremental error of low- versus high-precision computation? Is there evidence that regions assigned lower precision at 2K remain suitable for cached reuse at 4K?

A3. Thank you for suggesting these direct checks of the allocation signal. We added two diagnostics to examine its role in precision assignment and cross-resolution inheritance. Both use 16 paired UltraHR-eval4K prompts on ScaleDiff-FLUX.1-schnell with the SVDQuant W4A4 base and τ=0.10\tau=0.10. We keep the 1K and 2K stages identical and compare different 4K computation states against the output with the 4K stage run in bf16. Errors are regional RGB MSE on the 0–255 scale, measured after box-filter downsampling to 204822048^2.

Precision assignment. We compare the additional reconstruction error introduced by replacing W4A4 with nested W3A4. Regions are grouped by the lowest and highest quartiles of e(B)e(B) within each image.

Region groupW4 MSEW3 MSEAdditional MSE (W3 − W4)
Lowest residual quartile15.9932.3316.34
Highest residual quartile57.73133.8976.15

The highest-residual group incurs 4.66× greater additional error than the lowest-residual group, calculated from the group means across images. This ordering holds in all 16 images. The signal therefore identifies regions where reducing precision is more costly, supporting our assignment of higher precision to high-residual regions.

Cross-resolution inheritance. We group 4K regions according to their parents’ actual 2K precision assignments and measure their reconstruction error under cached reuse.

Parent’s 2K assignmentShare of 4K regions4K cache MSECoverage of lowest-cache-error quartile
Low precision54.69%117.577.34%
High precision45.31%257.022.66%

The lowest-cache-error quartile contains the 16 regions with the smallest cache error in each image. Children of low-precision 2K regions have less than half the mean cache error of children of high-precision regions, with lower error in all 16 images. They occupy 54.69% of the canvas yet contain 77.34% of the regions most suitable for reuse. Thus, the 2K allocation provides useful information about 4K reuse suitability, supporting further refinement within inherited high-precision regions.


Q4. Inheritance versus independent reselection. The inheritance ablation compares methods at the same threshold but different execution budgets. At matched executed BOPs, does inherited allocation still outperform independent 4K reselection? How often would independent reselection reactivate regions excluded by the inherited hierarchy?

A4. Thank you for this valuable suggestion. Matching the computation budget helps clarify the role of inheritance. Following your suggestion, we added a comparison on the first 500 UltraHR prompts, matching the number of active regions for each image. This gives identical allocated BOPs and closely matched executed BOPs.

Method4K active regionsAllocated BOPsExecuted BOPsHPSv2 ↑ImageReward ↑CLIP ↑FID ↓
Inheritance30.1%12.3%13.6%0.28270.62026.9083.7
Independent reselection, matched region count30.1%12.3%13.7%0.28260.62126.8784.1
Independent reselection, same threshold (Table 15)66.4%—21.2%0.28240.63226.92—

At the matched allocation budget, inheritance achieves comparable quality to full-canvas reselection, with slightly lower routing overhead. At the same threshold, it reduces executed BOPs from 21.2% to 13.6% while maintaining comparable quality. Together, these results show that inheritance preserves allocation quality while determining the 4K computation budget through the preceding stage, without a separately specified region quota.

For reactivation, same-threshold reselection reactivates 68.3% of the regions excluded by inheritance. Under the matched region budget, 22.1% of independently selected regions fall outside the inherited candidate set. Despite these different selections, quality remains comparable, supporting the effectiveness of the inherited candidate set.

Although the authors have provided detailed responses, my main concerns remain unresolved.

1.The new results address my concern about whether the deployed implementation preserves generation quality. However, Appendix E.3 also clarifies that both active precision groups use the base weights. The speedup over the same INT8 backend therefore comes from routing and cached reuse. It does not demonstrate an additional speed benefit from switching between the high- and low-precision states.

2.Table 10 makes this clearer. With allocation and caching fixed, mixed precision reduces FLUX BOPs from 13.85% to 13.61%, a relative reduction of about 1.7%, with very similar quality. Using the base precision for all active regions therefore retains almost all the computational savings in the main setting. The reduction on SDXL is larger, about 8.4%, which I acknowledge. Still, together with the lack of additional latency savings in the current executor, these results leave me unconvinced that mixed precision adds enough beyond the spatial caching policy.

3.The new precision-sensitivity and cross-resolution diagnostics directly address my question. They provide useful evidence that the residual helps identify regions where reducing precision or skipping refinement is less costly. I consider this concern largely addressed, although the diagnostics are limited to 16 images.

4.Thank you for adding the budget-matched comparison. It shows that inheritance and independent reselection achieve essentially the same quality at similar BOPs. This supports inheritance as a reasonable design choice. However, it does not show a clear additional benefit from tying subsequent refinement eligibility to the previous stage’s precision assignment, compared with independent selection.

I will therefore keep my weak-reject rating.

We thank the reviewer for the detailed and constructive feedback, and especially appreciate the recognition of PyraQuant’s natural cross-resolution allocation, inexpensive residual signal, and targeted analyses beyond aggregate benchmarks. We would like to further clarify how mixed precision and inheritance contribute to this framework.

The additional speedup from mixed precision is not large in the current setting, but its role goes beyond latency reduction. It allows regions that still need refinement to be processed at lower numerical cost while retaining generation quality, rather than choosing between full-precision computation and cached reuse. Caching determines whether a region should be computed, while mixed precision determines how much computation is spent on regions that remain active. The nested state is also derived from the existing quantized representation without extra calibration or model copies.

The matched-budget comparison shows that inheritance maintains comparable quality to independent reselection. Its benefit is that previous-stage precision decisions reduce the candidate set for later refinement: children of low-precision regions are already suitable for reuse and can be excluded, while children of high-precision regions remain eligible. Theoretically, this reduces the amount of candidate evaluation and selection at later resolutions and avoids repeatedly reconsidering the full canvas or specifying a new quota. This makes inheritance a more convenient hierarchical choice for progressive allocation while preserving generation quality.

Reviewer_Y5wvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PyraQuant allocates precision and cached reuse across stages and regions of progressive diffusion generation. Transition residuals guide inherited spatial choices, while lower precision is derived from the existing quantized representation. Experiments on 4K FLUX/SDXL pipelines report QDQ quality, effective precision/BOPs, and latency/memory from a separate deployment executor.

Critical strengths
  1. Useful exploitation of progressive structure. Existing transition residuals guide spatial refinement without an extra predictor or model pass. Inherited decisions reduce expensive high-resolution computation.

  2. Informative component and runtime analyses. Table 4 and Appendix E separate precision, caching, routing, inheritance, and INT8 execution. Wall-clock and memory reductions demonstrate practical savings beyond theoretical BOPs.

Critical weaknesses
  1. Quality and speed are measured with different numerical implementations. Quality uses high-/low-precision QDQ, while latency uses a separate INT8 executor (Section 4.1; Appendix E.3). The disclosed distinction prevents combining these results into a verified deployment operating point. The actual timed executor's quality is needed. The reported 1.81× speedup is relative to BF16, not an optimized W4/W3 baseline.

  2. The nested-precision contribution is small on FLUX and not isolated in runtime. Removing lower precision changes BOPs from 13.6% to 13.8%, versus 24.4% without caching; SDXL shows a larger change, from 24.3% to 26.5%. The BF16 routed/cached timing control isolates backend effects, not W4/W3 switching. A fixed-map runtime comparison is needed to establish the nested state's incremental value; “2.00 effective bits” includes skipped regions, not reduced weight storage.

  3. Baseline adaptation confounds the caching comparison. Appendix D.2 attributes much of the CacheQuant-style collapse to static activation quantization on three diagnostic prompts. Matching activation quantization over a representative prompt set is needed to separate the spatial policy's benefit from baseline adaptation failure.

  4. The novelty relative to region-adaptive progressive sampling is insufficiently positioned. The paper omits RALU [1] and Fresco [2], which already allocate refinement across regions and resolution stages. Although they adapt spatial resolution rather than numerical precision, they narrow the novelty of cross-resolution spatial allocation. The paper should clarify what its residual-guided precision/cache policy adds beyond these precedents.

[1] Wongi Jeong et al. Training-free Mixed-Resolution Latent Upsampling for Spatially Accelerated Diffusion Transformers. arXiv:2507.08422, 2025.

[2] Shikang Zheng et al. From Sketch to Fresco: Efficient Diffusion Transformer with Progressive Resolution. arXiv:2601.07462, 2026.

Questions for authors
  1. Can you report quality, latency, and memory from the exact timed executor, including its numerical differences from QDQ?

  2. With routing, caching, and backend fixed, what quality–latency benefit does the lower-precision state add?

  3. How competitive is temporal reuse under matched activation quantization, beyond the three-prompt diagnostic?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer Y5wv (1/3)

Q1. Can you report quality, latency, and memory from the exact timed executor, including its numerical differences from QDQ?

A1. Thank you for requesting this joint evaluation. We evaluated the exact timed executor on all 2,000 UltraHR-eval4K prompts and report its quality, latency, and memory together below. Latency and final-stage peak memory follow the original L40S measurement protocol. FID and KID use the UltraHR reference images.

MethodLatency ↓Peak memory ↓FID ↓KID ×10³ ↓IS ↑CLIP ↑HPSv2 ↑ImageReward ↑
FLUX INT8 full canvas41.0 s28.1 GiB45.711.811.6027.460.28220.713
FLUX PyraQuant executor24.4 s17.3 GiB42.99.311.7127.410.28460.731
SDXL INT8 full canvas68.9 s9.8 GiB37.73.912.4428.180.29240.747
SDXL PyraQuant executor59.5 s8.9 GiB38.64.612.3928.050.29080.769

Against the same INT8 backend, PyraQuant achieves 1.68× speedup and 38% lower peak memory on FLUX, with improvements in five quality metrics and nearly unchanged CLIPScore. On SDXL, it achieves 1.16× speedup and 9% lower peak memory, with comparable generation quality. These results jointly validate generation quality and deployment efficiency on the same numerical execution path.

For numerical correspondence, the QDQ path evaluates the assigned weight states—W4/W3 on FLUX and W8/W4 on SDXL—with the base activation quantization. The deployment path uses the same allocation rule, threshold, routing, and cache mechanism, and processes active regions using the base quantized weights converted for INT8 execution. Both high- and low-precision active regions therefore use the base weights in this path. Activations use A8, compared with A4 on FLUX and A8 on SDXL in QDQ; SDXL convolutions use 16-bit precision. The quality results above are measured directly from this deployment configuration, together with its reported latency and memory.


Q2. With routing, caching, and backend fixed, what quality–latency benefit does the lower-precision state add?

A2. Thank you for this question. We evaluated the first 500 UltraHR prompts per model, recording and replaying the full method’s region assignments and cache decisions. We then replaced only the low-precision regions with high-precision computation to test whether these regions can retain generation quality at lower numerical precision.

ModelConfigurationEffective bits ↓Relative BOPs ↓HPSv2 ↑ImageReward ↑CLIP ↑PSNR ↑ / LPIPS ↓
FLUXLow replaced by high2.20100%0.28230.61826.8726.21 / 0.134
FLUXFull method2.0198.3%0.28270.62026.9025.66 / 0.148
SDXLLow replaced by high6.13100%0.28960.74927.9123.69 / 0.162
SDXLFull method5.3391.6%0.28920.75627.8723.67 / 0.165

Quality and bit-level costs are evaluated through QDQ; PSNR and LPIPS use the corresponding uniform quantized base as the reference.

With routing and caching fixed, the lower-precision state reduces effective bit-width by 8.6% on FLUX and 13.1% on SDXL while maintaining comparable generation quality. This supports its role as a refinement option for regions that can tolerate lower numerical precision.

The two configurations have nearly identical measured latency under the shared deployment path described in our response to Question 1. The controlled comparison establishes an additional precision–quality benefit beyond routing and caching, using a nested representation that requires no separate weight copy or additional calibration.

AuthorsOpenReview ↗

Author Response to Reviewer Y5wv (2/3)

Q3. How competitive is temporal reuse under matched activation quantization, beyond the three-prompt diagnostic?

A3. Thank you for this helpful suggestion. We extended the comparison to all 2,000 UltraHR-eval4K prompts, using the corrected CacheQuant-style implementation and matching its activation quantization to the dynamic per-token A8 used by our RTN-W8 base. We retain its temporal reuse setting (N=5N=5), anchors, and cache error correction. Under these matched activation settings, PyraQuant retains its advantage across all six reported quality metrics.

MethodBOPs (%) ↓FID ↓KID (×10³) ↓IS ↑CLIP ↑HPSv2 ↑ImageReward ↑
fp16100.037.33.812.4028.200.29290.749
RTN-W833.837.13.612.6528.220.29250.739
CacheQuant-style, matched dynamic A8≈7.041.55.811.7527.910.28100.550
PyraQuant on RTN-W824.337.33.812.4028.060.29040.768

FID and KID are measured against the UltraHR reference images.

Compared with matched-activation CacheQuant-style, PyraQuant achieves lower FID (37.3 vs. 41.5) and higher HPSv2 (0.2904 vs. 0.2810) and ImageReward (0.768 vs. 0.550). It also matches the fp16 baseline on FID and KID while retaining a similar HPSv2 score. These full-benchmark results show that PyraQuant’s quality advantage persists after correcting the baseline and matching activation quantization.

Temporal reuse uses fewer BOPs, while PyraQuant reduces computation to 24.3% of the fp16 pipeline and better preserves generation quality.

AuthorsOpenReview ↗

Author Response to Reviewer Y5wv (3/3)

Our responses to Questions 1–3 address most of the concerns listed under Critical Weaknesses. Below, we further clarify Weakness 4 regarding the distinction between PyraQuant and RALU/Fresco.

Weakness 4. The novelty relative to region-adaptive progressive sampling is insufficiently positioned. The paper omits RALU [1] and Fresco [2], which already allocate refinement across regions and resolution stages. Although they adapt spatial resolution rather than numerical precision, they narrow the novelty of cross-resolution spatial allocation. The paper should clarify what its residual-guided precision/cache policy adds beyond these precedents.

Response to Weakness 4. While all three methods pursue efficient generation, their core research objectives differ. RALU and Fresco investigate how adaptive spatial resolution can reduce sampling cost. PyraQuant addresses a quantization problem: how to exploit regional and cross-stage differences in refinement requirements to reduce numerical precision while preserving generation quality.

RALU prioritizes edge-sensitive regions for earlier upsampling and addresses noise–timestep mismatches. Fresco uses temporal stability to guide regional upsampling while maintaining consistency through a unified noise field. Their allocation determines when and where to increase spatial resolution. PyraQuant retains the host pipeline’s resolution schedule and determines which regions require high-precision refinement, which can use lower-precision refinement, and which can reuse cached predictions at the final stage.

Three connected mechanisms support this objective. First, transition residuals guide regional precision and reuse decisions without an additional model forward pass for scoring. Second, lower-precision states are derived from the same stored quantized codes. This allows regions that still benefit from refinement to continue computing at reduced precision, without an additional model copy or recalibration. Third, precision allocation determines subsequent refinement eligibility: only the children of regions retained at high precision remain candidates for refinement at the next resolution, while regions outside these paths use cached predictions. The policy therefore connects the precision of current computation with the scope of subsequent refinement.

Our contribution lies in this residual-guided precision/cache policy and its cross-stage inheritance. RALU and Fresco provide relevant precedents for regional resolution adaptation; PyraQuant introduces distinct computation states and dependencies to address precision allocation within progressive generation. We will make these objectives and mechanisms explicit in the related-work discussion and contribution statement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.