- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes QuadMerge, a training-free token-reduction method for diffusion inference. It replaces global token matching with a budgeted multi-resolution partition: a CFG-residual importance map assigns 1x1, 2x2, or 4x4 groups, with a max-risk rule, importance-weighted pooling, timestep-adaptive compression, and plan caching. On PixArt-alpha, 30% token reduction costs 0.09 FID versus 1.05 for importance matching and 12.48 for ToMe; the gap grows at 70%.
Critical strengths
The formulation is clean. Image redundancy is spatially structured, and a budgeted multi-resolution partition is a sensible alternative to global matching. The max-risk rule keeps boundaries out of coarse groups, and the merge-attend-broadcast design leaves the dense interface untouched. The ablations over beta4, TAC, random maps, and scoring variants are thorough.
Critical weaknesses
-
The central efficiency claim fails at the operating point the paper recommends. At r=.30, end-to-end latency is 2.500 s against 2.550 s for the uncompressed baseline, a 2% difference measured on five images with one warm-up and no variance. The 30% figure is a token ratio: full-transformer FLOPs drop by 12.8% because only blocks 7-20 are patched and every MLP stays dense. At r=.50 the numbers are 19.7% FLOPs and 8.2% latency; at r=.70 about 25% and 13.1%. Table 2 is worse for the claim: the 14-step baseline reaches 37.25 FID at 1.714 s and 90.9 TFLOPs, while QuadMerge at r=.30 reaches 36.93 at 2.322 s and 91.4 TFLOPs. The FID difference is 0.32, within noise, and the baseline is 35% faster. Composing 14 steps with r=.30 gives 37.20 at 1.694 s, a 1.2% latency gain. The method wins only in the r=.70 regime that the authors call robustness rather than deployment. The SD2.1 2.2x speedup is the exception, but it is a 2022 UNet.
-
The planner and schedule are not isolated, and the evidence base is thin. At r=.70, QuadMerge without TAC is 57.29, close to importance matching at 56.91, and importance matching with TAC is 56.30; the large gain (48.99) appears only when both are combined. The two TAC instantiations use different caps (.75 versus .95), so this is not schedule-matched. There is no planner-only result at r=.30 or .50, the recommended settings. The only backbones are PixArt-alpha (2023, 0.6B) and SD2.1 (2022, UNet) with 20-step DDIM, and the structure claim rests on FID/CLIP plus qualitative panels, not a localized structural metric.
Questions for authors
- Why not use newer models?
- Can you report latency as mean and standard deviation over more than five images, with a breakdown into attention, MLP, text encoder, VAE, and planner overhead? At r=.30, K and TFLOPs are almost identical across QuadMerge, ToMe, and importance matching (2868.7 versus 2868.0, 113.16 versus 113.10), yet latency spans 2.500 to 2.690 s. The 2% gap against the baseline and the 2.2x SD2.1 speedup both need variance and a breakdown. On SD2.1 latency is also flat from r=.30 to .70 (0.727 versus 0.738 s), which does not fit a simple attention-scaling story.