- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
ComE-ToMe uses calibrated window attention (CWA) to estimate and compensate for local token-merging errors, together with a compensation-aware merging strategy. The method improves the quality–latency trade-off of diffusion models without training.
Critical strengths
Strengths
- Clear motivation and a practical approach to recovering details lost through token merging.
- Consistent improvements over merging baselines, supported by component ablations and direct error analysis.
Critical weaknesses
Weaknesses
-
Limited validation of the core approximations. CWA assumes that local errors capture most of the full merging error and that averaged external keys adequately approximate the softmax denominator. Table 11 provides useful evidence, but covers only one PixArt block at one merging ratio. Moreover, approximation quality in logit space does not directly establish denominator accuracy, particularly for sharply peaked attention. The conditions under which these approximations remain reliable are unclear.
-
Limited evidence of generalization to larger models. The image experiments include PixArt, SD3.5 Medium, and additional SDXL results, but omit FLUX. Video evaluation is restricted to Wan2.1-1.3B and short clips. Since the method relies on attention locality and key similarity, its effectiveness on larger video models and longer videos remains uncertain.
Questions for authors
Questions
- Extend Table 11 across layers, merging ratios, and models. Report how well local errors capture the full error and how accurately CWA estimates the local correction.
- Measure the error in the approximated softmax denominator, especially for sharply peaked attention, and examine its impact on compensation quality.
- Add quality–latency comparisons on FLUX and a larger Wan model, including longer videos, and check whether the core approximations still hold.
The method is well motivated and the results are promising, but the reliability of its approximations and its broader applicability need stronger evidence. I would consider raising my score if the additional analyses and experiments address these concerns and confirm a competitive quality–latency trade-off.
9 comments
Response to Reviewer UQ5b (2)
W1 and Q2 : Approximation quality in logit space does not establish the accuracy of the softmax denominator, particularly for sharply peaked attention, and the conditions under which the approximations remain reliable are unclear.
- Thank you for this insightful comment. While Proposition 1 concerns the logit space, we have analyzed in Appendix A.3 the error of the softmax denominator itself. We fix one query with logits and divide the keys into the query window and the external windows of keys each. The softmax denominator is
CWA keeps and replaces the keys of each with their average, whose logit is the mean logit . The CWA denominator is therefore
Each per-window ratio equals a KL divergence:
where is the attention of the query over the keys in , and is the uniform distribution over the same keys. Since cancels in , the relative denominator error factorizes as
is the attention mass on the external windows, and is the fraction of this mass that the averaged keys miss.
- is small when the attention is local (small ) or the external windows are near-uniform (small ). As the reviewer pointed out, can become large for sharply peaked attention, where a large external mass (large ) concentrates on a few keys (large ). We report both factors and in Table 20 of Appendix D.4 (Table R2).
Table R2 : Denominator error and its two factors in the CWA blocks of each model.
| Block | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PixArt- | 15 | 0.47 | 0.32 | 0.15 | 0.47 | 0.29 | 0.14 | 0.46 | 0.25 | 0.12 |
| 18 | 0.42 | 0.37 | 0.16 | 0.41 | 0.32 | 0.13 | 0.41 | 0.26 | 0.11 | |
| 20 | 0.38 | 0.33 | 0.13 | 0.39 | 0.31 | 0.12 | 0.38 | 0.26 | 0.10 | |
| SD3.5 Medium | 13 | 0.39 | 0.23 | 0.09 | 0.38 | 0.22 | 0.08 | 0.36 | 0.20 | 0.07 |
| 15 | 0.36 | 0.27 | 0.10 | 0.35 | 0.26 | 0.09 | 0.33 | 0.25 | 0.08 | |
| 17 | 0.21 | 0.31 | 0.07 | 0.20 | 0.30 | 0.06 | 0.18 | 0.30 | 0.06 | |
| FLUX.1-dev | 36 | 0.35 | 0.38 | 0.13 | 0.33 | 0.36 | 0.12 | 0.31 | 0.34 | 0.10 |
| 43 | 0.37 | 0.34 | 0.13 | 0.35 | 0.33 | 0.11 | 0.33 | 0.31 | 0.10 | |
| 44 | 0.33 | 0.39 | 0.13 | 0.31 | 0.38 | 0.12 | 0.30 | 0.36 | 0.11 | |
| Wan2.1-1.3B | 12 | 0.45 | 0.28 | 0.13 | 0.44 | 0.25 | 0.11 | 0.42 | 0.22 | 0.09 |
| 15 | 0.53 | 0.35 | 0.19 | 0.52 | 0.31 | 0.16 | 0.50 | 0.26 | 0.13 |
Response to Reviewer UQ5b (3)
- For sharply peaked attention, we group query-window pairs in the CWA blocks by the peak attention probability (Table 21 of Appendix D.4, Table R3). Peaked pairs () show larger errors, but they form only a small fraction of all pairs. The errors over all pairs therefore stay close to those of the non-peaked pairs, and sharply peaked attention has little impact on the overall compensation quality.
Table R3 : Denominator and output errors grouped by in the CWA blocks.
| Pairs (%) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| PixArt- | 88.9 | 0.12 | 0.058 | 0.057 | 0.11 | 0.094 | 0.087 | 0.10 | 0.123 | 0.111 | |
| 11.1 | 0.34 | 0.130 | 0.156 | 0.28 | 0.221 | 0.241 | 0.21 | 0.312 | 0.318 | ||
| All | 100 | 0.14 | 0.066 | 0.068 | 0.13 | 0.108 | 0.104 | 0.11 | 0.144 | 0.134 | |
| SD3.5 Medium | 98.5 | 0.08 | 0.043 | 0.037 | 0.07 | 0.075 | 0.068 | 0.07 | 0.102 | 0.091 | |
| 1.5 | 0.37 | 0.141 | 0.149 | 0.33 | 0.242 | 0.258 | 0.28 | 0.319 | 0.325 | ||
| All | 100 | 0.08 | 0.044 | 0.039 | 0.07 | 0.078 | 0.071 | 0.07 | 0.105 | 0.095 | |
| FLUX.1-dev | 93.1 | 0.12 | 0.108 | 0.101 | 0.11 | 0.157 | 0.143 | 0.09 | 0.189 | 0.169 | |
| 6.9 | 0.31 | 0.212 | 0.198 | 0.26 | 0.315 | 0.289 | 0.23 | 0.378 | 0.341 | ||
| All | 100 | 0.13 | 0.115 | 0.108 | 0.12 | 0.168 | 0.153 | 0.10 | 0.202 | 0.181 | |
| Wan2.1-1.3B | 81.8 | 0.13 | 0.134 | 0.126 | 0.11 | 0.222 | 0.205 | 0.09 | 0.271 | 0.248 | |
| 18.2 | 0.30 | 0.245 | 0.226 | 0.26 | 0.413 | 0.372 | 0.21 | 0.504 | 0.442 | ||
| All | 100 | 0.16 | 0.154 | 0.144 | 0.14 | 0.257 | 0.235 | 0.11 | 0.313 | 0.283 |
Response to Reviewer UQ5b (1)
We thank the reviewer for the careful review and for finding our approach well motivated and practical. We respond to each weakness and question below, and the corresponding changes are marked in blue in the revised manuscript.
We denote by Δf the merging error of the attention output, by ΔfW its in-window component, which comes from the keys in the query window, and by Δf^W the estimate of ΔfW by CWA. CWA relies on two core approximations. (A1) Locality assumes that the in-window error ΔfW captures most of the full merging error Δf. (A2) The averaged-key approximation assumes that the softmax denominator can be approximated with the averaged external keys (Eq. 5).
W1 and Q1 : Table 11 covers only one PixArt-α block at one merging ratio, and it is unclear how well local errors capture the full error and how accurately CWA estimates the local correction.
ϵloc=∥Δf∥∥Δf−ΔfW∥,ϵres=∥Δf∥∥Δf−Δf^W∥,ϵcal=∥ΔfW∥∥Δf^W−ΔfW∥.
ϵloc is the residual error of a local compensation that subtracts the in-window error ΔfW, which tests (A1). ϵres is the residual error after CWA compensation. ϵcal is the error of the CWA estimate, which tests (A2).
Table R1 : Output errors in the CWA blocks of each model.