acceptodds
Under review as a conference paper at ICLR 2027

Compensating Token Merging for Efficient Diffusion Transformers

Abstract

Token merging approaches accelerate diffusion transformers (DiTs) by aggregating redundant tokens into representative ones to reduce the quadratic cost of attention. Recent approaches further employ advanced selection criteria to identify redundant tokens for aggregation. Despite their effectiveness, they still suffer from significant merging errors, the discrepancies between full-token and merged-token outputs, leading to blurred textures and local artifacts. However, directly computing these errors requires attention over the full tokens, which negates the efficiency gain of token merging. To address this, we propose ComE-ToMe, a novel framework that compensates for the merging errors efficiently. Specifically, we introduce calibrated window attention (CWA) that estimates the errors within local neighborhoods to restore the lost details. By computing attention within local windows while incorporating outside-window tokens through their averaged keys, CWA effectively captures fine details without the cost of full attention. We further introduce compensation-aware merging (CAM), which adapts token merging for CWA by preserving boundary tokens and using cumulative CFG importance to stably protect salient regions across denoising steps. Extensive experiments across various DiT architectures demonstrate that ComE-ToMe consistently outperforms prior approaches while maintaining efficiency. We will make our code publicly available online upon acceptance.

Primary area:
generative models
Keywords:
Diffusion, Token merging

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UQ5bOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

ComE-ToMe uses calibrated window attention (CWA) to estimate and compensate for local token-merging errors, together with a compensation-aware merging strategy. The method improves the quality–latency trade-off of diffusion models without training.

Critical strengths

Strengths

  • Clear motivation and a practical approach to recovering details lost through token merging.
  • Consistent improvements over merging baselines, supported by component ablations and direct error analysis.
Critical weaknesses

Weaknesses

  1. Limited validation of the core approximations. CWA assumes that local errors capture most of the full merging error and that averaged external keys adequately approximate the softmax denominator. Table 11 provides useful evidence, but covers only one PixArt block at one merging ratio. Moreover, approximation quality in logit space does not directly establish denominator accuracy, particularly for sharply peaked attention. The conditions under which these approximations remain reliable are unclear.

  2. Limited evidence of generalization to larger models. The image experiments include PixArt, SD3.5 Medium, and additional SDXL results, but omit FLUX. Video evaluation is restricted to Wan2.1-1.3B and short clips. Since the method relies on attention locality and key similarity, its effectiveness on larger video models and longer videos remains uncertain.

Questions for authors

Questions

  1. Extend Table 11 across layers, merging ratios, and models. Report how well local errors capture the full error and how accurately CWA estimates the local correction.
  2. Measure the error in the approximated softmax denominator, especially for sharply peaked attention, and examine its impact on compensation quality.
  3. Add quality–latency comparisons on FLUX and a larger Wan model, including longer videos, and check whether the core approximations still hold.

The method is well motivated and the results are promising, but the reliability of its approximations and its broader applicability need stronger evidence. I would consider raising my score if the additional analyses and experiments address these concerns and confirm a competitive quality–latency trade-off.

9 comments
AuthorsOpenReview ↗

Response to Reviewer UQ5b (1)

We thank the reviewer for the careful review and for finding our approach well motivated and practical. We respond to each weakness and question below, and the corresponding changes are marked in blue in the revised manuscript.

We denote by Δf\Delta f the merging error of the attention output, by ΔfW\Delta f_W its in-window component, which comes from the keys in the query window, and by Δf^W\Delta \hat f_W the estimate of ΔfW\Delta f_W by CWA. CWA relies on two core approximations. (A1) Locality assumes that the in-window error ΔfW\Delta f_W captures most of the full merging error Δf\Delta f. (A2) The averaged-key approximation assumes that the softmax denominator can be approximated with the averaged external keys (Eq. 5).

W1 and Q1 : Table 11 covers only one PixArt-α\alpha block at one merging ratio, and it is unclear how well local errors capture the full error and how accurately CWA estimates the local correction.

  • Thank you for this constructive suggestion. Following the suggestion, we have extended Table 19 of Appendix D.4 (Table 11 in the original manuscript) to three reduction ratios, up to three CWA blocks per model, and four models (Table R1). Following Reviewer ih3v, we report three relative errors that show the remaining error directly:

ϵloc=∥Δf−ΔfW∥∥Δf∥,ϵres=∥Δf−Δf^W∥∥Δf∥,ϵcal=∥Δf^W−ΔfW∥∥ΔfW∥.\epsilon_{\text{loc}} = \frac{\Vert \Delta f - \Delta f_W\Vert}{\Vert \Delta f\Vert}, \qquad \epsilon_{\text{res}} = \frac{\Vert \Delta f - \Delta \hat f_W\Vert}{\Vert \Delta f\Vert}, \qquad \epsilon_{\text{cal}} = \frac{\Vert \Delta \hat f_W - \Delta f_W\Vert}{\Vert \Delta f_W\Vert}.

ϵloc\epsilon_{\text{loc}} is the residual error of a local compensation that subtracts the in-window error ΔfW\Delta f_W, which tests (A1). ϵres\epsilon_{\text{res}} is the residual error after CWA compensation. ϵcal\epsilon_{\text{cal}} is the error of the CWA estimate, which tests (A2).

  • ϵloc\epsilon_{\text{loc}} stays small in every block. This indicates that the in-window error ΔfW\Delta f_W captures most of the merging error, which supports (A1). ϵcal\epsilon_{\text{cal}} stays at most 0.089, and ϵres\epsilon_{\text{res}} differs from ϵloc\epsilon_{\text{loc}} by at most 0.036 in every block. This indicates that CWA performs comparably to a local compensation with ΔfW\Delta f_W. We have added these results to Table 19 of Appendix D.4.

Table R1 : Output errors in the CWA blocks of each model.

Blockr=0.3r=0.3r=0.5r=0.5r=0.7r=0.7
ϵloc\epsilon_{\text{loc}}ϵres\epsilon_{\text{res}}ϵcal\epsilon_{\text{cal}}ϵloc\epsilon_{\text{loc}}ϵres\epsilon_{\text{res}}ϵcal\epsilon_{\text{cal}}ϵloc\epsilon_{\text{loc}}ϵres\epsilon_{\text{res}}ϵcal\epsilon_{\text{cal}}
PixArt-α\alpha150.0670.0700.0200.1190.1120.0320.1660.1520.043
180.0750.0790.0310.1080.1090.0450.1390.1340.054
200.0600.0610.0280.0950.0890.0400.1240.1120.048
SD3.5 Medium130.0310.0300.0060.0650.0630.0110.0950.0910.016
150.0490.0450.0090.0930.0870.0150.1270.1190.022
170.0570.0450.0250.0810.0640.0310.0990.0760.036
FLUX.1-dev360.1100.1040.0420.1620.1480.0570.1960.1760.066
430.1220.1120.0310.1760.1580.0450.2080.1850.054
440.1170.1080.0410.1730.1550.0550.2060.1820.062
Wan2.1-1.3B120.1310.1250.0300.2300.2130.0520.2880.2650.065
150.1870.1770.0460.3040.2780.0730.3750.3390.089
AuthorsOpenReview ↗

Response to Reviewer UQ5b (2)

W1 and Q2 : Approximation quality in logit space does not establish the accuracy of the softmax denominator, particularly for sharply peaked attention, and the conditions under which the approximations remain reliable are unclear.

  • Thank you for this insightful comment. While Proposition 1 concerns the logit space, we have analyzed in Appendix A.3 the error of the softmax denominator itself. We fix one query with logits zjz_j and divide the keys into the query window WW and the external windows EiE_i of NWN_W keys each. The softmax denominator is

Z=ZW+ZE,ZW=∑j∈Wezj,ZE=∑iZi,Zi=∑j∈Eiezj.Z = Z_W + Z_E, \qquad Z_W = \sum_{j \in W} e^{z_j}, \qquad Z_E = \sum_{i} Z_i, \qquad Z_i = \sum_{j \in E_i} e^{z_j}.

CWA keeps ZWZ_W and replaces the keys of each EiE_i with their average, whose logit is the mean logit zˉEi\bar z_{E_i}. The CWA denominator is therefore

Z^=ZW+Z^E,Z^E=∑iZ^i,Z^i=NWezˉEi.\hat Z = Z_W + \hat Z_E, \qquad \hat Z_E = \sum_{i} \hat Z_i, \qquad \hat Z_i = N_W e^{\bar z_{E_i}}.

Each per-window ratio equals a KL divergence:

log⁡ZiZ^i=log⁡1NW∑j∈Eiezj−zˉEi=KL(U∥p(i)),\log \frac{Z_i}{\hat Z_i} = \log \frac{1}{N_W}\sum_{j \in E_i} e^{z_j - \bar z_{E_i}} = \mathrm{KL}(U \Vert p^{(i)}),

where p(i)p^{(i)} is the attention of the query over the keys in EiE_i, and UU is the uniform distribution over the same keys. Since ZWZ_W cancels in Z−Z^Z - \hat Z, the relative denominator error factorizes as

ϵZ=Z−Z^Z=ZEZ⋅κ,κ=1−Z^EZE=∑iZiZE(1−e−KL(U∥p(i)))∈[0,1).\epsilon_Z = \frac{Z - \hat Z}{Z} = \frac{Z_E}{Z} \cdot \kappa, \qquad \kappa = 1 - \frac{\hat Z_E}{Z_E} = \sum_{i} \frac{Z_i}{Z_E} \left(1 - e^{-\mathrm{KL}(U \Vert p^{(i)})}\right) \in [0, 1).

ZE/ZZ_E/Z is the attention mass on the external windows, and κ\kappa is the fraction of this mass that the averaged keys miss.

  • ϵZ\epsilon_Z is small when the attention is local (small ZE/ZZ_E/Z) or the external windows are near-uniform (small κ\kappa). As the reviewer pointed out, ϵZ\epsilon_Z can become large for sharply peaked attention, where a large external mass (large ZE/ZZ_E/Z) concentrates on a few keys (large κ\kappa). We report both factors and ϵZ\epsilon_Z in Table 20 of Appendix D.4 (Table R2).

Table R2 : Denominator error ϵZ\epsilon_Z and its two factors in the CWA blocks of each model.

Blockr=0.3r=0.3r=0.5r=0.5r=0.7r=0.7
ZE/ZZ_E/Zκ\kappaϵZ\epsilon_ZZE/ZZ_E/Zκ\kappaϵZ\epsilon_ZZE/ZZ_E/Zκ\kappaϵZ\epsilon_Z
PixArt-α\alpha150.470.320.150.470.290.140.460.250.12
180.420.370.160.410.320.130.410.260.11
200.380.330.130.390.310.120.380.260.10
SD3.5 Medium130.390.230.090.380.220.080.360.200.07
150.360.270.100.350.260.090.330.250.08
170.210.310.070.200.300.060.180.300.06
FLUX.1-dev360.350.380.130.330.360.120.310.340.10
430.370.340.130.350.330.110.330.310.10
440.330.390.130.310.380.120.300.360.11
Wan2.1-1.3B120.450.280.130.440.250.110.420.220.09
150.530.350.190.520.310.160.500.260.13
AuthorsOpenReview ↗

Response to Reviewer UQ5b (3)

  • For sharply peaked attention, we group query-window pairs in the CWA blocks by the peak attention probability pmax⁡p_{\max} (Table 21 of Appendix D.4, Table R3). Peaked pairs (pmax⁡≥0.1p_{\max} \ge 0.1) show larger errors, but they form only a small fraction of all pairs. The errors over all pairs therefore stay close to those of the non-peaked pairs, and sharply peaked attention has little impact on the overall compensation quality.

Table R3 : Denominator and output errors grouped by pmax⁡p_{\max} in the CWA blocks.

pmax⁡p_{\max}Pairs (%)r=0.3r=0.3r=0.5r=0.5r=0.7r=0.7
ϵZ\epsilon_Zϵloc\epsilon_{\text{loc}}ϵres\epsilon_{\text{res}}ϵZ\epsilon_Zϵloc\epsilon_{\text{loc}}ϵres\epsilon_{\text{res}}ϵZ\epsilon_Zϵloc\epsilon_{\text{loc}}ϵres\epsilon_{\text{res}}
PixArt-α\alpha<0.1< 0.188.90.120.0580.0570.110.0940.0870.100.1230.111
≥0.1\ge 0.111.10.340.1300.1560.280.2210.2410.210.3120.318
All1000.140.0660.0680.130.1080.1040.110.1440.134
SD3.5 Medium<0.1< 0.198.50.080.0430.0370.070.0750.0680.070.1020.091
≥0.1\ge 0.11.50.370.1410.1490.330.2420.2580.280.3190.325
All1000.080.0440.0390.070.0780.0710.070.1050.095
FLUX.1-dev<0.1< 0.193.10.120.1080.1010.110.1570.1430.090.1890.169
≥0.1\ge 0.16.90.310.2120.1980.260.3150.2890.230.3780.341
All1000.130.1150.1080.120.1680.1530.100.2020.181
Wan2.1-1.3B<0.1< 0.181.80.130.1340.1260.110.2220.2050.090.2710.248
≥0.1\ge 0.118.20.300.2450.2260.260.4130.3720.210.5040.442
All1000.160.1540.1440.140.2570.2350.110.3130.283
Reviewer_R23iOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Summary:

The paper introduces ComE-ToMe, a training-free approach to mitigate detail degradation from token merging in DiTs. Based on the observations that merging mainly impairs high-frequency details and that late-stage denoising attention exhibits pronounced locality, the paper proposes compensating for merging errors locally. The framework consists of:

CWA: Computes the local attention discrepancy between original and merged tokens—calibrating the normalization via average out-of-window keys—and adds the residual back to the global output.

CAM: Allocates representative tokens according to cumulative CFG importance and boundary positions to curb information loss in critical regions and across window boundaries.

Experiments on PixArt-α, SD3.5 Medium, and Wan2.1-1.3B demonstrate that the approach achieves superior generation quality compared to existing token merging methods at matched merging ratios while maintaining inference speedup.

Critical strengths
  1. CWA is a lightweight mechanism to compensate for merging errors. It requires no training and works in a plug-and-play manner. This approach does not depend on a specific merging algorithm. It easily integrates into existing token merging methods, showing strong versatility and practical value.
Critical weaknesses
  1. Hyperparameters like the accumulation starting step, decay factor, and boundary weight appear to be empirical. Currently, the ablation study is only performed on one base model(PixArt-α). I am concerned about how robust these parameters are across different architectures.
Questions for authors

1.For Wan2.1, CWA is restricted to spatial windows within each latent frame. Although keys from other frames help calibrate attention normalization, cross-frame merging errors are not directly compensated by CWA. My concern is whether these residual errors affect temporal consistency.

2.The merging range follows Imp-ToMe, covering the middle half of the network, while CWA is applied only to the latter half of these merged layers. How sensitive are the reported gains to this choice? In particular, what happens if CWA is applied to all merged layers or shifted earlier?

3.The accumulation start step, decay factor, and boundary weight are heuristically designed. Although the same settings are used across models, the reported hyperparameter ablations are limited to PixArt-α. My concern is how sensitive the gains are to these choices on other architectures. Sensitivity analyses on SD3.5 and Wan2.1 would help clarify whether these defaults remain effective without model-specific tuning.

7 comments
AuthorsOpenReview ↗

Response to Reviewer R23i (1)

We thank the reviewer for the positive assessment of CWA as a lightweight and plug-and-play mechanism. We respond to the weakness and each question below, and the corresponding changes are marked in blue in the revised manuscript.

W1 and Q3 : The CAM hyperparameters (the accumulation starting step, the decay factor, and the boundary weight) are ablated only on PixArt-α\alpha, and their robustness across architectures is unclear.

  • Thank you for this suggestion. Following the suggestion, we have extended in Table 14 of Appendix D.1 (Table R1) the ablation on the decay factor α\alpha and the boundary weight γ\gamma (Table 6) to SD3.5 and Wan2.1-1.3B.
  • (1) The cumulative CFG map improves quality on both models. With α=0.9\alpha=0.9, FID on SD3.5 decreases from 33.71 to 32.56, and VBench on Wan2.1 increases from 68.96 to 69.93. (2) The boundary term further improves quality on both models. With γ=0.25\gamma=0.25, FID on SD3.5 decreases from 32.56 to 32.05, and VBench on Wan2.1 increases from 69.93 to 70.48. This indicates that α=0.9\alpha=0.9 and γ=0.25\gamma=0.25 remain effective on both models.
  • We have also added in Table 15 of Appendix D.1 (Table R2) results for different steps at which CAM starts to accumulate CFG scores on PixArt-α\alpha. Starting at step 10, the middle of the denoising trajectory, gives the best FID, sFID, and CLIP score, and FID changes by at most 0.54 across all starting steps. We conjecture that this is because early steps form the global structure of the image, such that their CFG scores are noisy and less informative for merging. A late start, on the other hand, leaves fewer steps to accumulate the CFG map.
  • Due to the limited rebuttal period, we could conduct the ablation on the starting step only on PixArt-α\alpha. We are running the same ablation on SD3.5 and Wan2.1-1.3B, and we will post the results during the discussion period once they are complete and add them to the final version.

Table R1 : Ablation on α\alpha and γ\gamma of CAM (SD3.5 at r=0.7r=0.7, Wan2.1-1.3B at r=0.5r=0.5).

α\alphaγ\gammaSD3.5 FID ↓\downarrowSD3.5 sFID ↓\downarrowWan2.1 VBench ↑\uparrow
0033.7154.8568.96
0.9032.5654.2469.93
0.90.2532.0553.9270.48
0.980.2532.2854.0569.92

Table R2 : Ablation on the start step of CFG accumulation in CAM (PixArt-α\alpha, r=0.7r=0.7).

Starting stepFID ↓\downarrowsFID ↓\downarrowCLIP ↑\uparrow
0 (first step)40.7348.4830.97
540.5447.9430.99
10 (middle, ours)40.2646.9231.02
1540.8049.9531.02
AuthorsOpenReview ↗

Response to Reviewer R23i (2)

Q1 : CWA does not directly compensate for cross-frame merging errors on Wan2.1, which may affect temporal consistency.

  • Thank you for this insightful question. ComE-ToMe does not produce cross-frame merging errors on Wan2.1, since it merges tokens only along the spatial dimensions within each frame. Both the merging windows and the CWA windows therefore lie inside a single frame. CWA averages the keys of every window outside the query window, including the windows in other frames. We have clarified this in Appendix B.3.2.
  • Within-frame merging may still affect temporal consistency in a different way. ComE-ToMe merges each frame separately, and two consecutive frames may thus merge different tokens. Their merging errors may then differ, which could appear as flickering across frames.
  • This effect is small in practice, since consecutive frames have similar content. Token merging therefore tends to select similar tokens in consecutive frames, which results in similar merging errors across frames. To verify this, we have added in Table 12 of Appendix C.4 (Table R3) four temporal-consistency categories of VBench on Wan2.1-14B at r=0.3r=0.3, using the same setting as Table 10. ComE-ToMe stays within 0.6 points of the full-token model in all four categories, while reducing the latency by 14.2%. Its temporal flickering score drops by only 0.3 points (99.6 vs. 99.3), and its motion smoothness remains identical (99.2). This indicates that ComE-ToMe preserves temporal consistency.

Table R3 : VBench temporal consistency on Wan2.1-14B (r=0.3r=0.3). SC: subject consistency, BC: background consistency, TF: temporal flickering, MS: motion smoothness.

MethodSC ↑\uparrowBC ↑\uparrowTF ↑\uparrowMS ↑\uparrowLatency Δ\Delta
Wan2.1-14B96.799.399.699.2-
ComE-ToMe (ours)96.298.799.399.2-14.2%

Q2 : Sensitivity of the gains to the placement of CWA, including applying CWA to all merged layers or shifting it earlier.

  • Thank you for this question. Following the suggestion, we have added in Table 16 of Appendix D.1 (Table R4) results of three CWA placements on PixArt-α\alpha and SD3.5 at r=0.7r=0.7, where the earlier-half placement uses the same number of CWA layers as ours. Both models show the same trend. CWA is most effective in the latter half of the merged layers, and applying CWA to all merged layers further improves quality at a higher cost.
  • (1) Applying CWA to all merged layers brings FID close to that of the full-token model. FID reaches 37.07 on PixArt-α\alpha (full-token: 37.62) and 27.82 on SD3.5 (full-token: 27.95). This suggests that CWA compensates for much of the merging error. CWA on all merged layers, however, reduces latency by only 17.3% on SD3.5, compared with 32.1% for our setting.
  • (2) Shifting CWA to the earlier half degrades quality substantially, even with the same number of CWA layers (e.g., FID of 48.84 vs. 32.05 on SD3.5). CWA in the earlier half loses its effect, since the remaining merged layers merge the compensated tokens again. CWA in the latter half avoids this problem, since it compensates the tokens just before they leave the merged layers.

Table R4 : CWA placement on PixArt-α\alpha and SD3.5 (r=0.7r=0.7). Both placements use the same number of CWA layers.

CWA placementFID ↓\downarrowsFID ↓\downarrowCLIP ↑\uparrowLatency Δ\Delta
PixArt-α\alphaFull-token model37.6238.6131.75-
All merged layers37.0743.5531.25-11.5%
Earlier half of merged layers48.7754.5230.64-21.7%
Latter half of merged layers (ours)40.2646.9231.02-21.7%
SD3.5Full-token model27.9548.5032.15-
All merged layers27.8251.7932.03-17.3%
Earlier half of merged layers48.8464.3431.68-32.1%
Latter half of merged layers (ours)32.0553.9231.93-32.1%
AuthorsOpenReview ↗

Response to Reviewer R23i (3)

Follow-up on W1 and Q3 : Ablation on the starting step of CFG accumulation on SD3.5 and Wan2.1-1.3B.

  • As promised, we have completed the ablation on the starting step on SD3.5 and Wan2.1-1.3B (Table R5). We vary the starting step from the first step to three quarters of the denoising trajectory, using the same settings as Table R1.
  • Starting at the middle of the denoising trajectory gives the best results on both models. It gives the best FID (32.05) and sFID (53.92) on SD3.5 and the best VBench score (70.48) on Wan2.1-1.3B. PixArt-α\alpha shows the same trend (Table R2).
  • Starting at the first step gives the worst FID on SD3.5 (33.90) and the worst VBench score on Wan2.1-1.3B (67.04). This is consistent with our conjecture that CFG scores at early steps are noisy, since early steps form the global structure of the image.
  • We start the accumulation at the middle of the trajectory for all models without model-specific tuning.

Table R5 : Ablation on the starting step of CFG accumulation in CAM (SD3.5 at r=0.7r=0.7 with 40 steps, Wan2.1-1.3B at r=0.5r=0.5 with 50 steps). The starting steps correspond to steps 0, 10, 20, and 30 on SD3.5 and steps 0, 12, 25, and 37 on Wan2.1-1.3B.

Starting stepSD3.5 FID ↓\downarrowSD3.5 sFID ↓\downarrowWan2.1 VBench ↑\uparrow
First step33.9056.7067.04
1/4 of the trajectory32.4157.2070.04
1/2 of the trajectory (middle, ours)32.0553.9270.48
3/4 of the trajectory32.9458.5369.56
Reviewer_ih3vOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ComE-ToMe, a training-free method that compensates for attention errors caused by token merging. It combines calibrated window attention (CWA) for local error correction with compensation-aware merging (CAM) for token allocation.

Critical strengths
  • CWA is the main contribution. It corrects the output after merging rather than only improving token selection. Adding CWA to Imp-ToMe reduces FID from 45.72 to 38.36 at r=0.5 (Table 4).

  • Experiments show consistent quality improvements across PixArt-α, SD3.5, and Wan2.1, with little added latency over Imp-ToMe. The calibration ablation also shows a clear advantage over ordinary window attention (Table 10).

Critical weaknesses
  • The error analysis is limited to one PixArt-α block at r=0.7. The cosine similarities and norm ratios in Table 11 do not directly show the remaining full error after compensation. Proposition 1 also establishes optimality in logit space, not accuracy of the softmax denominator. Direct measurements of residual error and denominator error across more blocks and models would better support the approximation.

  • The practical benefit varies across settings. On PixArt-α, latency falls by only 1.7% at r=0.3. On Wan2.1, r=0.5 reduces latency by 25.8%, but VBench drops from 77.31 to 70.48 (Table 3). Video quality remains substantially below the full-token baseline.

Questions for authors

See weaknesses please.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ih3v (1)

We thank the reviewer for the careful reading and for recognizing CWA as the main contribution of our work. We respond to each weakness below, and the corresponding changes are marked in blue in the revised manuscript.

W1 : The error analysis covers only one PixArt-α\alpha block at r=0.7r=0.7, and it does not directly measure the residual error after compensation or the accuracy of the softmax denominator.

  • Thank you for this valuable suggestion. Following the suggestion, we have extended Table 19 of Appendix D.4 (Table 11 in the original manuscript) to more blocks, reduction ratios, and models (Table R1). Instead of cosine similarities and norm ratios, we measure two errors directly. Let Δf\Delta f be the merging error of the attention output, that is, the difference between the outputs with and without merging, and let Δf^W\Delta \hat f_W be the local correction estimated by CWA. The residual error after compensation is

ϵres=∥Δf−Δf^W∥∥Δf∥.\epsilon_{\text{res}} = \frac{\Vert \Delta f - \Delta \hat f_W\Vert}{\Vert \Delta f\Vert}.

The denominator error ϵZ\epsilon_Z is the relative error between the softmax denominator ZZ of a query, which sums the exponentiated logits over all keys, and its approximation Z^\hat Z by CWA, both of which we define below. We conduct this analysis on PixArt-α\alpha, SD3.5 Medium, FLUX.1-dev, and Wan2.1-1.3B. Even at the high reduction ratio r=0.7r=0.7, ϵres\epsilon_{\text{res}} averages 0.14 on the image models, indicating that CWA removes about 86% of the merging error. ϵZ\epsilon_Z stays between 0.06 and 0.19. This indicates that the errors in both the local correction and the denominator remain small across blocks, reduction ratios, and models.

  • We have also analyzed the error of the softmax denominator (Appendix A.3), which shows when the denominator approximation is reliable. We fix one query with logits zjz_j and divide the keys into the query window WW and the external windows EiE_i of NWN_W keys each. The softmax denominator is

Z=ZW+ZE,ZW=∑j∈Wezj,ZE=∑iZi,Zi=∑j∈Eiezj.Z = Z_W + Z_E, \qquad Z_W = \sum_{j \in W} e^{z_j}, \qquad Z_E = \sum_{i} Z_i, \qquad Z_i = \sum_{j \in E_i} e^{z_j}.

CWA keeps ZWZ_W and replaces the keys of each EiE_i with their average, whose logit is the mean logit zˉEi\bar z_{E_i}. The CWA denominator is therefore

Z^=ZW+Z^E,Z^E=∑iZ^i,Z^i=NWezˉEi.\hat Z = Z_W + \hat Z_E, \qquad \hat Z_E = \sum_{i} \hat Z_i, \qquad \hat Z_i = N_W e^{\bar z_{E_i}}.

Each per-window ratio equals a KL divergence:

log⁡ZiZ^i=log⁡1NW∑j∈Eiezj−zˉEi=KL(U∥p(i)),\log \frac{Z_i}{\hat Z_i} = \log \frac{1}{N_W}\sum_{j \in E_i} e^{z_j - \bar z_{E_i}} = \mathrm{KL}(U \Vert p^{(i)}),

where p(i)p^{(i)} is the attention of the query over the keys in EiE_i, and UU is the uniform distribution over the same keys. Since ZWZ_W cancels in Z−Z^Z - \hat Z, the relative denominator error factorizes as

ϵZ=Z−Z^Z=ZEZ⋅κ,κ=1−Z^EZE=∑iZiZE(1−e−KL(U∥p(i)))∈[0,1).\epsilon_Z = \frac{Z - \hat Z}{Z} = \frac{Z_E}{Z} \cdot \kappa, \qquad \kappa = 1 - \frac{\hat Z_E}{Z_E} = \sum_{i} \frac{Z_i}{Z_E} \left(1 - e^{-\mathrm{KL}(U \Vert p^{(i)})}\right) \in [0, 1).

ZE/ZZ_E/Z is the attention mass on the external windows, and κ\kappa is the fraction of this mass that the averaged keys miss. Therefore, ϵZ\epsilon_Z is small when the attention mass on the external windows (ZE/ZZ_E/Z) is small or when the attention within each external window is close to uniform (small κ\kappa). We report both factors in Table 20 of Appendix D.4.

Table R1 : Residual error after compensation and denominator error in the CWA blocks of each model.

Blockr=0.3r=0.3r=0.5r=0.5r=0.7r=0.7
ϵres\epsilon_{\text{res}}ϵZ\epsilon_Zϵres\epsilon_{\text{res}}ϵZ\epsilon_Zϵres\epsilon_{\text{res}}ϵZ\epsilon_Z
PixArt-α\alpha150.0700.150.1120.140.1520.12
180.0790.160.1090.130.1340.11
200.0610.130.0890.120.1120.10
SD3.5 Medium130.0300.090.0630.080.0910.07
150.0450.100.0870.090.1190.08
170.0450.070.0640.060.0760.06
FLUX.1-dev360.1040.130.1480.120.1760.10
430.1120.130.1580.110.1850.10
440.1080.130.1550.120.1820.11
Wan2.1-1.3B120.1250.130.2130.110.2650.09
150.1770.190.2780.160.3390.13
AuthorsOpenReview ↗

Response to Reviewer ih3v (2)

W2 : The practical benefit varies across settings. Latency falls by only 1.7% on PixArt-α\alpha at r=0.3r=0.3, and VBench on Wan2.1 drops from 77.31 to 70.48 at r=0.5r=0.5.

  • Thank you for this practical point. We agree that the practical benefit of ComE-ToMe depends on the setting. To examine this dependence, we have additionally evaluated ComE-ToMe on the larger Wan2.1-14B (Table 10 of Appendix C.4). The practical reduction ratio differs across models: r=0.5r=0.5 for the image models and Wan2.1-14B, and r=0.3r=0.3 for the smaller Wan2.1-1.3B. We explain the two cases below.
  • (1) We agree that ComE-ToMe reduces latency only slightly on PixArt-α\alpha at r=0.3r=0.3. Other ToMe-based methods show a similarly small reduction at this ratio (Table 1). The latency reduction grows with the reduction ratio. At r=0.5r=0.5, ComE-ToMe reduces latency by 17.3% on PixArt-α\alpha, and its FID remains nearly identical to that of the full-token model. We therefore regard r=0.5r=0.5 as the practical operating point for PixArt-α\alpha, and r=0.3r=0.3 as a quality-oriented setting.
  • (2) We agree that ComE-ToMe on Wan2.1-1.3B at r=0.5r=0.5 remains well below the full-token model. ComE-ToMe, however, loses less quality than other merging methods at comparable latency, and its quality drop is much smaller on the larger Wan2.1-14B. Table R2 compares the VBench drop from the full-token model between ComE-ToMe and Imp-ToMe, the strongest baseline. The results on Wan2.1-1.3B and Wan2.1-14B are from Table 4 and Table 10 of Appendix C.4, respectively. All training-free merging methods lose quality at high reduction ratios, and ComE-ToMe loses the least. On Wan2.1-1.3B, ToMeSD and ToMA drop even more than Imp-ToMe (Table 4). ComE-ToMe reduces the quality drop of Imp-ToMe by 35-54% across both models and both ratios.
  • On Wan2.1-14B at r=0.5r=0.5, the drop of ComE-ToMe is less than half of that of Imp-ToMe (2.78 vs. 6.04). ComE-ToMe at r=0.5r=0.5 even outperforms Imp-ToMe at r=0.3r=0.3 in VBench (78.05 vs. 77.90) and runs faster (341.0 s vs. 385.4 s). This indicates that ComE-ToMe provides a practical speed-quality trade-off on large video models, where acceleration is most needed.

Table R2 : VBench drop from the full-token model. Latency Δ\Delta is that of ComE-ToMe, and Reduction is the relative reduction of the drop.

ModelrrLatency Δ\DeltaImp-ToMeOursReduction
Wan2.1-1.3B0.3-16.7%-4.67-2.3649%
0.5-25.8%-10.49-6.8335%
Wan2.1-14B0.3-14.2%-2.93-1.7939%
0.5-22.7%-6.04-2.7854%
AuthorsOpenReview ↗

Response to Reviewer ih3v (3)

Summary. We summarize how our responses address each comment.

  • W1 (error analysis). We have extended the error analysis to four models, three reduction ratios, and up to three CWA blocks per model (Tables 19 and 20). Instead of cosine similarities and norm ratios, we directly measure the residual error after compensation and the denominator error. Both errors remain small across blocks, reduction ratios, and models. Appendix A.3 shows that the denominator error is small when the attention is local or the external windows are near-uniform.
  • W2 (practical benefit). At r=0.5r=0.5, ComE-ToMe reduces latency by 17.3% on PixArt-α\alpha with nearly the same FID as the full-token model (Table 1). On Wan2.1-1.3B and Wan2.1-14B, ComE-ToMe reduces the quality drop of Imp-ToMe by 35-54% (Table R2). The quality drop of ComE-ToMe is much smaller on the larger Wan2.1-14B, where acceleration is most needed (Table 10).

We hope that these results address the concerns of the reviewer, and we would be glad to answer any further questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

Dear Area Chair and Reviewers,

We thank the area chair and the reviewers for the time and care they put into reviewing our submission.

In this work, we propose ComE-ToMe, a training-free method that compensates for the errors of token merging in diffusion transformers using calibrated window attention (CWA) and compensation-aware merging (CAM). The reviewers find that CWA corrects the merging error itself rather than only improving token selection (ih3v), that CWA is lightweight and plug-and-play for existing merging methods (R23i), and that the approach is well motivated and practical (UQ5b). They also find that ComE-ToMe consistently improves quality across PixArt-α\alpha, SD3.5, and Wan2.1 with little added latency (ih3v, R23i, UQ5b).

We have addressed every comment below and uploaded a revised manuscript, where all changes are marked in blue so that they are easy to locate. Table and figure numbers in our responses follow the revised version. The main revisions are summarized as follows:

  • Analysis of the error of the softmax denominator (Sec. 3.3 and Appendix A.3).
  • Extended error analysis across blocks, reduction ratios, and models, which replaces Table 11 of the original manuscript (Tables 19-21 in Appendix D.4).
  • Attention locality and rank-one structure on larger models (Table 17 in Appendix D.2).
  • Results on FLUX.1-dev (Table 3).
  • Results on Wan2.1-14B, 10-second videos, and temporal consistency (Tables 10-12 in Appendix C.4).
  • Quality-latency trade-off curves (Fig. 7).
  • Ablation studies on the CAM hyperparameters and the placement of CWA (Tables 14-16 in Appendix D.1).
  • More qualitative results on FLUX.1-dev and Wan2.1-14B (Figs. 15 and 16 in Appendix C.5).

We believe that the revision resolves the concerns raised, and we would be glad to answer any further questions during the discussion period.

Best regards,

The authors

AuthorsOpenReview ↗

Summary of the discussion

Dear SACs, ACs, and Reviewers,

We thank you for the constructive discussion. All three reviewers confirmed that our responses resolved their concerns, and all three now recommend acceptance.

Strengths noted by the reviewers.

  • CWA corrects the merging error itself, rather than only improving token selection, and "experiments show consistent quality improvements across PixArt-α, SD3.5, and Wan2.1, with little added latency over Imp-ToMe" (ih3v).
  • CWA is lightweight, training-free, and plug-and-play for existing merging methods, "showing strong versatility and practical value" (R23i).
  • ComE-ToMe is well motivated and shows "consistent improvements over merging baselines, supported by component ablations and direct error analysis" (UQ5b).

Reviewer concerns and our responses.

Reviewer ih3v

  • Error analysis on more blocks and models (W1). We directly measured the residual and denominator errors on four models, three reduction ratios, and multiple CWA blocks. Both errors remain small (Tables 19 and 20).
  • Practical benefit (W2). ComE-ToMe at a higher reduction ratio runs faster than the strongest baseline at a lower ratio, with better quality (Fig. 7). On both Wan2.1 models, it substantially reduces the quality drop of the strongest baseline (Tables 4 and 10).
  • Response: "Thanks for the rebuttal as well as the new results. My concerns are solved."

Reviewer R23i

  • Robustness of the CAM hyperparameters (W1, Q3). The default settings give the best results on PixArt-α, SD3.5, and Wan2.1-1.3B (Tables 14 and 15).
  • Temporal consistency (Q1). On Wan2.1-14B, ComE-ToMe stays close to the full-token model in all four temporal-consistency categories of VBench (Table 12).
  • Placement of CWA (Q2). CWA is most effective in the latter half of the merged layers. Applying CWA to all merged layers brings FID close to that of the full-token model, and our placement retains most of this gain at a much lower cost (Table 16).
  • Response: "My concerns have been addressed by the additional experiments, and I have no further questions."

Reviewer UQ5b

  • Error analysis across layers, ratios, and models (W1, Q1). The in-window error accounts for most of the merging error, and the residual error after CWA stays close to that of a local compensation using the exact in-window error (Table 19).
  • Accuracy of the softmax denominator (W1, Q2). We derived the conditions under which the approximation is reliable: the attention is local, or it is close to uniform within each external window (Appendix A.3). The measured denominator error remains small, and sharply peaked attention has little impact on the overall error (Tables 20 and 21).
  • Larger models and longer videos (W2, Q3). ComE-ToMe achieves the best quality among merging methods at every reduction ratio on FLUX.1-dev and Wan2.1-14B, and outperforms Imp-ToMe on 10-second videos (Tables 3, 10, and 11). Attention locality and the rank-one structure also hold on both models (Table 17).
  • Response: "The direct measurements of softmax denominator and compensation errors address my main concern about CWA. The FLUX and Wan2.1-14B results, including the longer videos, also make the quality–latency trade-off clearer."

We will incorporate all additional results and clarifications into the final version. We thank you again for your time and valuable feedback.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.