acceptodds
Under review as a conference paper at ICLR 2027

TORE: Token-Wise Taylor Order Selection via Decomposed Residual Error Estimation for Diffusion Acceleration

Abstract

Diffusion models excel at image and video generation, but iterative denoising incurs substantial inference latency. Recent token-adaptive acceleration methods select Taylor prediction orders for the full-model residual using shallow-layer proxy scores. However, candidate orders are evaluated on these proxy features but used to predict the full-model residual, creating a proxy–target mismatch: the proxy-preferred order may not minimize residual prediction error. To address this mismatch, we propose TORE (Taylor Order selection via Residual Error estimation), which selects token-wise Taylor orders based on estimated prediction errors for the full-model residual. Yet direct error evaluation requires future ground-truth residuals, which are unavailable during order selection. TORE therefore estimates prediction error by analyzing how errors arise in Taylor extrapolation. Specifically, it decomposes prediction error into truncation error and finite-difference derivative error, then estimates both contributions from cached residuals to preselect token-wise orders for upcoming skip steps. Extensive experiments show that TORE achieves speedup on FLUX.1-dev and on Wan2.1 while maintaining high generation quality and fidelity.

Primary area:
generative models
Keywords:
Diffusion transformers, inference acceleration, feature caching, token-wise Taylor order selection, residual error estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_K7UkOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TORE, a training-free diffusion acceleration method that selects token-wise Taylor orders using estimated prediction errors of the full-model residual. It estimates truncation and finite-difference derivative errors from cached residuals, addressing the mismatch between shallow-proxy scoring and residual prediction. Orders are preselected at full-computation steps, eliminating probes at skip steps. Experiments cover text-to-image, text-to-video, and class-to-image generation.

Critical strengths

[S1] Presentation

The paper is generally easy to follow. In particular, Figure 2 illustrates TORE’s improvement over shallow-proxy selection, while Figure 3 demonstrates the limitations of the shallow proxy, making the motivation easy to understand.

[S2] Clear Novelty

The order selection criterion and the elimination of probing at skip steps are clear distinctions from prior work.

Critical weaknesses

[W1] Limited Scope of Novelty

Token-wise selection of different Taylor predictors already exists in TAP. Moreover, the basic perspective of decomposing Taylor prediction error is already present in TaylorSeer. Therefore, several of the paper’s main contribution claims appear to require more restrained wording.

[W2] Limited Experimental Evidence

a) The experimental evaluation is not sufficiently extensive. For FLUX.1-dev, only 200 images are generated and evaluated, and no confidence intervals are reported. This makes it difficult to establish that TORE consistently outperforms the baselines in ImageReward and other metrics. Although I acknowledge that the 200-prompt DrawBench evaluation is a widely used protocol, I believe a more robust evaluation is needed. For example, RALU (CVPR 2026) and JiT (CVPR 2026) evaluate over 4,000 generated images, employing various metrics such as GenEval and T2I-CompBench.

b) The experimental setup is insufficiently described. It is unclear how many initial steps use full computation and how higher-order finite differences are handled before sufficient history becomes available.

Questions for authors

[Q1] Figure 7 and Table 4 demonstrate the effectiveness of adaptive selection, but they do not isolate the advantage of the proposed criterion over TAP’s proxy criterion. Please provide a comparison that keeps all other settings identical and changes only the selection criterion between TAP’s proxy and TORE’s criterion.

[Q2] What are the confidence intervals for the results in Table 1? Among the 200 images, on how many does TORE achieve the highest ImageReward score across the compared methods?

[Q3] If possible, include additional images for FLUX.1-dev and report the evaluation results calculated based on a total of 1,000 or more images.

[Q4] How many initial generation steps use full computation? How are higher-order finite differences handled before sufficient history becomes available?

[Q5] How do performance, latency, FLOPs, and peak VRAM usage vary with M? In particular, since finite differences from order 0 through M+1 are cached, peak VRAM usage may increase with M. Does this occur in practice?

6 comments
AuthorsOpenReview ↗

Response to reviewer

We have uploaded a revised manuscript with additional experiments and analyses in the Appendix, as referenced in our responses below.

For weakness 1: We thank the reviewer for this suggestion. As illustrated in Fig. 2, TAP selects Taylor orders using shallow-layer proxy errors, which may not reflect the errors of the actual prediction target. This proxy–target mismatch can lead to suboptimal order selection, as demonstrated in Fig. 3. Our main contribution lies in the order-selection criterion. Specifically, starting from the Taylor forecasting formulation, we decompose residual prediction error into truncation error and finite-difference derivative approximation error. We then derive computable estimates of both components from cached full-model residuals (Eqs. 8–10). By minimizing their combined estimate, TORE selects the Taylor order for each token and prediction horizon without requiring future ground-truth residuals. Regarding TaylorSeer, it does not provide a computable error estimation criterion for adaptive order selection during inference. TORE extends this analysis into a practical selection criterion directly aligned with the residual prediction target. Detailed derivations are provided in Appendix B.

For weakness 2 (a) and question 3: We thank the reviewer for this suggestion. We would like to clarify that our original DrawBench evaluation uses 200 prompts with five random seeds per prompt, totaling 1,000 generated images per method. To further strengthen the evaluation, we conducted additional experiments on GenEval, following the evaluation protocol used by JiT (CVPR 2026), with 553 prompts and four random seeds per prompt, totaling 2,212 images per method. Together with DrawBench, our evaluation now covers 3,212 generated images per method across two benchmarks, providing broader evidence of generation quality and efficiency.

MethodConfigurationGenEval ↑Latency (s) ↓Speedup ↑FLOPs (T) ↓
TaylorSeerN=3, O=20.66129.7902.35×1340.37
HiCacheN=3, O=20.65569.7182.36×1340.18
TC-PadéN=6, θ=1.00.66968.6222.66×1265.40
TAPN=30.65728.3282.76×1339.28
TOREN=5, M=60.68067.4063.10×1190.53

For weakness 2 (b) and question 4: We thank the reviewer for this question. To balance warm-up overhead with the availability of higher-order finite differences, TORE determines the number of initial full-computation steps based on MM, using W=max⁡(2,⌊M/2⌋+1)W=\max(2,\lfloor M/2\rfloor+1). Similar to the short warm-up strategy used in TaylorSeer (ICCV 2025), TORE progressively accumulates residual history to support higher-order forecasting. During the early stages, candidate orders are restricted by the available finite differences. This strategy enables forecasting without waiting for all high-order differences to become available, while progressively expanding the candidate orders as the cache grows.

For question 1: We thank the reviewer for this suggestion. To directly evaluate the advantage of TORE's criterion over TAP's shallow-proxy criterion, we conducted controlled experiments where only the order-selection criterion differs. Specifically, both methods extrapolate from the same full-computation anchor along a shared generation trajectory, using identical input states, residual caches, candidate Taylor predictors, and prediction distances. We compare the true residual prediction MSE of TAP, TORE, and the residual oracle, which selects the lowest-error candidate order for each token. We also report the relative prediction error gap to this oracle. TORE achieves lower prediction errors and smaller oracle gaps across all three settings and the advantage becomes more pronounced under longer forecasting intervals, supporting the effectiveness of TORE's residual-oriented selection criterion.

SettingHorizon kTAP True Residual MSE ↓TORE True Residual MSE ↓Oracle True Residual MSE ↓TAP Oracle Gap (%) ↓TORE Oracle Gap (%) ↓
N=3, M=41751.121728.406633.96818.4814.90
21610.3591478.3531172.27637.3726.11
N=5, M=611006.5791004.921905.87811.1210.93
22056.7851803.6441547.79832.8816.53
32939.3012009.0411643.35178.8622.25
44427.4392285.3552046.538116.3411.67
N=8, M=51425.632370.012329.88129.0312.17
21418.2281023.282901.99857.2313.45
33194.0512158.4171929.43565.5411.87
45433.6682997.4042359.719130.2727.02
57710.1133624.4092809.542174.4329.00
610282.8434181.6253191.315222.2131.03
713292.1194689.7503620.918267.0929.52
AuthorsOpenReview ↗

Response to reviewer

For question 2: We thank the reviewer for this suggestion. In our original DrawBench evaluation, we used five different random seeds for each of the 200 prompts, generating five images per prompt and a total of 1,000 images per method. However, we did not clearly specify this evaluation protocol or report confidence intervals in Table 1. We now provide 95% confidence intervals for all generation quality metrics, as presented in the table below. For the per-image ImageReward comparison, TORE achieves the highest score among the accelerated methods on 260, 275, and 250 out of 1,000 images (26.0%, 27.5%, and 25.0%) for the three Table 1 configurations, respectively: ((N,M)=(3,4),(5,6),(8,5)).

MethodImageReward ↑CLIP ↑PSNR ↑SSIM ↑LPIPS ↓
Full1.0414 (±0.1622)28.010 (±0.981)———
60% steps1.0069 (±0.1609)27.440 (±0.934)19.060 (±0.713)0.7401 (±0.0249)0.2477 (±0.0253)
50% steps1.0126 (±0.1618)27.607 (±0.894)17.408 (±0.652)0.6947 (±0.0254)0.3102 (±0.0253)
40% steps0.9418 (±0.1720)27.505 (±0.821)16.363 (±0.563)0.6604 (±0.0246)0.3599 (±0.0238)
TaylorSeer (N=2,O=1)1.0494 (±0.1557)27.873 (±0.909)22.142 (±0.846)0.8043 (±0.0186)0.1575 (±0.0175)
HiCache (N=2,O=2)†0.9913 (±0.1733)27.668 (±0.926)23.953 (±0.919)0.8109 (±0.0231)0.1394 (±0.0212)
TC-Padé (N=3,θ=1.0)0.9968 (±0.1628)27.608 (±0.951)23.035 (±0.838)0.8384 (±0.0218)0.1546 (±0.0156)
TAP (N=2)1.0224 (±0.1592)27.826 (±0.903)21.368 (±0.848)0.7928 (±0.0187)0.1817 (±0.0175)
TORE (N=3,M=4)1.0462 (±0.1622)28.043 (±0.947)24.450 (±0.977)0.8291 (±0.0196)0.1188 (±0.0163)
TaylorSeer (N=3,O=2)1.0266 (±0.1616)27.919 (±0.954)19.297 (±0.738)0.7345 (±0.0218)0.2391 (±0.0211)
HiCache (N=3,O=2)1.0275 (±0.1597)27.945 (±0.921)19.554 (±0.726)0.7384 (±0.0218)0.2319 (±0.0216)
TC-Padé (N=6,θ=1.0)0.9808 (±0.1679)27.509 (±1.011)22.034 (±0.808)0.8043 (±0.0246)0.2067 (±0.0194)
TAP (N=3)0.9781 (±0.1677)27.559 (±0.940)18.683 (±0.678)0.7281 (±0.0224)0.2588 (±0.0219)
TORE (N=5,M=6)1.0364 (±0.1628)28.011 (±0.974)24.387 (±0.971)0.8210 (±0.0209)0.1302 (±0.0174)
TaylorSeer (N=5,O=2)1.0528 (±0.1607)27.819 (±0.984)16.657 (±0.531)0.6546 (±0.0244)0.3600 (±0.0212)
HiCache (N=4,O=2)1.0252 (±0.1580)27.879 (±0.925)18.023 (±0.648)0.6893 (±0.0238)0.2942 (±0.0221)
TC-Padé (N=8,θ=1.4)0.9469 (±0.1680)27.430 (±0.989)20.644 (±0.729)0.7679 (±0.0249)0.2728 (±0.0217)
TAP (N=5)0.9431 (±0.1669)27.585 (±0.973)15.583 (±0.600)0.6410 (±0.0218)0.3998 (±0.0206)
TORE (N=8,M=5)1.0308 (±0.1678)28.020 (±0.947)21.699 (±0.824)0.7582 (±0.0236)0.2126 (±0.0215)

For question 5: We thank the reviewer for this question. We conducted a sensitivity analysis on FLUX.1-dev by varying (M) from 0 to 6 while fixing (N=7). We evaluate generation fidelity, latency, FLOPs, and peak VRAM. As illustrated in Appendix Fig.10 (page 15), increasing MM introduces additional computational and memory overhead but also improves generation fidelity, with diminishing gains at higher orders. Importantly, MM only determines the maximum candidate order, while the actual prediction order is adaptively selected for each token. Therefore, increasing MM does not necessarily require higher-order Taylor extrapolation for all tokens. These observations highlight the trade-off between prediction fidelity and computational cost, motivating a moderate choice of MM that balances both.

I thank the authors for the detailed response and the additional experiments. However, several of my concerns remain unresolved, as detailed below.


[W1] Limited Scope of Novelty

Just to clarify, my concern in W1 was not whether TaylorSeer [1] provides a computable criterion for adaptive order selection. I agree that it doesn't. My concern was that the error decomposition itself, which the paper presents as one of its main contributions, already appears in TaylorSeer. This point was not addressed in the response, so let me elaborate.

The decomposition is already in TaylorSeer (Sec. 3.2, "Error Bounds Analysis", Eqs. (11)–(12)). TaylorSeer bounds the order-mm prediction error as a truncation term, Mm+1(m+1)!∣k∣m+1\frac{M_{m+1}}{(m+1)!}|k|^{m+1}, plus a finite-difference term, ∑i=1mCii!⋅∣k∣i∣N∣i−1\sum_{i=1}^{m} \frac{C_i}{i!} \cdot \frac{|k|^i}{|N|^{i-1}}. It also discusses the same trade-off that motivates TORE (paper lines 85–92): a higher order reduces the truncation error but adds finite-difference error that grows with NN.

TORE's estimator is essentially TaylorSeer's bound, with the unknown constants filled in by cached finite-difference norms. The truncation estimate (Eq. (8)) is TaylorSeer's first term with Mm+1≈dm+1/Nm+1M_{m+1} \approx d_{m+1}/N^{m+1}. The finite-difference estimate (Eq. (10)) can be rewritten as ∑i=1mi di+12N⋅kii! Ni−1\sum_{i=1}^{m} \frac{i\, d_{i+1}}{2N} \cdot \frac{k^i}{i!\, N^{i-1}}, which is TaylorSeer's second term with Ci=i di+12NC_i = \frac{i\, d_{i+1}}{2N}.

However, the paper does not mention this connection. TaylorSeer is cited only as a forecasting method (paper lines 56 and 130), while the decomposition is claimed as the paper's own contribution (paper lines 101–104 and Sec. 3.3.2). The response itself says that TORE extends "this analysis," which suggests the analysis already exists in TaylorSeer.

So I see the error decomposition as prior work. In my view, TORE's contribution is narrower: a concrete estimate of TaylorSeer's constants, and its use as a token-wise selection score.

[1] Liu, Jiacheng, et al. "From reusing to forecasting: Accelerating diffusion models with TaylorSeers." ICCV, 2025.


[W2-a, Q2] Statistical Significance of Quality Gains

Looking at the new confidence intervals, the gains in ImageReward and CLIP Score do not seem statistically meaningful. The differences fall within the intervals, and TORE is behind TaylorSeer in ImageReward in two of the three settings. TORE also wins on only 25.0–27.5% of images, which is close to chance. This is not a minor issue of trailing on one metric: for T2I, ImageReward is the only reported metric that reflects image quality itself, whereas PSNR, SSIM, and LPIPS all measure similarity to the full model. As it stands, the evidence supports better fidelity to the full model, but the contribution to image quality remains unclear.


[Q4] Availability of Higher-Order Differences

I'm not sure how often TORE actually selects high orders. Using an order up to MM requires M+2M+2 cached full steps, which takes up much of the trajectory when NN is large, and the response also mentions that candidate orders are restricted early on. In Fig. 8, the finite-difference error already exceeds the truncation error at order 1. If most selected orders are 0–2, the large MM values in the main tables are mostly nominal. Could the authors show the distribution of selected orders for the main configurations?


[Q5] Sensitivity to MM

The response presents results that were not actually measured. It says that fidelity, latency, FLOPs, and peak VRAM were evaluated, but according to the caption of Fig. 10, all curves other than peak VRAM "are illustrative and are not experimental measurements." Illustrative curves should not be presented as results in a rebuttal, and the conclusion drawn from them (fidelity improves with MM with diminishing returns) is therefore unsupported. The VRAM measurement does address my memory concern, but the other panels need to be either actually measured or removed.

Reviewer_qHVrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TORE chooses a Taylor prediction order separately for each token and skipped denoising step. It estimates prediction error directly from cached full-model residuals, combining Taylor truncation error with finite-difference derivative error. Orders are chosen at full-computation steps and reused without additional probing during skipped steps. Results on FLUX, Wan2.1, and DiT are promising. The main unresolved questions are whether the score reliably selects good orders in the deployed setting, whether the finite-difference correction adds value, and whether adaptation is preferable to simpler predictors at the same cost.

Critical strengths
  1. Order selection uses the actual prediction target. Selecting from full-model residual history avoids relying on a shallow feature whose preferred order may differ from that of the residual being forecast. Figure 3 makes the mismatch concrete: the shallow proxy favors order two even when the residual oracle prefers zero or one for 87.65% of evaluated tokens. This is a clear motivation for revisiting the selection criterion.
  2. The estimator accounts for two sources of prediction error. The method includes both the omitted Taylor terms and the error introduced by finite-difference derivatives. This explains why a higher order need not produce a better forecast. Choosing separate order maps for different prediction horizons is also sensible, since an order that works near the anchor may become inaccurate farther away.
  3. The reported gains extend across several generation tasks. On FLUX, one operating point reaches 3.14× speedup with 24.387 dB PSNR and 0.1302 LPIPS, compared with 3.00×, 22.034 dB, and 0.2067 for TC-Padé. Wan2.1 and DiT results provide evidence beyond a single image model. Precomputing the order maps also avoids the need for an extra probe at each skipped step.
Critical weaknesses
  1. The central order-selection claim needs validation in the deployed regime. The score approximates vector prediction error using leading terms and sums of magnitudes. These approximations discard cancellation and may affect different orders differently. Figure 3 demonstrates a problem with the previous proxy but does not show that TORE chooses near-optimal orders on those same tokens. Figure 7 is useful positive evidence, including a 7% error reduction, but only tests orders up to two without recursive error propagation. The higher-order settings in the main results need a direct comparison with the best candidate order on states already affected by caching.
  2. The additional value of the finite-difference correction is unclear. The component ablation turns adaptive selection on or off as a whole. It does not test whether adding the derivative-error correction improves upon truncation-only scoring, although this balance is central to the proposed estimator. Comparing the two scores with identical candidate forecasts, available history, and FULL/skip locations would isolate the correction’s contribution. Both excess prediction error relative to the best order and final generation fidelity would be informative, along with sensitivity to the correction’s weight.
  3. Adaptive selection is not compared with an equally resourced simpler predictor. Table 4 shows a substantial fidelity gain when adaptation is added to residual prediction: PSNR rises from 17.529 to 20.115 dB. It also raises latency from 4.56 to 4.90 s and FLOPs by roughly 7%. This establishes a useful tradeoff, but not whether adaptation is the best use of those extra resources. The complete method’s favorable baseline comparisons do not isolate this question. A quality–cost sweep against validation-tuned fixed orders, allowing each method to choose its full-step interval, would establish the incremental benefit.
Questions for authors
  1. On identical candidate forecasts, how do TORE and the shallow-proxy selector compare with the oracle in both the diagnostic setting and a main-table setting with maximum order four to six? Please measure excess prediction error and order-selection accuracy by horizon, including trajectories affected by earlier cache errors.
  2. How does the combined score compare with truncation-only scoring when the candidate forecasts, history, and FULL/skip locations are fixed? Does the finite-difference correction consistently improve prediction and final fidelity, and how sensitive is that gain to its weight?
  3. Does adaptation outperform tuned fixed-order predictors at matched latency or FLOPs? Please specify the fallback orders before enough historical FULL residuals are available and include any warmup computation in this comparison.
3 comments
AuthorsOpenReview ↗

Response to review

We thank the reviewer for their valuable feedback. We have uploaded a revised manuscript with additional experimental results and analyses included in the Appendix. In the following responses, we address the reviewer's concerns and refer to the corresponding results in the revised manuscript.

For weakness 1 and question 1: We thank the reviewer for this suggestion. We conducted additional controlled experiments comparing TORE, TAP, and the residual oracle under identical input states, residual caches, candidate Taylor predictors, and prediction distances. The evaluation is performed on actual TORE-generated trajectories, including states affected by previous caching errors. Additional full-model evaluations provide ground-truth residuals without altering the generation process. We evaluate three settings, ((N,M)=(3,4),(5,6),(8,5)), covering different forecasting horizons and higher-order configurations. We report oracle order agreement and relative excess prediction MSE at each horizon. Agreement measures the percentage of tokens whose selected orders match the residual oracle. Excess MSE denotes the relative increase in true residual prediction MSE over the oracle. TORE achieves higher oracle agreement and lower excess prediction error at every evaluated horizon, with larger advantages at longer forecasting distances.

SettingHorizon kTAP Agreement (%) ↑TORE Agreement (%) ↑TAP Excess MSE (%) ↓TORE Excess MSE (%) ↓
N=3, M=4136.7741.4418.4814.90
234.7354.7937.3726.11
N=5, M=6134.3537.4311.1210.93
220.9943.0732.8816.53
322.6757.5478.8622.25
420.4071.33116.3411.67
N=8, M=5132.5033.4029.0312.17
210.0437.7857.2313.45
35.3148.0265.5411.87
46.4363.11130.2727.02
55.0667.78174.4329.00
65.2668.69222.2131.03
77.2471.25267.0929.52

For weakness 2 and question 2: The two error estimates play complementary roles. As illustrated in Appendix Fig.8 (page 14), the estimated truncation error generally decreases with increasing Taylor order, while the finite-difference error increases. Consequently, truncation-only selection tends to favor higher orders, whereas finite-difference-only selection favors order zero. TORE balances these competing effects to determine suitable prediction orders. We conducted an additional ablation study to evaluate truncation-only scoring, finite-difference (FD)-only scoring, and their combination. Truncation-only scoring generally favors higher Taylor orders, while FD-only scoring favors order zero, as the estimated FD error increases monotonically with order. Considering either component alone therefore leads to a biased order-selection criterion. TORE combines both terms to balance the benefits of higher-order prediction against finite-difference approximation errors.

MethodLatency (s) ↓Speedup ↑FLOPs (T) ↓ImageReward ↑CLIP ↑PSNR ↑SSIM ↑LPIPS ↓
Trunc-only4.873.34×1110.24511.025027.81017.5110.68120.3166
FD-only4.673.48×1065.72681.021027.79317.3280.67540.3254
TORE4.903.32×1116.12441.041427.97120.1670.74520.2254

Second, we performed a sensitivity analysis by varying the FD weight λ\lambda in$\hat{E}_m = \hat{E}_m^{\mathrm{trunc}} + \lambda \hat{E}_m^{\mathrm{FD}}.$ Here, λ=0\lambda=0 corresponds to truncation-only scoring, and λ=1\lambda=1 is the original TORE criterion. These results show that adding the FD correction improves upon truncation-only scoring, while overly increasing its influence degrades performance. This supports our design choice that effective order selection requires balancing truncation error and finite-difference approximation error, rather than relying on either effect alone.

FD weight λ\lambdaImageReward ↑CLIP ↑PSNR ↑SSIM ↑LPIPS ↓
01.025027.81017.5110.68120.3166
0.51.041427.95319.3770.72110.2427
11.046627.97120.1670.74520.2254
21.029927.92618.5840.70430.2712
AuthorsOpenReview ↗

Response to reviewer

For weakness 3 and question 3: We thank the reviewer for this suggestion. To further evaluate the benefit of adaptive order selection, we conducted a quality–latency comparison between TORE and fixed-order Taylor predictors across different computational budgets. Specifically, we vary the full-computation interval (N∈{2,3,4,5,6,8,10,12})(N \in \lbrace 2,3,4,5,6,8,10,12 \rbrace) to obtain different latency levels. For this controlled experiment, we adopt a specific warm-up strategy to ensure that sufficient historical residuals are available before forecasting begins. For example, a fixed second-order predictor uses three warm-up steps, while TORE accumulates sufficient history to construct the required finite differences. No lower-order fallback is needed after warm-up, and all warm-up computations are included in the reported latency. As shown in Appendix Fig.9 (page 15), TORE achieves a more favorable PSNR–latency trade-off than the evaluated fixed-order predictors. At comparable latency, TORE consistently achieves higher reconstruction fidelity. Interestingly, no single fixed Taylor order performs best across all computational budgets. Higher-order predictors can be beneficial at shorter forecasting intervals, but their performance deteriorates as the interval increases due to accumulated prediction errors. This observation further motivates adaptive order selection. By selecting suitable orders for individual tokens, TORE maintains higher generation fidelity across different acceleration levels.

Reviewer_qHVrOpenReview ↗

Response after rebuttal

Thank you for the responses and additional experiments. They clarify several points, but my concerns about the limited contribution and the strength of the empirical evidence remain. The rebuttal does not sufficiently change my overall assessment, so I will maintain my rating of 2.

Reviewer_Z9vyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TORE, a training-free diffusion model acceleration method that reduces computational overhead by predicting the full model residuals of skipped denoising steps. The main contribution of this paper is a new error proxy metric directly targeting residual prediction, replacing the shallow-feature-based proxy metric in TAP. By combining truncation error and finite-difference approximation error decomposed from Taylor approximation, TORE constructs a a new error proxy metric for candidate predictors of orders 0 to M without computing the true residuals of future steps, thereby guiding order selection. Experimental results on several benchmarks show that, compared with TaylorSeer, HiCache, TC-Pade, and TAP, TORE achieves a better trade-off between generation quality and inference latency.

Critical strengths

The paper offers a new perspective on why increasing the Taylor approximation order fails to consistently enhance results. Through decomposing the approximation error into truncation and finite-difference components, the authors justify the presence of an optimal Taylor order rather than a monotonic improvement with higher orders.

Critical weaknesses

1.This paper should more clearly distinguish the existing motivation for adaptive order selection, which has been recognized by methods like TAP and does not provide a new research motivation, from its actual methodological innovation, namely the order-selection criterion based on estimates of truncation error and finite-difference approximation error and directly targeting residual prediction.

2.Existing experiments have not sufficiently validated the advantage of TORE’s proposed order-selection criterion over TAP’s existing method. There is still a lack of direct controlled experiments comparing the true residual prediction errors of TORE, TAP, and the residual oracle under the same experimental conditions. In addition, Table 4 does not separately evaluate order selection using only the truncation error estimate, only the finite-difference error estimate, and the combination of both error estimates.

Questions for authors

1.Can the authors conduct rigorous controlled experiments comparing TORE, TAP, and the residual oracle under the same residual cache, candidate predictor set, and prediction distance settings? Reporting the true residual prediction errors of the predictors selected by each method and their gaps to the oracle would help more directly validate the effectiveness of TORE’s proposed order-selection criterion.

2.When the residual cache has not yet accumulated enough complete computation steps, how does TORE initialize high-order finite differences? During the initial warm-up stage, does TORE restrict the candidate Taylor orders according to the number of currently available historical residuals, or do they adopt another initialization strategy?

3.Can the authors further analyze the respective contributions of the two error components in TORE’s order-selection criterion? This would help clarify whether the proposed error decomposition and the combination of the two error components indeed contribute to more effective order selection.

3 comments
AuthorsOpenReview ↗

response to reviewer

For weaknesse 1 :We thank the reviewer for this suggestion. Our main contribution lies in how the optimal order is determined, rather than the motivation for adaptive selection itself. As discussed in Section 3.2, TAP selects Taylor orders using prediction errors on shallow-layer proxy features, which may not reflect the errors of the actual prediction target, namely the full-model residual. To address this mismatch, TORE introduces a residual-oriented error estimation criterion derived from the Taylor forecasting formulation. Specifically, we decompose the prediction error into truncation error and finite-difference derivative approximation error (Eq. 6), and derive computable estimates for both components (Eqs. 8–10). This allows TORE to estimate the residual prediction error of each candidate order using cached residuals, without requiring future ground-truth residuals. Therefore, the key distinction from TAP is the proposed error estimation criterion, which guides order selection based on the predicted accuracy of the actual residual target rather than a shallow proxy.

For weakness 2.1 and question 1: We thank the reviewer for this suggestion. We conducted additional controlled experiments on FLUX.1-dev, where TAP and TORE use identical input states, residual caches, candidate Taylor predictors, and prediction distances. Only the order-selection criterion differs. The residual oracle selects the candidate with the lowest true prediction error, providing a reference for evaluating both methods. We report the true residual prediction MSE and the relative gap to the oracle under three forecasting configurations. Oracle Gap denotes the relative increase in true residual prediction MSE over the oracle. TORE has greater relative improvements under longer forecasting intervals.

SettingHorizon kTAP True Residual MSE ↓TORE True Residual MSE ↓Oracle True Residual MSE ↓TAP Oracle Gap (%) ↓TORE Oracle Gap (%) ↓
N=3, M=41751.121728.406633.96818.4814.90
21610.3591478.3531172.27637.3726.11
N=5, M=611006.5791004.921905.87811.1210.93
22056.7851803.6441547.79832.8816.53
32939.3012009.0411643.35178.8622.25
44427.4392285.3552046.538116.3411.67
N=8, M=51425.632370.012329.88129.0312.17
21418.2281023.282901.99857.2313.45
33194.0512158.4171929.43565.5411.87
45433.6682997.4042359.719130.2727.02
57710.1133624.4092809.542174.4329.00
610282.8434181.6253191.315222.2131.03
713292.1194689.7503620.918267.0929.52

For weakness 2.2 and question 3: The two error estimates play complementary roles. As illustrated in Appendix Fig.8 (page 14), the estimated truncation error generally decreases with increasing Taylor order, while the finite-difference error increases. Consequently, truncation-only selection tends to favor higher orders, whereas finite-difference-only selection favors order zero. TORE balances these competing effects to determine suitable prediction orders. We conducted an additional ablation study to evaluate truncation-only scoring, finite-difference (FD)-only scoring, and their combination. Truncation-only scoring generally favors higher Taylor orders, while FD-only scoring favors order zero, as the estimated FD error increases monotonically with order. Considering either component alone therefore leads to a biased order-selection criterion. TORE combines both terms to balance the benefits of higher-order prediction against finite-difference approximation errors.

MethodLatency (s) ↓Speedup ↑FLOPs (T) ↓ImageReward ↑CLIP ↑PSNR ↑SSIM ↑LPIPS ↓
Trunc-only4.873.34×1110.24511.025027.81017.5110.68120.3166
FD-only4.673.48×1065.72681.021027.79317.3280.67540.3254
TORE4.903.32×1116.12441.041427.97120.1670.74520.2254

For question 2 :We thank the reviewer for this question. To balance warm-up overhead with the availability of higher-order finite differences, TORE determines the number of initial full-computation steps based on MM, using W=max⁡(2,⌊M/2⌋+1)W=\max(2,\lfloor M/2\rfloor+1). Similar to the short warm-up strategy used in TaylorSeer (ICCV 2025), TORE progressively accumulates residual history to support higher-order forecasting. During the early stages, candidate orders are restricted by the available finite differences. This strategy enables forecasting without waiting for all high-order differences to become available, while progressively expanding the candidate orders as the cache grows.

AuthorsOpenReview ↗

response to reviewer

Note: We have uploaded a revised manuscript with additional experimental analyses in Appendix C. Please refer to the updated PDF for the corresponding figures and details.

Reviewer_Z9vyOpenReview ↗

Reply to the authors' rebuttal

Thank you for the detailed response and additional experiments.The authors' experimental response regarding TORE, TAP, and the residual oracle addresses my main concern. By keeping settings the same, the experiments provided by the authors demonstrate the advantage of the TORE method over the TAP method on the Residual MSE metric. However, the goal of image generation models is to generate good images, so it would be more convincing to compare the results of TORE and TAP under the same settings using image quality metrics that describe result quality more directly (e.g., the metrics in Table 1). In addition, the authors' ablation experiments and clarification regarding cold start also address my other concerns. However, from the authors' response, I understand that the main innovation of the paper is merely a new method for selecting the order of Taylor expansion, and the novelty of this approach is limited. Therefore, I will maintain my ratio of 2.

Comments on the paper

1 comment

Dear Reviewers,

As we are approaching the end of reviewer-author discussions, please go through authors' responses and raise questions that you would like authors to reply.

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.