acceptodds
Under review as a conference paper at ICLR 2027

Mask-Aware Efficient Inference-Time Scaling for Image Inpainting

Abstract

Inference-time scaling has recently emerged as an effective paradigm for improving generation quality by allocating additional computation at test time. However, image inpainting remains largely unexplored in this setting, where complete-trajectory scaling is inefficient and structurally mismatched: it repeatedly scales the observed unmasked context and imposes a uniform computation budget on masked tokens with different refinement difficulties. Therefore, we identify two spatially non-uniform requirements for inpainting: inference-time scaling should exclude the observed unmasked region, while accounting for the varying difficulty across masked tokens. To this end, we propose a training-free, mask-aware framework for efficient inference-time scaling in image inpainting. For the unmasked context, Mask-Aware Initial Noise Construction and Anchor-Cached Background KV Cache remove observed-context redundancy by aligning background trajectories and reusing anchor background key/value states. For the masked region, Token-Aware Temporal Velocity Cache adapts masked-region updates by reusing velocity predictions for stabilized tokens and focusing computation on uncertain ones. Experiments on EditBench and BrushBench show that our method preserves scaling gains while reducing cost by up to and over standard scaling.

Primary area:
generative models
Keywords:
Image Inpainting, Inference-Time Scaling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4HNZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training-free framework for efficient inference-time scaling in image inpainting. The method has three main components: (1) mask-aware initial noise construction, which shares background noise across Best-of-N candidates while varying noise in the target region; (2) a so-called anchor-cached background KV cache, which runs one anchor trajectory fully and reuses its background key/value states for later candidates while recomputing only the target region plus a small boundary band; and (3) a token-aware temporal velocity cache, which progressively removes stabilized dynamic tokens from active computation to be reused in their latest velocity predictions. The method is evaluated on EditBench and BrushBench using FLUX.1 Fill, and the reported results show near-Best-of-N quality with a substantially lower normalized compute cost, corresponding to speedups of roughly 4.4x-4.6x over standard multi-candidate scaling.

Critical strengths

The strongest aspect of the paper is its simplicity and being practical. The proposed modifications are training-free and can be applied at inference time to a strong pretrained inpainting model. The decomposition into shared background trajectories, reusable background context, and I find the adaptive temporal reuse to be intuitive, and the reported efficiency gains are quite substantial: the method reduces normalized compute to about 0.21-0.23 of vanilla Best-of-N while largely preserving its quality. This makes the method potentially useful in settings where multi-candidate inpainting is otherwise prohibitively expensive

Critical weaknesses
  1. The novelty of the background-KV component is rather limited and is not positioned sufficiently against very close prior work. In particular, KV-Edit (ICCV 2025) already caches background K/V states and, during denoising, processes only foreground tokens while concatenating their current K/V with cached background K/V. At the attention-computation level, I find it to be very close to the proposed Anchor-Cached Background KV mechanism. The main distinction I can tell is how the reusable background trajectory are obtained: KV-Edit obtains it from inversion, whereas this paper aligns Best-of-N candidates using shared background noise and takes the K/V states from an anchor candidate. LazyDiffusion also predates this work and establishes the broader idea of spending iterative computation only on the masked region, while retaining the global image context. Neither KV-Edit nor LazyDiffusion appears to be discussed or cited. As a result, I view the main novelty as the adaptation of this idea to cross-candidate inference-time scaling through background-trajectory alignment, rather than the background KV-cache mechanism itself. This, to me, substantially weakens the claimed novelty.

  2. It is not clear how the Token-Aware Temporal Velocity Cache handles tokens that have stopped changing. The paper says it skips these tokens and reuses their previous velocity predictions. But what happens to their keys and values in each attention layer? Other tokens still need these to attend to the skipped tokens. Do they keep the old keys and values, remove the tokens from attention, or do something else? These choices can affect speed, memory use, and image quality. The paper should explain this so others can reproduce the method and understand the reported savings.

Questions for authors

Clarification 1: When a dynamic token is declared stable by the Token-Aware Temporal Velocity Cache and removed from the active set, what then happens to that token's K/V states in every subsequent transformer block and denoising step? Are its most recently computed per-layer K/V states cached and concatenated with the K/V of active tokens, or is the token removed entirely from the attention context? If it is removed, please specify the resulting attention sequence and justify why the lost dynamic-token context is safe. If stale K/V is retained, please include this cache in the method description and in the memory/compute accounting. This answer could materially affect my assessment of both soundness and efficiency.

Clarification 2: Please clarify the technical novelty of the Anchor-Cached Background KV mechanism relative to KV-Edit. My reading is that both methods compute queries only for the editable/dynamic tokens while providing cached background K/V as attention memory; the main difference appears to be the provenance of the cache (an inversion trajectory in KV-Edit versus an aligned anchor candidate here). If there is a more fundamental difference, explaining it and positioning the contribution relative to KV-Edit and LazyDiffusion would materially affect my assessment of novelty.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4HNZ (1/3)

We thank the reviewer for acknowledging the practicality and substantial efficiency gains of our training-free method, and for highlighting the need to clarify both its novelty relative to closely related work and its handling of stabilized tokens.

W1: The novelty of the background-KV component is rather limited and is not positioned sufficiently against very close prior work. In particular, KV-Edit (ICCV 2025) already caches background K/V states, and LazyDiffusion predates the broader idea of focusing iterative computation on the masked region.

Q2: Please clarify the technical novelty of the Anchor-Cached Background KV mechanism relative to KV-Edit. My reading is that both methods compute queries only for the editable/dynamic tokens while providing cached background K/V as attention memory; the main difference appears to be the provenance of the cache (an inversion trajectory in KV-Edit versus an aligned anchor candidate here). If there is a more fundamental difference, explaining it and positioning the contribution relative to KV-Edit and LazyDiffusion would materially affect my assessment of novelty.

R1 (1/2) : Our key contribution is not merely to identify reusable computation, but to design the sampling process to make computation reusable. We thank the reviewer for pointing out KV-Edit [1] and LazyDiffusion [2], which are closely related to our work. We would like to clarify that our contribution goes beyond simply exploiting existing computational redundancy. Instead, we explicitly design the joint sampling structure across different candidates to actively create reusable intermediate computational states while preserving the diversity required for candidate exploration.

KV-Edit [1] supports image editing by constructing and reusing source-image-related background computational states, whereas our method actively establishes cross-candidate background computational reusability through structured sampling coupling. Specifically, KV-Edit primarily focuses on background preservation during image editing. It obtains background K/V states associated with the source image through an inversion-based editing process and uses them as attention context for foreground generation. Its computation reuse mainly relies on caching and utilizing existing background states, rather than treating sampling correlations across different generation trajectories as design variables to actively establish consistency among cross-candidate intermediate computational states. In contrast, we move the consideration of computation reuse to the initial noise sampling stage. By explicitly controlling the cross-candidate sampling correlation structure, we improve the alignment of background denoising trajectories, enabling reliable reuse of background computation that would otherwise be difficult to share directly across candidates. More importantly, this design preserves the marginal distribution of each candidate's initial noise while retaining independent stochastic exploration within the masked region, allowing candidate diversity and computation sharing to be achieved jointly.

LazyDiffusion [2] reduces redundant computation through a specially designed and trained model architecture, whereas our method requires neither additional training nor modifications to the pretrained model architecture and instead actively establishes computational reusability through structured sampling coupling. Specifically, LazyDiffusion employs a specially designed and trained Encoder–Decoder architecture to separate global image context encoding from iterative generation within the masked region, thereby reducing redundant computation at the architectural level. In contrast, our method operates directly on existing pretrained models and incorporates computation-reuse considerations into the initial noise sampling process. By explicitly controlling sampling correlations among candidates, we actively establish relationships between their background trajectories, enabling effective reuse of otherwise repeated background computation and thereby reducing inference costs.

References

[1] Zhu et al., KV-Edit: Training-Free Image Editing for Precise Background Preservation, ICCV 2025.

[2] Nitzan et al., Lazy Diffusion Transformer for Interactive Image Editing, ECCV 2024.

AuthorsOpenReview ↗

Response to Reviewer 4HNZ (2/3)

R1 (2/2) : Identical observed backgrounds do not necessarily imply identical intermediate computational states across different generation candidates. In standard Best-of-N, each candidate independently samples the full initial noise. Even when candidates share the same input image and editing instruction, their background tokens may follow different intermediate denoising trajectories. Consequently, directly reusing the background K/V states of one candidate for other independently sampled candidates may introduce mismatches in intermediate features and degrade generation quality.

To address this issue, we propose Reuse-Oriented Structured Sampling Coupling, implemented through Mask-Aware Initial Noise Construction. Specifically, we share the initial background noise across candidates while retaining independent noise sampling within the masked region. This design has two closely related properties. First, the marginal distribution of each candidate's initial noise remains standard Gaussian. Second, the joint sampling correlation structure across candidates is explicitly modified to improve the alignment of background denoising trajectories while preserving independent stochastic exploration within the masked region.

Building upon this design, a fully computed Anchor Candidate provides reusable background K/V states for subsequent candidates, while tokens near the mask boundary remain dynamically computed to maintain contextual consistency between the generated content and the observed background. Furthermore, Token-Aware Temporal Velocity Cache reduces redundant computation for stabilized tokens within the masked region, allowing the overall framework to reduce both cross-candidate spatial redundancy and within-candidate temporal redundancy.

We further conduct experiments to evaluate the practical impact of the above technical differences. Specifically, we evaluate two configurations of KV-Edit [1] under our image inpainting setting: KV-Edit/dev uses FLUX.1-dev, the original backbone adopted in the KV-Edit paper, whereas KV-Edit/Fill adapts the KV-Edit method to FLUX.1-Fill-dev, enabling a direct comparison with our method under the same pretrained backbone.

Table R4. Quantitative comparison with KV-Edit under the image inpainting setting.

MethodCLIP ↑HPS2.1 ↑HPSv3 ↑PickScore ↑VQA ↑ImageReward ↑
KV-Edit/dev30.08830.24092.534321.19060.84160.5010
KV-Edit/Fill30.19380.25374.833321.57610.87010.8074
Ours30.07800.25885.513821.74700.87720.8361

These results demonstrate that, under the evaluated image inpainting setting, our method achieves better overall generation quality than the adapted KV-Edit baseline, even when both methods use the same FLUX.1-Fill-dev pretrained backbone. This further validates the practical effectiveness of our approach and provides direct empirical evidence of its performance advantages over existing background K/V reuse methods.

In summary, we emphasize that the core technical contribution of our work is neither the KV Cache operation itself nor merely a change in the source of cached background K/V states. Instead, we propose a training-free joint sampling and computation-reuse mechanism that preserves the marginal distribution of each candidate's initial noise. By explicitly controlling the cross-candidate sampling correlation structure, our method transforms intermediate background computational states that would otherwise be difficult to share reliably across candidates into effectively reusable computational resources, while preserving independent exploration within the masked region.

References

[1] Zhu et al., KV-Edit: Training-Free Image Editing for Precise Background Preservation, ICCV 2025.

[2] Nitzan et al., Lazy Diffusion Transformer for Interactive Image Editing, ECCV 2024.

AuthorsOpenReview ↗

Response to Reviewer 4HNZ (3/3)

W2: It is not clear how the Token-Aware Temporal Velocity Cache handles tokens that have stopped changing. The paper says it skips these tokens and reuses their previous velocity predictions, but what happens to their keys and values in each attention layer?

Q1: When a dynamic token is declared stable by the Token-Aware Temporal Velocity Cache and removed from the active set, what then happens to that token's K/V states in every subsequent transformer block and denoising step? Are its most recently computed per-layer K/V states cached and concatenated with the K/V of active tokens, or is the token removed entirely from the attention context? If it is removed, please specify the resulting attention sequence and justify why the lost dynamic-token context is safe. If stale K/V is retained, please include this cache in the method description and in the memory/compute accounting. This answer could materially affect my assessment of both soundness and efficiency.

R2: Frozen tokens stop participating in query-side computation but remain accessible through cached per-layer K/V states. We apologize that the current manuscript does not clearly describe the handling of frozen tokens, which may have caused confusion. When a dynamic token is declared stable and removed from the active set, it stops participating in subsequent Query computation but is not removed entirely from the Attention Context.

Specifically, we retain the token's most recently computed Key and Value states in each Transformer Block and reuse them throughout the remaining denoising steps. Although frozen tokens no longer undergo fresh feature computation, their cached K/V states remain available to other active tokens, preserving the necessary contextual information. For the frozen token itself, we reuse its latest Velocity Prediction from the last step in which it was active, without executing further Transformer Forward passes for that token.

In terms of efficiency, this design reduces computation by shrinking the number of active Queries and associated feature updates, while retaining frozen tokens as attention context. Additional memory overhead mainly comes from caching their per-Transformer-Block K/V states, which are reused rather than recomputed in subsequent steps.

Manuscript revisions. In the revised manuscript, red marks changes to the original content, and blue marks newly added content.

To address W1/Q2, Section 2 (page 3) adds KV-Edit and LazyDiffusion and distinguishes their editing/reuse settings from cross-candidate Best-of-N sampling coupling; Section 3.2 (page 4) and Appendix G.1 (page 21) state the unchanged per-candidate Gaussian marginal and altered cross-candidate correlation.

To address W2/Q1, Appendix C (page 16), in the paragraph "Noise construction and anchor cache," clarifies that active queries attend to the step-aligned anchor text/background states, current active-token K/V, and retained last-active per-block K/V of frozen tokens. Frozen tokens remain in the attention context and reuse their last active-step velocity prediction.

Reviewer_mWo6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes to accelerate the Best-of-N inference-time scaling method for inpainting. It observes that there is significant redundancy between image samples during the inpainting process, because the unmasked parts stay fixed across all generated samples. It further observes that some generated tokens converge to their final values faster than others, so they can be treated as part of the known region once they have converged. Based on these observations, the authors cache the key-value (KV) pairs of the known regions of the images, which saves computation across diffusion steps and across inpainting samples. Using the same initial noise on the known region (background) turns out to be important for the method to work.

Critical strengths
  • The paper is very well written and easy to follow.
  • It achieves a strong speed-up over the baselines while retaining almost the same quality as Best-of-N.
  • The approach is sound and simple.
  • The experiments are extensive.
Critical weaknesses
  • The only weak point I could identify is the limited novelty of the method. KV-caching is a standard method to accelerate inference, but i'm unsure if the noise-sharing and stabilized caching token are novel enough.
  • The paper does not measure the diversity of the proposed method's outputs. Sharing the background noise could reduce the diversity of the exploration, so an analysis is needed to see whether the method creates a trade-off, especially when the masked/inpainted area covers most of the image.
Questions for authors

a) Does sharing the background noise affect the diversity of the generated unknown region when a very large part of the image is masked? I understand that this case becomes less of an inpainting task and closer to pure generation, since there is not enough information to infer the masked area.

b) Could the background contain so little information that different initial noise would significantly change its semantics?

c) Could the authors comment on Figure 2b: why does the variance increase again in the later denoising steps?

7 comments
AuthorsOpenReview ↗

Response to Reviewer mWo6 (1/3)

We thank the reviewer for the positive assessment of the paper's presentation, simplicity, and extensive experiments. We address the concerns about novelty and candidate diversity, together with the three specific questions.

W1: The only weak point I could identify is the limited novelty of the method. KV-caching is a standard method to accelerate inference, but I'm unsure if the noise-sharing and stabilized caching token are novel enough.

R1: Our contribution lies in constructing cross-candidate computational reusability and adapting computation within each candidate, rather than introducing a new KV Cache operation. We agree that KV Cache is an established inference-acceleration technique. However, our main contribution is to address the structured computational redundancy of Best-of-N image inpainting by designing both the sampling relationships among candidates and the allocation of computation within each candidate, while preserving random exploration in the region to be generated.

First, we find that semantic consistency does not imply that intermediate computational states are inherently reusable; instead, structured sampling can actively create the conditions for reliable computation reuse. Standard Best-of-N independently samples the full initial noise for every candidate, causing the same known background to evolve along different trajectories and making it difficult to share intermediate states. We therefore propose Mask-Aware Initial Noise Construction, which shares initial background noise across candidates while retaining independent sampling in the masked region. Under the assumption of independent standard Gaussian noises and a binary mask, this design preserves the marginal initial-noise distribution of each individual candidate but deliberately changes the cross-candidate correlation structure. As a result, background trajectories become more aligned while the masked region retains the random degrees of freedom required for diverse candidate exploration. This enables reliable cross-candidate reuse through Anchor-Cached Background KV. The contribution therefore goes beyond applying background caching: sampling itself is designed to make previously difficult-to-share cross-candidate computational states reusable.

Second, we further observe that even within the masked region, individual tokens have markedly different denoising dynamics and computational demands. Motivated by this finding, we introduce Token-Aware Temporal Velocity Cache, which adaptively determines how much computation different tokens require. This concentrates computation on tokens that still need refinement and further reduces temporal redundancy within each candidate trajectory. Overall, our contribution is a Best-of-N inpainting inference mechanism that combines actively constructed cross-candidate reusability with adaptive within-candidate computation allocation, concentrating computation where random exploration and continuing refinement are actually needed.

AuthorsOpenReview ↗

Response to Reviewer mWo6 (2/3)

W2: The paper does not measure the diversity of the proposed method's outputs. Sharing the background noise could reduce the diversity of the exploration, so an analysis is needed to see whether the method creates a trade-off, especially when the masked/inpainted area covers most of the image.

Q(a): Does sharing the background noise affect the diversity of the generated unknown region when a very large part of the image is masked? I understand that this case becomes less of an inpainting task and closer to pure generation, since there is not enough information to infer the masked area.

R2: Sharing background noise preserves independent sampling in the masked region, including under large masks. We agree that when the mask covers most of the image, inpainting increasingly approaches open-ended image generation, and it becomes especially important to retain randomness across candidates. However, Mask-Aware Initial Noise Construction does not reduce randomness in the region being generated: it shares noise only in the observed background and maintains independent sampling within the mask.

Specifically, under independent standard Gaussian noise and a binary mask, this construction preserves each candidate's standard Gaussian marginal prior while changing only the correlation structure across candidates. As the mask becomes larger, the shared-background region becomes smaller and the independently sampled region becomes correspondingly larger. In the limiting case where the entire image is masked, the initial-noise sampling reduces to the independent sampling used by standard Best-of-N. Thus, at the initial-noise level, our construction does not restrict the random exploration available inside a large masked region.

To further evaluate diversity under large masks, we divide 240 benchmark samples into four quartile groups by the proportion of mask tokens, with 60 samples per group, and measure candidate diversity using Mean Pairwise LPIPS. This metric averages pairwise LPIPS perceptual distances between generated candidates for the same input; higher values indicate greater perceptual differences among candidates.

Table R3. Candidate diversity and selected-output quality across mask-size quartiles (60 samples per group).

GroupAvg. Mask RatioMean Pairwise LPIPS ↑CLIP ↑HPSv3 ↑
Q121.24%0.06330.45.74
Q234.59%0.12030.15.43
Q345.59%0.17029.85.61
Q458.69%0.22330.65.56

As shown in Table R3, as the average mask ratio increases from 21.24% to 58.69%, Mean Pairwise LPIPS increases from 0.063 to 0.223; we do not observe a decline in candidate perceptual diversity in the larger-mask groups. In particular, Q4 has the highest Mean Pairwise LPIPS (0.223), while its CLIP Score and HPSv3 remain at 30.6 and 5.56, comparable to the other groups. Final generation quality does not show a consistently decreasing trend as the masked region grows.

These results indicate that, even with less observed background, our method preserves perceptual differences among candidates while maintaining competitive final generation quality. This accords with our design: shared background computation removes redundancy, while random exploration remains concentrated in the region requiring inpainting. We have included this mask-stratified diversity analysis in Section 4.4 (page 9), Table 5 (page 10).

AuthorsOpenReview ↗

Response to Reviewer mWo6 (3/3)

Q(b): Could the background contain so little information that different initial noise would significantly change its semantics?

R3: The observed background is conditioning information to preserve, while semantic exploration is assigned to the masked region. We agree that, when the observed background provides limited semantic information, different initial noises may affect the model's inference about the missing content. However, in image inpainting, the background is known conditioning information that should be preserved, rather than the main content explored by Best-of-N. Sharing background noise keeps the background denoising trajectories aligned across candidates, while independent masked-region noise still allows candidates to explore different plausible semantic completions. This design does not rely on the background carrying rich semantic information; it concentrates stochastic exploration in the region that actually needs generation.

Q(c): Could the authors comment on Figure 2b: why does the variance increase again in the later denoising steps?

R4: Figure 2(b) illustrates non-monotonic, token-dependent denoising dynamics. Figure 2(b) plots the temporal variance of Velocity Predictions for different masked-region tokens over local denoising windows in the original denoising process. The late-stage variance increase reflects the stage-dependent and non-monotonic nature of denoising dynamics. Related behavior has been observed in prior work. For example, Pyramid Attention Broadcast (PAB) [1] reports a U-shaped trend in the differences between Attention outputs at adjacent denoising steps: the variation decreases in the middle stages and rises again toward the end. This is consistent with the non-monotonic trend observed in our analysis. Thus, Figure 2(b) is intended to illustrate that different tokens exhibit heterogeneous, time-varying denoising dynamics, rather than claiming that token variances must decrease monotonically. We have clarified this observation in Section 3.4 (page 5).

Reference

[1] Zhao, Xuanlei, et al. “Real-Time Video Generation with Pyramid Attention Broadcast.” International Conference on Learning Representations (ICLR), 2025.

Manuscript revisions. In the revised manuscript, red marks changes to the original content, and blue marks newly added content.

To address W1, Section 2 (page 3) now discusses KV-Edit and LazyDiffusion, while Section 3.2 (page 4) and Appendix G.1 (page 21) clarify the contribution as constructing cross-candidate reusability through noise coupling and allocating computation adaptively. The Gaussian marginal and changed cross-candidate covariance are made explicit.

To address W2/Q(a), Section 4.4 (page 9) and Table 5 (page 10) add mask-stratified Mean Pairwise LPIPS and quality results for 240 samples; Appendix G.1 (page 21) covers the fully masked limit and Q(b), including why observed background need not carry rich semantics. To address Q(c), Section 3.4 (page 5) clarifies the non-monotonic, token-dependent variance in Figure 2(b), with the PAB reference. Additional temporal-reuse evidence and references appear in Appendix G.3 (page 22).

Reviewer_xyMSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training-free method to make Best-of-N image inpainting faster. It shares background noise across the candidates and reuses background KV cache from one fully computed anchor candidate. Within the edited region, tokens with stable velocity predictions reuse their last prediction. Experiments on EditBench and BrushBench show that the method keeps most of the quality gains of full Best-of-N.

Critical strengths
  1. The method makes good use of the inpainting setting. Since all candidates use the same input image, much of the background computation is repeated. Sharing this computation is a sensible way to save cost while allowing different completions inside the mask.
  2. The ablations help explain why the main components are needed. In particular, reusing background KV states without sharing the background noise leads to a clear drop in quality. This provides useful evidence that aligning the background trajectories matters for reliable reuse.
  3. The reported FLOPs savings are substantial. The method uses roughly one-fifth to one-quarter of the computation required by full Best-of-N, while keeping the output scores reasonably close. This is a useful trade-off for generating and selecting from many candidates.
Critical weaknesses
  1. The method relies on a single anchor, but its sensitivity to the anchor choice is unclear. A poor anchor may affect later candidates through the shared background states and reduce the diversity of the search.

  2. Tokens are permanently frozen based on only a few recent predictions. Short-term stability does not guarantee that a token will remain stable as surrounding content changes. And the influence of mask size and shape remain unknown for large masks leaving less reusable background and complex boundaries potentially reducing the method's benefits.

Questions for authors
  1. Sharing one anchor is a simple way to reduce repeated background computation. It would be better to show how final quality and candidate diversity vary across anchor seeds, particularly when the anchor produces a poor completion.

  2. The token-wise stability check is a practical way to decide where to reuse predictions. It would be better to compare the cached velocities with later full-compute predictions to show whether short-term stability supports freezing tokens for all remaining steps.

5 comments
AuthorsOpenReview ↗

Response to Reviewer xyMS (1/3)

We thank the reviewer for recognizing how our method exploits the structure of image inpainting, the role of background-trajectory alignment, and the substantial FLOPs savings. We address the concerns about anchor selection, irreversible token freezing, and mask size below, grouping the related weaknesses and questions.

W1: The method relies on a single anchor, but its sensitivity to the anchor choice is unclear. A poor anchor may affect later candidates through the shared background states and reduce the diversity of the search.

Q1: Sharing one anchor is a simple way to reduce repeated background computation. It would be better to show how final quality and candidate diversity vary across anchor seeds, particularly when the anchor produces a poor completion.

R1: The Anchor Candidate provides reusable background context without determining the content of subsequent candidates. Under our Mask-Aware Initial Noise Construction, all candidates share the initial background noise, while the noise inside the masked region is sampled independently for each candidate. Subsequent candidates reuse only the anchor's background K/V states, while their target-region tokens are dynamically generated from their own initial noise. Thus, the masked region retains the freedom to explore different completions.

To assess the sensitivity to anchor selection, we additionally evaluate three Anchor Seeds (4000, 4001, and 4002), measuring the anchor's own generation quality, the mean quality of subsequent reuse candidates, and candidate diversity using Mean Pairwise LPIPS. Mean Pairwise LPIPS averages the LPIPS perceptual distances over pairs of generated candidates for the same input; a higher value indicates greater perceptual differences among candidates and, accordingly, greater measured diversity.

Table R1. Generation quality and candidate diversity under different anchor seeds. Reuse Mean reports the mean quality of subsequent reuse candidates, not the selected Best-of-N result.

Anchor SeedEvaluationCLIP ↑HPS2.1 ↑HPSv3 ↑PickScore ↑VQA ↑ImageReward ↑Mean Pairwise LPIPS ↑
4000Anchor32.29570.30408.257424.09630.92300.8311—
4000Reuse Mean31.35120.29497.450923.60410.84720.88130.1863
4001Anchor31.76130.30918.077624.21700.93850.8683—
4001Reuse Mean31.87660.29638.298323.68820.82820.89060.1816
4002Anchor31.88840.30136.265324.32430.79800.8884—
4002Reuse Mean32.13640.29498.470223.43170.88750.91290.1988

As shown in Table R1, a lower-quality anchor does not necessarily lead to lower-quality subsequent candidates. For example, the anchor with Seed 4002 has an HPSv3 of only 6.27, lower than 8.26 and 8.08 for Seeds 4000 and 4001, respectively. Nevertheless, its subsequent reuse candidates achieve a mean HPSv3 of 8.47, without a corresponding decline relative to the other anchor settings. The remaining quality metrics likewise show no consistent deterioration as anchor quality decreases.

For candidate diversity, the Mean Pairwise LPIPS values are 0.1863, 0.1816, and 0.1988 for the three anchor seeds, remaining broadly comparable. In particular, Seed 4002, whose anchor has the lowest HPSv3, does not exhibit reduced pairwise LPIPS among its reuse candidates. These results indicate that anchor quality does not have a simple, direct correspondence with the quality or diversity of subsequent candidates. This is consistent with our design: the anchor supplies reusable background context, while independent masked-region initialization preserves the degrees of freedom required for multi-candidate exploration. Across the tested anchor settings, we do not observe a systematic quality decline or candidate collapse caused by a lower-quality anchor.

AuthorsOpenReview ↗

Response to Reviewer xyMS (2/3)

W2: Tokens are permanently frozen based on only a few recent predictions. Short-term stability does not guarantee that a token will remain stable as surrounding content changes.

Q2: The token-wise stability check is a practical way to decide where to reuse predictions. It would be better to compare the cached velocities with later full-compute predictions to show whether short-term stability supports freezing tokens for all remaining steps.

R2: Irreversible Token-Aware Temporal Velocity Cache is a selective approximation, not an assumption of perfectly unchanged future predictions. We agree that short-term stability in a token's Velocity Prediction cannot strictly guarantee stability throughout all subsequent denoising steps. However, Token-Aware Temporal Velocity Cache (TVC) does not require cached predictions to match later full-compute predictions exactly; its objective is to reduce redundant computation while largely preserving final generation quality.

To assess the reliability of irreversible freezing, we further compare the velocities cached by the original TVC with predictions obtained through later full computation. The original TVC exhibits relatively low prediction errors for most of denoising: 0.22%, 0.55%, and 0.70% at zero-based steps 12, 16, and 24, respectively. The error rises more noticeably near the end of denoising, reaching 2.06% at step 29. Importantly, despite this late-stage prediction discrepancy, the original TVC still maintains good final generation quality.

Table R2(a). Cached-velocity prediction error at selected zero-based denoising steps under the original TVC and two reactivation controls.

Zero-Based StepOriginal TVCReactivate Every 5 StepsContinuously Reactivate in Last 5 Steps
120.22%0.22%0.22%
160.55%0.46%0.55%
240.70%0.61%0.70%
292.06%0.72%0.97%

To further examine the actual effect of late-stage errors on generation quality, we introduce two comparison settings: periodically reactivating tokens every five steps and continuously reactivating them during the last five steps. Although these additional computations reduce the last-step prediction error to 0.72% and 0.97%, respectively, the original TVC achieves broadly comparable final generation quality across the evaluation metrics. In contrast, the original TVC requires only 0.229 normalized FLOPs, compared with 0.274 and 0.246 for the two reactivation controls. Thus, while the original TVC incurs some approximation error toward the end of denoising, the error does not produce a noticeable deterioration in the final generation quality under these settings, and irreversible freezing avoids additional repeated computation.

Table R2(b). Final generation quality and normalized FLOPs under different temporal-reuse settings. Lower FLOPs are better.

SettingCLIP↑HPS2.1↑HPSv3↑Pick↑VQA↑IR↑FLOPs↓
No TVC30.050.25905.5921.740.8750.8330.279
TVC30.080.25885.5121.750.8770.8360.229
Reactivate /530.030.25835.5521.750.8850.8340.274
Active last 529.940.25715.5621.710.8750.8290.246

This observation is also consistent with prior studies on the sensitivity of final generation quality to intermediate prediction errors. Hunter et al. [1] investigate how computational perturbations introduced at different stages of the diffusion process propagate to the final generated image. Their experiments find that the influence of comparable perturbations on the final output generally decreases as denoising progresses: later-stage errors tend to affect the final image less than earlier-stage errors. T-GATE [2] further shows that reusing cached Cross-Attention outputs after a certain denoising stage until the end of sampling can largely preserve generation quality. These studies support the observation that changes in intermediate computational states do not necessarily translate into substantial losses in final generation quality.

References

[1] Hunter, Rosco, et al. “Fast Sampling Through the Reuse of Attention Maps in Diffusion Models.” International Journal of Computer Vision, 2025.

[2] Liu, Haozhe, et al. “Faster Diffusion via Temporal Attention Decomposition.” Transactions on Machine Learning Research (TMLR), 2025.

AuthorsOpenReview ↗

Response to Reviewer xyMS (3/3)

W3: The influence of mask size and shape remain unknown for large masks leaving less reusable background and complex boundaries potentially reducing the method's benefits.

R3: Temporal reuse mitigates the loss of spatial-reuse benefits as the masked region grows. We acknowledge that, as the mask grows, fewer background tokens remain available for reuse, which may reduce the acceleration obtained from Anchor-Cached Background KV alone. However, our framework combines cross-candidate spatial reuse with temporal reuse within the dynamic region, allowing TVC to mitigate the loss of spatial efficiency in large-mask cases.

To evaluate this, we stratify 240 benchmark samples into four quartile groups according to the proportion of mask tokens, with 60 samples per group, and measure normalized FLOPs for background spatial reuse alone and for spatial reuse combined with TVC. As shown in Table R3, as the average mask ratio increases from 21.24% to 58.69%, the cost of background-only reuse rises from 0.17 to 0.41, confirming that spatial-reuse gains decrease when fewer background tokens can be reused.

Table R3. Efficiency across mask-size quartiles (60 samples per group). FLOPs ratios are normalized to standard Best-of-N and computed as the ratio of summed FLOPs within each group.

GroupAvg. Mask RatioBG-KV / BoN ↓BG-KV + TVC / BoN ↓CLIP ↑HPSv3 ↑
Q121.24%0.170.1430.45.74
Q234.59%0.260.2230.15.43
Q345.59%0.330.2629.85.61
Q458.69%0.410.3230.65.56

Importantly, TVC effectively mitigates this trend. With TVC, the normalized FLOPs decrease further to 0.14, 0.22, 0.26, and 0.32 across Q1–Q4, and its additional normalized cost savings increase from approximately 0.03 in Q1 to 0.09 in Q4. In the largest dynamic-region group (Q4), TVC further reduces computation by approximately 22.4% relative to background spatial reuse alone, improving the overall speedup from 2.41× to 3.11× over standard Best-of-N. Meanwhile, CLIP Score and HPSv3 remain broadly comparable across groups, without a consistent decline as the dynamic region grows.

These results show that our efficiency gains do not depend solely on the amount of reusable background. When masks are small, cross-candidate background K/V reuse removes substantial spatial redundancy; as masks grow and spatial-reuse opportunities diminish, TVC continues to reduce temporal redundancy by selectively reusing the Velocity Predictions of stabilized dynamic tokens. The spatial and temporal reuse mechanisms are therefore complementary and provide meaningful computation savings across different mask sizes.

Manuscript revisions. In the revised manuscript, red marks changes to the original content, and blue marks newly added content.

To address W1/Q1, Appendix G.2 (page 22) adds the three-anchor comparison in Table 12 (page 22), separating anchor quality, reuse-candidate mean quality, and candidate diversity.

To address W2/Q2, Section 4.4 (page 9) adds the final-quality and normalized-FLOPs comparison for irreversible TVC and both reactivation controls in Table 4 (page 10); Appendix G.3 (page 22) and Table 13 (page 22) provide the later full-compute velocity checks.

To address W3, Section 4.4 (page 9) and Table 5 (page 10) report all four mask-size groups, with spatial-only and combined-reuse costs, candidate diversity, and selected-output quality.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal Period

We sincerely thank all reviewers for their constructive feedback. We are encouraged that reviewers highlighted our method's "soundness, simplicity, and practicality" (xyMS, mWo6, 4HNZ), and "intuitive" adaptive temporal reuse (4HNZ). Reviewers also praised our experimental results for demonstrating substantial efficiency gains, strong speed-ups (xyMS, mWo6, 4HNZ). All three reviewers found the presentation "clear and easy to follow" (xyMS, mWo6, 4HNZ).

During the rebuttal, we provided targeted experimental evidence and technical clarifications to address the reviewers' key concerns:

For xyMS, we systematically investigated anchor sensitivity, temporal caching reliability, and efficiency across different mask sizes, providing additional evidence for the robustness and effectiveness of our framework. Following these analyses, the reviewer explicitly confirmed that "most of my concerns have been addressed" and maintained a positive rating.

For mWo6, we directly addressed the initial concerns about technical novelty and potential diversity loss by clarifying our sampling–computation co-design and introducing quantitative diversity comparisons with standard Best-of-N across different mask sizes. Following our clarification of the technical novelty and additional experiments, the reviewer positively reassessed our technical contribution, acknowledging that "most of my concerns have been addressed" and raising their score.

For 4HNZ, we addressed the two central concerns regarding technical novelty relative to KV-Edit and LazyDiffusion and the implementation of stabilized-token caching. We clarified the core technical distinctions between our method and KV-Edit and LazyDiffusion, supplemented our response with additional experimental comparisons, and provided detailed implementation clarifications. The reviewer explicitly acknowledged that "These address my implementation concern" and further stated that "I recognize its practical value and would not object to acceptance if the other reviewers and AC find the contribution sufficient.".

Overall, the rebuttal has strengthened the empirical support for our method, clarified its technical contributions, and led to positive reviewer feedback. Importantly, our core contribution is not KV caching itself, but a sampling–computation co-design that actively constructs cross-candidate computational reusability by aligning background denoising trajectories while preserving independent exploration in the masked region. Furthermore, Token-Aware Temporal Velocity Cache reduces within-candidate temporal redundancy, jointly enabling efficient spatial and temporal computation reuse. We sincerely thank all reviewers and the Area Chair for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.