acceptodds
Under review as a conference paper at ICLR 2027

Beyond Attention Masks: Instruction Anchoring for Efficient In-Context Diffusion Generation

Abstract

In-context diffusion transformers concatenate instruction, target, and reference tokens into a single sequence for joint attention. Reference-side computation must be repeated at every denoising step, with computation growing rapidly as more references are added. Decoupling the reference tokens from the target enables exact K/V reuse across denoising steps. This isolation, however, prevents the reference tokens from attending to the instruction, degrading instruction following and reference fidelity. This trade-off cannot be resolved by attention-mask design alone. We introduce AnchorCache, a parameter-free token-layout and attention-mask co-design that inserts static text anchors. The anchors condition the reference representations on the instruction during cache construction, after which the resulting reference K/V are reused exactly across denoising steps. This structural conversion initially degrades generation quality. We recover the lost performance with teacher-forced velocity distillation, followed by a short on-policy stage that queries the teacher at student-visited states. To our knowledge, this is the first use of on-policy distillation for architectural recovery in diffusion models. Across benchmarks spanning image, speech, and video generation, AnchorCache matches full-attention quality. Its efficiency gains increase with the reference context, reaching a 6.40× speedup in DiT inference. Code is available in an anonymous repository: https://anonymous.4open.science/r/AnchorCache-FD70/.

Primary area:
generative models
Keywords:
Diffusion Transformer, Image generation, Video generation, Model Acceleration, Condition Cache

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_thXHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AnchorCache, an efficient inference method for in-context diffusion transformers. The key observation is that isolating reference tokens from the evolving target enables exact K/V caching, but also removes instruction conditioning from the reference representations. AnchorCache addresses this trade-off by introducing static instruction anchors that condition the reference branch during cache construction while preserving exact K/V reuse. Since this architectural modification initially degrades generation quality, the authors further propose a two-stage recovery procedure consisting of teacher-forced velocity distillation followed by short on-policy distillation on student-visited states. Experiments on image, speech, and video generation show quality close to full attention while achieving up to 6.40× DiT inference speedup.

Critical strengths
  1. AnchorCache directly addresses the tension between exact reference caching and instruction-aware conditioning with a simple static-anchor design, without introducing additional learnable modules. The method retains almost the same inference efficiency as isolated caching while substantially recovering generation quality.
  2. The method is evaluated across image, speech, and video generation, and consistently achieves quality close to the full-attention models with substantial speedups. The ablations also clearly separate the contribution of static anchors and on-policy recovery, and show that additional teacher-forced training cannot reproduce the gain from the on-policy stage.
Critical weaknesses
  1. Applying AnchorCache changes the attention topology and requires a non-trivial teacher-forced distillation stage followed by on-policy refinement. Therefore, the practical benefit depends on whether this one-time adaptation cost can be sufficiently amortized over inference, which is not analyzed in detail.
  2. Most detailed ablations are performed on Qwen-Image-Edit, while speech and video are each evaluated on a single model with more limited analysis. This makes the broad cross-modal generality claim somewhat less thoroughly supported than the image-generation results.
Questions for authors

see weakness

Reviewer_hR3iOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper looks at how to reuse reference KV Cache in in-context Diffusion Transformers. With full attention, the reference information has to be recomputed at every denoising step. Simply isolating the reference tokens makes caching possible, but some instruction information is lost. The authors propose AnchorCache, which adds static text anchors so that reference tokens can get instruction information without depending on the changing denoising target. This allows the reference KV to be reused. Since this changes the attention structure, the authors also use two-stage distillation to recover generation quality. The method is tested on image, speech, and video tasks. On Qwen-Image-Edit, it achieves a 6.40× speedup for DiT denoising while largely preserving the original model's generation quality.

Critical strengths

1. The method is clever and has a good degree of novelty.

The authors address the conflict between reference KV reuse and instruction awareness by introducing static text anchors. The design is fairly simple and does not require any new learnable parameters. I think this is one of the valuable parts of the paper.

2. The speedup is clear, with some practical value.

The authors test the method on image, speech, and video tasks, which suggests it has some generality. Reducing repeated computation makes a lot of sense for tasks with multiple denoising steps and many reference inputs. They also include ablations to check whether the method works as intended.

Critical weaknesses

There are no concrete experiments on KV Cache memory usage.

  1. Appendix D analyzes computational complexity, and Figure 5 shows how the number of reference images affects inference latency. But I couldn't find a comparison of actual KV Cache memory usage or peak GPU memory.

  2. Since the method needs to cache reference K/V across layers, I think it would be useful to report memory usage for Full Attention, Isolated Cache, and AnchorCache under the same settings. Ideally, the authors could also show how memory usage changes as the number of references increases. This would give a more complete picture of the actual cost.

Questions for authors

Q1. Can you report the memory usage of KV Cache?

The details are in the weakness section above.

Q2. How much training compute does the two-stage distillation require?

Appendix E reports 30k steps for Stage 1 and 500 steps for Stage 2, but does not give the actual training time. Could the authors provide the GPU-hours, GPU setup, or wall-clock training time for each stage? This would help readers understand the extra training cost needed to get the speedup.

Reviewer_Y1Q1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AnchorCache, which enables exact K/V caching of reference tokens in in-context diffusion transformers while preserving instruction conditioning. The key observation is that isolating reference tokens for caching also removes their access to the instruction. AnchorCache resolves this by introducing a parameter-free, static copy of the instruction that is attended only by the reference branch. To recover the quality lost during this conversion, the method further uses teacher-forced velocity distillation followed by a short on-policy stage. Across image, speech, and video generation, AnchorCache matches full-attention quality while achieving up to a 6.40× DiT inference speedup.

Critical strengths
  1. The paper clearly explains why exact reference caching removes instruction conditioning and addresses this issue with a parameter-free static copy of the instruction. At nearly the same speedup (6.40× vs. 6.45×), AnchorCache improves the OmniContext score from 7.680 to 8.185 over the isolated-cache baseline, reaching performance comparable to full attention.

  2. Generality across modalities and architectures. The same design is evaluated on image editing, few-step image generation, speech, and video models, consistently improving over the isolated-cache baseline while maintaining quality close to full attention. This supports the claim that AnchorCache is a broadly applicable design for in-context diffusion transformers rather than a model-specific modification.

Critical weaknesses
  1. The efficiency analysis is incomplete. AnchorCache requires full-parameter distillation for roughly 30.5k steps per model, but the paper does not report the training compute, hardware or GPU-hours, or memory overhead, including the storage required for cached reference K/V. Since efficiency is a central claim, these costs should be reported alongside the inference speedup to make the trade-off among performance, memory, and compute clear.

  2. The comparison with acceleration methods (Table 2) is not entirely fair. Table 2 compares AnchorCache with existing acceleration methods such as TeaCache, SageAttention2, ToPi, and SVDQuant in terms of generation quality and speedup on OmniContext. However, these baselines are training-free or require only lightweight calibration, while AnchorCache requires 2-stage distillation. Training-based methods discussed in the related work, such as OminiControl2 and EasyControl, are not included in the comparison, making it difficult to assess AnchorCache against methods that also require training.

Questions for authors

1. How are positional encodings (e.g., MSRoPE in Qwen-Image-Edit) assigned to the static text anchors? Do the ST tokens share the same position IDs as the LT tokens?

2. What is the performance of AnchorCache immediately after the structural conversion, before any recovery training? Table 3 starts from the Stage 1 checkpoint, so the quality drop introduced by the structural conversion itself is not reported.

3. Could the authors report the training compute for the recovery stages, including hardware and GPU-hours, as well as the inference-time memory overhead of caching the reference K/V?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.