acceptodds
Under review as a conference paper at ICLR 2027

RefAdapt-DiT: Adaptive Joint Attention for Reference-Conditioned Diffusion Transformers

Abstract

Diffusion Transformers (DiTs) have become the standard backbone for high-quality generative modeling, yet deploying them in conditional generation tasks remains computationally prohibitive because bidirectional joint attention repeatedly processes large reference streams. While existing optimization schemes mitigate generic temporal redundancy, they typically rely on coarse-grained static reuse and overlook the distinct dynamics of references and targets. Specifically, we observe that reference representations often evolve slowly along the generation trajectory, while the target often assigns little attention mass to them; reference drift and this target-to-reference exposure jointly shape how strongly stale reference states affect the target. To exploit these patterns, we introduce \RefAdapt, a training-free framework for adaptive control of joint attention between references and targets. Instead of rigid static strategies, \RefAdapt combines consecutive target-Q change with previously observed target-to-reference attention mass to control reference computation adaptively at block granularity. Under ultra-few-step settings, \RefAdapt enables speedups of up to on 4-step MiniMax H3 and on 8-step Qwen Image Edit, while maintaining comparable visual quality.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Diffusion Transformers (DiTs), Reference-Conditioned Generation, Adaptive Joint Attention, Ultra-Few-Step Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CQLAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes RefAdapt-DiT, a training-free method for accelerating reference-conditioned diffusion transformers. The authors argue that reference K/V reuse should depend not only on how much the reference features change, but also on how strongly the target attends to them. Through empirical analysis, they observe that reference K/V generally evolves more slowly than target K/V in shallow-to-middle blocks, while this difference diminishes or even reverses in deeper blocks. They also find a correlation between target feature changes and reference K/V changes, and show that target-to-reference attention exposure helps predict the impact of stale reference K/V on target attention outputs. Based on these observations, RefAdapt-DiT combines current target-Q changes with historical attention exposure to dynamically determine when to refresh reference K/V. Specifically, it reuses cached reference K/V in earlier blocks and resumes reference computation once the combined score exceeds a threshold, recomputing the deeper blocks. Experiments on Qwen Image Edit and MiniMax H3 demonstrate speedups of 2.58× and 1.564×, respectively, while largely preserving generation quality.

Critical strengths

1. Practical inference acceleration with largely preserved generation quality. RefAdapt-DiT achieves substantial inference speedups on both Qwen Image Edit and MiniMax H3, while largely preserving generation quality. Compared with the evaluated acceleration baselines, the method provides higher speedups with competitive generation performance, demonstrating its practical effectiveness.

2. Reference-aware analysis and caching strategy. The motivation is straightforward and reasonable: whether reference K/V can be safely reused should depend not only on how much the reference features change, but also on how those changes affect the target attention output. The authors support this argument through controlled stale-K/V experiments, showing that exposure-conditioned reference drift correlates more strongly with target-attention perturbations than reference drift alone. These observations provide a useful motivation for the proposed caching strategy.

Critical weaknesses

1. The ablation study does not convincingly establish the effectiveness of the proposed controller. In Table 4, the Exposure Only baseline achieves a slightly lower speedup than RefAdapt-DiT (1.459× vs. 1.482×), but obtains higher PSNR (32.24 vs. 31.24) and SSIM (0.9958 vs. 0.9937). This raises questions about whether incorporating target-Q changes is actually necessary and whether they can serve as a reliable proxy for the target-side impact of stale reference K/V. Moreover, the controller signals have different numerical scales, yet the authors use the same threshold for all adaptive variants. A more comprehensive fidelity–speed trade-off comparison with independently tuned thresholds is needed to demonstrate the advantage of the proposed controller. In addition, this ablation is conducted on a separate 50-step trajectory rather than the actual few-step deployment settings.

2. The presentation is difficult to follow, and several important implementation details are insufficiently explained. The paper is unnecessarily complicated in its presentation, making it difficult to follow the motivation and the connections between different parts of the method. Some important quantities are not clearly defined. For example, Section 3.3 introduces the joint endpoint scalar EKVE^{KV} without specifying how the individual K/V drifts are combined. The numerical values of the stability constant ϵ\epsilon, controller threshold θ\theta, and maximum cache age Δmax⁡\Delta_{\max} are also not provided. Furthermore, the figures and tables often lack sufficient discussion of their findings, such as the additional insights provided by the right-hand profiles in Figure 1. The implementation of ResComp, which is used to obtain the additional acceleration results, is also insufficiently described. These issues make the paper unnecessarily difficult to understand and reproduce.

Questions for authors
  1. Speed–fidelity trade-off. Could the authors clarify how the shared threshold in Table 4 was selected? Since different controller signals have different numerical scales, would the advantage of the proposed controller remain consistent if the thresholds were independently tuned for each variant? In particular, does incorporating target-Q change provide additional benefits over using attention exposure alone?

  2. Implementation details and clarity. Could the authors provide clearer explanations of the key formulas and figures? In particular, how exactly is EKVE^{KV} computed, what values are used for the numerical stability constant ϵ\epsilon, controller threshold θ\theta, and maximum cache age Δmax⁡\Delta_{\max}, and how is ResComp implemented?

  3. Few-step ablations. Could the authors further validate the proposed controller through ablation studies under the actual 4-step MiniMax H3 and 8-step Qwen Image Edit settings? In particular, does combining target-Q change and attention exposure consistently outperform the individual signals in these deployment settings?

I am open to raising my rating if the authors can satisfactorily address these concerns, particularly regarding the effectiveness of the proposed controller and the robustness of the ablation comparisons.

Reviewer_gB1mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In ref conditioned DiT, reference token KV has slower changes across denoising steps, hence the paper caches this. Reference streams are recomputed only when the stale reference is noticed during runtime. The pipeline is training free and can speed up inference.

Critical strengths

S1. The thought is novel, and the pipeline requires no training. The method's speed up scales with number of references which is neat.

S2. The pipeline also works on distilled models which generally have little room for speedup. This can reduce latency in real time video and image generation methods.

Critical weaknesses

W1. What is Rescomp? It is defined as orthogonal resolution compression but nothing has been referenced or defined anywhere? This makes the paper feel very incomplete. The numbers on the abstract or tables depend on rescomp but no explanation of what this is? I couldnt find any published work that corresponds to "Rescomp"

W2. The experimentation needs more work - reference specific methods need to be compared with and baselines need run at corresponding speed. The ablations are weak - the method is not much better than using just exposure. In addition to this, the paper has some missing details, values of θ\theta and Δ\Delta max are never reported

Questions for authors

see weaknesses.

Reviewer_fWCrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RefAdapt-DiT, a training-free acceleration framework for reference-conditioned Diffusion Transformers. The authors show that reference representations evolve more slowly than target representations and that the impact of stale reference features depends on target-to-reference attention exposure. Based on these observations, they introduce a depth-adaptive reference reuse strategy that achieves substantial inference speedups while maintaining competitive generation quality.

Critical strengths
  1. Clear motivation. The paper identifies an interesting distinction between reference representation drift and its actual impact on target generation.
  2. Insightful empirical analysis. The investigation of reference–target dynamics and attention exposure provides useful insights into joint-attention diffusion models.
  3. Simple and effective method. The proposed training-free framework is intuitive, computationally efficient, and easy to integrate into existing diffusion architectures.
Critical weaknesses
  1. Figure clarity and presentation. Some figures are not sufficiently intuitive, making the main observations difficult to understand at first glance. In particular, Figure 1 could better highlight the depth-dependent differences between reference and target dynamics. Figure 3 would benefit from a direct visual comparison between raw reference drift and exposure-weighted drift, making the central motivation more immediately convincing. Figure 4 could also be simplified to better illustrate the adaptive reuse mechanism.
  2. Generality of the observations. Some mechanism analyses are based on a limited number of generation trajectories. Evaluating these observations across more independent prompts would provide stronger evidence of their generality.
Questions for authors
  1. How sensitive is the method to the controller threshold and maximum cache age?
  2. Do the observed relationships between reference drift, attention exposure, and target perturbation generalize across different prompts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.