acceptodds
Under review as a conference paper at ICLR 2027

Internal Reference Guidance via Latent Token Merging for Diffusion Sampling

Abstract

Inference-time diffusion sampling guidance plays a key role in enabling high-performance image synthesis by diffusion models. Existing methods, including Classifier-Free Guidance (CFG), typically maintain two denoising branches throughout the iterative denoising process to guide predictions by extrapolating between the two branches, which amounts to increased overhead in terms of the number of function evaluations (NFE). In this paper, we investigate whether effective guidance signals can be constructed within a single denoising branch. We achieve this by shifting guidance extrapolation from the prediction space to the intermediate latent-token feature space within the diffusion backbone. Specifically, we construct an internal reference within the cross-attention blocks by merging and unmerging latent tokens and perform guidance by extrapolating the original attention features away from this coarsened reference. Extensive experiments on diverse datasets show that the proposed Merging and Unmerging Guidance (MUG) serves as a general-purpose plug-in module that improves both U-Net- (\ie, SDXL) and MMDiT-based (\ie, Flux.1-schnell/dev) diffusion models in image fidelity, prompt alignment, and human preference. Notably, MUG fills an important applicability gap by enabling inference-time guidance for natively single-branch models, particularly guidance-distilled models (\eg, DMD2), for which conventional CFG is not inherently supported. MUG generalizes across different tasks, sampling regimes, and guidance paradigms while complementing existing guidance methods from CFG to recent condition-free techniques.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion model, diffusion sampling, image generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_icBeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MUG, a training-free, single-branch diffusion guidance method that constructs degraded internal references through token merging and performs CFG-style extrapolation in feature space to improve generation quality without additional model evaluations.

Critical strengths

The qualitative and quantitative results are promising.

The paper and method are easy to follow.

Critical weaknesses

MUG improves generation quality, but the paper does not sufficiently explain why extrapolating away from merged features leads to better predictions.

MUG is applied to cross-attention. Could it also be applied to self-attention and FFN layers?

Questions for authors

MUG can be viewed as enhancing attention outputs. Could directly scaling attention outputs achieve similar improvements, for example, by multiplying them by a constant factor or using token-wise attention entropy to identify important tokens and amplify their outputs with a factor greater than 1?

4 comments

Dear Reviewer,

The authors have provided their responses to the raised concerns and revisions. Please review their responses and join the discussion before the end of the author-review discussion phase (Nov 18, AoE). Thanks.

Best,

AC

Dear Reviewer,

We hope that our response has addressed your questions and comments, and would greatly appreciate that we have ample time to address any remaining questions or concerns, as fewer than two days remain in the discussion phase.

Thank you again for your time and effort in reviewing and making our manuscript better.

Authors of Submission 36194

Thank you for the authors' responses, which have addressed most of my concerns. However, the proposed method alters the structure of the originally generated images. Similarly, reducing the number of sampling steps in multi-step generation can also lead to structural changes. This raises the question of whether the proposed method offers any advantages over simply reducing the number of sampling steps.

Dear Reviewer icBe,

We are very glad that our additional evidence and discussions have addressed most of your concerns! We sincerely thank you for this insightful suggestion: using an earlier sampling prediction as the degraded reference is both natural and computationally attractive.

Initial one-shot test. We first conduct a rapid validation on 500 prompts using SDXL with 50-step sampling. We cache the earlier prediction at step 20 and use it for guidance once, at step 50:

=\epsilon(50)+\lambda\bigl[\epsilon(50)-\epsilon(20)\bigr].$$ | **Method** | **CLIP ↑** | **ImageReward ↑** | **PickScore ↑** | **$T_{\mathrm{E2E}}$ (s/img) ↓** | **$\Delta$Mem (MB) ↓** | |---|---:|---:|---:|---:|---:| | Baseline | 0.319 | 0.303 | 20.96 | 6.814 | 327.67 | | Early Exit as Degradation (step 50 vs. step 20; single update) | 0.319 | 0.307 | 20.98 | 6.928 | 327.80 | | Early Exit as Degradation (20-step lag; repeated) | 0.320 | 0.705 | 21.33 | 6.928 | 327.92 | | Merge–Unmerge as Degradation (ours) | **0.324** | **0.849** | **22.45** | 7.943 | 328.95 | The one-shot reference provides only a marginal improvement. We hypothesize that a single guidance update is too weak because it acts only after most of the trajectory has already been determined. The large state and noise-level gap between steps 20 and 50 may further mix the desired quality direction with ordinary denoising evolution. **Repeated 20-step-lag guidance.** We therefore design a stronger variant. Starting from step 21, each step uses the cached prediction from 20 sampling steps earlier: $$\epsilon_{\mathrm{guided}}(i) =\epsilon(i)+\lambda\bigl[\epsilon(i)-\epsilon(i-20)\bigr], \qquad i=21,\ldots,50.$$ This produces 30 guidance updates without any additional U-Net forward pass. It significantly improves over the one-shot variant while retaining near-baseline latency and memory. **Trade-offs.** Earlier-step degradation is highly efficient because it reuses existing predictions, but compares different latent states and noise levels. MUG instead constructs its clean and degraded retrievals at the **same $x_t$, timestep, and text condition**, yielding stronger improvements at the cost of additional cross-attention computation. **Few-step applicability.** Earlier-step guidance requires sampling history and is therefore unavailable for a 1-step trajectory. MUG remains applicable, as directly demonstrated by the 1-step DMD2 results in Table 2 and the “Compatibility with guidance-distilled models” discussion in Sec. 4: MUG improves CLIP from 0.333 to 0.336, ImageReward from 0.822 to 1.004, and PickScore from 22.27 to 22.31, but early exist is **not usable** in this case. Therefore, MUG has better generalizability as a universal guidance method. Based on our initial experiments, we also have good reasons to believe that earlier-step guidance **may not work well under a few-step (e.g., 2-8 steps) sampling set-up**, which are however an immensely important set-up in real applications and products. We sincerely thank you again for this inspiration. This temporal-reference idea deserves further development and systematic exploration, particularly regarding the temporal lag, activation window, and combination of multiple earlier predictions. Authors of Submission 36194
Reviewer_3it9OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MUG is a training-free, inference-time guidance method that constructs a coarsened reference by merging image latent tokens within a single denoising branch. It steers the original features away from this reference, improving generation quality and prompt alignment on SDXL, Flux, and DMD2 without additional full-denoiser evaluations. It does incur extra attention computation and complements, rather than replaces, CFG where available.

Critical strengths

Strengths

  • Applying CFG-style extrapolation to intermediate features, using a token-merged internal reference, is a simple and original idea.
  • MUG requires no additional training or model parameters and can be applied to pretrained, single-branch models for which conventional CFG is less straightforward.
  • The method improves generation quality without additional full-denoiser evaluations, offering a favorable quality–latency trade-off relative to more expensive guidance methods.
  • Its effectiveness is demonstrated across different architectures and sampling regimes, including few-step generation. MUG can complement existing guidance methods rather than requiring them to be replaced.
Critical weaknesses

Weakness

  • As a standalone guidance method, MUG performs significantly worse than CFG in terms of prompt alignment when applied to SDXL—particularly regarding CFG’s ability to enhance semantic representation. Consequently, it is more compelling to view MUG as a complement to CFG rather than a replacement for it.
  • Some design choices remain empirical. For example, the paper compares window-based token selection with fully random selection, but does not systematically examine different window sizes or other structured selection strategies.
  • The theoretical analysis explains how feature perturbations affect the model output, but does not establish why the resulting direction should consistently improve image quality or prompt alignment. The proposed mechanism remains partly conjectural.
  • The human preference study covers only 14 participants and 15 prompts. A larger study would provide stronger support for the perceptual improvements claimed.
Questions for authors

See the weakness

3 comments

Dear Reviewer,

The authors have provided their responses to the raised concerns and revisions. Please review their responses and join the discussion before the end of the author-review discussion phase (Nov 18, AoE). Thanks.

Best,

AC

Dear Reviewer,

We hope that our response has addressed your questions and comments, and would greatly appreciate that we have ample time to address any remaining questions or concerns, as fewer than two days remain in the discussion phase.

Thank you again for your time and effort in reviewing and making our manuscript better.

Authors of Submission 36194

Dear Reviewer 9Yrc,

We sincerely hope that our response has well addressed your questions. As the rebuttal phase comes to an end, we are ready to respond to and clear any of your remaining concerns.

We also kindly refer you to our response to questions from other reviewers, particularly R1Q1 to R1Q3 and R3Q2 to R3Q3, because we believe these response additionally address your questions, justifying other key design choices and why the MUG direction is useful.

Authors of Submission 36194

Reviewer_ecSiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Merging and Unmerging Guidance (MUG), a training-free diffusion guidance method that constructs a degraded reference through token merging and unmerging within cross-attention layers. It performs feature-level extrapolation without requiring a second denoising branch. Experiments on SDXL, Flux, and DMD2 show improvements in generation quality and efficiency over existing guidance methods, particularly PLADIS.

Critical strengths
  1. MUG provides a simple and effective way to perform diffusion guidance within a single denoising branch, without additional training or a second full model evaluation. This makes it particularly useful for guidance-distilled models.

  2. The experiments are extensive, covering different architectures, datasets, and sampling steps. MUG consistently improves generation quality over PLADIS, while introducing relatively little computational overhead.

Critical weaknesses
  1. Necessity of Cross-Attention. Why is it necessary to recompute cross-attention after token merging? It would be useful to compare this with directly merging and unmerging the original cross-attention output, which could avoid the additional attention computation.

  2. Guidance Direction Analysis. The paper lacks an analysis of how the MUG guidance direction relates to the CFG guidance direction and changes in image quality. Visualizing these directions across denoising steps could help explain why MUG works.

  3. Comparison with Feature Sharpening. MUG appears closely related to feature sharpening, i.e., Zsharp=Z+λ(Z−Smooth(Z))Z_{\mathrm{sharp}} = Z + \lambda (Z - \mathrm{Smooth}(Z)). Although the paper evaluates token blurring, a direct comparison with a properly tuned feature-sharpening baseline would help clarify what MUG offers beyond this simpler operation.

Questions for authors
  1. Why is it necessary to recompute cross-attention after token merging? Could the authors compare this with directly merging and unmerging the original cross-attention output? I would like to know whether the additional attention computation is essential to the performance gains.

  2. Could the authors analyze how the guidance direction introduced by MUG relates to that of CFG? For example, comparing the directions in the model output space across denoising steps and relating them to changes in image quality would help clarify the mechanism.

  3. MUG seems closely related to feature sharpening, where the difference between the original and smoothed features is amplified. Although token blurring is evaluated, could the authors compare MUG with an explicitly tuned feature-sharpening baseline? This would help establish whether the gains come from the proposed guidance mechanism or a more general sharpening effect.

  4. Can MUG be extended to video generation models? Since random token merging may introduce inconsistent perturbations across frames, I wonder whether it would affect temporal consistency and lead to flickering artifacts.

3 comments

Dear Reviewer,

The authors have provided their responses to the raised concerns and revisions. Please review their responses and join the discussion before the end of the author-review discussion phase (Nov 18, AoE). Thanks.

Best, AC

Thank you for the clarifications, and I maintain my original score after considering the rebuttal.

Dear Reviewer ecSi,

We sincerely appreciate your positive assessment of our work. We once again thank you for your time and effort in reviewing and making our manuscript better.

Authors of Submission 36194

Comments on the paper

9 comments

R1Q2: Guidance direction analysis. We thank the reviewer for this very helpful suggestion. We agree that the original analysis characterized how MUG's layer-local perturbations propagate to the model output, but did not directly compare the resulting output-space direction with CFG or relate that geometry to generation quality. We have added this analysis to the Further Analysis section in Appendix C of the revised PDF, under the new paragraph “Output-space relationship between MUG and CFG.”

Specifically, at identical (xt,t,c)(x_t,t,c) states, we first isolate MUG's intrinsic effect on the prompt-conditioned prediction and measure

g_{\mathrm{MUG}}(t)=\epsilon_{\mathrm{MUG,cond}}(t)-\epsilon_{\mathrm{cond}}(t),$$ where $g_{\mathrm{MUG}}$ is a conditional-branch diagnostic used to compare MUG's standalone output-space direction with CFG. It does not imply that MUG is applied only to the conditional branch when combined with CFG. Following our default CFG+MUG design, MUG is applied separately to both the conditional and unconditional branches. We therefore additionally measure the actual change induced on top of CFG, $$g_{\mathrm{MUG}\mid\mathrm{CFG}}(t) =\epsilon_{\mathrm{CFG+MUG}}(t)-\epsilon_{\mathrm{CFG}}(t) =(1+\gamma)\bigl(\epsilon_{\mathrm{MUG,cond}}-\epsilon_{\mathrm{cond}}\bigr) -\gamma\bigl(\epsilon_{\mathrm{MUG,uncond}}-\epsilon_{\mathrm{uncond}}\bigr),$$ where we use the same conditional-anchored extrapolation form as in our manuscript: $\epsilon_{\mathrm{CFG}}=\epsilon_{\mathrm{cond}}+\gamma(\epsilon_{\mathrm{cond}}-\epsilon_{\mathrm{uncond}})$, and analogously for $\epsilon_{\mathrm{CFG+MUG}}$. Here, $\gamma$ denotes the CFG extrapolation strength and is distinguished from MUG's own guidance scale $\eta$. Thus, $g_{\mathrm{MUG}}$ isolates the prompt-conditioned MUG direction, whereas $g_{\mathrm{MUG}\mid\mathrm{CFG}}$ captures the complete two-branch effect used in our CFG+MUG experiments. The MUG–CFG cosine similarity is slightly negative at the earliest steps, crosses zero during the first quarter of denoising, and gradually increases to approximately $0.55$ near the end (Fig. 10, left). The cosine curves for $g_{\mathrm{MUG}}$ and $g_{\mathrm{MUG}\mid\mathrm{CFG}}$ nearly overlap, showing that the two MUG-induced changes maintain similar alignment with CFG, while $g_{\mathrm{MUG}\mid\mathrm{CFG}}$ has a larger norm relative to CFG (Fig. 10, middle). We interpret this overlap only as similar alignment with CFG, rather than as direct evidence that $g_{\mathrm{MUG}}$ and $g_{\mathrm{MUG}\mid\mathrm{CFG}}$ are mutually aligned. Importantly, the orthogonal-component analysis in Fig. 10 (right) shows that more than $93\\%$ of the MUG direction is orthogonal to CFG through most of the trajectory and remains above $82\\%$ at the final steps. Thus, MUG is neither a duplicate nor a fixed rescaling of CFG. Instead, it provides a substantial complementary component whose relationship to CFG changes over denoising. The denoising-window ablation in Fig. 11 provides a complementary temporal view: early activation produces the largest gains among the individual windows, middle activation retains a smaller but consistent benefit, and late-only activation contributes little, while applying MUG throughout sampling gives the largest overall improvement. Notably, the quality gains are concentrated in early and middle denoising even though the MUG–CFG cosine is higher later. This temporal mismatch indicates that MUG's benefit is not explained by merely reinforcing CFG: its largely non-CFG-parallel component is most useful while global content and structure are being established. These findings also explain why MUG can remain complementary to CFG in our quantitative results.

R1Q3: Feature sharpening. We thank the reviewer for suggesting feature sharpening as a simple and important control. We agree that, under the coarse-projector approximation in Eq. (19), MUG can superficially resemble amplification of a high-frequency residual, and that the token-blurring result alone does not rule out a properly tuned sharpening explanation. We therefore implement feature sharpening as Zsharp=Z+η(Z−Blur⁡k,σ(Z))Z_{\mathrm{sharp}}=Z+\eta\bigl(Z-\operatorname{Blur}_{k,\sigma}(Z)\bigr) and sweep the kernel size kk, Gaussian standard deviation σ\sigma, and sharpening strength η\eta. This experiment uses exactly the same evaluation settings as the main experiments in our manuscript.

MethodCLIP ↑ImageReward ↑PickScore ↑
Token Blurring (k=3k=3, σ=1.5\sigma=1.5)0.3270.62422.04
Feature Sharpening (k=3k=3, σ=1.0\sigma=1.0, η=3\eta=3)0.3290.67922.34
Feature Sharpening (k=3k=3, σ=1.5\sigma=1.5, η=3\eta=3)0.3290.68422.34
Feature Sharpening (k=5k=5, σ=1.0\sigma=1.0, η=3\eta=3)0.3290.69222.35
Feature Sharpening (k=5k=5, σ=1.5\sigma=1.5, η=3\eta=3)0.3290.70822.34
Feature Sharpening (k=3k=3, σ=1.5\sigma=1.5, η=6\eta=6)0.3300.70922.30
Feature Sharpening (k=3k=3, σ=1.5\sigma=1.5, η=9\eta=9)0.3300.70722.26
MUG (ours)0.3330.89022.47

As shown, the tuned sharpening variants improve over token blurring, confirming that the reviewer's proposed control is substantially stronger than using a blurred feature directly. Nevertheless, their best results across the sweep are 0.330 CLIP, 0.709 ImageReward, and 22.35 PickScore, compared with 0.333/0.890/22.47 for MUG. Increasing the sharpening strength from η=6\eta=6 to 99 does not close the gap and slightly reduces ImageReward and PickScore, suggesting that the difference is not simply insufficient amplification.

The two operations also construct fundamentally different directions. Feature sharpening uses a fixed, data-agnostic spatial low-pass operator and extrapolates along Z−Blur⁡(Z)Z-\operatorname{Blur}(Z). MUG instead forms Z′=UAttn⁡(PHWQ,K,V)Z'=U\operatorname{Attn}(PHW_Q,K,V): coarsened image queries perform a new, content-dependent retrieval from the same text keys and values, and MUG extrapolates along the contrastive residual Z−Z′Z-Z'. Because attention is nonlinear, UAttn⁡(PHWQ,K,V)U\operatorname{Attn}(PHW_Q,K,V) is generally not equivalent to blurring an already-computed attention output. The controlled comparison therefore indicates that MUG's gain comes from its condition-preserving counterfactual retrieval, rather than from generic feature sharpening alone. We will include this tuned baseline and clarify this distinction in the revised manuscript.

R1Q4: Extension to video generation. Thank you for raising this important question. We agree that a naive video extension, in which each frame independently samples its merge map, may introduce frame-dependent perturbations and temporal flickering. Our preliminary experiments have confirmed this concern. We therefore extend MUG to Mochi-1-preview with a temporally tied design. Specifically, at each denoising step and layer, Frame-shared MUG samples one spatial merge map and applies it to all temporal slices, while merging token values only within each slice. This removes frame-dependent randomness without averaging features across time or suppressing motion.

Following the Mochi evaluation protocol of Hyung et al. [1], we use 100 VBench prompts and report both frame-quality and temporal metrics under otherwise identical generation settings. We compare the original Mochi model, a naive frame-independent extension, and our frame-shared extension. Best results are shown in bold, and second-best results are underlined.

MethodImaging quality ↑Aesthetic ↑Motion smoothness ↑Dynamic degree ↑Temporal flickering ↑
Mochi0.5240.5070.985‾\underline{0.985}0.870‾\underline{0.870}0.976‾\underline{0.976}
+ Naive frame-independent MUG0.6290.552‾\underline{0.552}0.9800.8720.969
+ Frame-shared MUG0.626‾\underline{0.626}0.5540.9890.8670.978

As noted, both MUG variants improve frame-level quality over Mochi. However, independently sampling the merge map per frame reduces Motion Smoothness and Temporal Flickering scores, consistent with the reviewer's concern. Sharing the map across frames recovers and surpasses the baseline on both temporal metrics (0.989 vs. 0.985 in Motion Smoothness and 0.978 vs. 0.976 in Temporal Flickering), while retaining nearly the same Imaging Quality as naive MUG and a comparable Dynamic Degree. These results demonstrate MUG's potential to be extended to video generation without sacrificing temporal consistency when its spatial perturbation is tied across frames. We will add this experiment and clarify Frame-shared MUG as the default video extension.

References.

[1] Hyung et al., Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling, CVPR 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.