acceptodds
Withdrawn from ICLR 2027

Directional Attention Guidance for Diffusion Models via Sparse Hopfield Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Classifier-Free Guidance (CFG) improves sample quality in diffusion models, but its dual-pass inference and null-condition training limit its use in few-step regimes. Attention-space guidance is a single-pass alternative, yet why sparse-versus-dense attention guidance works remains elusive. We analyze attention extrapolation through Modern Hopfield dynamics and establish two directional properties of the sparse-dense discrepancy under shared conditioning that identify it as an acceleration direction toward the dominant stored pattern. Building on this, we propose Directional Attention Guidance (DAG), a training-free, single-pass rule that amplifies only the component of the discrepancy aligned with the retrieval direction; stability follows from a weak contraction property, and an analysis inside real cross-attention layers shows that this alignment is observed throughout sampling. DAG generalizes across UNet and MMDiT backbones and multi-step and few-step regimes, improving text alignment and human-preference scores on every backbone tested, including FLUX.1, FLUX.2, and Qwen-Image, at a small added cost per image over the base sampler.

Primary area:
generative models
Keywords:
diffusion models, guidance, classifier-free guidance, attention, sparse attention, modern Hopfield networks, text-to-image generation, training-free, few-step sampling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_imymOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the computational overhead of traditional Classifier-Free Guidance (CFG), which requires both conditional and unconditional predictions, as well as the lack of a directional theoretical explanation for existing attention-space guidance methods. It proposes Directional Attention Guidance (DAG), a training-free guidance method. Based on Modern Hopfield retrieval theory, the method analyzes the discrepancy between sparse and dense attention and selectively amplifies the component aligned with the sparse retrieval direction through projection to improve image generation quality. Experiments across different architectures, including SDXL, FLUX, and Qwen-Image, as well as multi-step and few-step sampling regimes, demonstrate that DAG generally achieves better performance than the baselines and PLADIS on the GenEval text–image alignment benchmark and several human-preference metrics.

Critical strengths
  1. To address the lack of a directional theoretical explanation for existing attention-space guidance methods, this paper proposes DAG. The method requires no additional training or modifications to model weights, performs guidance within a single model forward pass, and can be combined with existing guidance methods.

  2. The authors evaluate the proposed method across various model architectures and sampling-step settings. DAG consistently improves GenEval scores on all tested models, while also achieving overall improvements in several human-preference proxy metrics.

Critical weaknesses
  1. Although the paper emphasizes that DAG does not require an additional full model forward pass, it still introduces extra computations for attention and projection. In Table 2, DAG consistently incurs higher inference latency than both the base sampler and PLADIS.

  2. The evaluation coverage is still limited. The experiments primarily focus on GenEval and MS-COCO, without evaluations on other commonly used benchmarks covering more diverse prompt types, such as DrawBench (Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding), PartiPrompts (Scaling Autoregressive Models for Content-Rich Text-to-Image Generation), and Pick-a-Pic (Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation). Furthermore, comparisons with some relevant methods are missing, such as ADG (Angle Domain Guidance: Latent Diffusion Requires Rotation Rather Than Extrapolation). ADG and DAG are closely related in their research motivations. ADG investigates angular alignment in the model prediction space, whereas DAG studies directional geometry in attention space. Although ADG is cited in the Introduction, the paper does not provide an experimental comparison with it.

  3. The theoretical analysis has significant shortcomings.

  • The claim regarding the common attractor in Corollary 1(b), does not hold and can be disproved by a simple two-pattern Hopfield counterexample. Consider a one-dimensional system with two stored patterns:

Xi=[1,−1],α=32,β>1.X_i=[1,-1],\qquad \alpha=\frac{3}{2},\qquad \beta> 1.

In a neighborhood of x=1x=1, sparse retrieval has already reached the collapse threshold, so

T32(x)=1,TDense(x)=tanh⁡(βx).T_{\frac{3}{2}}(x)=1,\qquad T_{\rm Dense}(x)=\tanh(\beta x).

Therefore, for any finite β>1\beta>1,

lim⁡x→1r(x)=1−tanh⁡β>0.\lim_{x\to1}r(x)=1-\tanh\beta>0.

For any λ>0\lambda>0,

T′(1)=1+λ(1−tanh⁡β)≠1.T'(1)=1+\lambda(1-\tanh\beta)\ne1.

  • The authors interpret the projected component as a theoretically justified convergence-accelerating direction. However, Theorem 1 establishes directional alignment for the full residual, which does not directly guarantee that the component retained after DAG's projection remains a convergence-accelerating direction. Moreover, orthogonality in the attention probability space does not guarantee orthogonality in the output space after the Value mapping.

  • In real attention layers, Keys and Values are typically obtained through separate linear projections, whereas the underlying Hopfield model in this paper uses the same pattern matrix for similarity computation and retrieval output reconstruction. The authors should further clarify how independent Key and Value projections affect the retrieval error, directional alignment, and fixed-point conclusions.

Questions for authors

Please see the weaknesses.

Reviewer_9hw6OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Directional Attention Guidance (DAG), a training-free method that modifies cross-attention using a projected difference between sparse and dense attention. The authors motivate the method through Hopfield retrieval dynamics and report improvements in text–image alignment and image quality across several architectures and sampling regimes, without requiring an additional full model evaluation.

Critical strengths

I find the results interesting. However, I am not an expert in diffusion guidance or Hopfield networks, so my confidence in assessing the theoretical contribution and novelty is limited.

Critical weaknesses

--

Questions for authors

--

Reviewer_6bd3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper starts from the finding in the PLADIS paper that interpreting attention from a Hopfield network view and introducing α\alpha-entmax, one type of sparse attention, improves generation quality. To further improve this, the paper calibrates the difference between sparse attention and dense attention used for extrapolation in PLADIS. In that process, similar to what is used in studies like APG, the authors propose a method that projects the guidance vector onto the output of the actual sparse attention to decompose the direction, and separates these two directions to apply them individually. At this time, among various scales, applying the scheme that uses only the same direction as sparse attention shows the best performance, and the authors implement this on top of various models and guidance methods such as CFG/APG and PLADIS to demonstrate its effectiveness.

Critical strengths

It showed consistent performance improvements across most works, and in particular, it has a strength in that it works reciprocally well with existing guidance methods as well as models using guidance distillation to achieve additional performance gains. Also, for single- or few-step denoiser models and distilled models, methods that apply guidance per step struggle to obtain performance improvements because sampling finishes before sufficient guidance is applied, whereas in the case of DAG, there is an advantage in that it can give guidance and improve quality with only a single forward pass even for such models.

Critical weaknesses

In this paper, in the same manner as APG, when adding the extrapolation value between the sparse attention signal and the original attention, the difference is decomposed into two terms, a term parallel to the sparse attention term and an orthogonal term, and ζ\zeta is used to mix the decomposed signals of these two directions. However, looking at the ablation experiments in the paper, the performance was best when ζ\zeta was set to 0, and therefore this paper only uses the term parallel to sparse attention; when this happens, the CFG-style extrapolation term eventually degenerates back into a scaling of the sparse term.

In the previous PLADIS paper, from this perspective, it was reported that there was already a performance improvement even by using only sparse attention before extrapolation, and additional performance improvement was obtained through extrapolation between sparse attention and original attention. Viewed from that perspective, I think the performance improvement obtained in this paper might come from scaling the sparse attention (α\alpha-entmax) term in the pre-extrapolation baseline of PLADIS, so it seems necessary for this paper to verify where the exact performance improvement comes from.

Questions for authors
  • Looking at Tables 1 and 2, it is described as if this paper uses DAG in addition to PLADIS, where the contributions of PLADIS are 1) using sparse attention in cross-attention and 2) extrapolating (sparse attention - dense attention) through this to give CFG-style guidance. However, in DAG, keeping only the direction of sparse attention and removing the orthogonal part from this extrapolation vector goes in the direction of eliminating contribution 2) of PLADIS, so I wonder how they were used together to obtain additional performance improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.