acceptodds
Under review as a conference paper at ICLR 2027

HARMONY: Adaptive Self-Guidance through Hierarchical Subnetwork Probing

Abstract

Guidance improves diffusion generation, but its quality and cost depend on the auxiliary prediction used to construct the correction. We introduce HARMONY, which obtains complementary guidance signals from a diffusion transformer's own hierarchy and adapts their influence across time and space. An early readout of the residual stream supplies a weak prediction without repeating transformer blocks, while an optional block-dropped pass provides a second direction. Agreement between these probes controls when and where guidance is applied. A conditional error model relates population agreement to optimal shrinkage and explains the effects of unequal probe quality and correlated errors. Across the reported ImageNet, text-to-image, and video evaluations, HARMONY improves generation quality while its Lite variant requires only two additional readout-head evaluations. On DiT-XL/2, the full method reduces FID from 2.27 to 1.93 with increased recall. The framework connects the choice of a weak prediction with adaptive guidance strength through information already available during inference.

Primary area:
generative models
Keywords:
diffusion models, adaptive guidance, self-guidance, image generation, video generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_KrMfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free self-guidance method for diffusion transformers. It obtains weaker predictions by applying the model’s shared output head to cached intermediate residual states, thereby avoiding repeated transformer-block computation. In its lite variant, the differences between the final and intermediate predictions serve as depth-based guidance directions, while the agreement between two depth probes adaptively modulates guidance across denoising timesteps and spatial tokens. The full version additionally incorporates a block-dropped width probe, combines the depth and width directions through norm matching. Both methods apply an orthogonal projection to the resulting gated correction. The paper motivates its agreement-based gating mechanism through a conditional error model and evaluates the method on various image and video generation tasks using multiple DiT backbones. Experiment results show consistent improvements over CFG and existing training-free guidance methods, while the Lite variant achieves a favorable quality–cost trade-off with only a small inference overhead.

Critical strengths
  1. The observation that cached intermediate residual states can be decoded into useful weak predictions is both interesting and practically relevant. It enables HARMONY-Lite to construct self-guidance signals without repeating transformer-block computation or training an auxiliary model, resulting in a favorable quality–cost trade-off.

  2. Extensive experimental validation is presented across different models and tasks, including class-conditional image generation, text-to-image generation, and text-to-video generation. The consistent improvements across DiT, SiT, SD3, FLUX, and Wan backbones, together with the component ablations, controlled study, latency analysis, and human evaluation, provide broad empirical support for the proposed method.

Critical weaknesses
  1. Some important recent diffusion guidance methods [1-3] are neither discussed nor compared against. This is particularly important for Internal Guidance [3], which also constructs guidance from intermediate and final diffusion-transformer predictions.

  2. The width probe is closely inherited from the stochastic block-dropping mechanism of S²-Guidance. Although the appendix provides partial ablations of the direction combination and gating components, it does not present a fully component-matched progression from the original S²-Guidance implementation to full HARMONY. Consequently, the novelty and empirical contribution specifically attributable to the depth probe and cross-probe agreement are not completely isolated.

  3. The method appears to have limited generalizability to U-Net-based, few-step models, or shallower diffusion models. This is exacerbated by a range of heuristic design choices in the method. These factors undermine the practical applicability of the method.

  4. A minor comment is that it is unclear how the method name “HARMONY” was derived. Clarifying whether it is an acronym or simply refers to the agreement between hierarchical probes would improve presentation.

[1] Hyung et al. “Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling”. CVPR 2025.

[2] Zhang et al. “Guiding a Diffusion Model by Swapping Its Tokens”. CVPR 2026.

[3] Zhou et al. “Guiding a Diffusion Transformer with the Internal Dynamics of Itself”. CVPR 2026.

Questions for authors
  1. Given that the width probe of the full method is closely related to S²-Guidance, could the authors provide a progressively component-matched ablation starting from the original S²-Guidance implementation? This would clarify how much of the improvement is specifically attributable to the depth probe and cross-probe agreement rather than to the other modifications.

  2. On real diffusion models, does the agreement score correlate with the actual reduction in prediction error on forward-noised validation samples, where the target noise or velocity is known? Such an analysis would materially strengthen the central motivation of the method.

Reviewer_Vs3COpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HARMONY, a diffusion Transformer self-guidance method that requires no additional training. It constructs weak predictions by reading intermediate layer features through a shared final prediction head and adaptively adjusts the strength of guidance in both time and space based on consistency among multiple probes. The Lite version uses two intermediate layer readouts, while the Full version further incorporates random block dropping for prediction. The paper evaluates quality and computational overhead across tasks including class-conditional image generation, text-to-image generation, and video generation.

Critical strengths
  1. Low-overhead inference design has practical value. Lite version reuses the intermediate states and final prediction heads from the normal forward pass, eliminating the need for auxiliary training or redundant computation of Transformer blocks.
  2. The experimental coverage and mechanism analysis are quite comprehensive.
Critical weaknesses
  1. This paper lacks a discussion of a series of highly relevant works, which significantly reduces the novelty of the core methods it presents. Internal Guidance and RAE v2 involve extrapolating the intermediate predictions from the same forward process to the final predictions, which is directly related to the basic approach in this paper. This paper does not conduct discussions or experimental comparisons with these works. And the further "Full version" looks like just incorporating the existing methods.
  2. There is inconsistency between the experimental records in the main text and those in the supplementary material. For example, the result in the last row of Table 5 in the main text does not match the results in Table H.1 in the supplementary material.

[1] "Guiding a diffusion transformer with the internal dynamics of itself." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [2] "Improved baselines with representation autoencoders." arXiv preprint arXiv:2605.18324.

Questions for authors

Given that the core idea of its method is similar to that of existing works and this manuscript has not conducted experimental comparisons or related discussions, I believe the novelty of this work is relatively limited. Therefore, I am temporarily giving a weak rejection.

Reviewer_Q54jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

Final justification after rebuttal

Firstly, I appreciate the authors' effort in the rebuttal

Since most of my concerns are now clear, therefore I want to reassess my score to acceptance

Thanks


This is an emergency review.

The problem formulation is clear. Many diffusion test-time guidance methods (most notably CFG and other perturbation-based guidance methods such as SAG, PAG, and SEG) require additional model evaluations or computational costs (i.e., NFEs) to produce a weak signal. However, this work focuses on obtaining such a weak signal without introducing an additional model. By leveraging intrinsic signals from DiT, specifically intermediate hidden states hh, the proposed method aims to avoid the additional computational cost.

This constitutes the depth-wise approach. In contrast, the width-wise approach, which resembles Chen et al., 2026, requires additional NFEs (≈0.9\approx 0.9) but achieves similar or better performance than the depth-wise approach. It also outperforms methods such as PAG (Ahn et al., 2024), which require an additional NFE (=1=1).

The authors further propose that stronger agreement between distinct probe signals can justify applying stronger guidance. Based on this intuition, they introduce adaptive guidance weights across both denoising steps and spatial tokens, applicable to both the depth-only and depth-and-width configurations.

Critical strengths

S1. Strong Performance

As shown in Table 2, HARMONY achieves good FID performance compared to a similar method (Chen et al., 2026), demonstrating that its importance-aware dropping works well. In particular, the well-designed ablation study in Table 5 demonstrates the effectiveness of the proposed method.

Critical weaknesses

W0. Presentation Quality

In particular, the quality of Figure 1 is poor. There are no clear definitions of (a)(a), (b)(b), or α\alpha in the figure (Although I found the definitions of (a)(a) and (b)(b) in Section 3.2 and α\alpha in Section 3.1, these notations are not properly explained in the figure or its caption). Also, Πf⊥\Pi_f^\perp is labeled as "radial projection," but isn't its intended operation the removal of the radial component?

I think the overall presentation quality is low. For example, there are overlapping elements (e.g., "O" and the direction arrow).

Furthermore, what does each color represent in the left panel of Figure 6? The figure provides no legend explaining these colors. What do the two blue circles in the bottom-left corner of the figure represent (i.e., DiT-S)?

W1. Unclear Assumptions

Perturbation-driven guidance methods, most notably Hong et al., 2023, and CFG (Ho & Salimans, 2022) provide clear rationales for defining their respective weak predictions. Karras et al., 2024 may share a similar intuition with this work; however, the rationale for treating an intermediate readout as a meaningful weak prediction remains unclear.

I think this work was partly motivated by Chen et al., 2024. The authors also introduce an additional width probe, but this approach appears quite similar to Chen et al., 2026. Thus, I think the novelty is limited, particularly considering Chen et al., 2026, unless the authors can provide a convincing rationale for why the proposed depth direction should improve generation.

W2. Experimental Scope

There are already several guidance methods that do not require additional neural function evaluations (NFEs), such as Kim & Sim, 2025, Li et al., 2024 (SG-prev), and Sadat et al., 2025. Therefore, I think the experimental comparison is insufficient. Please include comparisons with at least two additional relevant baselines under matched inference budgets.

Questions for authors

Please see the Weaknesses above.

Reviewer_AuDdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free guidance framework, namely HARMONY, for diffusion transformers. It constructs weak predictions using intermediate residual-stream readouts and an optional block-dropped forward pass, referred to as depth and width probes, respectively. By combining both resulting guidance directions with adaptively adjusted strengths, followed by an orthogonal projection, HARMONY further improves generation quality. Extensive experiments across diverse model scales and generation tasks demonstrate the overall effectiveness of the proposed framework.

Critical strengths
  1. The paper conducts extensive experiments across diverse model scales and generation tasks, demonstrating the overall effectiveness of the proposed framework. Additional experiments investigate the sensitivity of the method to various hyperparameters and design choices, providing useful insights into its empirical behavior.

  2. The proposed method is training-free and can be readily integrated into existing diffusion transformers without modifying model parameters, making it practically valuable.

Critical weaknesses
  1. The novelty of the proposed framework and the contribution of its individual components are not sufficiently distinguished. The framework combines several existing ideas: weak-model guidance, intermediate-layer readouts, stochastic block dropping, adaptive guidance scheduling, and orthogonal projection. While their integration is reasonable, it remains unclear which component represents the central methodological contribution beyond these existing techniques.

  2. The theoretical results do not directly justify several key choices in the implemented algorithm, which remains largely heuristic. Specifically, Eq. (5) relies on strong assumptions without establishing why probe agreement should reliably indicate a beneficial correction direction. Its further approximation using sample-wise cosine similarity introduces another substantial gap between theory and implementation. Eq. (8) merely characterizes the response of intermediate readouts to late residual updates, without proving that the resulting direction can reduce prediction errors. Similarly, Eq. (9) presents a rather trivial bound arising from the designed constraints, which offers little insight into why the method should work. More fundamentally, the entire theoretical analysis seems to take the effectiveness of weak probes for granted, while overlooking the most fundamental question: why should the difference between a full prediction and a weak prediction (especially the proposed depth-based scheme) point toward reducing the actual prediction error? Overall, the theoretical analysis appears to be a contrived post-hoc rationalization of heuristic design choices, rather than a principled theoretical foundation for the proposed method.

  3. The paper implies excessive AI involvement and insufficient authorial oversight. This is reflected in the seemingly contrived theoretical arguments that are weakly connected to the actual method, the distinctly AI-styled figures (particularly Figure 1), and the excessively defensive writing throughout the paper. Rather than presenting a clear and coherent scientific narrative, the manuscript frequently appears to rationalize its design choices and preempt potential criticisms. While the use of AI tools is not inherently problematic, the overall presentation raises concerns about whether the authors have sufficiently engaged in the critical reasoning, validation, and refinement of their own work. From a presentation perspective, I find it difficult to identify the core original contributions and assess the extent of the authors' intellectual ownership of the proposed ideas.

Questions for authors
  1. In Table 5, why are the configurations combining the width probe alone with either the temporal or spatial gate omitted? Moreover, since the agreement-based gates require two guidance directions, how are they computed in the width-only configuration with both gates enabled (i.e. the last line)?

  2. Why are the additional errors introduced by each probe assumed to have zero mean? Could these errors exhibit a systematic directional bias? Fuithermore, why are the additional errors from different probes assumed to be uncorrelated? Given that the probes share substantial computations and representations, wouldn't their errors also contain shared directional components?

  3. How does the proposed method compare with existing guidance methods in terms of visual generation quality? Providing side-by-side qualitative comparisons using actual generated samples would help better illustrate the improvements and offer a more intuitive understanding of the proposed method's effectiveness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.