acceptodds
Withdrawn from ICLR 2027

ScFlow: Spectrum-Aware Correction for Stabilizing Training-Free Spatial Control in Rectified Flow Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Rectified flow models (RFMs) have emerged as an alternative to score-based diffusion models (SDMs). Although training-free spatial-condition generation was successful in SDMs, so far, there has been no systematic study on RFMs. In this paper, we pinpoint that prior methods do not perform as well on RFMs. We focus on energy-guided sampling as a representative paradigm, and show that RFMs have a highly anisotropic latent-feature Jacobian compared to SDMs at early diffusion stage, which can unevenly emphasize or neglect different components of the control signal. We then develop Spectral Correction Flow (ScFlow), a training free spatial control framework targeting RFMs. ScFlow consists of two correction components, channel-wise whitening and prompt reweighting. Compared to the uncorrected baseline, ScFlow improves DINO self-similarity by 14.1% and CLIP score by 11.8% on average across evaluated RFMs. Extensive experiments demonstrate that ScFlow provides better controllability on RFMs and outperform state-of-the-art training-free methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
controllable image generation, diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LuxfOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates instability in training-free energy-guided spatial control for rectified flow models. The authors observe stronger early-stage anisotropy in the latent-feature Jacobians of the evaluated RFMs compared with score-based diffusion models, and identify it as a potential contributor to unstable guidance. They propose ScFlow, which introduces two inference-time corrections: channel-wise whitening of guidance gradients and prompt embedding reweighting. Experiments on ImageNet-R-TI2I across multiple spatial conditions and model backbones, supplemented by component ablations, sensitivity analyses, and human evaluation, demonstrate improved spatial and textual alignment over the uncorrected FreeControl baseline.

Critical strengths
  1. Simple corrections with substantial empirical support. The proposed components are straightforward and require no additional model training. The experiments demonstrate improved spatial and textual alignment over FreeControl and favorable trade-offs against other training-free methods. The comparisons, component ablations, sensitivity studies, and human evaluation collectively support the method’s practical usefulness.

  2. Clear and easy-to-follow presentation. The paper progresses logically from observed control failures to spectral analysis and then to the proposed corrections. The mathematical formulation and figures make the problem setup, motivation, and implementation accessible, helping readers follow the reasoning and assess the empirical evidence.

Critical weaknesses
  1. Limited insight into the origin of the spectral differences. While the paper documents stronger early-stage Jacobian anisotropy in the evaluated RFMs, it provides limited explanation of why their spectra differ from those of score-based diffusion models. In particular, the connection between the observed phenomenon and rectified-flow training or rectification remains unclear. It would be nice that if the authors can provide a more insightful explanation of this phenomenon to strengthen the contribution.

  2. Limited benchmark coverage. The quantitative spatial-control evaluation is primarily conducted on ImageNet-R-TI2I, a relatively small benchmark. Although multiple condition types and model backbones are evaluated, these experiments largely reuse the same underlying images, and the prompt-variation studies are also derived from the original evaluation set. This limits the evidence for generalization to more diverse scenes and conditions. Evaluation on an additional spatial-control benchmark, such as the prompt-driven conditional generation dataset released with Ctrl-X, would strengthen the empirical conclusions. I would be happy to raise my score if the authors provide convincing results on an additional benchmark that address the generalization concern,.

Questions for authors
  1. Figure.6 presents results with C1 and C2 combined. It would be interesting to see how C2 performs on its own and whether it already brings improvements over the baseline. I am also curious about how much C2 helps with more complex prompts, such as the long descriptive and multi-object prompts explored in the appendix.

  2. It would be useful to compare whitening with simpler alternatives, such as global gradient normalization or per-channel standardization. Could these methods achieve similar improvements with appropriately tuned guidance strengths? This would help clarify whether accounting for correlations between channels provides additional benefits.

Reviewer_XRsuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the instability of training-free spatial control methods when transferred from score-based diffusion models (SDMs) to rectified flow models (RFMs). Focusing on energy-guided sampling, the authors observe that RFMs exhibit stronger spectral anisotropy in their latent-feature Jacobians during early sampling stages, potentially causing certain control directions to dominate while suppressing others.

To address this issue, the authors propose ScFlow, consisting of channel-wise whitening and prompt reweighting. The former normalizes the channel covariance of guidance gradients, while the latter modifies text embeddings to influence cross-attention and Jacobian anisotropy. Experiments on InstaFlow and Rectified Diffusion demonstrate improvements over FreeControl in spatial and semantic alignment. However, the theoretical justification, comparisons with simpler alternatives, and applicability to modern flow-based architectures remain insufficiently established.

Critical strengths
  1. The paper identifies an interesting instability phenomenon when transferring existing energy-guided spatial control methods from SDMs to RFMs. The comparison of Jacobian spectral properties provides a potentially useful perspective on this behavior.

  2. The proposed corrections are training-free and relatively straightforward to integrate into existing energy-guided sampling frameworks. The experimental results show improvements over the uncorrected FreeControl baseline across multiple spatial conditions.

Critical weaknesses
  1. The connection between Jacobian spectral anisotropy and spatial control failure is not sufficiently established. The actual guidance gradient depends on both the singular values and the projection of the energy gradient onto the corresponding singular vectors. Moreover, channel-wise whitening operates on the covariance of a particular guidance gradient rather than directly correcting the Jacobian spectrum. It remains unclear whether the improvements are specifically attributable to the proposed spectral explanation.

  2. The proposed corrections resemble conventional gradient preconditioning and attention scaling, but comparisons with simpler alternatives are missing. Gradient clipping, global gradient normalization, per-channel standardization, and adaptive guidance scaling could potentially achieve similar improvements. Without these comparisons, the independent technical contribution of ScFlow is difficult to assess.

  3. The discussion of recent training-free control methods is incomplete. FreeControl: Efficient, Training-Free Structural Control via One-Step Attention Extraction [1] has already demonstrated efficient training-free structural control on FLUX.1-dev but is not discussed. Although it primarily uses RGB references rather than sparse spatial conditions, it remains highly relevant to the broader problem of training-free structural control on modern flow-based models.

  4. The complete method is evaluated only on UNet-based RFMs, despite the importance of modern Transformer-based flow models such as SD3.5 and FLUX. Although the appendix provides results for channel-wise whitening on SD3 and SD3.5, prompt reweighting is not evaluated on these architectures. Since this component contributes substantially to the improvements in the main ablation, the generality of the complete framework remains unverified.

  5. The experimental evaluation does not adequately establish the quality-efficiency trade-off. The main experiments use 40 inference steps and additional gradient-guidance steps, while few-step generation is an important advantage of RFMs. Furthermore, the SDS refinement stage accounts for much of the LPIPS improvement, although higher LPIPS does not necessarily indicate better image quality. The human evaluation also omits the spatial conditions and prompts, preventing direct assessment of controllability.

Minor Weaknesses

  1. Figures 3 and 4 are difficult to read due to small fonts, crowded curves, and overlapping annotations. These figures contain the central evidence supporting the spectral analysis and should be substantially redesigned.

  2. Figure 7 is placed in the Conclusion section despite presenting additional qualitative results. It would be more appropriate in the Experiments or Limitations section.

  3. The repeated use of generic labels such as C1 and C2 makes the method description and ablation figures unnecessarily difficult to follow. Descriptive component names would improve clarity.

Ref:

[1] FreeControl: Efficient, Training-Free Structural Control via One-Step Attention Extraction. (NeurIPS)

Questions for authors

In rebuttal, I want to see:

  1. Can the authors demonstrate that the observed improvement is specifically due to correcting Jacobian spectral anisotropy rather than conventional gradient normalization or preconditioning?

  2. Can the authors explain why uniformly scaling text embeddings improves controllability, given that this operation affects both attention Keys and Values? Would Key-only or Value-only scaling produce similar results?

  3. Why is the complete method not evaluated on SD3.5 or FLUX? How does it compare with recent training-free structural control methods on modern flow-based backbones?

  4. How does ScFlow perform under substantially smaller inference budgets, such as 4, 8, or 16 steps? What improvements remain when SDS refinement is excluded?

Reviewer_tnbJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates why FreeControl-style energy-guided spatial control transfers poorly to rectified models. It reports stronger early-stage concentration in latent-feature Jacobian spectra and proposes two inference-time corrections. channel-wise gradient whitening (C1) and prompt-embedding scaling (C2). Experiments cover five spatial-condition types on InstaFlow and Rectified Diffusion, with additional C1-only results on SD3/SD3.5.

Critical strengths
  1. Table 8 reports that unrefined ScFlow improves CLIP from 0.268 to 0.278 and DINO self-similarity distance from 0.088 to 0.076, at the same reported 15 s runtime as FreeControl. This supports a useful contribution from the corrections themselves.

  2. The main experiments cover two UNet-based rectified models, while Appendix A.2 shows C1-only improvements on SD3 and SD3.5. The sensitivity analyses and blinded human quality comparison provide additional support for the practical recipe.

Critical weaknesses
  1. For invertible RtR_t, Eq. (6) implies

    \qquad \|\widetilde g_t\|_2^2 = CN.$$ Thus C1 removes channel means and fixes the total gradient magnitude, in addition to whitening channel covariance. Figure 6 lacks a centered, norm-matched baseline and a C2-only ablation. Consequently, it does not establish whether full covariance correction is necessary for the gains. Furthermore, C2 scales values as well as keys, confounding attention temperature with attention-output magnitude.
  2. Section 4.2 does not clearly specify whether JuJ_u differentiates native noise/velocity predictions or a common sampling drift. Equivalent parameterizations can satisfy Ju=atI+btJϵJ_u = a_t I + b_t J_\epsilon, under which the proposed statistic S(J)S(J) is not invariant. Figure 3a therefore requires an explicit output-parameterization and noise-level check before attributing the difference to rectified dynamics. Figure 3b provides additional evidence, but Eq. (3) also depends on uj⊤wu_j^\top w: a concentrated Jacobian spectrum alone does not establish that the actual spatial-control signal loses relevant directions.

  3. Provide the selected feature layers, power-iteration stopping rule, practical trace estimator, and treatment of singular RtR_t. The inverse square root in Eq. (6) is undefined for singular covariance without a specified regularization or pseudoinverse.

  4. The spatial-control objective comes from FreeControl [1]; C1 uses standard ZCA whitening [2]; latent-feature Jacobian SVD and prompt-dependent geometry were studied in [3]; and inference-time attention-temperature modification appears in ERG [4]. Please cite and distinguish the latter three. These precedents narrow the contribution to the rectified-model diagnosis and the effectiveness of this combination; they do not establish that the complete method was previously proposed.

  5. Section 2: “many of these work primarily focuses” should be “many of these works primarily focus.” Section 7: “designing better energy objective” should be “designing a better energy objective.” Appendix C.2: “for fair comparison purpose” should be “for a fair comparison.”

Questions for authors
  1. On the 100 pairs used for sensitivity analysis, use identical seeds, the same 15-of-40 guidance schedule, and no SDS refinement. Compare tuned FreeControl, centered scalar-normalized guidance CNG^t/∥G^t∥F\sqrt{CN}\widehat G_t/\|\widehat G_t\|_F, C1, C2 alone, and C1+C2 with comparable guidance-scale tuning. Does C1 retain an advantage in CLIP and DINO distance?

  2. For Figure 3, identify the differentiated outputs, feature layers, and noise coordinates. Do the differences persist when outputs are expressed as comparable sampling drifts at matched noise levels? Also, do the actual energy gradients concentrate in the leading Jacobian singular directions, as required by the interpretation of Eq. (4)?

  3. With C1 and the guidance schedule fixed, compare key-only, value-only, and joint scaling. Does the improvement persist when value amplitude is unchanged?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.