acceptodds
Withdrawn from ICLR 2027

Persistence Forcing: Exploiting Feature Specialization in Pixel-Space Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pixel-space diffusion Transformers (DiTs) directly operate on high-dimensional visual data, yet their hidden representations typically undergo uniform refinement across depth. Natural images, however, are inherently organized at different levels of granularity. Global structure can often be represented compactly, whereas local textures and fine details require richer representations. Motivated by this, we introduce heterogeneous refinement in pixel-space DiTs, assigning different feature groups distinct refinement budgets across depth. Consequently, an ordered feature specialization emerges: sparsely refined features predominantly encode global visual structure, whereas more frequently refined features increasingly specialize toward localized, high-frequency details. We refer to these two groups as persistent and active features, respectively. Building on this emergent specialization, we introduce Persistence Forcing (PerF), which explicitly exploits this persistent–active feature organization for pixel-space image generation. This enables persistent features to continuously condition actively refined features, allowing stable global information to guide the ongoing refinement of finer visual details. During generative sampling, this interaction further induces a meaningful guidance direction that promotes coherent global structure and naturally complements classifier-free guidance. On ImageNet , PerF-L achieves FID of , approaching of JiT-H with only half the parameters, while PerF-H further achieves FID of and on ImageNet and , respectively.

Primary area:
generative models
Keywords:
Diffusion models, Image generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_M4hvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript lets different feature channels in a pixel-space diffusion Transformer (JiT) go through different numbers of updates (heterogeneous refinement). The authors find that the channels updated less tend to encode global structure, while the channels updated more tend to encode high-frequency details. Based on this, they propose Persistence Forcing (PerF), which uses the persistent features to modulate the active features, and then builds an extra sampling guidance (Persistence Guidance, PG) by switching this pathway on and off. They report that this consistently beats the JiT baselines on ImageNet 256/512.

Critical strengths
  1. The motivation and the method connect fairly clearly. The manuscript first observes the feature specialization, then shows that specialization alone is not enough (Table 1), then introduces the conditioning, and finally arrives at PG. The logic is complete.
  2. The cost-effectiveness of the method is also decent. PerF-L (471M) gets an FID of 1.91, close to the 1.86 of JiT-H (953M). Even after counting the extra forward pass for PG, the compute is still lower than JiT-H.
  3. The writing and the figures/tables are fairly clear.
Critical weaknesses
  1. I think the biggest point of contention in this manuscript is the causal chain from heterogeneous refinement to specialization to performance gain, and also the similarity to Register Guidance. Throughout the Introduction and the Method, the authors keep telling the reader that heterogeneous refinement lets different channels get different refinement budgets, that these channels then spontaneously develop into persistent global features and active local features, and that this improves generation. But the results they actually report come from the combination of heterogeneous refinement only + P-to-A conditioning + Persistence Guidance. This means the current experiments cannot isolate or verify whether heterogeneous refinement is a necessary factor for the final gain. I think the authors should at least run Uniform + conditioning, or Uniform + conditioning + PG, to show whether heterogeneous refinement is really effective. To me, it is entirely possible that the real story is a different one. Maybe the persistent specialization produced by heterogeneous refinement is not what matters, and the authors have simply added an extra conditioning branch by hand, randomly dropped this branch during training, and then used the two predictions for self guidance at sampling time.

  2. I think adding REPA to the main table does not mean much. REPA is not a necessary part of the core PerF mechanism. It could be a supplementary result, but it should not stand in for the main PerF results. What readers want to see is how much PerF itself improves things, or what the potential risks are, rather than a result that looks strong but has little meaning. At least for me, it is not a plus.

  3. I think the Persistence Guidance component is actually very similar to the Register Guidance proposed in "Registers Matter for Pixel Space Diffusion Transformers". Compared with that method, you train with random P-to-A dropping, while Register Guidance trains with random register dropping. It feels like there is not that much difference. Also, what Figure 3(b) shows as the thing you truly rely on is the existence of an internal conditioning pathway that can be switched on and off. But does this pathway really have to come from the persistent features produced by heterogeneous refinement? The manuscript does not answer this question clearly either (and this is actually a small version of the question in point 1, namely whether heterogeneous refinement is necessary for PG).

  4. The FLOPs overhead currently seems to be underestimated. PG + CFG actually needs 3 predictions, but Table 11 only reports the increase in single forward FLOPs. If you compute it, the actual sampling FLOPs come out to about 1.6x the baseline (compared with a normal CFG JiT without PG). That is quite a big gap.

  5. Besides that, I also have some suggestions on writing details. For example, the Introduction says "Global structure can often be represented compactly, whereas local textures and fine details require richer representations (Burt & Adelson, 1983; Mallat, 1989; Schwartz & Simoncelli, 2001). ... This motivates us to reconsider whether all hidden features should receive the same amount of refinement in pixel-space DiTs." I feel that "global structure can be represented compactly" leads to the conclusion that representation capacity (width) can differ, while what the paper manipulates is the number of updates (refinement budget). These two are clearly different, and the argument connecting them is missing.

  6. The PG training and inference details are currently incomplete. The authors say "We therefore stochastically disable the persistent-conditioning pathway during training... xup\mathbf{x}_u^p ... xc∅\mathbf{x}_c^{\varnothing}", but they do not explain how exactly "disable" is implemented (set to zero? removed?). The reason why the CFG interval [0.1,1] differs from the PG interval [0,1] is also not explained. In addition, key hyperparameters such as the P-to-A dropout probability of 0.1 and the REPA weight of 0.1 are not ablated.

  7. Some of the hyperparameter choices are also not transparent. The CFG scales are 2.8/2.7/3.0/2.9 and the PG scales are 3.0/2.8/1.9/2.0, but there is no explanation of how these scales were chosen.

  8. The current manuscript lacks a discussion of limitations.

Questions for authors
  1. Can the authors clarify whether heterogeneous refinement itself is necessary for the gains of PerF?

  2. How is Persistence Guidance methodologically different from Register Guidance in "Registers Matter for Pixel Space Diffusion Transformers"?

  3. Why are the main results reported with REPA rather than using PerF alone as the primary setting?

  4. Could the authors report the actual sampling cost of PerF when PG is enabled?

  5. Could the authors better justify the connection between compact representation and refinement budget?

  6. Could the authors provide more details and ablations for the PG training and sampling configuration?

  7. Could the authors discuss the main limitations of PerF?

Reviewer_S26sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper starts with a critical observation that natural images have different levels of granularity within one image, but standard DiT models do not explicitly model this behavior. The authors make part of the d-dimension feature updatable while the remaining dimensions are bypassed, where the exact configuration shifts across layers. This structural bias induces a clear feature organization but does not improve generation. Therefore, the authors further make information flow from persistence features to active features, achieving generation improvement.

Critical strengths
  1. The overall presentation is clear.
  2. The motivation is clear, and the method is really inspired by the motivation. So, the entire manuscript is coherent.
Critical weaknesses
  1. I feel the goal of improving generation itself limits the potential contribution of this work. It might be a more interesting and insightful topic to study why vanilla DiT models do not naturally converge to such a behavior, and why the latent/pixel difference matters in this observation. Currently, this paper simply provides yet another method to improve generation, which we already have a bunch of in the community, and they are not likely to be adopted by anyone in the future.
  2. The authors state that the vanilla JiT model does not have feature partition based on granularity. Since details are not fully disclosed, I can only guess here. But I guess the authors simply merge some continuous dimensions as a group and visualize it. However, it is possible that a granularity-based feature partition is actually within the vanilla JiT, but dimensions within one group are actually scattered across discontinuous elements. So, the structural bias does not create such a feature partition behavior, but only makes it more perceivable by placing dimensions in the same feature group together.
  3. A very specific question concerns me. The authors align the computation using ∑l=1Ldl2=Ld2\sum_{l=1}^L d_l^2=Ld^2. So, the two models share the same layer count (LL), and we can see the smallest dld_l must be smaller than dd. Hence, the largest dld_l should be larger than dd. This suggests that the same patch is encoded using more elements in the new model than in the vanilla model. This might be the real root of improvement. (Update: I check the appendix, and it is true that the largest dld_l is larger than dd)

If both point 2 and 3 are true, the whole story would be like this. Given the granularity-based feature partition, certain computation within the vanilla JiT is somehow redundant: the dimensions corresponding to persistent features do not receive as frequent and significant updates as active features. The authors simply delete all the redundancy at first. But we know DiT attention is bidirectional, so this deletion also prevents active features from gaining information from the persistent features, which leads to the initial generation drop in section 2.3. The authors further add something like a compressed unidirectional attention back to the backbone. In this way, the redundancy of updating persistent features at every block is squeezed. The authors exploit this released space to add more computation so that the total computation is still aligned to the vanilla JiT, leading to generation improvement. Therefore, what the community can actually learn is only "updating persistent features at every block is redundant", which is not insightful enough for publication.

Questions for authors
  1. Please clarify how the visualization of the vanilla JiT in Figure 1 is conducted.
Reviewer_GmpeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a heterogeneous architecture that allocates varying refinement budgets across network depth, departing from the uniform feature processing of existing methods. Specifically, the model implicitly partitions features into two groups: persistent features, which encode global visual structure, and active features, which capture localized, high-frequency details. Experiments on ImageNet demonstrate that this approach achieves an FID score comparable to the JiT baseline, with performance gains becoming increasingly pronounced as model size and image resolution scale.

Critical strengths
  • The motivation of heterogeneous is justifiable.
Critical weaknesses

Overall, the paper writing is very challenging to read and this gives me a serious concern of the overuse of AI for writing.

  • It is not clear how table 1 and figure 1 partition features into 2 groups. Though L149 says that feature groups are determined implicitly through output projection but how this decomposition work is not described and does it require training?

Exposition:

  • The intro flow needs to be improved. It starts with the problem of uniform refinement, then provides the observation to motivate the heterogeneous, and describe the proposed solution in the third paragraph. This is reasonable story telling but the third is still mixed between what the paper tackles and what done in past work. It is better to clearly separate them by first talk about the what other have done after the first paragraph before bringing the motivation and the method.
  • Not just the intro, whole paper needs work since current manuscript is hard to follow. For example, reading entire method I still fail to understand how feature partitions work.
Questions for authors
  • How bypassed features are performed? Similar to carrying over or just omit
  • How to decide which features to bypass, what depth to do so?
  • What exactly constraction-expansion schedule?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.