acceptodds
Under review as a conference paper at ICLR 2027

What and When to Align: Hierarchy-Decoupled Representation Alignment for Pixel-Space Diffusion

Abstract

Representation alignment accelerates the training of diffusion models by leveraging pretrained visual representations. However, existing methods treat these pretrained visual representations as uniform alignment targets, overlooking differences in semantic granularity and their roles in diffusion training. We investigate these differences in pixel-space diffusion models and find that alignment targets with stronger component-level semantics are associated with faster early improvements in generation quality, whereas category-level semantics provide complementary object-level supervision throughout training. Based on these findings, we propose HierarchY-Decoupled Representation Alignment (HYDRA), which decouples supervision into Scaffold and Anchor alignments depending on their semantic roles. HYDRA further introduces Position-wise Feature Centering to mitigate position-dependent biases in pretrained features used as alignment targets, and a timestep-conditioned projector to adapt alignment to representations that vary across diffusion timesteps. HYDRA improves generation performance across model scales on ImageNet with classifier-free guidance. JiT-B/16 reaches baseline-level FID with 3 fewer training epochs and further achieves 3.15, while JiT-H/16 improves the final FID from 1.86 to 1.74. Our results highlight that effective representation alignment requires considering not only what representations to align, but also when they should guide generative learning.

Primary area:
generative models
Keywords:
Diffusion Models, Representation Alignment, Pixel-Space Diffusion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QsCDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies representation alignment for pixel-space Just image Transformers (JiT). It associates late-intermediate DINOv2 features with component-level similarity and final-block features with category-level similarity, then proposes HYDRA: early Scaffold alignment to the sum of centered Blocks 9 and 10, continuous Anchor alignment to centered Block 11, and separate timestep-conditioned projection heads. On class-conditional ImageNet generation at 256 x 256, the reported 600-epoch FID improves from 3.66 to 3.15 for JiT-B/16, from 2.36 to 2.06 for JiT-L/16, and from 1.86 to 1.74 for JiT-H/16. The B model reaches FID 3.65 after 200 epochs. The paper includes target-selection and scheduling ablations, feature-similarity and gradient analyses, timing measurements, and a shorter DINOv3 experiment.

Critical strengths
  1. A useful improvement to a fixed generative backbone across three scales. Table 2 reports consistent FID reductions over JiT-B/L/H at 600 epochs, with especially substantial improvement for the B model, and near-baseline quality after 200 epochs. This establishes a practical benefit without changing the inference backbone. Appendix C.6 also reports training costs rather than relying only on epoch counts: HYDRA takes 0.68 seconds per iteration versus 0.47 for JiT and 0.74 for PixelREPA, with 7.48M projector parameters versus PixelREPA's 24.03M. Comparable final quality with a cheaper training head is useful even if the small FID differences are not robust. These figures make the efficiency claim assessable, although the threefold reduction in epochs should not be read as a threefold wall-clock speedup.

  2. Several important design choices receive direct empirical checks. Tables 3-5 separate the two objectives, their stopping schedules, and timestep conditioning. Table 4 is particularly relevant: stopping Scaffold at epoch 150 while retaining Anchor yields FID 3.15, compared with 3.22 when neither stops and 3.29 when both stop. Appendix C additionally checks loss weights, centering, target combinations, and an alternative encoder. This provides more support for the final recipe than a single aggregate improvement would.

Critical weaknesses
  1. The evidence does not sufficiently isolate semantic granularity as the explanation for the proposed division of roles. The abstract appropriately describes an association, but Section 3.1.2 goes further in attributing early alignment effectiveness to semantic granularity. Section 3.1 and Figure 3 compare mean cosine similarity for category-only and component-only matches, but neither the main text nor the appendices specify the PartImageNet split, image/pair counts, patch-to-part assignment, balancing, or whether category-only pairs can come from the same image while component-only pairs come from different images. Without a pair matched on neither attribute and appropriate controls, raw cosine changes can also reflect feature geometry or image/position effects. Table 1 establishes that particular intermediate targets work better early; Tables 3-4 establish that the particular two-target recipe and schedule work better. They do not isolate whether the semantic distinction, rather than a second head or a different target geometry, explains that benefit. The local gradient measurements in Appendix C.4 do not resolve this attribution. Please specify and control the similarity protocol and compare against a matched two-head same-target control, or consistently narrow the explanation to an empirical association and target-selection heuristic. This matters because the semantic explanation is a central contribution, beyond the recipe itself.

  2. The reliability of the small component effects and the acceleration relative to the closest alignment approach remain uncertain. In Table 2, the FID advantage over PixelREPA is 0.02/0.05/0.07 for B/L/H; the added Anchor benefit in Table 3 is 0.06, and Scaffold stopping in Table 4 improves FID by 0.07. No repeated-run or sampling-uncertainty estimates are reported. The lower reported training-head cost is a valid benefit even if these FID margins do not persist. However, the 200-epoch acceleration comparison omits PixelREPA at the corresponding budget, so it does not establish the early quality/cost trade-off against that close alternative. Appendix B also explicitly replaces iREPA's convolutional projector with an MLP. That is a reasonable target-isolation experiment, but the early curves should be labeled as iREPA features with an MLP; a faithful original projector is needed only for claims about the complete iREPA method. A matched early learning curve with PixelREPA and uncertainty for the decisive B/16 target/schedule comparisons would make the explanation and acceleration claims substantially more convincing. I am not requesting repeated runs for every scale or requiring state-of-the-art FID; the concern is whether the measurements distinguish the proposed contribution from existing alignment strategies.

Questions for authors
  1. For Figure 3, what are the PartImageNet split, sample and pair counts, label-to-patch rule, and within-image versus cross-image pairing rules? Do the semantic trends persist with balanced image/position controls and a neither-matched reference? Please also clarify whether a matched two-head same-target control has been evaluated; this would help distinguish complementary semantics from the architectural change itself.
  2. Are the PixelREPA entries in Table 2 rerun with the same training implementation, CFG sweep, and EMA/checkpoint selection, or taken from prior reports? What are the corresponding early-budget results and variability for HYDRA and PixelREPA? Do the decisive B/16 target/schedule improvements remain consistent across training/evaluation repeats? Appendix C.6 supplies useful iteration costs, so a comparison at equal elapsed training time would directly address the efficiency contribution.
  3. How are DINOv2 ViT-B/14 features spatially matched to the JiT 16 x 16 token grid for 256 x 256 generation? Appendix B identifies the encoder and alignment block but does not state the encoder image resizing/cropping or feature interpolation procedure. Please provide that mapping and confirm whether it is shared by the comparison methods and by the position-wise centering statistics.
6 comments

(W1 & Q1) Semantic Granularity & Alignment Target Roles, and Similarity Evaluation Protocol

1. Evidence for Semantic Granularity-Based Alignment Target Roles

1) Relationship between Semantic Granularity and Early Alignment across Vision Encoders

To further examine the generalizability of the relationship between Semantic Granularity and Early Alignment, we extend the semantic similarity analysis in Fig. 3 to multiple vision encoders, including DINOv3, SigLIP2, and V-JEPA 2.1. Following the reviewer's suggestion, we additionally include Neither Matched pairs as a reference. Based on the Category Matched Only, Component Matched Only, and Neither Matched similarities, we define the Semantic Granularity Ratio (R) as follows:

R=Component Matched Only−Neither MatchedCategory Matched Only−Neither MatchedR = \frac{\text{Component Matched Only} - \text{Neither Matched}}{\text{Category Matched Only} - \text{Neither Matched}}

As shown in Tab. C1, blocks with higher R generally achieve better Early Alignment performance across DINOv2, DINOv3, SigLIP2, and V-JEPA 2.1, although the blocks with the highest R and best FID do not always coincide exactly. These results provide cross-encoder empirical evidence supporting a consistent relationship between Semantic Granularity and Early Alignment effectiveness, despite differences in encoder architectures and learned representations.

EncoderMetricBlock7Block9Block10Block11
DINOv2-B/14Category Matched Only.147.127.123.248
Component Matched Only.174.208.221.271
Neither Matched.113.101.100.160
Ratio (R)1.7944.115‾\underline{4.115}5.2611.261
FID ↓11.9211.61‾\underline{11.61}11.5214.36
DINOv3-B/16Category Matched Only.231.234.246.366
Component Matched Only.271.361.371.321
Neither Matched.208.211.215.229
Ratio (R)2.7396.5225.032‾\underline{5.032}0.672
FID ↓13.2012.28‾\underline{12.28}12.2414.84
SigLIP2-B/16Category Matched Only.127.211.233.234
Component Matched Only.162.207.202.199
Neither Matched.092.147.150.155
Ratio (R)2.0000.938‾\underline{0.938}0.6270.557
FID ↓12.2913.19‾\underline{13.19}14.3515.26
V-JEPA 2.1-B/16Category Matched Only.128.133.131.141
Component Matched Only.140.164.194.217
Neither Matched.098.105.105.111
Ratio (R)1.4002.1073.423‾\underline{3.423}3.533
FID ↓14.0212.9412.68‾\underline{12.68}12.25

Table C1. Semantic granularity and alignment performance across vision encoders. FID is evaluated on ImageNet 256×256 using JiT-B/16 at 50 epochs.

2) Effectiveness of Role-Specific Alignment Targets

[Comparison with Matched Two-Head Same-Target Controls]

Projector 1 TargetProjector 2 Target50100150200
AnchorAnchor14.525.544.493.84
ScaffoldScaffold11.984.694.073.69
ScaffoldAnchor (HYDRA)12.774.834.053.65

Table C2. FID comparison of two-head same-target configurations and HYDRA across training epochs.

To investigate whether the performance improvement stems merely from adding a second alignment head or from the semantic complementarity of different target features, we conducted controlled experiments. Specifically, we compared two-projector configurations using the same target features for Scaffold and Anchor, respectively, against HYDRA, which aligns features from different encoder layers.

As shown in Tab. C2, Scaffold (Two-head) achieved the best FID of 11.98 at 50 epochs. However, HYDRA showed progressively better relative performance as training proceeded, achieving the lowest FID of 3.65 at 200 epochs. These results suggest that while Scaffold-based alignment can be particularly effective during early training, HYDRA provides additional benefits as training progresses by jointly leveraging target features from different encoder layers.

2. Semantic Similarity Evaluation Protocol

To address the reviewer's concerns regarding the reproducibility of Fig. 3, we clarify the PartImageNet similarity evaluation protocol, using the ground-truth part segmentation annotations provided by the dataset, including the dataset split, patch-to-part assignment, pair sampling, and balancing procedures, as summarized below.

ItemDescription
DatasetPartImageNet test split (2,398 images, 11 supercategories)
Patch-to-Part AssignmentA patch is assigned to a part if the part occupies at least 50% of its area.
Pairing RuleAll pairs are cross-image and sampled within the same supercategory.
Pair ConditionsS11: Both Matched; S10: Category Only; S01: Component Only; S00: Neither Matched
Sampling & Balancing5,450 pairs per condition (21,800 total), balanced across conditions within each supercategory
AggregationMacro-average across 11 supercategories
Feature Geometry ControlNeither-Matched (S00) reference and Position-wise Feature Centering
Image/Position ControlCross-image pairing and position-matched sampling (40,905 pairs per condition)

Table C3. Summary of the PartImageNet Semantic Similarity Evaluation Protocol.

Detailed experimental settings will be provided in the revised Appendix.

[Balanced Image/Position Controls] To examine whether the observed trends are influenced by spatial position, we enumerated all cross-image part pairs within each supercategory and computed the distance between their part centroids on the 16×16 patch grid. Before matching, same-part pairs lay closer together than different-part pairs (mean centroid distance: S11 4.44, S01 4.49 vs. S10 5.69, S00 5.66 patches). Within each supercategory, we then binned pair distances into 1-patch-wide bins and, in each bin, subsampled all four conditions (S11, S10, S01, S00) down to the smallest condition's count. This yielded 40,905 pairs per condition in total, with nearly identical mean centroid distances (4.68–4.70 patches). We repeated the sampling with five random seeds (seed std ≤ .001).

As shown in Tab. C4, cosine similarities changed by at most .004 relative to Fig. 3 across all conditions and blocks after distance matching, and the semantic trends remained unchanged (S01 > S10 from Block 7† onward). The small changes may partly reflect the Position-wise Feature Centering applied in our evaluation: without centering (raw Block 11), the changes are slightly larger (≤ .006). These results indicate that our semantic similarity analysis is robust to spatial position differences.

ConditionBlock 3†Block 5†Block 7†Block 9†Block 10†Block 11†
S11.184→.183 (−.001).201→.201 (+.000).227→.227 (+.000).257→.257 (+.000).273→.273 (+.000).397→.396 (−.001)
S10.137→.133 (−.004).141→.139 (−.002).147→.145 (−.002).127→.126 (−.001).123→.123 (+.000).248→.248 (+.000)
S01.134→.135 (+.001).146→.148 (+.002).174→.176 (+.002).208→.206 (−.002).221→.218 (−.003).271→.267 (−.004)
S00.099→.098 (−.001).103→.102 (−.001).113→.113 (+.000).101→.102 (+.001).100→.100 (+.000).160→.160 (+.000)

Table C4. Mean cosine similarity before → after spatial distance matching (DINOv2-B/14; † = Position-wise Feature Centering). Values in parentheses are the changes

(W2 Q2) Reliability and Training Efficiency

1. Comparison with PixelREPA at 200 Epochs

We clarify that the PixelREPA results in Tab. 2 were taken from the original paper. To further assess early-stage convergence and training efficiency, we independently trained PixelREPA and HYDRA for 200 epochs in our own environment and evaluated both methods using five sampling seeds after conducting a CFG scale sweep. Based on the per-iteration training costs reported in Appendix C.6, we additionally include HYDRA at 217 epochs to enable a comparison under approximately matched wall-clock training time.

MethodEpochsec/iter ↓Seed 0Seed 1Seed 2Seed 3Seed 4Mean ± Std. ↓
PixelREPA2000.743.943.903.883.893.913.90 ± 0.02
HYDRA2000.683.653.663.663.673.683.66 ± 0.01
HYDRA2170.683.623.593.633.633.613.62 ± 0.02

Table C5. Comparison of PixelREPA and HYDRA in terms of FID and training efficiency at early training stages.

As shown in Tab. C5, at 200 epochs, HYDRA achieves a lower mean FID than PixelREPA (3.66 vs. 3.90) while reducing per-iteration training time by 8.1% (0.68 vs. 0.74 s/iter). Furthermore, under an approximately matched wall-clock training budget, HYDRA at 217 epochs further improves the mean FID to 3.62, demonstrating both faster convergence and improved training efficiency.

2. Reliability across Sampling Seeds

All experimental results reported in the original submission were evaluated using sampling seed 0. To further assess the reliability of these results, we additionally evaluated the 600-epoch HYDRA models across five sampling seeds for all three model scales.

ModelSeed 0Seed 1Seed 2Seed 3Seed 4Mean ± Std.
JiT-B/16 + HYDRA3.153.143.213.203.203.18 ± 0.03
JiT-L/16 + HYDRA2.062.052.052.062.042.05 ± 0.01
JiT-H/16 + HYDRA1.741.801.741.751.771.76 ± 0.03

Table C6. FID stability of HYDRA across five sampling seeds for different model scales.

As shown in Tab. C6, the results exhibit low variability across sampling seeds for all three model scales, supporting the stability of HYDRA's generation performance under sampling randomness.

3. iREPA Notation

We have revised the manuscript to explicitly label iREPA as "iREPA (w/ MLP)" in Fig. 1 and clarify the use of an MLP projector in the caption of Tab. 1.

 

(Q3) Spatial Feature Matching between DINOv2 and JiT

[Implementation Details]

We clarify the spatial feature mapping and preprocessing procedures between DINOv2 and JiT.

1. Token Grid Alignment

Following the PixelREPA implementation, we use timm-based resampling to change the patch size of DINOv2 ViT-B/14 from 14 to 16 by adapting the pretrained Patch Embedding and Positional Embedding. Therefore, for a 256×256 input, both DINOv2 and JiT-B/16 produce the same 16×16 token grid, enabling direct one-to-one alignment at each spatial position. Accordingly, no additional feature interpolation is required.

2. Consistent Teacher Feature Extraction

Across all experiments, we apply the same input preprocessing and feature extraction procedures to the teacher encoder. This spatial mapping is consistently maintained across all comparison methods, including PixelREPA. Furthermore, the same teacher input preprocessing and feature extraction procedures are used to compute the statistics for Position-wise Centering.

Reviewer_fzoTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces HYDRA, a representation alignment method for pixel-sapce diffusion models. The authors explore the features from different layers of DINOv2 and find the intermediate layers contain more information. Based on this, they propose HYDRA, which is validated only on ImageNet 256x256.

Critical strengths
  1. This paper explores how features from different DINOv2 layers affect diffusion training.

  2. HYDRA is validated on ImageNet 256x256 across JiT-B, JiT-L and JiT-H, improving the FID and achieving faster convergence.

Critical weaknesses
  1. My main concern is the novelty of the proposed method. The core idea of HYDRA is to extend REPA by using features from both intermediate layers and the final layer, and applying different training schedules to these alignment objectives. The overall method seems like a relatively straghtforward extension of existing representation alignment approaches. More importantly, this paper does not compare with other REPA methods, and even a simple multi-layer REPA baseline.

  2. The experiments are only conducted on ImageNet 256x256, and all the main results are conducted on JiT architecture. There are no experiments on the other datasets, higher resolutions, or more practical tasks such as T2I generation. The method is also not evaluated on other pixel-sapce diffusion models. This makes it difficult to judge how well HYDRA generalizes beyond the current setting.

  3. The method relies on the observation that specific DINOv2 blocks contain different levels of semantic information. However, different visual encoders may have different feature hierarchies, so the same block selection may not always work. This paper is not sufficient to show that the proposed design generalizes to different visual encoders.

Questions for authors
  1. Could the authors compare HYDRA with a simple multi-layer REPA baseline that directly aligns features from several DINOv2 blocks?

  2. COuld the authors provide experiments on other pixel-space diffusion models or datasets?

  3. How should the Scaffold and Anchor layers be selected when using a different visual encoder?

  4. Why is Scaffold Alignment stopped at 150 epochs? How sentitive are the results to this choice?

8 comments

(W1) Concern about the Novelty

[Core Idea]

The core novelty of this work consists of a new key finding on the relationship between the Semantic Granularity of block-wise features in Vision Encoders and Alignment performance, and the separation of alignment roles based on Semantic Granularity and the design of appropriate Alignment Schedules for each role.

1. Key Findings: Semantic Granularity and Alignment Performance

Our key finding is that, although the Semantic Representation capability of Vision Encoders generally improves with network depth, among blocks containing sufficient Semantic information, the relative ratio of Component-level to Category-level information (R in Tab. B2) is more closely associated with Early Alignment performance than Feature Depth alone. In particular, features with higher Component-level ratios tend to achieve better Early Alignment performance.

To further support this key finding, we conducted the following two additional analyses.

1) Deeper Features Do Not Necessarily Yield Better Early Alignment

Block (DINOv2)01234567891011
Top-1 Acc. (%)11.2316.7421.8629.8138.5047.5054.0058.9866.3471.4876.1881.08

Table B1: Layer-wise linear probing Top-1 accuracy (%) of DINOv2 on ImageNet.

As shown in Tab. B1, the layer-wise Linear Probing results of DINOv2 show that ImageNet Top-1 Accuracy consistently increases from Block 0 to Block 11, suggesting that deeper features contain richer Semantic information. In contrast, Tab. B2 shows that Block 10 achieved the best Early Alignment performance with an FID of 11.52, whereas the deeper Block 11 yielded a worse FID of 14.36. These results indicate that improvements in Semantic Representation capability do not necessarily translate into better Early Alignment performance.

Instead, Early Alignment performance showed a stronger association with the relative proportion of Component-level and Category-level information (R). The Ratio (R) is calculated based on the similarities of Component Matched Only and Category Matched Only, as defined in Fig. 3 of the main paper. Specifically, the Neither Matched similarity is subtracted from both values as a baseline, and R is calculated as follows:

R=Component Matched Only−Neither MatchedCategory Matched Only−Neither MatchedR = \frac{\text{Component Matched Only} - \text{Neither Matched}}{\text{Category Matched Only} - \text{Neither Matched}}

As shown in Tab. B2, Block 10 achieved the highest R (5.261) and the best FID (11.52), whereas Block 11 exhibited a lower R (1.261) and worse FID (14.36). These results suggest that Semantic Granularity, particularly a higher relative proportion of Component-level information, is more closely associated with Early Alignment performance than Feature Depth alone.

EncoderMetricBlock7Block9Block10Block11
DINOv2-B/14Category Matched Only.147.127.123.248
Component Matched Only.174.208.221.271
Neither Matched.113.101.100.160
Ratio (R)1.7944.115‾\underline{4.115}5.2611.261
FID ↓11.9211.61‾\underline{11.61}11.5214.36

Table B2: Semantic granularity and alignment performance in DINOv2. FID is evaluated on ImageNet 256×256 using JiT-B at 50 epochs.

2) Relationship between Semantic Granularity Ratio and Early Alignment across Vision Encoders

To further validate this finding across different Vision Encoders, we analyzed DINOv3, SigLIP2, and V-JEPA 2.1, which were trained using different training paradigms (Tab. B3). Despite their distinct feature hierarchies, all three encoders exhibited a consistent trend: blocks with higher Component/Category Ratios (R) tended to achieve better Early Alignment performance. These results provide further empirical support for the relationship between Semantic Granularity and Early Alignment performance beyond DINOv2.

EncoderMetricBlock7Block9Block10Block11
DINOv3-B/16Category Matched Only.231.234.246.366
Component Matched Only.271.361.371.321
Neither Matched.208.211.215.229
Ratio (R)2.7396.5225.032‾\underline{5.032}0.672
FID ↓13.2012.28‾\underline{12.28}12.2414.84
SigLIP2-B/16Category Matched Only.127.211.233.234
Component Matched Only.162.207.202.199
Neither Matched.092.147.150.155
Ratio (R)2.0000.938‾\underline{0.938}0.6270.557
FID ↓12.2913.19‾\underline{13.19}14.3515.26
V-JEPA 2.1-B/16Category Matched Only.128.133.131.141
Component Matched Only.140.164.194.217
Neither Matched.098.105.105.111
Ratio (R)1.4002.1073.423‾\underline{3.423}3.533
FID ↓14.0212.9412.68‾\underline{12.68}12.25

Table B3: Semantic granularity and alignment performance across vision encoders. FID is evaluated on ImageNet 256×256 using JiT-B at 50 epochs.

2. Method Design: Hierarchy-Decoupled Representation Alignment

The novelty of HYDRA lies in distinguishing not only the roles of alignment targets but also when each alignment should be applied, based on these key findings.

Therefore, HYDRA is not merely an extension of multi-layer REPA that jointly aligns features from multiple layers, but rather a method that establishes and implements a design principle for what to align (What) and when to align (When), guided by semantic granularity.

 

(W1, Q1) Comparison with Other REPA Methods and Simple Multi-layer REPA Baseline

Method (DINOv2 Block) / Epoch50100150200
REPA (8 Block feature)13.944.924.163.80
REPA (10 Block feature)13.104.834.153.79
REPA (11 Block feature)15.185.694.624.13
iREPA (11 Block feature) (w/ MLP)12.385.074.363.99
Anchor (Two-head)14.525.544.493.84
Scaffold (Two-head)11.984.694.073.69
HYDRA (Ours)12.774.834.053.65

Table B4. FID comparison of different DINOv2 alignment targets and methods on ImageNet 256×256 using JiT-B/16 across training epochs.

Comparison with REPA Variants: REPA and iREPA were originally proposed for Latent Diffusion, and PixelREPA [ref] highlighted the performance limitations of directly applying these methods to Pixel-space Diffusion. Therefore, for a fair comparison, we adjusted the Alignment Loss Weight of REPA-based methods to 0.1, following PixelREPA. Accordingly, we included PixelREPA in the Main Performance Tab. 2, while presenting the results of REPA and iREPA in the training-stage FID comparison in Fig. 1. As shown in Tab. 2, HYDRA consistently achieves lower FID than PixelREPA across all model scales (B, L, and H).

Comparison with Alignment Baselines: We conducted additional controlled experiments comparing REPA baselines using different DINOv2 blocks as Alignment Targets, as well as Anchor (Two-head) and Scaffold (Two-head) configurations, where both heads align with the same target feature. As shown in Tab. B4, HYDRA achieved the best FID of 3.65 at 200 epochs, outperforming not only the best-performing Single-layer REPA baseline but also both Two-head configurations.

(Conclusion) These findings support that HYDRA's performance gains cannot be explained solely by increasing the number of Alignment Heads, but rather stem from the separation of alignment roles based on Semantic Granularity and the design of appropriate Alignment Schedules for each role.

 

(W2, Q2) Generalization and Effectiveness

To further evaluate HYDRA's generalizability, we conducted additional experiments at higher image resolutions and extended our evaluation to another pixel-space diffusion model.

1. Generalization to Higher Resolution (ImageNet 512×512)

ModelEpochFID ↓
JiT-B/326004.02
JiT-B/32 + HYDRA2004.03

Table B5. FID comparison between JiT-B/32 and HYDRA on ImageNet 512×512.

To evaluate HYDRA at higher resolutions, we conducted additional experiments using JiT-B/32 on ImageNet 512×512. As shown in Tab. B5, HYDRA achieved an FID of 4.03 in only 200 epochs, nearly matching the baseline FID of 4.02 obtained after 600 epochs. This demonstrates that HYDRA achieves comparable generation quality with only one-third of the training epochs, extending its convergence acceleration effect to 512×512 resolution. The 600-epoch HYDRA experiment is currently at 300 epochs and is expected to finish within approximately one day. We will include the results in the rebuttal once available.

2. Generalization to Other Pixel-Space Diffusion Models (PixelU)

Model600 Epochs
PixelU-B/163.40
PixelU-B/16 + HYDRA3.18

Table B6. FID comparison between PixelU-B/16 and HYDRA.

To further evaluate HYDRA's generalizability beyond JiT, we conducted additional experiments on PixelU-B/16. As shown in Tab. B6, HYDRA achieved an FID of 3.18, compared to 3.40 for the PixelU-B/16 baseline at 600 epochs. These results demonstrate that HYDRA's effectiveness extends beyond JiT to other pixel-space diffusion architectures.

[a] Guo, Zipeng, et al. "Pixelu: A u-shaped transformer for efficient end-to-end pixel diffusion." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

(W3 & Q3) Cross-Encoder Scaffold–Anchor Target Selection

[Block Selection Criteria Based on Semantic Granularity]

HYDRA's Scaffold does not necessarily require two features, and Blocks 9 and 10 in DINOv2 were selected from the Scaffold candidates based on the encoder's Semantic Granularity analysis. Therefore, rather than fixing the same blocks across different Vision Encoders, Scaffold and Anchor can be selected according to the feature characteristics of each encoder. To this end, we propose the following consistent Block Selection Recipe.

  • Candidate Block Selection: As shown in Tab. B1, the layer-wise Linear Probing results of DINOv2 demonstrate that Top-1 Accuracy gradually increases with network depth, suggesting that deeper features contain richer Semantic information. Furthermore, Tab. 1 in the main paper shows that features from later blocks with sufficient Semantic information generally achieve better Alignment performance than those from earlier blocks. Based on these observations, we divide the Vision Encoder blocks into two halves and select the latter half as Alignment Target candidates.

  • Scaffold Selection: Among the candidate blocks, we select those with higher Ratios (R), as defined in Tab. B2, as Scaffold Targets. This aims to leverage features that contain sufficient Semantic information while emphasizing relatively stronger Component-level information.

  • Anchor Selection: Among the candidate blocks, we select the block with the highest Category Matched Only Similarity as the Anchor Target. This aims to leverage Category-level Semantic information to provide continuous Supervision throughout training.

As shown in Tab. B3, our Semantic Granularity analysis of DINOv2, DINOv3, SigLIP2, and V-JEPA 2.1 reveals distinct hierarchical distributions of Component-level and Category-level information across encoders. Therefore, the key principle of HYDRA lies not in selecting specific blocks, but in distinguishing the roles of Scaffold and Anchor based on each encoder's Semantic Granularity. This Block Selection Recipe can be consistently applied to Vision Encoders with different feature hierarchies, and the Early Alignment results across DINOv3, SigLIP2, and V-JEPA 2.1 further support its effectiveness.

 

(W3 & Q4) Scaffold Early-Stopping Rationale and Sensitivity

The 150-epoch early stopping point for Scaffold Alignment was selected based on the observed Scaffold Alignment Loss dynamics. We further evaluated its sensitivity by comparing the final FID scores across different stopping epochs.

1. Rationale for the 150-Epoch Stopping Point

As shown in Tab. B7, the Scaffold Alignment Loss of JiT-B decreased rapidly during the first 100 epochs, after which the rate of decrease gradually slowed. Similar trends were observed in JiT-L/H (Tab. B8), indicating that Scaffold Alignment progresses primarily during early training, with diminishing improvements thereafter. Based on these consistent trends, we selected 150 epochs as the stopping point across model scales, although the optimal epoch may vary with model scale.

Epoch50100150200250300350400450500550600
Scaffold Loss ↓0.50370.44110.43180.42670.42360.42130.41960.41810.41670.41580.41500.4141

Table B7. Scaffold alignment loss of JiT-B/16 across training epochs without early stopping. Each value represents the average loss over the preceding 50 epochs.

Scaffold Loss ↓50100150
JiT-L/160.46300.39890.3874
JiT-H/160.45230.39220.3805

Table B8. Scaffold alignment loss of JiT-L/16 and JiT-H/16 during early training.

2. Sensitivity to the Scaffold Alignment Stopping Epoch

Stop Epoch100150200500
FID ↓3.193.153.223.24

Table B9. Effect of the Scaffold alignment stopping epoch on the FID of JiT-B/16 trained for 600 epochs.

To evaluate the sensitivity to the stopping epoch, we compared the final FID scores by varying the stopping point to 100, 150, 200, and 500 epochs (Tab. B9). The results show that stopping at 150 epochs achieved the best FID of 3.15, while 100 epochs yielded a comparable FID of 3.19. Extending supervision to 200 and 500 epochs resulted in slightly worse FID scores of 3.22 and 3.24, respectively. These results support the effectiveness of early stopping and indicate that the performance is not highly sensitive to the exact stopping epoch.

Reviewer_SzPBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies representation alignment for pixel-space Just image Transformers, the central observation is that late-intermediate DINOv2 features exhibit stronger component-level similarity, whereas final-block features provide stronger category-level similarity. Based on this , the proposed method, HYDRA, assigns these representations different roles: centered Blocks 9 and 10 form a temporary Scaffold target that is stopped after 150 epochs, while centered Block 11 forms an Anchor target retained throughout training. The method also introduces position-wise feature centering and separate timestep-conditioned projectors. On ImageNet 256x256, HYDRA improves JiT-B/L/H and reaches approximately the 600-epoch JiT-B baseline FID after 200 epochs.

Critical strengths
  1. The paper provides an informative analysis of representation hierarchy. The analysis of ImageNet similarities, clustering, and block-wise target comparisons give a concrete account of how intermediate and final DINOv2 features differ. This is more informative than treating the teacher representation as a single fixed target.

  2. ** The proposed HYDRA successfully translates the analysis into a coherent design**. The temporary Scaffold loss, persistent Anchor loss, position-wise feature centering, and timestep-conditioned projectors each address a specific observation. The design is technically sound to interpret and evaluate.

  3. The empirical evaluation shows clear improvement across scales. At 600 epochs, HYDRA improves FID from 3.66 to 3.15 on JiT-B, 2.36 to 2.06 on JiT-L, and 1.86 to 1.74 on JiT-H. Tables 3-5 further isolate the two losses, stopping schedule, and timestep conditioning.

Critical weaknesses
  1. The semantic-granularity explanation remains correlational. Block depth changes feature statistics and optimization geometry in addition to semantic granularity. The results establish that the selected targets work, but not that granularity is the causal factor.

  2. Hierarchical complementarity lacks a matched control. The full model adds a second projector, target, and loss. A dual-branch control using repeated targets, or swapped targets with the same schedules, is needed to isolate semantic complementarity from extra supervision and parameters.

  3. The fixed stopping rule is validated mainly at one scale. Table 4 studies the 150-epoch cutoff on JiT-B/16, but the same schedule is transferred to JiT-L/H. It is unclear whether the stopping point follows representation dynamics or this specific training budget.

  4. Generalization and effectiveness under large-scale/resolution settings remain limited. The main evidence is ImageNet 256x256 with DINOv2, and the DINOv3 result does not revalidate the proposed hierarchy. More results, such as ImageNet 512x512 or simple text-to-image experiments on MS-COCO could further strength the effectiveness.

Questions for authors

See the weakness, I would condiser raising my score if these concenrs were well addressed.


Most of my concerns except the generalization were well adddressed with the authors' detailed rebuttal, I decide to maintain my postive rating as weak accept. Thanks for the authors' efforts and they should incorporate these results in the revision.

5 comments

(W1) Lack of Causal Evidence for Semantic Granularity

[Hypothesis]

We hypothesize that the semantic granularity of alignment targets affects the difficulty of image generation, as the subsequent diffusion blocks must reconstruct images from intermediate representations shaped by these targets. In particular, abstract Category-level Features may impose more challenging optimization constraints on image reconstruction than Component-level Features during the early stages of training. Accordingly, we suggest that Component-level Features may provide more effective targets for Early Alignment. This stage-dependent effectiveness of different semantic granularities is a key finding underlying HYDRA's design, motivating the distinction between what to align and when to align.

[Experimental Validation]

To further investigate this hypothesis and the generalizability of our proposed method, we conducted additional experiments using DINOv3, SigLIP2, and V-JEPA 2.1, in addition to the original DINOv2. In Tab. A1, we analyze the relative proportion of Component-level and Category-level Semantic information (Semantic Granularity Ratio R) across different Vision Encoders and examine its relationship with Early Alignment performance (FID).

The Ratio (R) is calculated based on the similarities of Component Matched Only and Category Matched Only, as defined in Fig. 3 of the main paper. Neither Matched refers to cases where neither the component nor the category matches, and its similarity is subtracted from both values to serve as a baseline. The Ratio is formally defined as follows:

R=Component Matched Only−Neither MatchedCategory Matched Only−Neither MatchedR = \frac{\text{Component Matched Only} - \text{Neither Matched}}{\text{Category Matched Only} - \text{Neither Matched}}

This allows us to compare the relative proportions of Component-level and Category-level Semantic information after removing the common similarity baseline.

[Results]

EncoderMetricBlock7Block9Block10Block11
DINOv2-B/14Category Matched Only.147.127.123.248
Component Matched Only.174.208.221.271
Neither Matched.113.101.100.160
Ratio (R)1.7944.115‾\underline{4.115}5.2611.261
FID ↓11.9211.61‾\underline{11.61}11.5214.36
DINOv3-B/16Category Matched Only.231.234.246.366
Component Matched Only.271.361.371.321
Neither Matched.208.211.215.229
Ratio (R)2.7396.5225.032‾\underline{5.032}0.672
FID ↓13.2012.28‾\underline{12.28}12.2414.84
SigLIP2-B/16Category Matched Only.127.211.233.234
Component Matched Only.162.207.202.199
Neither Matched.092.147.150.155
Ratio (R)2.0000.938‾\underline{0.938}0.6270.557
FID ↓12.2913.19‾\underline{13.19}14.3515.26
V-JEPA 2.1-B/16Category Matched Only.128.133.131.141
Component Matched Only.140.164.194.217
Neither Matched.098.105.105.111
Ratio (R)1.4002.1073.423‾\underline{3.423}3.533
FID ↓14.0212.9412.68‾\underline{12.68}12.25

Table A1: Semantic granularity and alignment performance across vision encoders. FID is evaluated on ImageNet 256×256 using JiT-B at 50 epochs.

  • DINOv2: Block 10 achieved the highest Semantic Granularity Ratio (R = 5.261) and the best FID (11.52). At the deeper Block 11, R decreased to 1.261, while the FID worsened to 14.36.

  • DINOv3: Block 9 achieved the highest R (6.522), while the adjacent Block 10 showed a relatively high R (5.032) and the best FID (12.24). In contrast, at Block 11, R decreased to 0.672, while the FID worsened to 14.84.

  • SigLIP2: Block 7 achieved the highest R (2.000) and the best FID (12.29). As block depth increased, R decreased to 0.557, while the FID worsened to 15.26.

  • V-JEPA 2.1: Unlike the other encoders, R consistently increased with block depth, reaching its highest value (3.533) at Block 11. Block 11 also achieved the best FID (12.25) among the evaluated blocks, further supporting the relationship between Semantic Granularity and Early Alignment performance.

These results demonstrate that, although the optimal block depth varies across encoders, features with a higher relative proportion of component-level semantic information tend to achieve better Early Alignment performance. This supports our hypothesis that Semantic Granularity is closely associated with Alignment performance and that component-level features can provide more effective alignment targets during the early stages of training. Furthermore, these findings suggest a systematic, potentially automatable strategy for selecting alignment targets based on semantic granularity, enabling HYDRA to be extended to diverse vision encoder architectures.

(W2) Lack of Matched Controls for Hierarchical Complementarity

[Comparing repeated-target and different-target dual-branch configurations]

The performance improvement achieved by our proposed method is not merely due to the addition of a second head, but rather stems from the semantic complementarity of the distinct target features.

To verify this, we compared the FID scores of three configurations while keeping all other training settings unchanged: Anchor-Anchor, where both projectors use the Anchor target; Scaffold-Scaffold, where both projectors use the Scaffold target; and our proposed HYDRA (Scaffold-Anchor).

Projector 1 TargetProjector 2 Target50100150200
AnchorAnchor14.525.544.493.84
ScaffoldScaffold11.984.694.073.69
ScaffoldAnchor (HYDRA)12.774.834.053.65

Table A2. FID comparison of same-target dual-branch configurations and HYDRA across training epochs.

As shown in Tab. A2, Scaffold-Scaffold achieved the best FID of 11.98 at 50 epochs. However, as training progressed, HYDRA (Scaffold-Anchor) showed greater relative improvements, achieving the lowest FID of 3.65 at 200 epochs. These results suggest that Scaffold-based Alignment can be effective during the early stages of training, while HYDRA provides complementary alignment supervision from target features with different Semantic Granularities, potentially facilitating more effective Representation Learning as training progresses.

 

(W3) Limited Cross-Scale Validation of the Stopping Rule

The early stopping of Scaffold Alignment at 150 epochs was designed by considering the initial variation in Scaffold Alignment Loss and the diminishing benefits of prolonged supervision. Specifically, we analyzed the Scaffold Alignment Loss trajectory of JiT-B trained without early stopping and observed that the loss variations during early training began to stabilize around 150 epochs, which motivated our choice of the stopping point.

1. Scaffold Alignment Loss Variation and Generation Performance in JiT-B

Epoch50100150200250300350400450500550600
Scaffold Loss ↓0.50370.44110.43180.42670.42360.42130.41960.41810.41670.41580.41500.4141

Table A3. Scaffold alignment loss of JiT-B across training epochs without early stopping. Each value represents the average loss over the preceding 50 epochs.

Stop Epoch100150200500
FID ↓3.193.153.223.24

Table A4. Effect of the Scaffold alignment stopping epoch on the FID of JiT-B trained for 600 epochs.

As shown in Tab. A3, the Scaffold Alignment Loss of JiT-B decreased rapidly during the first 100 epochs, after which the rate of decrease gradually slowed. This indicates that Scaffold Alignment progresses rapidly during early training, while additional improvements gradually diminish.

To examine whether this trend is associated with generation performance, we compared the final FID scores by varying the stopping epoch to 100, 150, 200, and 500 (Tab. A4). The results show that stopping at 150 epochs achieved the best FID of 3.15.

2. Scaffold Alignment Loss Variation in JiT-L/H

Scaffold Loss ↓50100150
JiT-L/160.46300.39890.3874
JiT-H/160.45230.39220.3805

Table A5. Scaffold alignment loss of JiT-L/H during early training. Each value represents the average loss over the preceding 50 epochs.

Similar early decreasing trends in Scaffold Alignment Loss were also observed in JiT-L/H (Tab. A5). Based on these consistent trends across model scales, we applied the same 150-epoch stopping point to JiT-L/H, although the optimal stopping epoch may vary with model scale.

(W4) Limited Validation in Large-Scale and High-Resolution Settings

Through additional experiments, we confirmed that HYDRA's Semantic Granularity-based design principle extends beyond DINOv2 to other Vision Encoders, and that its training convergence acceleration is maintained at a higher resolution of 512×512 on ImageNet. To validate these findings, we analyzed the feature hierarchies of different Vision Encoders and conducted high-resolution image generation experiments.

1. Generalization across Vision Encoders

A consistent relationship between Semantic Granularity and Early Alignment performance was also observed in DINOv3, SigLIP2, and V-JEPA 2.1. As shown in Tab. A1 of (W1), these encoders, trained under different training paradigms, exhibit distinct feature hierarchies. Nevertheless, features with higher Component/Category Ratios consistently tend to achieve better Early Alignment performance. These findings suggest that HYDRA's design principle is not limited to the specific feature hierarchy or training paradigm of DINOv2.

2. Generalization to Higher Resolution

To evaluate HYDRA at higher resolutions, we conducted additional experiments using JiT-B/32 on ImageNet 512×512. As shown in Tab. A6, HYDRA achieved an FID of 4.03 in only 200 epochs (the 600-epoch run is ongoing at 300 epochs, with completion expected within a day; results will be added once available), nearly matching the baseline FID of 4.02 obtained after 600 epochs. This demonstrates that HYDRA achieves comparable generation quality with only one-third of the training epochs, extending its convergence acceleration effect to 512×512 resolution.

ModelEpochFID ↓
JiT-B/326004.02
JiT-B/32 + HYDRA2004.03

Table A6. FID comparison between JiT-B/32 and HYDRA on ImageNet 512×512.

Comments on the paper

3 comments

We sincerely thank all reviewers for their valuable comments and constructive feedback. We apologize for the delay in our responses.

We are currently revising the manuscript to further clarify the novelty and key contributions of our work, while also incorporating the additional experiments conducted during the rebuttal period into the Appendix.

We will upload the revised manuscript as soon as possible.

Thank you again for your time, patience, and thoughtful feedback.

Dear Reviewers,

Please carefully review the authors’ rebuttal and engage in further discussion with the authors to clarify any remaining concerns and questions. Your active participation during the discussion period is greatly appreciated.

Best, AC

We have uploaded a revised version of our manuscript incorporating the feedback received during the rebuttal discussion.

The revision includes clarifications to the motivation and methodology, as well as additional experimental results and analyses to further strengthen our findings. All revisions are highlighted in red\textcolor{red}{red} for ease of review.

We sincerely appreciate the reviewers' valuable feedback and constructive suggestions. We welcome any further comments or questions.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.