- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies representation alignment for pixel-space Just image Transformers (JiT). It associates late-intermediate DINOv2 features with component-level similarity and final-block features with category-level similarity, then proposes HYDRA: early Scaffold alignment to the sum of centered Blocks 9 and 10, continuous Anchor alignment to centered Block 11, and separate timestep-conditioned projection heads. On class-conditional ImageNet generation at 256 x 256, the reported 600-epoch FID improves from 3.66 to 3.15 for JiT-B/16, from 2.36 to 2.06 for JiT-L/16, and from 1.86 to 1.74 for JiT-H/16. The B model reaches FID 3.65 after 200 epochs. The paper includes target-selection and scheduling ablations, feature-similarity and gradient analyses, timing measurements, and a shorter DINOv3 experiment.
Critical strengths
-
A useful improvement to a fixed generative backbone across three scales. Table 2 reports consistent FID reductions over JiT-B/L/H at 600 epochs, with especially substantial improvement for the B model, and near-baseline quality after 200 epochs. This establishes a practical benefit without changing the inference backbone. Appendix C.6 also reports training costs rather than relying only on epoch counts: HYDRA takes 0.68 seconds per iteration versus 0.47 for JiT and 0.74 for PixelREPA, with 7.48M projector parameters versus PixelREPA's 24.03M. Comparable final quality with a cheaper training head is useful even if the small FID differences are not robust. These figures make the efficiency claim assessable, although the threefold reduction in epochs should not be read as a threefold wall-clock speedup.
-
Several important design choices receive direct empirical checks. Tables 3-5 separate the two objectives, their stopping schedules, and timestep conditioning. Table 4 is particularly relevant: stopping Scaffold at epoch 150 while retaining Anchor yields FID 3.15, compared with 3.22 when neither stops and 3.29 when both stop. Appendix C additionally checks loss weights, centering, target combinations, and an alternative encoder. This provides more support for the final recipe than a single aggregate improvement would.
Critical weaknesses
-
The evidence does not sufficiently isolate semantic granularity as the explanation for the proposed division of roles. The abstract appropriately describes an association, but Section 3.1.2 goes further in attributing early alignment effectiveness to semantic granularity. Section 3.1 and Figure 3 compare mean cosine similarity for category-only and component-only matches, but neither the main text nor the appendices specify the PartImageNet split, image/pair counts, patch-to-part assignment, balancing, or whether category-only pairs can come from the same image while component-only pairs come from different images. Without a pair matched on neither attribute and appropriate controls, raw cosine changes can also reflect feature geometry or image/position effects. Table 1 establishes that particular intermediate targets work better early; Tables 3-4 establish that the particular two-target recipe and schedule work better. They do not isolate whether the semantic distinction, rather than a second head or a different target geometry, explains that benefit. The local gradient measurements in Appendix C.4 do not resolve this attribution. Please specify and control the similarity protocol and compare against a matched two-head same-target control, or consistently narrow the explanation to an empirical association and target-selection heuristic. This matters because the semantic explanation is a central contribution, beyond the recipe itself.
-
The reliability of the small component effects and the acceleration relative to the closest alignment approach remain uncertain. In Table 2, the FID advantage over PixelREPA is 0.02/0.05/0.07 for B/L/H; the added Anchor benefit in Table 3 is 0.06, and Scaffold stopping in Table 4 improves FID by 0.07. No repeated-run or sampling-uncertainty estimates are reported. The lower reported training-head cost is a valid benefit even if these FID margins do not persist. However, the 200-epoch acceleration comparison omits PixelREPA at the corresponding budget, so it does not establish the early quality/cost trade-off against that close alternative. Appendix B also explicitly replaces iREPA's convolutional projector with an MLP. That is a reasonable target-isolation experiment, but the early curves should be labeled as iREPA features with an MLP; a faithful original projector is needed only for claims about the complete iREPA method. A matched early learning curve with PixelREPA and uncertainty for the decisive B/16 target/schedule comparisons would make the explanation and acceleration claims substantially more convincing. I am not requesting repeated runs for every scale or requiring state-of-the-art FID; the concern is whether the measurements distinguish the proposed contribution from existing alignment strategies.
Questions for authors
- For Figure 3, what are the PartImageNet split, sample and pair counts, label-to-patch rule, and within-image versus cross-image pairing rules? Do the semantic trends persist with balanced image/position controls and a neither-matched reference? Please also clarify whether a matched two-head same-target control has been evaluated; this would help distinguish complementary semantics from the architectural change itself.
- Are the PixelREPA entries in Table 2 rerun with the same training implementation, CFG sweep, and EMA/checkpoint selection, or taken from prior reports? What are the corresponding early-budget results and variability for HYDRA and PixelREPA? Do the decisive B/16 target/schedule improvements remain consistent across training/evaluation repeats? Appendix C.6 supplies useful iteration costs, so a comparison at equal elapsed training time would directly address the efficiency contribution.
- How are DINOv2 ViT-B/14 features spatially matched to the JiT 16 x 16 token grid for 256 x 256 generation? Appendix B identifies the encoder and alignment block but does not state the encoder image resizing/cropping or feature interpolation procedure. Please provide that mapping and confirm whether it is shared by the comparison methods and by the position-wise centering statistics.
6 comments
2. Semantic Similarity Evaluation Protocol
To address the reviewer's concerns regarding the reproducibility of Fig. 3, we clarify the PartImageNet similarity evaluation protocol, using the ground-truth part segmentation annotations provided by the dataset, including the dataset split, patch-to-part assignment, pair sampling, and balancing procedures, as summarized below.
| Item | Description |
|---|---|
| Dataset | PartImageNet test split (2,398 images, 11 supercategories) |
| Patch-to-Part Assignment | A patch is assigned to a part if the part occupies at least 50% of its area. |
| Pairing Rule | All pairs are cross-image and sampled within the same supercategory. |
| Pair Conditions | S11: Both Matched; S10: Category Only; S01: Component Only; S00: Neither Matched |
| Sampling & Balancing | 5,450 pairs per condition (21,800 total), balanced across conditions within each supercategory |
| Aggregation | Macro-average across 11 supercategories |
| Feature Geometry Control | Neither-Matched (S00) reference and Position-wise Feature Centering |
| Image/Position Control | Cross-image pairing and position-matched sampling (40,905 pairs per condition) |
Table C3. Summary of the PartImageNet Semantic Similarity Evaluation Protocol.
Detailed experimental settings will be provided in the revised Appendix.
[Balanced Image/Position Controls] To examine whether the observed trends are influenced by spatial position, we enumerated all cross-image part pairs within each supercategory and computed the distance between their part centroids on the 16×16 patch grid. Before matching, same-part pairs lay closer together than different-part pairs (mean centroid distance: S11 4.44, S01 4.49 vs. S10 5.69, S00 5.66 patches). Within each supercategory, we then binned pair distances into 1-patch-wide bins and, in each bin, subsampled all four conditions (S11, S10, S01, S00) down to the smallest condition's count. This yielded 40,905 pairs per condition in total, with nearly identical mean centroid distances (4.68–4.70 patches). We repeated the sampling with five random seeds (seed std ≤ .001).
As shown in Tab. C4, cosine similarities changed by at most .004 relative to Fig. 3 across all conditions and blocks after distance matching, and the semantic trends remained unchanged (S01 > S10 from Block 7† onward). The small changes may partly reflect the Position-wise Feature Centering applied in our evaluation: without centering (raw Block 11), the changes are slightly larger (≤ .006). These results indicate that our semantic similarity analysis is robust to spatial position differences.
| Condition | Block 3† | Block 5† | Block 7† | Block 9† | Block 10† | Block 11† |
|---|---|---|---|---|---|---|
| S11 | .184→.183 (−.001) | .201→.201 (+.000) | .227→.227 (+.000) | .257→.257 (+.000) | .273→.273 (+.000) | .397→.396 (−.001) |
| S10 | .137→.133 (−.004) | .141→.139 (−.002) | .147→.145 (−.002) | .127→.126 (−.001) | .123→.123 (+.000) | .248→.248 (+.000) |
| S01 | .134→.135 (+.001) | .146→.148 (+.002) | .174→.176 (+.002) | .208→.206 (−.002) | .221→.218 (−.003) | .271→.267 (−.004) |
| S00 | .099→.098 (−.001) | .103→.102 (−.001) | .113→.113 (+.000) | .101→.102 (+.001) | .100→.100 (+.000) | .160→.160 (+.000) |
Table C4. Mean cosine similarity before → after spatial distance matching (DINOv2-B/14; † = Position-wise Feature Centering). Values in parentheses are the changes
(W2 Q2) Reliability and Training Efficiency
1. Comparison with PixelREPA at 200 Epochs
We clarify that the PixelREPA results in Tab. 2 were taken from the original paper. To further assess early-stage convergence and training efficiency, we independently trained PixelREPA and HYDRA for 200 epochs in our own environment and evaluated both methods using five sampling seeds after conducting a CFG scale sweep. Based on the per-iteration training costs reported in Appendix C.6, we additionally include HYDRA at 217 epochs to enable a comparison under approximately matched wall-clock training time.
| Method | Epoch | sec/iter ↓ | Seed 0 | Seed 1 | Seed 2 | Seed 3 | Seed 4 | Mean ± Std. ↓ |
|---|---|---|---|---|---|---|---|---|
| PixelREPA | 200 | 0.74 | 3.94 | 3.90 | 3.88 | 3.89 | 3.91 | 3.90 ± 0.02 |
| HYDRA | 200 | 0.68 | 3.65 | 3.66 | 3.66 | 3.67 | 3.68 | 3.66 ± 0.01 |
| HYDRA | 217 | 0.68 | 3.62 | 3.59 | 3.63 | 3.63 | 3.61 | 3.62 ± 0.02 |
Table C5. Comparison of PixelREPA and HYDRA in terms of FID and training efficiency at early training stages.
As shown in Tab. C5, at 200 epochs, HYDRA achieves a lower mean FID than PixelREPA (3.66 vs. 3.90) while reducing per-iteration training time by 8.1% (0.68 vs. 0.74 s/iter). Furthermore, under an approximately matched wall-clock training budget, HYDRA at 217 epochs further improves the mean FID to 3.62, demonstrating both faster convergence and improved training efficiency.
2. Reliability across Sampling Seeds
All experimental results reported in the original submission were evaluated using sampling seed 0. To further assess the reliability of these results, we additionally evaluated the 600-epoch HYDRA models across five sampling seeds for all three model scales.
| Model | Seed 0 | Seed 1 | Seed 2 | Seed 3 | Seed 4 | Mean ± Std. |
|---|---|---|---|---|---|---|
| JiT-B/16 + HYDRA | 3.15 | 3.14 | 3.21 | 3.20 | 3.20 | 3.18 ± 0.03 |
| JiT-L/16 + HYDRA | 2.06 | 2.05 | 2.05 | 2.06 | 2.04 | 2.05 ± 0.01 |
| JiT-H/16 + HYDRA | 1.74 | 1.80 | 1.74 | 1.75 | 1.77 | 1.76 ± 0.03 |
Table C6. FID stability of HYDRA across five sampling seeds for different model scales.
As shown in Tab. C6, the results exhibit low variability across sampling seeds for all three model scales, supporting the stability of HYDRA's generation performance under sampling randomness.
3. iREPA Notation
We have revised the manuscript to explicitly label iREPA as "iREPA (w/ MLP)" in Fig. 1 and clarify the use of an MLP projector in the caption of Tab. 1.
(Q3) Spatial Feature Matching between DINOv2 and JiT
[Implementation Details]
We clarify the spatial feature mapping and preprocessing procedures between DINOv2 and JiT.
1. Token Grid Alignment
Following the PixelREPA implementation, we use timm-based resampling to change the patch size of DINOv2 ViT-B/14 from 14 to 16 by adapting the pretrained Patch Embedding and Positional Embedding. Therefore, for a 256×256 input, both DINOv2 and JiT-B/16 produce the same 16×16 token grid, enabling direct one-to-one alignment at each spatial position. Accordingly, no additional feature interpolation is required.
2. Consistent Teacher Feature Extraction
Across all experiments, we apply the same input preprocessing and feature extraction procedures to the teacher encoder. This spatial mapping is consistently maintained across all comparison methods, including PixelREPA. Furthermore, the same teacher input preprocessing and feature extraction procedures are used to compute the statistics for Position-wise Centering.
(W1 & Q1) Semantic Granularity & Alignment Target Roles, and Similarity Evaluation Protocol
1. Evidence for Semantic Granularity-Based Alignment Target Roles
1) Relationship between Semantic Granularity and Early Alignment across Vision Encoders
To further examine the generalizability of the relationship between Semantic Granularity and Early Alignment, we extend the semantic similarity analysis in Fig. 3 to multiple vision encoders, including DINOv3, SigLIP2, and V-JEPA 2.1. Following the reviewer's suggestion, we additionally include Neither Matched pairs as a reference. Based on the Category Matched Only, Component Matched Only, and Neither Matched similarities, we define the Semantic Granularity Ratio (R) as follows:
R=Category Matched Only−Neither MatchedComponent Matched Only−Neither Matched
As shown in Tab. C1, blocks with higher R generally achieve better Early Alignment performance across DINOv2, DINOv3, SigLIP2, and V-JEPA 2.1, although the blocks with the highest R and best FID do not always coincide exactly. These results provide cross-encoder empirical evidence supporting a consistent relationship between Semantic Granularity and Early Alignment effectiveness, despite differences in encoder architectures and learned representations.
Table C1. Semantic granularity and alignment performance across vision encoders. FID is evaluated on ImageNet 256×256 using JiT-B/16 at 50 epochs.
2) Effectiveness of Role-Specific Alignment Targets
[Comparison with Matched Two-Head Same-Target Controls]
Table C2. FID comparison of two-head same-target configurations and HYDRA across training epochs.
To investigate whether the performance improvement stems merely from adding a second alignment head or from the semantic complementarity of different target features, we conducted controlled experiments. Specifically, we compared two-projector configurations using the same target features for Scaffold and Anchor, respectively, against HYDRA, which aligns features from different encoder layers.
As shown in Tab. C2, Scaffold (Two-head) achieved the best FID of 11.98 at 50 epochs. However, HYDRA showed progressively better relative performance as training proceeded, achieving the lowest FID of 3.65 at 200 epochs. These results suggest that while Scaffold-based alignment can be particularly effective during early training, HYDRA provides additional benefits as training progresses by jointly leveraging target features from different encoder layers.