- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies self-supervised representation learning for pixel-space diffusion. The authors observe that self-distillation methods that align noisy features with cleaner targets do not consistently improve pixel-space generation, and further show that high feature similarity can coexist with poor spatial correspondence. To address this, they propose CORE, which combines Correlated Dual Observations (CDO) with Contextual Representation Grounding (CRG). CDO trains a shared generator on paired noisy and cleaner observations, while CRG uses masked contextual prediction through Contextual Feature Prediction (CFP) and Pixel-Guided Correction (PGC) to encourage spatially grounded representations tied to pixel prediction errors. CORE is evaluated on ImageNet using JiT backbones at 256×256 and 512×512 resolutions and is compared with JiT and several self-supervision baselines.
Critical strengths
- The paper studies an interesting and relevant problem: representation-alignment and self-distillation methods that are effective in latent-space diffusion do not consistently transfer to pixel-space models such as JiT. The analysis showing that high feature similarity can coexist with poor spatial correspondence provides a useful motivation for revisiting representation supervision in pixel-space diffusion.
- Unlike the adapted latent-space self-distillation methods, which do not consistently improve over JiT in pixel space, CORE consistently achieves better performance than vanilla JiT across the evaluated settings.
Critical weaknesses
- The current comparisons do not establish training efficiency under a matched compute budget. The methods are mainly compared after the same number of training epochs, although their per-step training costs differ substantially. In particular, CORE requires two full generator evaluations (CDO) and an additional shared-prefix evaluation (CRG), while the compared self-supervision baselines also have different auxiliary training costs. Therefore, comparisons at the same number of epochs do not correspond to matched training cost. Reporting training FLOPs, GPU-hours, or compute-matched performance across methods would provide a clearer assessment of training efficiency.
- The ablation suggests that the contribution of CRG is relatively limited compared with CDO. On JiT-B/16, CDO alone improves FID from 4.45 to 4.17, while adding the full CRG further improves it only to 4.09; meanwhile, IS decreases from 260.7 to 257.2. Since CRG is the component directly motivated by contextual grounding, stronger evidence would be helpful to establish that this relatively small incremental gain is robust and meaningful.
- There are a few minor reporting and presentation inconsistencies. Some JiT settings in Tab. 1 and Tab. 2 report FID but omit the corresponding IS values. In addition, the “No Image” points in the spatial-retrieval panel of Fig. 1(c) appear vertically misaligned despite belonging to the same intervention row.
Questions for authors
- Can the authors provide a compute-matched comparison, particularly between CORE and vanilla JiT, using training FLOPs or GPU-hours rather than primarily comparing methods after the same number of training epochs? Given the additional computation introduced by CDO and CRG, this would help clarify whether CORE still provides an advantage under a comparable training budget.
- Can the authors provide additional evidence that CRG contributes meaningfully beyond CDO-only? In Tab. 3, adding the full CRG only improves FID from 4.17 to 4.09, while IS decreases from 260.7 to 257.2.
5 comments
Response to Reviewer GQni(Comments 3-4)
3. Fig. 1(c)
Thank you for pointing out the misalignment of the “No Image” points. This results from inconsistent vertical placement in the figure: the two points correspond to the same input intervention, not different experimental settings. We will correct their vertical positions and label layout to avoid confusion.
Meanwhile, to show more directly how training supervision responds to spatial information, we added an experiment analyzing the sensitivity of supervision to spatial information. Holding the model weights, queried region, and each method's own supervision targets fixed, we exchange nearby and distant image patches within the context accessible to each method. We then measure the resulting changes in gradient directions at intermediate features and shared backbone parameters.
| Method | Feature-gradient change | Backbone-parameter gradient change |
|---|---|---|
| JiT | 0.2637 | 0.5396 |
| Self-Flow | 0.2789 | 0.4541 |
| SRA | 0.308 | 0.4797 |
| SRA+DTS+AS | 0.0638 | 0.2556 |
| CORE | 0.565 | 0.8508 |
Under this diagnostic setting, CORE exhibits stronger responses at both levels. This indicates that rearranging spatial context changes not only CORE's feature-level supervision but also, more substantially, the learning signal propagated to the generative backbone.
This analysis more directly reflects our motivation: pixel-space generation requires not merely high feature similarity, but training supervision that can exploit spatial context and establish a connection to pixel prediction. We will use this analysis as the new Fig. 1(c) in the revision and update both the figure layout and the accompanying text.
4. IS reporting and result selection
The missing IS values in Tables 1 and 2 correspond to the 200-epoch settings of JiT-L/16 and JiT-B/32, respectively. The original paper we cite reports only FID for these two configurations, and the corresponding 200-epoch checkpoints have not been officially released. We therefore have not supplemented their IS values. We will explicitly state the reason for these missing entries in the revision.
Regarding the decrease in IS from 260.7 to 257.2 after adding CRG in the 200-epoch ablation, we would like to clarify that we select the evaluation configuration with the best FID and report its corresponding IS, rather than independently selecting the best FID and the best IS. To examine whether this observation persists under longer training, we added the following 600-epoch ablation results. The table below reports results under a common evaluation protocol, including our re-evaluation of JiT, with IS reported at each method's best-FID configuration:
| Method | FID ↓ | Corresponding IS ↑ |
|---|---|---|
| JiT$^{*}$ | 3.6266 | 269.372 |
| CDO-only | 3.5845 | 265.4418 |
| CORE | 3.3784 | 269.4496 |
JiT is re-evaluated under the same protocol as CDO-only and CORE.
Under the same result-selection rule, adding CRG improves FID from 3.5845 to 3.3784, while the corresponding IS also increases from 265.4418 to 269.4496. This shows that the IS decrease observed at 200 epochs is not an inevitable trade-off introduced by CRG. With longer training, the complete CORE achieves both better FID and better IS than CDO-only. Figure 4 in the original manuscript presents the FID–IS trade-off curves across different CFG values at both 200 and 600 epochs, showing the overall relationship between these two metrics.
Thank you for the additional experiments. Although the compute-matched L/16 result is helpful, I believe compute-matched comparisons with JiT are also necessary for H/16 and the settings in Tab. 2, where the gains at the same number of epochs are smaller. The 600-epoch ablation shows a clearer benefit from CRG, but CDO-only improves FID over JiT by only 0.08 while requiring more computation. Could you compare JiT, CDO-only, and CORE at the same training cost?
Response to Reviewer GQni(Comments 1-2)
We thank the reviewer for the careful review and for recognizing the value of our research question and diagnostics, and CORE's consistent gains over JiT. We have added experiments addressing training cost, component contributions, and result presentation.
1. Compute-matched
To compare the training cost of CORE with other self-supervised methods, we measured steady-state training step times. CORE's context prediction and pixel correction tasks do introduce additional computation per step. However, all losses are jointly optimized rather than updating parameters separately for each branch. Therefore, the number of optimizer updates remains unchanged at the same number of training epochs and effective batch size. Moreover, the auxiliary components are used only during training, so the network architecture, sampling procedure, and computational cost at inference remain unchanged. The comparison is shown below:
Compared with SRA+DTS+AS, CORE adds 28.54 ms per step, approximately 20.9%. This overhead comes from the additional spatial supervision computation.
To further examine the relationship between training cost and generation quality, we use the measured per-step time ratio above as a reference and select 338 epochs as an additional evaluation checkpoint based on 600 × 93.15 / 165.16 ≈ 338. We report the CORE-L/16 results at this checkpoint below:
At 338 epochs, CORE-L/16 already outperforms JiT-L/16 trained for 600 epochs, demonstrating better generation quality with fewer training epochs and parameter updates. After 600 epochs, its FID further improves to 2.02, surpassing the 2.11 reported by PixelREPA-L/16 with external representation supervision.
At a larger scale, CORE-H/16 achieves 1.77 FID with approximately 953M generator parameters, outperforming the 1.82 FID reported by JiT-G/16 with approximately 2B parameters. These results further support CORE's effectiveness as model size increases and its value in improving generation quality without adding inference-time components.
2. Ablation
Thank you for highlighting the relative contributions of CDO and CRG in the original ablation. The original 200-epoch experiments reflect a shorter training stage and do not fully capture the differences between components under longer training. We therefore extended the JiT-B/16 ablation experiments to 600 epochs:
At 600 epochs, CORE achieves 3.38 FID, compared with 3.58 for CDO-only, showing a larger additional gain than at 200 epochs. Thus, the observation that “most gains come from CDO” at the shorter training stage does not persist under longer training: the gains of the complete method cannot be attributed solely to an additional forward pass, and CRG provides more pronounced additional benefits after longer training.
CDO primarily provides CRG with spatially corresponding observations of the same image at different noise levels. The cleaner observation supplies the targets for contextual feature prediction and candidate directions for pixel correction. Both paths also receive generation supervision, keeping the learning of these paired observations connected to the denoising task. CRG further turns this information into context prediction and pixel-error-guided supervision. In other words, CDO provides the observations needed for supervision, while CRG determines how this information is used to shape generative representations.
Moreover, CORE-L/16 reaches 2.02 FID, outperforming the PixelREPA-L/16 with external representation supervision (2.11). CORE-H/16 achieves 1.77 FID with ≈953M parameters, surpassing JiT-G/16 (1.82, ≈2B). These results support CORE's scalability without additional inference components.