- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DINO Forcing (DF), a method for incorporating pretrained visual representations into flow-matching image generators without assigning those representations a separate noisy generative trajectory. Training generally uses two passes through the same network: 1) a look-ahead pass which predicts a representation from the noisy image; 2) a conditioned pass which receives the look-ahead pass prediction and optimizes both the image flow-matching objective and an alignment objective against frozen pretrained features. At inference, the predicted representation is carried forward from one sampling step to the next, eliminating a separate representation ODE and representation-specific noise schedule. The experiments cover class-conditional ImageNet generation in latent space and pixel space, results show that in setting of latent space without guidance, DF achieves SOTA FID score at 2x fewer epochs, while in setting with guidance, DF's FID score still lags behind 'SFD+LDiT/1' (2.35 vs. 1.06)
Critical strengths
- The proposed replacement of feature co-denoising with supervised feature prediction is reasonable and well motivated.
- The experiment verification is very clear, table 7 clearly shows that with the same augmented architecture, unsupervised self-conditioning and image-supervised self-conditioning improve generation, but pretrained-feature supervision produces a much larger improvement.
Critical weaknesses
- Although DF achieves best latent-space FID score in 'unguided' setting, the guided setting latent-space FID score is significantly higher than existing method 'SFD+LDiT/1', 'REED', and 'iREPA', this make the claim in abstract "outperforms the state of the art in latent space at 2× fewer epochs than prior methods" somwhat not fully supported
- Epoch efficiency is not the same as training computation efficiency: the paper repeatedly emphasizes faster convergence and a twofold reduction in training budget. However, DF generally performs an additional gradient-free forward pass for each training step, it would be good to further report training FLOPs.
Questions for authors
- In sec 5.1, is the training stability sensitive to training recipes such as "10% of the time optimization on just the 1st forward pass", "5% of the time to make the model more stable", etc.
- Why does guidance provide relatively little improvement in latent space, and how do feature predictions change as guidance strength increases?
2 comments
Thanks the authors for detailed experiment on computational cost comparison and explanations on training stability and CFG limitation. I think my concerns have been addressed and I'll keep the positive rating
We thank the reviewer for finding our work motivating and also for their questions and comments. We have answered their concerns below as well as have updated the paper accordingly with the changes marked in green.
Our claim for 2x fewer epochs over prior methods is indeed applicable only for the unguided setting. We have updated the manuscript to specifically mention that.
We report training FLOPs in Table 5. The FLOP counts include DINO feature extraction and the additional gradient-free forward pass. Even with this cost, DINO Forcing outperforms REG (2.61 vs. 2.70 FID) with about half its training FLOPs (101,343 vs. 197,493), and outperforms REGLUE (2.47 vs. 2.50 FID) with about 36% fewer FLOPs (126,678 vs. 198,812). We have added the FLOP comparison in the main paper in Table 8 (page 10) and also additional discussion about the training cost in Section 5.6 (also on page 10).
Table 5: Training FLOPs of SiT-XL models (unguided FID). Computational costs include DINO feature extraction and follow Table 13 of REGLUE.
We did not observe any training instabilities (e.g., loss spikes or divergence) with any of these settings; both choices were made to improve performance and not for stability. Training on Pass 1 alone 10% of the time teaches the model to make good DINO predictions from the all-zero condition used at the first sampling step. We ablate this ratio in Appendix C ("How often should we train on Pass 1 alone?"). Conditioning on the ground-truth DINO features 5% of the time teaches the model to behave consistently once its predicted features are already close to the true ones, as happens in late sampling steps.
With guidance, the predicted DINO features become much more class-specific: a linear probe trained on real DINO features recovers the correct class with 89.1% top-1 accuracy without guidance and 96.0% with CFG (Table 4). Since these features already act as a much stronger condition than the class label, CFG, which guides through the class label, has little room to improve the image further. This is in line with prior work built on DINO features, such as Latent Forcing and RAE, which also rely on AutoGuidance. We leave further investigation, e.g., internal guidance or balancing the DINO and class conditions, to future work. We have added a discussion about the CFG failure in the paper in Appendix C (page 15) for future readers.
Table 4: Class recovery from predicted DINO features with a linear probe.