- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies three controls in consistency mid-training: teacher discretization K, trajectory refresh stride R, and student supervision count S. It propose to combine uniform cell subsampling with caching a teacher pool and processing complementary halves across two optimizer updates. The central empirical findings are a 41.8% reduction in reported interval peak memory and approximately 30.6% less training-loop time, with small, protocol-dependent FID changes.
Critical strengths
- It reports the empirical results carefully. In particular, it distinguishes conditional gradient unbiasedness from optimization equivalence and approximate equal-time comparisons from exactly matched budgets.
- The experiments show clearly that reducing student supervision saves activation memory while cacheing the teacher improves the throughput.
Critical weaknesses
- The contribution largely combines standard sampling choices. First, Prop 1 is the classical mean and covariance of simple random sampling without replacement. The technique is structurally closely related to establish minibatch random shuffling. The difference is that the pool here is regenerated online. Second, Prop 2 follows directly from gradient Lipschitz bounding a local gradient difference. However, it does not establish any advantage over fresh sampling or characterize the actual Radam trajectory.
- Eq. 1 is more like a criterion for configuration-selection rather than a solid allocation algorithm.
- The baseline of reducing the teacher trajectory’s batch size is missing. The primary comparison holds on a minibatch size of 8. If we reduce both B and R by half, it still match average teacher image evaluations and student cells per update. How would these perform?
- CMT’s original motivation is to provide an effective initialization for subsequent flow-map training and evaluates improvements through that downstream stage. Here, all tested transfers from the primary 30k checkpoints fail the step-300 gate, which cannot establish that reuse causes the same problem. This does not show the saved mid-training resources produce a useful final model. The intended benefit for downstream application is unclear.
- All experiments are on CIFAR-10. The longest primary runs comprise of 10k + 30k update, which is much shorter than the cited 300k update reference. The empirical contribution is marginal.
Questions for authors
See weakness
Reply
Proposition 1 uses the classical finite-population formulas for the mean and covariance. Section 2 (p. 2) connects the order of complementary updates to random reshuffling. The revised title is “BudgetCMT: Resource Controls for Consistency Mid-Training”.
Proposition 2 bounds the change in the gradient of the held-out half of a pool after one update (Section 3.2, p. 4). Starting from one saved RAdam state, we measured a 7.43% change in that gradient in relative L2 norm. The difference between the complementary and independent two-step parameter displacements was 21.38% of the independent displacement norm (Figure A2, p. 22). Complementary reuse covered all 96 cells per cycle. Independent reuse nevertheless had lower final FID at both NFEs in all three seeds. Against sparse fresh at matched time, BudgetCMT used 24.5–27.6% fewer teacher image evaluations. Its mean 1/2-NFE FID50k differences were −0.043/−0.017, with both descriptive 95% paired intervals spanning zero (Appendix G, pp. 43–44).
We tested a B=4, R=1 baseline against B=8, R=2, keeping K=12 and S=48. At the last shared complete cycle, both used the same ordered teacher inputs and the same counts of teacher image evaluations and student cells. The smaller pool made twice as many teacher calls. At matched time, BudgetCMT minus small-pool FID50k averaged −0.300/−0.074 at 1/2 NFE. Both descriptive 95% paired intervals span zero (Appendix G, pp. 43–44). We describe Eq. (1) as a selection criterion and give the measurement procedure in Section 3.3 (pp. 3, 5).
In the 300k CIFAR-10 pair, sparse fresh and BudgetCMT took 62.90 and 43.25 RTX 3090 hours (31.2% less for BudgetCMT). The same 100k post-training recipe took 23.04 and 23.05 A100 hours. Both stage times include evaluations. Final 1/2-NFE FID50k was 3.474/2.285 for sparse fresh and 3.439/2.312 for BudgetCMT. We also report the official-checkpoint control and earlier 30k transfer failures (Section 4.7, p. 9, and Appendix B.8, pp. 31–32). In one matched-time FFHQ64 pair with a 61.81M-parameter student, BudgetCMT used 26.15% fewer teacher image evaluations and completed 47.69% more updates. Its 1/2-NFE FID50k was higher, 18.544/14.013 versus 16.801/12.461 for sparse fresh (Appendix H, Tables A24–A25, p. 45).