acceptodds
Under review as a conference paper at ICLR 2027

BudgetCMT: Joint Allocation of Teacher Compute and Student Supervision Budgets in Consistency Mid-Training

Abstract

Consistency mid-training (CMT) trains a few-step student on intermediate states of online teacher ODE trajectories. Generating these trajectories incurs teacher computation, while retaining student activations incurs memory cost. We introduce BudgetCMT, a joint allocation framework with three controls: teacher discretization , trajectory refresh stride , and student supervision budget . Uniform sampling without replacement preserves the dense gradient in conditional expectation at a fixed parameter state and teacher pool, with covariance given by the finite-population correction. Complementary two-step reuse covers the pool once per cycle while halving teacher evaluations. On CIFAR-10, reducing supervision from 128 to 64 cells lowers reported interval peak GPU allocation by 41.8% in a three-seed 10k-update study; quality intervals allow both improvement and degradation. In a separate three-seed 30k-update continuation, reuse reduces loop time by a median of 30.6%. Its 2-NFE FID is 1.40% higher at equal updates and 1.42% lower at a pre-specified approximate equal-time endpoint. A single paired run from a second warm start shows the same direction, with a smaller equal-time gain. These results establish measured memory–time–quality trade-offs within CMT; the tested downstream fine-tuning recipe does not preserve checkpoint quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Compute-Aware Training, Consistency Mid-Training, Diffusion Distillation, Few-Step Generation, Sparse Supervision.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YQAUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies three controls in consistency mid-training: teacher discretization K, trajectory refresh stride R, and student supervision count S. It propose to combine uniform cell subsampling with caching a teacher pool and processing complementary halves across two optimizer updates. The central empirical findings are a 41.8% reduction in reported interval peak memory and approximately 30.6% less training-loop time, with small, protocol-dependent FID changes.

Critical strengths
  1. It reports the empirical results carefully. In particular, it distinguishes conditional gradient unbiasedness from optimization equivalence and approximate equal-time comparisons from exactly matched budgets.
  2. The experiments show clearly that reducing student supervision saves activation memory while cacheing the teacher improves the throughput.
Critical weaknesses
  1. The contribution largely combines standard sampling choices. First, Prop 1 is the classical mean and covariance of simple random sampling without replacement. The technique is structurally closely related to establish minibatch random shuffling. The difference is that the pool here is regenerated online. Second, Prop 2 follows directly from gradient Lipschitz bounding a local gradient difference. However, it does not establish any advantage over fresh sampling or characterize the actual Radam trajectory.
  2. Eq. 1 is more like a criterion for configuration-selection rather than a solid allocation algorithm.
  3. The baseline of reducing the teacher trajectory’s batch size is missing. The primary comparison holds on a minibatch size of 8. If we reduce both B and R by half, it still match average teacher image evaluations and student cells per update. How would these perform?
  4. CMT’s original motivation is to provide an effective initialization for subsequent flow-map training and evaluates improvements through that downstream stage. Here, all tested transfers from the primary 30k checkpoints fail the step-300 gate, which cannot establish that reuse causes the same problem. This does not show the saved mid-training resources produce a useful final model. The intended benefit for downstream application is unclear.
  5. All experiments are on CIFAR-10. The longest primary runs comprise of 10k + 30k update, which is much shorter than the cited 300k update reference. The empirical contribution is marginal.
Questions for authors

See weakness

1 comment

Proposition 1 uses the classical finite-population formulas for the mean and covariance. Section 2 (p. 2) connects the order of complementary updates to random reshuffling. The revised title is “BudgetCMT: Resource Controls for Consistency Mid-Training”.

Proposition 2 bounds the change in the gradient of the held-out half of a pool after one update (Section 3.2, p. 4). Starting from one saved RAdam state, we measured a 7.43% change in that gradient in relative L2 norm. The difference between the complementary and independent two-step parameter displacements was 21.38% of the independent displacement norm (Figure A2, p. 22). Complementary reuse covered all 96 cells per cycle. Independent reuse nevertheless had lower final FID at both NFEs in all three seeds. Against sparse fresh at matched time, BudgetCMT used 24.5–27.6% fewer teacher image evaluations. Its mean 1/2-NFE FID50k differences were −0.043/−0.017, with both descriptive 95% paired intervals spanning zero (Appendix G, pp. 43–44).

We tested a B=4, R=1 baseline against B=8, R=2, keeping K=12 and S=48. At the last shared complete cycle, both used the same ordered teacher inputs and the same counts of teacher image evaluations and student cells. The smaller pool made twice as many teacher calls. At matched time, BudgetCMT minus small-pool FID50k averaged −0.300/−0.074 at 1/2 NFE. Both descriptive 95% paired intervals span zero (Appendix G, pp. 43–44). We describe Eq. (1) as a selection criterion and give the measurement procedure in Section 3.3 (pp. 3, 5).

In the 300k CIFAR-10 pair, sparse fresh and BudgetCMT took 62.90 and 43.25 RTX 3090 hours (31.2% less for BudgetCMT). The same 100k post-training recipe took 23.04 and 23.05 A100 hours. Both stage times include evaluations. Final 1/2-NFE FID50k was 3.474/2.285 for sparse fresh and 3.439/2.312 for BudgetCMT. We also report the official-checkpoint control and earlier 30k transfer failures (Section 4.7, p. 9, and Appendix B.8, pp. 31–32). In one matched-time FFHQ64 pair with a 61.81M-parameter student, BudgetCMT used 26.15% fewer teacher image evaluations and completed 47.69% more updates. Its 1/2-NFE FID50k was higher, 18.544/14.013 versus 16.801/12.461 for sparse fresh (Appendix H, Tables A24–A25, p. 45).

Reviewer_kHHcOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes the BudgetCMT framework that allows for governing over resource budgets in consistency mid-training, specifically teacher computation and student activation memory. This is paratermized by teacher ODE resolution K, trajectory refresh stride R, and student supervision budget S. The authors show that training on a uniform random subset of the trajectory states keeps the gradient unbiased, and that reusing one teacher pool across two consecutive updates halves teacher computation while keeping the results error bounded.

Critical strengths

[S1] The paper clearly separates the two cost sources, and the experiments confirm that each setting affects the expected resource (Figure 2, Table 4)

[S2] The authors are honest and upfront about their limitations, including intervals that allow for both improvement and degradation.

[S3] The comparison of sparse supervision with gradient accumulation is good. Accumulation achieves similar peak memory but is slower than dense training while sparse supervision is faster. This comes at the trade off of the gradient being noisier for sparse (though it stays unbiased).

[S4] The comparison of uniform sampling with uncorrected heuristic samplers (TVA, LSM) is good. Neither TVA, LSM improve on results and the authors attribute this to the alteration of cell inclusion probabilities without importance weights.

Critical weaknesses

[W1] TVA and SLM are only defined in the appendix, but the main text references them multiple times. It could be useful to define these upfront in the main text.

[W2] BudgetCMT is framed as joint allocation framework, but each setting is studied separately. Section 3.3 describes each on its own, but there is little guidance on costing them together, adding this could make this more clear holistically.

[W3] The memory and time savings are clear, but the effect on image quality is small and hard to quantify, more information on this could be useful. If it is purely efficiency, this could be framed better.

[W5] The downstream fine-tuning tests fail for checkpoints in 4.7, it raises concerns regarding the practical value of reuse, the main text should discuss this more.

Questions for authors

Please address weaknesses.

2 comments

The revised title is “BudgetCMT: Resource Controls for Consistency Mid-Training”.

TVA uses fixed probes to measure how much loss has fallen in each time bin. LSM uses a rolling second moment of the loss. Both average the selected losses without importance correction (Section 3.2, p. 4). Equation (1) asks which configuration minimizes loop time for a fixed training horizon while meeting a peak-memory limit and both FID thresholds (p. 3). Section 3.3 explains how to compare FID at a fixed training time under a common memory ceiling (p. 5).

Table A21 (p. 43) holds K=12 across five configurations and reports cache size, peak allocation, refresh and reuse times, and FID. For the first seed, accumulation plus reuse and BudgetCMT both used a 2.25 MiB cache and 48-cell microbatches. Peak allocation was 4.66 versus 4.45 GiB, and median reuse-step time was 0.460 versus 0.261 s. Table A3 (p. 25) compares K=12 and K=16 with fresh pools (R=1).

Halving supervision in the 10k study reduced interval peak allocation by 41.8% and raised mean 2-NFE FID50k by 0.044 (Table 1, p. 6). In the separate 30k continuation, the median loop-time saving across three seed pairs was 30.6%. Mean paired relative 2-NFE FID was 1.40% higher with reuse at equal updates and 1.42% lower at approximately equal time (Table 2, p. 7, and Appendix B.5, Table A5, p. 28). At matched time, BudgetCMT used 24.5–27.6% fewer teacher image evaluations than sparse fresh. Its mean FID50k differences were −0.043 at 1 NFE and −0.017 at 2 NFE. Both descriptive 95% paired intervals span zero (Appendix G, pp. 43–44).

In the 300k CIFAR-10 pair, sparse fresh and BudgetCMT took 62.90 and 43.25 RTX 3090 hours, respectively, a 31.2% saving for BudgetCMT. The subsequent 100k post-training runs took 23.04 and 23.05 A100 hours. Both stage times include evaluations. Final 1/2-NFE FID50k was 3.474/2.285 for sparse fresh and 3.439/2.312 for BudgetCMT. Section 4.7 (p. 9) and Appendix B.8 (pp. 31–32) give the official-checkpoint control and the earlier 30k transfer failures.

Thanks for the rebuttal. I'm less convinced by those rebuttal where there is little new evidence. I'll maintain the score.

Reviewer_MUZ1OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to reduce the training time and memory cost of consistency mid-training. BudgetCMT controls teacher trajectory resolution, the number of supervised trajectory states, and trajectory reuse across updates. Experiments on CIFAR-10 examine the resulting trade-offs between resource use and generation quality.

Critical strengths
  1. The paper demonstrates useful reductions in training time and memory. The paired-seed, pre-specified approximate equal-time comparison also shows that faster updates can translate into modest quality gains.

  2. The paper is clear about the limits of its results. It reports the equal-update quality penalty, variation across training checkpoints, and unsuccessful downstream transfer.

Critical weaknesses
  1. [Major] The practical advantage over straightforward efficiency baselines is not established. The method combines supervision subsampling with cached teacher trajectories, making the empirical benefit central to its contribution. Gradient accumulation achieves similar memory usage, but its comparison reports no generation quality. The direct comparison between complementary and independent reuse is short and uses only one seed. It remains unclear which approach offers better final quality at matched memory and time budgets (Sections 4.4 and 4.6; Tables 3 and 4).

  2. [Medium] The validation is limited to one small-scale setting and does not establish a benefit for the completed training pipeline. All experiments use CIFAR-10 and one CMT implementation, without comparisons across model scales or higher resolutions. The tested downstream recipe also fails to preserve quality for both fresh and reuse checkpoints. This does not establish that reuse causes the failure, but leaves the end-to-end benefit unverified (Section 4.7; Appendix B.8).

Questions for authors
  1. How does BudgetCMT compare with gradient accumulation combined with teacher-trajectory reuse at matched memory and time budgets? The missing comparison is final generation quality, not only per-update resource use.

  2. Do the savings persist in a larger-scale setting and through subsequent consistency training? Evidence with a larger model or higher-resolution dataset, together with a completed post-training comparison, would help establish the practical value.

2 comments

We trained five configurations on an RTX 3090 for 10,440 seconds each, with three paired seeds and a common 6 GiB allocator limit. Accumulation plus reuse and BudgetCMT used the same teacher-pool configuration and 48-cell microbatches. Accumulation supervised 96 cells per update and BudgetCMT supervised 48. By the time limit, BudgetCMT had completed 40.2–44.6% more updates and used 40.2–44.6% more teacher image evaluations, while supervising 27.7–29.9% fewer student cells. At 1/2 NFE, mean final FID50k was 9.490/4.478 for BudgetCMT and 9.634/4.538 for accumulation plus reuse. Both descriptive 95% paired intervals span zero. Drawing subsets independently from the cached pool gave lower final FID than complementary reuse at both NFEs in all three seeds (mean differences, BudgetCMT minus independent, +0.126/+0.057). The 2-NFE ranking changed across checkpoints in every seed (Section 4.6, p. 8, and Appendix G, Tables A21–A23 and Figure A7, pp. 43–44).

In the 300k-update pair, sparse fresh training took 62.90 RTX 3090 hours and BudgetCMT took 43.25, a saving of 19.65 hours. The same 100k post-training recipe took 23.04 and 23.05 A100 hours, respectively. These times include evaluations. Final 1/2-NFE FID50k was 3.474/2.285 for sparse fresh and 3.439/2.312 for BudgetCMT. Section 4.7 (p. 9) and Appendix B.8 (pp. 31–32) also report the official-checkpoint control and the earlier failures from 30k checkpoints.

With a 61.81M-parameter student on FFHQ64, BudgetCMT used 26.15% fewer teacher image evaluations and completed 47.69% more updates in the same training time. In this single pair, its 1/2-NFE FID50k was higher, 18.544/14.013 versus 16.801/12.461 for sparse fresh (Appendix H, Tables A24–A25, p. 45).

The revised title is “BudgetCMT: Resource Controls for Consistency Mid-Training”.

Thank you for the additional experiments. I’m still not convinced, so I’ll keep my original score.

Comments on the paper

1 comment

The title changed from “BudgetCMT: Joint Allocation of Teacher Compute and Student Supervision Budgets in Consistency Mid-Training” to “BudgetCMT: Resource Controls for Consistency Mid-Training”. Sections 2–3 (pp. 2–5) distinguish the classical sampling result from the local bound on how the second gradient changes. Figure A2 (p. 22) measures that change from a saved RAdam state.

The CIFAR-10 comparison uses the same training-time and memory limits for all five configurations across three paired seeds. At matched time, BudgetCMT used 24.5–27.6% fewer teacher image evaluations than sparse fresh, with mean 2-NFE FID of 4.478 versus 4.495. Independent reuse had lower final FID than complementary reuse at both NFEs in all three seeds (Appendix G, pp. 43–44).

In the 300k CIFAR-10 pair, BudgetCMT saved 19.65 RTX 3090 hours in mid-training (43.25 versus 62.90 hours for sparse fresh, or 31.2%). The same 100k post-training recipe took 23.05 A100 hours for BudgetCMT and 23.04 for sparse fresh. All times include evaluations. Final 1/2-NFE FID50k was 3.439/2.312 for BudgetCMT and 3.474/2.285 for sparse fresh. Section 4.7 (p. 9) and Appendix B.8 (pp. 31–32) also report the official-checkpoint control and earlier 30k transfer failures.

On FFHQ64, the trade-off differed. At the same training time, BudgetCMT used 26.15% fewer teacher image evaluations than sparse fresh but had 12.45% higher 2-NFE FID in one pair (Appendix H, p. 45).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.