- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies where to place temporal boundaries when a diffusion or flow model is split into equal-capacity specialists, only one of which is active at a given sampling time. Complexity-Balanced Splitting (CBS) first trains a lightweight auxiliary flow, estimates a positive complexity monitor along time, and chooses boundaries that equalize the cumulative monitor mass. The two proposed monitors use a density-weighted spatial Jacobian/Dirichlet-energy quantity and trajectory acceleration, respectively.
CBS is tested with SiT on ImageNet-256, JiT on ImageNet-64, and a UNet on CIFAR-10. Relative to uniform time intervals with the same number of specialists, the proposed boundaries consistently improve FID and related metrics while keeping the active parameter count and per-step FLOPs unchanged. The largest reported improvement is from FID 6.24 to 4.03 for SiT-XL with classifier-free guidance. Total training and storage cost still grow with the number of specialists.
Critical strengths
-
The boundary-selection rule is simple, inexpensive, and appears robust. It replaces an otherwise combinatorial search with a one-dimensional cumulative-mass calculation, and the auxiliary-flow ablations suggest that the resulting boundaries are not overly sensitive to the proxy model.
-
The empirical result is consistent across three architectures, multiple datasets, different numbers of specialists, latent and pixel-space models, and both monitor choices. The improvement over uniform splitting is often substantial. This breadth is the main reason I am on the positive side despite the concerns below.
Critical weaknesses
-
I see the theoretical development as motivation rather than a justification of the particular monitors. The assumptions linking local approximation error to the chosen complexity quantities are strong, and the Dirichlet-energy bound does not directly match the weighted estimator used in practice. I also could not reconcile the acceleration monitor with the usual local interpolation error proportional to step size squared times acceleration; this would normally lead to a square-root density rather than simply equalizing acceleration. The method may still work well, but the current claims should be framed as theory-inspired unless this link is tightened.
-
Unequal intervals also change the effective timestep sampling distribution seen by each specialist, so the experiments do not fully isolate better boundaries from better training reweighting. A monolithic model trained with the induced global timestep density is missing, as is a comparison with the same total number of optimization updates. Most results also appear to use a single training run. Given the extra training and storage cost of specialist models, end-to-end resource measurements would be more informative than active FLOPs alone.
Questions for authors
-
What happens if a single model is trained with exactly the global timestep density induced by the CBS intervals, with total optimization updates matched? This seems necessary to attribute the gain to specialization and boundary placement.
-
Can the authors report multiple training seeds for the main ImageNet result, or at least for a representative smaller setting? The current gains are promising, but run-to-run variation is not shown.
-
Could the authors clarify the exponent used for the acceleration monitor and compare it with the density suggested by the local interpolation-error formula? A held-out velocity-prediction error by interval would also help validate the proposed notion of complexity.
4 comments
Rebuttal (2/2)
W2 / Q2: Multiple training seeds
We ran three independent training seeds for each model on CIFAR-10:
| Model | FID, mean ± std (3 seeds) |
|---|---|
| Single network | 3.55 ± 0.03 |
| Uniform (0.33, 0.66) | 3.51 ± 0.02 |
| CBS (0.20, 0.78) | 2.73 ± 0.02 |
CBS improves FID by approximately 0.8 over both controls, with observed run-to-run standard deviations of only 0.02–0.03. These independent seeds provide evidence of repeatability on a second architecture and dataset (App. O).
Q3: Held-out velocity-prediction errors
We computed at 100 timesteps on 10K ImageNet validation latents, comparing the SiT-B/2 single network, uniform experts, and CBS experts (400K steps per network; App. N, Fig. 3).
The largest improvement occurs in the low-noise interval , where the acceleration monitor is highest. Relative to the single network, CBS reduces prediction error there by 1.6% on average and up to 2.1%, versus 1.4% / 1.8% for uniform experts. For the single network, the maximum integrated interval error is 0.290 under CBS boundaries versus 0.300 under uniform boundaries. Both observations are consistent with the complexity profile identifying an effective split.
The two expert configurations have similar integrated prediction errors (0.758 each, versus 0.764 for the single network) despite different FIDs. We report the per-timestep curves alongside the generation results rather than relying solely on a global velocity-MSE statistic.
W2: Training and inference resources
We added SiT-B/2 measurements (App. P):
| Resource | Single | CBS, N=3 |
|---|---|---|
| Total / active parameters | 130M / 130M | 390M / 130M |
| GFLOPs per sampling step | 23.01 | 23.01 |
| Main-model training GPU-hours | ≈106 | ≈318 |
| Training wall-clock | ≈53 h on 2×H200 | ≈53 h on 6×H200 (3 concurrent jobs) |
| Peak sampling memory | 15.8 GB | 15.3 GB (sequential loading) |
| Sampling img/s, no CFG / CFG 1.5 | 3.84 / 1.78 | 3.71 / 1.75 |
CBS retains the same active parameters and per-step FLOPs, with approximately 2–3% lower sampling throughput. The three experts require three times the main-model training GPU-hours and stored parameters; parallel training achieves comparable wall-clock time when the GPUs are available. The GPU-hour estimates cover main-model training, with auxiliary-model and monitor costs discussed separately in App. C.
Manuscript updates: Secs. 2.2/3.1 and Apps. A/M (theoretical scope and exponent); Secs. 4.4–4.5 and App. I (induced density and matched updates); Apps. N/O/P (held-out error, seeds, resources); Conclusion (limitations). The monitor's assumptions are discussed in the cited sections and appendices.
We hope that the additional experiments and clarifications address the reviewer's concerns and strengthen the overall contribution of our work.
Dear Reviewer,
Thank you for your time and effort in reviewing the submissions. We kindly ask you to check the authors' responses and engage in further discussion if necessary. Please let us know promptly whether your concerns have been addressed or if any issues remain unresolved.
Your timely feedback would be greatly appreciated. Thank you for your cooperation!
Thank you for the response and updated experiments. My concerns have been resolved, and I will keep my current score.
Rebuttal (1/2)
We thank the reviewer for the thoughtful theoretical assessment and the positive evaluation of our experiments. We address the theoretical questions and matched-update controls here; Part 2 covers training seeds, held-out errors, and resource measurements.
W1 / Q3: Theory and acceleration exponent
What the theory establishes. For a positive monitor under the integrated-burden assumption (A2), equalizing its cumulative mass exactly minimizes our min–max surrogate (Eqs. 3–5). Equal expert capacity (A1) motivates the objective; whether a particular monitor captures neural-network difficulty (A3) is evaluated empirically. We make this distinction in Sec. 2.2. Similarly, the spectral argument concerns unweighted Dirichlet energy, whereas our tractable, density-weighted Jacobian estimator is a motivated proxy (Sec. 3.1, App. A). Its empirical support comes from the boundary-sensitivity results (Tab. 4), both monitors (App. E), and auxiliary-model ablations (App. C).
Why use acceleration rather than its square root? In piecewise-linear interpolation, local error scaling as h2∥x¨∥ leads to a square-root knot density. Our setting partitions work across equal-capacity networks, and assumes the interval burden is proportional to ∫m(t)dt (A1–A2). Choosing m(t)=E∥x¨t∥ then yields exponent one in this surrogate; we describe it as a modeling choice, not a consequence of the spline error formula. Applying a square-root profile moves the SiT-B/2 split from (0.40, 0.77) to about (0.36, 0.72) (App. M). The empirical justification for the chosen monitor and resulting boundaries is in our experiments, rather than a claim of FID optimality.
W2 / Q1: Exact induced timestep density and matched updates
Three equally trained CBS experts with boundaries (0.40, 0.77) induce the aggregate density p(t)=1/[3(ti−ti−1)] on interval i, approximately (0.83, 0.90, 1.45). We trained a single SiT-B/2 model with precisely this timestep distribution:
At 400K updates, CBS-induced sampling improves the single network by 1.43 / 0.90 FID; CBS experts achieve 30.51 / 10.72 with the same updates per expert. At matched 1.2M aggregate updates, the reweighted single network yields 29.30 / 10.68, compared with 30.51 / 10.72 for CBS. This establishes the contribution of timestep sampling in this B/2 setup and makes the aggregate training cost transparent (Sec. 4.4, App. I).
Longer matched-update experiment (SiT-S/2):
At 2.4M total updates, CBS improves FID by 1.89 / 1.38 relative to a single network trained with standard timestep sampling. This is a separate, longer-training setting from the B/2 density control.
Our main controlled claim concerns boundary placement among equally sized expert ensembles: CBS consistently outperforms uniform boundaries with the same expert count, per-expert training schedule, total updates, and inference FLOPs (Tab. 1). This isolates the value of the complexity-balanced split independent of the extra cost of choosing a multi-expert design.