- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SWIM freezes a diffusion backbone and trains a lightweight recurrent controller to select which transformer blocks to execute at each denoising step. The controller conditions on the current latent, prompt embedding, timestep, and a trajectory-curvature signal. A hard top-k rule keeps the per-step block budget fixed while allowing the identities of executed blocks to vary by sample and timestep. The method is evaluated on SD3.5, FLUX 2, PixelDiT, and CogVideoX.
Critical strengths
- The same-backbone FLUX 2 result is promising: SWIM reduces reported CMACs from 23.8T to 18.32T while changing the T2I-CompBench++ mean from 0.74 to 0.73. If all costs are measured consistently, this is a meaningful efficiency-quality trade-off.
- The PixelDiT controls support the value of learned dynamic routing. At essentially identical compute, SWIM obtains FID 3.69, compared with 4.52 for static selection and 55.25 for random anchored selection. This shows that block identity and temporal adaptation matter, rather than only the number of executed blocks.
Critical weaknesses
- The curvature signal appears circular or substantially more expensive than reported. The gate uses to decide which blocks to execute at step , but Appendix A.1 defines this quantity through a Hessian-vector product of the model output energy, requiring gradient computation and a double backward pass. It is unclear how this signal can be obtained before deciding which blocks to execute. If it is computed from the previous step, the indexing and first-step behavior are not specified. More importantly, the reported CMAC and latency accounting does not state whether these backward passes are included.
- The efficiency comparisons do not isolate the contribution of SWIM reliably. Several baselines use different backbones, schedules, or numbers of evaluations. The paper reports "active parameters," although every frozen block must remain stored because it may be selected later. The gate is separately trained for each backbone, so "model-agnostic" describes the framework rather than transferable gate weights. In addition, the proposed QNC metric adds a 20–25-scale CLIP score to a 0–1 benchmark mean, making the latter nearly irrelevant.
Questions for authors
- How is computed causally at inference time? Does its computation require executing the current backbone or performing backward/HVP operations, and are those costs included in every reported latency and CMAC value?
- How is the GRU trained over complete denoising trajectories? Please specify hidden-state initialization, rollout construction, gradient truncation, and how STE gradients are obtained for skipped blocks without evaluating them.
- Can the acceleration baselines be rerun on the same checkpoint, sampler, NFE, resolution, prompts, and seeds under matched CMAC or matched quality?
- Please distinguish stored model parameters from active parameters and report batch-size sensitivity, since block skipping only saves computation when all samples in a batch skip the same block.