- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how to allocate training compute in Drifting Models. The authors aim to address the question "is extra compute is better spent taking more steps toward the current frozen target ("realizing") or recomputing the field ("refreshing")?". Using a decomposition of the frozen-target residual into aligned and orthogonal parts, they show on ImageNet that more steps do bring the network closer to a frozen target.
Critical strengths
The practical question of how to split compute between refreshing and fitting is an impactful design choice for Drifting models.
Critical weaknesses
-
The scope is too narrow to support general conclusions: the entire experimental scope is just one method (Drifting), one configuration, 2 seeds, and a long-run comparison of only $ k =1 \text{ and } 2 $. This is not substantive evidence to support any general conclusions. In those long runs a "refresh" also brings a fresh batch of noise and labels, so part of the result may be the generic finding that fresh minibatches beat repeated steps at matched compute.
-
Section 3 shows that deeper fitting helps mainly at –, yet the FID experiment tests only . Table 1 shows that makes less net progress than at checkpoints 10k–30k, including the 30k state from which the downstream runs start. The diagnostic therefore explains neither FID outcome and the regime where deeper fitting actually works is never tested.
-
The 40 local probes are scored by held-out field-space loss yet Section 5.2 argues that field-space gains do not reliably transfer to FID. The probes also form five dependent cells rather than 40 independent trials. The evidence to support their claim is weak and in tension with the paper's own argument.
Questions for authors
-
Does the reversal hold at the depths where your diagnostic shows realization actually works (–)? Without at least one such run, how does Section 3 support the downstream conclusion?
-
Please report the per-iteration cost breakdown (refresh vs. optimizer update) and a FLOP-matched comparison alongside wall-clock.