acceptodds
Under review as a conference paper at ICLR 2027

Refresh or Realize? Compute Allocation in Drifting Models

Abstract

Drifting Models train a one-step generator by recomputing a finite-sample drift field at every iteration and taking an optimizer step toward the drifted target. The field says how generated samples should move, but the step is taken in parameters shared by all samples, so the motion the network actually makes need not match the motion it was given. This leaves a basic training question open: should extra compute go into fitting the current target more closely, or into recomputing the field? We study it on ImageNet . Holding the target fixed for optimizer steps and measuring the realized displacement, we find that deeper fitting does bring the network closer to the frozen target, and that the number of steps needed before it makes any net progress drops from about sixteen early in training to one later on. When the extra steps come for free, also lowers FID. Once they are paid for, the result flips: at approximately matched measured wall-clock, spending the budget on fresh fields gives lower FID than deeper fitting, on both training seeds. The target itself shows why a fresh field is worth so much. Redrawing the finite support rotates its direction far more than a parameter update does (cosine – against ), and a correction that is optimal in field space is not reliably better in FID than a parameter-free one. For Drifting, fitting each target well and spending compute well are different goals.

Primary area:
generative models
Keywords:
Drifting Models, training dynamics, compute allocation, target refresh, finite-sample estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ARwNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to allocate training compute in Drifting Models. The authors aim to address the question "is extra compute is better spent taking more steps toward the current frozen target ("realizing") or recomputing the field ("refreshing")?". Using a decomposition of the frozen-target residual into aligned and orthogonal parts, they show on ImageNet 256×256256 \times 256 that more steps do bring the network closer to a frozen target.

Critical strengths

The practical question of how to split compute between refreshing and fitting is an impactful design choice for Drifting models.

Critical weaknesses
  1. The scope is too narrow to support general conclusions: the entire experimental scope is just one method (Drifting), one configuration, 2 seeds, and a long-run comparison of only $ k =1 \text{ and } 2 $. This is not substantive evidence to support any general conclusions. In those long runs a "refresh" also brings a fresh batch of noise and labels, so part of the result may be the generic finding that fresh minibatches beat repeated steps at matched compute.

  2. Section 3 shows that deeper fitting helps mainly at k≥4k \geq 4–1616, yet the FID experiment tests only k=2k = 2. Table 1 shows that k=2k = 2 makes less net progress than k=1k = 1 at checkpoints 10k–30k, including the 30k state from which the downstream runs start. The diagnostic therefore explains neither FID outcome and the regime where deeper fitting actually works is never tested.

  3. The 40 local probes are scored by held-out field-space loss yet Section 5.2 argues that field-space gains do not reliably transfer to FID. The probes also form five dependent cells rather than 40 independent trials. The evidence to support their claim is weak and in tension with the paper's own argument.

Questions for authors
  1. Does the reversal hold at the depths where your diagnostic shows realization actually works (k=4k = 4–1616)? Without at least one such run, how does Section 3 support the downstream conclusion?

  2. Please report the per-iteration cost breakdown (refresh vs. optimizer update) and a FLOP-matched comparison alongside wall-clock.

Reviewer_D2XcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores compute allocation during the training of Drifting Models. The authors observe that after a single optimizer step (k = 1) there is still a "realization gap" to a frozen target. Early in training it can take up to 16 steps to get close to the frozen target due to orthogonal movements. Yet, when comparing deeper fitting of frozen targets (k = 2) against target refreshing (k = 1) under matched compute the default k = 1 method outperforms deeper fitting because, keeping the target frozen hardly updates the update direction, while refreshing the support provides new directional signal.

Critical strengths
  • The authors investigate how using deeper fitting in the standard optimization of Drifting Models impacts training under unequal and equal compute allocation. They honestly report that spending extra compute to reduce the realization gap overfits to the frozen target and negatively impacts training dynamics over refreshing targets at each iteration. Given that Drifting Models are a fairly new, but efficient and strong approach to image generation, this may be quite impactful to the community.
  • The authors decompose the updates into orthogonal and aligned updates, making the dynamics of the realization gap more tangible and showing how aligned updates become dominant over orthogonal updates with deeper fitting.
Critical weaknesses
  • The motivation for closing the realization gap is not clear to me. It is unclear, what observed properties are direct results of standard mini-batch SGD versus a property of optimizing Drifting Models. In standard SGD one gradient step also does not close the gap to a target and multiple gradient steps towards one batch target may lead to overfitting.
  • There is no discussion about AdamW optimization dynamics, such as gradient updates including averages of previous updates through momentum. Another way to make steps align more to the frozen targets could be to increase the influence of the latest update (lower beta_1). Although this changes overall optimization dynamics a discussion would have been interesting.
  • In the matched-compute setup it would have been interesting to see what happens if the optimization is run for 2 epochs over the same samples. Instead of two consecutive optimization steps using the same frozen support this would compare to just seeing each sample twice (isolating sample diversity from consecutive stepping).
  • For someone not deeply familiar with Drifting Models, the introduction could be more self-contained. Introducing clear definitions early and clearly distinguishing between background, motivation and novel contributions would make the work more accessible.
Questions for authors
  • Can the authors clarify what is expected optimization dynamics based on mini-batch SGD and AdamW and versus what is special to the setup of Drifting Models?
  • To what extent are the orthogonal movements early in deeper fitting caused by AdamW momentum (beta_1) carrying over previous update directions? Would lowering beta_1 align the steps to the frozen target faster?
  • In the matched-compute setup, how would k = 2 compare against running k = 1 for 2 epochs over the exact same sample set (so both setups see the exact same number of unique samples, but in random vs consecutive order)?
  • Would it be possible for the authors to introduce key concepts (such as the realization gap and support refreshing) earlier to make the write-up accessible to a broader audience?
Reviewer_BZtPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Drifting Models train a generator that produces a sample in one forward pass. During training, at each iteration, a vector field is constructed from real and generated samples, prescribing where each generated sample should move in space. The generator network then regresses toward these “drifted” targets. In the next iteration fresh samples are drawn, a new vector fields is constructed and the network is updated again. And so on so forth.

The question studied in this paper is where is best to spend compute. Regress more towards the targets at each iteration, or do more iterations?

The findings are that more per-iteration fitting helps, if free, but if one has to choose where to spend their budget then it's best to spend it doing more iterations with fresh samples.

For a methodological paper I would expect more expansive experiments and ablations.

Critical strengths
  1. A concrete, well-motivated question. Drifting prescribes movements for individual samples, but a shared network update only approximately executes them. Asking whether to fit longer per iteration, or recompute the target addresses an actual training-design choice.

  2. Several complementary experiments. The paper combines frozen-target diagnostics, training continuations, held-out local probes, support-resampling measurements, and affinity-floor audits. Together, these examine the interaction between field estimation, optimization, and implementation.

Critical weaknesses
  1. The main FID evidence is narrow. It covers two training seeds, one basic ImageNet configuration, and k=1 versus k=2. The comparisons are made for models continuing from existing checkpoints; importantly the paper does not test using k=2 throughout training from initialization. This supports a specific allocation result, with limited evidence for broader recommendations.

  2. The proposed mechanism is only partially established. The paper does not isolate whether the improvement comes from fresh support or from rebuilding the target at updated parameters. It seems that once larger k is consider there is another possibility that should be tested against the other two in the paper: do higher k but rebuild the target from the updated parameters after each micro-iteration.

  3. The theoretical novelty and predictive power are modest. The Jacobian-based displacement formula and fixed-kernel iteration are standard derivations. The residual decomposition is useful but elementary. None establishes when refreshing should outperform realization under equal compute, and the simplified dynamics do not quantitatively explain the actual AdamW trajectories.

Overall, my main concern is that, for a paper that proposes a specific training recommendation, there should be a lot more experiments with variations to the method to substantiate that this is a robust recommendation. For example, trying different optimizers, architectures, training from scratch (vs a common checkpoint, doing more ablations (e.g. 3).

Questions for authors
  1. Does refreshing still win when frozen-target fitting is implemented as efficiently as possible?

Appendix D.2 says targets are rematerialized during additional steps. Can you provide a cost breakdown and compare against caching the frozen target, or explain why caching is impractical?

  1. Does the reversal survive reasonable optimizer and averaging controls?

You match EMA updates and weight decay per refresh, but the time-matched arms have different refresh counts and optimizer-update counts. Does the result persist under alternative reasonable choices of EMA horizon, cumulative decay, and learning-rate schedule?

  1. How robust is the FID advantage across training randomness and starting points?

Are seeds 42 and 43 all the attempted training seeds? What uncertainty comes from FID evaluation? Does the advantage persist when branching from earlier checkpoints—or when training with each procedure from initialization?

  1. Is fresh support actually responsible for the local improvement?

Can you compare three branches: reuse the old target, rebuild the target using the same support, and rebuild it using fresh support?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.