acceptodds
Under review as a conference paper at ICLR 2027

Decided Early, Seen Late: The Committor of Neural Network Training

Abstract

Training a neural network compiles one global number, the loss, into millions of weights, and one number cannot pin them all down: two runs of one recipe can fit their data equally well and still differ in whether they generalize or in which attention head carries a capability such as in-context copying. We ask which source of chance makes that choice, when it is made, and whether anything in the network shows it while it is being made. To answer all three, we copy a checkpoint into replicas that differ only in the randomness that follows, as chemical physics does to estimate a committor, and divide the variance of an outcome into the part chance has yet to decide, the part already decided that a description of the network can read, and the part it cannot. The instrument recovers what was known by other means, such as early stability to training noise and the equal effect of every source of randomness on a ResNet's errors. It also shows that (1) equally good runs differ along only a few directions, along which chance moves a run, (2) which source of chance decides depends on the recipe, (3) outcomes which are decided at all are decided early, at a median of 7.5% of training in our recipes and long before they show in the run, and (4) only a task readout or a projection of the network's outputs reads the decision as it is made, whereas the diagnostics commonly used to watch training read it thousands of steps later yet pass the usual within-run test of a precursor. The instrument needs nothing more than the ability to restart a run from a checkpoint, so one can use it to date when an outcome of their own recipe is decided, to check whether a proposed diagnostic reads that decision or only its aftermath, and to forecast whether a run will generalize by letting a few replicas run briefly.

Primary area:
interpretability and explainable AI
Keywords:
training dynamics, committor analysis, underspecification, grokking

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Jt8zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies a know technique in glass physics directly into the training of a neural network. The set up looks simple, during training checkpoints at different training times are saved and from them a set of new trainings are spawned with the same parameters. The aim is to measure how much of the training is from that particular spawning point still left to chance. The authors claim that the contribution is this instrument. The authors separate the effect into three terms eq1 in the paper, where the observed variance is then assigned to chance (variability allowed still in the training procedure) and to a part that is not chance (further divided ni the observable part and the non observable part). Then using this instruments, the authors recover known effects in neural network training a provide 4 new observations.

Critical strengths

The main strength of the paper is the use of creativity to apply a physics instrument into the training of neural networks. It is based on previous work that sustains the paper and the literature seems to be well researched. The authors recover many known effects in network training and that provides credibility to the instrument. With the help of the instrument authors claim 4 new observations. I think the strongest could be the localisation of the committing point of a network. That, is when the training change variability is low, below 0.1% in the paper.

Critical weaknesses

The paper is somehow difficult to read at some parts, some concepts are used before they are defined in the paper. One of the claims marked as new in the introduction (number 3, Decide early) is marked as "recovered" in A.2. The instrument seems to be computationally expensive to use in practice, since many retrainings per saved checkpoint are needed.

For me is not clear if before the “decision” still something could be read by another instruments, since the presented one has a part not readable (beta) at any of the sampled steps.

Questions for authors

Could you clarify if the "decision" point is dependent on the sampling grid? And if before this could be in practice be even earlier due to the "non readable" part of the eq1 at every sample point?

Over the text it seems that the main bed used is the bed F, and some of the observations seems to be extrapolated from this runs. Could these extrapolations be particular of the F configuration?

How much computation is needed to run this instrument? Could you provide some numbers?

Reviewer_jqkhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which training outcomes remain sensitive to future randomness, when that sensitivity declines, and which network measurements predict the resulting outcomes. It combines checkpoint-forking experiments with a standard variance decomposition. Separate experiments assess the effects of individual sources of randomness.

Critical strengths

The paper addresses an extremely important question, which is to understand the effect of randomness on generalizing solutions. They propose a unifying framework obtained from a variance-decomposition. This framework provides quantitative estimates of the effect of different sources of randomness in diverse settings.

Critical weaknesses

Clarity. The authors use the word “committor” in the introduction itself, without describing it. This breaks the flow for a reader unfamiliar with the chemical physics literature cited in the paper. Some non-rigorous terms like “date”, “run”, “recipe”, “testbed”, used throughout the paper without proper definitions. The abstract is a bit vague: “divide the variance of an outcome into the part chance has yet to decide, the part already decidedthat a description of the network can read, and the part it cannot”. Sentences are long and complicated to follow.

Reproducibility. The head-identity experiment lacks sufficient detail about the training-data construction, including the underlying token source and repeated-span sampling procedure. This makes the positional shortcut and the experiment difficult to reproduce.

Novelty and relevance: Equation (1) follows from a standard variance identity, so the contribution rests primarily on the measurement framework and its empirical applications. Although the paper distinguishes its approach from [1] and [2], the significance of the additional measurements remains insufficiently established. Section 4 quantifies decision times but gives limited explanation of what determines them. Section 5 distinguishes decision-aligned readouts from later precursors, but the practical consequences of this distinction require stronger demonstration.

[1] Cecilia Summers, Michael J. Dinneen Proceedings of the 38th International Conference on Machine Learning, PMLR 139:9913-9922, 2021. [2] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, Michael Carbin Proceedings of the 37th International Conference on Machine Learning, PMLR 119:3259-3269, 2020.

Questions for authors

(1) Please define “committor” at its first use, with a simple training example.
(2) I would appreciate some more details on the exact procedure used for changing settings in Section 3. If mentioned in Appendix, this must be noted in the main text.
(3) The procedure for measuring γ\gamma is not mentioned explicitly (line 140).
(4) The sensitivity of these variance measurements is not reported.

Reviewer_SxU7OpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies sources of randomness in training deep neural networks. It proposes to compare the variance in discrete outcomes (generalizes vs not generalizes) over full independent training runs with the variance obtained from a mid-training fork. Through this, the paper aims to locate a decision time point where training dynamics settle on the outcome.

Critical strengths

Locating sources and time points in training of deep neural networks that are pivotal for generalisation is an interesting avenue.

Critical weaknesses

Unclean empirical setup

The overall execution of the experiments is unclean. Throughout is not clear which models have been used, what data they were trained on and how a mid-training fork could change severe hyperparameters such as width/depth of the network or the optimization type. The discrete outcome study never formalised what it means to generalize or when attention heads encode a specific capability.

Theory falls short

No connection to stochastic gradient descent theory is given, nor is the unbiased estimator claim validated in a rigorous manner. The formalism for Eq (1) assumes the network is a deterministic function of the state of the optimizer and its weights. This seems overly simplified given the stochastic nature of the training, the very paper tries to disentangle.

Questions for authors
  • How broad are your claims in terms of model classes?
  • How did you formalize the discrete outcomes of a training?
  • How did you implement the mid-training forks for hyperparameters that heavily change the network?
Reviewer_7cmbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at 4 different deep learning settings, and studies at which time during training the final generalization performance becomes effectively deterministic. The analysis is based on training several networks starting from different seeds.

Critical strengths

The overall idea is interesting: at what point during training do networks become "locked in" to either learning a generalizing or non-generalizing solution?

Critical weaknesses

-- This reads like it is written entirely by Claude. In my opinion, Claude writes in a way that is hard to read, repetitive, and unmotivating for anyone trying to read the paper. My reaction is: why should I read your paper, if it is not clear whether the authors read it themselves? More concretely, the paper is presented in a way that makes it very hard to evaluate the claims. For instance,

  • The experimental settings are not clearly stated: ResNet-18 on what? It is only on page 5 we learn that it is a ResNet on CIFAR-10.
  • Figures are unparseable. What is each of the panels doing in Figure 3? Unclear from the caption, and impossible to get anything rigorous from the text.
  • Why should I care about fraction of overall training time? Maybe you trained for 100x longer than you had to, which would artificially deflate the time at which the "fork" is reached.

Overall, the exposition issues in the current version mean that this needs a major revision before it can be accepted, and I would suggest that the authors submit to the next conference iteration.

Questions for authors

-- Another methodological question is: given a certain set of parameters, why inject randomness only in the order in which the data is shown? Why not perturb the weights slightly? This paper implicitly assumes one of the many different ways to define the source of stochasticity during training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.