acceptodds
Under review as a conference paper at ICLR 2027

Predicting and Steering the Emergence of Capabilities in Neural Networks

Abstract

Neural networks often acquire capabilities abruptly during training, from in-context learning to arithmetic and multi-step reasoning. Scaling laws describe when a capability emerges on average, but not whether a particular run will acquire it, or whether that outcome can still be changed. Here, we introduce the training committor, the probability that a capability emerges within a fixed horizon from a given checkpoint, which we estimate by restarting training from the checkpoint many times. In small transformers and in a language model trained on natural text, we find that most runs are effectively decided before the capability appears, and that the committor predicts which runs will acquire it. Notably, on the controlled tasks, it is more accurate than predictors based on loss or weight statistics. Beyond prediction, the committor reveals where to intervene: a short burst of training data aligned with the capability can rescue nearly all of the runs it marks as unlikely to succeed. Since its branches can be trained under any intervention, the committor can also compare interventions on the same run and pick the most effective one. Overall, our results establish capability emergence as a predictable and steerable property of the training state.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Training Committor, Capability Emergence, Training Dynamics, In-Context Learning, Grokking, Targeted Interventions

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UcPfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces the training committor, defined as the probability that a saved training checkpoint acquires a capability within a fixed horizon under a specified continuation protocol. To estimate this quantity, the authors launch multiple reseeded continuations from each parent checkpoint—typically sixteen branches—and use the fraction that acquire the capability as the forecast. Evaluation spans in-context learning, modular addition, permutation composition, sparse parity, a small natural-text GPT, and released Pythia/Pico checkpoints, with forecasts scored against each parent’s own continuation. The work reports strong ranking performance (e.g., AUC 0.963 on 93 held-out parents), compares the estimator with checkpoint-local predictors, and uses the forecast to select low-probability runs for targeted data pulses. The main contributions are a new estimator for training-checkpoint futures, evidence that most runs are settled early, and a proof of concept that the forecast can guide intervention selection and dosing.

Critical strengths
  1. The core framing is genuinely novel and potentially useful. Treating a training checkpoint as a state with a distribution of futures, and estimating a finite-horizon hitting probability by reseeded continuations, is a creative transfer from transition-path theory. This is consequential because it offers a principled alternative to reading a single observed trajectory, and the controlled-task ranking results are strong enough to merit discussion.

  2. The intervention results are striking as a proof of concept. In in-context learning, a short aligned data pulse converts 46/51 low-forecast parents that fail on their own, against 2/51 under ordinary training. Even if the selection and cost issues remain, this is a substantive empirical signal that low-committor states are not simply doomed and can be redirected by aligned data.

Critical weaknesses
  1. The estimand and the scoring protocol are not clearly separated, which complicates interpretation. The paper defines the protocol-conditional probability, yet in several places the wording suggests an estimate of “the checkpoint’s probability of success.” The primary reset protocol changes the optimizer state and data order, but the forecast is scored against the parent’s own continuation, which retains both. I noticed that the authors themselves report a systematic gap: reset minus outcome is −0.065-0.065, two of the 56 zero-bin parents acquire, and carry closes most of the gap. Because the paper uses not only ranking but also probability magnitude to set pulse doses and choose actions, a calibration failure under the scoring protocol would leave the “predict and steer” results insufficient to support the paper’s central conclusion.

  2. Practical value under realistic compute and decision constraints needs further demonstration. A full forecast requires 16 continuations; even a one-branch forecast costs about 950 times the 750 pulse updates saved by leaving 15 parents untreated, so the forecasting overhead far exceeds the intervention cost it saves. More importantly, the authors’ own waiting baseline reaches 69/7269/72, whereas committor targeting reaches 63/7263/72; the committor leads only before step 3,000. This means its advantage depends on a regime where decisions must be made very early and waiting is costly—a regime the paper does not characterize or justify. The action-selection experiment further weakens the “steering tool” claim: every parent selects the same action, and the per-parent oracle improves only from 91/11491/114 to 92/11492/114, so per-parent action choice contributes almost nothing. As presented, the method is closer to an offline diagnostic or proof of concept than a practical steering tool; a cost-matched decision analysis or a revised claim is needed.

Questions for authors
  1. What exactly is the target estimand? Please state whether the paper estimates qH,Πq_{H,\Pi} for a specific protocol or an “intrinsic” checkpoint probability. If the goal is to predict the parent’s own continuation, why is reset the primary protocol rather than carry or a protocol matched to the scored outcome? Please report calibration, Brier score, and decision-curve results under matched protocols, not only AUC under reset.
  2. What is held out, precisely? The current evaluation holds out the parent’s own continuation, but the 16 branches for that parent are used to estimate its own qq. Please clarify whether any checkpoint, initialization, task, or model family is truly held out. Were horizons, thresholds, cascade fractions, action libraries, and comparator choices pre-specified? If not, please provide nested cross-validation or a fully held-out evaluation with multiple-comparison correction.
  3. How dependent are the 16 branches? They share weights and differ only in reseeding and optimizer protocol. Please report branch correlation, effective sample size, sensitivity to B>16B>16, and whether the q=0/1 discretization drives the selection and dosing results.
  4. What supports per-parent action selection? If all parents choose the same action and the oracle gain is only 1/114, is the claim “the committor chooses the most effective action” or merely “the committor identifies one generally good action”? Please either enlarge the action library or revise the claim.
  5. Are low-forecast runs merely undertrained or more plastic? Please match treated and control parents on training step, loss, accuracy, weight norm, and optimizer state, and show that the pulse effect is not a generic effect of extra aligned data or additional updates.

If the authors can satisfactorily address the questions raised above, I will consider raising my rating accordingly.

5 comments
AuthorsOpenReview ↗

Response to Reviewer UcPf (1/2)

Thank you for your careful and specific review. We respond to each of your points below in the order mentioned.

Weakness 1 and Question 1. What the forecast estimates and how it is scored

We agree that the original submission did not make this distinction clear. To clarify, the forecast is used to estimate the probability that a run acquires a capability under one stated way of restarting it, and not an inherent property of the checkpoint. We rewrote Section 3 and added in Table 1 to make this clear in the text.

You are also right that our default restart (reset) differs from the run that it is scored against. As you suggested, we repeated the evaluation with restarts that keep optimizer states as well (carry), making sure to score both on the same outcomes. Section 4.1 now reports the main results, and the remaining tables are in Appendix B.

Original seeds, resetOriginal seeds, carryNew seeds, resetNew seeds, carry
AUC0.9270.9880.9280.997
Brier score0.0590.0360.1060.025
Average forecast minus outcome-0.065+0.024-0.118+0.019

As shown in the table, the carry removes the under-forecast that you mentioned. Additionally, none of the 83 runs with a zero carry forecast went on to gain the capability. The reset is kept as the default because it tests how much of the outcome is decided purely by the weights. It is also accurate when scored with its own kind of restart (AUC 0.982, Brier score 0.033). On the added decision curves, both do better than treating every run or none, except reset at the most extreme thresholds on the new seeds. The steering results hold under both kinds of restarts. Moreover, treating the 57 top ranked runs gives 63 acquisitions with either, and the rule that sets pulse length from the probability reaches 45.5 expected with carry while using about 34% fewer pulse updates (1,065 vs. 1,610), and 47.4 with reset. When a fixed threshold is used instead, it simply needs to be set for the kind of restart, since carry forecasts run slightly higher than reset (Appendix B.4).

Weakness 2. Practical value and the cost-matched analysis

We both revised the claim and added a cost-matched analysis as suggested (Section 4.5). We now state that the practical result is the pulse itself, applied by a simple rule that does not need a forecast. The rule is to wait to step 1,000 and then give a short pulse to every run that has not yet gained the capability. With this rule all 72 runs acquire the capability, compared to 16 under ordinary training. From our experiments, we clearly note that waiting and then treating does better than choosing runs by forecast, as you observed. We therefore do not claim that forecast-based allocation is the best way to choose runs.

We agree that the cost matters for practical use. However, as the first method to estimate this probability for a training checkpoint, the committor's main contribution is to figure out when a run's outcome is decided and if something can ever change it. It shows which runs are failing rather than slow, and it decides how long a pulse each run needs (Section 4.3). Neither of these can be read from a single training run, and, while expensive, they do offer important information that could not be obtained otherwise. In a practical sense, this information is how we arrived at the rule above.

To make the forecast more usable, we also made it cheaper through the revisions. By using shorter restarts that stop at step 1,000 with more added only when the first four disagree, the cost ends up being about 3% of the full forecast. The AUC is 0.990 compared to 1.0 for all sixteen short restarts (Section 4.2). The forecast is most useful when a decision about a run has to be made early, before simply training it longer would show whether it succeeds. At step 100, it predicts the run's later held-out score with R² of 0.88 to 0.95, compared to 0.32 to 0.49 for the best checkpoint statistic (Section 4.1). Finally, the steering in our title refers to the pulse, and this holds on new seeds, a wider model and a larger task (Table 3).

AuthorsOpenReview ↗

Response to Reviewer UcPf (2/2)

Question 2. Held-out evaluation and pre-specification

Thanks for your question. In the submission, only each run's own continuation was held out. To expand our study, we added six cohorts that the original results never used (Section 4.1, Table 1, Appendix B.4). The new seeds keep every setting of the original runs, the wider model changes only the model size, the 40-key task only the number of keys, and the added dihedral group is a task family that we never used in development. The results are below.

New cohortRunsAcquireAUC
In-context learning, new seeds72260.928
Wider model40140.871
40-key task4041.0
S5 composition16121.0
Dihedral group composition4091.0
Sparse parity30100.995

The horizons were set on separate pilot runs, and the rule for admitting baselines and the action library were fixed before scoring. The main choices made after seeing outcomes are below. None of them enters the forecast results in Section 4.1, and the table shows where each is used and its test.

ChoiceUsed inTestResult
Cascade fractionSection 4.2Nested cross-validationSame fraction in 71 of 72 folds
Dose lengthsSection 4.3Held-out, 60 new runs58.1 of 60 expected vs. 47.2 with shuffled doses
Natural-text horizonSection 4.6All horizons, 2,700 to 3,600AUC at least 0.981 at each
Released-model thresholdsAppendix FMarked as exploratory

After the multiple-comparison correction you asked for, 7 of 23 baseline comparisons remain significant, with the committor ahead in all 23 (Section 4.1).

Question 3. Dependence between the 16 restarts

Thanks for pointing this out. We have added this analysis to our paper (Appendix B.2). Restarts from the same checkpoint are close to independent when the run's own probability is controlled for. The remaining correlation between them is between -0.02 and +0.01, with the effective sample size at least 10.6 of the 16 restarts. Importantly, the result also holds with more restarts, with the AUC at 0.924 with 32 and 0.927 with 16. Finally, the 0 or 1 forecasts do not drive selection or dosing results, shown by three thresholds from 1/16 to 4/16 selecting the same runs.

Question 4. Per-run action selection

You are correct that the submission supported only the second. The more accurate claim is "identifies one generally good action" and we have revised the paper accordingly (Section 4.5). We also took your advice, and enlarged the action library. With four different kinds of action on 32 runs, choosing the kind of action for each run gains only a little over the best single action, helping support the revised claim (Appendix D.7). However, what varies is how long a pulse each one needs. On 60 new runs, choosing each run's pulse length from its own restarts beat the best fixed length whenever pulse updates have a cost, ignoring the cost of restarts (Section 4.3).

Question 5. Undertrained or more plastic

Thank you for raising this very important point. We agree that matching is important to truly isolate the effect of the pulse, and we have added exactly this comparison (Section 4.3, Table 3, Appendix D.2). First, we compared all runs at the same training steps, and across 144 of them, the ones with a low forecast do not differ from those with a high forecast in training loss, training accuracy, weight norm, or the optimizer's moment norms. In addition, in 49 pairs matched on all of these and on held-out accuracy, the pulse raises acquisition compared to ordinary training by 0.755 for low-forecast runs, compared to 0.224 for high-forecast ones. Held-out accuracy is the only statistic that does differ, so naturally we also paired each low-forecast and high-forecast run of the same held-out accuracy. In these 39 pairs, ordinary training brings 29 of the high-forecast runs to the capability but only 5 of the low-forecast runs. The pulse, instead, brings 34 of these low-forecast runs to it. Low-forecast runs are also not more plastic, since they learn random labels about as well as high-forecast runs.

Second, we had to be sure that the pulse is not a byproduct of extra updates or extra data. In every comparison, the pulsed and unpulsed copies of a run are started from the same checkpoint, so they share training steps, weights, and even optimizer state. On 42 runs the same number of ordinary updates rescues 2, more data of the kind the run has already seen rescues 3, and a random change is able to rescue none. The pulse here is able to rescue 39 of these 42 runs. This shows that the rescue comes from the data that is new to the run, instead of just the additional updates or more of the same data. This is explained further in our reply to Reviewer JvPA and in Section 4.3.

We hope this answers each of your five questions. If any remains open, please tell us and we will address it.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer UcPf,

Thank you again for your careful review, which helped us improve the paper. In our two comments and the revised PDF, we have addressed each of your questions, including the estimand, matched-protocol scoring, the cost-matched analysis, and the held-out cohorts.

As the discussion period ends on November 18 (AoE), we would greatly appreciate it if you could let us know whether our responses address your concerns. If they do, we would be grateful if you would consider raising your score. We are happy to clarify anything that remains open.

Best regards,

The Authors

Reviewer_4HaEOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

The authors examine whether the emergence of a capability within a fixed horizon during neural network training is predictable. They call their method the "training committor" which involves sampling different continuations from a current fixed point in training, and reporting the percentage that acquire the capability. They find that their predictor is quite reliable (most continuations end up at the same outcome), and that it is possible to steer that outcome in some cases with targeted training. This is an impactful finding given the importance of being able to predict a model run at scale.

Critical strengths
  • Important question
  • Simple method. I appreciate that you investigated both reset and carry versions
  • Interesting finding that most capabilities are predetermined early in training despite variability in training continuations.
  • Appendix E.2 shows how the method could be made more practical
Critical weaknesses
  • The main text doesn't always stand on its own, relying heavily on the 16 page appendix to explain the tasks, experimental protocols, alternative models, and intervention experiments.
Questions for authors
  • How do you imagine this technique being used in practice?
  • How would you revise the main text so that it doesn't rely as much on the appendix?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4HaE

Thank you for your positive review and for the two questions, which helped us to make the paper clearer. Both are answered below.

Question 1. How the technique would be used in practice

We revised the Conclusion to show this. Here we summarize the three main uses in practice, and the rest are outlined in Sections 4.3 to 4.6. First, it shows something that cannot be known otherwise. Restarts from a stalled checkpoint show whether the run was unlucky or unlikely to ever succeed in the same training, which a single run can never tell us.

Second, it can be used to fix a failing run. We can restart a run from its checkpoint with a change to its training, and see whether the restarts now succeed. Doing this showed us that a short pulse of new data fixes failing runs. With this pulse at step 1,000, all 72 runs gain the capability, instead of only 16 under ordinary training (Section 4.5).

Third, we make the forecast much cheaper in the revision, which allows it to be more usable in practice. We now start with four short restarts and add more only if they disagree. This ends up taking 5.8 restarts on average, which is about 3% of the original cost while also keeping an AUC of 0.990 compared to 1.0 with all sixteen short restarts (Section 4.2).

Weakness and Question 2. Making the main text stand on its own

We agree that the submission relied too much on the appendix. Following your suggestions we used the extra main text page to bring important material into the main text. It now points to the appendix 17 times compared to 29 before, and we made sure it can be read on its own without requiring the appendix. Every remaining pointer only adds supplementary detail for a result that the main text already states. For ease of understanding, we also added Table 1 in Section 3 to show what each restart keeps and what it randomizes, with the tasks and runs below it all in one place. Section 4 was also revised to describe the baselines when they first appear, explain how interventions are built, and say the dose rule in full (compared to being in the appendix before). Section 4 also has a new subsection on what decides whether a run acquires a capability.

We hope this addresses your questions and are happy to answer any further ones.

Thank you for the reply and for the improvements in readability. I raised the Presentation score, and I maintain my overall rating of 4, as it can't go any higher!

Thank you for your response and for your comments on readability. They were very valuable in improving our paper.

Reviewer_JvPAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces the notion of “training committor”, defined as probability that a capability emerges within a certain training steps from a given check-point. The probability is computed empirically by running the training multiple times from the given check-point. Then authors claim that the training committor “predicts” whether another training run started from the same model check-point leads to acquisition of the same capability. Authors make the claim based on experiments, where they evaluate small transformers' capabilities on in-context learning, modular arithmetic, and composition of permutation. Finally, the authors demonstrate that adding data, which is related to the task that the model is later evaluated upon will increase the model’s performance on the task after training.

Critical strengths

Introduction of the concept of committor to deep learning could be interesting. In chemical physics, this concept has been used not only to compute transition probability between chemical states, but also to understand mechanisms that underlies such transitions. Introduction of this concept could be productive if it helps us identify new mechanisms in training dynamics, as being done in chemistry.

Critical weaknesses

Overall, this paper significantly suffers from lack of clarity in its presentation as well as a large gap between what's claimed and actually demonstrated. Writing suffers from its vagueness and figures are hard to understand.

First, starting from the title, authors repeatedly claim that the training committor "predicts" whether additional training from a given model check-point results in the model acquiring a certain capability. However, training committor is a summary statics that requires repeated (~16) training runs from the given model check-point to begin with. Of course, there could be variabilities in stochastic gradient descent, but framing the above results as "prediction" is very misleading. This fact also makes the procedure not useful in practice.

Second, authors then claims that an addition of "a short burst of training data aligned with the capability" can make model learn the capability. However, the fact that a model trained on the added data related to evaluation performs better is not significant without a hypothesis explaining why we should expect otherwise.

Finally, there lacks a conceptual connection between the first and second contributions. It's reasonable to expect that a check-point that did not learn a certain capability after multiple repeated trainings to require addition of more training data related to the capability, and the conceptual framework did not yield new mechanistic insight nor practical value.

Questions for authors

I believe this manuscript requires substantial improvements in both its presentation and the thoroughness of its results before it is suitable for resubmission to future conferences.

5 comments
AuthorsOpenReview ↗

Response to Reviewer JvPA (1/3)

Thank you for the feedback. We answer all your points in order below. Importantly, thanks to your guidance we were able to add the mechanistic analysis that you pointed out as missing (Section 4.4, our third comment).

lack of clarity in its presentation as well as a large gap between what's claimed and actually demonstrated. Writing suffers from its vagueness and figures are hard to understand

We agree that the writing was too vague in places, and we revised the paper extensively to improve this. In several places, we had stated a conclusion without saying what exactly was measured. As an example of this, we wrote of "a rule that sets each pulse length from the forecast" without stating the rule. In the revision, Section 4.3 states the rule as no pulse for forecasts of at least 0.5, ten updates for those between 0 and 0.5, and 75 updates for a forecast of zero. Additionally, Table 1 was added to clearly show what each restart keeps and what is reset, with the tasks, runs and exactly what counts as "acquiring" a capability next to it. Table 3 also gives every intervention result, with the new replications and controls in one place. For clarity, we went through the text and figures as well and replaced terms that could be confusing with standard terms in the field. Examples of these include "baseline" for "comparator" and "threshold" for "bar". The captions of Figures 2 and 3 were also revised to say what each panel plots.

You also mention a gap between what we claim and what we show. Our revision fixes this by adding new experiments. Beyond the original 72 in-context-learning checkpoints we had, the revision adds six cohorts and a natural-text model (Sections 4.1 and 4.6, Table 1, Appendix B.4). The results for four of these and the natural-text model are below.

New cohortRunsAcquireAUC
In-context learning, new seeds72260.928
Wider model40140.871
Dihedral group composition, a new task family4091.0
Sparse parity, new runs30100.995
10M-parameter GPT on FineWeb-Edu3214 by update 2,8001.0

requires repeated (~16) training runs from the given model check-point to begin with. Of course, there could be variabilities in stochastic gradient descent, but framing the above results as "prediction" is very misleading

The committor is, as you explained, a summary of repeated training runs, and we should have explained more clearly why we decided to call it a prediction, in the title and rest of the manuscript. The reason for this is how it was tested. Every forecast is scored against the original run, which keeps its own optimizer state and minibatch order. This is not visible to any restart, so the forecast could have failed. Summarizing repeated simulations is also a standard way to predict very similar systems whose future is not deterministic. A great example of this is in ensemble weather prediction models that launch many simulations from perturbed estimates of the current atmosphere to predict the probability of future states (Molteni et al. 1996, https://doi.org/10.1002/qj.49712252905; Leutbecher and Palmer 2008, https://doi.org/10.1016/j.jcp.2007.02.014). To make this clearer, Section 3.2 now states both of these points when it defines the forecast. The restarts are needed because the strongest predictor read from the checkpoint alone reaches AUC 0.794, compared to 0.963 for the committor.

You also note that stochastic gradient descent varies. That variability is exactly what the restarts measure, and it is inherently why the prediction is a probability. Furthermore, if the minibatch order after the checkpoint decided the outcome, restarts from one checkpoint would actually split, which they do for 18 of our 93 checkpoints. For the other 75, however, all 16 restarts agree, and the original run matches them on 73 of those 75.

not useful in practice

We agree with the point about cost, since each forecast in the submission needed sixteen full restarts which is expensive. The revision makes this much cheaper by starting with only 4 short restarts, adding more if and only if the 4 disagree. In our measurements, this takes 5.8 restarts on average which is about 3% of the original cost. Importantly, the forecast is able to stay nearly as accurate during this (AUC 0.990 vs. 1.0 with all sixteen short restarts; Section 4.2). The cost is able to demonstrate something a single run cannot show, how likely a run is to fail and what has to change for a training to rescue it (Sections 4.3 to 4.5). This also led directly to a practical result in our paper that needs no restart at all. By using the forecast we were able to discover that the failing runs can be rescued by a short pulse of the right data, so we could give this pulse early on at step 1,000 to every run that did not yet have the capability. All 72 then acquire it, compared to 16 under ordinary training (Section 4.5).

AuthorsOpenReview ↗

Response to Reviewer JvPA (2/3)

not significant without a hypothesis explaining why we should expect otherwise

We fully agree that this can be seen as expected, since the added data matches the capability tested for. However, prior work suggests an opposite hypothesis, that a run which fails under ordinary training stays stuck. Deficits early in training can become permanent (Achille et al. 2019), networks trained only on part of their data generalize worse even when the rest is added (Ash and Adams 2020), networks trained continually lose an ability to learn (Dohare et al. 2024), and in-context learning can give way to in-weights learning (Singh et al. 2023). In these papers, further training on more data did not repair the network. Additionally, the methods that actually restored its ability to learn reset or changed some of its weights (Ash and Adams 2020; Dohare et al. 2024). Our pulse does not do this. It does not reset or edit any weights, and changes only the training data for fifty updates. We apologize for not making this motivation clear during the submission time. We have updated Section 4.3, which now opens with this hypothesis motivated by prior work and explains that the rescue could have easily failed instead.

To be sure that nothing simpler explains the rescue, we ran three controls. First, to test whether the failing runs are just simply slow, we ran one block of 16 runs to ten times longer than ordinary training (100,000 updates). Importantly, the gap to the pulsed runs did not change. Next, to see if extra related data helps, we gave 50 updates of more data of the kind the run already trains on. This rescued only 3 of the 42 runs. In contrast, the same 50 updates on three mappings the model did not see rescued 39 of the same 42. This shows that the rescue depends on the data being new to the run as well, not just related to the task. Finally, the runs could also not be learning the test answers, because evaluation uses mappings that were not in the training or pulse.

there lacks a conceptual connection between the first and second contributions

Thank you for this valuable feedback. We agree that the submission did not make this explicit, and clarify how the two are connected in the revision. The first contribution measures the probability under ordinary training where the forecast near zero says that continuations are unlikely to work. The second logically tries to ask whether anything can change that, and changing the training does. As you said, a failing run may well need more of the right data. However, a single failed run can never tell us whether something was unlucky or needs an entirely different training, only the committor can.

The connection is also important in a practical sense, because the forecast from the first helps control the intervention in the second. First, the forecast is able to decide how long a pulse each run gets (Section 4.3). In a replay on recorded outcomes, these lengths rescue 47 of the 48 runs, the same as the best fixed pulse, while using 27% fewer updates. Second, the same restarts can be used under any intervention, not just the ordinary training, allowing for comparison between possible interventions before any is actually used. Across 19 saved states, they compare 4 candidates and identify the one that works. On restarts that were not used for this, it is able to reach the capability 80% of the time, compared with only 36% for a random choice (Section 4.5). Section 4.3 now explicitly opens with the link between the two contributions.

AuthorsOpenReview ↗

Response to Reviewer JvPA (3/3)

did not yield new mechanistic insight

Thank you for noting the lack of mechanistic insight, and for offering the comparison in chemistry. To address this we used the committor in parallel to the way chemists do, to test what decides whether a run acquires in-context learning (added in Section 4.4 and Appendix D.7). In our task, the model was allowed to answer a question by looking back at an example in its context that had the same input and copying the label (Olsson et al. 2022; Reddy 2024). We tested whether learning to attend to this matching example is the step that can separate runs acquiring a capability from runs that do not. The experiments indicated that this was true.

First, having this attention is enough to fully solve the task. When a model is forced to attend to the matching example early in training, accuracy becomes perfect in all 72 runs, including ones that go on to fail. This demonstrates that every run can already use this attention, and the difference between runs is whether it forms.

Second, strengthening this attention actually rescues failing runs. To test this we added a training term that strengthens this attention for 50 updates. In six runs, almost certain to fail otherwise, the probability of acquiring the capability, which is the committor, rose from 0.010 to 0.750 at the strongest of the three settings. A random change of the same size, or the same push on the value and output weights instead, left it at 0.010.

Finally, the data pulse works through this attention. When we froze the query and key weights, which set this attention, the pulse's success fell from 0.783 to 0.143. If we instead freeze as many value and output weights, the success is left at 0.824. At step 1,000 forcing this attention brings all of the 106 runs that have not gained the capability to the threshold needed. The attention also is able to separate runs which we do not even edit. Among restarts from identical weights, the ones that will acquire tend to have stronger attention within 100 steps, ahead of held-out loss or accuracy.

These results together provide clear evidence that whether a run forms this attention helps decide whether it can gain a capability. This mechanistic discovery was possible through the committor, since none of the 86 standard statistics we fit reproduces the committor. Besides the in-context learning approach we explained here, the same kind of test on modular addition and permutation composition points to the MLP (Section 4.4).

We hope that we have addressed all your concerns, and are happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all the reviewers for their time, thoughtful feedback, and constructive suggestions. These reviews greatly improved the clarity and rigor of the paper. We have uploaded a revised manuscript.

The changes are summarized below.

  • New held-out cohorts. We added six cohorts that the original results never saw, including a wider model and a new task family. The forecast now reaches AUC from 0.871 to 1.0 on them (Section 4.1, Appendix B.4).

  • A cheaper forecast. The forecast was updated so that it can use a few short restarts, at about 3% of the original cost. The AUC remains high at 0.990 compared to 1.0 for all sixteen short restarts (Section 4.2).

  • Practical rule. The restarts showed a simple rule to use in practice. A short pulse at step 1,000 given to every run that has not gained the capability brings all 72 runs to it instead of only 16 under ordinary training (Section 4.5).

  • Additional steering results. Pulsing rescues 35 of 44 failing runs on new seeds, 11 out of 11 on a wider model, and 29 out of 36 on a larger task. Ordinary training rescues only 4, 0, and 0 runs respectively (Table 3).

  • New attention result. Strengthening one attention pattern raises the committor, or success rate, of failing runs from 0.010 to 0.750. The pulse is shown to work through this attention, since freezing the weights that set it cuts the pulse's success from 0.783 to 0.143 (Section 4.4).

  • Redesigned natural-text experiment. Restarts now forecast when each of the 32 runs gains induction as a capability, with a rank correlation of 0.977 (Section 4.6).

  • Clearer definitions and standard terms. We updated Section 3 to clearly define what the forecast estimates, with an added table and definitions. For clarity overall, we also replaced terms that could be confusing with standard terms in the field, such as "baseline" for "comparator" and "threshold" for "bar".

  • Expanded statistical analyses. We added calibration and decision curves with matched protocols, nested selection, a correction for multiple comparisons, and an analysis of the dependence between restarts (Sections 4.1 and 4.2, Appendices B and C).

We respond to each reviewer in more detail below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.