acceptodds
Withdrawn from ICLR 2027

Training-Path Equivalence: Validating Implementation Substitutions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Efficient neural kernels are typically validated through output and gradient agreement at fixed weights. Training introduces a different requirement: a replacement must preserve the reference behavior after repeated parameter and optimizer updates. We introduce Training-Path Equivalence (TPE), a framework that turns this requirement into an explicit substitution contract and uses short paired training paths to screen candidates. In a fresh evaluation of two previously frozen policies on 35 condition-seed units, the step-50 paired screen identifies all 20 contract rejections and all 15 passes; local checks at fixed numerical tolerances through step 200 miss three rejections and reject five passing units. A separately timed Math-to-Flash attention substitution, accepted in five fresh seeds under the lower-learning-rate 5,000-step contract, retains a 1.357x full GPU training-step speedup. Matched reference-seed calibration across nine conditions independently reproduces rejection and acceptance outcomes and quantifies the accepted pair's small functional gaps. Complementary long-horizon studies, replications up to 1.003B parameters, and optimizer-state interventions characterize how substitution decisions depend on training requirements. Together, these results establish paired-path validation as a practical complement to local numerical checks for selecting efficient training implementations.

Primary area:
optimization
Keywords:
training-path equivalence, implementation substitution, training dynamics, numerical validation, GPU kernels, optimizer state, efficient training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yWamOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work tackles the problem of numerical divergence when different (faster) kernels are substituted for known good ones. This work proposes a contract which specifies the divergence behaviour and a concrete way to test if a specific kernel substitution fits within the tolerances specified.

Critical strengths

This work identifies an important problem which is the long term numerical (behavioural) stability of different kernel implementations. This is particularly relevant now since there is significant diversity in hardware specific implementations in all parts of the stack (framework, parallelism, device specific kernels, etc.).

Critical weaknesses
  • This work does not fundamentally solve the soundness problem.
    • Training dynamic problems stemming from numerical and algorithmic differences in kernels can emerge way down the line during training[1] . This work essentially defers the correctness problem from a single invocation of the different kernels (the baseline) to a 50-step screen, while section 2.3 concedes that full paired training still determines the outcome.
    • This work also does not justify the 50 iteration prefix length for contract checking.
    • Practically, if the screen states a kernel selection is sound but it in fact diverges, we would not be able to know other than to train with the sound kernel end to end, which the paper already requires. What does the screen save?
  • The propagation bound is badly justified.
    • The work does not show that values the lipschitz factor takes on, and how it would change as training goes on. We don't know what factor to use before comprehensive measurements, and we also don't know if the same value would persist throughout the training.
    • Furthermore, even a small lipschitz factor, when multiplied like in equation 7 (and 22 in the appendix) would explode over many steps (e.g. 5000) even for very small values of L. In reality, much of the single-step differences could be corrected by later gradient updates.
Questions for authors
  • Would this methodology scale to larger models?
  • Suppose a divergence happens later in the training, how would the authors detect and perhaps defer to a more sound kernel?
Reviewer_mQWqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, authors study whether numerically similar implementations remain sufficiently close after repeated training updates. Training-Path Equivalence formalizes this requirement through a reference-conditioned contract specifying observables, tolerances, checkpoints, and training horizon. A frozen step-50 gradient-disagreement screen correctly predicts all 20 rejections and 15 passes in a fresh confirmation cohort, repeated fixed-tolerance local checks miss three rejections and reject five passing cases. An accepted Math-to-Flash attention substitution achieves a reported 1.357x full GPU training-step speedup. Additional experiments examine longer horizons, larger models, and optimizer-state mechanisms.

Critical strengths
  1. The controlled experiments are the strongest contribution of this manuscript, e.g., the fused/unfused AdamW intervention distinguishes first-moment from second-moment effects. Synchronizing the first-moment substantially reduces parameter and prediction disagreement, while synchronizing the second-moment doesn't. The update-norm-matched epsilon experiments also prevent an overly simple explanation based on epsilon alone.

  2. Authors also make useful distinctions in interpreting the results, the 50000-step experiment demonstrates that passing at the endpoint doesn't imply passing thoughout the monitored path. The accepted substitution is independently confirmed and timed using forward, backward, and optimizer updates, rather than preseting an isolated kernal speedup as a training speedup.

Critical weaknesses
  1. The main comparison seem to be not isolate the value of paired-path information, the proposed threshold is calibrated to predict final contract outcomes, while the repeated-local baseline retains numerical qualification thresholds and rejects after any failure. Besides, Table 2 provides useful early local-only and metadata comparisons, but E2 needs counterparts calibrated on the same historical results, together with comparable wall-time accounting.

  2. The 35 outcomes cover seven conditions with uniform labels within each condition, ten passing cases are exact controls, the other five come from one non-exact, lower learning rate attention condition. These primarily establish repeatability across fresh seeds for the selected conditions, but not reliable discrimination among unfamiliar substitutions near the acceptance boundary.

Questions for authors
  1. Can authors confirm that were same initialization, same-data reruns of each individual backend measured?

  2. How does the frozen step-50 rule perform on the mixed-output L8/L4 conditions in Table 19?

  3. On the E2 cohort, how do outcome calibrated local-feature and metadata-only predictors compare with the paired screen when calibration data and validation cost are controlled?

Reviewer_HqRTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper discusses whether a faster implementation can replace a reference implementation inside a training run, rather than only matching outputs and gradients at one frozen weight vector. The paper proposes a framework named Training-Path Equivalence to cast the replacement decision as a contract between a reference run and a candidate run which share initialization and data order. It further defines acceptance requirements including both a local output-and-gradient check and agreement on parameter distance, hidden CKA, and validation-loss gap. The same contract separates a terminal decision at the final step from a path-uniform decision over declared checkpoints. The paper validates the proposed method on small Transformers trained on TinyStories and WikiText-2, with concrete pairs PyTorch Math SDPA vs. Flash SDPA, LayerNorm variants, and fused vs. unfused AdamW, and varies the horizon, tolerance, and model scale up to 1.003B parameters.

Critical strengths
  1. The paper discovered and validated potential disagreement for a kernel-swap decision. The proposed contract separates two requirements, where terminal acceptance only ensures that the horizon lie inside tolerance, while path-uniform acceptance ensures every declared checkpoint to do so. The 50,000-step Math-to-Flash evaluations shows the split on the same five seeds, as the terminal rule passes 5/5, while the path-uniform rule passes 0/5. Figure 2 also shows that doubling the tolerance changes that path-uniform count.

  2. The proposed TPE validation strategy is shown to be effective in screening substitutions against a declared training contract. The reported step-50 rule tracks the later accept and reject labels, and related runs check the same decision under different learning rates, horizons, tolerances, and model scales. The confirmation cohort, local-check comparison, timing study, long-horizon profile, and larger-model replications are reasonably sufficient to support the usefulness of TPE.

Critical weaknesses
  1. The paper is hard to follow. The main text does not make the validation target, the formulas, or the experimental line auditable. The abstract reports “20 contract rejections and 15 passes,” a “step-50 paired screen,” and “condition–seed units” before naming the substitution, the three observables, or the tolerances. Section 2 then introduces QlocQ_{\mathrm{loc}}, zt(i)z_t^{(i)}, δm\delta_m, gmg_m, RCR_C, ρ50\rho_{50}, and DmTPED_m^{\mathrm{TPE}} without one worked example that plugs a real pair into the acceptance rule. Equation (7) is a Lipschitz accumulation bound that the text explicitly does not estimate and does not use as a certificate. A paper whose contribution is a decision procedure requires that procedure can be checked from the main text.

  2. The prospective-screen claim is stronger than the cohort that supports it.

i) It is not reasonable to consider the 35 units reusing five seed identifiers across seven conditions, rather than forming 35 separate outcomes. Section 8 states that the final labels are uniform within each condition. The 20/15 split is then five seeds copied across condition-level outcomes, as default A0 and N0 reject on both datasets (20), exact T0 passes on both (10), and low-learning-rate WikiText-2 A0 passes (5). Ten of the fifteen accepts are the exact control, which a faithful implementation should pass under any reasonable gate.

ii) The efficiency result does not leave this small-model contract. The timed network has 477,696 parameters and sequence length 128. The 1.357$\times$ speedup and the 10 MiB memory drop are measured there. At 355M parameters, learning rate 10−410^{-4}, and 20,000 steps, the same Math-to-Flash pair fails the terminal contract on 3/3 seeds, with median validation-loss gap 0.026. The paper reports this failure. The abstract still leads with the small-model acceptance and the 1.357×1.357\times figure as the systems benefit.

Questions for authors
  1. Please improve the presentation throughout the paper. The introduction should explain what the training-path contract validates and why this requirement matters in practice. The method section should be more self-contained, with a direct explanation of each formula and enough definitions to follow the acceptance rule without the appendix. The experimental section should also be reorganized so that the role of each study is clear before the results.

  2. Please confirm whether every E2 condition has one label across all five confirmation seeds, and state how many distinct condition-level decisions the 35 units represent. A held-out condition with mixed seed-level outcomes would change my reading of the 35/35 result in Table 1.

  3. Under any contract frozen before the large-model runs were inspected, does Math-to-Flash pass at 355M or at 1.003B? The reported 20,000-step run at learning rate 10−410^{−4} fails on 3/3 seeds. If no such acceptance exists, the 1.357$\times$ result should be scoped to the 477,696-parameter, sequence-128 workload.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.