acceptodds
Under review as a conference paper at ICLR 2027

Forward Similarity Does Not Guarantee Training Equivalence: Precision Placement in Differentiable Gram Solvers

Abstract

Changing the numerical precision of a differentiable Gram solver may leave predictions nearly unchanged while altering the computation used to learn its parameters. We study this distinction in controlled Spectral Koopman Attention (SKA) and an independent episodic ridge classifier, assigning FP32 or FP64 to separate solver stages without changing their exact-arithmetic definitions. Matched comparisons fix the model, input, and optimizer state; separately trained endpoints are evaluated through a common precision route. For SKA, we compare with the higher-precision solver route , a numerical reference rather than mathematical ground truth. Across 18 matched SKA validation comparisons from three training seeds, median relative discrepancies in logits, full-model gradients, and restored AdamW updates are , , and , respectively, each normalized by its corresponding reference norm. Prediction agreement can therefore miss differences in the learning computation. In a seven-seed matched classifier comparison, imposing externally prescribed ridge schedules shared across routes within each seed during training reduces the absolute between-route spread in post-training regularized-Gram conditioning by relative to learning ridge separately under each route. This shows that the observed conditioning separation depends on the training-time ridge policy. Precision choices in these solvers therefore warrant checks of parameter updates alongside predictions, with endpoint conditioning and task outcomes assessed separately.

Primary area:
optimization
Keywords:
differentiable solvers, numerical precision, mixed precision, numerical linear algebra, optimization, training dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_F2TeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes that the similarity in forward outputs of models doesn't mean they will have similar backward gradients or parameter updates. The authors learned numerical behaviors in SKA and episodic ridge classifiers via comparing the difference in fp32/64 precision across gram construction, cholesky factorization and solution-side solves. To validate that, they built experiments over 18 matched SKA comparisons and found that the mixed precision q110 yields a small median logit discrepancy compared with the FP64 q111, however, it incurs a large discrepancy in gradients and optimization. Overall, I think the experiments designed in this paper are quite responsible. The authors validate their claims via cross-platform runs, custom VJP backward re-computations and so on. However, the problem discussed here may not be that novel to me, and I will argue that this is more like an overcomplication of a common engineering problem.

Critical strengths
  1. The issue discussed is important, especially for the realistic training of LLMs. The authors experimentally proved that evaluating forward convergence is not sufficient. Furthermore, they also showed that gradients and VJPs are very sensitive to precision settings. I think the paper can provide a warning to those who works in linear attention or implicit AI.

  2. The experimental control and attribution of issue is strong and solid in this paper. Specifically, isolating gradient shifts by fixing optimizer states and inputs, proving backward discrepancy originates from the solver submap via VJPs, showing that algebraic dual reformulations attenuate sensitivity creates a tight. These steps convince a logical loop.

Critical weaknesses
  1. The models/benchmarks use are too small to support their claims. The authors did a lot of experiments, however, they are just restricted to small models and datasets. It is hard to say whether the finding in this paper is meaningful for modern LLMs. To my best knowledge, LLMs rarely run explicit cholesky gram solves in attention blocks due to the bottleneck of GPU memory.

  2. The authors over-engineer NLA basics in this paper. The "low-precision Cholesky autodiff is numerically shaky" is textbook knowledge. The paper used pages and a cluttered notation scheme to prove that, which may distract readers and cover the real contributions.

Questions for authors
  1. Does the proposed precision sensitivity happens similarly in current scale-up architectures, such as Mamba and RWKV? They replace explicit cholesky solves with chunkwise parallel scans or stateful recurrent solves.

  2. Have you tried that in a very long training (I will say over 50k steps), does a <10% AdamW update discrepancy will really degrade the final performance? I guess maybe the momentum buffers can smooth out such noise?

1 comment
AuthorsOpenReview ↗

Response to Reviewer F2Te

Thank you for your careful assessment of the experimental controls and for raising the question about AdamW’s momentum buffers.

We agree that the numerical sensitivity of Cholesky and its derivatives is well known. The paper spent too much space and notation on that point. The finding that changed our own precision choice was what happened when we transferred the ridge schedule. On WikiText, the setting with FP64 only for Gram construction (q100) finished all five runs when it learned its own ridge, with task loss similar to the FP64 reference. When q100 replayed the same-seed q111 ridge schedule from the start of training, four of five runs failed. Moving Cholesky to FP64 allowed all five to finish. We will give that result more prominence in the next version and use descriptive stage names in the main text.

Does momentum smooth out the discrepancy? In the ridge classifier, yes, in most of the matched comparisons we measured. With the saved AdamW state restored, the update discrepancy was smaller than the gradient discrepancy in 355 of 360 comparisons. For the same gradients, a first step from zero moments gave a larger discrepancy in all 360. Your suggestion about optimizer history fits this model well.

In SKA, the Table 3 updates already include the restored AdamW state. The median update discrepancy exceeded the median gradient discrepancy at both checkpoints: 6.14% versus 5.95% at step 100, and 14.44% versus 12.12% at step 1,000. All nine early update discrepancies were below all nine later values. The pooled median of 9.88% obscures that separation. Over the same interval, the median logit discrepancy rose from 0.83% to 1.37%.

Our longest runs ended at step 5,000. By then q110 and q111 had similar WikiText task cross-entropies under the same evaluation solver (2.38697 ± 0.00135 and 2.38716 ± 0.00098, mean ± sample SD over five seeds, Table 6). We did not train to 50,000 steps, so whether the update differences eventually affect loss remains open.

Scale and architecture. We have not tested Mamba or RWKV, and neither uses the regularized Cholesky solve studied here in the same way. A closer example is MesaNet, which uses conjugate gradients to solve a regularized least-squares problem at each token and was trained at scales up to 1B parameters. Tumma et al. also report instability in their own MesaNet training runs. This gives a concrete larger model in which to test update agreement, though its solver would need its own precision measurements. MesaNet, ICLR 2026 · Preconditioned DeltaNet, Appendix E.5

Reviewer_XdvaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how numerical precision choices affect neural network training, particularly whether similar forward predictions imply equivalent learning behavior. The experiments are done on discrepancies in predictions, gradients, optimizer updates, and post-training conditioning across different FP32/FP64 configurations. The results suggest that output similarity does not necessarily guarantee equivalent parameter updates, while learned regularization.

Critical strengths

The paper raises a good question about whether numerical precision choices that preserve similar forward predictions also preserve the underlying learning dynamics. This distinction is very relevant as mixed-precision computation becomes common in training large models.

Critical weaknesses
  1. The manuscript is extremely difficult to follow due to its dense-yet-fragmented presentation and confusing narrative. Several experiments involving different precision routes, optimizer configurations, datasets, and evaluation protocols are introduced without a clear structure connecting them to the central research question. The distinction between local update discrepancies, endpoint conditioning, and ridge interventions is particularly difficult to follow. The paper would benefit significantly from a focused presentation that clearly mentions the state of the art and its limitations, the authors' hypotheses, research contributions, experimental design, and key findings.

  2. While the paper demonstrates that similar forward predictions can produce different gradients and optimizer updates, it remains unclear whether these discrepancies translate into meaningful differences in downstream task performance. In Table 6, the final cross-entropy values across precision routes are largely similar, despite substantial differences in some conditioning statistics. Moreover, the authors acknowledge that the observed differences do not establish statistically significant task improvements or degradation. Consequently, the practical implications of the reported numerical discrepancies remain insufficiently demonstrated.

  3. The numerical sensitivity of matrix solvers and their derivatives is already well established in the literature. Although the paper extends this analysis to optimizer updates and learned conditioning statistics, it is unclear whether these observations constitute a sufficiently substantial conceptual or methodological research advance. Also, the paper does not provide a systematic strategy for selecting precision configurations based on numerical stability, computational cost, and task performance. Hence the findings offer limited actionable guidance for practitioners.

Questions for authors

Besides my concerns stated in the Weakness section, I would appreciate some clarity on the following questions:

  1. To what extent do the observed precision sensitivities generalize to larger models and commonly used mixed-precision training settings such as FP16 or BF16?

  2. Table 7 shows substantially smaller numerical discrepancies for the dual formulation compared to the primal formulation. Does this behavior persist across different ridge initialization regimes and matrix dimensions, or is it specific to the evaluated configuration?

  3. Table 3 reports relatively large gradient and update discrepancies despite cosine similarities close to 1. Could the authors provide a more detailed analysis of whether these differences are concentrated in particular parameter groups or primarily reflect changes in gradient magnitudes?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer Xdva

Thank you for asking how the result depends on matrix size, where the update difference is concentrated, and how far the findings generalize.

We agree that the submitted paper makes the reader work too hard to connect the experiments. Our general response gives the precision setting we would use on WikiText and the completion, loss and CPU timing results behind that choice. In the next version, we will put that comparison first and give the evaluation cohorts together before the results.

Larger models and lower precision. Our experiments cover FP32 and FP64 in SKA and the ridge classifier. We have not measured the effect sizes in larger models or with FP16 or BF16. The comparison we would carry over is to restore the same weights, batch and optimizer state and check the actual update when one solver stage changes precision.

Primal versus dual. We ran two follow-ups. First, at the original Table 7 classifier checkpoint, we held the 64 features, learned ridge and query examples fixed and varied the number of support examples. For q110, the ratio of dual to primal AdamW update discrepancy was 0.073 with 25 supports, 0.47 with 50, and 6.07 with 75. Each formulation was compared with its own FP64 reference. The dual gave the closer update at 25 and 50 supports. At 75, the primal gave the closer update.

At 25 supports, Table 7 gives a ratio of 0.113 on macOS. The new sweep ran on Linux. We verified that the checkpoint and the ordered support and query IDs matched at that size. We also varied the evaluation ridge from 0.0001 to 0.01. Both mixed settings favored the smaller system in every comparison that passed the FP64 reference check. We excluded the setting with 75 supports and ridge 0.0001 because the two FP64 reference gradients exceeded the agreement tolerance. This dimension sweep used one trained checkpoint.

Second, we kept the original dimensions (25 supports, 64 features) and trained the classifier with the FP64 primal solver from four ridge initializations, with five seeds per initialization. At step 5,000, the dual gave a closer update for both mixed-precision settings in all 20 trained models.

Update direction and parameter groups. Across the 18 q110–q111 pairs in Table 3, the median angle between updates was 5.64°, while the median absolute relative difference in their norms was 0.53%. For gradients, the corresponding values were 4.94° and 0.76%. This also explains how a cosine close to one can coexist with a 10% discrepancy. For two equal-norm vectors separated by 5.64°, the relative distance is 2 sin(5.64°/2), or about 9.8%.

The saved Table 3 measurements give whole-model summaries, so we used the separate 29 states from Table 2 to locate the difference within the model. For q110 versus q111, the key projection accounted for 79.0% of the squared gradient difference and 80.1% of the squared update difference, averaging the shares over states. The downstream readout accounted for 3.7% and 0.85%, respectively. Log ridge had the largest relative gradient discrepancy among the measured groups in every state, but contributed only about 0.01% of the squared whole-model gradient difference.

Reviewer_pccqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates numerical precision placement in differentiable Gram matrix solvers, focusing on whether close agreement in forward predictions implies equivalence in backward gradients and parameter updates.

The authors partition the solver into three computational stages—statistic construction, Cholesky factorization, and solution-side computation—assigning either FP32 or FP64 precision to each stage. Using a controlled Spectral Koopman Attention (SKA) model and an independent episodic ridge regression classifier, they compare forward predictions, full-model gradients, optimizer updates, and separately trained model endpoints across different precision routes.

Across 18 matched SKA validation comparisons from three training seeds, comparing the mixed-precision route q_100 against the higher-precision solver reference route q_111 yields median relative discrepancies of 0.962% in logits, 8.61% in full-model gradients, and 9.88% in restored AdamW updates, each normalized by its corresponding q_111 reference norm.

These results demonstrate that measurable discrepancies in gradients and parameter updates can persist even when forward outputs are close. Furthermore, local update sensitivity is influenced by specific computational stages and algebraically equivalent dual reformulations (evaluated at a single fixed checkpoint), whereas in the tested classifier setting, post-training conditioning separation strongly depends on the training-time ridge regularization policy. The separately trained endpoints exhibit regime-dependent behavior, and the experiments do not demonstrate that improvements in numerical conditioning lead to task performance gains.

Overall, the primary contribution of this paper is the insight that forward prediction similarity does not adequately reflect discrepancies in actual optimizer updates, while demonstrating that stagewise precision and solver implementation choices shape local update divergences and that training-time regularization policies strongly affect post-training conditioning separation.

Critical strengths

The paper decomposes the differentiable Gram matrix solver into three computational stages, namely statistic construction, Cholesky factorization, and solution-side computation, and systematically assigns FP32 or FP64 precision to each stage to construct controlled comparison routes.

At fixed model states, the study combines matched route comparisons (measuring discrepancies in forward outputs, full-model gradients, and restored AdamW updates) with custom backward-pass (VJP) replacement diagnostics and primal versus algebraically equivalent dual reformulation comparisons. At the training-endpoint level, it further pairs separately trained endpoint evaluations with targeted ridge regularization interventions.

Within the scope of each experimental setup, the empirical observations are directly supported by the controlled comparisons.

Critical weaknesses
  • The Conceptual Increment Beyond Established Numerical Sensitivity Needs Further Clarification. From a numerical analysis perspective, the fact that close agreement in forward function evaluations does not imply closeness in derivatives is a known property, and the authors note in §2 and §5 that the numerical sensitivity of linear solvers and their derivatives has been examined in prior literature. While §4.2–4.3 extend this analysis by quantifying discrepancies in full-model gradients and restored AdamW updates across solver stages, the paper does not fully clarify what new training insights this quantification provides beyond existing knowledge of derivative sensitivity. Although the matched optimizer-update measurements and common-route endpoint evaluations provide empirical information beyond the general observation of derivative sensitivity, the paper offers limited guidance on when the measured discrepancies are consequential enough to change a practical precision choice. This limits the practical significance of the contribution in my assessment.

  • Insufficient Mechanistic Links Connecting the Separate Empirical Observations. The paper appropriately separates its local and endpoint findings and does not claim a demonstrated causal chain across training or a task-performance benefit. My concern is therefore about the explanatory reach and significance of the contribution, rather than the validity of its stated empirical findings. Specifically, although the paper conducts targeted diagnostic and interventional analyses across multiple setups, the explanatory connections linking these individual findings remain underdeveloped.

Questions for authors

Substantive Training Insights Beyond Known Derivative Sensitivity (§2, §4.2–4.3, §5): Given that the numerical sensitivity of linear solvers and their derivatives is established in prior literature, what previously unknown, substantive insights into model training do the quantified discrepancies in full-model gradients and restored AdamW updates establish, and how can they inform practical precision or training decisions?

Mechanistic Links Between Local Discrepancies and Training Endpoints (§4.2–4.6): The paper separately reports local update discrepancies at fixed states (§4.2–4.3; §4.4, Table 7), differences across separately trained endpoints (§4.4, Table 6), and regularization-dependent conditioning separation (§4.5–4.6). Could the authors address two related questions: (a) Do the existing analyses support any connection between the local update discrepancies and the endpoint differences? If not, please clarify what conclusions about training outcomes can and cannot be drawn from the local diagnostics; and (b) What practical training significance do the observed conditioning differences hold without established task-performance benefits (§4.6, Table 8)?

Minor:

Generalizability Beyond Strictly Controlled Architectures (§3.2): Given the tightly controlled experimental setups (e.g., the simplified SKA architecture in §3.2), to what extent do the authors expect these findings and sensitivity patterns to hold when key architectural or experimental constraints are relaxed?

Presentation and Experimental Organization: While the technical reporting is detailed, the experimental protocols, evaluation cohorts (e.g., the 9-unit, 18-unit, 20-batch, 29-unit, and 36-unit setups), and extensive cross-references to Appendices A–C are somewhat fragmented across sections, making the main text dense to follow. Streamlining the presentation of these cohorts would improve readability.

1 comment
AuthorsOpenReview ↗

Response to Reviewer pccq

Thank you for distinguishing between whether the experiments support our findings and whether those findings matter for training.

(a) The learned ridge gives us a partial connection between the local measurements and the endpoints. Across the 29 matched SKA states in Table 2, the median absolute gradient on log ridge was 1.34 × 10⁻³ with an FP32 solution side (q110) and 5.37 × 10⁻⁷ with the FP64 solver (q111). Log ridge had the largest relative gradient discrepancy among the parameter groups we measured in every state. At the same time, it accounted for only about 0.01% of the squared whole-model gradient difference. These observations are compatible because the reference gradient on log ridge was itself very small.

The separately trained WikiText models also differed in their learned ridge. What we cannot establish is how the individual steps produced that difference. On WikiText, the signed differences in the AdamW step on log ridge went both ways. The matched comparisons tell us which solver stage changes an update and by how much at a given state. They do not show a consistent drift toward the larger final ridge.

(b) At the endpoints we could test the role of ridge directly. Keeping every other weight fixed and substituting the same-seed q111 ridge closed a median 98.0% of q110’s absolute gap in log condition number on WikiText (Figure 1). The practical consequence showed up during training: with FP32 Cholesky, all five runs finished when they learned their own ridge, one finished when replaying the q111 schedule, and none finished with ridge fixed at 0.001.

The classifier answers a different part of your question. Sharing each seed’s ridge schedule across precision settings reduced the spread in final conditioning by 97.4%. In a separate comparison, a prescribed schedule lowered the condition number in all seven seeds, while held-out loss fell in only four (paired p = 0.375, Table 8). The conditioning measurements tell us how strongly the ridge policy affected the trained systems. We would still use held-out loss to choose between these classifiers.

One clarification about Table 3: the 18 pairs compare q110 with q111. Both perform Gram construction and Cholesky factorization in FP64 and differ only on the solution side. Our SKA block has no residual bypass or MLP. A residual branch would provide another path for the gradient, and we have not measured how much that would change the whole-model discrepancy.

We agree that the comparison groups take too much work to keep track of. In the next version, we will introduce them in one table that gives the checkpoint, batches or episodes, and number of training seeds for each experiment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviewers and Area Chair

We thank all three reviewers and the area chair for their careful reading of our paper.

Several reviewers asked what these measurements add to the known sensitivity of solver derivatives. For the WikiText model we studied, we would compute the Gram matrix and its Cholesky factorization in FP64 and keep the solution side in FP32 (q110). The clearest reason came from testing whether a ridge schedule learned at one precision could be used at another.

We compared three ways of setting the ridge during training. Each run either learned its own ridge, replayed from the start the ridge trajectory learned by the same seed with the FP64 solver (q111), or kept the ridge fixed at 0.001. The entries below are completed runs out of five (Tables 6 and 27).

Solver stages in FP64Learned ridgeReplayed q111 ridgeFixed ridge (0.001)
Gram only (q100)5/51/50/5
Gram and Cholesky (q110)5/55/55/5
All three stages (q111)5/55/55/5

With a learned ridge, q100 finished every run and had the lowest observed mean task cross-entropy at step 5,000. The means were 2.38608 for q100, 2.38697 for q110 and 2.38716 for q111, all evaluated with the q111 solver. Yet both settings with an FP32 solution side learned a ridge 6.2–8.8 times as large as q111’s in every seed. The mean ridge values were 0.0216, 0.0223 and 0.0031, respectively. Judging q100 by completion and loss under its own learned ridge alone would have missed a practical limitation. Four of five runs failed when replaying the q111 schedule, and all five failed with fixed ridge. Every failure occurred at Cholesky.

The local update comparison points to the same stage. At a separate q111-trained checkpoint (seed 0, step 1,000, 20 batches on an A100), moving Cholesky to FP64 changed the mean logit discrepancy from 2.75% to 2.67%, while the mean Adam update discrepancy fell from 42.2% to 17.4% (Table 4). In the separate training runs with replayed or fixed ridge, moving that same stage to FP64 allowed all five seeds to finish under each policy (Table 27).

In our CPU benchmark, a forward and backward pass took 142.3 ms with q110 and 164.6 ms with q111, about 16% longer. Two runs with the same stage precisions took 139.8 and 147.7 ms, so we treat the timing comparison as approximate. We have no GPU timings. For this workload, q110 finished under every ridge policy we tested, and extending FP64 to the solution side gave no observed task-loss benefit at 5,000 steps.

The original submission PDF does not report the per-route learned-ridge values, the CPU timings or the follow-up analyses described in these replies. We will include those results in the next version and introduce the evaluation cohorts in one table before presenting the results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.