acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing How Optimisers Shape Learning through Operator Projections

Abstract

Loss and accuracy track progress, but do not distinguish what an optimiser can change from the direction it chooses or the structure that changes during learning. We introduce an operator-projection framework for bias-free piecewise-linear networks. At each training state, it separates locally attainable operator change, steering within that space, and parameter motion invisible to the sampled operators to first order. Deep linear and MNIST-1D experiments use this reference to examine existing optimisers and a reference-following control. SGD exhibits low alignment despite nearly complete estimated reachability. Increasing reference alignment does not monotonically improve test accuracy, and the control reduces input-dependent gating even when activation slopes remain nonzero. Most strikingly, its pronounced decline in raw operator diversity after fitting nearly disappears after removing the common-logit component that softmax ignores. Across three activations, this component accounts for – of the squared net operator displacement between the measured snapshots. Finite-step checks further distinguish local steering from realised operator change when gates vary. The framework provides a diagnostic instrument for interpreting optimiser behaviour and the evolution of the learned function, rather than a proposed optimisation algorithm.

Primary area:
learning theory
Keywords:
implicit bias, implicit regularization, training dynamics, deep linear networks, spectral dynamics, singular value dynamics, Jacobian, optimization, low-rank bias, ReLU networks, theory of deep learning, gradient descent

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_yEYnOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a framework for analysing optimiser updates in bias-free piecewise-linear networks, where each input has a local linear map P(x)P(x) such that f(x)=P(x)xf(x)=P(x)x. The authors define a gradient-based target change for these maps and project it onto the changes that the current network can realise to first order. The minimum-norm weight update producing this projection serves as a reference. The authors use this reference to separate what the network can change from the direction an optimiser chooses, while identifying parameter motion that leaves the sampled maps unchanged to first order.

Experiments on deep linear regression and MNIST-1D show that SGD can move these maps in a direction substantially different from the attainable target, even when weight changes could realise nearly all of the target change. A reference-following training rule that approximates the reference step shows that following the reference more closely does not consistently improve test accuracy. The authors also find that this rule's apparent decline in the diversity of these maps across inputs after fitting largely disappears after removing a common-logit component that does not affect softmax predictions.

Critical strengths
  • The paper offers an interesting and distinctive diagnostic perspective on optimisers: it uses projections in the space of these local linear maps to separate what the current network can locally change from how an optimiser directs those changes.
  • The mathematical construction in Sections 2–3 is explicit; Proposition 1 and the worked example in Section 3.1 make clear what each quantity measures.
Critical weaknesses
  1. The justification and diagnostic meaning of the reference remain unclear.

The reference uses an equally weighted Frobenius distance to the per-sample targets, but the paper does not justify why this provides an informative baseline for optimiser behaviour. Samples differ in noise, difficulty and redundancy; deviation from the reference may reflect how an optimiser prioritises these samples, yet the aggregate diagnostics do not explain such trade-offs. The diagnostics also retain prediction-invariant common-logit changes (Section 4.4), and closer alignment does not consistently improve accuracy.

The paper has not established why this reference is appropriate for diagnosing optimiser behaviour, nor what the measured deviations reveal about learning. These are central questions about the contribution, not minor points requiring clarification. To establish the framework's value, the paper needs to show that the measured deviations admit a meaningful interpretation—for example, in terms of sample priorities or properties of the learned function—or yield a substantive finding about optimisers beyond the geometric relationships already implied by the construction. Without such evidence, I am unable to assess what substantive understanding the framework adds.

  1. The main diagnostic comparisons do not sufficiently characterise the learning process.

The main comparisons use selected checkpoints around fitting, leaving unclear whether the observations persist across training. REF's alignment already changes across later phases (Appendix D.3), but comparable profiles are missing for SGD and Adam, and for reachability and the kernel fraction. Each optimiser is also measured on its own trajectory, mixing the update rule with the state it reaches. Measurements across training (before, at and after fitting), supplemented by comparisons at common weights with specified optimiser states, would clarify these distinctions.

  1. The research question and main findings are difficult to extract.

The paper is difficult to follow: the abstract and introduction introduce many technical terms (e.g., "steering", "reference", "reference-following control") before making the research question clear. Dense numerical detail, frequent qualifications and evidence scattered across the appendix further obscure the main findings. The motivation and central conclusions need to be stated more directly.

Questions for authors

The main requests are in the Weaknesses section; I have no further questions at this stage.

5 comments

We thank the reviewer for their detailed comments and for recognising the diagnostic perspective and explicit mathematical construction. We agree that the paper should make clearer both the meaning of the reference and the conclusions supported by the comparisons.

The justification and diagnostic meaning of the reference remain unclear.

The reference asks which part of the training loss's operator-gradient request the current network can realise. Under the stated batch-average Frobenius inner product, the target is bi=−ηgixi⊤b_i=-\eta g_i x_i^\top, where gig_i is the loss gradient with respect to the output for sample ii. On the same batch, SGD gives uSGD∝M∗bu_{\mathrm{SGD}} \propto M^*b, hence MuSGD∝MM∗bMu_{\mathrm{SGD}} \propto MM^*b. The reference instead gives uref=M+bu_{\mathrm{ref}}=M^+b and Muref=Πrange⁡(M)bMu_{\mathrm{ref}}=\Pi_{\operatorname{range}(M)}b. The comparison therefore separates the component excluded by local feasibility from the reweighting of attainable directions. This is a specified geometric baseline, not a claim that its metric uniquely characterises learning or that following it should maximise test accuracy.

Equal weighting in the metric also does not mean identical contributions from every sample: the targets depend on the sample-specific loss gradients and inputs. Nevertheless, we agree that this does not resolve questions about noise or redundancy, and an aggregate cosine does not identify which samples an update favours. A different metric would define a different reference. Our present interpretation is relative to the declared metric and loss.

We evaluated the per-sample first-order decreases di(u)d_i(u) at common weights. Each seed of the selected SGD, Adam and REF runs was replayed exactly to checkpoints before, around and after fitting and at the end. At each checkpoint, the SGD increment, the truncated REF increment and, on Adam trajectories, Adam's increment with its actual state were evaluated on the same probe batch. Examples are grouped by their loss quintile at the checkpoint, before the update, and we report each quintile's share of the batch's first-order decrease; increments keep their actual magnitude, but the shares do not depend on it. All three rules concentrate the first-order decrease on the highest-loss fifth of the probe, and their profiles at common weights are close: at Adam's states the three are nearly identical, and at SGD's states REF places slightly more of the decrease on the highest-loss quintile than SGD. These measurements therefore do not show that the rules prioritise different samples; under the declared loss, the first-order decrease of every rule follows where the loss gradients are concentrated.

There are empirical findings beyond the projection identities. Table 1 shows that SGD has alignment 0.204–0.253 despite estimated reachability close to one. Near-complete reachability is not guaranteed by the construction: the shared weights can prevent the sampled operators from meeting their targets jointly. At these states, the measurements distinguish that possibility from reweighting of a largely attainable target. The CReLU truncation sweep then shows that higher reference alignment does not monotonically improve prediction: from k=20k=20 to k=200k=200 LSQR iterations per update (separately trained runs), alignment increases from 0.864 to 0.992 while accuracy decreases from 0.602 to 0.558, with all settings fitting the training set (Table 9). This supports separating local reference following from generalisation within the tested control family. REF uses a normalised target and a truncated solve, so these are observations about the implemented control rather than an exact-reference training trajectory.

The common-logit result concerns the properties of the learned function. Table 14 evaluates raw and class-centred operators on the same inputs and snapshots. For ReLU REF, raw diversity falls from 191.5 to 109.8, while class-centred diversity changes from 221.0 to 218.8; the same distinction occurs for CReLU and leaky ReLU. The common-logit component accounts for 86–98% of REF's squared net operator displacement between these snapshots across the three activations, against 1% for SGD. Thus the pronounced raw diversity decline does not establish a comparable simplification of the operator governing class differences. This does not imply that the class-centred operators themselves are unchanged.

We agree that removing this component from structural measurements does not resolve its role in the update diagnostics. The reported alignment and kernel fractions use the full-logit map MM. They should not be interpreted as invariant under replacing MM by the class-centred map QMQM, where Q=I−11⊤/CQ=I-\mathbf1\mathbf1^\top/C and CC is the number of classes. We therefore recomputed them under QMQM.

At the same checkpoints and probe inputs, with the reference solved separately as Mc+bM_c^+b, reachability, alignment and the comparisons between rules are essentially unchanged under QMQM: the mean change in alignment is below 0.03 for every rule, phase and architecture. Kernel fractions can only grow, since ker⁡M⊆ker⁡QM\ker M\subseteq\ker QM. SGD's remains zero, because Qb=bQb=b places its increment in the range of (QM)∗(QM)^*; Adam's rises by about 0.02; the truncated REF increment acquires a kernel fraction of about 0.03–0.26, the part of its motion that changes only the common logit. The full-logit comparisons therefore persist under the class-centred map, and the class-centred map additionally isolates REF's common-logit motion.

The main diagnostic comparisons do not sufficiently characterise the learning process.

We agree that the fitting-phase comparison does not establish persistence throughout training. Appendix D.3 already shows that REF's alignment changes with phase: on CReLU it decreases from 0.931 around fitting to 0.710 in the last quarter. The existing tables focus on selected phases and replayed fitting-snapshot states. We have therefore examined the recorded alignment profiles; we have also replayed each run exactly to checkpoints before, around and after fitting and at the end, to measure reachability and kernel fractions.

The alignment probes, recorded every 2000 updates on a fixed probe with each rule's live optimiser state, show different temporal patterns. For Adam, the candidate increment uses its actual moments and step counter. SGD's estimated alignment is approximately stable at the sampled checkpoints, whereas REF's is highest around fitting and declines afterwards. Adam's estimated alignment is already low before fitting and remains low at later sampled checkpoints. In particular, the early CReLU measurements show low alignment while all their reference solves meet the stated residual threshold. Many later Adam solves fail that threshold at the in-run budget. Re-solving them with 30000 iterations changes Adam's alignment by at most 0.012 and confirms the pattern; on CReLU every higher-budget solve passes, while on ReLU half remain slightly above the threshold. Passing the residual threshold is an operational numerical check, not a general certificate of projection accuracy. We will report the profiles with seed variation and identify the probes that fail the threshold.

The replay adds kernel fractions and reachability over the same phases. SGD's and REF's increments have kernel fractions near zero at every phase, whereas Adam's remain large throughout (above 0.4 at every checkpoint on both architectures). Reachability stays close to one along the SGD and REF trajectories and along Adam's CReLU trajectory. Along Adam's ReLU trajectory it falls from around fitting onward, with converged lower bounds around 0.85, so part of the target becomes locally unattainable at those states; this does not account for Adam's low alignment, which is far below what is attainable there (see the common-weight comparison below).

We also agree that comparing each rule on its own trajectory combines the update rule with the state it reaches. The handover experiment addresses dependence on training history, but it does not isolate candidate updates at common weights. Likewise, the matched-loss comparison in Table 8 controls the measured training-loss level, not the weights or optimiser histories. These results therefore do not identify an optimiser-intrinsic cause of the performance differences.

The same probes also score the SGD direction at Adam's own weights, providing a first comparison at a common state. On ReLU and leaky ReLU, SGD has higher estimated alignment than Adam at those weights. This is a difference between the candidate increments evaluated at identical weights and on the same probe. On CReLU, both estimated alignments are low: low alignment at these states is therefore not specific to Adam's increment. This does not imply that the two updates point in the same direction or quantify the relative contributions of state and update rule. These comparisons use the in-run solver budget; the systematic replay below re-solves them with a higher budget.

The systematic comparison confirms this on ReLU and CReLU. Each Adam run was replayed exactly, with its actual moments and counter, to checkpoints before, around and after fitting and at the end; at each checkpoint, the SGD, Adam and truncated REF increments were scored on the same probe batch against a 30000-iteration reference. On ReLU, SGD's increment is better aligned than Adam's at 17 of 18 checkpoints, by a factor of two to four on average. On CReLU, the two are equally aligned at every phase. At the same Adam states, the truncated REF increment reaches much higher alignment on both architectures, so better-aligned attainable increments exist at these states. SGD and REF are nevertheless less aligned at Adam's CReLU states than on their own trajectories. On ReLU, Adam's increment is therefore less aligned than SGD's at identical states; on CReLU, low alignment at Adam's states is shared by SGD. These measurements do not quantify the relative contributions of state and update rule. Leaky ReLU is covered only by the in-run probes.

The research question and main findings are difficult to extract.

We will state the question before introducing the diagnostics: how can we distinguish what the network can locally change from the direction selected by an optimiser, and interpret the resulting structural changes in relation to prediction? The experimental discussion will foreground the findings above, with each intervention defined before its result. We will also narrow the interpretation of the supporting controls. Smoothing REF's increments does not reproduce Adam in the tested settings, but this does not exclude a role for momentum within Adam. Handover effects depend on architecture, and matching training loss does not match training history. These experiments do not establish a complete explanation of Adam's advantage. The contribution is a diagnosis of local update geometry and learned structure, with its scope stated explicitly.

Reviewer_PgrPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper provides a diagnostic framework that disentangles parameter update steps into reachability (how much of the ideal update for every datapoint is achievable), operator alignment (how closely the optimizer follows that reachable direction), and invisible kernel motion for a bias-free piecewise-linear network. Using this framework, the authors study SGD, Adam and a reference-following control, primarily on MNIST-1D. For SGD they find that even though reachability is high, the updates have low alignment and artificially increasing the alignment does not monotonically improve test accuracy.

Critical strengths

The paper introduces a novel diagnostic framework to understand how optimizer updates move with regard to operator properties and “ideal” reference points, which is a useful perspective for analyzing and gaining novel insights into optimization dynamics.

Critical weaknesses
  • The authors primarily study SGD and Adam on small, bias-free, unnormalized networks, which is a simplified setup compared to standard practice. Importantly, AdamW (with decoupled weight decay) is commonly used over Adam and exhibits very different training dynamics. Additionally, modern architectures include normalization layers, which make networks agnostic to weight scaling. Discussing or evaluating these adaptations would be important for practical relevance, as the observed behavior may not translate to standard setups due to their different training dynamics.
  • The experimental section is far too dense and cluttered with noisy experimental details, making the main body not self-contained. Examples are:

Handover experiments test whether an optimiser’s effect depends on the state it inherits. With 6000 updates of each rule, Adam followed by REF scores .606 on ReLU against .560 for the reverse order, higher in all three paired seeds (differences .034, .073, .031).

The handover experiment is not clearly defined and results is just a list of numbers across seeds, rather than aggregation with standard deviation.

Smoothing REF’s weight increments lowers alignment and gives nonpositive mean accuracy changes in all twelve tested settings (Appendix D.14)

The experimental setup refers to Appendix section, but never defines what the twelve tested settings are in the main body.

In its current state, it is almost impossible to follow the exact experimental design and verify the results beyond the authors' high-level summary statements.

Questions for authors

How would the theory and observations translate to networks including normalization layers or state-of-the-art optimizers such as AdamW?

2 comments

We thank the reviewer for recognising the usefulness of the diagnostic perspective. We agree that the empirical scope and the definitions of the supporting experiments should be clearer.

How would the theory and observations translate to networks including normalization layers or state-of-the-art optimizers such as AdamW?

The bias-free piecewise-linear setting makes the relation between the operator and the represented function explicit: fW(x)=PW(x)xf_W(x)=P_W(x)x, with PW(x)P_W(x) equal to the input Jacobian away from gate boundaries. This allows us to study how a parameter update changes the full local input–output map. The setting is useful for controlled analysis, but the empirical conclusions remain conditional on these models and tasks. The additional comparisons with heavy ball, Muon, Shampoo, SOAP and K-FAC in Appendix D.17 broaden the optimiser comparison within this setting; they do not establish transfer to normalised architectures. Likewise, the CIFAR-10 experiment in Appendix D.19 is a feasibility check, not a replication of the MNIST-1D findings.

AdamW does not require changing the diagnostic construction for the current architectures. The diagnosis takes a candidate weight increment as input, so it can include AdamW's decoupled decay term. At a fixed state, linearity gives M(uadaptive+udecay)=Muadaptive+MudecayM(u_{\mathrm{adaptive}}+u_{\mathrm{decay}})=Mu_{\mathrm{adaptive}}+Mu_{\mathrm{decay}}. The total increment can therefore be evaluated, and the two contributions can also be examined separately. Whether the empirical profiles and accuracy comparisons persist under weight decay is a distinct question, which the existing Adam results do not settle.

For these networks, uniform decay of all weight matrices has a simple form in our geometry: (MW)i=L PW(xi)(MW)_i=L\,P_W(x_i). Thus the decay increment d=−γλWd=-\gamma\lambda W, where γ\gamma is the learning rate and λ\lambda the decay coefficient, produces the first-order operator change (Md)i=−γλL PW(xi)(Md)_i=-\gamma\lambda L\,P_W(x_i). Applied alone, decay preserves the gates when 1−γλ>01-\gamma\lambda>0; the complete AdamW update need not do so. For softmax cross-entropy, if every example in the probe batch is correctly classified, the decay component has nonpositive inner product with the operator-gradient target.

We evaluated this at Adam's fitting states, recovered by exact replay of the five principal seeds, by adding d=−γλWd=-\gamma\lambda W to Adam's increment for λ∈{0.01,0.1,1}\lambda\in\{0.01,0.1,1\}. The estimated kernel norm fraction of the decay increment exceeds 0.90, and its operator component points against the target in these measurements.At λ=1\lambda=1, adding decay lowers the estimated alignment and raises the total kernel fraction in every seed on both ReLU and CReLU; at smaller λ\lambda it does so in the seeds where the decay increment is larger than Adam's own, which is the case in eight of the ten seeds at λ=0.01\lambda=0.01. These are empirical observations at the tested states. A large kernel fraction describes the decomposition of the parameter increment; it does not imply that its operator change is small relative to Adam's, since Md=−γλLPMd=-\gamma\lambda L P. This comparison evaluates a hypothetical increment at fixed weights and optimiser state. It does not establish the behaviour of a full AdamW training trajectory.

We also trained AdamW at the principal ReLU and CReLU settings, with the same five data seeds and 12000 updates, selecting the learning rate in {10−3,3⋅10−3,10−2,3⋅10−2}\{10^{-3},3\cdot10^{-3},10^{-2},3\cdot10^{-2}\} and the decay in {0,0.01,0.1,1}\{0,0.01,0.1,1\} jointly on validation, where λ=0\lambda=0 are the Adam runs. Validation selects a nonzero decay on both architectures. On ReLU, AdamW's test accuracy is close to Adam's (paired difference 0.002±0.0180.002\pm0.018 over five seeds). On CReLU it is higher on average (0.094±0.1040.094\pm0.104 over the four seeds that fit under both rules), but the difference varies strongly across seeds. At AdamW's own fitting states, recovered by exact replay and scored against converged 30000-iteration reference solves, its increment is markedly better aligned than Adam's at Adam's states, while its decay part again points against the target. Training with decay therefore reaches states where the full increment follows the reference more closely, whereas adding the same term at fixed Adam states lowers alignment. The diagnostic separates the effect of the decay term on an increment from its effect on the trajectory; it does not identify the cause of the accuracy difference.

Normalisation requires more care. The input Jacobian remains a possible object of analysis where it is defined, but the identity fW(x)=PW(x)xf_W(x)=P_W(x)x need not hold, and the operator target derived from that identity cannot simply be reused. The appropriate extension depends on the normalisation scheme and whether its statistics are fixed or batch-dependent. If an exact parameter-scale symmetry leaves the input Jacobians unchanged, its tangent direction belongs to the kernel of the parameter-to-Jacobian map. However, that kernel may contain other directions, so its norm fraction would not measure weight-scale motion alone. We will make this distinction explicit and limit the present empirical claims to the tested architectures.

LayerNorm and RMSNorm compute statistics from each input in both training and evaluation. For a bias-free linear layer followed directly by such a normalisation, positive rescaling of all its weights is an exact symmetry in the idealised ϵ=0\epsilon=0 formulation, where the normalising denominator is nonzero. With the fixed positive ϵ\epsilon used in standard implementations, this invariance is generally only approximate. The input Jacobian remains defined where the network is differentiable, but the identity fW(x)=PW(x)xf_W(x)=P_W(x)x need not hold. BatchNorm couples examples when it uses batch statistics. With fixed running statistics at evaluation, a network with piecewise-linear activations is instead affine within each region; if batch statistics are used at evaluation too, the batch dependence remains. These extensions are outside the present empirical claims.

The handover experiment is not clearly defined and results is just a list of numbers across seeds, rather than aggregation with standard deviation.

We agree that the main text should define the protocol and report an aggregate comparison. The handover experiment trains with one optimiser for the first ss updates and the other for the remaining 12000−s12000-s, with s∈{0,250,1000,3000,6000,12000}s\in\{0,250,1000,3000,6000,12000\}. Adam uses learning rate 0.0030.003; REF uses target size η=0.5\eta=0.5 and 50 LSQR iterations. When Adam starts second, its moments and bias-correction counter start at the handover. Accuracy is averaged over the last quarter of training, and the experiment uses three paired seeds (Appendix D.12).

With 6000 updates of each rule, the paired accuracy difference between Adam-then-REF and REF-then-Adam is 0.046±0.0230.046\pm0.023 on ReLU and −0.029±0.058-0.029\pm0.058 on CReLU, where the deviations are sample standard deviations of the paired differences over the three seeds. The effect is consistent across seeds on ReLU but not on CReLU. The revised Section 4.6 gives the protocol and these aggregate differences. The experiment establishes order dependence in the tested setting; it does not exclude inherited state as a contributor to other performance differences.

The experimental setup refers to Appendix section, but never defines what the twelve tested settings are in the main body.

The twelve settings are the product of two activations, ReLU and CReLU, two target sizes, η∈{0.5,1}\eta\in\{0.5,1\}, and three smoothing coefficients, β∈{0.5,0.9,0.99}\beta\in\{0.5,0.9,0.99\}. REF's parameter increments are replaced by ut=βut−1+(1−β)ΔWtREFu_t=\beta u_{t-1}+(1-\beta)\Delta W_t^{\mathrm{REF}}, using 50 LSQR iterations. Each setting is compared with its same-seed β=0\beta=0 control over three seeds. Table 20 reports the paired accuracy differences as mean and standard deviation. All twelve means are nonpositive, with the largest decrease 0.075±0.0270.075\pm0.027. This intervention does not reproduce Adam's accuracy advantage in the tested configurations; it does not rule out momentum's contribution within Adam. The revised Section 4.3 now gives these settings and defines the intervention in the main text.

Reviewer_uLidOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper decomposes weight updates into reachability, alignment, and kernel fraction metrics. The paper does not offer a strong reason for doing so, and does not connect these measures to important phenomena or performance metrics.

The paper also has major clarity issues to the point that I believe it is not in a state fit for publication. I could not understand the abstract. As one example, the abstract contains the sentence "Most strikingly, its pronounced decline in raw operator diversity after fitting nearly disappears after removing the common-logit component that softmax ignores" without defining operator diversity (raw or not), or the common-logit component, or why this is in fact striking. The abstract contains no explanation of why these metrics might be worth tracking, nor does the intro or discussion.

Critical strengths

The experiments appear to have been done correctly.

Critical weaknesses

Fatal clarity issues

Lack of discussion of significance of these results, in other words, why the reader should care about these metrics

Questions for authors

Why are the proposed metrics important? how do they shed light on open questions in the field, or link to performance?

1 comment

We thank the reviewer for their feedback. We agree that the abstract should explain the purpose of the diagnosis before introducing its technical terms. In particular, the sentence about operator diversity assumes definitions given later in the paper. We will replace this wording with a direct explanation of the finding and its significance.

Why are the proposed metrics important? how do they shed light on open questions in the field, or link to performance?

Our aim is to understand how an optimiser changes the function represented by a network. A loss curve does not distinguish a change that the current network cannot realise from an attainable change that the update rule does not follow. It also does not reveal parameter motion that leaves the sampled local operators unchanged to first order. The framework separates these effects. The example in Section 3.1 makes the distinction explicit: three updates have the same first-order loss decrease but differ in their operator direction or invisible parameter motion.

The experiments show why this distinction matters. In Table 1, SGD has estimated reachability close to one but operator alignment between 0.204 and 0.253. Thus its low alignment is not primarily explained by an inaccessible target at these measured states. It reflects the reweighting induced by the parameterisation: SGD produces MM∗bMM^*b, whereas the reference produces the attainable projection of bb. The algebra identifies the possible effects; the measurements establish which effect occurs in these networks.

The link to predictive performance is examined directly, rather than assumed. In the CReLU truncation sweep, all settings fit the training set. Increasing the solver budget from 20 to 200 raises alignment from 0.864 to 0.992, while test accuracy decreases from 0.602 to 0.558 (Section 4.3; Table 9). Following the local reference more closely therefore does not monotonically improve generalisation. The reference-following rule is an experimental control for studying this relationship, not a proposed replacement for existing optimisers or an assertion that the reference is statistically optimal.

The structural result addresses a different ambiguity in interpreting learning. Adding the same quantity to every class logit leaves softmax probabilities unchanged. For ReLU REF, raw operator diversity decreases from 191.5 to 109.8 after fitting, whereas removing this common-logit component gives diversity 221.0 to 218.8 on the same inputs and snapshots (Section 4.4; Table 14). Across the three activations, the common-logit component accounts for 86–98% of the squared net operator displacement. The finding is that a pronounced decline in a raw structural statistic need not indicate a comparable simplification of the operator governing class differences. We do not infer that the class-centred operators themselves remain unchanged.

We will make these questions and findings explicit in the abstract and introduction, and organise the experimental discussion around them. We would also appreciate clarification of the soundness assessment. The review identifies presentation and significance concerns and notes that the experiments appear to have been performed correctly, but does not identify a specific error in the construction or evaluation. This would help us address any remaining technical concern directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.