- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a framework for analysing optimiser updates in bias-free piecewise-linear networks, where each input has a local linear map such that . The authors define a gradient-based target change for these maps and project it onto the changes that the current network can realise to first order. The minimum-norm weight update producing this projection serves as a reference. The authors use this reference to separate what the network can change from the direction an optimiser chooses, while identifying parameter motion that leaves the sampled maps unchanged to first order.
Experiments on deep linear regression and MNIST-1D show that SGD can move these maps in a direction substantially different from the attainable target, even when weight changes could realise nearly all of the target change. A reference-following training rule that approximates the reference step shows that following the reference more closely does not consistently improve test accuracy. The authors also find that this rule's apparent decline in the diversity of these maps across inputs after fitting largely disappears after removing a common-logit component that does not affect softmax predictions.
Critical strengths
- The paper offers an interesting and distinctive diagnostic perspective on optimisers: it uses projections in the space of these local linear maps to separate what the current network can locally change from how an optimiser directs those changes.
- The mathematical construction in Sections 2–3 is explicit; Proposition 1 and the worked example in Section 3.1 make clear what each quantity measures.
Critical weaknesses
- The justification and diagnostic meaning of the reference remain unclear.
The reference uses an equally weighted Frobenius distance to the per-sample targets, but the paper does not justify why this provides an informative baseline for optimiser behaviour. Samples differ in noise, difficulty and redundancy; deviation from the reference may reflect how an optimiser prioritises these samples, yet the aggregate diagnostics do not explain such trade-offs. The diagnostics also retain prediction-invariant common-logit changes (Section 4.4), and closer alignment does not consistently improve accuracy.
The paper has not established why this reference is appropriate for diagnosing optimiser behaviour, nor what the measured deviations reveal about learning. These are central questions about the contribution, not minor points requiring clarification. To establish the framework's value, the paper needs to show that the measured deviations admit a meaningful interpretation—for example, in terms of sample priorities or properties of the learned function—or yield a substantive finding about optimisers beyond the geometric relationships already implied by the construction. Without such evidence, I am unable to assess what substantive understanding the framework adds.
- The main diagnostic comparisons do not sufficiently characterise the learning process.
The main comparisons use selected checkpoints around fitting, leaving unclear whether the observations persist across training. REF's alignment already changes across later phases (Appendix D.3), but comparable profiles are missing for SGD and Adam, and for reachability and the kernel fraction. Each optimiser is also measured on its own trajectory, mixing the update rule with the state it reaches. Measurements across training (before, at and after fitting), supplemented by comparisons at common weights with specified optimiser states, would clarify these distinctions.
- The research question and main findings are difficult to extract.
The paper is difficult to follow: the abstract and introduction introduce many technical terms (e.g., "steering", "reference", "reference-following control") before making the research question clear. Dense numerical detail, frequent qualifications and evidence scattered across the appendix further obscure the main findings. The motivation and central conclusions need to be stated more directly.
Questions for authors
The main requests are in the Weaknesses section; I have no further questions at this stage.
5 comments
At the same checkpoints and probe inputs, with the reference solved separately as , reachability, alignment and the comparisons between rules are essentially unchanged under : the mean change in alignment is below 0.03 for every rule, phase and architecture. Kernel fractions can only grow, since . SGD's remains zero, because places its increment in the range of ; Adam's rises by about 0.02; the truncated REF increment acquires a kernel fraction of about 0.03–0.26, the part of its motion that changes only the common logit. The full-logit comparisons therefore persist under the class-centred map, and the class-centred map additionally isolates REF's common-logit motion.
The main diagnostic comparisons do not sufficiently characterise the learning process.
We agree that the fitting-phase comparison does not establish persistence throughout training. Appendix D.3 already shows that REF's alignment changes with phase: on CReLU it decreases from 0.931 around fitting to 0.710 in the last quarter. The existing tables focus on selected phases and replayed fitting-snapshot states. We have therefore examined the recorded alignment profiles; we have also replayed each run exactly to checkpoints before, around and after fitting and at the end, to measure reachability and kernel fractions.
The alignment probes, recorded every 2000 updates on a fixed probe with each rule's live optimiser state, show different temporal patterns. For Adam, the candidate increment uses its actual moments and step counter. SGD's estimated alignment is approximately stable at the sampled checkpoints, whereas REF's is highest around fitting and declines afterwards. Adam's estimated alignment is already low before fitting and remains low at later sampled checkpoints. In particular, the early CReLU measurements show low alignment while all their reference solves meet the stated residual threshold. Many later Adam solves fail that threshold at the in-run budget. Re-solving them with 30000 iterations changes Adam's alignment by at most 0.012 and confirms the pattern; on CReLU every higher-budget solve passes, while on ReLU half remain slightly above the threshold. Passing the residual threshold is an operational numerical check, not a general certificate of projection accuracy. We will report the profiles with seed variation and identify the probes that fail the threshold.
The replay adds kernel fractions and reachability over the same phases. SGD's and REF's increments have kernel fractions near zero at every phase, whereas Adam's remain large throughout (above 0.4 at every checkpoint on both architectures). Reachability stays close to one along the SGD and REF trajectories and along Adam's CReLU trajectory. Along Adam's ReLU trajectory it falls from around fitting onward, with converged lower bounds around 0.85, so part of the target becomes locally unattainable at those states; this does not account for Adam's low alignment, which is far below what is attainable there (see the common-weight comparison below).
We also agree that comparing each rule on its own trajectory combines the update rule with the state it reaches. The handover experiment addresses dependence on training history, but it does not isolate candidate updates at common weights. Likewise, the matched-loss comparison in Table 8 controls the measured training-loss level, not the weights or optimiser histories. These results therefore do not identify an optimiser-intrinsic cause of the performance differences.
The same probes also score the SGD direction at Adam's own weights, providing a first comparison at a common state. On ReLU and leaky ReLU, SGD has higher estimated alignment than Adam at those weights. This is a difference between the candidate increments evaluated at identical weights and on the same probe. On CReLU, both estimated alignments are low: low alignment at these states is therefore not specific to Adam's increment. This does not imply that the two updates point in the same direction or quantify the relative contributions of state and update rule. These comparisons use the in-run solver budget; the systematic replay below re-solves them with a higher budget.
The systematic comparison confirms this on ReLU and CReLU. Each Adam run was replayed exactly, with its actual moments and counter, to checkpoints before, around and after fitting and at the end; at each checkpoint, the SGD, Adam and truncated REF increments were scored on the same probe batch against a 30000-iteration reference. On ReLU, SGD's increment is better aligned than Adam's at 17 of 18 checkpoints, by a factor of two to four on average. On CReLU, the two are equally aligned at every phase. At the same Adam states, the truncated REF increment reaches much higher alignment on both architectures, so better-aligned attainable increments exist at these states. SGD and REF are nevertheless less aligned at Adam's CReLU states than on their own trajectories. On ReLU, Adam's increment is therefore less aligned than SGD's at identical states; on CReLU, low alignment at Adam's states is shared by SGD. These measurements do not quantify the relative contributions of state and update rule. Leaky ReLU is covered only by the in-run probes.
The research question and main findings are difficult to extract.
We will state the question before introducing the diagnostics: how can we distinguish what the network can locally change from the direction selected by an optimiser, and interpret the resulting structural changes in relation to prediction? The experimental discussion will foreground the findings above, with each intervention defined before its result. We will also narrow the interpretation of the supporting controls. Smoothing REF's increments does not reproduce Adam in the tested settings, but this does not exclude a role for momentum within Adam. Handover effects depend on architecture, and matching training loss does not match training history. These experiments do not establish a complete explanation of Adam's advantage. The contribution is a diagnosis of local update geometry and learned structure, with its scope stated explicitly.
We thank the reviewer for their detailed comments and for recognising the diagnostic perspective and explicit mathematical construction. We agree that the paper should make clearer both the meaning of the reference and the conclusions supported by the comparisons.
The reference asks which part of the training loss's operator-gradient request the current network can realise. Under the stated batch-average Frobenius inner product, the target is bi=−ηgixi⊤, where gi is the loss gradient with respect to the output for sample i. On the same batch, SGD gives uSGD∝M∗b, hence MuSGD∝MM∗b. The reference instead gives uref=M+b and Muref=Πrange(M)b. The comparison therefore separates the component excluded by local feasibility from the reweighting of attainable directions. This is a specified geometric baseline, not a claim that its metric uniquely characterises learning or that following it should maximise test accuracy.
Equal weighting in the metric also does not mean identical contributions from every sample: the targets depend on the sample-specific loss gradients and inputs. Nevertheless, we agree that this does not resolve questions about noise or redundancy, and an aggregate cosine does not identify which samples an update favours. A different metric would define a different reference. Our present interpretation is relative to the declared metric and loss.
We evaluated the per-sample first-order decreases di(u) at common weights. Each seed of the selected SGD, Adam and REF runs was replayed exactly to checkpoints before, around and after fitting and at the end. At each checkpoint, the SGD increment, the truncated REF increment and, on Adam trajectories, Adam's increment with its actual state were evaluated on the same probe batch. Examples are grouped by their loss quintile at the checkpoint, before the update, and we report each quintile's share of the batch's first-order decrease; increments keep their actual magnitude, but the shares do not depend on it. All three rules concentrate the first-order decrease on the highest-loss fifth of the probe, and their profiles at common weights are close: at Adam's states the three are nearly identical, and at SGD's states REF places slightly more of the decrease on the highest-loss quintile than SGD. These measurements therefore do not show that the rules prioritise different samples; under the declared loss, the first-order decrease of every rule follows where the loss gradients are concentrated.
There are empirical findings beyond the projection identities. Table 1 shows that SGD has alignment 0.204–0.253 despite estimated reachability close to one. Near-complete reachability is not guaranteed by the construction: the shared weights can prevent the sampled operators from meeting their targets jointly. At these states, the measurements distinguish that possibility from reweighting of a largely attainable target. The CReLU truncation sweep then shows that higher reference alignment does not monotonically improve prediction: from k=20 to k=200 LSQR iterations per update (separately trained runs), alignment increases from 0.864 to 0.992 while accuracy decreases from 0.602 to 0.558, with all settings fitting the training set (Table 9). This supports separating local reference following from generalisation within the tested control family. REF uses a normalised target and a truncated solve, so these are observations about the implemented control rather than an exact-reference training trajectory.
The common-logit result concerns the properties of the learned function. Table 14 evaluates raw and class-centred operators on the same inputs and snapshots. For ReLU REF, raw diversity falls from 191.5 to 109.8, while class-centred diversity changes from 221.0 to 218.8; the same distinction occurs for CReLU and leaky ReLU. The common-logit component accounts for 86–98% of REF's squared net operator displacement between these snapshots across the three activations, against 1% for SGD. Thus the pronounced raw diversity decline does not establish a comparable simplification of the operator governing class differences. This does not imply that the class-centred operators themselves are unchanged.
We agree that removing this component from structural measurements does not resolve its role in the update diagnostics. The reported alignment and kernel fractions use the full-logit map M. They should not be interpreted as invariant under replacing M by the class-centred map QM, where Q=I−11⊤/C and C is the number of classes. We therefore recomputed them under QM.