acceptodds
Under review as a conference paper at ICLR 2027

Common-Mode Collapse and Recovery in Direct Feedback Alignment

Abstract

Direct feedback alignment (DFA) trains hidden layers through fixed random projections of output error. With tanh hidden units and independent sigmoid outputs, plain stochastic gradient descent can stall near the loss of a constant predictor of class frequencies. We trace this stall to the error's common mode, the component shared across inputs. An exact mean–covariance decomposition separates a rank-one update formed by the mean teaching signal and mean presynaptic activity. Its leading component drives tanh units toward saturation. At initialization, random feedback provides no systematic correction of the shared error on average; readout learning limits its duration. A reduced model initialized from the network, without fitted parameters, predicts the concentration of activation sensitivity across 48 settings. On MNIST, class decodability largely survives collapse, but readout learning remains slow at a fixed learning rate. Adam learns faster despite deeper collapse. Calibrating the baseline readout to the class prior suppresses collapse and speeds learning; weaker feedback trades less collapse for slower learning. Replacing errors by their signs sustains collapse; subtracting the signal's batch mean prevents sustained collapse and improves learning in the tested setting. Related effects occur in deeper and convolutional networks and on CIFAR-10, with severity and cost depending on the readout, optimizer and input statistics.

Primary area:
optimization
Keywords:
direct feedback alignment, credit assignment, learning dynamics, activation saturation, representation collapse, biologically plausible learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ekRqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an early-training failure mode of direct feedback alignment (DFA): hidden representations become strongly concentrated and learning stalls near the loss of a class-prior predictor. The proposed explanation is that a shared component of output error, acting through fixed random feedback, mean presynaptic activity, and trainable biases, shifts tanh units toward saturation before the readout corrects the shared error.

The paper derives an exact mean–covariance decomposition and develops approximate dynamics for mean drift and sensitivity concentration. Predictions are evaluated across 48 settings and additional width–depth configurations. Frozen-feature experiments distinguish retained class decodability from slow readout optimization, while interventions involving calibration, centering, feedback strength, and optimizer choice test the mechanism and its limits. Related effects are examined in deeper networks, convolutional networks, and alternative objectives.

Critical strengths

A useful mechanistic account with quantitative predictions. The paper connects shared output error to hidden mean drift, saturation, and concentrated sensitivity, and evaluates a reduced model initialized from the network without fitted trajectory parameters. Its agreement across multiple experimental settings makes the contribution more informative than a descriptive observation of training plateaus.

A convincing distinction between decodability and optimization speed. The frozen-feature decoding and readout continuation experiments show that collapsed representations can retain substantial class information while supporting slow SGD learning. The centering and scalar-rescaling controls provide particularly useful evidence for an amplitude bottleneck at a fixed optimizer scale.

Critical weaknesses

The paper studies a specialised fixed-feedback learning rule, with its strongest quantitative evidence concerning small tanh networks under fixed-rate SGD. Adaptive optimization substantially shortens the stall, while the interventions have mixed effects across other settings. The paper should more clearly explain which important scientific or practical problem benefits from this analysis, and what guidance or transferable insight it provides beyond the examined transient. This affects my assessment of the contribution’s significance, rather than its algebraic correctness.

Questions for authors
  1. Scope and causal contribution of the rank-one approximation.

Equation (11) decomposes the shared-output-error update into a rank-one mean term and a gate–activity covariance term. Error centering removes both. Figure 6f measures the relative norms of these two contributions, while Table 11 reports their related mean-drift diagnostics, including substantial upstream contributions at initialization. Could the authors provide corresponding absolute component magnitudes and signed drift contributions for the two-class and squared-error settings where the reduced model misses gate concentration (Figure 2d)? In particular, is the discrepancy driven by changes in the mean-error signal during training, gate–error covariance, upstream feature motion, or other approximations in the reduced model? An intervention that separately removes the rank-one and covariance contributions would also help clarify their respective causal roles.

  1. Separating feedback orientation from the delivered mean-error drive.

The pre-aligned feedback control matches the Frobenius norm of the random feedback matrix, but this does not ensure that the projected mean-error signals have equal magnitudes. The paper acknowledges that the control therefore does not isolate the restoring contribution in Equation (5). Could the authors quantify the magnitude of the gated broadcast mean-error signal and the signed restoring contribution separately? If feasible, a control that matches the initial delivered mean-error magnitude while varying feedback orientation would help determine whether the reduced collapse and faster learning are primarily associated with the restoring direction or changes in the strength of the shared drive.

  1. Concrete utility and benefits under adaptive optimization.

What is the principal scientific or practical use of this analysis—for example, a transferable principle for biological credit assignment, guidance for hardware-constrained local learning, or improved DFA training? Please connect that use to the demonstrated evidence.

2 comments

Thank you for proposing specific causal tests. We added selective removal, signed drift measurements and shared-only routing; Figure 5 brings the orientation and adaptive-optimizer results into the main text.

Rank-one versus covariance. Removing the shared rank-one weight and bias terms preserves approximately the initial feature variance; removing shared gate–activity covariance alone leaves baseline compression nearly unchanged (Table 33). Removing shared error only in lower layers preserves 88% of top-layer variance and advances the 80% accuracy crossing from 1,200 to 910 updates (Table 36). All retain the true readout gradient and fluctuating-error term, including the gate–error residual.

Absolute and signed drift. Table 39 reports top-layer displacement norms, with signed projections divided by total displacement magnitude in parentheses. At initialization, means across three seeds are:

SettingShared rank/biasShared covarianceUpstream
Two-class0.0111 (−0.04)0.000123 (≈0)0.0543 (+0.91)
Unit-scale squared error0.163 (+0.14)0.000882 (≈0)0.436 (+0.86)

By update 50, squared error's rank/bias fraction rises to +0.75, while shared covariance remains near zero. Figure 22 retains the fluctuating-error and finite-step cross terms. These measurements distinguish upstream motion from local shared covariance.

Two checks locate the approximation errors. For squared error, supplying observed mean error reduces participation-minimum error from 0.390 to 0.035, implicating readout dynamics. For two classes, class-conditional Gaussian moments reduce participation reconstruction error from 0.510 to 0.009 at update 300 (Figure 21). These are measured-data diagnoses, distinct from initialization-only predictions.

Orientation at matched initial strength. Aligned, random and reversed feedback have initial mean-gated probe norms of 19.8 (rounded) in all three arms. Their signed restoring quantities q = ebar^T Wout Gamma_bar^2 B ebar are +31.06, +0.06 and −31.06. Here ebar is mean output error, Wout the readout, B the top-layer feedback matrix, and Gamma_bar the diagonal matrix of mean activation derivatives. Positive q denotes a corrective hidden contribution under the mean-only closure. Their minimum feature variances over the first 300 updates are 63%, 2.8% and 0.05% of initial values (Figure 5a; Table 35). The training-sample norm match does not constrain later accumulated drive.

The trajectory control uses B etilde + Bprime ebar: the fluctuating pathway stays random while shared error uses fixed aligned or reversed feedback. Shared-only alignment preserves 65% of initial variance versus 3% for ordinary DFA and reaches 80% accuracy in 920 versus 1,200 updates. Whole-error alignment preserves 63% and reaches it in 290 (Table 36), identifying an additional benefit from aligning input-dependent feedback.

Same-state virtual steps hold the fluctuating gradient exactly fixed. Alignment improves the measured top-layer-only loss decrease at every validation checkpoint in every seed at the nominal rate (Figure 19b). This establishes a local directional benefit alongside the training comparisons.

Utility under adaptive optimization. Centering persistent sign feedback advances the 80% crossing from 370 to 100 updates under Adam, with every seed improving; the benefit also holds at the original feedback scale. In a normalized Transformer it speeds fitting in all eight original pairs. At AdamW decay 0.03, it also brings forward sustained 95% held-out accuracy in all eight fresh pairs, with a 19% median paired reduction (Figure 5c; Table 47). The generalization benefit is regime-specific. The practical lesson is to control dominant shared drive separately from useful input-dependent feedback.

Thank you for proposing these controls. They let us distinguish the shared signal’s strength and direction and separate local from upstream contributions.

Reviewer_ekRqOpenReview ↗

Response to rebuttal and updated rating

Thank you for the detailed rebuttal. The added causal controls and drift diagnostics thoroughly address my technical questions, and the two adaptive-optimizer examples covering sign feedback under Adam and the normalized Transformer grokking task under AdamW help demonstrate utility beyond small tanh networks with fixed-rate SGD. For the final version, I hope the authors can further expand the validation and discussion on modern architectures. Overall, the paper is significantly strengthened, and I am raising my score from 2 to 3.

Reviewer_tnYqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study an early-training stall in direct feedback alignment (DFA), where hidden layers are trained via a fixed random projection of the output error. With tanh hidden units, independent sigmoid outputs, one-hot targets and vanilla SGD, a large error component arises that is shared across all inputs (which they call the "common mode"). The random feedback sends this shared error to every hidden unit with a different random coefficient. This pushes units towards saturation, i.e., the ends of the tanh function, such that hidden activity becomes nearly identical across inputs. The authors term this "representation collapse", and during it, the loss plateaus near that of a constant class-prior predictor. Using an exact mean-covariance decomposition of the DFA update, they then derive a "reduced model" which predicts the depth of this collapse across multiple settings without fitted parameters. They also show that class information remains linearly decodable from collapsed features, while readout learning slows down. Last, they test interventions such as centering the broadcast error, centering inputs, and recalibrating the readout bias to the class prior. With sign-error feedback, whose shared component does not decay, they show that subtracting its batch mean prevents sustained collapse. These experiments extend to deeper MLPs, a CNN and CIFAR-10.

Critical strengths
  1. The paper gives an intuitive mechanistic explanation of why random feedback fails to correct a shared error which is corrected by backpropagation. When the feedback is aligned with the readout, the resultant hidden-layer term is positive semidefinite, and therefore, reduces the mean error. Random feedback, however, provides no such systematic restoring term, which is the authors' main claim. I found this to be the most insightful part of the paper.
  2. The derivations I checked are correct -- the mean–covariance decomposition, the initial mean-error norm, and the large-dose participation asymptote.
  3. The authors test their theory quantitatively against multiple controlled variations (e.g., feedback gain, learning rates), instead of just a single setting. This makes their claims and assumptions convincing.
  4. I really like that the authors separate decodability from learning speed (Sec. 5). This is because class information is retained even after collapse, as shown empirically (88.688.6 \\% vs. 89.389.3\\% for held-out decoding). Whereas, readout learning slows down simply because feature amplitude reduces.
  5. The authors are transparent about the limits of their theory. They explicitly state the reduced model's failure modes in predicting participation rebound as well as the costs of some interventions (e.g., error centering destabilizing ReLU/GELU networks).
Critical weaknesses

Before I begin, I should caveat by saying that feedback alignment is somewhat outside my core expertise. For this reason, I am not able to fully assess the novelty of the work relative to existing DFA literature. My following comments therefore focus on whether or not the paper's claims are supported sufficiently by the presented evidence.

  1. My main concern is how much the phenomenon matters in practice. The baseline uses non-standard settings -- vanilla SGD, a small learning rate, and a feedback scale (g=1)(g=1) which is much larger than the original DFA setup (g≈0.03g\approx 0.03, L346–347). Using the original protocol, the authors show that the stall lasts only tens of updates (L80–82, Table 22). Moreover, standard choices such as softmax readouts, standardized inputs or Adam largely remove or shorten this effect. The CIFAR-10/100 results are the strongest evidence of broader relevance, but are mostly in the appendix.

  2. Relatedly, it is unclear to me that collapse is the sole factor making learning slow. Weaker feedback prevents collapse but learns slower (18971897 vs. 537537 updates). Adam learns faster despite deeper collapse, and Sec. 5 identifies small feature amplitude under a fixed SGD step size as the actual bottleneck. On the other hand, the title and present framing suggest representational collapse as the central problem. However, the evidence seems to suggest that collapse and slow learning are both consequences of the same shared error, rather than collapse causing slow learning.

  3. The reduced model's r=0.91r=0.91 is pooled across layers and settings with very different collapse levels. This model also misses participation rebound, overestimates shallow-layer collapse, and relies on fitted or phenomenological components for exit times. I think the authors should tighten their claim by saying that collapse prediction without fitted parameters is limited to just the onset.

  4. Last, I found the paper slightly hard to follow. The abstract reads as a list of findings without stating the paper's goal or main takeaway. I had to read till the end of the introduction section to understand the paper's contributions. Also, several paragraphs pack in many numbers before stating their point (e.g., L135–142), making it a dense read. Perhaps leading each paragraph with its conclusion would be helpful.

Questions for authors
  1. Under the original DFA protocol, the stall lasts only tens of updates, per table 22. In which practically relevant settings does collapse substantially slow learning?

  2. Given that weaker feedback prevents collapse but slows learning, and Adam learns faster despite deeper collapse, is collapse itself the cause of slow learning? Or is it the fixed SGD step size acting on small-amplitude features?

  3. What was the rationale (if any) behind choosing g=1g=1 for the baseline feedback scale? I ask this because the original DFA scale corresponds to roughly g=0.03g=0.03 at this width.

  4. How well does the reduced model predict participation minima within each layer or setting as opposed to being pooled across all cases?

  5. Do the authors observe a comparable stall, and comparable slowing of learning, for ReLU networks with softmax readouts and normalized inputs? I would expect the effect to be smaller, since with balanced classes a softmax readout starts at uniform predictions that already match the mean target, so the initial shared error is near zero.

2 comments

Thank you for identifying the restoring-direction argument and the decodability–learning-speed distinction as central insights. They now organize the revision and the direct tests below.

When does the shared signal materially cost learning time? New controls establish benefits under adaptive optimization, including at the original feedback scale. For persistent sign feedback, centering reduces median updates to 80% accuracy from 3,560 to 890 under SGD and from 370 to 100 under Adam. At the original scale under Adam, the comparison is 320 to 90; every paired seed improves (Table 26). This is distinct from the short transient under the original ordinary-error protocol.

In a two-block Transformer with layer normalization and softmax, shared error supplies a raw hidden gradient 4–5.5 times the fluctuating component at initialization. Centering speeds fitting in all eight original pairs: medians are 870 to 500 updates without decay and 1,410 to 730 at decay 1. At AdamW decay 0.03, it advances sustained 95% held-out accuracy in all eight fresh pairs, with a 19% median paired reduction (Figure 5c). Table 47 separates cohorts and decay regimes. Prior calibration also cuts the 80% crossing from 130 to 28 updates at SGD rate 0.1 (Table 8) and improves DFA validation performance on VOC2007 and WikiText-103 (Table 43).

Does collapse itself explain slow learning? We agree with your reading that collapse depth alone is not a measure of learning time. The shared signal suppresses feature variation; the resulting cost depends on optimizer scale, prior correction and hidden update direction. Frozen-feature centering and rescaling, with initial predictions preserved, isolate an amplitude bottleneck for readout SGD. Selective joint-training controls identify which signal causes compression: removing rank-one and bias contributions preserves approximately the initial variance, while removing gate covariance alone changes little in the baseline (Table 33).

Layer-specific removal adds a useful causal distinction. Removing shared error only from lower layers preserves 88% of initial top-layer variance and advances the 80% crossing from 1,200 to 910 updates (Table 36). Readout-only Adam reaches that crossing in 230 updates and weakens collapse; hidden-only Adam reaches it in 410 despite deeper concentration (Table 40). The revision therefore separates the origin of compression from its optimizer-dependent learning cost.

Why gain one? Unit-variance feedback defines the strong-drive baseline. It is not the original DFA scale or a proposed default. Original uniform feedback has entry standard deviation 1/sqrt(3d), about 0.033 at this width. Under SGD, scaling a fixed feedback matrix is exactly equivalent to scaling the hidden rate; changing the distribution is not. Across 41 distinct gain/rate configurations, early participation tracks the relative rate eta_out/(g eta_hidden) (Table 9). The scale sweep tests the mechanism beyond the baseline; reducing gain also weakens useful input-dependent updates.

Is agreement more than pooling? Correlations within layers 1–3 are 0.93, 0.90 and 0.86. Model mean absolute error versus a leave-one-condition-out constant is 0.128 versus 0.146 in layer 1, 0.112 versus 0.247 in layer 2, and 0.087 versus 0.258 in layer 3 (Table 38). Within gain, bias, rate, width and batch-size sweeps, correlations over setting-by-layer means are 0.98–0.99; the class-count sweep is the exception, 0.10 (Table 15). The model predicts early minima as well as onset; rebound and phenomenological exit dynamics remain separate claims.

ReLU, softmax and normalized inputs. Direct MLP tests at both feedback scales under SGD and Adam avoid early variance collapse and a detected plateau in every ordinary-DFA run (Table 41), as you anticipated. The Transformer supplies a measured case where normalization and softmax still allow substantial shared drive. The abstract now states the goal first, and Section 3 introduces the purpose and interpretation of each equation before the quantitative comparisons.

Thank you for pressing on where the effect matters. The new optimizer and Transformer comparisons give a more concrete answer to that question.

I thank the authors for their thorough and prompt response, and the substantial number of additional controls! All my concerns have been addressed, and I have raised my score to 3.

Reviewer_b9ddOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
3
Read review
Paper summary

The paper studies an early-training failure mode of direct feedback alignment. Starting from an exact mean–covariance decomposition of the hidden-layer update, it develops an approximate model in which shared output error drives hidden preactivations toward tanh saturation. The authors investigate how this compresses input-dependent activity, how well the model predicts early collapse, and whether interventions targeting the shared drive improve learning.

Critical strengths

The paper connects a clearly identified training phenomenon to an explicit decomposition of the update rule and tests consequences of the proposed mechanism.

The distinction between feature variation, class information, and the ability of a learning rule to exploit that information is conceptually useful.

Critical weaknesses

It could be my unfamiliarity with related works in FA/DFA, but I found the presentation hard to understand and follow. I cannot easily understand the defined and derived quantities. Normally, I have some progressive understanding as I go through derivations in more details. For this work, however, I think I have to stare at certain equations (e.g. Equation 5-7) for tens of minutes in order to continue reading. Numbers and facts are often enumerated without interpretations and implications. The presentation is really concerning as it obstructs understanding.

The practical application of the phenomenon seems limited. The principal analysis focuses on image classification with a sigmoid classification head and tanh activations.

Questions for authors

The paper goes back and forth on advantages and disadvantages of the analysis. Can you summarize for me: a) what can the analysis predict, b) what are the gaps, c) what are the potential reasons for the gap?

If common-mode collapse is avoided, would this make DFA practical? Figure 1d suggests DFA works better than BP. Are you claiming that DFA has better performance than BP? What obstacles remain to making DFA practical, and what potential advantages motivate its use relative to BP? This paper doesn't need to close all the gaps, but I want to understand DFA vs BP better.

How would the analysis apply to networks with ReLU activations and a softmax classification head?

2 comments

Thank you for identifying where the presentation obstructed understanding. We have rewritten the derivation and added a main-text figure. Short answers to your three questions are:

  1. Prediction: the model predicts early saturation and the concentration of sensitivity across units; recovery requires additional dynamics.
  2. Utility: controlling shared error can speed learning, including under Adam.
  3. ReLU/softmax: the exact gradient separation applies, but the tanh saturation dynamics do not. Direct normalized ReLU/softmax tests confirm the expected weak transient; a normalized Transformer supplies a case where centering speeds fitting and, under weak decay, held-out generalization.

The mechanism in words. Each hidden unit receives the same mean output error across examples, multiplied by its fixed random feedback coefficients. This can move tanh units toward saturation, reducing their response to input differences. Meanwhile, the readout learns the class prior and reduces that error. The analysis asks which process acts first.

Section 3 (p. 3) now introduces each quantity before deriving it. A gate is the activation's local slope. Equation 5 asks whether hidden feedback helps correct mean error: one gate converts broadcast error into preactivation drift, the other converts drift into activity change. Equation 6 estimates collapse risk as initial drift rate times readout-correction time. Equation 7 instead sums the drive along the observed trajectory; it is a measurement, not a prediction. Table 13 separates these objects from the reduced model.

What is predicted, and why do gaps occur? Across 48 settings, predicted early participation minima have condition-averaged correlation 0.91 and mean absolute error 0.10 on a 0–1 scale, compared with 0.64 and 0.31 for the initialization estimate (Table 15). High participation means gate energy is spread across units; it measures neither total sensitivity nor class information. Within-layer comparisons are in Table 38. The model describes early concentration; it does not predict the later participation rebound. Its exit extension is phenomenological.

We now diagnose the main discrepancies. For unit-scale squared error, using observed mean error substantially improves prediction, identifying the readout approximation as a source. For two classes, class-conditional activity distributions reconstruct participation much better than a single Gaussian (Appendix F.3). These checks explain which assumptions need refinement.

What benefit does this offer? Our claim is improved learning within DFA. Figure 1 uses a common nominal rate and does not establish superiority over tuned BP. Avoiding collapse addresses one obstacle; useful credit direction, feature conditioning and architecture-dependent optimization still determine competitiveness. Fixed feedback avoids forward–feedback weight matching, motivating biological and local-learning research.

The concrete benefits persist under Adam. Centering persistent sign feedback cuts median updates to 80% accuracy from 3,560 to 890 under SGD and from 370 to 100 under Adam, with every pair improving. In a normalized Transformer, centering speeds fitting in all eight original pairs. At AdamW decay 0.03 it also advances sustained 95% held-out accuracy in all eight fresh confirmation pairs, with a 19% median paired reduction (Figure 5c; Table 47). This generalization result is specific to weak decay. Prior calibration also improves DFA on multi-label recognition and language modeling (Table 43).

ReLU and softmax in detail. ReLU changes active sets rather than saturating on both sides. Softmax mean error vanishes when mean predictions match the target prior; balanced uniform predictions are one case. In the baseline MLP with ReLU, softmax and global standardization, all 40 ordinary-DFA runs avoid an early variance collapse or detected plateau across two feedback scales, SGD/Adam and both sampling schemes (Table 41). This confirms the scope distinction. The Transformer experiment instead measures the shared raw gradient directly and tests its removal, without assuming the tanh model (Appendix I).

Thank you again. Your questions about clarity and benefit led directly to the rewritten Section 3 and Figure 5; we hope these changes make the mechanism and its implications easier to assess.

I appreciate the clear explanations. I think the scope of where this applies is still a major limitation. I will slightly raise my score because of the improved clarity.

Comments on the paper

1 comment
AuthorsOpenReview ↗

general response

We thank the reviewers for recognizing the mechanistic contribution and the distinction between retained information and learning speed. Their questions led to a clearer derivation and direct tests of the shared signal's causal role and usefulness under adaptive optimization. The revision keeps the original question: when does shared output error reshape hidden activity before the readout corrects it?

The central result is that a common teaching signal can consume much of the hidden update while weakening input-dependent activity. In the tanh regime, we derive this process and predict early sensitivity concentration. The contribution is the mechanism, quantitative predictions and causal tests; the mean–covariance identity and error centering themselves are established tools.

What changed and where to look:

  • Clearer argument: Section 3 (p. 3) now derives the mean drift and mean-logit equations step by step, explaining the two derivative gates and separating the initialization estimate, reduced prediction and measured dose. Table 13 summarizes what each predicts.
  • Causal tests: Section 4 and Appendix F separate rank-one/bias, gate covariance and upstream effects. With initial gated strength matched, aligned, random and reversed feedback retain 63%, 2.8% and 0.05% of initial feature variance. Aligning only the shared pathway preserves 65% and advances the 80% accuracy crossing from 1,200 to 920 updates while retaining random input-dependent feedback (Tables 35–36).
  • A new main-text figure: Figure 5 (p. 9) brings together the orientation control, faster sign-feedback learning under SGD and Adam, and earlier held-out generalization in a normalized Transformer at weak AdamW decay. Each panel shows the paired seeds.
  • Prediction and explanation: Correlations within layers 1–3 are 0.93, 0.90 and 0.86; we also report absolute errors against a constant predictor and diagnose the two-class and squared-error discrepancies (Appendix F.3). Separate readout/hidden Adam controls explain why collapse depth alone cannot predict learning time (Table 40). Direct ReLU/softmax tests establish the expected weak transient with standardized inputs.

Where the intervention helps learning. Centering persistent sign feedback cuts median updates to 80% accuracy from 3,560 to 890 under SGD and from 370 to 100 under Adam; at the original feedback scale under Adam, the reduction is 320 to 90 (Table 26). Every paired seed improves. Thus the benefit extends beyond a small-rate SGD transient.

In a normalized modular-arithmetic Transformer, centering speeds fitting in all eight original pairs and brings forward transfer to held-out pairs whose reversals were trained. In a fresh confirmation at AdamW decay 0.03, centered DFA also reaches sustained 95% accuracy on the full held-out set earlier in all eight pairs, with a 19% median paired reduction (Figure 5c). This is a regime-specific speed benefit. Appendix I (p. 61) separates fitting, symmetry transfer and full generalization, and reports every decay cohort separately (Table 47).

Evidence in larger models. Prior calibration improves DFA validation performance in all five paired seeds in both a pretrained ResNet-18 on VOC2007 and a six-block Transformer on WikiText-103, using Adam/AdamW. Mean average precision rises from 41.63% to 43.18%; language cross-entropy falls from 6.70 to 6.26 nats per byte-BPE token (Table 43). Ordinary and calibrated DFA use the same selected rate in each task. These exploratory comparisons demonstrate a useful intervention within DFA. Calibration also changes readout optimization; the selective evidence for hidden-layer causality comes from the baseline tanh controls above.

The practical guidance is to measure shared drive separately from useful update direction and control it when it dominates. The revision preserves the original mechanistic focus while showing concrete benefits under adaptive optimization. The anonymous archive contains the code, paired records and reproduction commands; the individual replies answer each reviewer's questions directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.