acceptodds
Under review as a conference paper at ICLR 2027

Isotropic Updates Enable Backprop-Free Training of Modern Vision Architectures

Abstract

Forward Direct Feedback Alignment (FDFA) trains networks without backpropagation (BP) by delivering the output error directly to each layer through feedback matrices learned with forward-mode automatic differentiation. On modern vision architectures, namely deep non-convolutional models with normalization and residual connections such as ViT, MLP-Mixer, and Swin, however, FDFA and existing backpropagation-free methods fail to train well. We hypothesize that a key contributing factor is spectral concentration of the updates: the errors of all samples pass through the same feedback matrix, and in our diagnostics FDFA updates are skewed toward a few singular directions. Accumulated skewed updates inflate the weights anisotropically and lower the sensitivity of some feedback points, where single-pass feedback estimation then degrades and in turn sustains the skewed updates. We therefore introduce norm-preserving update orthogonalization (NPO), which makes the update of any optimizer isotropic while preserving its column and row spaces and its Frobenius norm, and apply it to FDFA to obtain IsoFDFA. Training diagnostics are consistent with the hypothesis, and NPO suppresses each stage. IsoFDFA trains MLP-Mixer, ViT, and Swin on CIFAR, matching or exceeding a BP baseline trained with the same learning rate on MLP-Mixer and ViT, and trains DeiT-S on ImageNet, where FDFA makes no progress. To our knowledge, this is the first result showing that a vision Transformer can be trained on ImageNet to non-trivial accuracy with learned feedback and a single forward-mode pass per step, without auxiliary local losses or backpropagation of errors across blocks.

Primary area:
optimization
Keywords:
backpropagation-free learning, direct feedback alignment

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_XjiFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes norm-preserving update orthogonalization (NPO) and applies it to FDFA as IsoFDFA. On CIFAR, IsoFDFA substantially improves over FDFA for MLP-Mixer, ViT, and Swin-T. On ImageNet, it reaches 25.8% top-1 accuracy, compared with 0.2% for FDFA and 79.9% for BP.

Critical strengths

1、IsoFDFA substantially improves FDFA on several non-convolutional CIFAR models.

2、 The paper includes useful controls, including BP+NPO, optimizer changes, and per-point feedback.

Critical weaknesses

1、The mechanism diagnostics focus mainly on one ViT/CIFAR-100 setting, so it is unclear how broadly they explain the gains.

2、ImageNet results use one seed and remain far below BP; the effect of pooled feedback at ImageNet scale is not isolated.

3、Performance drops sharply with fewer Newton--Schulz iterations, but the runtime and compute trade-offs are not fully reported.

Questions for authors

1、Do you have multi-seed ImageNet results?

2、How much does pooled feedback affect ImageNet performance?

3、How much of NPO's benefit comes from general optimization gains, given that BP+NPO also improves over BP?

2 comments
AuthorsOpenReview ↗

Response to Reviewer XjiF

Thank you for your positive assessment of the improvement over FDFA and of the control experiments. We have addressed each of your comments and questions as follows.

W1

"The mechanism diagnostics focus mainly on one ViT/CIFAR-100 setting, so it is unclear how broadly they explain the gains."

We added two things.

  • We ran the same diagnostics as in Figure 3 on MLP-Mixer and Swin-T and show them in Appendix C.4 (Figure 4). Both models show broadly the same trends as ViT. Under FDFA, the updates stay close to rank one, the weights and activations inflate, and the imbalance of the sensitivities widens greatly. Under IsoFDFA, these are suppressed. The numbers are given in Appendix C.4.
  • We added a Swin-T column to the ablation (Table 4) and now report mean ± standard deviation over three seeds. The conclusions of (i)–(vi) are unchanged on Swin-T. The one numerical difference is FDFA with pooled feedback, which stays at 2.6% on Swin-T (18.1% on MLP-Mixer and 16.2% on ViT).

W2 / Q1 / Q2

"ImageNet results use one seed and remain far below BP; the effect of pooled feedback at ImageNet scale is not isolated." / "Do you have multi-seed ImageNet results?" / "How much does pooled feedback affect ImageNet performance?"

We now report BP, FDFA, and IsoFDFA on ImageNet with three seeds and show the effect of pooling separately (Table 2).

Methodtop-1top-5Feedback memory (per GPU)
BP79.9 ± 0.0495.0 ± 0.04–
FDFA0.10 ± 0.010.51 ± 0.0213.3 GB
IsoFDFA29.2 ± 1.152.3 ± 1.313.3 GB
IsoFDFA (pooled)24.9 ± 2.746.7 ± 3.474 MB
  • Multi-seed results (Q1). BP, FDFA, and IsoFDFA all use three seeds. The top-1 accuracy of IsoFDFA is 29.2 ± 1.1% (the 25.8% of the submitted version was a single seed with pooling). FDFA is at chance level in all three seeds.
  • Effect of pooling (Q2). In the revision, feedback without pooling is the main result, shown next to the pooled variant. Pooling reduces the feedback memory from 13.3 GB to 74 MB, to about 1/180, and the drop in top-1 is nevertheless only about 4 points (29.2% → 24.9%).
  • Gap to BP (W2). That a gap to the 79.9% of BP remains is stated as a limitation in the conclusion. On the other hand, FDFA cannot train DeiT-S on ImageNet at all, and IsoFDFA makes it trainable up to 29.2%. To our knowledge, this is the first time a vision Transformer has been trained on ImageNet to non-trivial accuracy with learned feedback and one forward-mode pass per step, without local losses or backpropagation across blocks.

W3

"Performance drops sharply with fewer Newton--Schulz iterations, but the runtime and compute trade-offs are not fully reported."

We added measurements to Appendix C.7 (Table 7) and summarize them in Section 4.5 of the main text (the paragraph on computational cost). They are measured on CIFAR-100 with batch size 128 on an RTX A6000.

IterationsAccuracy (MLP-Mixer / ViT / Swin-T, mean over three seeds)Increase in step time over FDFA
13.9 / 1.9 / 3.8+3–5%
27.2 / 5.5 / 8.4+5–8%
551.6 / 45.1 / 49.7+5–11%
8 (default)50.9 / 45.9 / 49.7+10–15%

The peak memory equals that of FDFA regardless of the number of iterations. Reducing eight iterations to five saves only 2–5% of the step time, and with one or two iterations training does not succeed. The benefit of reducing the number of iterations is therefore small, and we use eight as the default. In comparison with BP, FDFA itself takes 1.9–2.4 times the time and 2.7–3.9 times the memory of BP, and the part added by NPO is small in comparison.

Q3

"How much of NPO's benefit comes from general optimization gains, given that BP+NPO also improves over BP?"

The effect of NPO can be divided into three parts: the gain of the orthogonalized update itself (which also operates under BP), limiting the inflation (under DFA and FDFA), and the recovery of the feedback estimate (under FDFA only). Indeed, the gain from NPO is in the order BP < DFA < FDFA on all four models (−1.0 to +11.1 points for BP, +7.9 to +34.6 for DFA, and +13.2 to +48.9 for FDFA). We believe that in FDFA the recovery of the estimate, an effect specific to it, adds to the general gain. These effects are not additive, however, so we cannot decompose the gain quantitatively. Details are given in the response to the common question in our response to all reviewers.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer XjiF

Dear Reviewer XjiF,

Thank you again for your careful review and for your positive assessment of the improvement over FDFA and of the control experiments. In the revision, the mechanism diagnostics are also shown for MLP-Mixer and Swin-T (Figure 4), the ImageNet results use three seeds and show the effect of pooled feedback separately (Table 2), the runtime for each number of Newton–Schulz iterations is reported (Table 7), and a control that applies NPO to DFA was added to distinguish the gain specific to FDFA from a general optimization gain (Table 11). We believe that these address the points you raised.

Since the discussion period closes on November 18 (AoE), we would be very grateful to hear whether, in your view, our response and the revised manuscript resolve your concerns. We will gladly answer any remaining questions.

Thank you very much for your time and consideration.

Reviewer_PGKbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the failure of Forward Direct Feedback Alignment to train modern architectures. The authors propose an explanation for this failure based on the concentration of updates along a few singular directions.

To address this problem, they propose Norm-Preserving Update Orthogonalization, which equalizes the nonzero singular values of optimizer updates while preserving their Frobenius norm.

Their proposed method improves on feedback alignment on CIFAR-10 and CIFAR-100. It also enables training DeiT on ImageNet-1k, although performance remains significantly below backpropagation.

Critical strengths

The paper presents a convincing experimental observation about the failure of feedback alignment on modern architectures. The strong difference between the classical FA and their proposed algorithm is interesting and motivates further investigation.

The diagnostics and the ablations provide useful evidence about the relationship between update spectra, training stability, and feedback estimation.

Critical weaknesses

I am not fully convinced that NPO is the most natural way to address the identified problem. The mathematical argument explains why equalizing singular values minimizes the spectral norm under constraints, but it is less clear why this should be the best transformation for learning. Have the authors considered alternatives like Newton like updates?

Questions for authors
  • Could the authors clarify (maybe theoretically) the connection between sensitivity imbalance, the estimation noise, and the failure of standard FA in Section 2.2? Do the diagnostics of Figure 2 show similar behavior for MLP-Mixer and Swin-T?

  • Why should equalizing all nonzero singular values be preferable to other possible corrections, such as second-order or Newton-like preconditioning? Have the authors investigated the role of the effective rank of the updates, or the use of a truncated-rank correction?

  • On ImageNet, the method uses pooled feedback matrices and different training settings compared with BP. How much do these changes influence the results?

  • In Figure 1, I would like to see the training curves of standard BP for the three architectures.

  • It would be useful to include the BP and BP+NPO reference values directly in Table 3, instead of requiring the reader to look back at Table 1.

  • The diagnostics in Figure 2 are shown only for ViT. It would be interesting to understand whether the same mechanism is observed in MLP-Mixer and Swin-T.

4 comments
AuthorsOpenReview ↗

Response to Reviewer PGKb (1/2)

Thank you for your positive assessment of the experimental observations and of the diagnostics and ablations. We have addressed each of your comments and questions as follows.

W1 / Q2

"[…] it is less clear why this should be the best transformation for learning. Have the authors considered alternatives like Newton like updates?" / "Have the authors investigated the role of the effective rank of the updates, or the use of a truncated-rank correction?"

Why equalization. Equalizing the singular values follows naturally from the failure mechanism that the revision derives in a simplified model (Section 2.2). In this mechanism, the skew of the updates toward a few directions inflates the activations, and this sets off the cycle. Equalizing the singular values removes the skew while preserving the norm of the update, and limits the inflation of the activations (Section 2.3). The problem we identify is the spectral concentration of the updates, and equalization addresses it directly.

Relation to Newton-type preconditioning.

  • NPO is a special case of matrix-type second-order preconditioning. The Shampoo update L^{-1/4} G R^{-1/4} without accumulated statistics (L = G G^T, R = G^T G) is the polar factor U V^T of G (Bernstein & Newhouse, 2024). Equalization is thus whitening by a preconditioner built from the current update alone. (The Newton–Schulz iteration in our implementation only computes the polar factor and uses no curvature.)
  • Our focus is the spectrum of the updates. With the same model, optimizer, and learning rate, BP trains (42–56% on CIFAR-100) and FDFA fails. The difference lies in how the error is delivered and in the resulting spectrum of the updates (stable rank 1–2 for FDFA and 3–8 for BP; Figure 3a). This does not rule out interactions with the curvature of the loss, but NPO controls the spectrum of the updates directly, without estimating curvature.
  • Curvature is costly without BP. A Hessian–vector product requires backpropagation or second-order forward-mode differentiation (Cobb et al., 2024), and K-FAC or Shampoo would build the preconditioner from a gradient estimate passed through feedback that is still being learned.

Experiment combining Shampoo with FDFA (Appendix C.9, Table 9). Since the polar factor corresponds to Shampoo without accumulation, we tested its counterpart: Shampoo with statistics accumulated throughout training (Gupta et al., 2018), applied to the weight matrices other than the head. Final-epoch accuracy (%) on CIFAR-100 after 100 epochs (the Shampoo rows are means over three seeds):

MLP-MixerViT
FDFA2.05.0
FDFA + Shampoo (best learning rate)44.240.5
FDFA + Shampoo direction, with the norm of the Adam step3.32.1
IsoFDFA50.945.9

Shampoo also mitigates the failure substantially (2–5% → 40–44%), which is consistent with attributing the failure to the spectral concentration of the updates (we did not measure the spectrum of the Shampoo updates). It stays 5–7 points below IsoFDFA, however, and is sensitive to the step size: at twice the best learning rate training collapses, and with the norm of the Adam step the accuracy rises to 7–10% and then collapses. Accumulated statistics do not guarantee an isotropic update at each step; our hypothesis is that an update that remains skewed, applied at the size of the Adam step, enters the same cycle as FDFA. NPO needs no accumulated statistics and, in most cases, no learning rate tuned on FDFA. This discussion was added to Section 3.

Role of the effective rank, and truncated-rank correction (Appendix C.10, Table 10). We added two experiments that equalize only partially (computed with an exact SVD); in both, the result is rescaled to the Frobenius norm of the original update.

  • Varying the degree of equalization. σ_i → σ_i^p (p = 1 is FDFA, p = 0 the equalization of IsoFDFA). For p = 0.75 / 0.5 / 0.25, MLP-Mixer gives 2.6 / 2.7 / 3.8% and ViT 1.6 / 2.0 / 30.3% (default IsoFDFA: 50.9 / 45.9%).
  • Correcting only the top ranks. Equalizing the top k singular values and discarding the rest, for k = 8 / 32, gives 3.7 / 7.4% on MLP-Mixer and 3.8 / 10.3% on ViT.

The default NPO outperformed all the partial corrections we tested. With p > 0, the singular values remain unequal; the top-k correction is isotropic within the retained subspace but discards the other directions. This suggests that, besides limiting the skew, keeping a broad update subspace matters, and it is consistent with the small effect of one or two Newton–Schulz iterations (Table 4 (vi)). The role of the effective rank is shown in Figure 3a: the stable rank of the updates stays at 1–2 under FDFA and is above 100 under IsoFDFA.

AuthorsOpenReview ↗

Response to Reviewer PGKb (2/2)

Q1 / Q6

"Could the authors clarify (maybe theoretically) the connection between sensitivity imbalance, the estimation noise, and the failure of standard FA in Section 2.2? Do the diagnostics of Figure 2 show similar behavior for MLP-Mixer and Swin-T?"

Theory (Q1). As you point out, this connection was hard to read in the submitted version. We rewrote Section 2.2 and added theory in a simplified model: each of the five stages (a)–(e) of the chain is stated as a proposition (proofs in Appendix A), and a new figure summarizes the five quantities, their links, and where NPO acts (Figure 2).

  • (a) When the estimation error is large, the same skewed update persists (Proposition 1); under BP, the component along the common direction of the inputs decays.
  • (b) When skewed updates persist, the scale of the activations grows in proportion to the number of steps, and faster the larger the share of the gradient along the common input direction (Proposition 2).
  • (c) The inflation of the activations breaks the balance of the sensitivities (Proposition 3).
  • (d) The imbalance of the sensitivities amplifies the cross-point noise (Proposition 4, Eq. (3)).
  • (e) The noise leaves an error in the feedback estimate (Proposition 5).

Stages (c)–(e) correspond to the connection that you ask about. Section 2.3 gives, in the same model, the condition under which NPO lowers the rate at which the scale grows. The theory covers a single linear map (Propositions 1 and 2) and a uniform inflation imposed from outside (Propositions 3 and 5); that the two connect in actual training and the cycle closes is a hypothesis, which we examine with the diagnostics of Section 4.2 and the intervention experiments of Section 4.5.

Diagnostics on MLP-Mixer and Swin-T (Q6). We ran the same diagnostics as in Figure 3 (Figure 2 of the submitted version) on MLP-Mixer and Swin-T and show them in Appendix C.4 (Figure 4). Both models show broadly the same trends as ViT. Under FDFA, the updates stay close to rank one, the weights and activations inflate, and the imbalance of the sensitivities widens greatly. Under IsoFDFA, these are suppressed. The numbers are given in Appendix C.4.

Q3

"On ImageNet, the method uses pooled feedback matrices and different training settings compared with BP. How much do these changes influence the results?"

Pooling. In the revision, feedback without pooling is the main result (Table 2).

Methodtop-1top-5Feedback memory (per GPU)
BP79.9 ± 0.0495.0 ± 0.04–
FDFA0.10 ± 0.010.51 ± 0.0213.3 GB
IsoFDFA29.2 ± 1.152.3 ± 1.313.3 GB
IsoFDFA (pooled)24.9 ± 2.746.7 ± 3.474 MB

Pooling reduces the feedback memory from 13.3 GB to 74 MB, to about 1/180, and the drop in top-1 is nevertheless only about 4 points. On CIFAR-100, the effect of pooling is even smaller: 0.8, 0.2, and 1.1 points (MLP-Mixer, ViT, Swin-T; Table 4 (v)).

Training settings. As you point out, IsoFDFA differs from BP in two settings (learning rate 2.5 × 10^-4, and weight decay also on LayerNorm). To examine their influence, we trained BP with the same setting as IsoFDFA (three seeds, 300 epochs; Appendix C.3).

Setting of BPtop-1top-5
Standard (learning rate 10^-3, no weight decay on LayerNorm)79.9 ± 0.0495.0 ± 0.04
Same as IsoFDFA (learning rate 2.5 × 10^-4, weight decay also on LayerNorm)78.6 ± 0.394.2 ± 0.1

The difference in settings changes the top-1 accuracy of BP by only 1.3 points; it has little influence on the results.

Q4

"In Figure 1, I would like to see the training curves of standard BP for the three architectures."

We added the training curves of BP for the three models to the bottom row of Figure 1 (black dashed lines). The submitted version showed only BP + NPO as a reference. The curves of BP + NPO, FDFA, and IsoFDFA are the same as in the submitted version. The curves of BP are, like the others, means over three seeds, with bands of ± one standard deviation. The final-epoch mean ± standard deviation over three seeds is in Table 1.

Q5

"It would be useful to include the BP and BP+NPO reference values directly in Table 3, instead of requiring the reader to look back at Table 1."

We added the reference values of BP and BP + NPO to the first row (BP/BP+NPO/FDFA) of the ablation table (Table 4 in the revision; Table 3 in the submitted version). The values are the same as in Table 1.

Reviewer_PGKbOpenReview ↗

Thank you for the detailed rebuttal

Thank you for the detailed answers and the many new experiments. I appreciate the rewrite of Section 2.2 with a proposition for each of the points.

I have a few remaining comments, mostly about the choice of full equalization.

1. Partial equalization (Table 10). The jump from p=0.25p = 0.25 (3.8% on MLP-Mixer) to p=0p = 0 (50.9%) is striking. A possible explanation is the size of the step along the top singular direction, rather than isotropy itself. Since every variant is rescaled to the same Frobenius norm, the more equal the singular values, the smaller the part of the step along the top direction. For a raw update with stable rank around 1.2, a rough estimate gives a step along the top direction about 2–3 times larger with p=0.25p = 0.25 than with NPO. Figure 5 shows that IsoFDFA on MLP-Mixer already fails when its step is 3 times larger. Would p=0.25p = 0.25 train with a learning rate about three times smaller? This would help separate the role of full isotropy from that of a smaller step along the dominant direction.

2. Feedback alignment on MLP-Mixer. In Appendix C.4, the cosine similarity between BlB_l and the mean Jacobian under IsoFDFA on MLP-Mixer stays around 0.02–0.04, yet IsoFDFA reaches 50.9%. Do the authors think that on this model the gain comes from limiting the inflation rather than the recovery of the feedback estimate?

Overall, I still find the revised paper clearer and better than the submitted version, and I continue to find the result interesting.

AuthorsOpenReview ↗

Response to the Follow-up Questions

Thank you very much for your careful reading of our response, for your kind words on the revised Section 2.2, and for the thoughtful follow-up questions.

"Would p = 0.25 train with a learning rate about three times smaller? This would help separate the role of full isotropy from that of a smaller step along the dominant direction."

We ran the experiment you propose, with p = 0.25 at a learning rate about three times smaller (3 × 10^-5) and, to cover a larger factor, ten times smaller (10^-5). The setting is otherwise that of Table 10 (mean ± standard deviation over three seeds).

UpdateLearning rateMLP-MixerViT
p = 0.2510^-4 (Table 10)3.8 ± 0.730.3 ± 1.1
p = 0.253 × 10^-52.6 ± 0.428.5 ± 0.2
p = 0.2510^-522.2 ± 0.319.8 ± 0.2
IsoFDFA (p = 0)10^-4 (Table 10)50.9 ± 0.445.9 ± 0.4
IsoFDFA (p = 0)3 × 10^-5 (Figure 5)42.6 ± 0.234.1 ± 0.3
IsoFDFA (p = 0)10^-5 (Figure 5)30.8 ± 0.322.4 ± 0.4

Your reading agrees with our mechanism. In Section 2.3, what NPO lowers is the step along the common input direction, which sets how fast the activations inflate (Proposition 2). The results partly support it. On MLP-Mixer, p = 0.25 still collapses at a learning rate three times smaller (2.6%), but no longer at one ten times smaller (22.2%). A smaller top step thus stabilizes training. Even so, it remains well below IsoFDFA at the same learning rate (42.6% and 30.8%). On ViT, p = 0.25 does not collapse, but stays below IsoFDFA at every learning rate.

We therefore think that both full isotropy and a small top step play a role. A small top step gives stability. But if it is obtained through a smaller learning rate, every direction shrinks by the same factor, so the skew of the update is unchanged and learning degrades. The range of stable learning rates is narrow (Section 2.4), and no learning rate we tested brings p = 0.25 close to IsoFDFA at 10^-4. Equalization lowers the top step while keeping the size of the update, so the other directions are not sacrificed. Removing the skew, and keeping the size of the update so that the learning rate need not be chosen anew, are the two requirements we place on our method (Section 2.3) and the reason why we use full equalization.

"In Appendix C.4, the cosine similarity between B_l and the mean Jacobian under IsoFDFA on MLP-Mixer stays around 0.02–0.04, yet IsoFDFA reaches 50.9%. Do the authors think that on this model the gain comes from limiting the inflation rather than the recovery of the feedback estimate?"

As you point out, the cosine under IsoFDFA on MLP-Mixer is low (0.02–0.04). However, we believe that this result is consistent with our view and with the theory of Section 2.2, and that it does not rule out the mechanism we propose. As stated in the common response, we divide the effect of NPO into (i) the gain of the orthogonalized update itself, (ii) limiting the inflation, and (iii) the recovery of the feedback estimate. Your question concerns the relation between (ii) and (iii). Effect (iii) arises through (ii) (Section 2.2 (c)–(e)), so the two cannot be separated completely. We think that both operate on MLP-Mixer, for three reasons.

  • The learned feedback matters for the accuracy. With NPO in both cases, fixed random feedback (DFA + NPO) reaches 24.1% on MLP-Mixer and learned feedback (IsoFDFA) 50.9% (Table 11). The two differ only in whether the feedback is learned. We expect (i) and (ii) to operate under DFA + NPO and (i)–(iii) under IsoFDFA, so this difference of 27 points indicates that (iii) operates on MLP-Mixer as well.
  • The cosine is indeed low, but it improves over FDFA. It is below 0.002 in absolute value under FDFA and 0.02–0.04 under IsoFDFA (Appendix C.4).
  • The low cosine is consistent with the imbalance of the sensitivities that remains on this model. The variance of the single-pass estimate has a term proportional to the imbalance χ_l (Eq. (3)). Under IsoFDFA, the median χ_l at epoch 30 is smallest on ViT, 67 on Swin-T, and about 10^3 on MLP-Mixer, and the cosine is about 0.3, 0.10–0.15, and 0.02–0.04, in the same order (Figures 3d and 4). On MLP-Mixer, the weight norm still grows to 4.3 times its initial value under IsoFDFA (2.4 times under BP), and χ_l falls by ten orders of magnitude from that of FDFA (5 × 10^13) but stays at about 10^3. NPO thus limits the imbalance substantially on this model too, but not completely, and the estimate remains noisy.

Effect (iii) thus also operates on MLP-Mixer. That the alignment stays lower there than on the other models can be understood from the remaining imbalance of the sensitivities. As noted in the common response, these effects are not additive, so the present experiments cannot give their shares. Separating them quantitatively is left for future work.

We hope that these answers address your remaining concerns. Thank you again for your careful review and for your constructive suggestions.

Reviewer_r87xOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IsoFDFA, which applies norm-preserving orthogonalization to optimizer updates in Forward Direct Feedback Alignment (FDFA). The authors hypothesize that spectrally concentrated updates induce weight inflation, sensitivity imbalance, and degraded feedback estimation. The proposed method substantially improves FDFA on a variety of architectures.

Critical strengths
  1. The empirical results are substantial. IsoFDFA dramatically improves FDFA across several modern vision architectures and benchmarks. I think it is a meaningful empirical contribution that helps advancing the field of alternatives to backpropagation.

  2. The ablations are informative and help identify the core contribution of the method. Indeed, the experiments illustrate how update spectra, weight inflation, sensitivity imbalance, and feedback alignment evolve during training. They also support the importance of update orthogonalization and scaling. (see however weakness 3).

With more work, the contributions of the paper could clearly be strengthen. I say this because I find this work interesting.

Critical weaknesses

Limited novelty relative to closely related work.

Boeshertz et al. (2026), Overcoming Rank Collapse in Feedback Alignment, already identify low-dimensional updates as a limitation of feedback alignment and use Muon-style orthogonalization to improve training and alignment. There is thus a substantial overlap with prior work.

Furthermore, PolarAdamW (Zhang, 2026) already applies polar orthogonalization to AdamW-preconditioned directions. The principal methodological distinction here is preserving the raw update's Frobenius norm, which amounts to a particular scaling of an established polar transformation. The associated spectral-norm optimality follows from classical matrix inequalities.

More broadly, the importance of orthogonality and spectral conditioning for stable signal and gradient propagation is well established, notably through the literature on dynamical isometry and Lipschitz-constrained networks (Pennington et al., 2017; Xiao et al., 2018; Araujo et al., 2023; Delattre et al., 2023). More specifically, FOTON (Caillon et al., 2025) already demonstrated the importance of weight orthogonality for stable forward-only learning in deep networks. Although these works address different mechanisms from update orthogonalization, they provide relevant context for the proposed approach and deserve some discussion in the related work section in my opinion.

While applying this approach to learned FDFA produces interesting results, I find the underlying methodological novelty limited.

Insufficient evidence separating the proposed mechanism and scaling rule from existing optimization techniques.

The comparisons do not establish whether norm-preserving orthogonalization remains superior to suitably tuned existing polar optimizers. Moreover, BP+NPO also improves performance, suggesting that some benefits are unrelated to the proposed feedback-specific instability mechanism. Finally, although ImageNet accuracy improves substantially, the gap to BP remains large (25.8% versus 79.9%), and no wall-clock or memory comparison demonstrates a practical computational advantage. I am not expecting a method to match BP all of a sudden (and I would be skeptical about it as BP had 40 years to "mature"), but given that avoiding sequential backpropagation is a central motivation, I would have liked to see some evidence of the actual computational advantages of IsoFDFA. Wall-clock time and peak memory comparisons, or at least a theoretical analysis of computational cost (FLOPs, sequential depth, memory requirements), would help establish the practical benefits of the proposed approach over BP.

I am open to revise my score if those weaknesses were adressed.

Questions for authors
  • Would IsoFDFA retain its advantage over PolarAdamW-style alternatives under comparable learning-rate and update-scale tuning? This seems particularly important given that norm preservation is the main algorithmic distinction.

  • Can the authors better isolate the benefits specifically attributable to stabilizing learned feedback, rather than to the general optimization improvements also observed with BP+NPO?

3 comments
AuthorsOpenReview ↗

Response to Reviewer r87x (1/2)

Thank you for your positive assessment of the empirical results and the ablations, and for pointing out concrete directions for improvement. We have addressed each of your comments and questions as follows.

W1

"Limited novelty relative to closely related work" (Boeshertz et al., 2026; PolarAdamW; dynamical isometry, Lipschitz-constrained networks, FOTON).

Status of the two papers. The ICLR 2027 Reviewer Guide (FAQ) does not require comparison with papers published at a peer-reviewed venue on or after July 17, 2026, or available only on arXiv. Boeshertz et al. (NeurIPS 2026) and PolarAdamW (arXiv only) fall under these. Both are nevertheless the works closest to ours, and the revision compares with both.

Boeshertz et al. (2026). We share the observation that low-dimensional updates hinder backpropagation-free training and that orthogonalizing the update helps. The differences are the following.

  • Learning rule and mechanism. They study FA with fixed random feedback; we study FDFA, where the feedback itself is estimated. The part of our mechanism in which skewed updates degrade the feedback estimate (Section 2.2 (c)–(e)) has no counterpart with fixed feedback, whereas the inflation caused by persistent skewed updates ((a), (b)) can also occur there. In a new control (Appendix C.11, Table 11), NPO improves DFA on all four models as well, but less than FDFA, and DFA + NPO stays below IsoFDFA.
  • Models. Their experiments use convolutional models. In our Table 1, FDFA trains ResNet-18 to some extent without NPO (44.3% on CIFAR-100); it fails on ViT, MLP-Mixer, and Swin-T (2–5%), which, together with DeiT-S on ImageNet, are our targets.
  • Size of the update. They use Muon with its RMS matched to AdamW through an empirical constant (Liu et al., 2025) and grid-search the learning rate for each model. NPO inherits the update size of the base optimizer for each matrix and step, needs no constant, and applies to any base optimizer (Section 3, Appendix C.6).

We also added Muon with a learning-rate sweep (Table 3). Tuned on FDFA, it is comparable to IsoFDFA, but at the learning rate chosen with BP it loses about 40 points on MLP-Mixer and ViT and 6.6 points on Swin-T (NPO: 0.5–1.7 points on Swin-T).

PolarAdamW (Zhang, 2026). NPO uses the same polar factor when applied to the same matrix, and the main difference is how the size of the update is set (Section 2.4; PolarAdamW also applies the weight decay outside the polar map). We do not claim a new polar transformation, and the optimality of NPO is cited as a classical result (Fan & Hoffman, 1955). The comparison is given under W2 / Q1.

Contributions. (i) A hypothesis for the mechanism by which single-pass FDFA fails on modern vision models, now derived in a simplified model and supported by diagnostics; (ii) the requirement that the update size be inherited from the base optimizer, so that the learning rate chosen with plain BP can in most cases be reused; and (iii) the first training of ViT-type models on ImageNet with learned feedback and a single forward-mode pass.

Dynamical isometry, Lipschitz constraints, FOTON. We added a paragraph to Section 3. These works control the spectrum of the weights or of the Jacobians, whereas we intervene on the spectrum of the updates. Orthogonalizing the weights instead of the update also improves FDFA substantially (ablation (ii): 23.6–37.8%), consistent with this literature, but stays below NPO on all three models (45.9–50.9%), presumably because the weights are corrected only after a skewed update has been applied.

Q2

"Can the authors better isolate the benefits specifically attributable to stabilizing learned feedback […]?"

We added a control that applies NPO to DFA, which does not learn its feedback (Appendix C.11, Table 11). The gain from NPO is in the order BP < DFA < FDFA on all four models (−1.0 to +11.1, +7.9 to +34.6, and +13.2 to +48.9 points). We interpret this as three effects: the gain of the orthogonalized update itself (also under BP), limiting the inflation (under DFA and FDFA), and the recovery of the feedback estimate (under FDFA only). These effects are not additive, so we cannot decompose the gain quantitatively. Details are given in the response to the common question in our response to all reviewers.

AuthorsOpenReview ↗

Response to Reviewer r87x (2/2)

W2 / Q1

"Would IsoFDFA retain its advantage over PolarAdamW-style alternatives under comparable learning-rate and update-scale tuning?"

We carried out this comparison (Section 4.4, Table 3, Appendix C.6; the four CIFAR-100 models, three seeds). NPO and PolarAdamW differ mainly in the effective learning rate, so with the learning rate tuned on FDFA, PolarAdamW reaches an accuracy comparable to that of IsoFDFA (Figure 5). The difference is whether that learning rate can be taken from BP. The revision states this as one of our two requirements (Section 2.3): IsoFDFA trains stably only in a narrow range of learning rates, so reusing the one tuned with BP matters in practice. Table 3 shows that NPO meets this requirement in most cases.

  • NPO: For Adam, RMSProp, and AdamW, the best learning rate of plain BP equals the best one of IsoFDFA or is one grid step away in all nine settings of ResNet-18, ViT, and Swin-T, and using it costs at most 6 points except for ViT with AdamW (37 points). MLP-Mixer is an exception (a loss of about 20 points): the accuracy of BP hardly depends on the learning rate and gives no guidance.
  • PolarAdamW: The best learning rate of BP is ten times that of FDFA on all four models, and using it costs 13–41 points.

The reason is given in Section 2.4 and Appendix C.6. For Adam-type optimizers, the step size per unit learning rate (ρ) is nearly the same under plain BP and IsoFDFA (see Figure 6), so NPO takes steps of the same size as plain BP. In PolarAdamW and Muon, the polar update is built into the optimizer, so the BP run used to choose the learning rate also uses it and favors larger learning rates; this is why each is compared with its own BP. Converting the learning rate of plain BP into that of a fixed-scale polar update would require the factor ρ√min(m,n) (Section 2.4), which is exactly the rescaling that NPO performs.

W2

"Wall-clock time and peak memory comparisons, or at least a theoretical analysis of computational cost (FLOPs, sequential depth, memory requirements), would help establish the practical benefits of the proposed approach over BP."

Advantage of FDFA-type methods. Under BP, the update of each layer waits for the error to propagate backwards (update locking; Jaderberg et al., 2017). DFA (Nøkland, 2016) and FDFA (Bacho & Chu, 2024) deliver the output error directly to each layer: only the L forward blocks are sequential (BP: L forward, then L backward), and once the output error is available, the block updates are independent and can be computed in parallel. This suits hardware in which layers learn locally and in parallel; DFA has been used to train large Transformers on optical hardware (Wang et al., 2026).

IsoFDFA inherits this advantage at a small additional cost. NPO is computed for each weight matrix from its own update alone, so the block updates remain independent. It adds 10–15% to the step time of FDFA and no peak memory (Table 7 in Appendix C.7; CIFAR-100, batch size 128, one RTX A6000).

ModelFDFA (ms)IsoFDFA (ms)Peak memory (GiB, both)
ResNet-18234.8269.6 (+15%)11.1
MLP-Mixer535.2602.0 (+12%)17.0
ViT193.5220.0 (+14%)8.9
Swin-T1156.11271.9 (+10%)42.1

The cost of NPO, O(K m n min(m, n)) for an m × n matrix with K iterations, depends on neither the batch size nor the number of tokens (cost estimate in Appendix C.7).

Aim of this work, and comparison with BP. Our aim is to make the modern architectures that FDFA could hardly train trainable while keeping the advantages of FDFA; we therefore evaluated the cost relative to FDFA. Our implementation executes the blocks sequentially on a single GPU, so the advantage of parallelism does not appear there: a step of IsoFDFA takes 2.0–2.7 times the time and 2.7–3.9 times the memory of BP (FDFA: 1.9–2.4 times the time; Appendix C.7, conclusion). Since the block updates, including NPO, are independent of each other, we expect a computational advantage over BP when they are executed in parallel; verifying this is future work.

"[…] the gap to BP remains large (25.8% versus 79.9%) […]"

What we would like to emphasize is that something that could not be trained at all has become trainable. FDFA stays at chance level on DeiT-S in all three seeds (top-1 0.10 ± 0.01%), and IsoFDFA raises this to 29.2 ± 1.1% by intervening only on the spectrum of the updates (three seeds, without pooling; the 25.8% of the submitted version was a single seed with pooled feedback). To our knowledge, this is the first time a vision Transformer has been trained on ImageNet to non-trivial accuracy with learned feedback and one forward-mode pass per step, without local losses or backpropagation across blocks. The remaining gap to BP is stated as a limitation in the conclusion; as you note, we do not expect a backpropagation-free method to match BP right away.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer r87x

Dear Reviewer r87x,

Thank you again for your detailed review and for pointing out concrete directions for improvement. In the revision, we added the comparison with PolarAdamW and Muon under learning-rate tuning (Table 3), a control that applies NPO to DFA to isolate the benefit specific to learned feedback (Table 11), and measurements of step time and memory (Table 7), and we extended the discussion of related work (Section 3). We believe that these address the weaknesses you raised.

Since the discussion period closes on November 18 (AoE), we would be very grateful to hear whether, in your view, our response and the revised manuscript resolve your concerns. We will gladly answer any remaining questions.

Thank you very much for your time and consideration.

Reviewer_BQddOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why Forward Direct Feedback Alignment (FDFA), a backpropagation-free method that learns feedback matrices with forward-mode differentiation, fails to train modern vision architectures such as ViT, MLP-Mixer, and Swin. The authors hypothesize a failure mechanism: since all samples share the same feedback matrix, the updates concentrate on a few singular directions, which inflates the weights anisotropically, creates a large sensitivity imbalance across feedback points, and degrades single-pass feedback estimation, forming a vicious cycle. To break this cycle, they propose norm-preserving update orthogonalization (NPO), which replaces the optimizer update with its polar factor rescaled to the original Frobenius norm. The resulting IsoFDFA trains MLP-Mixer, ViT, and Swin on CIFAR, matching a BP baseline with the same learning rate on MLP-Mixer and ViT.

Critical strengths
  1. A clear diagnosis of why FDFA fails, supported by careful diagnostics. The paper does not simply propose a fix, but traces the failure step by step, from skewed updates to weight anisotropy, sensitivity imbalance, and degraded feedback estimation, and shows that a single intervention on the update spectrum suppresses every stage. The improvement over FDFA is large (about 41 to 49 points on CIFAR-100 for non-convolutional models), and the ablations convincingly show that preserving the update norm is important.
Critical weaknesses
  1. Limited practical significance. The main motivations of backpropagation-free training are biological plausibility and hardware in which layers learn locally and in parallel, but the paper does not show any practical benefit in either direction. No wall-clock time, memory, or parallelism measurements are reported, even though single-pass FDFA still requires a forward-mode pass, within-block differentiation, the maintenance of feedback matrices, and eight Newton–Schulz iterations per weight matrix at every step. Meanwhile, the accuracy is still far from practical use: 25.8% top-1 on ImageNet compared with 79.9% for BP. As a result, it remains unclear what IsoFDFA can be used for today, and in which setting a practitioner would choose it over standard backpropagation.

  2. The claim of matching backpropagation relies on weak baselines, and the technical novelty is limited. The BP baseline on CIFAR uses a constant learning rate of 1e-4 for 100 epochs with minimal augmentation, reaching only about 42% on CIFAR-100 with ViT and MLP-Mixer, which is far below what these architectures typically achieve with a standard training recipe. "Matching BP" under this setting therefore says little about how close IsoFDFA is to well-tuned backpropagation; on ImageNet, where the standard DeiT recipe is used, the gap is more than 50 points. Moreover, NPO is essentially the polar-factor orthogonalization used in Muon and PolarAdamW, with the Frobenius norm of the raw update restored. NPO also improves BP itself on these architectures, so it is not fully clear how much of the gain is specific to the proposed failure mechanism rather than a generally better optimization step.

Questions for authors
  1. What is the intended practical use of IsoFDFA? Is there any setting, for example on specific hardware or in a parallel training scenario, where it already offers an advantage over standard backpropagation in time, memory, or energy?
  2. What are the wall-clock time and memory per training step of IsoFDFA compared with BP, including the forward-mode pass, feedback-matrix updates, and Newton–Schulz iterations?
  3. How does IsoFDFA compare with a well-tuned BP baseline on CIFAR, using a standard training recipe with a learning-rate schedule and stronger augmentation?
  4. Since NPO also improves BP on these architectures, how much of the gain of IsoFDFA over FDFA comes from resolving the proposed failure mechanism rather than from a better optimizer in general?
3 comments
AuthorsOpenReview ↗

Response to Reviewer BQdd (1/2)

Thank you for your positive assessment of the diagnosis of the failure mechanism and of the ablations. We have addressed each of your comments and questions as follows.

W1 / Q1

"Limited practical significance. […] it remains unclear what IsoFDFA can be used for today […]." / "Is there any setting […] where it already offers an advantage over standard backpropagation in time, memory, or energy?"

Positioning of this work. The aim of this work is to make the modern architectures that FDFA could hardly train trainable while keeping the advantages of FDFA. The original FDFA paper (Bacho & Chu, 2024) states that FDFA resolves backward locking and the weight transport problem, the two issues usually raised against the biological plausibility of BP, and that it is a local rule compatible with neuromorphic hardware. IsoFDFA changes neither the estimation of the feedback nor the delivery of the error, and NPO is computed locally for each matrix, so it inherits these properties. On the hardware side, delivering the error directly to each layer requires no sequential backward propagation across blocks and allows the blocks to be updated locally and in parallel, which suits hardware in which layers learn locally and in parallel; DFA has been used to train large Transformers on optical hardware (Wang et al., 2026). With fixed feedback, however, the accuracy falls short of BP, and FDFA, which learns the feedback, could hardly train ViT-type models (2–5% on CIFAR-100, chance level on ImageNet). We identified a cause of this failure and showed that an intervention on the spectrum of the updates alone makes training succeed.

Accuracy on ImageNet. What we would like to emphasize is that something that could not be trained at all has become trainable. FDFA stays at chance level on DeiT-S in all three seeds (top-1 0.10 ± 0.01%), and IsoFDFA raises this to 29.2 ± 1.1% (Table 2; the 25.8% of the submitted version was a single seed with pooled feedback). To our knowledge, this is the first time a vision Transformer has been trained on ImageNet to non-trivial accuracy with learned feedback and one forward-mode pass per step, without local losses or backpropagation across blocks. The remaining gap to BP is stated as a limitation in the conclusion.

Where it can be used today (Q1). In the current implementation on standard GPUs, there is not yet a setting in which IsoFDFA is faster or more memory-efficient than BP. The blocks are executed sequentially on a single GPU, so the advantage of parallelism does not appear, and a step of IsoFDFA takes 2.0–2.7 times the time and 2.7–3.9 times the memory of BP (Table 7). We did not measure energy consumption. Since the block updates, including NPO, are independent of each other, we expect a computational advantage over BP when they are executed in parallel; verifying this is future work.

To summarize: IsoFDFA inherits the structural advantages of FDFA (no sequential backward propagation across blocks, local updates) as they are, and the cost hardly increases over FDFA (the step time increases by 10–15%, and the memory is unchanged). In addition, as the revision makes explicit, NPO inherits the size of the update from the base optimizer, so in most settings the learning rate chosen with plain BP can be used as it is (Table 3), which saves the learning-rate search on FDFA. Most importantly, we identified one cause of the failure of backpropagation-free training on modern models and removed it. We believe that making modern models trainable where they previously were not is a necessary step toward practical backpropagation-free training.

Q2

"What are the wall-clock time and memory per training step of IsoFDFA compared with BP, including the forward-mode pass, feedback-matrix updates, and Newton–Schulz iterations?"

We added these to Appendix C.7 (Table 7). They are measured on CIFAR-100 with batch size 128 on one RTX A6000 (mean of three measurements of 50 consecutive steps after a 10-step warm-up; the time includes data loading).

ModelBP (ms)FDFA (ms)IsoFDFA (ms)BP (GiB)FDFA / IsoFDFA (GiB)
ResNet-1899.1234.8269.62.911.1
MLP-Mixer282.7535.2602.06.217.0
ViT94.0193.5220.02.48.9
Swin-T624.21156.11271.913.242.1

A step of FDFA includes the forward-mode pass, the differentiation within blocks, and the update of the feedback matrices, and takes 1.9–2.4 times the time and 2.7–3.9 times the memory of a step of BP. NPO with eight Newton–Schulz iterations adds 10–15% to the time of FDFA and leaves the memory unchanged. Thus, most of the additional cost of IsoFDFA is that of FDFA itself, and the part due to NPO is small.

AuthorsOpenReview ↗

Response to Reviewer BQdd (2/2)

W2 / Q3

"The claim of matching backpropagation relies on weak baselines […]." / "How does IsoFDFA compare with a well-tuned BP baseline on CIFAR […]?"

The training setting of Table 1 (Adam, learning rate 10^-4, 100 epochs) follows the original FDFA paper (Bacho & Chu, 2024); all methods and models share it, with no per-method tuning, including for BP. As you point out, a comparison under a recipe tuned for BP is also important, so we added one with the learning rate tuned for each method (Appendix C.8, Table 8): AdamW (weight decay 0.05), a cosine schedule with a 5-epoch warm-up, 200 epochs, RandAugment, random erasing, Mixup / CutMix, and label smoothing on CIFAR-100. The final-epoch accuracy at the best learning rate of each method (mean over three seeds) is as follows.

MethodResNet-18MLP-MixerViTSwin-T
BP79.471.067.075.5
FDFA55.42.65.32.3
IsoFDFA63.052.248.653.5

Under the recipe tuned for BP, the gap to BP widens. BP improves by 12–28 points and IsoFDFA by 1–6 points, so the gap becomes 16–22 points. This is not a matter of how the learning rate is chosen: BP benefits more from this recipe (schedule, strong augmentation, longer training), and even against BP at the learning rate that is best for IsoFDFA, the gap is 14–22 points. The accuracy "close to BP" of Table 1 therefore holds in the setting of Table 1 and not under this recipe; this is stated in Appendix C.8 and Section 4.3.

The claims of this work, on the other hand, are unchanged under this recipe. FDFA stays at 2.3–5.3% on the three non-convolutional models, and IsoFDFA reaches 48.6–53.5%. The difference of 43.3–51.2 points is larger than in the setting of Table 1 (40.9–48.9 points). Moreover, the recipe improves BP by 12–28 points while FDFA remains untrainable: the failure of FDFA is not due to a weak recipe and is not resolved by the schedule and data augmentation of this recipe. This is consistent with our explanation that its cause lies in the skew of the updates. IsoFDFA itself also gains 1–6 points from this recipe.

W2

"NPO is essentially the polar-factor orthogonalization used in Muon and PolarAdamW, with the Frobenius norm of the raw update restored."

That NPO uses the same polar factor as Muon and PolarAdamW is stated in Section 2.4 of the revision. The difference lies in how the size of the update is set, and this difference is essential to the aim of this work.

The two requirements. The revision makes explicit what we require of the transformation of the update (introduction, Section 2.3): it should remove the skew of the update, and it should keep the size of the update as set by the base optimizer, so that the learning rate need not be chosen anew. NPO is designed to meet both; preserving the norm is not an add-on but the second requirement itself.

Fixed-scale polar updates do not meet the second requirement. Muon and PolarAdamW set the size of the update by the shape of the matrix alone, and the polar update is built into the optimizer. The BP run used to choose the learning rate then also uses it and favors larger learning rates, so the learning rate does not carry over to FDFA: with PolarAdamW it is ten times too large on all four models (a loss of 13–41 points), and with Muon the loss is about 40 points on MLP-Mixer and ViT (Table 3).

NPO also meets the second requirement in most cases. NPO is applied afterwards to the output of an optimizer, so the optimizer (Adam, RMSProp, AdamW) and the learning rate chosen with plain BP can be used as they are. In all nine settings of ResNet-18, ViT, and Swin-T, the best learning rate of BP equals the best one of IsoFDFA or is one grid step away (Table 3; MLP-Mixer is an exception). Fixed-scale polar updates reach a comparable accuracy if the learning rate is searched again on FDFA (Appendix C.6), but NPO makes that search unnecessary in most cases.

Q4

"[…] how much of the gain of IsoFDFA over FDFA comes from resolving the proposed failure mechanism rather than from a better optimizer in general?"

We added a control that applies NPO to DFA, which does not learn its feedback (Appendix C.11, Table 11). The gain from NPO is in the order BP < DFA < FDFA on all four models (−1.0 to +11.1, +7.9 to +34.6, and +13.2 to +48.9 points). We interpret this as three effects: the gain of the orthogonalized update itself (also under BP), limiting the inflation (under DFA and FDFA), and the recovery of the feedback estimate (under FDFA only). These effects are not additive, so we cannot decompose the gain quantitatively. Details are given in the response to the common question in our response to all reviewers.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer BQdd

Dear Reviewer BQdd,

Thank you again for your careful review and for your positive assessment of the diagnosis of the failure mechanism. In the revision, we added measurements of step time and memory (Table 7), a comparison with BP under a recipe tuned for BP (Table 8), an explicit comparison with Muon and PolarAdamW (Section 2.4, Table 3), and a control that applies NPO to DFA to distinguish the proposed mechanism from a general optimization gain (Table 11); the conclusion now states the practical positioning and the limitations (Section 5). We believe that these address the concerns you raised.

Since the discussion period closes on November 18 (AoE), we would be very grateful to hear whether, in your view, our response and the revised manuscript resolve your concerns. We will gladly answer any remaining questions.

Thank you very much for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers (1/2)

We thank all reviewers for their careful reading and constructive comments. We have revised the manuscript accordingly; changes from the submitted version are shown in blue. Section, table, and figure numbers below refer to the revised version. The main changes are listed here, and individual questions are answered in the responses to each reviewer.

  • Theoretical clarification of the failure mechanism (Section 2.2, Figure 2, Appendix A). The failure mechanism is the one proposed in the submitted version; the revision makes it theoretically explicit. Section 2.2 now divides it into five stages (a)–(e) and states the relation at each stage as Propositions 1–5 in a simplified model (proofs in Appendix A). To make this easier to follow, we also added a figure of the mechanism (Figure 2). The theory concerns a simplified model; that the cycle closes in actual training is a hypothesis, which we examine with diagnostics (Section 4.2) and intervention experiments (Section 4.5). [PGKb Q1, r87x Q2, BQdd Q4, XjiF Q3]
  • Scope of the diagnostics (Appendix C.4, Figure 4). The diagnostics of Figure 3 are now also shown for MLP-Mixer and Swin-T. [PGKb Q6, XjiF W1]
  • Relation to Muon and PolarAdamW (Section 2.4, Table 3 in Section 4.4, Appendix C.6). We added a section that makes the relation between NPO and Muon / PolarAdamW explicit. They use the same polar factor, and the main difference is how the size of the update is set. The revision states that not having to choose the learning rate anew is one of our two requirements, alongside removing the skew (Section 2.3), and confirms experimentally that the learning rate chosen with plain BP can be reused in most cases. [r87x W1, Q1; BQdd W2]
  • Related work (Section 3, Appendix C.11). We added a discussion of dynamical isometry, Lipschitz-constrained networks, and FOTON, and of the relation to second-order preconditioning (K-FAC, Shampoo), and we describe the differences from Boeshertz et al. (2026) in more detail. We also added a control experiment that applies NPO to DFA with fixed feedback (Table 11). [r87x W1; PGKb W1, Q2]
  • Second-order preconditioning and partial equalization (Appendices C.9, C.10). We added an experiment that combines Shampoo with accumulated statistics with FDFA (Table 9), and experiments that vary the degree of equalization or equalize only the top singular directions (Table 10). [PGKb W1, Q2]
  • ImageNet results (Section 4.3, Table 2). BP, FDFA, and IsoFDFA are now reported as mean ± standard deviation over three seeds. IsoFDFA without pooling of the feedback matrices is the main result (top-1 29.2 ± 1.1%; FDFA 0.10 ± 0.01%), shown next to the pooled variant (24.9 ± 2.7%) and the memory of the feedback matrices. [XjiF W2, Q1, Q2; PGKb Q3; r87x W2; BQdd W1]
  • Setting of BP on ImageNet (Appendix C.3). We added BP trained with the same setting as IsoFDFA (learning rate, weight decay on LayerNorm): top-1 78.6 ± 0.3%, compared with 79.9% for BP in the standard setting. [PGKb Q3]
  • Comparison under a recipe tuned for BP (Appendix C.8, Table 8). We added a comparison of BP, FDFA, and IsoFDFA under a training recipe tuned for BP (AdamW, cosine schedule, RandAugment, Mixup / CutMix, etc., 200 epochs) with the learning rate tuned for each method. [BQdd W2, Q3]
  • Ablation (Section 4.5, Table 4). The ablation now uses three seeds and has a Swin-T column. The first row gives the reference values of BP and BP + NPO. The comparison with fixed-scale polar updates (PolarAdamW, Muon) was moved to the learning-rate sweep experiment (Section 4.4, Table 3); accordingly, the Muon row and the row "NPO without norm preservation (Muon scale)" of ablation (i) in the submitted version were removed. [XjiF W1, PGKb Q5]
  • Training curves of BP (Figure 1). The training curves of BP were added to Figure 1. [PGKb Q4]
  • Measured computational cost (Section 4.5, Table 7 in Appendix C.7). We added measurements of the time per step and the peak memory (BP, FDFA, IsoFDFA, and 1 / 2 / 5 / 8 Newton–Schulz iterations), together with a cost estimate. [r87x W2; BQdd W1, Q2; XjiF W3]
  • Practical positioning (Section 5). The conclusion now states the practical positioning and the limitations. [BQdd W1, Q1]
  • Other revisions of the main text (abstract, Sections 1, 2.1, 2.3, 5). In line with the changes above, we revised the wording of the main text. The abstract and the introduction now follow the five stages of Section 2.2 and list three contributions (mechanism, method, evaluation). The comparison with BP on CIFAR is described as "comes close to" BP instead of "matching or exceeding" it. Section 2.3 first states the two requirements on the transformation of the update and then introduces NPO as the operation that meets them. The conclusion is organized as conclusion, limitations, and future work. To fit within ten pages, we adjusted the main text as a whole and moved derivations and experimental details to the appendix.
AuthorsOpenReview ↗

Response to All Reviewers (2/2)

Reviewers r87x (Q2), BQdd (Q4), and XjiF (Q3) raised a common question, which we answer together here.

"How much of NPO's benefit comes from general optimization gains, given that BP+NPO also improves over BP?" (XjiF Q3)

NPO has both a general optimization benefit and an effect specific to FDFA. In FDFA, this specific effect adds to the general gain. We discuss them below.

1. Three effects. In the revision, we derive the failure mechanism of FDFA theoretically in a simplified model (Section 2.2, Appendix A). This makes it clear that the effect of NPO can be divided into the following three parts.

  • (i) The gain of the orthogonalized update itself. Making the update isotropic in most cases helps the optimization regardless of how the error is delivered. This also operates under BP and is a general optimization gain.
  • (ii) Limiting the inflation. When the feedback is not aligned with the Jacobian, the same skewed update persists and the weights and activations inflate (Section 2.2 (a), (b)). NPO limits this. Under BP, the component along the common input direction decays under the condition given in Appendix A (remark after the proof of Proposition 1), and we observe no such inflation (Figure 3).
  • (iii) Recovery of the feedback estimate. FDFA estimates where to deliver the error from perturbations. When the activations inflate, a perturbation at a point before a normalization layer hardly reaches the output, and the estimate is buried in cross-point noise (Section 2.2 (c)–(e), Propositions 3–5). By limiting the inflation, NPO also prevents this degradation of the estimate. This operates only in FDFA, where the feedback is estimated.

We therefore expect only (i) to operate under BP, (i) and (ii) under DFA with fixed feedback, and all of (i)–(iii) under FDFA.

2. Effect of NPO on the three methods. We checked this view by applying NPO to BP, DFA, and FDFA. The experiment that applies NPO to DFA is new in the revision (Appendix C.11, Table 11). The table below gives the accuracy (%) on CIFAR-100 without NPO → with NPO, and the difference (mean over three seeds; Tables 1 and 11).

Error deliveryExpected effectsResNet-18MLP-MixerViTSwin-T
BP (exact error)(i)67.2 → 66.2 (−1.0)42.8 → 53.9 (+11.1)41.7 → 44.3 (+2.6)56.4 → 61.8 (+5.4)
DFA (fixed feedback)(i), (ii)39.6 → 47.5 (+7.9)3.6 → 24.1 (+20.5)3.9 → 19.2 (+15.3)2.4 → 37.0 (+34.6)
FDFA (learned feedback)(i), (ii), (iii)44.3 → 57.5 (+13.2)2.0 → 50.9 (+48.9)5.0 → 45.9 (+40.9)2.8 → 49.7 (+46.9)

The gain is in the order BP < DFA < FDFA on all four models, which is consistent with the view above. NPO mostly raises the accuracy of BP and DFA, so a general optimization gain is indeed present. On top of it, the gain is largest for FDFA, and we believe that the recovery of the feedback estimate adds to it. We did not measure the inflation itself under DFA.

3. Evidence for effects (ii) and (iii). They are supported by the following two results.

  • Removing the cross-point estimation noise improves FDFA. With Adam fixed, replacing single-pass estimation by per-point estimation improves the accuracy from 2.0 to 24.0% on MLP-Mixer, from 5.0 to 18.7% on ViT, and from 2.8 to 22.9% on Swin-T (Table 4 (iv)). Per-point estimation removes the cross-point variance term of Eq. (3), so this shows that cross-point noise contributes to the failure of FDFA.
  • The inflation and the imbalance of the sensitivities occur under FDFA and not under BP. In Figure 3 (ViT), the stable rank of the updates of BP without NPO is 3–8, and the weight norm, the activation norm, and the ratio of the sensitivities stay stable throughout training. Under FDFA, the stable rank stays at 1–2 and the ratio of the sensitivities reaches 10^9. Under IsoFDFA, the weight norm returns to a value close to that of BP, and the activation norm and the ratio of the sensitivities return to within one order of magnitude of BP.

4. The tested changes of recipe and optimizer do not make FDFA trainable. Our experiments follow the training setting of the original FDFA paper. Even with a recipe tuned for BP, under which BP improves by 12–28 points, FDFA stays at 2.3–5.3% on the non-convolutional models (Appendix C.8, Table 8). Replacing the optimizer by RMSProp leaves FDFA at 1.9–2.6%, and adding NPO gives 42.1–46.1% (Table 4 (iii)). This shows that the tested changes of the training recipe and the optimizer alone do not resolve the failure of FDFA. What resolves it in our experiments is an intervention on the spectrum, and removing the skew of the update itself works best.

5. Limits of the separation. The results above support the interpretation that NPO suppresses the failure mechanism observed in FDFA. This distinction is, however, qualitative. The three effects are not additive, so these experiments cannot quantify how many points of the gain are due to each.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.