- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes norm-preserving update orthogonalization (NPO) and applies it to FDFA as IsoFDFA. On CIFAR, IsoFDFA substantially improves over FDFA for MLP-Mixer, ViT, and Swin-T. On ImageNet, it reaches 25.8% top-1 accuracy, compared with 0.2% for FDFA and 79.9% for BP.
Critical strengths
1、IsoFDFA substantially improves FDFA on several non-convolutional CIFAR models.
2、 The paper includes useful controls, including BP+NPO, optimizer changes, and per-point feedback.
Critical weaknesses
1、The mechanism diagnostics focus mainly on one ViT/CIFAR-100 setting, so it is unclear how broadly they explain the gains.
2、ImageNet results use one seed and remain far below BP; the effect of pooled feedback at ImageNet scale is not isolated.
3、Performance drops sharply with fewer Newton--Schulz iterations, but the runtime and compute trade-offs are not fully reported.
Questions for authors
1、Do you have multi-seed ImageNet results?
2、How much does pooled feedback affect ImageNet performance?
3、How much of NPO's benefit comes from general optimization gains, given that BP+NPO also improves over BP?
2 comments
Follow-up on Our Response to Reviewer XjiF
Dear Reviewer XjiF,
Thank you again for your careful review and for your positive assessment of the improvement over FDFA and of the control experiments. In the revision, the mechanism diagnostics are also shown for MLP-Mixer and Swin-T (Figure 4), the ImageNet results use three seeds and show the effect of pooled feedback separately (Table 2), the runtime for each number of Newton–Schulz iterations is reported (Table 7), and a control that applies NPO to DFA was added to distinguish the gain specific to FDFA from a general optimization gain (Table 11). We believe that these address the points you raised.
Since the discussion period closes on November 18 (AoE), we would be very grateful to hear whether, in your view, our response and the revised manuscript resolve your concerns. We will gladly answer any remaining questions.
Thank you very much for your time and consideration.
Response to Reviewer XjiF
Thank you for your positive assessment of the improvement over FDFA and of the control experiments. We have addressed each of your comments and questions as follows.
W1
We added two things.
W2 / Q1 / Q2
We now report BP, FDFA, and IsoFDFA on ImageNet with three seeds and show the effect of pooling separately (Table 2).
W3
We added measurements to Appendix C.7 (Table 7) and summarize them in Section 4.5 of the main text (the paragraph on computational cost). They are measured on CIFAR-100 with batch size 128 on an RTX A6000.
The peak memory equals that of FDFA regardless of the number of iterations. Reducing eight iterations to five saves only 2–5% of the step time, and with one or two iterations training does not succeed. The benefit of reducing the number of iterations is therefore small, and we use eight as the default. In comparison with BP, FDFA itself takes 1.9–2.4 times the time and 2.7–3.9 times the memory of BP, and the part added by NPO is small in comparison.
Q3
The effect of NPO can be divided into three parts: the gain of the orthogonalized update itself (which also operates under BP), limiting the inflation (under DFA and FDFA), and the recovery of the feedback estimate (under FDFA only). Indeed, the gain from NPO is in the order BP < DFA < FDFA on all four models (−1.0 to +11.1 points for BP, +7.9 to +34.6 for DFA, and +13.2 to +48.9 for FDFA). We believe that in FDFA the recovery of the estimate, an effect specific to it, adds to the general gain. These effects are not additive, however, so we cannot decompose the gain quantitatively. Details are given in the response to the common question in our response to all reviewers.