acceptodds
Under review as a conference paper at ICLR 2027

Conditioned Direct Feedback Alignment via Activity and Error Geometry

Abstract

Direct feedback alignment (DFA) trains hidden layers with fixed random projections of the output error. Even when this feedback provides useful credit, unequal scales across activity or error directions can distort the local update. We study conditioned DFA (nDFA), a family that adapts established inverse-moment preconditioning to either side of this update. An aligned linear analysis describes how activity conditioning changes spectral learning rates and early-stopping risk. Synthetic experiments show gains from both stabilization and changes in update direction. Confirmatory CIFAR-10 experiments show that activity conditioning, including its FOOF formulation, improves tuned DFA at matched measured training time. Controls support a role for centered correlations beyond the tested mean and diagonal alternatives, while early-only conditioning retains most of the benefit with less training work. Conditioning also benefits backpropagation, and timing interventions do not establish a mechanism specific to random feedback. Error conditioning improves short-budget models and changes update directions, but its small additional improvement under stable training does not survive correction for multiple comparisons. Predicted benefits on image-background benchmarks are not confirmed. These findings establish practical benefits and important limits of conditioning learning with fixed random feedback.

Primary area:
optimization
Keywords:
direct feedback alignment, credit assignment, local learning, preconditioning, activity geometry, biologically inspired learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_S4bkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates conditioned direct feedback alignment (nDFA), a family of learning rules that improves credit assignment in fixed random feedback networks by applying inverse-second-moment conditioning to either the presynaptic activity or the local error signals. Through a linearized spectral analysis and extensive experiments across synthetic and real-world datasets like CIFAR-10, the authors demonstrate that activity conditioning mitigates the distorting effects of high-variance nuisance directions, thereby stabilizing training and improving optimization speed compared to tuned DFA baselines. While activity conditioning provides robust gains and a favorable early-only efficiency tradeoff, the study establishes that error conditioning offers more limited benefits under stable training and that these geometric interventions do not resolve the fundamental stationary mismatch inherent in random feedback learning.

Critical strengths
  1. The paper provides a geometric characterization of DFA failure modes by demonstrating how activity anisotropy weights task-irrelevant nuisance directions, and it proves that nDFA effectively reduces the input-side condition number to reweight these updates toward task-relevant directions.

  2. The authors establish a practical and statistically validated efficiency tradeoff through early only activity conditioning on CIFAR-10, showing that applying inverse-moment operators during initial training phases retains most accuracy gains while significantly reducing the total computational work compared to persistent conditioning.

Critical weaknesses
  1. The authors find that error conditioning improves short budget models but its additional benefit fails to survive the global correction under stable training on CIFAR-10. This outcome leaves a gap between the theoretical family of two factor conditioned DFA and the empirical evidence, as the study does not isolate whether this failure stems from the low rank nature of gated DFA errors or from an inherent limitation of the error side inverse moment normalization when combined with activity conditioning.

  2. The primary empirical hypothesis suggests that nDFA should help most when high variance directions are task irrelevant nuisance, yet this prediction is not confirmed on the image background benchmarks. The failure to transfer synthetic nuisance geometry to real image backgrounds indicates that the spectral learning rate analysis may rely on an idealized task orientation assumption that does not hold for the complex nuisance structure found in naturalistic datasets.

Questions for authors

For weakness 1: Provide further evidence or a more detailed ablation to determine if the fragile estimation of error moments is the primary bottleneck preventing error conditioning from yielding sustained gains in converged models?

For weakness 2: Given that the predicted benefits on image background tasks were not confirmed, how do the authors reconcile the gap between the synthetic spectral gain and the realized performance on real world datasets where nuisance directions may not be as clearly separated?

3 comments

Thank you for identifying these two questions. We designed follow-ups for both. They establish an additional error-side benefit and provide a direct measurement of the geometric difference between the synthetic and image-background tasks.

1. Is error-moment estimation the bottleneck?

Better estimation helps, while the larger study also resolves a benefit with the original estimator. On the stable CIFAR-10 protocol, with 48 paired seeds, adding the error factor early to activity-only conditioning (A) gives:

Error-moment recipeAccuracy gain over A; individual 95% CI
Minibatch moments, as in the paper+0.28 points [0.11, 0.45]
Averaged moments+0.65 points [0.49, 0.82]

The averaged recipe also beats the minibatch rule by +0.37 points. All six accuracy and cross-entropy comparisons pass Holm correction. The minibatch estimate is consistent with the original small positive estimate; both the larger sample and smaller correction family improve sensitivity. This follow-up tests the gain over A, not the original full criterion, which also required a diagonal-error control.

We separated estimation quality from the other recipe changes in two ways. First, at saved model states the minibatch-conditioned direction agrees with a pooled reference at cosine 0.72–0.81, versus 0.93–0.96 for an independent pooled estimate. Second, in a Transformer control that changes only error-moment averaging, averaged moments improve test perplexity from 232 to 189 in 12/12 seeds. This isolates an averaging benefit more directly than the MLP recipe, which also changes calibration and inverse refresh.

The Transformer results extend the error-side benefit beyond the MLP. In the 36M Transformer it lowers DFA test perplexity from 391 with A to 195; the full error factor also beats its diagonal control. At three times the training budget, the corresponding comparison is 264 to 145, with the same absolute early-error window. Concentrated local errors therefore coexist with useful conditioning. These are fixed-budget results, not a claim about converged models or a complete explanation of the mechanism (Fig. 5A–C; Appendix M.1–M.2).

2. Why did the synthetic prediction not transfer to image backgrounds?

We now measure the relevant geometry by redrawing backgrounds while holding each foreground fixed. This separates background variation from foreground-locked variation. Applying the inverse activity operator changes the class-mean-to-background energy ratio as follows:

TaskRatio after conditioning / ratio before
Synthetic nuisance×3.72
Synthetic task-aligned control×0.09
MNIST + backgrounds×0.93
Fashion-MNIST + backgrounds×0.41

The operator enriches this task proxy relative to nuisance in the synthetic nuisance regime, but not in the image mixtures. Their class means overlap high-variance directions, so visible background does not by itself establish the task orientation assumed by the spectral analysis. This gives a measurable boundary to the prediction (Fig. 5E; Appendix M.4).

These three-seed measurements use class means as a task proxy, so they support a geometric interpretation rather than a complete causal account. The denser-checkpoint audit agrees with the original benchmark outcome. The revision now provides both the confirmed error-factor gains and the geometry measurements requested in your review.

AuthorsOpenReview ↗

additional tests

Following our earlier response, we would like to add new evidence on your two main questions. Studies undertaken in response to Reviewer Jpt2's follow-up now include the diagonal-error control and a direct geometry intervention. We appreciate your role in motivating this work, which clarifies both the value of error conditioning for fixed-feedback learning and the role of background geometry. The new studies are in Appendix C and Fig. 7; main Fig. 5A,E highlight the error controls and prospective orientation comparison.

Additional evidence on error-moment estimation. A new, separately frozen study on 48 fresh seeds adds the diagonal-error control to the earlier comparison against activity conditioning alone (A). With averaged moments, it meets the original five-part criterion under its own correction family. The full error factor beats A by +0.70 points [95% CI 0.52, 0.87] and a diagonal error factor, tuned over ten settings, by +0.82 [0.65, 0.99]; no diagonal-factor gain is statistically established. Without re-selecting the error settings, the gain over A is +1.05 [0.89, 1.22] at batch 128 and +0.73 [0.56, 0.91] at batch 512. This establishes an additional benefit from the full error factor under the tested averaged-moment recipe and across these batch sizes.

Additional evidence on background geometry. Your question motivated a direct intervention: we rotated every background relative to the class-mean subspace with its spectrum, foregrounds and labels unchanged. The intervention establishes that background orientation affects the benefit of conditioning: in the paper's mixtures conditioning gains 2.7 and 2.1 accuracy points when the leading background variance masks the class-mean directions, and loses 0.2 and 0.3 when it is orthogonal to them. The interaction has the same sign in all 20 seeds; BP shows the same dependence. We then recorded the sign of this dependence as a prediction before training two further mixtures, and it was confirmed in all eight declared tests (20 fresh seeds); BP again shows the same dependence there. This is a prospective replication across mixture strengths using the same images and rotations.

The intervention sharpens our earlier interpretation. The ratio predicts the opposite ordering, so we now retain it as a descriptive measurement rather than the predictive boundary proposed in our first reply. Input ablations show that the advantage of conditioning remains detectable with the class-mean component of the input removed, including in all four new cells. The stronger prediction of no advantage with that component alone held in two of four. This supports a more precise hypothesis about usable signal beyond class means, while leaving the full mechanism open. Your questions helped us move from a plausible account to controlled evidence about when conditioning aids local learning.

Reviewer_S4bkOpenReview ↗

Final Decision for this submission:

Thank you for the rigorous follow-up experiments and the detailed response!

  • On W1, it shows that off-diagonal error structure is useful, and that the earlier null result was not an inherent limitation of error-side conditioning.

  • On W2, the background-rotation intervention shows that the benefit of conditioning depends on how background variance is oriented relative to the class-mean subspace. The revised abstract states clearly that the predicted image-background benefits are not confirmed. I find this scoping appropriate.

My concerns have been addressed, and I will raise my rating to 3. For confidence, I do not have direct research experience in this field so I will keep 2. Hope you may understand.

Reviewer_Jpt2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks into conditioned Direct Feedback Alignment (nDFA), which uses inverse-second-moment preconditioning on presynaptic activity and local error signals in DFA. The goal is to reduce anisotropic update directions and improve learning using fixed random feedback. The major contributions are: A linear-Gaussian analysis of how activity conditioning influences spectral learning rates and early stopping behaviour. Synthetic experiments to investigate the effects of nuisance directions, stabilisation, and directional reweighting. CIFAR-10 experiments show improvements over tuned DFA with matched training-time budgets. Ablations look at centred correlations, conditioning schedules, and activity versus error-side preconditioning. The results show that activity conditioning provides consistent benefits, including an efficient early-only variant. However, additional benefits from error conditioning have not been statistically established under stable training, and predicted improvements on image-background benchmarks have not been validated.

Critical strengths
  • The paper provides extensive synthetic and real-data experiments with carefully designed controls. The CIFAR-10 evaluation includes tuned DFA, BP, and FD-DFA baselines, matched measured training time, and ten paired seeds. Holm correction across 21 comparisons further strengthens the statistical evidence.

  • The separation of activity-side and error-side conditioning provides a useful perspective on approximate credit assignment. Norm-matching experiments demonstrate directional benefits beyond simple update rescaling, while covariance controls highlight the contribution of centered correlations.

  • The early-only conditioning experiments also establish a useful accuracy - computation tradeoff. Importantly, the authors transparently report unsuccessful predictions and limitations, improving the credibility of the findings.

Critical weaknesses

Major Weaknesses

  1. The activity-conditioning operator is largely adapted from established FOOF/KFAC-style preconditioning, rather than introducing a fundamentally new optimization mechanism. The theoretical analysis assumes an aligned linear regime and does not explain how useful feedback alignment emerges. Furthermore, similar improvements under BP suggest that the benefits arise primarily from general preconditioning rather than a mechanism specific to DFA. This limits the theoretical novelty of the contribution.

2.The main practical results are limited to BatchNorm-equipped MLPs on CIFAR-10, without experiments on CNNs, Transformers, or larger datasets. The approach also requires layer-wide moment statistics and matrix operations, introducing computational and memory overhead. Although sample-space solutions reduce some costs, their scalability to larger architectures remains insufficiently demonstrated. These limitations weaken the practical significance of the reported improvements.

Minor Weaknesses

  1. The expected benefits on image-background benchmarks are not confirmed, and some experiments show degraded cross-entropy. Further analysis of real-data nuisance geometry would help explain this discrepancy.

  2. The experiments reveal substantial dependence on BatchNorm, damping, and conditioning schedules. A more systematic analysis of these interactions would help establish when the approach can be reliably applied.

  3. The manuscript is comprehensive but dense, with multiple experimental protocols and damping conventions distributed across the appendices. A more concise presentation of the main findings and practical recommendations would improve accessibility.

Questions for authors
  1. Can the authors provide results on CNNs or deeper architectures to demonstrate that the proposed conditioning remains effective beyond MLPs?

  2. How do memory usage, FLOPs, and wall-clock overhead scale with layer width and inverse-refresh frequency? How do these costs compare with conventional preconditioning methods?

  3. Since conditioning also benefits BP, can the authors provide additional evidence identifying which improvements are specifically attributable to DFA's approximate credit assignment?

  4. What explains the discrepancy between the synthetic nuisance experiments and the unsuccessful image-background predictions? Could layerwise covariance or task-relevant subspace analysis clarify this?

  5. How sensitive are the additional error-side gains to batch size, damping, and normalization? Could adaptive damping improve their stability?

4 comments

Thank you for the five concrete questions. We addressed them with frozen follow-up studies and added the results to Section 4.5, Fig. 5 and Appendix M.

Q1. Does conditioning extend beyond MLPs?

Yes: six-block Transformer language models with LayerNorm, trained from scratch on a WikiText-103 prefix, show matched-time DFA gains at 13M and 36M parameters. Block-DFA supplies fixed random feedback to each block and differentiates exactly within it. With twelve paired seeds, the 36M test perplexities are:

BudgetRaw DFAActivity (A)A + early error factor
20 minutes543391195
60 minutes536264145

Each factor improves cross-entropy in 12/12 seeds and passes its prespecified correction. The error factor operates for the first five minutes at both budgets. A separate pre-vocabulary-feedback study retains the activity gain and a smaller error gain (Table 59). These are fixed-budget gains; BP remains stronger, and their magnitude under Launay et al.'s longer full-corpus recipe is untested.

In a small CNN without normalization, conditioned DFA is +2.4 validation-accuracy points ahead at equal updates. At equal time, the confirmed gain is for BP (+0.39 test points), not DFA. We distinguish these endpoints (Appendix M.3).

Q2. How do time, memory and FLOPs scale?

On an A100 at batch 256, sample-space A costs 1.7–1.9× a raw step through width 8192, with at most 18% more peak allocated memory. At width 8192, the cached dense FOOF implementation costs 5.4×. The advantage depends on batch size: A reaches 3.0× at batch and width 1024. For dense FOOF at width 2048, changing inverse refresh from every step to every 100 steps reduces step time from 26.4 to 8.9 ms. These compare implemented recipes, not the solve alone.

Fig. 5D and Appendix M.5 report the full profiles, estimated FLOPs and operation counts beside Adam, FOOF, KFAC and Shampoo. Sample-space solves avoid width-by-width factor storage when the batch is smaller than the layer. The reported matched-time performance gains already include conditioning overhead.

Q3. What is specific to approximate credit, and what is new beyond FOOF/KFAC?

In these Transformers the DFA-minus-BP gain differences pass Holm correction in three separately frozen studies. A is slightly worse under BP at matched time, while completing fewer updates, and E adds no reliable BP gain. This establishes a credit-dependent benefit at matched time under separately selected settings; it does not imply a universal DFA-specific mechanism. The MLP benefit shared with BP remains in the paper.

The activity operator is FOOF's. The additional factor uses local DFA-error moments, rather than KFAC's Fisher factor, and improves over both A and a diagonal-error control in Transformers. Table 1 now makes these relationships explicit. Appendix M.6 also gives a descent condition without perfect alignment, separating input geometry from effective credit; it is a linear-flow result, not a theory of alignment acquisition or AdamW.

Q4. Why did the image-background prediction fail?

Redrawing backgrounds at fixed foregrounds lets us measure the geometry directly. The inverse activity operator changes the class-mean-to-background energy ratio by ×3.72 in the synthetic nuisance task, versus ×0.93 on MNIST mixtures and ×0.41 on Fashion mixtures. Conditioning therefore enriches this task proxy only in the synthetic nuisance case. These three-seed measurements provide evidence for a mismatch to the model's task-orientation premise, with the inference scoped to the measured class-mean proxy (Fig. 5E; Appendix M.4).

Q5. Error-side sensitivity and adaptive damping.

Averaged early error conditioning improves on A by +0.65 accuracy points [95% CI 0.49, 0.82] in the 48-seed MLP study, with improved cross-entropy. In a Transformer control changing only moment averaging, perplexity improves from 232 to 189 in 12/12 seeds. Thus estimation matters under both BN and LayerNorm. Optimization batch size was held fixed. Damping was validation-selected; DFA selected grid-edge values, so weaker damping is untested. The ridge scales with the moment trace; a further adaptive rule would need separate validation.

For readability, the new main figure summarizes these findings, Table 1 consolidates method conventions, and Appendix M opens with a study registry. The Discussion gives practical guidance: start with activity conditioning and test averaged error conditioning in an early window using validation-selected damping.

I thank the authors for their detailed rebuttal and additional experiments. The new Transformer results, computational profiling, and error-conditioning analysis strengthen the empirical contribution and address several of my concerns.

However, some important limitations remain:

  1. The Transformer results are encouraging, but conditioning does not yield a statistically significant matched-time DFA improvement on CNNs. Moreover, the conditioned Transformer models remain substantially behind BP.
  2. The credit-dependent improvements are interesting, but they do not establish a DFA-specific mechanism. The theoretical analysis still does not explain feedback alignment.
  3. The error-conditioning gains remain sensitive to implementation choices, particularly the feedback convention. Batch-size sensitivity and adaptive damping remain unexplored, and the 48-seed MLP follow-up omits the original diagonal-error control.
  4. Although overhead is better characterised, practical comparisons against optimised KFAC/Shampoo implementations remain missing.
  5. The proposed geometric diagnostic offers a plausible explanation for the negative results, but does not establish a causal mechanism.

Overall, the rebuttal meaningfully strengthens the paper, but the concerns regarding novelty, robustness, and broader practical applicability are only partially resolved. I therefore maintain my original assessment. I will increase my score if the above concerns are addressed.

Thank you for your follow-up and for recognizing the strengthened empirical evidence. Following our earlier response, we ran targeted studies addressing your five remaining concerns. These additional results support a clearer account of when conditioning helps fixed-feedback learning. Appendix C and Fig. 7 give the protocols and results; main Fig. 5A,E highlight the error controls and prospective geometry. Each new training study froze its arms, seeds, endpoint and correction family before confirmation.

1. Matched-time CNN improvement and the gap to BP.

The new CNN experiment establishes a matched-time DFA benefit beyond MLPs. Motivated by the saved validation curves, we fixed a 60-second budget before new training, re-selected each learning rate and used 20 fresh seeds. Conditioned DFA is +0.75 test-accuracy points ahead [95% CI 0.38, 1.13] (Holm-adjusted p = 0.003, 17/20 seeds), with lower cross-entropy. BP shows no gain (−0.04 [−0.30, 0.23]); the DFA-minus-BP difference also passes correction (+0.79 [0.32, 1.26], p = 0.009). Thus the benefit is demonstrated at equal time and depends on the credit rule at this budget. The 200-second result remains unchanged (+0.08 points [−0.27, 0.44]), so the CNN advantage is budget-dependent.

In the three Transformer cohorts, both factors together remove 42–52% of the test cross-entropy gap between raw DFA and BP (Appendix C.1). BP remains stronger. The practical contribution is a substantial improvement under an unchanged fixed-feedback pathway, with conditioning costs included in the budget.

2. Evidence about alignment under fixed feedback.

Your question motivated direct measurements at 148 saved Transformer states and at 176 snapshots along training (four seeds). We measure the cosine between the raw block-DFA direction and the exact gradient, over all 24 conditioned matrices, at networks trained by each rule. At the best 36M checkpoints it is 0.26 against 0.15 after A (+0.106 [0.078, 0.134], 12/12 seeds; 13M: 0.31 against 0.16, 12/12). Alignment rises while the full error factor is active (0.36 at the end of its window). At the first feed-forward matrices, local-error participation rank falls from 158 to 1.0 within a minute of raw DFA training, recovers to 20 under A, and reaches 46 while the error factor is active.

Because the measured direction is raw DFA, this difference concerns the states reached by learning; it is not merely the immediate reweighting of the vector being measured. The results connect error conditioning to less concentrated errors and better-aligned trained states, strengthening the empirical account of approximate credit. They do not derive alignment acquisition or establish rank recovery as its cause. After 3,600 seconds the difference is not resolved (0.17 against 0.14), while the perplexity gain persists.

3. Error-factor controls, batch size and adaptive damping.

The missing diagonal control and batch-size tests are now complete. A separately frozen study on 48 fresh seeds meets the original five-part criterion with averaged moments and its own five-test family. The full factor beats A by +0.70 points [0.52, 0.87] and a diagonal error factor, tuned over ten settings, by +0.82 [0.65, 0.99]. Both accuracy and loss improve. Without re-selecting the error settings, the gain over A is +1.05 [0.89, 1.22] at batch 128 and +0.73 [0.56, 0.91] at batch 512. All thirteen tests in the primary and batch-transfer families pass Holm correction. This supports a useful contribution from off-diagonal error structure under the tested recipe and transfer across batch sizes, rather than a gain restricted to one selected batch.

We also tested spectrum-relative adaptive damping. It preserves the MLP benefit (+0.79 [0.62, 0.96] over A) and reproduces the output-error Transformer arm (perplexity 195.8 against 194.9). Under pre-vocabulary feedback it improves on A (+0.28 nats [0.13, 0.44], 11/12 seeds), but does not remove the higher-loss level or improve reliably on the fixed ridge. Thus the error factor remains useful under both feedback conventions, although its magnitude and consistency depend on the convention. The appendix reports the associated rank-recovery diagnostic. These new tests address the missing controls directly; they do not retrospectively alter the original ten-seed criterion.

Continuing our additional response to your follow-up, we address the optimizer comparison and the geometric mechanism.

4. Direct comparison with reference K-FAC and Shampoo.

We now provide the practical comparison you requested: unmodified reference implementations (K-FAC: Pauloski et al., 2021; Distributed Shampoo: Shi et al., 2023), validation-selected settings and equal measured training time on ten fresh seeds.

Shampoo-DFA improves tuned raw DFA by +2.23 points [1.93, 2.53]. A and Shampoo have no detected accuracy difference (A: 67.8%; Shampoo: 67.9%; A minus Shampoo: −0.12 [−0.43, 0.18]); Shampoo has lower cross-entropy (0.929 against 0.943). FOOF-DFA is the most accurate tested DFA rule (69.45%; +1.57 [1.32, 1.81] over Shampoo-DFA), with higher cross-entropy. At the trained width A uses 0.27 GiB against 1.14 for Shampoo; the step costs are included in the matched-time comparison. Under BP, Shampoo reaches 69.5% against 66.8% for AdamW. K-FAC reaches 66.0%, against 67.1% for AdamW in the same implementation. Its selection remains at grid edges in damping after three extensions in its favour and in refresh period; it is the best setting found in this search.

These comparisons establish where activity conditioning is practically useful alongside a strong optimizer: similar observed accuracy to Shampoo-DFA with less measured memory, while Shampoo offers better predictive loss. Shampoo's success with the same DFA pseudo-gradient also supports conditioning as a useful tool for approximate credit. Our factor-specific controls add knowledge that this optimizer ranking alone cannot provide: which activity and error statistics contribute. Appendix C.5 reports the width, batch-size and refresh-cost measurements, including the limits of the sample-space advantage.

5. Controlled evidence about background geometry.

We agree that the initial diagnostic alone could not establish a mechanism. Your question led to an intervention: rotate the backgrounds' leading variance inside (0°) or orthogonal to (90°) the class-mean subspace, holding their spectrum, total variance, foregrounds and labels fixed. In the paper's mixture conditioning raises test accuracy from 66.1% to 68.7% on MNIST and from 56.7% to 58.8% on Fashion-MNIST at 0°, and does not at 90° (90.2% to 90.0% and 79.2% to 79.0%). The interaction passes correction and has the same sign in all 20 seeds (cross-entropy −0.089 [−0.095, −0.082] on MNIST and −0.061 [−0.071, −0.052] on Fashion-MNIST).

This establishes that changing background orientation changes the benefit in these constructed tasks. We then recorded the direction as a prediction before training two further mixtures (foreground weights 0.25 and 0.45; 20 fresh seeds). All eight declared tests are detected with the predicted sign, and BP again shows the same dependence. At 0° conditioning raises test accuracy from 41.6% to 45.9% and 38.5% to 41.4% at 0.25, and from 78.9% to 80.7% and 67.0% to 68.1% at 0.45; at 90° it does not help. This is prospective evidence across new mixture strengths using the same images and rotations. The two orientations also differ in difficulty. A second, nuisance-dominant mixture is uninformative, because neither rule learns at 0°.

The controlled result also makes the interpretation more precise: the class-mean energy ratio predicts the opposite ordering, so we retain it as a descriptive measurement rather than a validated explanation. The advantage remains detectable after removing the class-mean component (+3.8 to +4.8 points in all four new cells); the stronger prediction of no advantage with that component alone held in two of the four new cells. The evidence therefore supports orientation-dependent benefits and motivates a hypothesis about usable signal beyond class means. It does not establish an exclusive causal mechanism or transfer to independent natural-image benchmarks.

How this changes the assessment. The new studies add the missing diagonal control, batch transfer, adaptive damping, a matched-time CNN gain, reference optimizers and a prospective orientation test. They strengthen the robustness and practical scope of the empirical contribution. We would appreciate your reassessment in light of these direct tests. Appendix C (six studies), Fig. 7 and Tables 28–38 are new; main Fig. 5A,E present the error controls and prospective orientation comparison.

Reviewer_swYdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes conditioned Direct Feedback Alignment, which they call nDFA. DFA's local update is an outer product of the local error and the presynaptic activity, and the authors wrap inverse-second-moment preconditioning around either side of it to address one concrete failure mode: even when random feedback provides a usable descent direction, unequal variance across activity or error directions still distorts the local update. They give three rules — A conditions the activity side only, E the error side only, and K is the two-sided Kronecker form.

I count three contributions. First, they bring the FOOF/KFAC-style activity operator plus a new error-side operator into DFA training with fixed random feedback. Second, under post-alignment, linear-Gaussian assumptions they work out a spectral analysis showing that conditioning essentially reweights each eigendirection's learning rate by its variance, suppressing high-variance directions, with a bound on the conditioning-number gain. Third, the experiments are solid: an eight-cell synthetic suite, historical MNIST and Fashion-MNIST runs, and a preregistered, frozen-protocol ten-seed CIFAR-10 confirmation with paired seeds and a global Holm correction.

The authors draw the boundaries themselves, clearly. Activity conditioning reliably improves a tuned DFA in stressed regimes, gains beyond a scalar step-size change, and passes the statistical bar at matched measured training time; its FOOF and early-only variants also work, the latter saving roughly a third of the training work. Error conditioning helps the short-budget historical runs, but its small stable-training increment fails the multiple-comparison correction, and the authors honestly report this as not establishing an additional benefit. The predicted gains on image-background benchmarks do not reproduce, and the same operator also helps BP, which points to a generic-preconditioning mechanism rather than anything DFA-specific.

Critical strengths

Strength 1: Honest reporting of negative results, with rigor uncommon in this area. The authors employ preregistration, frozen protocols, Holm global correction, and paired seeds to make the question of when and whether conditioning helps fixed-random-feedback learning reliably answerable and clearly bounded. In particular, the negative results — the error-side gain failing correction, the image-background prediction not reproducing, and the mechanism not being DFA-specific — are presented plainly, which is uncommon in top-tier submissions. As a result, readers can accept the positive claim, namely that the activity-side gain is real and exceeds scalar rescaling, without concern for overselling. In biologically-inspired learning, where results depend heavily on implementation details and tuning, evidence that is reproducible, falsifiable, and bounded is more valuable than an improved performance number.

Strength 2: The separation of the activity and error factors yields directly actionable conclusions. The paper's most insightful contribution is decomposing the two factors typically coupled in FOOF/KFAC and testing them separately in the DFA context, yielding the clear finding that the activity factor is robust while the error factor is weak; a mean-preserving diagonal-covariance control demonstrates the independent contribution of centered correlations; and an early-window experiment translates this into a compute-saving recommendation. This factor-level ablation answers not merely whether conditioning helps, but which side, when, and under what geometry, providing concrete, actionable criteria for subsequent work and establishing a baseline for the direction.

Critical weaknesses

Weakness 1: The theoretical contribution is limited; alignment acquisition is not explained, and no DFA-specific mechanism is established. The spectral analysis in Section 2.1 rests on post-alignment, linear-Gaussian assumptions, and its core result — inverse-second-moment reweighting of per-eigendirection learning rates and a conditioning-number bound — is a direct application of results long established in KFAC and natural gradient. The authors concede that the theory does not explain alignment acquisition and does not establish a DFA-specific mechanism. More importantly, the same operator applied to BP is equally effective, indicating that the gain arises from generic preconditioning rather than a DFA-specific law. The paper is therefore relatively weak on theoretical novelty and mechanistic specificity, and somewhat limited in depth for ICLR. This bears directly on the assessment: the contribution is positioned as rigorous empirical characterization plus application transfer, rather than a new theoretical insight.

Weakness 2: Two preregistered hypotheses were not confirmed, narrowing the positive contribution. The two hypotheses explicitly tested — an additional stable-training error-side benefit, and predicted gains on image-background benchmarks — both return negative results, the former failing Holm correction and the latter worsening cross-entropy. The robustly established positive conclusion is thus confined largely to the claim that activity conditioning improves a tuned DFA in stressed MLP settings, and even this depends on the BN normalization recipe, as Appendix J shows extreme losses in its absence. Combined with the qualifications that FOOF is compute-intensive with no FLOP advantage and DFA receives no preferential benefit, the transferability and practical impact are narrower than the abstract suggests. This bears on the actual value of the contribution: honest negative results are commendable, but they also objectively reduce the scope of the positive claim.

Questions for authors

The following questions, if answered differently, would materially affect the assessment; the final two are suggestions that do not affect this decision.

  1. Is the error-side failure attributable to the fragility of estimating the error covariance, or is the method itself ineffective? The main text notes that gated DFA errors can be low-rank and that under-damped inversion amplifies poorly estimated directions, and Appendices F–I show that retuned error damping can nearly suppress the gain of E. Please clarify whether higher damping, low-rank regularization, or shrinkage was applied to the error covariance to test whether the error side still yields no benefit when adequately estimated. This determines whether the error-side negative is an implementation issue or a substantive conclusion.
  2. Why did the predicted image-background gains fail to transfer? The high-variance-equals-nuisance geometry of the synthetic suite did not produce a gain on real backgrounds. Please provide evidence on how the anisotropy of the activity covariance and its alignment with the task and background subspaces differ fundamentally between the synthetic and real settings. This determines the predictive reach of the synthetic model.
  3. Why is the two-sided K form largely absent from the main confirmation? The main text reports A, E, and their combination from the historical studies, but K does not appear to be a core arm in the joint CIFAR-10 confirmation. Please clarify the relationship between the two-sided Kronecker form and FOOF-DFA, and whether it improves over A alone.

Suggestions, not affecting this decision: 1)Could conditioning be extended to convolutional or Transformer architectures via KFAC-style block approximations, to test whether the activity-side gains persist on deeper, more modern architectures? 2)Can the early-only benefit be formalized independently of the learning-rate schedule, to yield a more general criterion for when to switch conditioning on and off?

2 comments

Thank you for the careful assessment. Your questions led to a larger error-estimation study, a counterfactual background analysis, and extensions beyond MLPs. These strengthen the positive contribution while preserving the original results.

Q1. Does the error factor help when its moment is adequately estimated?

Yes. On the stable CIFAR-10 protocol with 48 paired seeds, early K improves on activity-only conditioning (A) by +0.28 accuracy points [individual 95% CI 0.11, 0.45] with the paper's minibatch estimator. Averaged error moments raise the gain to +0.65 [0.49, 0.82] and beat minibatch K by +0.37 points. All six accuracy and cross-entropy comparisons pass Holm correction. The original estimate is consistent with this result; sample size and correction-family size both changed. The 48-seed study does not repeat the original diagonal-error control.

We examined the regularizers you asked about. Higher damping was tested, but very strong damping makes the operator nearly scalar. At saved states, diagonal shrinkage improves agreement with a pooled-reference update; rank-32 truncation does not materially improve it. These are estimator diagnostics, not training comparisons. We therefore tested averaging in training. In a Transformer control holding calibration, damping and refresh fixed, averaging alone improves test perplexity from 232 to 189 in 12/12 seeds. The MLP averaged recipe additionally changes calibration and refresh (Fig. 5A; Appendix M.1–M.2).

Q2. What differs between synthetic nuisance and image backgrounds?

We redrew backgrounds at fixed foregrounds and measured the activity operator's effect on class-mean energy relative to background energy. The ratio rises ×3.72 in the synthetic nuisance task but changes by ×0.93 on MNIST mixtures and ×0.41 on Fashion mixtures. On the images, class means overlap high-variance directions and the measured proxy is not enriched. These three-seed measurements support a mismatch to the synthetic task-orientation premise and clarify why visible background alone is insufficient evidence for the predicted regime. The conclusion concerns the measured class-mean proxy (Fig. 5E; Appendix M.4).

Q3. Where is K in the confirmation, and how does it differ from FOOF-DFA?

K was included in a dedicated ten-seed cohort (Appendix A.8), which we should have made easier to find. Early K's small positive gain did not pass the original correction; the larger study above now establishes a gain over A. Applying E throughout training was less accurate than A in the original cohort, so our follow-ups apply E early and continue with A.

FOOF-DFA uses FOOF's activity operator with DFA credit. K adds a left factor formed from the local DFA-error moment, rather than KFAC's Fisher factor. Table 1 now distinguishes the moment estimators, solves, norm matching and optimizers; norm matching precedes AdamW. In Transformers, the full error factor also beats the diagonal control.

Theory, scope and your two suggestions.

The Transformer studies now provide the credit-rule comparison you requested: both factors help DFA at matched time, and the DFA-minus-BP gain differences pass correction in three separately frozen studies. At 36M parameters, perplexity falls from 543 to 391 with A and to 195 with early K; at three times the budget it falls from 536 to 264 and 145. BP remains stronger overall; A is slightly worse under BP at matched time and E adds no reliable gain. This is a credit-dependent empirical effect, not an established alignment-acquisition mechanism. The models use LayerNorm rather than BN. A small CNN adds per-update DFA evidence, although only BP gains there at matched time (Fig. 5B–C; Appendix M.2–M.3).

Appendix M.6 adds a linear descent calculation without perfect alignment: the activity factor acts on input geometry and the error factor on effective credit. A positive-definite symmetric part of effective credit is sufficient for descent, not necessary at a particular state. It does not cover AdamW or explain alignment acquisition. We explicitly credit the established activity operator to FOOF.

The constant-learning-rate timing study establishes an early-over-late benefit without a decay schedule; the longer Transformer study retains the early-error gain with its absolute window fixed. A general switching criterion remains open. Fig. 5D and Appendix M.5 add measured costs across width and refresh frequency. The revised main figure and Section 4.5 bring these answers together.

AuthorsOpenReview ↗

additional tests

Following our earlier response, we would like to share additional evidence from studies undertaken in response to Reviewer Jpt2's follow-up. These studies also address your questions about error estimation and background geometry. Thank you for motivating this deeper investigation: it has helped establish more precisely where conditioning supports local learning with fixed feedback. The studies are in Appendix C and Fig. 7; main Fig. 5A,E highlight the new error controls and prospective orientation comparison.

Additional evidence on Q1: error factor. A new, separately frozen study on 48 fresh seeds includes the diagonal-error control missing from our earlier follow-up. With averaged moments, it meets the original five-part criterion under its own correction family. The full error factor beats activity conditioning alone (A) by +0.70 points [95% CI 0.52, 0.87] and a diagonal error factor, tuned over ten settings, by +0.82 [0.65, 0.99]. Without re-selecting the error settings, the gain over A is +1.05 [0.89, 1.22] at batch 128 and +0.73 [0.56, 0.91] at batch 512. These controls establish an additional benefit from the full error factor under this averaged-moment recipe.

Related architecture and alignment evidence. The new CNN study establishes an early matched-time benefit for DFA: at a 60-second budget fixed before new training, with 20 fresh seeds, conditioned DFA gains +0.75 points [0.38, 1.13] and BP does not. The 200-second comparison is unchanged, with no established DFA gain. In the Transformers, networks trained with the full error factor reach states with better-aligned raw DFA directions (cosine with the exact gradient 0.26 against 0.15 after A at 36M, 12/12 seeds) in the 1,200-second runs; after 3,600 seconds the difference is not resolved. This supplies evidence about alignment in trained networks while keeping alignment acquisition an open theoretical question.

Additional evidence on Q2: image backgrounds. Your question led us to test the geometry directly, rotating every background relative to the class-mean subspace with its spectrum, foregrounds and labels unchanged. The intervention establishes that background orientation affects the benefit of conditioning: in the paper's mixtures conditioning gains 2.7 and 2.1 accuracy points when the leading background variance masks the class-mean directions, and loses 0.2 and 0.3 when it is orthogonal to them. The interaction has the same sign in all 20 seeds; BP shows the same dependence. We then recorded the sign of this dependence as a prediction before training two further mixtures, and it was confirmed in all eight declared tests (20 fresh seeds); BP again shows the same dependence there. This is a prospective replication across mixture strengths using the same images and rotations.

The intervention also refines the interpretation in our first reply. The ratio predicts the opposite ordering, so we now retain it as a descriptive measurement rather than an explanation. Input ablations show that the advantage of conditioning remains detectable with the class-mean component of the input removed, including in all four new cells. The stronger prediction of no advantage with that component alone held in two of four. These results sharpen the hypothesis toward usable task information beyond class means, without establishing an exclusive mechanism. We appreciate the opportunity to replace the initial diagnostic account with these more informative controlled tests.

Comments on the paper

2 comments

Thank you for the constructive reviews. We focused the follow-ups on your shared questions: error-moment estimation, transfer beyond MLPs, computational cost, and the geometry of image backgrounds.

The main new findings are:

  • Error conditioning adds a reliable benefit. In the 48-seed CIFAR-10 MLP study, an early error factor improves on activity conditioning by +0.28 accuracy points with minibatch moments and +0.65 with averaged moments. Accuracy and cross-entropy comparisons pass the prespecified Holm correction.
  • Both factors help Transformer learning with fixed feedback. In the 36M model, activity conditioning lowers test perplexity from 543 to 391, and the early error factor lowers it further to 195, at matched measured training time. The gains persist at the longer budget. Paired BP controls establish a credit-dependent benefit at matched time in these models.
  • The geometry analysis gives a concrete answer to the transfer question. Redrawing backgrounds at fixed foregrounds shows that conditioning enriches class-mean energy relative to background energy in the synthetic nuisance task, whereas the image mixtures lack this enrichment. These measurements distinguish the tested geometries and clarify why visible background alone does not establish the model's premise.

The revised manuscript brings these answers together in Section 4.5 and the five panels of Fig. 5. Table 1 clarifies the relationship to FOOF and KFAC. Appendix M reports the protocols, all prespecified comparisons, CNN results, width/refresh cost profiles, and a descent calculation without perfect alignment.

The original confirmation remains unchanged. Each confirmatory follow-up was frozen before its new runs were evaluated and has a separate correction family. Public test sets had prior exposure; the feedback-convention study also reuses earlier seed IDs and BP endpoints, as disclosed in Appendix M. Our reviewer-specific replies provided below give the direct answers and supporting results and we hope clarify the points raised by reviewers.

AuthorsOpenReview ↗

additional tests

Following our earlier general response and the subsequent discussion with Reviewer Jpt2, we would like to summarize the additional evidence now available. Thank you for the constructive questions: they prompted targeted experiments that clarify how conditioning supports learning with fixed, approximate feedback. The revised manuscript and our additional reviewer comments report four advances:

  • Error-factor robustness. A separately frozen study with 48 fresh seeds includes the tuned diagonal-error control. With averaged moments, the full error factor improves both accuracy and cross-entropy over activity conditioning alone and the diagonal control, meeting the declared five-part criterion. The gains transfer across the tested batch sizes without reselecting the error settings.
  • Transfer beyond MLPs. A new 20-seed CNN study establishes a matched-time DFA gain at a prespecified early budget, alongside the Transformer gains already reported. The CNN advantage is budget-dependent. Transformer measurements also associate full error conditioning with less concentrated local errors and better-aligned trained states, refining the mechanism hypothesis.
  • Practical comparisons. Validation-selected reference K-FAC and Shampoo implementations now place the method alongside established preconditioners and quantify accuracy, predictive-loss and memory trade-offs.
  • Controlled geometry. Rotating backgrounds at fixed spectrum and variance changes the benefit of conditioning. The direction of this interaction was then predicted before training and confirmed at two further mixture strengths using the same images. This refines our earlier interpretation: the class-mean energy ratio remains a descriptive measurement, rather than a validated explanation of the effect.

Main Figure 5 and Appendices B–C bring this evidence together. The supplement has also been reorganized by scientific question, with clearer captions and less repetition. The original confirmation results remain unchanged; the additional studies have separate protocols and statistical families. Together, these comparisons strengthen the paper's central contribution: identifying which activity and error statistics help learning under imperfect credit, and under which tested conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.