acceptodds
Under review as a conference paper at ICLR 2027

Backpropagation Misaligns with the Hierarchy of Brain Responses to Images

Abstract

Backpropagation is the core learning mechanism underlying deep learning. However, whether and how this algorithm is implemented in the brain remains highly debated. In particular, while forward activations of pretrained models reliably map onto the cortical hierarchy of visual processing, it is unknown whether backpropagated gradients exhibit a similar correspondence. Here, we address this question using functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) recordings of human brain responses to natural images. For this, we extend standard encoding analyses of forward activations to map backpropagated gradients onto neural data. Focusing on a recent self-supervised vision model (DINOv3) and reproducing results on eight vision models, we find that backpropagated gradients can reliably predict both fMRI and MEG signals, specifically in higher-level visual cortex and for later latencies. However, the spatial and temporal organization of these backpropagated gradients in the brain diverges from the patterns expected under a biologically plausible backpropagation mechanism: specifically, both the order in which gradients are computed and their spatial organization diverge from the temporal and spatial hierarchies of the human brain. Together, these results suggest that, although deep networks and the brain may share similar representational content, they likely rely on fundamentally different mechanisms to learn those representations.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Self-Supervised Learning, Vision models, Dinov3, Backpropagation, NeuroAI, Learning mechanisms, fMRI, MEG, neuro-inspired learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TsMBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors suggest to extend standard encoding analyses of forward activations to map backpropagated gradients onto neural data. They show that backpropagation signals can predict neural activation, beyond the predictable information provided by forward activations. They validate it across 8 conventional AI models predicted on fMRI and MEG signals.

Critical strengths

The paper propose to extend standard encoding analyses of forward activations. Motivated by literature on biologically plausible learning methods, arguing that top-down neural activities can propagate backpropagation-like signals, the authors suggest to test this hypothesis by using backpropagation signals to predict neural activities. This is a novel approach, and can provide valuable information to the open question of how learning mechanisms are implemented in the brain. Their results suggest, that backpropagation signals indeed provide information beyond can be explained solely by forward activation. They analyze temporal and hierarchical behavior, and demonstrate interesting insights on the predictive patterns and the training progress.

Critical weaknesses

I find the interpretation of Fig 2 (E,F) as an overclaim - the textual description of the results doesn't fully aligned with the visual results. I prefer more accurate description of the visible results, and supporting claims like: "shallower layers peak earlier, whereas deeper layers align later with brain responses (Fig. 2E)." with a numeric evaluation since the time of the two peaks seems similar across depth.

The paper oversimplify existing claims for biologically learning models - most do not argue for a simple clean forward - backward separation in the brain (a forward phase followed by a backward propagation), and many suggest more complex time dynamics with interacting bottom-up and top-down activities.

Questions for authors

How can these results be interpret when relaxing the assumption of a clean forward backward separation?

2 comments
AuthorsOpenReview ↗

Layer-wise peak-time statistics clarify the temporal hierarchy

We thank the reviewer for this thoughtful review and for highlighting the novel approach, successful isolation of gradient signals, and the temporal and hierarchical analyses.

W1 — Peak-time hierarchy is now quantified. We now quantify peak times in a new column (Fig. 2G–H) and compare forward and backward timing (Supp. Fig. 11). Forward peak time increases with depth (r = .96, p = 1e-6); backward peak time does not (r = .19, p = .5). We improved the text accordingly.

W2 — Scope now targets sequential backpropagation. We agree. We test only the standard, sequential form of backpropagation, and we now state this clearly in the Introduction and Discussion.

Q1 — The spatial result does not require a separate backward phase. Our predictions assume that the backward pass follows the forward pass. Backward layers do peak later on average, though no layer peak differs significantly between forward and backward (paired Wilcoxon, BH-FDR q ≥ .5; Supp. Fig. 11). This latency complements the result that residual backprop becomes significantly positive at later latencies (0.5–1.7 s; Fig. 4F). But crucially, backward peaks show no depth order (Fig. 2H: r = .19, p = .5), unlike forward peaks (Fig. 2G: r = .96, p = 1e-6). Together, these results are consistent with learning-related signals emerging after initial image processing, but not with the specific reverse-ordered temporal hierarchy predicted by sequential backpropagation.

If we relax the sequential assumption, gradients still predict brain responses (Fig. 3) and the reversed fMRI hierarchy remains (Fig. 2D; Supp. Fig. 10H), this reversed fMRI hierarchy is also now reproduced for a new benchmark of 6 video models like VJEPA-2 (Supp. Fig. 10). The MEG result therefore argues only against a strict, sequential temporal hierarchy of gradient backpropagation; it does not rule out simultaneous or recurrent learning models. We revised the text to make this scope explicit.

This review helped us improve the paper by:

  • quantifying peak times per layer, with statistics (Fig. 2G–H, Supp. Fig. 11);
  • stating precisely the type of standard, sequential form of backpropagation we test in the manuscript (abstract, intro, discussion)
  • adding a benchmark of six video models where results are reproduced, with V-JEPA 2's spatial and temporal hierarchies (Supp. Fig. 10);

All new figures and text are in the paper; we tried to keep this response brief, but are of course happy to discuss any detail.

Thank you again for your thoughtful review. We hope our revisions have addressed your feedback.

Would you have any remaining questions or points you’d like us to address or clarify?

Reviewer_GwYgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a fundamental and long-debated question: whether the brain implements backpropagation or a functionally equivalent learning mechanism. Unlike previous works that primarily compare the correspondence between forward activations of deep networks and brain activity, this study further investigates whether backpropagated gradients can also be mapped onto human visual brain activity. Methodologically, the authors extend standard encoding analysis from forward features to backward gradient features, using linear ridge regression to map model features onto fMRI and MEG neural responses to natural images. The core experiments focus on DINOv3, with replications across multiple additional vision models.

Critical strengths
  1. The research question is important and novel. This paper examines a highly fundamental question: it asks not only whether AI representations resemble the brain, but also whether AI learning signals resemble the brain. This goes beyond the vast majority of works that focus solely on forward representational similarity, offering clear novelty and potential impact.

  2. The simultaneous use of fMRI and MEG covers both spatial and temporal dimensions. The paper uses both fMRI and MEG to examine spatial distribution and temporal dynamics, respectively. This multi-modal design is well matched to the scientific question.

  3. Replication across multiple vision models increases robustness. The paper reports replications on multiple additional models beyond DINOv3, which strengthens the robustness of the conclusions.

Critical weaknesses
  1. The interpretation of mapping backpropagated gradients to brain signals remains conceptually insufficient. The central argument relies on a strong interpretative leap: the loss gradient in the model can be viewed as a kind of backward activation corresponding to the brain's learning signal. However, from the provided content, this mapping remains rather indirect. Different loss functions yield different gradient structures. Also, for self-supervised models (DINOv3), the training objective does not match the subjects' task state during scanning. Subjects are engaged in recognition or oddball tasks (passive viewing), without explicit learning objectives or behavioral errors matching the model’s training loss.

  2. The definition and interpretation of Residual Backpropagation need to be more rigorous. The authors state that Residual Backpropagation scores are obtained from backpropagated features after removing the variance explained by Forward activations. This analytical direction is reasonable, but current information is insufficient. Since forward and backward features may inherently share complex non-linear structures, simple linear residualization cannot guarantee that what remains is a purely gradient-specific signal.

  3. Stronger baselines and exclusion of alternative explanations are needed. It is recommended to add more controls, such as gradients generated from random labels or random losses, gradients from untrained models matching the real model structure. Without these controls, it is difficult to assert that what is observed is "backprop-specific."

  4. Methodological details are lacking. The manuscript would benefit from a clearer description of key methodological choices.

  5. The conclusion is slightly stronger than the evidence supports. A more prudent claim is that model backward gradients can linearly predict part of the brain signals, but their spatiotemporal organization does not support a simple, hierarchical, biologically plausible backpropagation mapping.

Questions for authors
  1. Have you compared gradients from untrained models, gradients from random losses or objectives, random features matched in dimensionality and norm to forward features, and gradient norm rather than the full gradient pattern?

  2. The paper claims that backward encoding is not later than forward encoding. Does this "not later" refer to peak time, onset time, or the center of the significant time window? How is this compared statistically?

  3. Subjects are in recognition/oddball tasks, while model gradients come from training loss. How do the authors rule out the interpretative bias caused by this task mismatch? Have you considered using objective functions more consistent with the subjects' tasks or online prediction error proxies?

3 comments
AuthorsOpenReview ↗

Five new controls support gradient specificity

We thank the reviewer for these constructive comments and for highlighting the novel question, complementary fMRI–MEG design, and cross-model replication.

W1 — Backprop and residual-backprop features predict brain responses across objectives in 2 benchmark of vision models and video models. Figures 3–4 show that gradients sigificantly predict fMRI and MEG responses, even beyond forward activations alone. This shows a correspondence between model gradients and brain responses. We agree that it does not prove that the brain computes the same error signals. The Introduction now states more clearly the premise: the cortex learns from prediction errors, even during passive viewing. DINOv3 learns in a self-supervised way, without labels, which makes it the closest available analogue (Sec. 2.2), and the pattern holds across 8 image models with different losses (Supp. Fig. 8) and now in a new benchmark of 6 video models, including self-supervised video model V-JEPA 2 (Supp. Fig. 10).

W2 — Randomized gradients do not reproduce the residual. Thanks to this review, we added controls with gradients computed from random teacher targets and random linear objectives. Both keep the model and images fixed but remove the true training signal; neither yields a significant positive residual in fMRI or MEG (Supp. Fig. 12B–C, G–H). Thus, shared image and model structure cannot explain the residual. We also corrected the text to highlight more precisely that the residual is the gain R(Forward + Backprop) − R(Forward) (Eq. 1), not a removal of variance from the gradients.

W3 — Five new controls support gradient specificity. We agree and add five controls (Supp. Fig. 12): dimension- and norm-matched noise, random targets, random objectives, untrained models of the same architecture, and gradient magnitude. All controls support gradient specificity. While untrained-model gradients already give a weak gain, trained-model gradients give a significantly stronger gain (Fig.3, 4, Supp. Fig. 12).

W4 — Methods now specify each key analysis. We clarified the residual definition and now describe the peak-time measure, the new controls and the video models (Methods; Supp. Figs. 10–12). Please tell us if any detail remains unclear.

W5 — Conclusions now precisely match the evidence. We agree and adopt your wording in the abstract and Discussion: while gradients linearly predict part of the brain signal, their spatiotemporal organization does not support a simple, hierarchical backpropagation mapping.

Q1 — Five complementary controls are now reported. We now test these alternatives in Supp. Fig. 12: untrained models (D, I), random targets (B, G), random objectives (C, H), Gaussian noise (A, F) and gradient magnitude (E, J). Only full gradients of the trained model give a strong gain.

Q2 — Peak timing is now more precisely defined and tested. We define peak time as the mean time at which the score exceeds 95% of its maximum, following previous litterature (Raugel et al., 2025, Neurips). Backward layers do peak later on average, though no layer peak differs significantly between forward and backward (paired Wilcoxon, BH-FDR q ≥ .5; Supp. Fig. 11). This latency complements the result that residual backprop becomes significantly positive at later latencies (0.5–1.7 s; Fig. 4F). Crucially, backward peaks show no depth order (Fig. 2H: r = .19, p = .5), unlike forward peaks (Fig. 2G: r = .96, p = 1e-6). Together, these results are consistent with learning-related signals emerging after initial image processing, but not with the specific reverse-ordered temporal hierarchy predicted by standard backpropagation.

Q3 — Task scope and proxy objective are now explicit. Participants mostly view images passively; the tasks only keep them alert and engaged, and no task matches a model loss. Under predictive coding, a self-supervised objective is the closest match among those tested (Introduction; Sec. 2.2). Future datasets with online prediction-error proxies would be a promising next step.

Your review helped us improve the paper by:

  • adding five controls for gradient specificity that strengthen the conclusions (Supp. Fig. 12);
  • adding two new plots showing dynamics of peak times across layers (Fig. 2G–H, Supp. Fig. 11);
  • improving the description of the residual and expanding the Methods;
  • tempering the conclusion in the abstract and Discussion.

All new figures and text are in the paper; we tried to keep this response brief, but are of course happy to discuss any detail.

Reviewer_GwYgOpenReview ↗

Replying to five new controls support gradient specificity

Thank you for the substantial revision. The new randomized-target and randomized-objective controls, the comparison with untrained models, and the clarification of the residual metric directly address several of my concerns. I also appreciate the more cautious conclusion. These changes improve the paper, and I am inclined to raise my score to borderline. Additionally, I noticed several minor typographical and spelling errors throughout the manuscript, such as the final two sentences of the abstract and the “//”. I would encourage the authors to conduct a thorough proofreading of the entire paper to ensure the final version is well-polished.

Thank you very much for your feedback and for pointing out the typos, which we will carefully address!

Your thoughtful reviews have helped us significantly strengthen the paper.

Would you have any remaining questions or suggestions for improvement that might bring your score to a 4? We would be happy to address them and further improve the manuscript.

Reviewer_jYAwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study addresses a central question: does the brain actually use backpropagation to learn? The authors recorded human brain responses to natural images using fMRI and MEG. They extended encoding analysis—a method traditionally used to examine forward activations—to investigate backpropagation gradients. This approach was applied to eight different visual models, with a primary focus on the self-supervised visual model DINOv3.

The findings present a dual picture: on one hand, backpropagation gradients can reliably predict fMRI and MEG signals, particularly within higher-level visual cortex and at later temporal latencies. On the other hand, the computational sequence and spatial organization of these gradients within the brain do not align with the spatiotemporal hierarchy observed in the human brain, nor do they conform to the patterns expected from a biologically plausible backpropagation mechanism.

Critical strengths
  1. The authors propose an innovative approach to investigate whether the gradients used in backpropagation, rather than merely the forward-pass activations, correspond to signals in the brain.

  2. The findings aren't model-specific, they replicate across 8 diverse architectures and losses, and even track DINOv3's training dynamics from scratch.

Critical weaknesses
  1. This analysis is based on a single self-supervised learning (SSL) model, DINOv3-S; the "EMA teacher-student" mechanism employed by this model bears little resemblance to biological principles, and thus, using it as an analogy for how the brain learns lacks a solid foundation.

  2. DINOv3 works on static images with no recurrence or temporal dynamics, yet learning signals may unfold over time, something this model just can't capture by design.

  3. fMRI and MEG can't resolve synapse-level or layer-specific learning signals, so the paper can only say the data doesn't match backprop's predicted pattern, it can't really settle whether the brain does backprop or not.

Questions for authors
  1. The gradients were squashed from ~4M dimensions via random projection + PCA, could that preprocessing itself have destroyed the top-down hierarchical structure?

  2. Residual backprop best predicts late-latency, higher-level cortical responses, is that really a "learning signal", or just gradients re-encoding high-level semantic features already implicit in the forward path?

  3. Forward activations and gradients derive from the same weights and are highly correlated, can a linear encoding model really disentangle them

5 comments
AuthorsOpenReview ↗

A six-model video benchmark extends the analysis beyond static images, and new controls strengthen the results

We thank the reviewer for this careful review and for highlighting the innovative approach of encoding backpropagated gradients in the brain, its replication across eight diverse models, and the training-dynamics analysis.

W1 — Results generalize across eight image models. Our results replicate across 8 image models with different architectures and losses (Fig. 3, Supp. Fig. 8). We agree that the EMA teacher–student scheme is not biological (Discussion, Modeling limitations). We chose DINOv3 because of its self-supervised learning objective, which makes it the closest analogue to continuous learning in the brain among the available open-source deep vision models (Introduction; Sec. 2.2).

W2 — Results now extend to six video models. Good point. We started with static images to avoid temporal confounds, and we now added six video models spanning four types of objectives (Supp. Fig. 10). V-JEPA 2, the only self-supervised video model that learns in latent space, shows the highest forward, backprop and residual backprop encoding scores (Supp. Fig. 10A–F). Its gradients show the same misalignment as DINOv3: a reversed spatial hierarchy (Supp. Fig. 10H: r = −.48, p = 1e-4) and no temporal order (Supp. Fig. 10J: r = .25, p = .2).

W3 — fMRI and MEG resolve cortical organization across space and time. We agree: our data test the predicted pattern at the cortical level, not synaptic mechanisms – though resulting from them (Discussion, Measurement limitations). Still, the same data recover the forward hierarchy in space and time (Fig. 2C, 2G), while backward gradients show the opposite order in space (Fig. 2D) and no order in time (Fig. 2H).

Q1 — Encoding is robust across PCA and projection sizes. We reduce each layer independently, so the reduction never mixes layer identities. Backprop encoding remains stable from 50–384 PCA dimensions and 200–1600 projection dimensions (Supp. Fig. 9), making a compression-driven reversal unlikely, though not impossible; we added this caveat to the Limitations.

Q2 — Randomized objectives do not reproduce the residual. By design, the residual excludes all variance that forward features predict linearly. Thanks to this review, we added gradients computed from random teacher targets and random linear objectives. Both keep the model and images fixed but remove the true training signal; neither yields a significant positive residual in fMRI or MEG (Supp. Fig. 12B–C, G–H). Thus, the residual depends on the true training objective, not on re-encoded forward features.

Q3 — Gradients predict brain responses beyond forward activations. We compare encoding scores, not individual regression weights. Residual Backpropagation is R(Forward + Backprop) − R(Forward) (Eq. 1), with features of same total dimension in both models. A positive residual therefore means that gradients improve brain-response prediction beyond forward activations.

Your review helped us improve the paper by:

  • adding a benchmark of six video models where results are reproduced (Supp. Fig. 10);
  • adding random-target and random-objective controls that strengthen the conclusion that the residual depends on the true training objective (Supp. Fig. 12).

All new figures and text are in the paper; we tried to keep this response brief, but are of course happy to discuss any detail.

Thank you for the response and the added analyses. While some of my questions have been addressed, several points still require further clarification. I will therefore keep my current score.

Thank you for your follow-up.

It seems we have addressed each of your original concerns through additional analyses and clarifications, including the new video-model benchmark and all five successful gradient-specificity controls.

Could you please specify which points you feel still require clarification? We’d be happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.