acceptodds
Under review as a conference paper at ICLR 2027

Understanding How Inference Shapes Weight Updates in Predictive Coding: From Label Noise to the State-to-Weight Interface

Abstract

Predictive coding networks infer neural activities through iterative energy minimization and learn network parameters through local synaptic updates. In this work, we find that, under symmetric label noise, PC maintains its fit to clean samples while memorizing fewer incorrect labels than backpropagation. By analyzing state inference and weight updates separately, we find that inference absorbs an incorrect label into the hidden states rather than passing it on to the weights. For the same image, an incorrect label induces a larger feedforward prediction error than the true label, yet the residuals after inference are similar, yielding a higher state absorption ratio. Our theoretical analysis characterizes this separation, showing how inference absorbs prediction errors through state changes and how the resulting states shape both the residuals and parameter gradients used for weight updates. At fixed weights and inference states, using inferred inputs attenuates the weight-gradient response to label changes and alters its direction; replacing these inputs with feedforward states leads to greater noise memorization during training. PC also memorizes more class-structured incorrect labels than those from symmetric corruption. Together, these results show how inference filters target-induced changes before local updates accumulate in shared weights.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Predictive Coding, Energy-based Model, Noise Memorization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2MJkOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper investigates why and how predictive coding (PC) networks trained with symmetric label noise memorise fewer incorrect labels than backpropagation (BP). On CIFAR-10 the authors show that PC fits clean examples and its test accuracy quite well, while BP fits the wrong labels late in training and its test accuracy falls. Control experiments argue that this is a feature of PC's inference process and do this by testing a set of other options including how wider networks, inference length, and removing misclassified clean examples affect PC. The authors also use the forward update rule (PC+F), interpolation of the layer-wise input (from forward to the inference state), and an ePC step-size sweep to give more evidence for their finding. Reading the feedforward input instead of the inferred input increases noise memorization, showing that the inference state really does matter in a clean and well constructed comparison.

Critical strengths

The investigation explored in this paper really does matter. Careful, step-by-step analysis of what actually happens inside a PC network during learning is needed if we want to understand how PC differs from BP. It is also needed to understand whether PC’s useful properties, such as absorbing or ignoring label noise, can be kept or transferred to other networks. I value this direction of research.

Furthermore, the central experimental idea is quite clean and neat. PC+F and the λ intepolation process keeps the inferred target fixed and changes only the input the weight update reads, which isolates one part of the learning rule. The fixed-weight paired probes, using the same image and weights under the true and the wrong label, are also a good way to look inside a single update without the confounds of full training.

Finally, the authors do a lot of work to isolate potential confounds including random-label capacity, network width, inference time length, optimizers, more datasets etc.

Critical weaknesses

This paper has a lot of weaknesses. The weaknesses are almost all centred around the lack of human readability of this work and the lack of structure that it has. As an overarching point, this work should be far more clear about where this paper fits into the field, exactly how experiments are constructed and what they mean, and what the takeaways of this work should be. Experimental clarity and paper narrative are almost completely missing from the main text. As a major point, almost none of the figures are readable and all have some major issue with how they are formatted or described. Some explanations are present in the appendix but the main work cannot be read without constant referencing to the appendix for missing information and context (some of which is not even present there).

To be more specific about these issues, I want to give some examples.

  1. Very short introduction: The introduction barely describes any of the existing work on understanding the difference between PC model and BP model learning
  2. No experimental setup section: Experiments are poorly (or not at all) described in the text, and the reader has to move around the paper, and to other papers, to understand them. This is largely because the main text has no clear setup section. Basic terms are not defined where they are used. For example, it is never explained in the main text what a corrupted sample is or how symmetric corruption is generated. Model names appear without definition, for example “Fashion-MNIST/MLP5” (L198). Hyperparameters that matter are spread across appendix tables etc.
  3. Inference length choices are never explained: The number of inference steps clearly matters a lot. Some experiments use 7 steps (Table 4), the reference run for filtering uses T = 5 (Appendix B.2), and Figure 4 uses 5, 10, 15 and 20. I could not find where or how these values were chosen, or why T is not fixed or tuned in a rigorous way. Because the main claim is about what inference does to the weight update, this needs to be explicit. It also means this paper is probably impossible to reproduce.
  4. Figures are hard to read and poorly explained: Several figures (e.g. Figure 2, Figure 6) have legends that overlap the curves and very small axis text. They do not look as though they were checked carefully before submission. The captions are also in general far too short and some can't even be understood just by reading the main text. Some figures with major issues: Figure 2d (cannot be understood clearly), Figure 3 (“Width multiplier” is never defined), Figure 5 (not clearly defined in text), Figures 7 and 8 (too close together), Figure 8c (s0, s1 and s2 are never explained) etc.
  5. No discussion and no clear conclusions: The main text goes from results straight to a one-paragraph conclusion. There is no discussion of what the findings mean, how they relate to earlier PC-vs-BP results, or what a reader should take away.

All in all, the paper is hard to follow: it is often unclear what is plotted, why, and how. In its current form the presentation does not allow a reader to properly assess the contribution. This is a pity given that I think that the actual question being explored is important.

Questions for authors

Equation 7 mixes the inferred and forward states linearly with λ. This is a kind of linear interpolation. By why not instead use the value of the state at different values of t? Is the state from 0->T really a linear interpolation?

How exactly are corrupted samples generated for the earlier figures of your work? And could you possibly identify corrupted samples in the actual underlying dataset using the PC models?

3 comments

Thank you for carefully reading our paper and recognizing the value of the research question. We acknowledge your concerns about readability and experimental clarity. To meet the page limit, we excessively compressed the experimental rationale, captions, and figures, making it difficult to understand how the experiments were constructed and what questions their results answered. We have revised the manuscript as follows.

Research context and experimental setup. In the first paragraph of the Introduction, we added related work on the learning properties of PC, clarified its relationship to our research question, and directed readers to the more detailed related-work discussion in the appendix. Section 2 now brings together the label-corruption procedures, models and datasets, hyperparameter selection, and evaluation metrics, with references to the full protocols in Appendix A.

Inference steps and hyperparameters. Hyperparameters were searched using TPE on datasets without label corruption, and the selected configurations were subsequently used in the noisy-label experiments. Each method uses its own searched configuration, which explains why the number of inference steps differs across configurations. For the width and inference-step experiments, we separately searched the remaining hyperparameters for each setting. Appendix A now collects the configurations actually used and maps them to the corresponding experiments in the main text.

Figures and captions. We regenerated and reorganized the figures, using consistent method colors: red for PC, green for PC+F, and blue for BP. We also assigned consistent line styles to wrong-label fitting, clean fitting, true-label recovery, and test accuracy. The captions now provide additional experimental settings and explain the axes and markers. We also adjusted legend placement and panel layouts.

Narrative structure and discussion. We reorganized Sections 4 and 5 so that each analysis answers a question raised by the preceding results. Section 4 holds the inference procedure fixed and changes the input read by the weight update to establish how the use of inferred states affects noise memorization. Section 5 first shows how state adaptation absorbs output displacement, then examines which directions remain in the residual. It subsequently investigates how these residuals and adapted inputs form weight gradients, and finally connects individual updates to memory accumulation through cross-example gradient coherence and continuation experiments. The main-text Discussion explains what these findings mean for understanding PC and designing learning rules; the appendix provides further mechanistic discussion, limitations, and future directions.

Q1: Why use linear interpolation rather than states at different points along the inference trajectory?

Thank you for this suggestion. Linear interpolation directly controls the composition of forward and final inferred states in the update input; it does not assume that the actual inference trajectory is linear. Following your suggestion, we added intermediate-state experiments: after completing the same inference procedure, we keep each layer’s final inferred target fixed and replace only the weight-update input with hth^t, a state along the actual inference trajectory.

Section 4.2 now presents these two interventions together. The intermediate-state experiments show that reading inferred activities before they reach their final states can already reduce wrong-label memorization. We also conducted an independent hyperparameter search for a selected intermediate-state rule and included the results in the same comparison. The full configurations and training trajectories are provided in Appendix C. Together, these experiments show that partially incorporating inferred activities can already achieve low noise memorization while maintaining high test accuracy.

Q2: How are corrupted examples generated? Can PC identify mislabeled examples in the dataset?

We now specify the procedure in Section 2. For symmetric label noise at rate ρ\rho, we randomly select the corresponding fraction of training examples and uniformly replace each selected label with one of the other classes, excluding its original class. The images themselves remain unchanged.

For label-error identification, Appendix G presents the forward-displacement distributions of correctly labeled and mislabeled examples, including CIFAR-10N human annotations and synthetic symmetric noise. The histograms show that, at the end of training, PC retains a high-displacement region enriched in mislabeled examples, suggesting that this signal can help identify some label errors. However, the distributions overlap substantially: difficult correctly labeled examples can also have large displacements. Appendix G therefore discusses incorporating cross-example gradient directions as a direction for further research.

Dear authors,

Thank you for your responses, especially to my questions which now are a bit more clear. I have examined the paper once again but I still find a number of aspects of it to be inadequately formulated and formed if the intention is for it to be published in it's current state.

First, I don't believe my core concerns about human readability have been addressed. The paper still has very little narrative and jumps from point to point with little flagging or clearly trackable logic as to why the next test/step is necessary, interesting, or informative. Figures are still far too small to be readable in an A4 printed format and I needed to zoom/lean-in to read all axis labels and text. The 'Experimental Setup' section help alleviate some of my previous concerns but is so rushed that it does not actually describe in any detail the system you are simulating and does not set up any of the notation for the rest of the paper. In lieu of a method section the paper should do this somewhere (could also be in the introduction). Finally, though you indicate that you added narrative structure to Sections 4, 5, and the Discussion, I still find these to be head-spinningly disorienting. One specific request would be to not just mention what you study next but why. The discussion, for my liking, is also far too short and does not cover the impact of this work

In all, I would still recommend major changes to this paper before it could be considered a readable contribution. In your rebuttal you mention that: To meet the page limit, we excessively compressed the experimental rationale, captions, and figures, making it difficult to understand how the experiments were constructed and what questions their results answered., I think the degree of compression you have tried to achieve is hurting this paper significantly. If the kind of contribution you would like to make has so many separate and independent contributions that do not fit easily into this page limit I might recommend an alternative venue (full journal submission) where the page limit does not impact your communication.

Finally, since the changes in this version (compared to the previous version) are not indicated in any way, it is difficult to track exactly what changed.

Given all of the above, I opt not to modify my current recommendation for this paper's publication.

Best wishes, Reviewer 2MJk

Dear Reviewer 2MJk,

Thank you for your follow-up review.

Regarding the size of the figures: we apologize for overlooking the axes and legends previously, we have now enlarged them as much as possible. As for the experimental setup, we have revised this section and included references to the more detailed descriptions and notation provided in the appendix.

Regarding the logic of the article, we realized that the previous descriptions were more from our perspective as the ones who implemented the models and ran the experiments, which created logical gaps from a reader's point of view. We have therefore thoroughly rewritten this section to ensure smoother transitions. In the current version, changes made since the previous revision are highlighted in red, while key differences between the previous revision and the original submission are highlighted in blue. We hope this makes it easier for you to track the changes.

Thank you once again for your review, and we apologize for the additional effort caused due to the unmarked changes previously.

Best regards, The Authors

Reviewer_dB3EOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates why predictive coding (PC) networks memorize fewer incorrect labels than backpropagation (BP). The authors examine the state-to-weight interface, showing how target-conditioned inference influences not only local learning targets but also the gradients used for weight updates. The question of how target information is accommodated by neural activities versus consolidated into synaptic weights is meaningful, and the paper provides some interesting mechanistic observations. However, I remain unconvinced that the observed behavior reflects a useful inductive bias rather than PC's limited fitting ability and optimization dynamics. The manuscript also suffers from substantial presentation issues. Overall, I lean toward rejection.

Critical strengths
  1. Meaningful mechanistic question. Understanding how inference mediates the transfer of target information into synaptic weights is an important question in PC learning. The characterization of inference-dependent, direction-selective gradient filtering provides useful insights into how PC training and inference jointly shape weight updates.
Critical weaknesses
  1. Insufficient separation from limited fitting ability. The reduced noisy-label memorization may largely reflect PC's limited fitting ability or optimization inefficiency rather than a beneficial inductive bias. The random-label fitting and width experiments demonstrate that PC can memorize arbitrary associations, but do not establish comparable fitting ability to BP under matched learning conditions.

  2. Unclear scientific significance and incomplete mechanistic explanation. The practical or general significance of reduced noisy-label memorization remains unclear, particularly since BP achieves higher test accuracy at validation-selected checkpoints (Appendix K). Furthermore, while the gradient analyses are informative, their causal connection to selective long-term memorization is not fully established. It remains uncertain whether the paper identifies a general learning principle or characterizes a consequence of PC's optimization dynamics in a relatively narrow setting.

Questions for authors

Clarification requests relevant to my assessment:

  1. How can the reduced noisy-label memorization be distinguished from PC's limited fitting ability?

The ability to memorize arbitrary labels does not necessarily establish comparable fitting efficiency. What evidence rules out the possibility that PC simply learns difficult or irregular associations less effectively than BP?

  1. What is the significance of reduced memorization at the final checkpoint when BP achieves better validation-selected generalization?

Appendix K acknowledges that BP achieves higher validation-selected test accuracy across symmetric corruption settings. Is the claimed contribution a generalization advantage, or primarily a characterization of late-stage learning dynamics? Why should the latter be considered a consequential learning property beyond what can be achieved through standard model selection?

Suggested analyses and additional feedback:

To help resolve the above concerns, comparing noisy-label memorization at matched clean-fitting levels, rather than matched epochs, would be particularly informative. A more targeted intervention, such as applying PC+F only to the readout layer, could also help clarify the relationship between the observed gradient filtering and accumulated memorization.

The presentation would also benefit from substantial revision. Figure 2(d), for example, combines different training protocols without clearly explaining the differing meanings of the horizontal axis, marker conventions, and dataset compositions. Several figures contain overlapping legends or insufficiently readable panels. More self-contained captions and clearer explanations of the experimental rationale would significantly improve readability.

Minor Comments

The Appendix Contents page contains incorrectly ordered entries. Main figure captions should more consistently specify experimental settings, including corruption rates and evaluation protocols.

3 comments

Thank you for your review and concrete suggestions regarding clean fitting and layer-specific interventions. We have added these analyses and revised the presentation. Figures now use consistent method colors (PC: red; PC+F: green; BP: blue) and metric-specific line styles, with clearer captions and legend placement. We also reorganized the appendix contents and revised the experimental sections to explain each experiment’s rationale, setup, results, and implications.

1. How can the reduced noisy-label memorization be distinguished from PC's limited fitting ability?

In our random-label experiments, PC is indeed less effective than BP. Figure 2(c) measures how many fixed random labels each method fits under the same weight-update step budget.

Figure 2(d) compares this demonstrated memorization ability with the incorrect labels actually memorized during mixed training, at matching corrupted-subset sizes. For example, at a corruption rate of 0.2, both methods can fit all random labels in a pool of the corresponding size, yet PC memorizes far fewer incorrect labels during mixed training. The fitting-ability gap between PC and BP therefore does not fully explain this difference.

Following your suggestion, Section 3.3 compares wrong-label fitting against clean fitting. Within their shared clean-fitting range, PC does not consistently fit fewer incorrect labels than BP; BP subsequently reaches higher clean fitting and memorizes more incorrect labels. We therefore tested whether unresolved clean examples prevent PC from turning to incorrect labels.

At each epoch, we retain correctly classified clean examples and all corrupted examples, removing unresolved clean examples. Clean fitting on the retained subset reaches a high level comparable to BP, while wrong-label memorization increases only modestly. Removing unresolved clean examples therefore does not produce BP-like growth in noise memorization.

Together with the random-label controls, these results direct the question toward how PC learns from mixed data and motivate the subsequent analysis of inference and weight updates.

2. What is the significance of reduced memorization at the final checkpoint when BP achieves better validation-selected generalization?

Our central question is why the two learning rules handle incorrect labels differently as training continues, and what this reveals about designing PC learning methods.

Prior work introduced modifications such as Forward Update to improve deep PC training. Under the tested configurations, PC+F, PC+SF, and ePC exhibit more pronounced late-stage noise memorization than standard PC. Thus, modifications that improve training can also change how networks learn from label noise.

Our interpolation and intermediate-state experiments show that partially incorporating inferred activities reduces wrong-label memorization while maintaining high test accuracy. These findings explain how inferred states affect learning through the update inputs and provide a basis for improving PC training while retaining this property. We have clarified this significance in the Discussion.

3. What happens when only the readout or hidden-layer update rules are changed?

Following your suggestion, Appendix C.4 reports continuations from the same PC checkpoint under four rules: unchanged PC, forward inputs only at the readout, forward inputs only at the hidden layers, and forward inputs at all layers. The branches share starting weights, data order, and inference settings, with identically restarted weight optimizers.

Changing only the readout produces a trajectory close to PC. Changing only the hidden layers produces behavior closer to PC+F: clean fitting and test accuracy continue to improve, while wrong-label fitting increases modestly.

Together with the layerwise probes in Appendix D.1 (Figure 20), this clarifies the roles of different layers: the strongest residual attenuation within a single inference episode occurs at the readout, whereas changing hidden-layer update inputs produces the main training differences in these continuations.

Thank you for the authors' efforts to address my concerns and improve the presentation.

However, my main concern remains insufficiently resolved. As shown in the revised Figure 3(a), PC does not consistently exhibit lower wrong-label fitting than BP at comparable clean-fitting levels, while failing to reach BP's higher clean-fitting levels. Although the filtering experiment is informative, its improved clean fitting is measured only on the retained subset and therefore does not establish comparable fitting ability on the full clean set. It thus remains unclear whether PC's reduced noise memorization reflects selective suppression of incorrect labels rather than differences in overall fitting ability and optimization dynamics.

Additionally, although the presentation has improved, I still find the lack of clarity concerning, particularly for a paper centered on mechanistic analyses. For these reasons, I maintain my negative recommendation.

Thanks for your follow-up review.

We acknowledge that “PC does not consistently exhibit lower wrong-label fitting than BP at comparable clean-fitting levels, while failing to reach BP's higher clean-fitting levels.”. However, we don’t compare how well the two methods fit clean samples.

The filter experiment examines a specific explanation derived from the memorization process of BP. Research on learning with noisy labels has found that networks will first fit clean samples. During this early learning phase, the gradients are dominated by clean samples; after the clean samples are fitted, their contribution to the gradients diminishes, and the gradients of mislabeled samples become dominant instead, driving the network to memorize noisy sample[1].

The filter experiment eliminates this source for PC. Once the fit on clean samples reaches a specific threshold, subsequent training epochs utilize only the currently correctly classified clean samples alongside all mislabeled samples. Consequently, the retained clean samples contribute almost no error, leaving the learning signal to be predominantly driven by the mislabeled samples—precisely the condition that triggers memorization in BP. Under these circumstances, PC's fit on the entirety of the mislabeled samples merely increases from 14.3% to 17.7%. Thus, the unfitted clean samples fail to account for PC's noise suppression. Section 5 elaborates on the underlying cause: inference absorbs the majority of the changes demanded by the false labels, rendering the residual gradients far more inconsistent within the same label category (Section 5.4).

Regarding the presentation, we have revised the manuscript based on Reviewer 2MJk's comments, and we kindly ask if you could take another look. We have highlighted the changes from this round in red. Additionally, we will provide a clearer description of the filtering experiment in the final revision.

[1] Liu, Sheng, et al. "Early-learning regularization prevents memorization of noisy labels." Advances in neural information processing systems 33 (2020): 20331-20342.

Reviewer_SssmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates why predictive coding (PC) models outperform those trained with backpropagation (BP) when there are incorrectly labeled training examples. They look at how the inference step in PC influences what the network learns, and find that the incorrect labels do not directly affect the weights but are absorbed into the network's hidden activities during inference. In the process, they rule out an alternative hypothesis that reduced memorization capacity, rather than inference process, is what leads to improved performance. They then explore the factors in inference and training responsible for these improvements by comparing PC against the PC+F learning rule, which uses the feedforward activity rather than inferred activity during weight update. Later analyses aim to show that PC inference acts as a kind of filter that builds up associations across training examples, which provides a mechanism for memorizing consistent patterns.

Critical strengths
  1. The authors are rigorous in testing possible confounders for the improved performance of PC.
  2. Involving PC+F as an intermediary between PC and BP allows the authors to disentangle the two ways in which PC differs from BP (final prediction vs forward prediction, and inferred target weight update vs chain rule). This allows the effects of forward vs final prediction to be measured independently.
Critical weaknesses
  1. In Figure 6, PC achieves both higher test accuracy and lower memorization of noisy labels than PC+F and all the intermediate rules. The results do not demonstrate a favorable trade-off from interpolating toward feedforward inputs, so what practical or scientific insight does the interpolation between PC and PC+F provide beyond comparing the two endpoints?
  2. It's not clearly spelled out why ePC is invoked in Section 4.3, nor why frozen checkpoints and Proposition 1 are introduced in Section 5. The same can be said for Prop 2 in Section 5.3 and many of the math-heavy sections. In these later sections the authors do not build up the necessary logic to help the readers understand their scientific goals or why their experiments/analysis are necessary to reach these goals. As a result it is extremely difficult to follow the text and I often found myself trying to infer the author's line of reasoning. It is especially hard to do so when the text is so terse and largely states the results without much clarification. I strongly suggest that the authors rewrite sections 4 and 5 in a way that provides the reader with a single narrative arc that is easy to follow, rather than listing every technical result without context.
  3. While the authors mention prior work, much of this is mentioned in Appendix I rather than in the main text, and I believe there is a distinction between simply citing an earlier result and making its relationship to new analysis clear at the point where that analysis is presented. Without this, it is hard to assess which of the many results in this paper are novel contributions, extensions of prior work, or simply given for context.
Questions for authors
  1. Predictive coding as an algorithm typically requires more compute to do one weight update than backprop. Do you compare the performance of PC and BP when they are given the same amount of compute?
  2. Predictive coding requires multiple inference steps per weight update, making it potentially more computationally expensive than backpropagation. The paper compares methods over similar training durations or number of weight updates, but it is unclear whether their computational budgets are matched. Have the authors compared PC and BP under equal computational budgets (eg. training FLOPs, wall-clock time, or validation-based early stopping)? Controlling for this would help determine whether the supposed advantages of PC (eg. in test accuracy later in training) persist when computational cost is taken into account.
4 comments

Thank you for your review. We accept your comments on the motivation of the experiments, the narrative structure and the positioning relative to related work. The revision reorganizes Sections 4 and 5, stating the purpose, setup and meaning of each analysis. The first paragraph of the introduction now places the paper among prior work on how PC relates to BP and states the question it addresses: how the states obtained by inference enter the shared weights through local updates. Existing methods are introduced with their origin and role where they are used.

1. PC already outperforms PC+F and the intermediate rules. What more does the interpolation experiment show?

The interpolation experiment uses the PC+F working configuration. It asks two things: whether partially introducing the inferred state suppresses wrong-label memorization, and how the proportion introduced affects true-label recovery. Introducing the inferred state only in part already reduces wrong-label fitting markedly. The interpolation therefore shows how noise memorization changes with the composition of the update input, which complements the comparison between the two endpoints, PC and PC+F. Section 4.2 now states this purpose and conclusion.

We study those intermediate rules further for the following reason. Prior benchmark studies and work on deep PC show that standard PC is hard to train in deep networks, especially in convolutional networks with a limited inference budget, and Forward Update alleviates these problems. This raises the question of whether an update rule that improves training can also reduce wrong-label memorization by retaining part of the inferred state. The interpolation and intermediate-state experiments provide evidence on this question.

2. Why introduce ePC, frozen checkpoints and the theoretical propositions, and how do these analyses form one line of argument?

We agree with your concerns. In the revision, Section 4 of the main text focuses on PC and PC+F and on the interpolation and intermediate-state interventions. ePC and the other extension experiments have moved to Appendix C, as supplementary analyses that change the inference process.

We reorganized Sections 4 and 5 around two connected questions: which part of PC learning reduces noise memorization, and how does it do so?

Section 4 uses PC+F to separate the role of inferred targets from that of the inputs read by the weight update. Keeping inference unchanged while replacing inferred inputs with forward inputs increases noise memorization, showing that retaining the inferred targets alone is insufficient.

Section 5 then traces how inferred states shape learning, from target mismatch to accumulated memory. It first shows that state adaptation absorbs much of the output displacement, then uses controlled probes and theory to explain which components remain in the residual. It next examines how this residual and the adapted input jointly shape weight gradients. Finally, cross-example gradient coherence and continuation experiments connect these local effects to memory formation over training.

Frozen-checkpoint probes hold the learned weights fixed, allowing us to vary the target or update rule without introducing differences in training history. The theoretical analysis explains how state changes account for the reduction in target mismatch, and why finite-step inference leaves different amounts of that mismatch in the residual depending on its direction. In the revised version, we explained the purpose of each analysis before presenting it and stated the conclusion supported by its results.

3. Are PC and BP compared under matched computational budgets? (Q1, Q2)

Thank you for raising this issue. The original experiments primarily matched the number of weight updates and reported validation-selected performance, including the CIFAR-10N results in Appendix B.2, with the implications discussed in Appendix I. Following your suggestion, we added a wall-clock-matched comparison and included both the budget-endpoint and validation-selected results from these runs in Appendix A.4.

Wall-clock-matched comparison.

For each seed, we use the measured time required for PC to complete 50 epochs on CIFAR-10/VGG5 with symmetric corruption rate 0.4 as the budget. Both methods run on the same H100 with the same training and evaluation procedures. Timing includes data loading, per-epoch evaluation, and compilation. BP uses its benchmark configuration and original 50-epoch learning-rate schedule, then holds the terminal learning rate until the budget is reached. Runs stop after a complete epoch, so their actual duration slightly exceeds the target budget.

MethodEpochCumulative wall-clock time (s)Test accuracy (%)Wrong-label fitting (%)Clean fitting (%)
PC50109.1 ± 0.676.4 ± 0.414.3 ± 0.291.4 ± 0.2
PC (ES)36–4795.7 ± 13.476.3 ± 0.613.1 ± 1.490.7 ± 0.6
BP (epoch-matched)5086.4 ± 0.670.0 ± 0.460.0 ± 2.298.5 ± 0.1
BP (time-matched)64–65109.7 ± 1.169.1 ± 0.967.0 ± 1.698.9 ± 0.1
BP (ES)16–2034.3 ± 3.578.4 ± 0.77.6 ± 1.689.5 ± 1.3

Results are means ± sample standard deviations over three seeds; varying epochs are reported as ranges. ES checkpoints are selected using a clean validation split comprising 30% of the original test set; test accuracy is evaluated on the remaining 70%, as mentioned in the Appendix A.3, Table 5.

Under this matched-time setting, PC retains lower wrong-label fitting and higher test accuracy at the budget endpoint. Giving BP additional training time further increases wrong-label memorization and decreases test accuracy. The endpoint difference therefore persists after matching wall-clock time.

Checkpoint selection and learning trajectories.

Validation-based selection yields another clear result: BP reaches high test accuracy earlier and outperforms PC at its selected checkpoint. PC performs similarly at its selected and final checkpoints. Together, these results show that BP achieves better predictive performance through model selection, while PC maintains lower noise memorization as training continues. Our study investigates the learning mechanism behind these different trajectories.

The PC/PC+F comparison further examines this mechanism. Both methods perform the same inference procedure with the same number of inference steps and have similar per-step costs. Changing only the input read by the weight update increases wrong-label memorization. In this controlled comparison, the increase can therefore be attributed to the update-input change. Together with the clean-example filtering experiment in Section 3.3, these results direct the explanation toward how the weight update uses inferred states.

Scientific significance and implications for learning-rule design.

We use the difference in learning under label noise to investigate how inferred states become weight learning. The PC/PC+F comparison reveals that inference does more than provide local targets: whether the weight update reads adapted inputs also changes what the network ultimately memorizes. This identifies a concrete mechanism linking inference to parameter learning.

This understanding matters for extending PC learning methods. Prior work introduced modifications such as Forward Update to improve deep-network training, while our results show that these modifications also change PC’s noise-memorization behavior. As PC is made easier to train and extended to deeper networks, it is therefore valuable to examine which learning properties are retained and which change. The interpolation and intermediate-state experiments offer a concrete direction: retaining part of the inferred activity reduces wrong-label memorization while maintaining high test accuracy. This provides a testable design approach for combining improved training with this learning property in deeper architectures, and motivates broader evaluation across models and tasks.

I thank the authors for performing the additional experiments that we requested, such as the wall-clock-matched comparison. However, I agree with reviewer 2MJk that this paper remains difficult to parse even after extensive rewriting. Sections 4 and 5 are less dense than before, but still requires more work to make the main ideas accessible. In Section 5, the text overwhelmingly describes what the authors did rather than explain why each experiment follows naturally from the preceding one. Another example is Eq. 4, where A(y)A(y) is introduced with no explanation as to why this is relevant or a useful quantity to calculate. Because of this, my score will remain at 2.

Thanks for your follow-up review. Regarding A(y)A(y), we accidentally commented it out during the highlighting phase. There was originally an introductory sentence before the equation: "We measure their norms and summarize their separation using the per-image absorption ratio." This absorption ratio is the quantity shown in Figure 5b, and we will restore it in the revision. As for the reason for calculating it, we included the following explanation: "A larger ratio means that a smaller fraction of the output displacement remains as a residual after inference." Consequently, it directly answers the section title: "WHERE THE WRONG-TARGET DISPLACEMENT GOES."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.