- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper investigates why and how predictive coding (PC) networks trained with symmetric label noise memorise fewer incorrect labels than backpropagation (BP). On CIFAR-10 the authors show that PC fits clean examples and its test accuracy quite well, while BP fits the wrong labels late in training and its test accuracy falls. Control experiments argue that this is a feature of PC's inference process and do this by testing a set of other options including how wider networks, inference length, and removing misclassified clean examples affect PC. The authors also use the forward update rule (PC+F), interpolation of the layer-wise input (from forward to the inference state), and an ePC step-size sweep to give more evidence for their finding. Reading the feedforward input instead of the inferred input increases noise memorization, showing that the inference state really does matter in a clean and well constructed comparison.
Critical strengths
The investigation explored in this paper really does matter. Careful, step-by-step analysis of what actually happens inside a PC network during learning is needed if we want to understand how PC differs from BP. It is also needed to understand whether PC’s useful properties, such as absorbing or ignoring label noise, can be kept or transferred to other networks. I value this direction of research.
Furthermore, the central experimental idea is quite clean and neat. PC+F and the λ intepolation process keeps the inferred target fixed and changes only the input the weight update reads, which isolates one part of the learning rule. The fixed-weight paired probes, using the same image and weights under the true and the wrong label, are also a good way to look inside a single update without the confounds of full training.
Finally, the authors do a lot of work to isolate potential confounds including random-label capacity, network width, inference time length, optimizers, more datasets etc.
Critical weaknesses
This paper has a lot of weaknesses. The weaknesses are almost all centred around the lack of human readability of this work and the lack of structure that it has. As an overarching point, this work should be far more clear about where this paper fits into the field, exactly how experiments are constructed and what they mean, and what the takeaways of this work should be. Experimental clarity and paper narrative are almost completely missing from the main text. As a major point, almost none of the figures are readable and all have some major issue with how they are formatted or described. Some explanations are present in the appendix but the main work cannot be read without constant referencing to the appendix for missing information and context (some of which is not even present there).
To be more specific about these issues, I want to give some examples.
- Very short introduction: The introduction barely describes any of the existing work on understanding the difference between PC model and BP model learning
- No experimental setup section: Experiments are poorly (or not at all) described in the text, and the reader has to move around the paper, and to other papers, to understand them. This is largely because the main text has no clear setup section. Basic terms are not defined where they are used. For example, it is never explained in the main text what a corrupted sample is or how symmetric corruption is generated. Model names appear without definition, for example “Fashion-MNIST/MLP5” (L198). Hyperparameters that matter are spread across appendix tables etc.
- Inference length choices are never explained: The number of inference steps clearly matters a lot. Some experiments use 7 steps (Table 4), the reference run for filtering uses T = 5 (Appendix B.2), and Figure 4 uses 5, 10, 15 and 20. I could not find where or how these values were chosen, or why T is not fixed or tuned in a rigorous way. Because the main claim is about what inference does to the weight update, this needs to be explicit. It also means this paper is probably impossible to reproduce.
- Figures are hard to read and poorly explained: Several figures (e.g. Figure 2, Figure 6) have legends that overlap the curves and very small axis text. They do not look as though they were checked carefully before submission. The captions are also in general far too short and some can't even be understood just by reading the main text. Some figures with major issues: Figure 2d (cannot be understood clearly), Figure 3 (“Width multiplier” is never defined), Figure 5 (not clearly defined in text), Figures 7 and 8 (too close together), Figure 8c (s0, s1 and s2 are never explained) etc.
- No discussion and no clear conclusions: The main text goes from results straight to a one-paragraph conclusion. There is no discussion of what the findings mean, how they relate to earlier PC-vs-BP results, or what a reader should take away.
All in all, the paper is hard to follow: it is often unclear what is plotted, why, and how. In its current form the presentation does not allow a reader to properly assess the contribution. This is a pity given that I think that the actual question being explored is important.
Questions for authors
Equation 7 mixes the inferred and forward states linearly with λ. This is a kind of linear interpolation. By why not instead use the value of the state at different values of t? Is the state from 0->T really a linear interpolation?
How exactly are corrupted samples generated for the earlier figures of your work? And could you possibly identify corrupted samples in the actual underlying dataset using the PC models?
3 comments
Dear authors,
Thank you for your responses, especially to my questions which now are a bit more clear. I have examined the paper once again but I still find a number of aspects of it to be inadequately formulated and formed if the intention is for it to be published in it's current state.
First, I don't believe my core concerns about human readability have been addressed. The paper still has very little narrative and jumps from point to point with little flagging or clearly trackable logic as to why the next test/step is necessary, interesting, or informative. Figures are still far too small to be readable in an A4 printed format and I needed to zoom/lean-in to read all axis labels and text. The 'Experimental Setup' section help alleviate some of my previous concerns but is so rushed that it does not actually describe in any detail the system you are simulating and does not set up any of the notation for the rest of the paper. In lieu of a method section the paper should do this somewhere (could also be in the introduction). Finally, though you indicate that you added narrative structure to Sections 4, 5, and the Discussion, I still find these to be head-spinningly disorienting. One specific request would be to not just mention what you study next but why. The discussion, for my liking, is also far too short and does not cover the impact of this work
In all, I would still recommend major changes to this paper before it could be considered a readable contribution. In your rebuttal you mention that: To meet the page limit, we excessively compressed the experimental rationale, captions, and figures, making it difficult to understand how the experiments were constructed and what questions their results answered., I think the degree of compression you have tried to achieve is hurting this paper significantly. If the kind of contribution you would like to make has so many separate and independent contributions that do not fit easily into this page limit I might recommend an alternative venue (full journal submission) where the page limit does not impact your communication.
Finally, since the changes in this version (compared to the previous version) are not indicated in any way, it is difficult to track exactly what changed.
Given all of the above, I opt not to modify my current recommendation for this paper's publication.
Best wishes, Reviewer 2MJk
Dear Reviewer 2MJk,
Thank you for your follow-up review.
Regarding the size of the figures: we apologize for overlooking the axes and legends previously, we have now enlarged them as much as possible. As for the experimental setup, we have revised this section and included references to the more detailed descriptions and notation provided in the appendix.
Regarding the logic of the article, we realized that the previous descriptions were more from our perspective as the ones who implemented the models and ran the experiments, which created logical gaps from a reader's point of view. We have therefore thoroughly rewritten this section to ensure smoother transitions. In the current version, changes made since the previous revision are highlighted in red, while key differences between the previous revision and the original submission are highlighted in blue. We hope this makes it easier for you to track the changes.
Thank you once again for your review, and we apologize for the additional effort caused due to the unmarked changes previously.
Best regards, The Authors
Thank you for carefully reading our paper and recognizing the value of the research question. We acknowledge your concerns about readability and experimental clarity. To meet the page limit, we excessively compressed the experimental rationale, captions, and figures, making it difficult to understand how the experiments were constructed and what questions their results answered. We have revised the manuscript as follows.
Research context and experimental setup. In the first paragraph of the Introduction, we added related work on the learning properties of PC, clarified its relationship to our research question, and directed readers to the more detailed related-work discussion in the appendix. Section 2 now brings together the label-corruption procedures, models and datasets, hyperparameter selection, and evaluation metrics, with references to the full protocols in Appendix A.
Inference steps and hyperparameters. Hyperparameters were searched using TPE on datasets without label corruption, and the selected configurations were subsequently used in the noisy-label experiments. Each method uses its own searched configuration, which explains why the number of inference steps differs across configurations. For the width and inference-step experiments, we separately searched the remaining hyperparameters for each setting. Appendix A now collects the configurations actually used and maps them to the corresponding experiments in the main text.
Figures and captions. We regenerated and reorganized the figures, using consistent method colors: red for PC, green for PC+F, and blue for BP. We also assigned consistent line styles to wrong-label fitting, clean fitting, true-label recovery, and test accuracy. The captions now provide additional experimental settings and explain the axes and markers. We also adjusted legend placement and panel layouts.
Narrative structure and discussion. We reorganized Sections 4 and 5 so that each analysis answers a question raised by the preceding results. Section 4 holds the inference procedure fixed and changes the input read by the weight update to establish how the use of inferred states affects noise memorization. Section 5 first shows how state adaptation absorbs output displacement, then examines which directions remain in the residual. It subsequently investigates how these residuals and adapted inputs form weight gradients, and finally connects individual updates to memory accumulation through cross-example gradient coherence and continuation experiments. The main-text Discussion explains what these findings mean for understanding PC and designing learning rules; the appendix provides further mechanistic discussion, limitations, and future directions.
Q1: Why use linear interpolation rather than states at different points along the inference trajectory?
Thank you for this suggestion. Linear interpolation directly controls the composition of forward and final inferred states in the update input; it does not assume that the actual inference trajectory is linear. Following your suggestion, we added intermediate-state experiments: after completing the same inference procedure, we keep each layer’s final inferred target fixed and replace only the weight-update input with ht, a state along the actual inference trajectory.
Section 4.2 now presents these two interventions together. The intermediate-state experiments show that reading inferred activities before they reach their final states can already reduce wrong-label memorization. We also conducted an independent hyperparameter search for a selected intermediate-state rule and included the results in the same comparison. The full configurations and training trajectories are provided in Appendix C. Together, these experiments show that partially incorporating inferred activities can already achieve low noise memorization while maintaining high test accuracy.
Q2: How are corrupted examples generated? Can PC identify mislabeled examples in the dataset?
We now specify the procedure in Section 2. For symmetric label noise at rate ρ, we randomly select the corresponding fraction of training examples and uniformly replace each selected label with one of the other classes, excluding its original class. The images themselves remain unchanged.
For label-error identification, Appendix G presents the forward-displacement distributions of correctly labeled and mislabeled examples, including CIFAR-10N human annotations and synthetic symmetric noise. The histograms show that, at the end of training, PC retains a high-displacement region enriched in mislabeled examples, suggesting that this signal can help identify some label errors. However, the distributions overlap substantially: difficult correctly labeled examples can also have large displacements. Appendix G therefore discusses incorporating cross-example gradient directions as a direction for further research.