- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper takes the linear-Gaussian version of MCPC (predictive coding model trained with a Langevin E-step and a Hebbian M-step) and asks when it generalizes and when it memorizes. The main idea is simple: averaging the Hebbian update over the sampled latent cloud adds the posterior covariance C to the latent second moment, so C acts as a ridge. Most of the results are for the linear case. The authors also show a CelebA tanh-decoder section showing MCPC samples faces from the prior.
Critical strengths
The core mechanism is clean and general. E[zzᵀ] = mmᵀ + C is a one-liner, but connects it to a closed-form ridge for a finite, biased chain. Also Fig. 4 matches the formula with no free parameters. I have also found interesting the causal ridge-only vs noise-only test as a piece of experimental design. More papers should do this instead of just correlating.
Critical weaknesses
-
Writing is extremely dense, sentences extremely short, and the notation very heavy. It was quite challenging to go through the main sections of the paper even just once. Also, a lot of the results are in the supplementary material, making it hard to understand everything.
-
Practical value is limited. By their own admission, preventing memorization via the sampler costs likelihood compared with a tuned explicit ridge. So why not just add the ridge? Also, the recommendation to “early stop or use more data” is quite general, and not specific to MCPC.
-
“Why MCPC generalizes” is answered only for a linear-Gaussian model with tied weights, a posterior-mean start and the accumulating rule.
Given that, in my opinion, the paper is not in a presentable state to be published, and given my concerns on the practical value, I recommend rejection.
Questions for authors
No direct questions
Thank you for the detailed response and additional experiments. You have adequately addressed my specific concerns, and I appreciate the clarifications and corrections. That said, I am still leaning toward rejection, primarily because of the limited scope and incremental nature of the contribution. For a future submission, I would encourage you to strengthen the connection between the theory and nonlinear models, and more clearly establish what new insights the analysis provides beyond existing results on implicit regularization.
I appreciate the effort that went into the rebuttal, but I will retain my original rating.