acceptodds
Under review as a conference paper at ICLR 2027

Why MCPC generalizes

Abstract

Monte Carlo predictive coding (MCPC) infers latent causes by Langevin sampling, which makes it a generative model. We ask when an MCPC network generalizes during training and when it begins to memorize. Its local Hebbian updates average over the sampled cloud of latent states, so the cloud's spread acts as a ridge. For a linear–Gaussian coder we derive this ridge in closed form. It sets a threshold on the data spectrum, and directions farther above it are learned sooner. Population directions lie far above and set the generalization onset. Directions that exist only because the training set is finite lie just above and set a later memorization onset. The network generalizes without memorizing in the window between the two onsets. More data widens this window, and past a critical data size memorization never begins. Longer or more chains move the memorization onset but cannot prevent it. A larger step size can prevent it, but only by also losing the signal. The theory's learning flow reproduces both measured onsets. A pre-registered causal test shows that the ridge, not the sampling noise, sets the memorization onset. On CelebA faces more data likewise delays memorization while sample quality peaks at about the same step. To make MCPC generalize rather than memorize, stop training between the two onsets or train on more data than the critical size.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Monte Carlo Predictive Coding, generalize, memorization, generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7n2yOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper takes the linear-Gaussian version of MCPC (predictive coding model trained with a Langevin E-step and a Hebbian M-step) and asks when it generalizes and when it memorizes. The main idea is simple: averaging the Hebbian update over the sampled latent cloud adds the posterior covariance C to the latent second moment, so C acts as a ridge. Most of the results are for the linear case. The authors also show a CelebA tanh-decoder section showing MCPC samples faces from the prior.

Critical strengths

The core mechanism is clean and general. E[zzᵀ] = mmᵀ + C is a one-liner, but connects it to a closed-form ridge for a finite, biased chain. Also Fig. 4 matches the formula with no free parameters. I have also found interesting the causal ridge-only vs noise-only test as a piece of experimental design. More papers should do this instead of just correlating.

Critical weaknesses
  • Writing is extremely dense, sentences extremely short, and the notation very heavy. It was quite challenging to go through the main sections of the paper even just once. Also, a lot of the results are in the supplementary material, making it hard to understand everything.

  • Practical value is limited. By their own admission, preventing memorization via the sampler costs likelihood compared with a tuned explicit ridge. So why not just add the ridge? Also, the recommendation to “early stop or use more data” is quite general, and not specific to MCPC.

  • “Why MCPC generalizes” is answered only for a linear-Gaussian model with tied weights, a posterior-mean start and the accumulating rule.

Given that, in my opinion, the paper is not in a presentable state to be published, and given my concerns on the practical value, I recommend rejection.

Questions for authors

No direct questions

Reviewer_FkVLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates why Monte Carlo predictive coding (MCPC) generalizes and under what conditions it begins to memorize training data. The argument is that sampling from the latent posterior acts as an implicit ridge regularizer within the Hebbian update. Using a linear Gaussian model, the authors derive this term analytically to show which directions of data covariance get captured and which get suppressed. They then examine how dataset size, model capacity, and sampling hyperparameters steer these dynamics. They validate the theory using synthetic tests and simple datasets like MNIST and CelebA.

Critical strengths

The main strength is the theoretical analysis connecting posterior sampling to implicit regularization in MCPC. The derivation of the effective ridge induced by finite Langevin chains is useful, and the spectral analysis provides an intuitive explanation for why population-level structure can be learned before finite-sample fluctuations. I also appreciate the causal experiments that separate the effects of the covariance term from sampling noise.

Critical weaknesses

My main concern is the noticeable gap between the theoretical results and the paper’s broader claims. The analysis relies on quite restrictive assumptions: a linear Gaussian model, spectral alignment, and a constant sampling step size (for the fixed-point results). In contrast, the empirical setup uses an adaptive step size, and the text openly acknowledges that convergence to the aligned manifold is not guaranteed. More crucially, Appendix E.7 suggests that the drop in memorization in Table 1 is primarily driven by the regularization of stiff directions rather than the spectral thresholding emphasized in the main theory. The authors should disentangle these mechanisms much more clearly.

My second major concern is the nonlinear evaluation. In the CelebA experiments, roughly half of the generated samples undergo post-hoc sign correction before evaluation. Because this directly changes the distribution of prior samples, I am not convinced the reported FID reflects the generative capacity of the model itself. As it stands, the link between the nonlinear empirical results and the theoretical predictions remains rather weak.

Questions for authors

Q1) Could you clarify how much of the memorization behavior in the main experiments is explained by the spectral retention threshold vs the regularization of stiff directions discussed in Appendix E.7? These seem to be distinct mechanisms.

Q2) Could you report FID on CelebA without applying the post-hoc sign correction? It is critical to see how much the reported fidelity actually depends on this step.

Q3) The abstract claims that increasing the sampling step size can prevent memorization only at the cost of losing the signal. However, Appendix B.7 seems to allow regimes where sufficiently strong signal directions survive while fluctuation directions are suppressed. Is the trade-off claimed in the abstract specific to the experimental regime?

Reviewer_Wst3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates why Monte Carlo predictive coding (MCPC) can generalize before memorizing its training data. The authors analyze a two layer linear Gaussian predictive coder and show that finite Langevin sampling induces an effective ridge regularization. They derive a closed form characterization of this regularization and show how it determines which data directions are learned during training. Controlled experiments support the proposed mechanism, while additional experiments investigate related behavior in nonlinear image models.

Critical strengths
  1. Theoretical contribution. The paper derives a closed form characterization of the effective ridge regularization induced by finite Langevin sampling and connects it to a threshold governing which data directions are learned. This provides a mechanistic explanation for how MCPC can generalize before memorizing finite sample fluctuations, and how sampling parameters influence this behavior.

  2. Controlled experiments and systematic analysis. The authors use controlled interventions to distinguish the effects of induced regularization from sampling noise, providing evidence that regularization is the primary mechanism behind delayed memorization. They also systematically investigate the effects of dataset size, model width, and sampling parameters. The transparent reporting of preregistered predictions that were not supported by the experiments is another strength.

Critical weaknesses
  1. Limited theoretical scope and empirical validation. The theoretical analysis is restricted to a two layer linear Gaussian predictive coder with tied maps. While the authors investigate nonlinear image models, these experiments provide only partial evidence for extending the theoretical conclusions to more complex settings. The paper also notes that natural image spectra generally do not satisfy the assumptions underlying the theory, limiting the extent to which these experiments directly validate its predictions.

  2. Practical implications and empirical limitations. Some preregistered predictions are not supported by the experiments. Furthermore, explicitly tuned ridge regularization achieves comparable or better held out likelihood in some settings, leaving the practical advantage of MCPC over conventional regularization less clear.

Questions for authors
  1. How might the theoretical analysis be extended to deeper or nonlinear predictive coding models, and what are the main challenges in doing so?

  2. Given that explicitly tuned ridge regularization can achieve comparable or better held out likelihood in some settings, what advantages does MCPC offer over explicit regularization, if any?

Comments on the paper

1 comment

Thank you for the detailed response and additional experiments. You have adequately addressed my specific concerns, and I appreciate the clarifications and corrections. That said, I am still leaning toward rejection, primarily because of the limited scope and incremental nature of the contribution. For a future submission, I would encourage you to strengthen the connection between the theory and nonlinear models, and more clearly establish what new insights the analysis provides beyond existing results on implicit regularization.

I appreciate the effort that went into the rebuttal, but I will retain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.