acceptodds
Under review as a conference paper at ICLR 2027

Local M-Covers for Replicated Learning

Abstract

Replicated learning temporarily introduces multiple copies of a model, modifying the training dynamics in ways that can improve generalization while allowing the replicas to contract back to a single deployable model. We ask whether the internal structure of this temporary lift can be tuned so that these replica fluctuations induce task-relevant corrections. We study this through an -cover, which replicates and rewires a model's factor graph, and introduce a local -cover in which replica fluctuations interact through structured rather than uniform coupling. For a general model, we derive the dynamics of replica-difference modes near synchronization and show that generalization improves when their induced predictor displacement aligns with the source-posterior correction. In an analytically tractable random-feature model with two trainable linear layers, this yields a concrete link between source (teacher-prior)–representation mismatch and the preferred cover geometry, and predicts the transition between improved and degraded generalization. Across broader benchmarks, structured -covers outperform established replicated-learning methods across tasks, architectures, learning dynamics, and overparameterization regimes. These results identify the internal geometry of temporary replication as a new design axis for generalization: structured replica interactions can improve the final single-model predictor without increasing deployment-time model size.

Primary area:
learning theory
Keywords:
replicated learning, generalization, local entropy, structured optimization, random features, overparameterization, spectral learning theory, replica coupling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mo3BOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how structured interactions among replicated models can improve the generalization of a final single model. It proposes a local M-cover that combines each replica's task loss with a layerwise disagreement penalty. The formulation generalizes replicated SGD through nonuniform replica mixing and a function-sensitive parameter metric. Deployment uses a network obtained by averaging replica parameters.

The theoretical contribution is a local analysis near synchronization. The authors derive the dynamics of replica disagreements, conditions for their contraction, and the leading predictor displacement induced by their influence on the average model. They connect this displacement to a conditional improvement criterion under posterior-averaged squared error: sufficiently small displacements help when aligned with the correction toward the Bayes posterior predictor.

Validation combines synthetic experiments and broader benchmarks. A random-feature model with two trainable linear layers and a Gaussian source prior enables numerical evaluation of the theoretical alignment criterion. Ring-based experiments compare predicted improvement boundaries with simulations and investigate how source–representation mismatch affects preferred mixing. Additional experiments cover image classification, language modeling, physics-informed PDE solving, and time-series forecasting, comparing against single-model training, replicated SGD, Elastic SGD, and SAM where applicable. These benchmarks use a Euclidean coupling metric and report improvements over the single-model and replicated-learning baselines. Supplementary experiments examine alternative graph structures and different numbers of replicas.

Critical strengths
  • Descriptive contribution: explaining how replica interactions can affect generalization. The paper develops a local theory showing how temporary differences between model copies change the average model's training trajectory and final predictions. It connects this prediction change to a condition for improved generalization and evaluates the theoretical predictions in synthetic experiments. This provides a framework for understanding why the interaction pattern between replicas can matter.

  • Prescriptive contribution: proposing the local M-cover training framework. The local M-cover generalizes replicated SGD by introducing structured interactions between replicas and function-sensitive measure of their disagreement. It provides a concrete way to modify replicated training while keeping a single deployed model. Experiments using the Euclidean-metric report gains across image classification, language modeling, PDE solving, and time-series forecasting. This breadth supports the framework's potential beyond a single task or architecture.

Critical weaknesses
  • The connection between task properties and useful interaction structures is demonstrated only in a restricted setting. The random-feature experiments link source–representation mismatch to preferred mixing within the ring-diffusion family. However, the alternative graph structure experiments compare validation errors without checking whether the theory predicts their displacements or alignment. It will greatly support the analysis if more discussions about how structures mechanistically influence can be presented in the paper.
Questions for authors
  • Clarification of scope:
    • Is the connection between task properties and preferred interaction structure intended to extend beyond the ring-diffusion family? If so, what existing analysis or evidence connects the predicted displacement or alignment to the alternative graph structures' results?
    • The alignment condition seems general enough and applies to other methods, what does the analysis establish specifically about the advantages of local-M cover, besides providing a way to calculate the predictor displacement?

A theoretical comparison (like with rings) of predicted alignment across graph structures would be useful future work. I am not requesting a new experimental study for the rebuttal; the demonstration of the paper can be sharpened to emphasize the major contributions.

2 comments

W1) and Q1) Is the connection between task properties and preferred interaction structure intended to extend beyond the ring-diffusion family? If so, what existing analysis or evidence connects the predicted displacement or alignment to the alternative graph structures' results?

Thank you for highlighting this point. Our improvement criterion is not restricted to ring coupling: it applies to any interaction graph satisfying the assumptions of our local analysis.

The posterior-alignment criterion in Eq. (24) follows from the risk identity in Eq. (23). The MM-cover-induced predictor displacement Δf(T)\Delta f(T) is defined in Eqs. (19)-(21) and depends on the transverse operators HqH_q defined in Eq. (15).

The general graph structure, not just a one-dimensional ring, enters through these transverse operators appearing in Eq. (15), which depend on the eigenvalues κℓq\kappa_{\ell q} of the replica interaction matrices LℓL_\ell.

Thus, the theory applies to a wide variety of mixing topologies beyond the ring.

For illustration, Fig. 2 evaluates this general criterion for the ring-diffusion family parameterized by ρ\rho, using the finite-time specialization in Eq. (73).

We have clarified this more general scope in the manuscript by adding a sentence at the end of the General Theory section.

Q2) The alignment condition seems general enough and applies to other methods, what does the analysis establish specifically about the advantages of local-M cover, besides providing a way to calculate the predictor displacement?

We agree that the posterior-alignment condition is general and is not, by itself, specific to local MM-covers.

The specific contribution of our theory is the detailed explanation of how the posterior-alignment condition depends on the mixing topology via the transverse operators HqH_q of the mixing matrix, through Eqs. (15)–(17). Eq. (21) then shows how these forces accumulate along training.

We can then analyze how the posterior-alignment condition depends on the persistence of cover-to-cover fluctuation modes. Uniform rSGD is a special case: it has only one general consensus mode. Local MM-covers provide additional control. For the 1D ring mixing, we have another fluctuation mode parametrized by rho (related to the width of the local coupling). The theory in principle extends to understanding the contribution and designing mixing matrices that exploit additional modes.

One way to express this is that our theory shows that the M-cover is a multi-mode extension of rSGD (in terms of the mixing structure).

Q3) A theoretical comparison (like with rings) of predicted alignment across graph structures would be useful future work. I am not requesting a new experimental study for the rebuttal; the demonstration of the paper can be sharpened to emphasize the major contributions.

We agree that comparing predicted alignment across graph structures is a useful direction for future work.

As stated in the response to your previous question, this is doable by looking at the mixing mode distribution of these other graph structures, and how the lifetimes of resulting cover-to-cover fluctuation mode affect the generalization improvement.

Our conclusion points toward this by proposing to "adaptively learn richer layer- and time-dependent mixing spectra and eigendirections, guided by estimated source-prior alignment, beyond our single-parameter family." We have also clarified the general scope of the criterion to better emphasize the paper’s contributions.

We also encourage you to read our responses to the other reviewers and would appreciate your support for our paper.

Thank you for the clarifications. The added sentence on scope and the description of the M-cover as a multi-mode extension of rSGD address my question about the specific contribution. The connection between the predicted displacement and the alternative-topology results remains untested, which I accept as future work. I maintain my score.

Reviewer_qqkwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper trains several coupled copies of a network and averages them into a single model. Standard replicated SGD couples every copy uniformly to the mean while the authors replace this with a structured disagreement penalty built from a replica-graph Laplacian, such as a ring, optionally weighted by a functional metric. They motivate it as a simplified version of the graph-cover construction from statistical physics. Their theory says the temporary disagreements between copies leave a small, lasting shift in the final model, and that generalization improves when this shift points toward the Bayes-optimal predictor. The method is then verified on 7 diverse benchmarks spanning vision, language modeling, PDE solving and forecasting.

Critical strengths

The method also behaves consistently across very different settings. It plugs into SGD, AdamW and quasi-Newton optimizers, and into MLPs, ResNets, Transformers and PINNs, without architecture-specific changes. It never underperforms the single-model baseline, and where SAM helps, combining the two gives further gains. This suggests the coupling is a general-purpose component rather than something tuned to one benchmark.

Critical weaknesses

[W1] There is no related works section and it is not obvious why some of the baselines were not chosen. The following papers should be cited and aptly represented in the comparisons or discussions. I do not think SAM represents the entirety of prior work on this.

  1. Tongtian Zhu, Fengxiang He, Lan Zhang, Zhengyang Niu, Mingli Song, and Dacheng Tao. Topologyaware generalization of decentralized sgd. In International Conference on Machine Learning, pp. 27479–27503. PMLR, 2022.
  2. Zhu, T., He, F., Chen, K., Song, M. & Tao, D.. (2023). Decentralized SGD and Average-direction SAM are Asymptotically Equivalent. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:43005-43036 Available from https://proceedings.mlr.press/v202/zhu23e.html.
  3. Kong, L., Lin, T., Koloskova, A., Jaggi, M., & Stich, S. (2021, July). Consensus control for decentralized deep learning. In International Conference on Machine Learning (pp. 5686-5696). PMLR.
  4. Jain, S., Addepalli, S., Sahu, P. K., Dey, P., & Babu, R. V. (2023, June). Dart: Diversify-aggregate-repeat training improves generalization of neural networks. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 16048-16059). IEEE.
  5. Jolicoeur-Martineau, A., Gervais, E., Fatras, K., Zhang, Y., & Lacoste-Julien, S. (2023). Population parameter averaging (papa). arXiv preprint arXiv:2304.03094.

[W2] The replicated methods process M times as many samples per step as Base. So part of the gain over Base could just come from more data throughput. A large-batch Base (batch M·B, retuned learning rate) and an M×-epoch Base are needed. This also pertains to the significance of the results since all the results are within standard errors of each other right now.

Questions for authors

Please address the weaknesses.

The following was unclear to me -

  1. Beyond the coupling schedule and the cρ\frac{c}{\rho} multiplier, is there any algorithmic difference between the implemented method and Laplacian-coupled decentralized SGD on the same topologies?
  2. If not, what does the M-cover view contribute? Does it predict which topology or ρ\rho to choose in a way that existing topology-aware analyses of decentralized SGD do not?
2 comments

W1) and Q1) Related works and difference with decentralized SGD

We thank the reviewer and have added a related-work appendix. The first three papers study decentralized SGD, including Zhu et al. (2022), which is already cited in the introduction of our initial submission. Our formulation differs substantially. The main differences are:

  • We define cover interactions directly in the loss derived from the local M-cover lift, whereas Zhu et al. (2022) only defines parameter mixing in the SGD update.

  • As a consequence, the compatibility gradient can be incorporated into optimizers such as momentum SGD, AdamW, and weight decay, as pointed out in your assessment of the paper’s strengths. These combinations were not investigated in Zhu et al. (2022).

  • This more general framework allows us to benchmark across more diverse architectures and tasks (image classification, language modeling, PDE solving, and time-series forecasting).

  • Importantly, our theory addresses a different question: Zhu et al. (2022) derive topology-dependent stability and generalization-gap bounds, whereas we compute the finite-time predictor displacement induced by replica fluctuations. This allows us to characterize when that displacement improves posterior-averaged population risk, under our local gradient-flow assumptions.

  • Note moreover that, in principle, our framework also permits nonidentity functional metrics GℓG_\ell, although the benchmarks (other than Fig. 2) use Gℓ=IG_\ell=I.

  • Zhu et al. (2023) and Kong et al. (2021) study extensions of the decentralized-SGD framework, so the formulation distinctions above also apply. Zhu et al. 2023 use a related near-consensus expansion; but our analysis additionally computes accumulated predictor displacement (Eq. 21) and a conditional improvement boundary (Fig. 2). Kong et al. focus on controlling gossip-averaging rounds on a fixed graph, rather than designing the interaction geometry.

  • DART (Jain et al., 2023) and PAPA (Jolicoeur-Martineau et al., 2023) use direct parameter averaging, which is related to consensus coupling. Our contribution is focused on structured replica coupling. We already discuss related averaging approaches, including Elastic Averaging SGD.

W2) The replicated methods process M times as many samples per step as Base. So part of the gain over Base could just come from more data throughput. A large-batch Base (batch M·B, retuned learning rate) and an M×-epoch Base are needed. This also pertains to the significance of the results since all the results are within standard errors of each other right now.

Thank you for this important question.

First, we would also like to clarify that the results are not all within statistical uncertainty of each other: CIFAR-10 (updated results), TinyStories, ETTm2, and Fashion-MNIST show improvements over rSGD with 95% paired confidence intervals excluding zero.

Note that rSGD and M-cover already use identical data throughput, providing a matched-throughput comparison. Second, we would like to clarify that our experiments show that m-cover performs better than rSGD and base for the same amount of training data pool.

Concerning the number of data sampled per step with replacement (or batch size), larger batches or more epochs do not automatically improve Base performance. We additionally tested both Base controls on CIFAR-10 and Fashion-MNIST: larger batches and more epochs did not close the gap, and M-cover retained lower mean test error in every comparison.

On CIFAR-10, increasing Base’s batch size sixfold to 768 also matched the update count and yielded 6.066% test error. Training Base for 600 epochs at batch size 128 yielded 5.700%, versus 5.545% for M-cover on the same ten seeds.

On Fashion-MNIST, increasing Base’s batch size fivefold to 640, with validation-based learning-rate retuning, yielded 10.34% error. Training for 1,500 epochs yielded 9.76%, versus 9.44% for M-cover. All ten paired runs favored M-cover over the longer-trained Base, with a mean gain of 0.321 percentage points. The training curve showed no sustained improvement after epoch 300 (see new Fig. 7). Note that Fashion-MNIST’s paired gain over rSGD is 0.171 percentage points (95% CI [0.083, 0.259]).

Q2) Does [M-cover] predict which topology or to choose in a way that existing topology-aware analyses of decentralized SGD do not?

Yes, the M-cover viewpoint helps identify promising mixing regimes. In Fig. 2, the theory predicts improvement over Base through the predictor displacement in Eqs. (19)–(21) and the posterior-alignment criterion in Eq. (24), evaluated using Eq. (73). Theory identifies a small ρ≪1\rho \ll 1 regime where local ring coupling outperforms the baseline. The same theoretical criterion can, in principle, be evaluated for other parameterizations of the mixing matrix beyond 1D ring to identify beneficial regimes under its assumptions.

Reviewer_qqkwOpenReview ↗

Reply to Rebuttal

Makes sense to me.

I still advise the authors to slightly change the framing of the paper in the final version based on the new results (also put the related works in the main text). However, I think my soundness concerns are gone.

I am increasing to weak accept as I think the paper is overall interesting.

Reviewer_V7YROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how structured interactions among temporary model replicas affect generalization. It proposes a local M-cover framework, which couples replicas during training while retaining a single model at deployment. The paper analyzes local replica dynamics and posterior risk, using a random-feature model to relate source-representation mismatch to the preferred coupling geometry. It conducts experiments on MNIST, Fashion-MNIST, CIFAR-10, Tiny ImageNet, TinyStories, PINNs, and ETTm2 to assess generalization gains.

Critical strengths
  • Treating replica interaction geometry as a design choice is an interesting contribution beyond uniform coupling.
  • The local dynamics and posterior-risk analysis explain how temporary replication can alter a single deployed predictor.
  • Controlled random-feature studies provide interpretable checks of the proposed mechanism.
Critical weaknesses
  1. The implemented coupling may differ from the analyzed objective. Tiny ImageNet uses I−QbI-Q_b, while Eq. (8) yields I−Sym⁡(Qb)I-\operatorname{Sym}(Q_b). These operators differ unless QbQ_b is symmetric; the unspecified permutations leave this correspondence unresolved. (pp. 3, 22)

  2. The theory only partly explains the stochastic benchmarks. Eq. (21) models initial perturbations under deterministic flow, whereas TinyStories starts identical replicas and generates differences through stochastic optimization. The resulting noise-driven gains are not quantified by that formula. (pp. 5, 23)

  3. Evidence for broad generalization gains is uneven. CIFAR-10 and Tiny ImageNet paired confidence intervals include zero; PINN results use one seed and evaluation-grid selection. These disclosed limitations weaken the broad improvement claim. (pp. 9, 21, 23, 25)

Questions for authors
  1. Can the posterior-alignment criterion guide coupling selection without access to the teacher posterior?

  2. How much does the assignment of replica interactions across layers matter beyond the mixing spectrum itself?

1 comment

W1) Tiny ImageNet uses I-Q, while Eq. (8) yields I-Sym(Q).

We thank the reviewer for identifying this mismatch. We agree that these operators do not coincide for this particular task.

To resolve this issue, we reran Tiny ImageNet M-cover using the symmetric ring-heat kernel used in Fashion-MNIST with Q=Q⊤Q=Q^\top. It now agrees with Eq. (8).

We kept the original training settings and evaluated on the same eight final seeds. The resulting mean error is 42.424%, compared with 42.427% for the original implementation, 42.499% for Base, and 42.503% for rSGD. Thus, the symmetric implementation yields essentially the same observed mean performance.

We have updated Table 1 and revised appendix accordingly.

W2) The theory only partly explains the stochastic benchmarks.

The theory is based on gradient descent flow (GD) by design. GD theory isolates how interaction geometry controls replica fluctuations and how their effects accumulate into predictor displacement and changes in posterior-averaged population risk in the large batch limit.

Recent papers study minibatch stochasticity in simplified models, including Zhang et al., How Does Critical Batch Size Scale in Pre-training? (ICLR 2025), and Bordelon and Mori, Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model (2026). The latter trains only a linear readout on fixed features, whereas our model jointly trains two linear layers.

Recent theoretical developments therefore do not directly apply to our random feature plus two-layer model or its M-cover coupling. Extending them to this setting would require developing a theory not yet available for the Base model, then extending it to its M-cover. Such an extension lies beyond the scope of the present deterministic analysis, but we agree that it is an interesting direction for future work.

We have revised the future-work discussion accordingly.

W3) Evidence for broad generalization gains is uneven.

Note that Fashion-MNIST and TinyStories already show improvements over the reported baseline comparisons with confidence intervals excluding zero.

To strengthen the evidence on CIFAR-10, we ran 25 additional paired seeds with all hyperparameters fixed. Combining these with the original ten seeds gives mean errors of 5.500% for M-cover and 5.628% for rSGD, with a paired improvement of 0.128 percentage points (95% CI: [0.023, 0.233]). This CI does not cross zero.

On ETTm2, the five paired seeds give 95% confidence intervals for MSE improvements of [0.00008, 0.00040] over rSGD and [0.00121, 0.00337] over Base, both excluding zero; we have added the details to the appendix.

For PINNS, we have run 9 additional seeds which confirm that the mean improvement of m-cover is better than rSGD and base (see Appendix).

M-cover achieves lower mean error than both Base and rSGD on all seven benchmarks in Table 1. This consistency across benchmarks, together with CI excluding zero on multiple tasks, supports our conclusion that M-cover can improve generalization across diverse tasks and architectures.

We have updated Table 1 and revised the appendix to report the additional confidence intervals.

Q1) Yes, indirectly. Without a specified teacher prior, we cannot evaluate the posterior-alignment criterion directly or use it to compute an optimal mixing matrix.

Nevertheless, this criterion can guide which mixing coupling regimes to explore without the exact knowledge of the teacher source. In the random-feature model of Fig. 2, the theory predicts a transition to improvement over Base under standard assumptions on the input distribution for small nonzero ρ\rho; where local ring coupling outperforms empirically both Base and the fixed uniform rSGD reference in various models.

Q2) In principle, an ideal mixing matrix depends not only on its spectrum but also on its alignment with the eigendirections of the teacher prior. The random-feature model of Sec. 4 makes a shared-basis assumption in function space. In Fig. 3, the teacher prior and the population random-feature covariance are diagonal in the same basis. We therefore vary the amount of teacher power in each feature direction while holding those directions fixed. In practice, the teacher’s dominant directions may be rotated relative to the representation modes.

A direction for future work is to use training or validation signals to learn the mixing eigendirections adaptively across layers and over time. This is what is hinted at in the last sentence of the paper "and adaptively learn richer layer- and time-dependent mixing spectra beyond our single-parameter family."

The simpler setting of the current paper already shows that gains are possible without using this detailed knowledge of the eigendirection of source prior. We have added a mention of this interesting opportunity in the Discussion. Thank you for raising this question.

Comments on the paper

1 comment

Human assessment: Most of my concerns have been addressed. I increased my score. I am not an expert in this field so I mostly rely on AI for review, and put a low confidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.