acceptodds
Under review as a conference paper at ICLR 2027

Evidence-based Model Alignment for Uncoordinated One-Shot Federated Learning

Abstract

One-Shot Federated Learning is a paradigm in which a global model is learned from data distributed across multiple clients in a single communication round. This setting becomes particularly challenging when the training process is uncoordinated, i.e., with clients starting from different initializations and training for different numbers of epochs. We analyze this problem from a Bayesian perspective, showing how the permutation symmetries of client posteriors can be exploited to improve the resulting global model. Based on these observations, we propose FLAME, a principled approach that identifies client permutations for model alignment by maximizing the Bayesian evidence of the merged model, and then performs Bayesian merging, thus handling permutation symmetries and weight uncertainty in a unified manner. We show on standard benchmarks that our proposal outperforms competing approaches that typically rely either on permutation-based matching or on client loss curvature alone.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
One-shot Federated Learning, Model Merging, Bayesian Inference, Laplace Approximation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ghFzOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the model merging problem in uncoordinated one-shot federated learning and proposes a method called FLAME. It first employs Laplace Approximation to approximate the local posterior distribution of each client, then identifies the optimal neuron permutation by maximizing the approximate Bayesian Evidence of the merged model, and finally performs Bayesian Merging on the aligned posteriors. The paper also provides theoretical derivations and empirical evaluations to demonstrate the effectiveness of the proposed method.

Critical strengths

1、The work demonstrates a degree of methodological innovation.

This paper proposes a model alignment framework based on Bayesian Evidence, which incorporates posterior parameter uncertainty into the process of matching and permuting neurons—moving beyond a reliance solely on traditional Euclidean weight similarity—thereby offering a fresh perspective on model alignment and merging.

2、The theoretical derivation is comprehensive.

The paper provides a detailed theoretical derivation of the proposed Bayesian Evidence objective. It demonstrates that under the assumption of an isotropic posterior, this objective reduces to the conventional "weight matching" approach, thereby establishing a theoretical link between the new method and existing techniques.

Critical weaknesses
  1. There is a lack of direct validation for the core Bayesian evidence alignment criterion.

While this paper selects neuron permutations by maximizing an approximate Bayesian evidence, the method is evaluated primarily based on final classification accuracy, so there is no direct analysis of the relationship between the evidence score and the performance of the merged model. Consequently, whether higher evidence consistently correlates with better model merging outcomes is unclear. And the marginal benefit of using the full Bayesian evidence, as opposed to relying solely on the curvature-weighted misalignment term, varies significantly across datasets, and on CIFAR-10, it almost no improvement. Therefore, I have skeptical regarding the necessity and stability of the full Bayesian evidence objective

  1. The experimental validation across different model architectures is insufficient.

The primary experiments focus on LeNet and smaller-scale CNNs. In the experiments involving ResNet-18 on the CIFAR-100 dataset, all methods perform poorly, with FLAME achieving a maximum accuracy of only 9.9%. The authors think that this may reflect the inherent limitations of one-shot aggregation in deep networks. But the CIFAR-100 experiments are limited to ResNet-18, it is unclear whether this subpar performance stems from the difficulty of the dataset, the model architecture, or a combination of both. I recommend to include additional experiments(such as using a standard CNN on CIFAR-100 or a ResNet-18 on CIFAR-10)to more thoroughly verify the effectiveness of the proposed method across diverse model architectures.

Questions for authors
  1. Can you provide further analysis on the relationship between the Bayesian Evidence scores for different candidate neuron arrangements and the performance of the final merged model?

  2. In the CIFAR-100/ResNet-18 experiments, both FLAME and other methods performed poorly. Can you include additional experiments, specifically using a smaller CNN on the CIFAR-100 dataset and a ResNet-18 model on the CIFAR-10 dataset, to disentangle the effects of dataset difficulty from those of model architecture, and to further validate the effectiveness of FLAME across different architectures?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer ghFz (1/2)

Thank you for the constructive review and the two concrete questions. Both motivated additional experiments and analyses, which are now included in the revised paper.

Q1: Does higher evidence imply a better merge?

Overall, evidence is consistent with final accuracy: FLAME achieves higher evidence than Cycle in 203 of the 215 runs across the main table, the quantity-skew table, and Figure 2 (5–20 clients), and higher accuracy in 170. Table 2 now reports evidence alongside the accuracy of each alignment. We further investigated this relationship in three ways (new Section 5.3).

On alignments not optimized for evidence. To test whether evidence reflects merge quality beyond FLAME's own alignment, we considered four alignment strategies that do not optimize evidence (Π=I, Git Re-Basin, Cycle, and isotropic weight matching). For each strategy, we computed its evidence and evaluated the accuracy after applying the same Bayesian merge. We then computed the Spearman correlation between evidence and accuracy across the four strategies, separately for each run (10 clients, α = 0.1, 5 seeds per dataset). The table reports the mean correlation across the five runs for each dataset:

Alignment strategiesFashionMNISTSVHNCIFAR-10
All four alignments+0.84+0.96+0.92
Three non-trivial (excluding Π=I)+0.70+0.90+0.80

The consistently positive correlations indicate that alignment strategies with higher evidence tend to produce more accurate merged models. This relationship persists even when excluding the unaligned baseline, suggesting that evidence captures differences in merge quality also among non-trivial alignment strategies.

The full evidence against the misalignment term alone. We reran the term ablation with the corrected pipeline (10 clients, α = 0.1, random epochs, 5 seeds), using the same Bayesian merge in every row. Accuracy (%):

AlignmentFashionMNISTSVHNCIFAR-10
Π=I37.8 ± 3.938.5 ± 4.222.4 ± 1.8
Git Re-Basin50.2 ± 3.560.1 ± 4.331.4 ± 1.5
Cycle (official C2M3)58.1 ± 2.069.4 ± 1.935.7 ± 1.4
Misalignment61.4 ± 1.970.4 ± 2.236.9 ± 1.0
+ Volume/Occam61.1 ± 1.668.9 ± 2.637.3 ± 0.9
+ Prior-location (FLAME)60.9 ± 1.770.8 ± 2.737.1 ± 1.3

Your reading was right: the three evidence-based variants are within two percentage points of one another, with no statistically significant differences. We have therefore revised our claims about the relationship between the volume and prior-location terms and model accuracy. We retain the full evidence because it is the quantity derived by our formulation, is the score validated in Q1, and performs as well as any subset at no additional computational cost. With the full evidence, FLAME outperforms Cycle on all three datasets.

Q2: Dataset difficulty or architecture?

We ran both experiments you suggested (Appendix C.5). FLAME accuracy (%), α = 0.1:

ConfigurationCIFAR-10CIFAR-100
CNN, 10 clients37.19.2
ResNet-18, 5 clients43.112.4
  • ResNet-18 is not the bottleneck. On CIFAR-10, it is level with the CNN with 5 clients (43.1 vs. 41.3, within the standard error), and the CNN performs similarly with 10 clients and fixed epochs (40.6, Table 4). The different numbers of clients used on CIFAR-100 (10 and 5) are thus unlikely to explain the difference between the two architectures.
  • CIFAR-100 under this partition is the hard part: each client sees only a few of the 100 classes, and every one-shot method stays low.
  • The ranking of the methods is preserved. On ResNet-18 / CIFAR-10, FLAME reaches 43.1%, against 40.5% for Cycle, 32.5% for FedFisher and 26.6% without alignment. On CIFAR-100, FLAME improves on the unaligned merge by 6 to 7 points with both architectures, and on Cycle with the CNN (+0.57, 5/5 seeds); with ResNet-18 the two are level (12.4 vs. 12.3).

On the variation across datasets. We clarify this point in Section 5.3 and General Response 2/2. Curvature affects the alignment only for weights whose values are informed by the clients' data. This applies to 34–100% of the hidden-layer weights of LeNet on FashionMNIST, where curvature improves accuracy by 1.67 points. In contrast, it applies to fewer than 4% of the CNN weights on SVHN and CIFAR-10. In the latter settings, Prop. 4.2 predicts that the evidence reduces to weight matching, consistent with the observed effect of curvature on accuracy being below half a point.

We hope this addresses your two questions and would be happy to clarify any remaining points.

Thank you for your response. Your response addresses some of my concerns, but since there are no significant differences among the three evidence-based variants, I will maintain my current score (although you have revised your earlier claims, I think the remaining issue prevents me from raising my score). I will increase my confidence score later. I suggest submitting the rebuttal earlier next time to allow more time for discussion.

btw, where's your (2/2)?

Thank you for the quick reply, despite our late response. The new experiments and their replication took the whole rebuttal period, and we are sorry this left so little time for discussion.

We agree that the three evidence-based variants are not significantly different from each other. Still, evidence-based alignment outperforms Euclidean matching, and the evidence ranks alignments consistently with their accuracy (Section 5.3). We keep the full evidence because it is the quantity our derivation yields, rather than a subset of terms chosen after seeing the results.

About (2/2): this is a typo, and we apologize for the confusion. In the end we fit our whole reply into a single comment, so it is complete as it stands.

Thank you also for raising your confidence.

Reviewer_fqiSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies one-shot federated learning under uncoordinated client training and proposes FLAME. FLAME selects client permutations by maximizing the closed-form evidence of the merged Laplace-approximated posterior and then merges under the selected alignment. The paper provides theoretical connections to weight matching in the isotropic case.

Critical strengths
  1. The paper is well-organized and easy to follow.
  2. The theorectical analysis is good. The unified Bayesian formulation is conceptually clean, and the closed-form evidence objective provides a principled alternative to Euclidean weight matching for permutation selection.
  3. The ablation showing that the curvature-induced misalignment term is the key ingredient is useful.
Critical weaknesses
  1. The central evidence-based procedure is not consistent in the algorithm. The evidence objective is derived for full Gaussian posteriors, but the alignment step uses only a diagonal approximation of the posterior, while the final Bayesian merge uses Kronecker-factored Hessians. Please provide more analysis or explaination on the objective selection.
  2. the proposed FLAME only tested in three simple classification tasks. The model, dataset are too small.
Questions for authors
  1. More analysis on when and why selecting the Gaussian posteriors\Kronecker-factored Hessians and so on.
  2. The performance should be tested under complex tasks.
1 comment
AuthorsOpenReview ↗

Reply to Reviewer fqiS

Thank you for the careful review. We leveraged your main concern as an opportunity to further improve FLAME.

W1. Consistency of the objective.

In the original submission, alignment relied on a diagonal approximation, whereas merging used K-FAC. The revised FLAME consistently uses a single Gaussian representation for each client throughout the pipeline: its damped K-FAC Laplace posterior (Section 4.2). Specifically, the misalignment term of the evidence uses the full K-FAC precisions, model selection (keeping the best rounded permutation) uses the evidence of the corresponding model, and merging returns the exact mode of the product of the same posteriors. The only remaining approximation concerns the volume term: since the log-determinant of a sum of Kronecker products has no closed form, we compute it from the marginal variances of the same K-FAC posteriors.

Evaluation of the impact of approximations. Following your suggestion, we added a 2 × 2 comparison in Appendix D.4, crossing diagonal and K-FAC posteriors for alignment with diagonal and K-FAC posteriors for merging. We use the same clients (10 clients, independent initializations, 3 datasets, α ∈ {0.1, 0.025}, 3 seeds; 18 runs). Mean accuracy (%):

AlignmentDiag. mergeK-FAC merge
Diag. evidence24.543.3
K-FAC evidence24.545.4

The K-FAC merge is decisive, and with it the K-FAC evidence adds 2 points over the diagonal one. The consistent configuration is the best of the four. Our exact merge also improves on the FedFisher merge under every alignment (see also General Response 2/2 and Table 14 of the appendix).

Why Gaussian posteriors and Kronecker factors. We added this discussion to Section 4.2. Gaussian posteriors are the family for which the product of client posteriors divided by the prior (Eq. 1), and therefore the evidence of the merged model (Thm. A.2), have a closed form. The Laplace approximation provides these posteriors from already-trained models, without retraining (Daxberger et al., 2021). Regarding the curvature approximation, the full Hessian requires O(d²) memory (about 10¹⁴ entries for ResNet-18), whereas the diagonal approximation discards within-layer correlations that matter for merging (Jhunjhunwala et al., 2024). K-FAC retains the input and output correlations of each layer at O(Σℓ d_in² + d_out²) cost, and its eigendecomposition makes the product and prior division exact within each layer. Finally, factored damping (Ritter et al., 2018) keeps every factor invertible, as in the implementation provided by the Laplace library.

W2. Harder tasks.

We added (tables in the reply to Reviewer YgAF and in Appendix C.5–C.6):

  • ResNet-18 on CIFAR-10: FLAME 43.1%, against 40.5% for Cycle, 32.5% for FedFisher and 26.6% without alignment.
  • ResNet-50 on CIFAR-10, as a first test of scale: FLAME improves on the unaligned merge by 5.4 points, and on Cycle by 1.1.
  • CIFAR-100, with the CNN and with ResNet-18: FLAME improves on the unaligned merge by 6 to 7 points.

Transformer-scale models are discussed as future work, with the implementations they require.

Reviewer_YgAFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies one-shot federated learning (FL), a paradigm in which a global model is learned from data distributed across multiple clients in a single communication round. The paper proposes FLAME (Federated Learning with Alignment by Model Evidence). FLAME approximates each client's posterior with a Laplace approximation and chooses client permutations by maximizing the Bayesian evidence of the merged model. The same posterior/curvature information is then used for Bayesian merging. Experiments on FashionMNIST, SVHN, and CIFAR-10 datasets with LeNet/CNN models, and different setting of Dirichlet heterogeneity, number of clients, initialization, and local training epochs. The results show that FLAME generally outperforms other baselines significantly.

Critical strengths

S1. Strong theoretical support.

S2. Experiments consider several dimensions including: level of non-IID data, number of clients, and initialization, and local training epochs. The authors also provide useful ablation study on the evidence objective.

Critical weaknesses

W1. The model used in the paper is still small. It will be more benefit if the authors could discuss or provide result on larger models or with different model family. Can FLAME practically scale to ResNet-50, ViT, or transformer-scale models?

W2. It seems that the cost of FLAME are not trivial since FLAME requires calculation for Laplace approximations. Full Hessians. Considering to FedAvg which is much more simpler with only weight aggregation. One of my concern is that how does FLAME compared against common initialization + simple one-shot (or multi-short) aggregation, considering both accuracy and total communication/computation? The authors should clearly discuss this points in the paper.

Questions for authors

See above

3 comments
AuthorsOpenReview ↗

Reply to Reviewer YgAF (1/2)

Thank you for the positive assessment and the practical questions.

W1: Larger models. We added ResNet-18 (11.2M parameters) on CIFAR-10 and CIFAR-100 (5 clients), and the CNN on CIFAR-100 (10 clients); independent initializations, α = 0.1, 5 seeds (Appendix C.5). As a first test of scale, we also added ResNet-50 (23.5M parameters, bottleneck blocks without normalization layers) on CIFAR-10, with 5 clients and 5 seeds, as for ResNet-18. FLAME, Cycle and Π = I use the same merge; FedFisher uses its own.

Model / DatasetClientsFLAMECycleΠ = IFedFisher
ResNet-18 / CIFAR-10543.1 ± 4.340.5 ± 4.026.6 ± 3.332.5 ± 2.7
CNN / CIFAR-100109.2 ± 0.38.6 ± 0.42.8 ± 0.24.2 ± 0.6
ResNet-18 / CIFAR-100512.4 ± 0.512.3 ± 0.65.5 ± 0.22.4 ± 0.3
ResNet-50 / CIFAR-10524.0 ± 2.822.9 ± 2.518.6 ± 2.321.9 ± 3.5
  • On ResNet-18 / CIFAR-10, FLAME improves on the unaligned merge by +16.5, on Cycle by +2.55 and on FedFisher by +10.6.
  • On ResNet-50, FLAME improves on the unaligned merge by +5.4 points and on Cycle by +1.1.
  • The pipeline scales to this size unchanged: the K-FAC factors grow with the squared layer widths, and the search is linear in the number of clients, since every client is aligned to the merged posterior, in contrast to quadratic scaling of C2M3.
  • For ViTs and other transformers, suitable approaches to parameter matching and scalable Laplace approximations are available (Verma & Elbayad, 2024; Yang et al., 2024), providing relevant building blocks for extending FLAME to these architectures.

W2: Cost. FLAME builds upon each client's posterior approximated as a K-FAC Laplace approximation (Appendix B.1). In the following we discuss the main costs of our approach:

  • Communication. Each client sends its weights and its K-FAC factors, the same message as FedFisher-KFAC: 4.0× the weights for LeNet, 5.0× for the CNN, 9.4× for ResNet-18 and 7.5× for ResNet-50 (about half if the symmetric factors are sent as triangles). We note that the computation of Laplace posterior is still needed by any method that performs a Bayesian merge, so this cost is not specific to FLAME.
  • Client computation. Local training plus one pass over the local data to accumulate the K-FAC factors.
  • Server computation, median per run on a shared GPU (upper bounds):
ModelFLAME weight-matching startFLAME evidence searchCycle
LeNet / FashionMNIST11 s7 min2.5 min
CNN / SVHN, CIFAR-101 min4 min10 min
ResNet-18 / CIFAR-1002.5 min9.5 min9.5 min
ResNet-50 / CIFAR-104.7 min63 min21 min

FLAME's server step costs about as much as running C2M3: faster on the CNNs, similar on ResNet-18, slower on LeNet and on ResNet-50, where our search has not yet converged within the step budget used for the smaller models. We would also like to note that the server-side computational costs of FLAME and C2M3 scale differently with the number of clients: linearly and quadratically, respectively, as detailed in the Appendix.

AuthorsOpenReview ↗

Reply to Reviewer YgAF (2/2)

W2: Common initialization with one-shot or multi-round FedAvg. We added this comparison (10 clients, α = 0.1, fixed local epochs, 5 seeds; Appendices C.3–C.4). The FedAvg rows use the same total local training as FLAME: multi-round FedAvg splits the local epochs over the rounds. Upload is the total transmitted per client, relative to one copy of the weights.

MethodRoundsUploadFMNISTSVHNCIFAR-10
Shared init + FedAvgone shot1×37.645.930.4
Shared init + FedAvg2 rounds2×55.467.641.4
Shared init + FedAvg5 rounds5×77.983.855.3
Shared init + FedAvg10 rounds10×78.885.859.0
Shared init + FLAMEone shot4–5×64.472.342.5
Independent init + FLAMEone shot4–5×57.571.440.6
Independent init + FLAME alignment, then 10 FedAvg rounds10 rounds4–5× + 10×80.486.460.0

One shot. Even without a common initialization, FLAME is 10 to 25 points above one-shot FedAvg with one. With a common initialization, every matcher returns the identity, and FLAME reduces to its Bayesian merge with the prior centred at that initialization. Over the four values of α, this merge is the most accurate one-shot method in 8 of 12 settings, and it improves on FedFisher, RegMean and FedBEns by 3 to 5 points (Appendix C.3).

Multi-round. One-shot FLAME is on par with two FedAvg rounds. At the same upload as one FLAME message (about 5 rounds), multi-round FedAvg is clearly more accurate. When clients share an initialization and stay available for several synchronized rounds, multi-round training is the right tool. FLAME targets the case where this is not possible: a single round, with clients trained independently before the federation (pre-existing models, different checkpoints).

Combining both. If independently trained clients can join FedAvg rounds afterwards, FLAME's alignment provides the starting point: the server aligns the clients once and averages them, then runs FedAvg. After 10 rounds this is above FedAvg from a shared initialization on all three datasets. Aligning first also improves FedAvg at every round and in every seed (Appendix C.4, Table 8). This start includes the clients' initial local training.

Computation. On the client, FLAME adds one pass over the local data to accumulate the K-FAC factors. The server cost is given in our reply on the cost of FLAME above. We state this scope explicitly in the main text and in the limitations.

Reviewer_YgAFOpenReview ↗

Office comment

Thank you for your clarification. The reply helps a lot. Since I have already give a positive score, I will keep it as it is.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (2/2): Summary of new results

Expanded experimental evaluation. In accordance with the reviewers' feedback, we substantially expanded the experimental suite compared to our initial submission. Given that the refinements in both the baselines and our proposed approach must be read in light of this broader evaluation protocol, we invite the reviewers to assess the results across the full body of experiments now reported, rather than weighing them against the preliminary version.

Overall results. We summarize the main results below, averaging across models and experimental scenarios. For each setting, experiments are performed over 5 seeds, and every comparison is paired on the same clients. The following main table now reports results from 120 runs: 3 datasets × 4 values of α × 2 local-epoch settings (random and fixed) × 5 seeds. Mean accuracy (%) is reported over the resulting 24 settings:

FLAMECycle (official C2M3 + our merge)Π = I (our merge, no alignment)C2M3RegMeanOTFusionFedAvg
51.048.328.027.924.423.316.5

The main take-home is that our joint modelling of the alignment and merging is necessary. In particular, under the same Bayesian merge, FLAME outperforms the unaligned merge by +23.0 points. Under the same alignment, the Bayesian merge outperforms plain averaging by +20.9 points. Alignment alone (C2M3, OTFusion) and curvature alone (Π = I) both stay at or below 28%.

A closer comparison with the corrected Cycle baseline.

SettingFLAME − Cyclep-value (paired t-tests)
Main setting (Table 1 and 4): All+2.65< 0.001
FashionMNIST+2.200.07
SVHN+3.33< 0.001
CIFAR-10+2.41< 0.001
ResNet-18 on CIFAR-10 (new setting)+2.550.02
CNN on CIFAR-100 (new setting)+0.570.009

FLAME achieves higher average accuracy than Cycle across all considered datasets, client configurations, and additional architectures.

A comparison on alignment and merging strategies. We trained 120 new sets of clients and crossed three alignments with three merge rules on them (Appendix D.4). Mean accuracy (%):

Alignment \ MergeFedAvg mergeFedFisher mergeBayesian merge (ours)
None16.620.828.3
Cycle27.744.447.8
Evidence match (ours)29.546.850.5 (FLAME)

Overall FLAME provides:

  • Better alignments, independently of the merging technique: FLAME's alignments outperform alternatives under different merge rules, and the improvements consistently carry over to merging procedures other than our Bayesian merge.
  • Better merging across alignments: using FLAME's alignment, our Bayesian merge improves accuracy by +3.64 points over the FedFisher merge.

When does Bayesian evidence improve over weight matching?

We observe that the improvement follows the combination of the merging and matching. Bayesian evidence provides gains in the regimes expected from our theoretical analysis: the evidence maximization departs from weight matching when the clients' data have a dominant influence on the merged weights, namely, when the data precisions are not prior-dominated. In detail:

  • LeNet / FashionMNIST: this holds for 34–100% of the hidden-layer weights in at least two clients, and the curvature adds +1.67 points.
  • CNN / SVHN and CIFAR-10: it holds for fewer than 4% of the hidden-layer weights. Prop. 4.2 then predicts that the evidence reduces to weight matching, and the curvature changes the accuracy by less than half a point there (+0.45 averaged over all 120 runs of the main table).

We note that the isotropic matching also improves over Cycle, highlighting the usefulness of optimizing over the merged models rather than with consistency between clients. FLAME thus uses the curvature where it is informative and falls back on weight matching, without loss, where it is not.

AuthorsOpenReview ↗

General response (1/2): a correction, and a revised FLAME

We thank the reviewers for their constructive feedback. We are happy that they found the unified Bayesian formulation conceptually clean (fqiS), the theory comprehensive (ghFz, YgAF), and the link to weight matching useful.

Mainly, reviewer concerns are focused on:

  • A consistent use of the posterior approximation (fqiS);
  • Larger models and harder datasets (all);
  • The cost of the method, and a comparison with coordinated training (YgAF);
  • Direct evidence that a higher Bayesian evidence gives a better merge (ghFz).

The revised PDF addresses all four; changes are marked in red. Before the individual replies, we report a correction that affects the experiments, and the improvements it led to.

1. Correction: our Cycle baseline

While preparing the additional experiments, we identified an error in the implementation of our proposed baseline, Cycle, which uses the state-of-the-art model alignment method C2M3 (Crisostomi et al., 2024), followed by our Bayesian merge. The original implementation mixed two permutation conventions and used a faulty step size, so it returned alignments close to the identity. This is why Cycle tracked Π = I in the submitted Table 1. Cycle has been fixed properly, resulting in a much stronger baseline; statements in the submission that relied on the faulty one have all been revised.

We apologize for the error. Experiments reported in every table and figure in the revision were rerun from scratch with the corrected code.

2. An improved, consistent FLAME (Reviewer fqiS)

The revision also led us to strengthen FLAME, in line with Reviewer fqiS's suggestions. Specifically, alignment, model selection, and merging now rely on a more consistent single-Gaussian representation of the client posteriors. In particular:

  • Each client's posterior is its K-FAC Laplace approximation, with the standard factored damping of Ritter et al. (2018). The misalignment term of the evidence now uses the full K-FAC precisions, instead of just their diagonal, and the merge returns the exact mode of the product of the same posteriors (Eq. 1).

  • We improved the permutation search by starting from the weight-matching solution of the isotropic case (Prop. 4.2), and then maximizing the K-FAC evidence over relaxed permutations. Every 50 steps, the permutations are rounded and scored with the full evidence, and the best one is kept, so the search never decreases the evidence.

  • The only remaining further approximation in the evidence computation is the volume term: the log-determinant of a sum of Kronecker products has no closed form, so we evaluate it from the marginal variances of the same posteriors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.