- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the model merging problem in uncoordinated one-shot federated learning and proposes a method called FLAME. It first employs Laplace Approximation to approximate the local posterior distribution of each client, then identifies the optimal neuron permutation by maximizing the approximate Bayesian Evidence of the merged model, and finally performs Bayesian Merging on the aligned posteriors. The paper also provides theoretical derivations and empirical evaluations to demonstrate the effectiveness of the proposed method.
Critical strengths
1、The work demonstrates a degree of methodological innovation.
This paper proposes a model alignment framework based on Bayesian Evidence, which incorporates posterior parameter uncertainty into the process of matching and permuting neurons—moving beyond a reliance solely on traditional Euclidean weight similarity—thereby offering a fresh perspective on model alignment and merging.
2、The theoretical derivation is comprehensive.
The paper provides a detailed theoretical derivation of the proposed Bayesian Evidence objective. It demonstrates that under the assumption of an isotropic posterior, this objective reduces to the conventional "weight matching" approach, thereby establishing a theoretical link between the new method and existing techniques.
Critical weaknesses
- There is a lack of direct validation for the core Bayesian evidence alignment criterion.
While this paper selects neuron permutations by maximizing an approximate Bayesian evidence, the method is evaluated primarily based on final classification accuracy, so there is no direct analysis of the relationship between the evidence score and the performance of the merged model. Consequently, whether higher evidence consistently correlates with better model merging outcomes is unclear. And the marginal benefit of using the full Bayesian evidence, as opposed to relying solely on the curvature-weighted misalignment term, varies significantly across datasets, and on CIFAR-10, it almost no improvement. Therefore, I have skeptical regarding the necessity and stability of the full Bayesian evidence objective
- The experimental validation across different model architectures is insufficient.
The primary experiments focus on LeNet and smaller-scale CNNs. In the experiments involving ResNet-18 on the CIFAR-100 dataset, all methods perform poorly, with FLAME achieving a maximum accuracy of only 9.9%. The authors think that this may reflect the inherent limitations of one-shot aggregation in deep networks. But the CIFAR-100 experiments are limited to ResNet-18, it is unclear whether this subpar performance stems from the difficulty of the dataset, the model architecture, or a combination of both. I recommend to include additional experiments(such as using a standard CNN on CIFAR-100 or a ResNet-18 on CIFAR-10)to more thoroughly verify the effectiveness of the proposed method across diverse model architectures.
Questions for authors
-
Can you provide further analysis on the relationship between the Bayesian Evidence scores for different candidate neuron arrangements and the performance of the final merged model?
-
In the CIFAR-100/ResNet-18 experiments, both FLAME and other methods performed poorly. Can you include additional experiments, specifically using a smaller CNN on the CIFAR-100 dataset and a ResNet-18 model on the CIFAR-10 dataset, to disentangle the effects of dataset difficulty from those of model architecture, and to further validate the effectiveness of FLAME across different architectures?
4 comments
Thank you for your response. Your response addresses some of my concerns, but since there are no significant differences among the three evidence-based variants, I will maintain my current score (although you have revised your earlier claims, I think the remaining issue prevents me from raising my score). I will increase my confidence score later. I suggest submitting the rebuttal earlier next time to allow more time for discussion.
btw, where's your (2/2)?
Thank you for the quick reply, despite our late response. The new experiments and their replication took the whole rebuttal period, and we are sorry this left so little time for discussion.
We agree that the three evidence-based variants are not significantly different from each other. Still, evidence-based alignment outperforms Euclidean matching, and the evidence ranks alignments consistently with their accuracy (Section 5.3). We keep the full evidence because it is the quantity our derivation yields, rather than a subset of terms chosen after seeing the results.
About (2/2): this is a typo, and we apologize for the confusion. In the end we fit our whole reply into a single comment, so it is complete as it stands.
Thank you also for raising your confidence.
Reply to Reviewer ghFz (1/2)
Thank you for the constructive review and the two concrete questions. Both motivated additional experiments and analyses, which are now included in the revised paper.
Q1: Does higher evidence imply a better merge?
Overall, evidence is consistent with final accuracy: FLAME achieves higher evidence than Cycle in 203 of the 215 runs across the main table, the quantity-skew table, and Figure 2 (5–20 clients), and higher accuracy in 170. Table 2 now reports evidence alongside the accuracy of each alignment. We further investigated this relationship in three ways (new Section 5.3).
On alignments not optimized for evidence. To test whether evidence reflects merge quality beyond FLAME's own alignment, we considered four alignment strategies that do not optimize evidence (Π=I, Git Re-Basin, Cycle, and isotropic weight matching). For each strategy, we computed its evidence and evaluated the accuracy after applying the same Bayesian merge. We then computed the Spearman correlation between evidence and accuracy across the four strategies, separately for each run (10 clients, α = 0.1, 5 seeds per dataset). The table reports the mean correlation across the five runs for each dataset:
The consistently positive correlations indicate that alignment strategies with higher evidence tend to produce more accurate merged models. This relationship persists even when excluding the unaligned baseline, suggesting that evidence captures differences in merge quality also among non-trivial alignment strategies.
The full evidence against the misalignment term alone. We reran the term ablation with the corrected pipeline (10 clients, α = 0.1, random epochs, 5 seeds), using the same Bayesian merge in every row. Accuracy (%):
Your reading was right: the three evidence-based variants are within two percentage points of one another, with no statistically significant differences. We have therefore revised our claims about the relationship between the volume and prior-location terms and model accuracy. We retain the full evidence because it is the quantity derived by our formulation, is the score validated in Q1, and performs as well as any subset at no additional computational cost. With the full evidence, FLAME outperforms Cycle on all three datasets.
Q2: Dataset difficulty or architecture?
We ran both experiments you suggested (Appendix C.5). FLAME accuracy (%), α = 0.1:
On the variation across datasets. We clarify this point in Section 5.3 and General Response 2/2. Curvature affects the alignment only for weights whose values are informed by the clients' data. This applies to 34–100% of the hidden-layer weights of LeNet on FashionMNIST, where curvature improves accuracy by 1.67 points. In contrast, it applies to fewer than 4% of the CNN weights on SVHN and CIFAR-10. In the latter settings, Prop. 4.2 predicts that the evidence reduces to weight matching, consistent with the observed effect of curvature on accuracy being below half a point.
We hope this addresses your two questions and would be happy to clarify any remaining points.