- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Spectral Graph Merging (SGM), a model-merging method that operates in a graph-spectral domain. For each layer, the delta matrix is lifted to a symmetric bipartite block matrix X_i. Then a shared graph is constructed from the adjacency matrix of the row-normalized mean task vector. Each X_i is decomposed into J frequency bands through a Chebyshev-approximated spectral graph wavelet filter bank. Afterward, the lowest band (j = 0) is aggregated by dense summation, instead, the higher bands (j > 0) are fused with a TIES-style sign election, followed by quantile-based soft-thresholding with a band-dependent sparsity level ρ_j. The fused bands are mapped back to weight space and rescaled to produce the final merged model, that is evaluated on the MergeBench and OptMerge benchmarks.
Critical strengths
- Fresh perspective and clear presentation. The manuscript is well written and easy to follow. Treating a layer’s task update as a signal on the neuron-interaction bipartite graph, and performing multi-resolution analysis in that domain, is an interesting idea for model merging. It also links the merging literature to graph signal processing (GSP) and weight-space learning.
- Informative ablations. The paper reports a broad set of ablations, covering graph support, band selection, fusion rule and energy rescaling. These help clarify the contribution of each component and support the narrative.
- Under-explored direction. Mitigating task interference with tools from GSP appears to be largely unexplored, and this work offers a careful study of it.
Critical weaknesses
- Relation to SVD-based methods. My main concern is that spectral graph wavelet decompositions are closely related to SVD-based decompositions, especially on bipartite graphs. This places SGM close to shared-subspace SVD methods such as Iso-C/Iso-CTS, TSV and KnOTS. From the current analysis and evaluation, I cannot determine whether SGM is a genuinely different approach or a reformulation of existing subspace methods in different notation. A more explicit theoretical and empirical comparison with these methods would substantially strengthen the paper.
- Inconsistent baseline coverage across benchmarks. The two benchmarks are evaluated against different sets of methods. Table 2 (OptMerge) is considerably more exhaustive, while Table 1 (MergeBench) omits several relevant baselines, including the SVD-based methods most closely related to SGM. The reason for this asymmetry is not explained. Evaluating a consistent and broader set of baselines on both benchmarks would make the empirical claims more convincing.
- Possibly misconfigured baseline. In Table 2, Iso-C performs dramatically worse than all other methods. This suggests a possible implementation issue or insufficient hyperparameter tuning. Please double-check this baseline.
- Readability of Figure 2. Figure 2 appears almost flat, and the relevant differences are only visible in the small inset. A logarithmic scale for the normalized energy, or a different normalization, would make the separation between models visible at a glance.
- Limited evaluation settings. Model-merging papers commonly evaluate on vision benchmarks (e.g., ViT models across 8/14/20 tasks) and NLP benchmarks (e.g., GLUE), in addition to LLM benchmarks. The present evaluation covers only the large (vision-)language model setting. Results in the standard vision and GLUE settings would complement the analysis and ease comparison with prior work.
Questions for authors
- Configuration consistency.
- Which configuration (fixed or adaptive J) produced the results in Tables 1 and 2?
- Why does the baseline in Figure 8 appear to differ from the full-SGM result in Table 4?
- Which model and setting produced the results in Table 6?
- Hyperparameter selection.
- What held-out data was used to select the hyperparameters (e.g., α and ζ)?
- Did the baselines receive an equivalent tuning budget?
- How were TSV, Iso-C, CART and KnOTS tuned on OptMerge?
- Is the sharp performance drop of Iso-C due to an implementation error or a configuration issue?
- Citations and typos. I noticed several incorrect citation attributions (e.g., DARE and RegMean are attributed to the wrong works), as well as a number of typos and editing leftovers. Please carefully proofread the manuscript and verify all references for the camera-ready version.