- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates expert-pair selection for model merging, focusing on preserving joint task performance when combining experts generated from a shared checkpoint. It proposes a fixed-budget spectral selection framework that integrates cross-task screening, layerwise pair geometry, separate validation, and base-model fallback. Experiments across six checkpoints show improved strict joint success over Top-1 averaging, with spectral selection outperforming quality-only and cosine-based baselines on several key metrics, while highlighting the conditional nature of expert composability.
Critical strengths
1.The paper identifies a practical limitation of model merging: individually beneficial task experts may not preserve both tasks' performance when composed. Its focus on strict joint improvement over the base model is meaningful and avoids the limitations of average-score-based evaluation.
2.The proposed method combines cross-task quality screening, layerwise singular-subspace overlap, bounded validation, and base-model fallback. The matched quality-only and global-cosine baselines help isolate the contribution of spectral pair ranking.
Critical weaknesses
-
Limited methodological novelty. The core components, task-quality screening, singular-subspace similarity, coefficient calibration, and validation-based selection, are related to existing model-merging approaches. The paper should more clearly distinguish its contribution from the integration of existing model-merging and parameter-geometry approaches.
-
Insufficient theoretical support for spectral ranking. The theoretical analysis explains why proximity and orthogonality alone do not guarantee composability, but does not directly establish why layerwise subspace overlap predicts better joint task retention.
-
Under the matched protocol, spectral ranking improves strict joint success by only 2.6 percentage points over quality-only ranking and 0.07 percentage points in mean worst-task gain over global-cosine ranking. Without statistical significance tests or confidence intervals for these comparisons, it is difficult to determine whether the incremental gains are robust.
-
Inconsistent superiority across models. Although the proposed method outperforms Top-1 averaging on all six checkpoints, it does not achieve the best strict joint success on Qwen2.5-7B or OLMo 3-7B, where Direct Search and TIES perform better, respectively. This limits the evidence for its general superiority over existing merging methods.
-
Incomplete evaluation of practical efficiency. Although the paper includes sensitivity analyses for several design choices, it does not provide a sufficiently comprehensive, matched assessment of end-to-end runtime and computational overhead, including SVD and candidate screening.
-
The proposed method still exhibits a 17.1% regression rate and a 13.0% abstention rate in the primary aggregate. Moreover, the evaluation focuses on two-task composition, so the results do not establish that the method will scale effectively to merging multiple experts simultaneously.
Questions for authors
See weaknesses
4 comments
Response to Reviewer PrWg — Part 2/4
The review's Questions section states “See weaknesses”. Q1–Q3 across Parts 2–4 organize additional responses to W4–W6.
Q1. Matched baseline comparison (W4)
The new matched comparison uses the same 525 configurations, 200 validation items/task, at most 50 proposals/task pair, and strictly positive validation gains on both tasks. All methods return Base if none passes; testing uses the original held-out data. TIES selects retention from 5%, 10%, 20%, 50%, and 100% by validation; Direct Search selects one expert maximizing worst-task validation gain under the same gate and fallback rule.
These implementation details apply to the additional matched 525-configuration comparison in Appendix E.3, not the original system-level comparison in Table 2.
TIES implementation. Deltas are delta_i = theta_i - theta_0 relative to the common Base. All floating-point model parameters are included, with shared parameters counted once and buffers excluded. For each expert, trimming retains the largest absolute entries globally across its complete delta vector, at retention r = 0.05, 0.10, 0.20, 0.50, 1.00; both experts use the same r. At each coordinate, the sign of the sum of retained updates is elected, and the nonzero retained updates agreeing with that sign are averaged. A zero sum produces a zero merged update. The merged delta is added to Base with scale 1, without additional normalization or rescaling.
TIES ranks screened feasible pairs by the paper's quality-only score Q, without spectral information, and retains up to ten distinct pairs. Its budget is at most 10 pairs × 5 retention rates × 1 scale = 50 proposals. The original alpha/beta coefficient grid is not additionally applied to TIES.
Direct Search implementation. All 5,000 realized candidates are ranked on selection data by descending q = min(g_A, g_B), then descending m = (g_A + g_B)/2, then ascending candidate ID; the first 50 enter validation. Here each selection gain is the candidate's selection score minus Base's selection score; for accuracy, gains use pp. Direct Search returns one expert, without combining candidates.
For both baselines, each proposal uses the original 200 validation examples per task. Among proposals with strictly positive gains on both tasks, selection maximizes worst-task validation gain, then mean gain, then fixed proposal order; if none passes, return Base. The held-out test is used only for the final selected output, with no test-based selection or post-test fallback.
| Method | SJ, % (count/525) | WG (pp) | Regression, % (count/525) | Abstention, % (count/525) |
|---|---|---|---|---|
| TIES | 48.0 (252/525) | +0.42 | 22.5 (118/525) | 13.5 (71/525) |
| Direct Search | 50.3 (264/525) | +0.49 | 20.8 (109/525) | 13.1 (69/525) |
| Ours | 53.1 (279/525) | +0.61 | 17.1 (90/525) | 13.0 (68/525) |
These new matched results favor Ours in aggregate SJ, WG, and regression, with shared validation and fallback opportunities. They neither replace the original model-specific exceptions nor establish statistical significance against TIES or Direct Search. The Quality-only and Global Cosine controls isolate the spectral ranking component.
Efficiency continues in Part 3, Q2; regression and three-expert results in Part 4, Q3.
Response to Reviewer PrWg — Part 3/4
Continuation of Part 2: Q2 responds to W5.
Q2. Computational overhead (W5)
Timing uses Qwen2.5-3B on one dedicated node with eight NVIDIA A100 80GB GPUs, one model replica per GPU, BF16 inference, and batch size 16 per GPU. Prompts and generation-length limits are identical across methods; subspace computation uses FP32.
For each method, five pools are each timed five times: 25 complete runs. One run covers one full candidate pool, seven tasks, and all 21 task pairs. The table reports mean wall-clock hours per pool on the eight-GPU node, not summed GPU-hours.
Each candidate is screened once per task. For retained experts, each subspace is computed once and reused across task pairs within that pool. Model outputs are not reused between methods or independent timing repetitions.
| Stage | Quality-only | Global Cosine | Spectral |
|---|---|---|---|
| Pool generation (h) | 6.0 | 6.0 | 6.0 |
| Screening and shared preprocessing (h) | 160.0 | 160.0 | 160.0 |
| Ranking, including applicable SVD (h) | 0.2 | 7.4 | 32.6 |
| Reconstruction and validation (h) | 19.8 | 19.8 | 19.8 |
| Post-generation total (h) | 180.0 | 187.2 | 212.4 |
| Post-generation time ratio | 1.000× | 1.040× | 1.180× |
| Generation-to-selection total (h) | 186.0 | 193.2 | 218.4 |
Spectral adds 18% wall-clock time over Quality-only after generation; including generation, the increase is approximately 17.42%. The original 35,000 screening and at most 2,100 calibration task-evaluation counts exclude SVD and reconstruction; these timings include both. The measurements cover Qwen2.5-3B only and do not establish the same overhead across models.
Validation-budget sensitivity is evaluated with spectral weight 0.5, rank cap 8, and K=50 fixed:
| Pair shortlist B | SJ (%) | WG (pp) | Regression (%) |
|---|---|---|---|
| 5 | 51.6 | +0.56 | 18.5 |
| 10 | 53.1 | +0.61 | 17.1 |
| 20 | 53.5 | +0.62 | 17.0 |
Increasing B from 10 to 20 adds 0.4 pp SJ and 0.01 pp WG. These results describe the outcome–budget trade-off; runtime is assessed separately above.
W1–W4 are addressed in Part 1; Q1 is in Part 2; Q3 continues in Part 4.
Response to Reviewer PrWg — Part 4/4
Continuation of Part 3: Q3 responds to W6.
Q3. Regression, abstention, and three-expert composition (W6)
The 17.1% held-out regression rate shows that the validation gate does not guarantee improvement. Abstentions return Base, contribute zero gain, and remain in the denominator. The primary 525 outcomes comprise 279 strict successes, 88 weak-only improvements, 90 regressions, and 68 abstentions (13.0%).
We additionally evaluate three-expert composition on Qwen2.5-1.5B and 3B using all four three-task combinations of GSM8K, MATH-500, MBPP, and ROCStories and the five original pools: 2 models × 4 task triplets × 5 pools = 40 configurations. This is separate from the primary 525-configuration study.
Each task supplies up to 50 screened candidates. Triplets take one expert per task, with distinct IDs and fixed 1/3 coefficients. Using the original 200 validation items/task and distinct-triplet budgets below, all three gains must be strictly positive. Selection maximizes worst-task validation gain, then mean gain, then fixed ID order; otherwise return Base. All 40 configurations, including fallbacks, enter the rates and mean WG.
Three-expert selection rule. Each task's candidate set retains up to 50 experts with nonnegative selection gain, ordered by that task's gain descending and then fixed candidate ID order. Form feasible triplets by taking one candidate from each task's set, requiring three distinct IDs. Sorting the IDs gives a canonical tuple: deduplicate these tuples to obtain the feasible set , so alternate task assignments of the same experts are counted once.
For , the scores are
,
,
.
The pairwise overlaps retain the two-expert layerwise definition, rank cap 8, and attention/MLP weights. Quality-based and spectral-guided selection rank by and , respectively, descending, with score ties resolved by canonical ID tuple. They validate proposals. Random selection samples the same number uniformly without replacement from the same deduplicated set. An empty feasible set returns Base.
Per-task Top-1 chooses each task's highest-ranked expert independently. If these IDs repeat, it returns Base as a structural abstention; lower-ranked experts are not substituted. With distinct IDs, its single proposal uses the same fixed equal weights and validation gate. These structural fallbacks also remain in the 40-configuration denominator.
| Method | Maximum validated triplets | Success count | Three-task SJ (%) | WG (pp) |
|---|---|---|---|---|
| Top-1 equal-weight merge | 1 | 9/40 | 22.5 | -0.18 |
| Random selection | 10 | 11/40 | 27.5 | +0.08 |
| Quality-based selection | 10 | 13/40 | 32.5 | +0.23 |
| Spectral-guided selection | 10 | 15/40 | 37.5 | +0.31 |
The aggregate difference over quality-based selection is two additional successes out of 40 (+5.0 pp) and +0.08 pp WG. It is not uniform across models:
| Model | Method | Success count | Three-task SJ (%) | WG (pp) |
|---|---|---|---|---|
| Qwen2.5-1.5B | Quality-based | 5/20 | 25.0 | +0.09 |
| Qwen2.5-1.5B | Spectral-guided | 5/20 | 25.0 | +0.08 |
| Qwen2.5-3B | Quality-based | 8/20 | 40.0 | +0.37 |
| Qwen2.5-3B | Spectral-guided | 10/20 | 50.0 | +0.54 |
On 1.5B, joint success is tied and spectral-guided WG is 0.01 pp lower. The improvement occurs on 3B.
This exploratory three-expert result is driven by 3B and establishes neither statistical significance nor general scalability. Pairwise compatibility does not guarantee joint retention; the joint merge is evaluated directly.
Response to Reviewer PrWg — Part 1/4
Thank you. SJ denotes strict joint success; WG denotes mean worst-task gain; pp denotes percentage points.
All rebuttal analyses retain the original selection, validation, and held-out test partitions used in the submission; no new data split was introduced.
W1. Contribution relative to existing merging and geometry methods
Screening, singular-subspace similarity, coefficient calibration, and validation have precedents. Our contribution is the formulation and controlled evaluation of joint base-relative retention for separately selected local perturbation experts, under bounded validation with unconditional abstention accounting. RandOpt uses sample-select-ensemble; here the decision concerns weight-space composition. The joint-retention objective, failure analysis, and bounded selection procedure define this problem, while matched controls isolate the contribution of pair ranking. We do not claim to introduce singular-subspace similarity itself.
W2. What the theory establishes and what supports spectral ranking
The analysis establishes that proximity and orthogonality do not ensure joint improvement. Its exact decomposition separates individual effects from interactions, motivating cross-task screening and validation. It does not prove that lower overlap yields better retention; overlap is an empirical prioritization signal.
The shuffled-overlap control permutes overlap assignments within each model-task-pair-pool feasible set, holding quality scores, screened candidates, K=50, B=10, five coefficient proposals, and the validation gate fixed. Random tie-breaking changes only exact quality ties, preserving distinct-quality ordering. Each randomized control uses 20 repetitions of the same 525 configurations; entries report mean ± standard deviation, with SJ dispersion in pp. Spectral ranking is the original 525-configuration reference.
Relative to Quality-only, spectral ranking replaces an average of 2.4 of ten shortlisted pairs (24%) and retains 7.6 under the same validation budget.
These controls favor retaining pair-specific geometry, without establishing a causal mechanism or universal overlap-success relationship. As noted in Related Work, Demystifying Mergeability reports a positive association with leading right-subspace alignment for fine-tuned CLIP pairs. Our empirical claim concerns the screened local-perturbation setting.
W3. Magnitude and uncertainty of the incremental gains
Table 2's +2.6 pp annotation subtracts the displayed SJ values for Ours and Global Cosine, 53.1% and 50.5%. Appendix D.1 gives exact count-based differences of +4.0 pp over Quality-only and +2.7 pp over Global Cosine: 279/525, 258/525, and 265/525 successes, respectively. Thus, the +2.6 annotation refers to Global Cosine, not Quality-only.
Paired hierarchical bootstrap uses 10,000 resamples, model strata and task-pair/pool grouping, with identical configuration indices for Ours and its comparator. Estimates and 95% CIs are:
All intervals include zero, so the observed gains do not establish superiority at the 95% level. Paired outcomes change in both directions: spectral ranking has 33 exclusive successes versus 12 for Quality-only, and 28 versus 14 for Global Cosine. The 525 configurations share pools, tasks, and assessment items and are not independent trials. The randomized controls provide complementary evidence, not statistical significance for these comparisons.
W4. Variation across models
In the original system-level comparison, Direct Search leads on Qwen2.5-7B (61.0% versus 59.0% SJ) and TIES leads on OLMo 3-7B (53.3% versus 51.4%). These exceptions preclude a claim of universal superiority.
The matched ranking results use 105 configurations per model. SJ rates and count-based differences are:
The aggregate benefit is distributed across four checkpoints, with a tie on OLMo.
Continued: Part 2/Q1 baselines; Part 3/Q2 efficiency; Part 4/Q3 regression and three experts.