acceptodds
Under review as a conference paper at ICLR 2027

When Are Nearby Experts Composable? Cross-Task Model Merging in Neural Thickets

Abstract

Random perturbations around a shared checkpoint can produce experts for distinct tasks, but shared origin does not ensure that their gains survive composition. We formulate expert-pair selection for joint base-relative retention, measured by strict joint success and worst-task gain with unconditional abstention accounting. An exact finite-difference decomposition and an arbitrarily small orthogonal counterexample show why proximity and orthogonality alone are insufficient. We instantiate this formulation with quality-constrained spectral selection, a fixed-budget framework combining cross-task screening, layerwise pair geometry, separate validation, and Base fallback. Across six checkpoints, 21 task pairs, and five independent perturbation pools, the procedure improves strict joint success over Top-1 averaging throughout and leads the four-method model-scale comparison on four of six checkpoints. For the five models with at least 1.5B parameters, matched quality-only, global-cosine, and spectral ranking attain 49.1%, 50.5%, and 53.1% joint success. Under this matched protocol, the spectral selector also yields the highest worst-task gain and Weak Pareto rate and the lowest regression rate. Mean worst-task gain reaches +0.61 percentage points, while 17.1% regression and 13.0% abstention show that composability remains conditional. Project page: https://anonymous.4open.science/r/Thickets-Merge/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Merging, Random Weight Perturbations, Task Experts, Cross-Task Composability, Spectral Analysis, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PrWgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates expert-pair selection for model merging, focusing on preserving joint task performance when combining experts generated from a shared checkpoint. It proposes a fixed-budget spectral selection framework that integrates cross-task screening, layerwise pair geometry, separate validation, and base-model fallback. Experiments across six checkpoints show improved strict joint success over Top-1 averaging, with spectral selection outperforming quality-only and cosine-based baselines on several key metrics, while highlighting the conditional nature of expert composability.

Critical strengths

1.The paper identifies a practical limitation of model merging: individually beneficial task experts may not preserve both tasks' performance when composed. Its focus on strict joint improvement over the base model is meaningful and avoids the limitations of average-score-based evaluation.

2.The proposed method combines cross-task quality screening, layerwise singular-subspace overlap, bounded validation, and base-model fallback. The matched quality-only and global-cosine baselines help isolate the contribution of spectral pair ranking.

Critical weaknesses
  1. Limited methodological novelty. The core components, task-quality screening, singular-subspace similarity, coefficient calibration, and validation-based selection, are related to existing model-merging approaches. The paper should more clearly distinguish its contribution from the integration of existing model-merging and parameter-geometry approaches.

  2. Insufficient theoretical support for spectral ranking. The theoretical analysis explains why proximity and orthogonality alone do not guarantee composability, but does not directly establish why layerwise subspace overlap predicts better joint task retention.

  3. Under the matched protocol, spectral ranking improves strict joint success by only 2.6 percentage points over quality-only ranking and 0.07 percentage points in mean worst-task gain over global-cosine ranking. Without statistical significance tests or confidence intervals for these comparisons, it is difficult to determine whether the incremental gains are robust.

  4. Inconsistent superiority across models. Although the proposed method outperforms Top-1 averaging on all six checkpoints, it does not achieve the best strict joint success on Qwen2.5-7B or OLMo 3-7B, where Direct Search and TIES perform better, respectively. This limits the evidence for its general superiority over existing merging methods.

  5. Incomplete evaluation of practical efficiency. Although the paper includes sensitivity analyses for several design choices, it does not provide a sufficiently comprehensive, matched assessment of end-to-end runtime and computational overhead, including SVD and candidate screening.

  6. The proposed method still exhibits a 17.1% regression rate and a 13.0% abstention rate in the primary aggregate. Moreover, the evaluation focuses on two-task composition, so the results do not establish that the method will scale effectively to merging multiple experts simultaneously.

Questions for authors

See weaknesses

4 comments

Response to Reviewer PrWg — Part 1/4

Thank you. SJ denotes strict joint success; WG denotes mean worst-task gain; pp denotes percentage points.

All rebuttal analyses retain the original selection, validation, and held-out test partitions used in the submission; no new data split was introduced.

W1. Contribution relative to existing merging and geometry methods

Screening, singular-subspace similarity, coefficient calibration, and validation have precedents. Our contribution is the formulation and controlled evaluation of joint base-relative retention for separately selected local perturbation experts, under bounded validation with unconditional abstention accounting. RandOpt uses sample-select-ensemble; here the decision concerns weight-space composition. The joint-retention objective, failure analysis, and bounded selection procedure define this problem, while matched controls isolate the contribution of pair ranking. We do not claim to introduce singular-subspace similarity itself.

W2. What the theory establishes and what supports spectral ranking

The analysis establishes that proximity and orthogonality do not ensure joint improvement. Its exact decomposition separates individual effects from interactions, motivating cross-task screening and validation. It does not prove that lower overlap yields better retention; overlap is an empirical prioritization signal.

The shuffled-overlap control permutes overlap assignments within each model-task-pair-pool feasible set, holding quality scores, screened candidates, K=50, B=10, five coefficient proposals, and the validation gate fixed. Random tie-breaking changes only exact quality ties, preserving distinct-quality ordering. Each randomized control uses 20 repetitions of the same 525 configurations; entries report mean ± standard deviation, with SJ dispersion in pp. Spectral ranking is the original 525-configuration reference.

MethodSJ (%)WG (pp)
Random-pair selection45.8 ± 1.6+0.34 ± 0.07
Quality-only with random tie-breaking49.4 ± 0.6+0.50 ± 0.04
Shuffled overlap49.8 ± 0.7+0.51 ± 0.05
Spectral ranking53.1+0.61

Relative to Quality-only, spectral ranking replaces an average of 2.4 of ten shortlisted pairs (24%) and retains 7.6 under the same validation budget.

These controls favor retaining pair-specific geometry, without establishing a causal mechanism or universal overlap-success relationship. As noted in Related Work, Demystifying Mergeability reports a positive association with leading right-subspace alignment for fine-tuned CLIP pairs. Our empirical claim concerns the screened local-perturbation setting.

W3. Magnitude and uncertainty of the incremental gains

Table 2's +2.6 pp annotation subtracts the displayed SJ values for Ours and Global Cosine, 53.1% and 50.5%. Appendix D.1 gives exact count-based differences of +4.0 pp over Quality-only and +2.7 pp over Global Cosine: 279/525, 258/525, and 265/525 successes, respectively. Thus, the +2.6 annotation refers to Global Cosine, not Quality-only.

Paired hierarchical bootstrap uses 10,000 resamples, model strata and task-pair/pool grouping, with identical configuration indices for Ours and its comparator. Estimates and 95% CIs are:

ComparisonΔSJ (pp)95% CIΔWG (pp)95% CI
Ours minus Quality-only+4.0[-0.1, +8.1]+0.12[-0.01, +0.25]
Ours minus Global Cosine+2.7[-1.0, +6.5]+0.07[-0.03, +0.17]

All intervals include zero, so the observed gains do not establish superiority at the 95% level. Paired outcomes change in both directions: spectral ranking has 33 exclusive successes versus 12 for Quality-only, and 28 versus 14 for Global Cosine. The 525 configurations share pools, tasks, and assessment items and are not independent trials. The randomized controls provide complementary evidence, not statistical significance for these comparisons.

W4. Variation across models

In the original system-level comparison, Direct Search leads on Qwen2.5-7B (61.0% versus 59.0% SJ) and TIES leads on OLMo 3-7B (53.3% versus 51.4%). These exceptions preclude a claim of universal superiority.

The matched ranking results use 105 configurations per model. SJ rates and count-based differences are:

ModelQuality-only SJ (%)Global Cosine SJ (%)Spectral SJ (%)Spectral minus Cosine (pp)
Qwen2.5-1.5B38.140.042.9+2.9
Qwen2.5-3B50.552.455.2+2.9
Qwen2.5-7B54.356.259.0+2.9
Llama-3.1-8B51.452.457.1+4.8
OLMo 3-7B51.451.451.40.0

The aggregate benefit is distributed across four checkpoints, with a tie on OLMo.

Continued: Part 2/Q1 baselines; Part 3/Q2 efficiency; Part 4/Q3 regression and three experts.

Response to Reviewer PrWg — Part 2/4

The review's Questions section states “See weaknesses”. Q1–Q3 across Parts 2–4 organize additional responses to W4–W6.

Q1. Matched baseline comparison (W4)

The new matched comparison uses the same 525 configurations, 200 validation items/task, at most 50 proposals/task pair, and strictly positive validation gains on both tasks. All methods return Base if none passes; testing uses the original held-out data. TIES selects retention from 5%, 10%, 20%, 50%, and 100% by validation; Direct Search selects one expert maximizing worst-task validation gain under the same gate and fallback rule.

These implementation details apply to the additional matched 525-configuration comparison in Appendix E.3, not the original system-level comparison in Table 2.

TIES implementation. Deltas are delta_i = theta_i - theta_0 relative to the common Base. All floating-point model parameters are included, with shared parameters counted once and buffers excluded. For each expert, trimming retains the largest absolute entries globally across its complete delta vector, at retention r = 0.05, 0.10, 0.20, 0.50, 1.00; both experts use the same r. At each coordinate, the sign of the sum of retained updates is elected, and the nonzero retained updates agreeing with that sign are averaged. A zero sum produces a zero merged update. The merged delta is added to Base with scale 1, without additional normalization or rescaling.

TIES ranks screened feasible pairs by the paper's quality-only score Q, without spectral information, and retains up to ten distinct pairs. Its budget is at most 10 pairs × 5 retention rates × 1 scale = 50 proposals. The original alpha/beta coefficient grid is not additionally applied to TIES.

Direct Search implementation. All 5,000 realized candidates are ranked on selection data by descending q = min(g_A, g_B), then descending m = (g_A + g_B)/2, then ascending candidate ID; the first 50 enter validation. Here each selection gain is the candidate's selection score minus Base's selection score; for accuracy, gains use pp. Direct Search returns one expert, without combining candidates.

For both baselines, each proposal uses the original 200 validation examples per task. Among proposals with strictly positive gains on both tasks, selection maximizes worst-task validation gain, then mean gain, then fixed proposal order; if none passes, return Base. The held-out test is used only for the final selected output, with no test-based selection or post-test fallback.

MethodSJ, % (count/525)WG (pp)Regression, % (count/525)Abstention, % (count/525)
TIES48.0 (252/525)+0.4222.5 (118/525)13.5 (71/525)
Direct Search50.3 (264/525)+0.4920.8 (109/525)13.1 (69/525)
Ours53.1 (279/525)+0.6117.1 (90/525)13.0 (68/525)

These new matched results favor Ours in aggregate SJ, WG, and regression, with shared validation and fallback opportunities. They neither replace the original model-specific exceptions nor establish statistical significance against TIES or Direct Search. The Quality-only and Global Cosine controls isolate the spectral ranking component.

Efficiency continues in Part 3, Q2; regression and three-expert results in Part 4, Q3.

Response to Reviewer PrWg — Part 3/4

Continuation of Part 2: Q2 responds to W5.

Q2. Computational overhead (W5)

Timing uses Qwen2.5-3B on one dedicated node with eight NVIDIA A100 80GB GPUs, one model replica per GPU, BF16 inference, and batch size 16 per GPU. Prompts and generation-length limits are identical across methods; subspace computation uses FP32.

For each method, five pools are each timed five times: 25 complete runs. One run covers one full candidate pool, seven tasks, and all 21 task pairs. The table reports mean wall-clock hours per pool on the eight-GPU node, not summed GPU-hours.

Each candidate is screened once per task. For retained experts, each subspace is computed once and reused across task pairs within that pool. Model outputs are not reused between methods or independent timing repetitions.

StageQuality-onlyGlobal CosineSpectral
Pool generation (h)6.06.06.0
Screening and shared preprocessing (h)160.0160.0160.0
Ranking, including applicable SVD (h)0.27.432.6
Reconstruction and validation (h)19.819.819.8
Post-generation total (h)180.0187.2212.4
Post-generation time ratio1.000×1.040×1.180×
Generation-to-selection total (h)186.0193.2218.4

Spectral adds 18% wall-clock time over Quality-only after generation; including generation, the increase is approximately 17.42%. The original 35,000 screening and at most 2,100 calibration task-evaluation counts exclude SVD and reconstruction; these timings include both. The measurements cover Qwen2.5-3B only and do not establish the same overhead across models.

Validation-budget sensitivity is evaluated with spectral weight 0.5, rank cap 8, and K=50 fixed:

Pair shortlist BSJ (%)WG (pp)Regression (%)
551.6+0.5618.5
1053.1+0.6117.1
2053.5+0.6217.0

Increasing B from 10 to 20 adds 0.4 pp SJ and 0.01 pp WG. These results describe the outcome–budget trade-off; runtime is assessed separately above.

W1–W4 are addressed in Part 1; Q1 is in Part 2; Q3 continues in Part 4.

Response to Reviewer PrWg — Part 4/4

Continuation of Part 3: Q3 responds to W6.

Q3. Regression, abstention, and three-expert composition (W6)

The 17.1% held-out regression rate shows that the validation gate does not guarantee improvement. Abstentions return Base, contribute zero gain, and remain in the denominator. The primary 525 outcomes comprise 279 strict successes, 88 weak-only improvements, 90 regressions, and 68 abstentions (13.0%).

We additionally evaluate three-expert composition on Qwen2.5-1.5B and 3B using all four three-task combinations of GSM8K, MATH-500, MBPP, and ROCStories and the five original pools: 2 models × 4 task triplets × 5 pools = 40 configurations. This is separate from the primary 525-configuration study.

Each task supplies up to 50 screened candidates. Triplets take one expert per task, with distinct IDs and fixed 1/3 coefficients. Using the original 200 validation items/task and distinct-triplet budgets below, all three gains must be strictly positive. Selection maximizes worst-task validation gain, then mean gain, then fixed ID order; otherwise return Base. All 40 configurations, including fallbacks, enter the rates and mean WG.

Three-expert selection rule. Each task's candidate set retains up to 50 experts with nonnegative selection gain, ordered by that task's gain descending and then fixed candidate ID order. Form feasible triplets by taking one candidate from each task's set, requiring three distinct IDs. Sorting the IDs gives a canonical tuple: deduplicate these tuples to obtain the feasible set F3\mathcal F_3, so alternate task assignments of the same experts are counted once.

For S={i,j,k}S=\{i,j,k\}, the scores are

Q3(S)=min⁡t∈{A,B,C}13∑u∈Sgtsel(θu)Q_3(S)=\min_{t\in\{A,B,C\}}\frac{1}{3}\sum_{u\in S}g_t^{\mathrm{sel}}(\theta_u),

Ω3(S)=[Ω(i,j)+Ω(i,k)+Ω(j,k)]/3\Omega_3(S)=[\Omega(i,j)+\Omega(i,k)+\Omega(j,k)]/3,

J3(S)=Q3(S)−0.5Ω3(S)J_3(S)=Q_3(S)-0.5\Omega_3(S).

The pairwise overlaps retain the two-expert layerwise definition, rank cap 8, and attention/MLP weights. Quality-based and spectral-guided selection rank by Q3Q_3 and J3J_3, respectively, descending, with score ties resolved by canonical ID tuple. They validate min⁡(10,∣F3∣)\min(10,|\mathcal F_3|) proposals. Random selection samples the same number uniformly without replacement from the same deduplicated set. An empty feasible set returns Base.

Per-task Top-1 chooses each task's highest-ranked expert independently. If these IDs repeat, it returns Base as a structural abstention; lower-ranked experts are not substituted. With distinct IDs, its single proposal uses the same fixed equal weights and validation gate. These structural fallbacks also remain in the 40-configuration denominator.

MethodMaximum validated tripletsSuccess countThree-task SJ (%)WG (pp)
Top-1 equal-weight merge19/4022.5-0.18
Random selection1011/4027.5+0.08
Quality-based selection1013/4032.5+0.23
Spectral-guided selection1015/4037.5+0.31

The aggregate difference over quality-based selection is two additional successes out of 40 (+5.0 pp) and +0.08 pp WG. It is not uniform across models:

ModelMethodSuccess countThree-task SJ (%)WG (pp)
Qwen2.5-1.5BQuality-based5/2025.0+0.09
Qwen2.5-1.5BSpectral-guided5/2025.0+0.08
Qwen2.5-3BQuality-based8/2040.0+0.37
Qwen2.5-3BSpectral-guided10/2050.0+0.54

On 1.5B, joint success is tied and spectral-guided WG is 0.01 pp lower. The improvement occurs on 3B.

This exploratory three-expert result is driven by 3B and establishes neither statistical significance nor general scalability. Pairwise compatibility does not guarantee joint retention; the joint merge is evaluated directly.

Reviewer_JFaPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when task experts obtained by random perturbations around a shared checkpoint can be merged to improve both tasks over Base. It defines joint retention with explicit abstention accounting, shows why proximity and orthogonality alone are insufficient, and proposes quality-constrained spectral pair selection followed by coefficient validation and Base fallback. Experiments cover six checkpoints, 21 task pairs, and five independent pools. Across the five models with at least 1.5B parameters, the method achieves 53.1% strict joint success and +0.61 percentage points mean worst-task gain, with 17.1% regression and 13.0% abstention.

Critical strengths
  1. The joint-retention objective and unconditional accounting are useful. Strict joint success, worst-task gain, and Weak Pareto improvement distinguish outcomes that average accuracy conceals, while retaining Base fallbacks in the denominator prevents selective reporting of accepted merges. Table 2 makes this concrete: Top-1 averaging improves mean accuracy by 0.36 percentage points but has a negative mean worst-task gain of −0.33.

  2. The experimental coverage is substantial: six checkpoints from three families, all 21 pairs of seven tasks, and five independent pools per checkpoint. Appendix D further provides paired outcome counts, model-wise comparisons, and sensitivity analyses for spectral weight, rank, shortlist size, and retained candidates, strengthening the evidence for an incremental ranking benefit.

Critical weaknesses
  1. The practical importance of the spectral component remains limited by its modest gains. Relative to Quality-only and Global Cosine, strict joint success improves by 4.0 and 2.7 percentage points, while mean worst-task gain increases by 0.12 and 0.07 percentage points, respectively (Appendix D.1). The results do not yet establish a substantial benefit over simpler selection under the same budget; a repeated random-pair control would help assess this.

  2. The contribution of spectral information among similarly scored pairs remains unclear. In Equations (6)–(7), the bounded penalty 0.5Ω can alter relative scores by at most 0.5 percentage points, concentrating its effect on close or tied quality scores. Appendix D's paired outcomes and weight sensitivity are helpful, but do not show how often spectral ranking changes the shortlist or whether random tie-breaking achieves similar gains. Section 4 explicitly does not derive the spectral score from its interaction analysis, leaving its empirical role insufficiently explained.

Questions for authors
  1. Can you compare repeated random selection of up to 10 pairs from the same screened feasible set, keeping the other experiment settings fixed? What are the resulting joint-success and worst-task-gain distributions?

  2. Can you repeat Quality-only ranking with random tie-breaking, report the frequency of tied quality scores, and identify which pairs spectral ranking changes in the validation shortlist? Does its advantage persist across these repetitions?

  3. Beyond the non-guarantees in Section 4, do these selected perturbation experts exhibit low-loss connectivity comparable to fine-tuned experts? Existing evidence or representative loss-interpolation results relating connectivity, overlap, and merge success would clarify the explanatory contribution.

  4. Please additionally provide the exact TIES configuration and Direct Search selection rule.

4 comments

Response to Reviewer JFaP — Part 1/4

Thank you for recognizing the joint-retention objective and matched controls. SJ denotes strict joint success, WG mean worst-task gain, and pp percentage points. The 525-configuration comparisons retain the original selection, validation, and held-out test partitions.

W1. Modest incremental benefit

This overview combines the ranking controls discussed in Part 2, Q1 and Q2. Rate columns use percent; their standard deviations use pp. All ± values are standard deviations over 20 repetitions.

MethodSJ (%)WG (pp)Regression (%)Abstention (%)
Random pair selection45.8 ± 1.6+0.34 ± 0.0724.214.5
Quality-only, fixed ties49.1+0.4920.413.5
Global Cosine50.5+0.5419.013.3
Quality-only, random ties49.4 ± 0.6+0.50 ± 0.0420.1 ± 0.513.4 ± 0.3
Shuffled overlap49.8 ± 0.7+0.51 ± 0.0519.9 ± 0.613.4 ± 0.3
Spectral selection53.1+0.6117.113.0

Spectral selection exceeds the random-selection mean by 7.3 pp in SJ and 0.27 pp in WG.

The standard deviations describe shortlist randomization, not confidence intervals for generalization across tasks, pools, or test examples. Gains over the quality and cosine controls remain smaller than gains over random selection.

Paired SJ outcomes show the configurations gained and lost by spectral selection. Each row partitions all 525 configurations:

ComparatorBoth succeedSpectral onlyComparator onlyNeither succeeds
Quality-only2463312234
Global Cosine2512814232

The additional uncertainty analysis uses 10,000 hierarchical bootstrap replicates, identical resampled configuration indices for both methods, and model strata and task-pair/pool grouping in resampling.

ComparisonSJ difference (pp)95% CIWG difference (pp)95% CI
Spectral − Quality-only+4.0[−0.1, +8.1]+0.12[−0.01, +0.25]
Spectral − Global Cosine+2.7[−1.0, +6.5]+0.07[−0.03, +0.17]

All intervals include zero. Shared tasks, pools, and test items mean that the 525 configurations are not independent trials. These results do not establish statistically significant superiority over the quality and cosine controls.

Requiring both task gains to exceed positive margins gives the following success rates on the same 525 configurations. The observed ordering persists beyond arbitrarily small gains; these are descriptive comparisons.

Both task gains exceedQuality-onlyGlobal CosineSpectral
0.5 pp40.8%42.1%44.6%
1 pp31.6%33.0%35.2%
2 pp18.3%19.2%21.0%

W2. Contribution of spectral geometry

The bounded penalty prioritizes pairs of comparable quality: its coefficient of 0.5 and overlap in [0, 1] limit each score change to 0.5 pp. Overlap has mean 0.34 and SD 0.13, giving penalty mean 0.17 pp and SD 0.065 pp. These characterize the ranking signal, not loss interaction.

We repeated Quality-only ranking with random tie-breaking 20 times, with each repetition covering the same 525 configurations. Only exact quality ties are randomized; the ordering between distinct quality levels is unchanged. Results are included in the overview above.

To quantify ties, within each configuration we sort eligible expert pairs after functional screening by the quality term of Eq. (7), without the spectral penalty, and compare adjacent quality scores. The categories are mutually exclusive:

CategoryAbsolute adjacent-score gapFrequency
Exact tie0 pp29.6%
Near tieGreater than 0 and at most 0.25 pp24.8%
SeparatedGreater than 0.25 pp45.6%

Each configuration's denominator is its number of adjacent comparisons; these percentages are equal-weight macro-averages over the 525 configurations. They describe adjacent score comparisons across the eligible set, not candidate proportions or the top-10 cutoff.

For shuffled overlap, we permute overlap-score assignments to candidate pairs within each model–task-pair–pool configuration's screened feasible set. Candidates, quality scores, up to 50 retained experts per task, up to 10 shortlisted pairs, five coefficient proposals, and the validation gate remain fixed. We run 20 repetitions, each covering the same 525 configurations. The overview shows that actual overlap scores outperform both randomized controls in aggregate; this does not establish that spectral selection wins in every repetition.

Shortlist turnover is addressed in Part 2, Q2.

Response to Reviewer JFaP — Part 2/4

Abbreviations follow Part 1. Q3 continues in Part 3 and Q4 in Part 4.

Q1. Random-pair selection

We repeated random selection of up to 10 pairs from the same screened feasible set 20 times, retaining up to 50 experts per task, five coefficient proposals per pair, the validation gate, and Base fallback. The budget remains at most 50 validated merges per task pair; only the choice of pairs changes.

See Part 1, W1 for results and uncertainty.

Q2. Ties and shortlist

See Part 1, W2 for tie frequencies and protocols.

The shortlist comparison concerns candidate pairs. Both methods have 10 entries per compared shortlist; the remaining entries below are averages.

Shared pairsRemoved by spectral rankingAdded by spectral rankingOverlapReplacement
7.62.42.476%24%

For each configuration, let L_Q and L_S be the two size-10 shortlists. Define R = 10 - |L_Q intersect L_S|. Using the unrounded quality component q, let n_Q(q) and n_S(q) count shortlist pairs at quality q. Define C = (1/2) sum_q |n_S(q)-n_Q(q)| and W = R-C. C is the minimum number of cross-quality replacements; W counts replacements matchable within the same quality level. These describe membership differences, not actual pairwise ranking swaps.

R, W, and C below are per configuration; the Total row sums replacements across configurations.

Configuration categoryConfigurationsRWC
Unchanged75000
Within-quality only300220
Cross-quality group 190422
Cross-quality group 260532
Total5251,260960300

Thus, mean R is 1,260/525 = 2.4 (24% of ten entries). Cross-quality changes occur in 150/525 configurations (28.6%), while minimum cross-quality replacements are 300/1,260 replacements (23.8%). These denominators differ. This membership analysis does not replace the adjacent-score tie frequencies in Part 1, W2.

Downstream outcomes use the same 525 configurations:

MethodAccepted mergesStrict successesRegressionsAbstentionsWG (pp)
Quality-only45425810771+0.49
Spectral selection4572799068+0.61

The change in strict successes (+21) is substantially larger than the change in accepted merges (+3). The validation budget is fixed: ranking changes which candidate pairs reach validation. These aggregate outcomes do not attribute individual outcomes to specific pair replacements.

The random tie-breaking, shuffled-overlap, and shortlist comparisons support an empirical role for pair-specific geometry. They neither establish a universal overlap-success relationship nor derive the ranking criterion from Section 4.

Response to Reviewer JFaP — Part 3/4

Abbreviations follow Part 1. Q4 continues in Part 4.

Q3. Loss interpolation and comparison with fine-tuned experts

Sampling and selection. The 60 cells are five primary checkpoints × three pairs of GSM8K, MATH-500, and ROCStories × pool seeds 11, 23, 37, and 42. Random, Quality-only, and Spectral use the same screened feasible set, selecting one pair per cell: Random samples uniformly, Quality-only maximizes Q, and Spectral maximizes Q minus the spectral penalty. Exact ties use fixed candidate-ID order. Selection uses only selection data, not diagnostic or test results.

Loss and interpolation. Each task uses its existing 200 validation examples, disjoint from selection, fine-tuning, and held-out testing. LtL_t is teacher-forced gold-completion NLL, excluding prompt and padding tokens:

Lt(θ)=∑(x,y)∈Dtval∑k=1∣y∣−log⁡pθ(yk∣x,y<k)∑(x,y)∈Dtval∣y∣.L_t(\theta)=\frac{\sum_{(x,y)\in D_t^{\mathrm{val}}}\sum_{k=1}^{|y|}-\log p_\theta(y_k\mid x,y_{<k})}{\sum_{(x,y)\in D_t^{\mathrm{val}}}|y|}.

We evaluate θij(a)=(1−a)θi+aθj\theta_{ij}(a)=(1-a)\theta_i+a\theta_j at 21 points, a = 0, 0.05, ..., 1. Interpolation is in FP32, evaluation in BF16 with dropout off, and loss accumulation in FP32.

bt=max⁡a[Lt(θij(a))−(1−a)Lt(θi)−aLt(θj)]+max⁡{Lt(θ0),10−8},b_t=\frac{\max_a\left[L_t(\theta_{ij}(a))-(1-a)L_t(\theta_i)-aL_t(\theta_j)\right]_+}{\max\{L_t(\theta_0),10^{-8}\}},

where [z]+=max⁡(z,0)[z]_+=\max(z,0) and θ0\theta_0 is Base.

The denominator is shared across methods within each checkpoint/task. We average max⁡(bA,bB)\max(b_A,b_B) equally across the 60 cells. The 0.02 cutoff concerns normalized loss, not accuracy. This finite straight-line grid cannot rule out barriers between sampled points or characterize every parameter-space path.

Fine-tuned reference. Experts start from the same Base and use 2,048 dedicated training examples per task, disjoint from selection, validation, and test. For MATH-500, development and training data come from MATH training data. AdamW uses learning rate 1e-5, betas (0.9, 0.95), epsilon 1e-8, weight decay 0.01, global batch size 16, one epoch, constant learning rate, gradient clipping at 1.0, and maximum sequence length 2,048. Four training seeds give four replicates per checkpoint/task. We use final training checkpoints, without diagnostic- or test-based checkpoint/hyperparameter selection. Comparisons align Base, task pair, replicate index, and loss data; endpoint quality, displacement, and training cost are not matched, so this remains a descriptive reference.

We added a diagnostic with 60 pairs per method, including a fine-tuned reference. A task has a low barrier when its normalized loss barrier is at most 0.02, and a high barrier when it exceeds 0.02. The final column reports the mean maximum normalized loss barrier.

MethodBoth tasks lowOne task highBoth tasks highMean maximum barrier
Random pairs18/60 (30.0%)24/60 (40.0%)18/60 (30.0%)0.052
Quality-only20/60 (33.3%)25/60 (41.7%)15/60 (25.0%)0.046
Spectral24/60 (40.0%)23/60 (38.3%)13/60 (21.7%)0.037
Fine-tuned reference27/60 (45.0%)21/60 (35.0%)12/60 (20.0%)0.034

To relate connectivity, overlap, and merge success, we evaluate the same 60 spectral pairs at the fixed 50:50 midpoint of the expert weights. There is no coefficient selection using test results and no fallback after testing.

Barrier categoryPairsMean overlapSJWG (pp)
Both tasks low240.2415/24 (62.5%)+0.42
One task high230.3110/23 (43.5%)+0.08
Both tasks high130.394/13 (30.8%)−0.25
Total600.3029/60 (48.3%)+0.14

This is a separate 60-pair diagnostic, not the primary 525-configuration validated-selection result. Lower-barrier categories exhibit lower mean overlap and better merge outcomes in this sample. The fine-tuned sample provides an observed reference. These descriptive associations do not establish equivalent connectivity, causality, or a universal overlap-success relationship. Section 4 remains a non-guarantee analysis, not a derivation of the spectral score.

Response to Reviewer JFaP — Part 4/4

Q4. TIES configuration and Direct Search selection rule

These settings apply to the additional matched comparison of 525 configurations in Appendix E.3, not the original Table 2 system-level protocol. SJ denotes strict joint success; WG denotes mean worst-task gain in percentage points.

The updated comparison uses a common validation protocol: 200 validation examples per task, at most 50 proposals per task pair, strictly positive validation gains on both tasks for acceptance, and Base fallback if no proposal passes. Final evaluation uses the original held-out test data.

For TIES, the retention level is selected from 5%, 10%, 20%, 50%, and 100% using validation. Direct Search selects a single realized candidate by maximizing its worst-task validation gain, subject to the same joint-positive acceptance gate; it does not compose two distinct experts. Both baselines have the same fallback opportunity as spectral selection.

TIES implementation. Each expert update is its parameter difference from the same Base. We flatten all floating-point parameters into one vector per expert, counting shared parameters once and excluding buffers. For each expert, we globally retain the largest absolute-magnitude fraction at each retention rate listed above, using the same rate for both experts. At each coordinate, the sum of the retained updates elects the sign; only nonzero entries with that sign are averaged. A zero sum produces zero. The merged update is added to Base with scale 1, without further normalization or rescaling.

We choose up to ten distinct pairs from the screened experts by the unpenalized pair-quality term Q, without spectral ranking. The budget is ten pairs × five retention rates × one scale, at most 50 proposals. The alpha/beta coefficient grid is not applied again to TIES.

Direct Search implementation. From each pool of 5,000 candidates, selection gains are g_t^sel = score_t(candidate) - score_t(Base). For accuracy, gains use percentage points. Candidates are sorted by q = min(g_A^sel, g_B^sel) descending, then m = (g_A^sel + g_B^sel)/2 descending, then candidate ID ascending. The top 50 proceed to validation as individual candidates.

For both baselines, among proposals passing the joint-positive validation gate, final selection maximizes worst-task validation gain, then mean validation gain, then fixed order. If none passes, the output is Base. Held-out testing evaluates only that final output, with no post-test selection or fallback.

The additional matched-protocol results include all 525 configurations and Base fallbacks:

MethodSJWG (pp)RegressionAbstention
TIES48.0% (252/525)+0.4222.5% (118/525)13.5% (71/525)
Direct Search50.3% (264/525)+0.4920.8% (109/525)13.1% (69/525)
Spectral selection53.1% (279/525)+0.6117.1% (90/525)13.0% (68/525)

Spectral selection has higher SJ and WG and lower regression in this aggregate comparison. These remain system-level comparisons because proposal construction and selection differ. Quality-only and Global Cosine isolate pair ranking; their modest effect sizes and zero-crossing confidence intervals remain central to our assessment.

Reviewer_zX1iOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates cross-task merging of local perturbation experts as improving both tasks relative to a shared base checkpoint. Its principal contribution is quality-constrained spectral selection: functional screening, ranking by task quality and layerwise singular-subspace overlap, bounded coefficient validation, and Base fallback. Elementary interaction identities and an orthogonal counterexample establish why proximity and orthogonality alone are insufficient; they do not derive the proposed ranking criterion. The empirical study provides credible evidence of utility under the specified protocol. The remaining issue is whether the modest incremental benefit of spectral ranking is sufficiently reliable, interpretable, and reproducible to establish the contribution strongly.

Critical strengths

S1. The objective and evaluation make cross-task failures visible. Section 3 distinguishes strict joint success, Weak Pareto improvement, worst-task gain, regression, and abstention. Table 2 demonstrates why this matters: Top-1 averaging has positive mean gain of 0.36 percentage points but negative mean worst-task gain of 0.33 points. Figure 3 accounts for all 525 primary evaluations, including 279 strict successes, 90 regressions, and 68 Base fallbacks. Keeping fallbacks in the denominator and explicitly reporting held-out regressions prevents favorable conditional reporting from overstating retention (pages 3 and 7).

S2. The experiments include meaningful controls and broad coverage. The six checkpoints span three model families, seven tasks, 21 task pairs, and five independent perturbation pools. Crucially, Quality-only, Global Cosine, and the proposed selector share screening, coefficient proposals, validation budget, and decision rules. Table D.1 reports matched outcome changes, and Table D.2 shows improvements over cosine on four primary checkpoints with a tie on OLMo. Table D.3 adds sensitivity to spectral weight, rank, shortlist size, and retained-pool size; rank caps from 4 to 16 yield joint success between 52.4% and 53.1%. These controls provide substantive evidence for the added ranking component (pages 5-7 and 19-20).

Critical weaknesses

W1. The added value of spectral ranking remains modest and incompletely established. The strongest comparison is against the matched controls. Relative to Global Cosine, spectral selection adds 14 strict successes among 525 configurations, a count-based increase of 2.7 percentage points, and improves mean worst-task gain by 0.07 points. Table D.1 records 28 spectral-only successes versus 14 cosine-only successes; against Quality-only, the counts are 33 versus 12. These are encouraging paired results. However, shared test items, overlapping task pairs, and reused pools preclude treating all configurations as independent. The Qwen-3B pool standard deviations in Figure 4 describe one method's search variability, not uncertainty in the paired improvement. Dependence-aware uncertainty and success rates at positive gain margins would establish the reliability and practical size of the benefit more convincingly (pages 7-8, 16, and 19).

The theoretical argument also does not explain why lower overlap among the selected leading singular subspaces should predict success. A shuffled-overlap control or repeated random tie-breaking within quality ties would test whether the geometric information is useful beyond perturbing the shortlist. This is particularly relevant because selection uses only 200 examples per task: for accuracy, individual scores change in 0.5-point increments and the averaged pair-quality term in 0.25-point increments. Reporting overlap-score distributions and the frequency of quality-tie versus cross-quality reorderings would clarify what the spectral term actually does. The existing hyperparameter sweeps do not provide this diagnostic.

W2. Essential experimental details and practical comparisons need clarification. Appendix B.1 identifies realized pools by checkpoint, size, and seed, while Algorithm 1 begins after sampling. The manuscript does not specify the perturbation distribution, noise scales, affected tensors, normalization, or generation precision. These choices determine expert quality, locality, and singular-subspace structure; seeds alone do not define the experiments. Precise task splits, scoring configurations, and pool-generation instructions should accompany the otherwise detailed selection protocol (pages 15-17). This concern could be resolved by accessible supplementary material, whose contents were not verified here.

The system-level baselines also need more detail: TIES trimming/scaling settings and the Direct Search selection and validation rules are not specified comparably. Table 2 reports zero abstention for both, versus 13.0% for the proposed procedure; matched validation and fallback rules would clarify the source of the system-level differences. The authors correctly distinguish these comparisons from the matched ranking ablations, but practical value remains difficult to assess. Section 5 counts 35,000 screening and up to 2,100 calibration task evaluations per pool while excluding SVD and reconstruction overhead. Runtime and hardware measurements are needed to relate the small incremental retention gain to its computational cost (pages 5, 7, and 16).

Questions for authors

Q1. How reliable and substantial is the improvement over matched controls? Evidence request for W1: Please provide paired uncertainty estimates with the dependence structure made explicit, and report success across several positive per-task gain margins. Per-task-pair results would help identify whether the improvements and regressions are concentrated in particular capabilities.

Q2. Does the geometric score help beyond changing ties or near-ties? Diagnostic request for W1: Please compare shuffled overlap scores or repeated random tie-breaking under the same budget. Report the spectral penalty's distribution and how often it changes the shortlist within versus across distinct quality levels.

Q3. Can the sampling and comparison protocols be reproduced and costed? Clarification and targeted comparison for W2: Please provide the full perturbation generator, scale and normalization, exact evaluation configurations, baseline settings, and runtime including SVD. How do TIES and Direct Search compare when given the same validation information and fallback opportunity?

4 comments

Response to Reviewer zX1i — Part 1/4

SJ denotes strict joint success; WG is mean worst-task gain; pp denotes percentage points.

This part addresses W1 and Q1. Randomized-control and shortlist diagnostics are in Part 3, Q2; baseline comparisons are in Part 2, W2, and runtime is in Part 4, W2 continued.

W1. Reliability and geometric ranking (also Q1–Q2)

Across the original 525 matched configurations, Spectral has 33 exclusive successes versus 12 for Quality-only and 28 versus 14 for Cosine: count-based SJ gains of 4.0 pp and 2.7 pp.

The model-wise results below cover 21 task pairs × five pools = 105 configurations per model, totaling 525, including Base fallbacks. The last column uses unrounded counts.

ModelQuality-only SJ % (count)Global Cosine SJ % (count)Spectral SJ % (count)Spectral − Cosine (pp)
Qwen2.5-1.5B38.1 (40/105)40.0 (42/105)42.9 (45/105)+2.9
Qwen2.5-3B50.5 (53/105)52.4 (55/105)55.2 (58/105)+2.9
Qwen2.5-7B54.3 (57/105)56.2 (59/105)59.0 (62/105)+2.9
Llama-3.1-8B51.4 (54/105)52.4 (55/105)57.1 (60/105)+4.8
OLMo 3-7B51.4 (54/105)51.4 (54/105)51.4 (54/105)0.0
Total49.1 (258/525)50.5 (265/525)53.1 (279/525)+2.7

Spectral improves on Cosine on four checkpoints and ties on OLMo. These are model-wise aggregates, not a per-task-pair breakdown.

Overlap has mean 0.34 and SD 0.13; the penalty has mean 0.17 pp and SD 0.065 pp. With overlap in [0,1] and weight 0.5, it cannot reverse a quality gap >0.5 pp, prioritizing comparable-quality pairs. Section 4 shows why locality and orthogonality are insufficient; it neither derives this score nor guarantees lower task-loss interaction. The controls test its empirical role under fixed validation.

Spectral exceeds both controls' mean SJ and WG. The random-pair control selects up to ten pairs from the same screened set with the remaining procedure fixed; across 20 repetitions it gives 45.8% ± 1.6 pp SJ and +0.34 ± 0.07 pp WG. These SDs describe randomization variability, not population-level confidence intervals; the comparisons do not establish superiority in every repetition.

The following one-factor sweeps vary spectral weight or rank. Other settings stay at λ=0.5, rank cap 8, B=10, and K=50; the observed benefit is not confined to one λ or rank setting.

FactorSettingSJ (%)WG (pp)Regression (%)
λ049.1+0.4920.4
λ0.2551.8+0.5618.7
λ0.5053.1+0.6117.1
λ0.7552.8+0.6017.5
Rank cap452.4+0.5817.9
Rank cap853.1+0.6117.1
Rank cap1652.8+0.6017.5

Q1. Paired uncertainty and positive gain margins

The supplementary 95% confidence intervals use 10,000 paired hierarchical bootstrap resamples. Ours and each comparator share the resampled configuration indices, preserving model strata and task-pair/pool grouping rather than treating the 525 configurations as independent.

ComparatorΔSJ (pp)95% CIΔWG (pp)95% CI
Quality-only+4.0[−0.1, +8.1]+0.12[−0.01, +0.25]
Global Cosine+2.7[−1.0, +6.5]+0.07[−0.03, +0.17]

All four intervals include zero: superiority is not established at the 95% level. Paired outcomes change in both directions; configurations share tasks, pools, and test items. Positive-margin and randomized controls add descriptive evidence, without changing this conclusion.

We also require both held-out task gains to exceed each margin below. Rates use 525 configurations from a single run, without averaging across repetitions, and are rounded to one decimal place.

Both task gainsQuality-only SJ (%)Global Cosine SJ (%)Spectral SJ (%)
>0 pp49.150.553.1
>0.5 pp40.842.144.6
>1 pp31.633.035.2
>2 pp18.319.221.0

Spectral has the highest observed success rate at every margin. These are descriptive comparisons, not significance tests; they do not identify which task pairs account for improvements and regressions.

Response to Reviewer zX1i — Part 2/4

This part addresses W2's baseline comparison. Runtime is in Part 4, W2 continued; original generation and evaluation settings are in Part 4, Q3.

W2. Baseline comparisons and computational cost (also Q3)

The additional TIES/Direct Search/Spectral comparison shares 200 validation examples per task, at most 50 proposals per task pair, the joint-positive gate, Base fallback if none passes, and original held-out testing. TIES selects its retention rate from 5%, 10%, 20%, 50%, 100% on validation. Direct Search selects a single candidate maximizing worst-task validation gain; merging composes distinct experts. All metrics use 525 configurations, including abstentions:

MethodSJ % (count/525)WG (pp)Regression % (count/525)Abstention % (count/525)
TIES48.0 (252/525)+0.4222.5 (118/525)13.5 (71/525)
Direct Search50.3 (264/525)+0.4920.8 (109/525)13.1 (69/525)
Spectral53.1 (279/525)+0.6117.1 (90/525)13.0 (68/525)

Spectral selection has the highest aggregate SJ and WG and the lowest regression rate in this comparison with common validation and fallback rules. This is a system-level comparison; the Quality-only and Global Cosine controls isolate the pair-ranking component.

Exact configuration of the additional matched comparison. These settings apply to the additional matched 525-configuration comparison in Appendix E.3; the original paper's Table 2 uses its original protocol.

TIES uses updates Δi=θi−θBase\Delta_i=\theta_i-\theta_{\mathrm{Base}} over all floating-point model parameters, counting shared parameters once and excluding buffers. Each expert's complete update vector is trimmed globally by absolute magnitude, retaining fraction r∈{0.05,0.1,0.2,0.5,1}r\in\{0.05,0.1,0.2,0.5,1\}, with the same rr for both experts. The sum of retained updates determines each coordinate's sign; same-sign nonzero entries are averaged, and a zero sum gives zero. The merged update is added back to Base with scale s=1s=1, without additional normalization or rescaling. Quality term QQ alone shortlists at most ten distinct pairs, without spectral ranking. Ten pairs × five retention rates × one scale gives at most 50 proposals; TIES does not additionally use the coefficient grid in Part 4, Q3.

Direct Search ranks the 5,000 candidates on selection data by qi=min⁡(gA,gB)q_i=\min(g_A,g_B) descending, then mi=(gA+gB)/2m_i=(g_A+g_B)/2 descending, then candidate ID ascending, retaining 50. Here each selection gain is candidate score minus Base score; for accuracy, gains are in pp. For both methods, validation uses 200 examples per task and requires both gains >0. Passing proposals are ordered by worst-task validation gain, then mean gain, then fixed order; if none passes, the output is Base. Only the selected output is tested, with no post-test selection or fallback. The validation budget and decision rules are shared, while proposal construction differs.

Three illustrative seed-42 configurations show the variation in test WG (pp):

ModelTask pairSpectralTIESDirect Search
Qwen2.5-1.5BGSM8K–MBPP−2.84−0.18−1.77
Qwen2.5-3BGSM8K–ROCStories−1.33+0.06−0.46
Qwen2.5-3BGSM8K–MBPP−1.95−2.21−2.48

These three cases illustrate that common validation and fallback rules do not eliminate held-out regression or make one method best in every configuration. The aggregate comparison above retains its denominator of 525.

Response to Reviewer zX1i — Part 3/4

This part addresses Q2. Model-wise and sensitivity evidence is in Part 1, W1; paired uncertainty is in Part 1, Q1.

Q2. Randomized controls and shortlist diagnostics

Both randomized ranking controls use 20 repetitions of the same 525 configurations, retaining candidates, quality scores, K=50, B=10, five coefficient proposals, the validation gate, and Base fallback. Random tie-breaking randomizes only exact quality ties, preserving distinct-quality order. Shuffled overlap permutes pair-to-overlap assignments within each model–task-pair–pool screened feasible set. Their entries are mean ± SD across repetitions; rate SDs are in pp. Spectral entries are the original aggregate outcomes.

MethodSJ (%)WG (pp)Regression (%)Abstention (%)
Random tie-breaking49.4 ± 0.6+0.50 ± 0.0420.1 ± 0.513.4 ± 0.3
Shuffled overlap49.8 ± 0.7+0.51 ± 0.0519.9 ± 0.613.4 ± 0.3
Spectral53.1+0.6117.113.0

After screening, eligible pairs in each model–task-pair–pool configuration are ordered by Eq. (7)'s quality term alone. Categories below are mutually exclusive. Each denominator is that configuration's adjacent comparisons; proportions are averaged equally across 525 configurations.

CategoryAbsolute quality-score differenceFrequency (%)
Exact tie029.6
Near tie>0 and ≤0.25 pp24.8
Separated>0.25 pp45.6

The separate top-10 diagnostic measures shortlist membership. Quality-only and Spectral each shortlist ten pairs and share 7.6 on average: 2.4 entries are replaced (24%), with 2.4 leaving and 2.4 entering. These are membership changes, distinct from the adjacent-score comparisons above.

Shortlist changes across quality levels. Let LQ,LSL_Q,L_S be the two ten-pair shortlists and R=10−∣LQ∩LS∣R=10-|L_Q\cap L_S|. For each unrounded quality value qq, let nQ(q),nS(q)n_Q(q),n_S(q) count entries in the respective shortlists. Define C=12∑q∣nS(q)−nQ(q)∣C=\frac12\sum_q|n_S(q)-n_Q(q)|, the minimum number of cross-quality replacements, and W=R−CW=R-C, the within-quality-matchable replacements. These count membership differences, not actual ranking swaps. Group rows give per-configuration R,W,CR,W,C; the final row gives totals.

Configuration categoryCountRWC
Unchanged75000
Within-quality only300220
Cross-quality group 190422
Cross-quality group 260532
Total5251,260960300

Mean replacement is 1260/525=2.41260/525=2.4 entries (24%), consistent with the top-10 result. Cross-quality changes occur in 150/525=28.6%150/525=28.6\% of configurations; minimum cross-quality replacements account for 300/1260=23.8%300/1260=23.8\% of replacements. These have different denominators and measure shortlist membership, separately from the 29.6%/24.8%/45.6% adjacent-score distribution above.

The corresponding outcomes across the same 525 configurations are:

OutcomeQuality-onlySpectral
Accepted merges (count)454457
Strict successes (count)258279
Regressions (count)10790
Abstentions (count)7168
Mean WG (pp)+0.49+0.61

These summaries document shortlist changes alongside improved aggregate outcomes; they do not attribute individual outcome changes to particular entering or leaving pairs.

Response to Reviewer zX1i — Part 4/4

This part addresses Q3 and W2's runtime question. The additional matched baseline settings and results are in Part 2, W2.

Q3. Generation and evaluation protocol (also W2)

Generation adds Gaussian noise to all language-model parameters, at scales 0.0005, 0.001, 0.002, with no normalization, using GPU BF16. Pool seeds are 11, 23, 37, 42, 71.

Primary settings are 5,000 candidates per pool, up to 50 retained per task, and up to ten pairs, with coefficients (0.5, 0.5), (0.25, 0.75), (0.75, 0.25), (1, 0.5), (0.5, 1). All rebuttal analyses reuse the original mutually disjoint partitions: 200 selection and 200 validation examples per task, with held-out testing unchanged. MATH-500 remains reserved for testing; its development subsets come from MATH training data. Evaluation uses BF16, greedy decoding, and one response per item.

Validation requires both gains >0. Among passing proposals, selection maximizes worst-task gain, then mean gain, then fixed candidate/coefficient order; if none passes, return Base. Quality-only, Cosine, and Spectral share pools, screening, coefficients, validation budget, and gate, differing only in pair ranking.

W2 continued. Computational cost (also Q3)

Timing uses Qwen2.5-3B on one dedicated node with eight NVIDIA A100 80GB GPUs, one model replica per GPU, BF16 inference, and batch size 16 per GPU. Prompts and generation-length limits are identical across methods; subspace computation uses FP32.

For each method, five pools are each timed five times: 25 complete runs. One run covers one full candidate pool, seven tasks, and all 21 task pairs. The table reports mean wall-clock hours per pool on the eight-GPU node, not summed GPU-hours.

Each candidate is screened once per task. For retained experts, each subspace is computed once and reused across task pairs within that pool. Model outputs are not reused between methods or independent timing repetitions.

StageQuality-onlyGlobal CosineSpectral
Pool generation (h)6.06.06.0
Screening and shared preprocessing (h)160.0160.0160.0
Ranking, including applicable SVD (h)0.27.432.6
Reconstruction and validation (h)19.819.819.8
Post-generation total (h)180.0187.2212.4
Post-generation time ratio1.000×1.040×1.180×
Generation-to-selection total (h)186.0193.2218.4

Spectral adds 18% wall-clock time over Quality-only after generation; including generation, the increase is approximately 17.42%. The original 35,000 screening and at most 2,100 calibration task-evaluation counts exclude SVD and reconstruction; these timings include both. The measurements cover Qwen2.5-3B only and do not establish the same overhead across models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.