acceptodds
Under review as a conference paper at ICLR 2027

Merging Destroys What You Paid to Teach

Abstract

Large Language Models (LLMs) are increasingly specialized by fine-tuning copies of a shared base model and then merging them into a single model. While model merging is typically evaluated by how well the merged model preserves downstream benchmark performance, these evaluations need not reveal whether the information acquired during specialization survives the merge. We demonstrate this gap on multilingual and coding specialists: merged models with similar benchmark performance can retain very different amounts of the knowledge acquired during specialization. To study this failure systematically, we introduce a suite of ten factual datasets designed to measure knowledge acquired during fine-tuning. Across these datasets, factual retention drops substantially after merging, worsens as more specialists are combined, and can be highly uneven across specialists. To better preserve this knowledge, we introduce LUMEN (Low-rank Update Merging with residual ENhancement), which jointly learns a coordinate-wise mixture of specialist updates and a low-rank correction for the merged model using only a small subset (%) of specialist data. Across merges of two to six specialists, LUMEN improves normalized factual retention by % on average over the strongest reported baseline at each merge size. At six specialists, LUMEN retains % pre-merge factual accuracy, compared with % for the strongest reported baseline. Together, our results establish factual retention as a distinct objective for model merging and show that substantially more of the knowledge acquired during specialization can be preserved at no additional inference cost.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Model Merging, LoRA, Representational Space, Fine-tuning, LLMs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_c8wvOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether model merging preserves the factual information acquired during specialist fine-tuning, arguing that standard task benchmarks can obscure substantial losses in factual recall. The authors introduce ten factual datasets and evaluate retention through the merged model’s accuracy normalized by each specialist’s pre-merge accuracy. Their experiments show substantial recall degradation, worsening retention as more specialists are merged, and asymmetric losses across specialists.

The paper also proposes LUMEN, which combines low-rank-parameterized, coordinate-wise mixing of specialist updates with a learned low-rank residual correction. Both components are optimized using a small calibration subset and materialized into a single checkpoint. LUMEN substantially improves normalized factual accuracy over the evaluated baselines, particularly for larger merges, and achieves the highest aggregate MergeBench score among the methods evaluated in the paper.

Critical strengths
  1. The paper identifies a practically meaningful evaluation gap. Distinguishing broad task performance from recall of specialist-taught information is useful for assessing whether merging preserves the value of specialization.

  2. LUMEN achieves substantial empirical improvements without increasing inference cost. The gains over the strongest reported baseline become large at higher merge sizes: at six specialists, normalized factual accuracy increases from 30.5 to 60.1.

Critical weaknesses
  1. The experiments do not isolate the contribution of the proposed merging mechanism from residual fine-tuning. LUMEN introduces both coordinate-wise gates and an additive low-rank correction, but the paper does not report gate-only, correction-only, or scalar-mixing-plus-correction ablations.

2.The aggregation of per-specialist retention in Figure 4/Table 5 and Table 2 needs clarification. For Open Library + ClinicalTrials, the displayed values average to 69.2%, whereas Table 2 reports 70.7%; for OpenAlex + ClinicalTrials-PI, they average to 56.6%, versus 58.0%. Please specify whether Table 2 uses the macro-average defined in Eq. (6) or another aggregation rule, and reconcile these values.

  1. section 3.2 describes validation-based hyperparameter selection, whereas section 5.1 explicitly reports per-cell oracle choices for dataless baselines and best-of-two learning-rate results for aTLAS. appendix A.4 does not identify the split used to maximize retention. Please specify the selection protocol for each experiment/table and clarify which choices use reporting/test performance.
Questions for authors
  1. Component attribution. Can you provide results for gates without correction, scalar mixing with the same correction, and correction-only adaptation of a conventional merge?

  2. Please specify LUMEN’s training objective: is it supervised cross-entropy on gold answers, distillation from the specialists, or another loss? If cross-entropy is used, is it computed only over answer tokens or over the full sequence? For the N-way experiments, how many candidate checkpoints are evaluated using the 125 selection probes per specialist, and are these probes also used for hyperparameter selection?

  3. Which specialists constitute the nested sequence in Table 3, and how were they selected from the eight specialists mentioned in Section 5.1?

6 comments

We thank reviewer c8wv for a careful and precise review! The reviewer raises 3 concerns. First, whether LUMEN's gain comes from its merging mechanism or from the correction. Second, why the per-specialist values in Figure 4 / Table 5 do not average to the aggregates in Table 2. Third, which hyperparameter and checkpoint choices used the reporting data. The reviewer also asks for LUMEN's training objective, how the selection probes are used, and how the ladder's specialists were chosen. All questions have helped improve the precision and strength of the paper, and have been answered below.

W1 / Q1. Component attribution. “The experiments do not isolate the contribution of the proposed merging mechanism from residual fine-tuning. […]”

Thank you for this insightful question. We acknowledge that this ablation study was missing from the original submission and have now incorporated it into the revised paper. In short, both components help: on the factual ladder at N=6, removing the correction costs 4.4 points, and replacing the gate with scalar mixing costs 2.5.

On our factual retention ladder, when merging N=6 specialists, the gate-only variant (without correction) achieves 66.7% retention. Scalar mixing + correction improves retention to 68.6%, but still falls short of the full LUMEN method, which achieves the highest retention of 71.1%.

Correction only (LUMEN’s rank-1 correction trained on top of task arithmetic, with the merge coefficients frozen at task arithmetic’s tuned α) keeps 100.8% at N=2 but falls to 21.3% at N=6, below even scalar mixing (aTLAS, 33.6%). A correction on top of a fixed merge cannot recover what the merge lost once several specialists are merged, so LUMEN’s gain does not come from fine-tuning a fixed merge.

MethodN=2N=3N=4N=5N=6
aTLAS98.786.156.846.833.6
Correction only100.880.647.827.821.3
LoRA on base17.519.318.420.618.6
Gate only101.996.384.375.766.7
Scalar mixing + correction102.898.287.1 ± 0.779.368.6 ± 0.7
LUMEN103.1 ± 0.598.4 ± 0.488.2 ± 0.381.5 ± 0.571.1 ± 0.3

Retention (%), macro-averaged as in Eq. 6. ± is the standard deviation over 3 seeds, where shown; other cells seed 1. aTLAS at its selected learning rate.

On MergeBench, the components do different jobs: the gate alone gives the best GSM8K score (56.6), and the gate with the correction (LUMEN) raises safety from 45.0 to 67.6. Together, LUMEN has the highest average (46.9 ± 1.7, vs 46.1 ± 1.8 for scalar mixing with the same correction). On MergeBench, correction only reaches an average of 43.3 against LUMEN’s 46.9. It keeps less safety (58.3 vs 67.6) and fewer Aya facts (77.8% vs 95.4%).

MethodInstrGSM8KMultiCodeSafetyAvgAya
Correction only27.947.144.838.458.343.377.8
Gate only29.856.646.538.645.043.388.9
Scalar mixing + correction30.9 ± 2.649.4 ± 5.945.7 ± 0.236.9 ± 2.267.8 ± 2.646.1 ± 1.890.4 ± 5.3
LUMEN32.4 ± 2.350.5 ± 5.546.0 ± 0.238.2 ± 1.867.6 ± 2.546.9 ± 1.795.4 ± 3.3

So the gain comes from learning the mixture together with the correction, not from fine-tuning a fixed merge. With the same correction at N=6, a fixed merge retains 21.3%, learned scalars 68.6%, and learned gates 71.1%. The gate's edge over scalars grows with N, to 1.1–2.5 points from N=4 on.

W2. Aggregation in Table 2 vs. Figure 4 / Table 5

Thank you for catching this. Table 2 did not use Eq. 6: it reported the ratio of mean accuracies (mean merged accuracy over mean specialist accuracy), which exceeds the mean-of-ratios (Eq. 6) when the specialist that kept more also had the higher accuracy. All aggregates in the revised paper use (Eq. 6) on the per-specialist values shown, computed from the same runs.

Open Library + ClinicalTrials has a second cause: its per-specialist values in Figure 4 / Table 5 (62.9 / 75.4) were copied from Task Arithmetic’s training run with hyperparameter λ = 0.45, not the selected λ = 0.55. The correct values are 63.8 / 77.1. The aggregation rule changes five of the six cross-dataset rows, not only the two that were identified:

PairMean of displayed valuesSubmitted Table 2Revised (Eq. 6)
Open Library + ClinicalTrials69.270.770.5
OpenAlex + ClinicalTrials-PI56.658.056.6
Open Library + OpenAlex67.167.767.0
UniProt + Caselaw57.057.257.0
Open Library + ClinicalTrials-PI57.157.257.1

OpenAlex + ClinicalTrials is unchanged (74.3). Every change is under 1.5 points, and none affects a conclusion. On the new full test sets, the two aggregation rules differ by at most 0.05 points.

W3. Selection protocol. “Please specify the selection protocol for each experiment/table and clarify which choices use reporting/test performance.”

We agree the submission mixed two protocols: the data-free baselines and aTLAS's learning rate were chosen per cell on the reporting probes, and LUMEN's checkpoint on the 125 selection probes. In the revised paper, no choice uses test data, and every learned method is evaluated at its final step.

BenchmarkWhat is tunedChosen on
Factual ladderBaselines' coefficients and learning rates (aTLAS per N); LUMEN's gate learning rate, once at N=4125 selection facts per specialist, excluded from every reported number
MergeBenchEvery method's coefficients and learning ratesHeld-out loss on 200 calibration rows per task
FusionBenchLUMEN's correction learning rate and steps; baselines at FusionBench defaultsA held-out part of LUMEN's ≤ 5% training slice
Pairwise merges (Table 2)Data-free baselines' merging coefficientEach baseline's best coefficient per pair on the reporting probes, as in the submission. Table 2 contains only data-free baselines, so each is shown at its strongest and the loss we measure is conservative.

aTLAS is tuned at every N while LUMEN is tuned once, which favours the baseline.

Q2. Training objective and selection probes. “Please specify LUMEN’s training objective […] how many candidate checkpoints are evaluated using the 125 selection probes per specialist, and are these probes also used for hyperparameter selection?”*

LUMEN minimizes cross-entropy on the gold answer tokens only. The loss covers the tokens after <ans>; the prompt tokens are masked.

L=−1∑(x,y)∈B∣y∣∑(x,y)∈B∑t=1∣y∣log⁡pW(yt∣x,y<t)\mathcal{L} = -\frac{1}{\sum_{(x,y)\in\mathcal{B}} |y|}\sum_{(x,y)\in\mathcal{B}}\sum_{t=1}^{|y|} \log p_{W}\left(y_t \mid x, y_{<t}\right)

For each calibration fact entity > relation > <ans> value, the loss is the cross-entropy of value given everything up to <ans>, averaged over the answer tokens. The targets are the gold answers, not the specialists’ outputs, so there is no distillation. On MergeBench, the loss is likewise computed only on response tokens. The gates, the correction and the scalar coefficients are trained jointly on this single loss. We have added this to Section 4 of the revised paper (uploaded soon).

In the submission, LUMEN’s step was chosen among 6 evaluations (every 200 steps) on the 125 selection probes. Now no checkpoint is chosen: every learned method is evaluated at its final step. Yes, the same selection facts are used for hyperparameter selection, and they never enter a reported number: data-free baselines compare 4–16 settings on them, aTLAS four or five learning rates per merge size, and aTLAS×K nine (lr, weight decay) settings at N=2 and N=4, reused at every other N.

Q3. The ladder’s specialists. “Which specialists constitute the nested sequence in Table 3, and how were they selected from the eight specialists mentioned in Section 5.1?”

The ladder adds specialists in this order: NIH RePORTER, Wikidata, OpenAlex, ClinicalTrials, UniProt, Caselaw. The order was randomly selected before any ladder method was run. The other two (Open Library, ClinicalTrials-PI) enter in the B and C orderings below. Further, we also tested two more orderings, and found that LUMEN’s lead was consistent across all orderings we tested.

Across all 3 orderings, LUMEN leads the best prior method at every merge size, by 22–33 points at N ≥ 4. The 2 ordering results (selected methods) are shown below: Ordering B (OpenAlex, Caselaw, NIH RePORTER, UniProt, Open Library, Wikidata):

MethodN=2N=3N=4N=5N=6
Task Arithmetic61.726.48.35.24.4
RegMean64.931.117.913.29.8
Iso-C75.061.152.141.430.6
Iso-CTS72.059.149.042.932.2
aTLAS92.974.857.644.234.3
aTLAS×K66.565.651.650.741.9
LUMEN100.294.488.279.570.0

Ordering C (ClinicalTrials, Wikidata, UniProt, NIH RePORTER, ClinicalTrials-PI, Caselaw):

MethodN=2N=3N=4N=5N=6
Task Arithmetic72.525.110.17.24.3
RegMean64.533.418.914.311.4
Iso-C88.474.653.930.823.6
Iso-CTS76.570.952.333.525.0
aTLAS96.479.256.233.926.1
aTLAS×K85.171.062.743.834.6
LUMEN101.895.587.765.856.9

Absolute retention does depend on which specialists are merged (ordering C is the hardest, 56.9% at N=6), but LUMEN leads the best prior method at every merge size in all three orderings, by 22–33 points at N ≥ 4.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. However, a few follow-up questions remain.

  1. Why are the scores in Table W1 significantly higher than those in Table 3 of the original paper? (For example, when N=6, the score for LUMEN increases from 60.1 to 71.1, and for aTLAS from 26.8 to 33.6.) Where does this improvement come from?

  2. There are "MATH" and "SYNTAX" columns in Table 4 of the original paper. Why are there not these two columns in the Rebuttal W1 table?

Reviewer_wgfFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates factual knowledge retention in model merging, highlighting that strong performance on standard benchmarks does not necessarily indicate that knowledge acquired by individual experts during fine-tuning is preserved. To evaluate this issue, the authors construct a factual retention benchmark from nine public datasets and one synthetic dataset. Experiments across different merging settings reveal declines in factual recall and uneven knowledge retention across experts.

To address this problem, the authors propose LUMEN, which jointly learns low-rank coordinate-wise mixing gates and residual corrections using a small amount of calibration data. The learned parameters can be folded into a single merged model without introducing additional inference overhead.

Experiments on Qwen3.5-4B with two to six experts show that LUMEN improves normalized factual retention by an average of 21.5 percentage points over the strongest reported baselines. In the six-expert setting, retention reaches 60.1%, compared with 30.5% for the baseline. On the five-expert Llama-3.2-3B MergeBench setting, LUMEN achieves the highest average task score of 41.3, while also improving factual probe performance.

Overall, the paper highlights factual retention as an important complementary metric to standard task performance when evaluating model merging methods.

Critical strengths

A key strength of this work is that it highlights a gap in current model-merging evaluation: strong downstream performance does not necessarily mean that knowledge acquired during fine-tuning has been preserved. The factual-retention benchmark provides a useful complement to standard capability evaluations. Building on this observation, LUMEN improves retention as more specialists are merged while still producing a single checkpoint with no additional inference cost, making the method practically relevant for model consolidation.

Critical weaknesses

1.The paper lacks ablation studies of LUMEN’s two core components. It is currently unclear whether the gains come from coordinate-wise gating, the low-rank correction, or their combination. Results for each component in isolation would help clarify their respective contributions. The aTLAS ×K comparison only shows that increasing the number of mixing parameters is insufficient and does not replace a direct ablation of LUMEN.

2.The paper lacks sufficient comparisons with recent representative model-merging methods. The experiments do not include relevant approaches such as TSV-Merge, Iso-CTS, or WUDI-Merging. Notably, TSV-Merge is discussed in the paper but omitted from the empirical evaluation. Consequently, the current results establish LUMEN’s advantage only over the selected classical baselines, rather than its competitiveness against more recent state-of-the-art methods.

Questions for authors

What exact objective is used to optimize LUMEN’s gates and low-rank residual corrections? Is the loss computed over the full sequence or only over the answer tokens following <ans>? Although the paper specifies the sampling strategy and optimization hyperparameters, this training detail appears to be missing and is important for both reproducibility and interpreting the reported gains.

3 comments

We thank reviewer wgfF for the constructive review! They ask whether LUMEN's gain comes from the gate, the correction or both, how it compares with TSV-Merge, Iso-CTS and WUDI-Merging, and what its training objective is. We answer the first two with new experiments and clarify the third. Along the way we found LUMEN was under-tuned in the submission (its gate barely moved), so all LUMEN numbers below use the retuned recipe as mentioned in the General response.

W1. Ablation of LUMEN’s two components. “The paper lacks ablation studies of LUMEN’s two core components. […]”

Thank you for this insightful question! We acknowledge that this ablation study was missing from the original submission and have now included it in the revised paper. Our results demonstrate that both the mixing and correction components play important roles in LUMEN’s performance, with removing either component leading to noticeable performance degradation.

On our factual retention ladder, when merging N=6 specialists, the gate-only variant (without correction) achieves 66.7% retention, while LoRA on base achieves only 18.6%. Correction only (LUMEN’s rank-1 correction on a task-arithmetic merge) retains 21.3%. In comparison, the full LUMEN method achieves 71.1% retention, outperforming all three ablations.

On MergeBench, each component alone reaches the same average (gate only 43.3, correction only 43.3), and LUMEN, which combines them, reaches 46.9. The gate alone gives the best GSM8K score (56.6); adding the correction raises safety from 45.0 to 67.6.

So most of the gain comes from the learned mixing (66.7% for the gate alone vs. 21.3% for the correction alone at N=6), and combining the two is best on both benchmarks.

N=2N=3N=4N=5N=6
scalars only (aTLAS)98.786.156.846.833.6
correction only100.880.647.827.821.3
LoRA on the base, 4.3M params17.519.318.420.618.6
gate only101.996.384.375.766.7
LUMEN (gate + correction)103.1 ± 0.598.4 ± 0.488.2 ± 0.381.5 ± 0.571.1 ± 0.3

W2. Recent baselines. “The paper lacks sufficient comparisons with recent representative model-merging methods. […] such as TSV-Merge, Iso-CTS, or WUDI-Merging.”

We thank the reviewer for bringing this up. We have since added TSV-Merge, Iso-C, Iso-CTS, and WUDI-Merging on the three benchmarks: our factual ladder, MergeBench, and the FusionBench vision benchmark. LUMEN outperforms all four on every benchmark, and its lead widens as more models are merged. With six specialists on the factual ladder, it retains 71.1% against 31.7% for the best of them (Iso-CTS). On FusionBench with 20 tasks, it is 10.9 points ahead.

MethodLadder N=6MergeBench AvgFusionBench 8 tasksFusionBench 20 tasks
TSV-M2.138.083.173.2
Iso-C31.037.580.470.3
Iso-CTS31.737.278.873.5
WUDI7.838.184.867.1
LUMEN71.1 ± 0.346.9 ± 1.786.8 ± 0.184.4 ± 0.2
Fine-tuned (ceiling)10053.790.489.8

*Ladder: factual retention (%) on Qwen3.5-4B. MergeBench: mean of five task scores on Llama-3.2-3B (the average uses GSM8K for math). FusionBench: average accuracy (%) when merging CLIP ViT-B/32 specialists. All methods are run on the same checkpoints; per-task FusionBench results are in Appendix (paper uploaded soon)

  • Factual ladder: at N=6, the recent methods keep at most 31.7% of the taught facts. LUMEN keeps 71.1%.
  • MergeBench: all four trail LUMEN by about 9 points.
  • FusionBench: LUMEN’s lead grows from 2.0 points at 8 tasks to 10.9 at 20.

Q1. Training objective. “What exact objective is used to optimize LUMEN’s gates and low-rank residual corrections? Is the loss computed over the full sequence or only over the answer tokens following <ans>?”

LUMEN minimizes cross-entropy on the gold answer tokens only. The loss covers the tokens after <ans>; the prompt tokens are masked.

L=−1∑(x,y)∈B∣y∣∑(x,y)∈B∑t=1∣y∣log⁡pW(yt∣x,y<t)\mathcal{L} = -\frac{1}{\sum_{(x,y)\in\mathcal{B}} |y|}\sum_{(x,y)\in\mathcal{B}}\sum_{t=1}^{|y|} \log p_{W}\left(y_t \mid x, y_{<t}\right)

For each calibration fact entity > relation > <ans> value, the loss is the cross-entropy on the value tokens only, given everything up to <ans>. The targets are the gold answers, not specialist outputs, so there is no distillation. On MergeBench, the loss is likewise computed only on response tokens. The gates, the correction and the scalar coefficients are all trained jointly on this one loss. We have added this to Section 4 of the revised paper (uploaded soon).

Dear Reviewer wgfF,

Thank you again for your thoughtful feedback. We have carefully considered your comments and incorporated substantial revisions into the updated manuscript (uploaded now). We believe our responses and revisions address your main concerns.

As the discussion period is coming to a close tomorrow, we would greatly appreciate it if you could take another look at our responses and updated manuscript and let us know if you have any remaining concerns or questions.

Thank you again for your time, feedback, and consideration of our manuscript.

Sincerely,

The Authors

Dear Reviewer wgfF,

As the discussion period comes to a close, we wanted to follow up on our rebuttal and revised manuscript.

We have directly addressed your main concerns through new component ablations, comparisons with recent model merging methods, and clarification of the training objective.

Importantly, LUMEN substantially outperforms all three baselines you specifically requested (TSV-Merge, Iso-CTS, and WUDI-Merging) across our factual retention benchmark, MergeBench, and FusionBench. On factual retention with six specialists, LUMEN achieves 71.1%, compared to 31.7% for the strongest of these baselines (Iso-CTS), an improvement of 39.4 percentage points. On MergeBench, LUMEN achieves an average score of 46.9, compared to 38.1 for WUDI. On FusionBench's 20-task benchmark, LUMEN achieves 84.4% accuracy, compared to 73.5% for Iso-CTS, an improvement of 10.9 percentage points.

We believe these results, together with the additional ablations, directly address your concerns and substantially strengthen the paper. We would greatly appreciate it if you could reconsider your assessment in light of the new evidence and let us know if any concerns remain.

Thank you for your time and consideration.

Best, The Authors

Reviewer_R4cNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that model merging is usually judged on broad benchmarks, which can hide whether the merged model still remembers the specific facts each specialist was fine-tuned on. The authors build ten fact-recall datasets the base model can't answer, fine-tune specialists on them, and show that standard merges lose many of these facts, more so as more specialists are merged, and often unevenly between specialists. They then propose LUMEN, which learns per-weight mixing gates built from low-rank factors plus a small low-rank correction on a few thousand specialist examples, and folds both into one checkpoint. LUMEN keeps about twice as many facts as the best baseline when six specialists are merged.

Critical strengths
  • The question is clear and useful. Showing that merges with nearly the same benchmark scores keep very different amounts of what was actually taught is a good warning for how merging is evaluated.
  • The fact datasets where the base model scores zero make a clean testbed that others could reuse.
  • I like some of the analysis findings, especially that TIES wins whenever the specialists share relation types and loses whenever they don't.
  • Everything folds into a single checkpoint, so the gains come at no inference cost.
Critical weaknesses
  • The gate and the low-rank correction are never ablated separately. The correction doesn't use the specialists' updates at all, so it is basically a rank-one LoRA trained on the calibration facts. Without gate-only and scalar-mixing-plus-correction results, I can't tell whether the gain comes from better mixing or just from some fine-tuning after the merge.
  • I'm not convinced by the capacity control. The row-block version of aTLAS can represent plain aTLAS, yet it does far worse and even falls below untuned Task Arithmetic with two specialists, which looks like a tuning problem rather than evidence that more parameters don't help. Its learning rate isn't reported.
  • On MergeBench, LUMEN is trained on the calibration split but only compared with training-free merges. Its higher average also comes mostly from the safety score, and it isn't the best merge on instruction following, math, multilingual or coding. A learned baseline on the same data is needed.
  • Retention isn't measured relative to the base model. On the Aya probe the base model already gets a large share of the specialist's correct answers, so the baselines look better than they are, and that probe is small with the specialist itself only getting about half right.
  • Everything uses one seed, a small test set per specialist, and a single ordering of six specialists that is never disclosed, with no error bars. Since retention depends a lot on which datasets are merged, I'd like to know the specialists, see another ordering, and see LUMEN on the synthetic dataset where merging fails worst.
  • The related work misses several close papers that should be in the background/literature review:
    • Yang et al., Representation Surgery for Multi-Task Model Merging, ICML 2024. They train a small module after merging to fix the merged model's representation drift, which is very close to LUMEN's correction term and its motivation.
    • Wang et al., Localizing Task Information for Improved Model Merging and Compression, ICML 2024. They show each task's information survives in the merged weights and can be recovered with per-weight masks, which relates to the per-weight gate and to why held-out facts come back.
    • Arnob et al., Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts, COLM 2025. They study how merging degrades as more experts are combined and compare sparse with low-rank experts, which relates to the scaling experiment.
    • Meng et al., Locating and Editing Factual Associations in GPT, NeurIPS 2022, and Gekhman et al., Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?, EMNLP 2024. The first traces where facts are recalled and edits them with rank-one updates, relevant to the block-restoration experiment, and the second shows facts new to the model behave differently in fine-tuning, which supports the paper's main distinction.
Questions for authors
  • How do gate-only and scalar-mixing-plus-correction variants do on the scaling ladder?
  • What learning rate and regularization did the row-block aTLAS use?
  • Which specialists, in what order, form the ladder?
  • What do the syntax and function-name probes ask, and what does the checker test?

Minor Issues to fix

  • The average improvement in the abstract is in percentage points, not percent.
  • The calibration pool size reads as growing with the number of specialists, which I think is a typo.
  • Section 3 says baselines are tuned on validation, while the appendix says each cell uses its own best coefficient.
  • For some pairs, the per-specialist values in the imbalance figure don't average to the aggregate in the main table.
  • The initialization of the low-rank factors isn't stated.
  • Typos: "that replace", "MERGEBENCHalong", stray ",." after the retention equation.
4 comments

We thank reviewer R4cN for their insightful comments that have helped make this paper stronger overall.

W1 / Q1. Gate vs. correction. “The gate and the low-rank correction are never ablated separately. […] How do gate-only and scalar-mixing-plus-correction variants do on the scaling ladder?”

Thank you for this insightful question! We acknowledge that this ablation was missing, and have now added it to the paper. We find that both components help the performance of LUMEN, and that ablating either of them decreases performance.

We ablated the mixing and the correction components of LUMEN separately, giving the following variants:

  • LUMEN (gate + correction)
  • Gate only (no correction)
  • Correction only (Task Arithmetic + correction)
  • Scalar mixing + correction
  • aTLAS (per-tensor scalar mixing, no correction)
  • LoRA on base (rank 2, no merge)

We ran these ablations on both our factual retention benchmark and MergeBench.

On the factual retention ladder at N=6, the gate only retains 66.7%. Correction only, the rank-one LoRA on the calibration facts that the reviewer describes, retains 21.3%, and a LoRA on the base model (rank 2, 4.3M parameters) retains 18.6%. Scalar mixing + correction, the second variant the reviewer asked for, retains 68.6%. LUMEN retains 71.1%, leading scalar mixing + correction by 1.1–2.5 points from N=4 on.

MethodN=2N=3N=4N=5N=6
aTLAS98.786.156.846.833.6
Correction only (on Task Arithmetic)100.880.647.827.821.3
LoRA on base17.519.318.420.618.6
Gate only101.996.384.375.766.7
Scalar mixing + correction102.898.287.1 ± 0.779.368.6 ± 0.7
LUMEN103.1 ± 0.598.4 ± 0.488.2 ± 0.381.5 ± 0.571.1 ± 0.3

On MergeBench, the gate alone gives the highest GSM8K score (56.6), and adding the correction to the gate (LUMEN) raises safety from 45.0 to 67.6, within one std of scalar mixing + correction. LUMEN also has the highest instruction-following score (32.4), giving it the highest MergeBench average (46.9).

MethodInstrGSM8KMultiCodeSafetyAvgAyaSYNTAXFUNCNAME
aTLAS (scalar only)20.336.344.934.035.834.363.077.758.4
Correction only27.947.144.838.458.343.377.848.188.8
Gate only29.856.646.538.645.043.388.996.081.4
Scalars + correction30.9 ± 2.649.4 ± 5.945.7 ± 0.236.9 ± 2.267.8 ± 2.646.1 ± 1.890.4 ± 5.389.6 ± 8.589.6 ± 2.2
LUMEN32.4 ± 2.350.5 ± 5.546.0 ± 0.238.2 ± 1.867.6 ± 2.546.9 ± 1.795.4 ± 3.387.9 ± 9.790.9 ± 1.3

So the gain comes from learning the mixture together with the correction, not from fine-tuning a fixed merge. With the same correction at N=6, a fixed merge retains 21.3%, learned scalars 68.6%, and learned gates 71.1%. The gate's edge over scalars grows with N, to 1.1–2.5 points from N=4 on.

W2 / Q2. Tuning of row-block aTLAS. “The row-block version of aTLAS can represent plain aTLAS, yet it does far worse […] Its learning rate isn’t reported.”

The reviewer is right: the aTLAS×K result was a tuning artifact. Retuned, it rises from 56.5% to 90.9% at N=2 and becomes the strongest baseline from N=4 onward, but LUMEN still leads it by 28–33 points at N ≥ 4.

The submitted aTLAS×K used lr 0.01, AdamW with weight decay 0.1, a constant schedule and zero-initialized coefficients. We suspect the weight decay pulled all 948K coefficients toward 0, keeping the merge close to the base model. Sweeping (lr, wd) with a cosine schedule at N=2 and N=4, we found (0.03, 0) best at both and use it at every N. We also retuned plain aTLAS: the selection facts pick an lr of 0.03–0.3 depending on N, and every table reports aTLAS at its selected lr.

aTLAS×Klr 0.003lr 0.01lr 0.03lr 0.1lr 0.3
N=2, weight decay 0.120.756.575.881.179.9
N=2, weight decay 021.370.190.976.2—
N=4, weight decay 0.119.737.050.953.549.2
N=4, weight decay 020.341.360.232.6—
aTLASlr 0.01lr 0.03lr 0.1lr 0.3
N=298.098.698.798.3
N=380.083.986.185.7
N=445.956.855.854.9
N=537.937.942.446.8
N=630.131.733.633.6
  • Retuning fixes the artifact. Without weight decay and at a higher learning rate, aTLAS×K rises from 56.5% to 90.9% at N=2, now well above Task Arithmetic (67.0%).
  • More parameters help, but don't close the gap. aTLAS×K overtakes aTLAS from N=4 on, yet still trails LUMEN by 28–33 points.

W3. A learned MergeBench baseline. “On MergeBench, LUMEN is trained on the calibration split but only compared with training-free merges. […] A learned baseline on the same data is needed.”

We agree with the reviewer’s comment, and have thus added two trainable baselines (aTLAS and a LoRA on the base model) as well as four other data-free baselines (TSV-M, Iso-C, Iso-CTS and WUDI) to the MergeBench evaluation. We find that both trainable baselines score the lowest averages in the table (aTLAS 34.3, LoRA 33.3), below every training-free merge.

(1) Trainable baselines on the same data. To address the first point, that LUMEN is trained on the calibration split but compared only with training-free merges, we have added two trainable baselines: aTLAS and a rank-4 LoRA on the base model, trained on all five MergeBench tasks at once. Both use the same calibration rows per task and LUMEN’s recipe.

InstrGSM8KMultiCodeSafetyAvgAya keptSYNTAXFUNCNAMEJSONP.S.
aTLAS (200 steps)20.336.344.934.035.834.363.077.758.485.582.6
LoRA on base9.137.947.332.240.233.361.1100.980.191.830.4
LUMEN32.4 ± 2.350.5 ± 5.546.0 ± 0.238.2 ± 1.867.6 ± 2.546.9 ± 1.795.4 ± 3.387.9 ± 9.790.9 ± 1.397.488.0

Moreover, on the reviewer's second point: we agree that the submitted MergeBench average gained mostly from safety. Retrained with the new recipe, LUMEN is the best merge on both instruction following (32.4) and safety (67.6), has the highest average (46.9 ± 1.7), and is within 4.2, 2.1 and 2.3 points of the best merge on GSM8K, multilingual and code.

W4. Base-relative retention. “Retention isn’t measured relative to the base model. […]”

On every fact dataset the base model scores 0%, so base-relative retention is identical to the paper’s metric and the ladder results are unchanged. On Aya, we acknowledge that the reviewer is correct: the base already answers 41 of the specialist’s 108 correct items. Not subtracting the base was a design choice made to avoid harder-to-interpret negative numbers in the results tables. Subtracting the base keeps the relative ordering of different methods, and lowers the baselines far more than LUMEN.

AyaSoupTARegMeanTIESLUMEN
Retention (submitted)63.968.566.779.692.6
Base-subtracted38.146.042.965.187.3

Subtracting the base hurts the baselines most: Task Arithmetic loses 22.5 points and RegMean 23.8, while LUMEN loses only 5.3. On the 67 items that only the specialist knows, LUMEN still retains the most (72%, vs. 69% for TIES), though at this sample size the gap is within noise. For this reason our primary evidence remains the fact datasets: about 1,000 facts per specialist, where the base scores 0% and LUMEN's lead is large.

W5 / Q3. Seeds, test size, orderings, synthetic set. “Everything uses one seed, a small test set per specialist, and a single ordering of six specialists that is never disclosed […] see LUMEN on the synthetic dataset where merging fails worst.”

LUMEN’s results are stable across seeds and test sets. Over three seeds the standard deviation is at most 0.5 points, and on the full held-out set the 95% confidence intervals are about ±1.4 points.

Test size. For the factual benchmark, we now evaluate on the full held-out set: about 1,000 facts per specialist instead of 125. At this size, each retention score is accurate to about ±1.4 points (at most ±2.1; 95% confidence interval). We can see that the std is in fact far smaller than LUMEN’s 22–33-point lead at N ≥ 4.

Seeds. All LUMEN results (except on the new ladder orderings), on our factual benchmark and MergeBench, are now averaged over three seeds. On the ladder, LUMEN varies by at most 0.5 points across seeds, while on MergeBench we observe an std of 1.7 on the average score.

The ladder adds specialists in this order: NIH RePORTER, Wikidata, OpenAlex, ClinicalTrials, UniProt, Caselaw. The order was randomly chosen before any ladder method was run. We also tested two more orderings (selected methods shown below):

Ordering B (OpenAlex, Caselaw, NIH RePORTER, UniProt, Open Library, Wikidata):

MethodN=2N=3N=4N=5N=6
Task Arithmetic61.726.48.35.24.4
RegMean64.931.117.913.29.8
Iso-C75.061.152.141.430.6
Iso-CTS72.059.149.042.932.2
aTLAS92.974.857.644.234.3
aTLAS×K66.565.651.650.741.9
LUMEN100.294.488.279.570.0

Ordering C (ClinicalTrials, Wikidata, UniProt, NIH RePORTER, ClinicalTrials-PI, Caselaw):

MethodN=2N=3N=4N=5N=6
Task Arithmetic72.525.110.17.24.3
RegMean64.533.418.914.311.4
Iso-C88.474.653.930.823.6
Iso-CTS76.570.952.333.525.0
aTLAS96.479.256.233.926.1
aTLAS×K85.171.062.743.834.6
LUMEN101.895.587.765.856.9

Absolute retention does depend on which specialists are merged (ordering C is the hardest, 56.9% at N=6), but LUMEN leads the best prior method at every merge size in all three orderings, by 22–33 points at N ≥ 4.

Moreover, on the synthetic set, all methods, including LUMEN, perform poorly. All learned methods sit at 11–12% (LUMEN 12.2, aTLAS 11.7); data-free methods at 7% or less.

retention
Iso-CTS0.0
TSV-M0.1
task arithmetic4.3
WUDI7.0
aTLAS11.7
LUMEN12.2

Q4. SYNTAX and FUNCNAME probes. “What do the syntax and function-name probes ask, and what does the checker test?”

A4: We take 400 coding instructions from the coding specialist’s training data (Magicoder-Evol-Instruct), none used for calibration, and decode greedily. SYNTAX: the generated Python parses. FUNCNAME: it defines at least one function whose name also appears in the taught solution. No code is executed; both are reported relative to the coding specialist.

W6 and minor issues. “The related work misses several close papers […]”

Thank you for the pointers. The revised paper(uploaded soon) discusses all five: Representation Surgery (Yang et al., 2024), Localizing Task Information (Wang et al., 2024), sparse adapters for merging experts (Arnob et al., 2025), locating and editing factual associations (Meng et al., 2022), and fine-tuning on new knowledge (Gekhman et al., 2024). Unlike Representation Surgery, which keeps a separate module per task, LUMEN folds everything into a single checkpoint.

Further, in the new draft, all minor issues are fixed: percentage points in the abstract, the calibration pool size, a consistent selection protocol, per-specialist values that match Table 2, the initialization of the low-rank factors, and typos.

Dear Reviewer R4cN,

Thank you again for your thoughtful feedback. We have carefully considered your comments and incorporated substantial revisions into the updated manuscript (uploaded now). We believe our responses and revisions address your main concerns.

As the discussion period is coming to a close tomorrow, we would greatly appreciate it if you could take another look at our responses and updated manuscript and let us know if you have any remaining concerns or questions.

Thank you again for your time, feedback, and consideration of our manuscript.

Sincerely,

The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Improved training recipe

At submission, LUMEN's gate was trained at lr 1e-3 and barely moved (logit std 0.0007), so it stayed near uniform; the submitted gains came almost entirely from the scalar coefficients and the correction. We now train with lr 3e-2, weight decay 0 and a cosine schedule over 1,200 steps, and report the final step with no checkpoint selection (the submission selected the step on held-out facts). The learned baselines (aTLAS, aTLAS×K) use the same recipe.

On the factual benchmark, every method is now scored on the full held-out test set (about 1,000 facts per specialist), with all hyperparameters chosen on a separate validation set (125 facts per specialist) that never enters a reported number. LUMEN results are averaged over 3 seeds. On MergeBench, we now use GSM8K alone for math, because MATH is at most 1.7% for every model, the math specialist included, so it cannot separate merges. Under the paper's original definition, LUMEN scores 42.0 (mean of three seeds), still above every baseline.

Trained this way, the gate alone recovers most of the gain (66.7% of LUMEN's 71.1% at N=6). Adding the correction to the gate, which gives the full LUMEN, yields the best result at every merge size and lifts the MergeBench average from 43.3 to 46.9 (see the component ablation above). Together, these changes explain why the numbers differ from the submitted tables.

AuthorsOpenReview ↗

New Experiments: LUMEN outperforms all compared merging methods on FusionBench

We have now added new comparisons highlighting LUMEN’s performance in additional, and rather unanticipated, settings:

LUMEN transfers to vision: on FusionBench, LUMEN outperforms every method we ran. LUMEN was designed for factual retention in language models. We tested whether its results transfer to a different modality: CLIP-ViT-B/32 image classification on FusionBench’s 8-task and 20-task evaluation suites. LUMEN uses at most 5% of each task’s labels (three seeds). We find that LUMEN achieves an average accuracy of 86.8% on the 8 task suite, with the next best prior at 84.8% (WUDI), and a fine-tuned specialist ceiling of 90.4%. However, our most impressive result is observed on the 20 tasks suite where LUMEN achieves an 84.4% average accuracy compared to the next best prior at 73.5% (Iso-CTS), and a fine-tuned specialist ceiling of 89.8%. This result highlights both effective generalization across modalities and LUMEN’s ability to maintain strong performance as the number of merged tasks increases.

Average Test Accuracy (%)8 tasks20 tasks
Fine-tuned specialists (ceiling)90.489.8
Best data-free merge84.8 (WUDI)73.5 (Iso-CTS)
Test-time adaptation (AdaMerging)82.6 (AdaMerging)69.1 (AdaMerging)
LUMEN (≤5% labels)86.8 ± 0.184.4 ± 0.2

Moreover, we provide the complete, per-task table for 8 and 20 task FusionBench (Bold = best merge per columns; test accuracy [%] ):

  • LUMEN beats every method we evaluate, and its lead grows with the number of tasks: +2.0 points over the next best at 8 tasks, and +10.9 at 20.
  • At 20 tasks, LUMEN keeps 94% of the fine-tuned accuracy, and it beats test-time adaptation without seeing a single test image.

8 tasks

SUN397CarsRESISCEuroSATSVHNGTSRBMNISTDTDAvg
fine-tuned (ceiling)74.978.595.199.197.398.999.679.790.4
task arithmetic57.055.764.773.477.968.596.147.167.6
TIES67.064.274.374.577.769.494.154.071.9
MagMax66.760.172.275.281.270.293.146.870.7
TSV-M67.671.684.793.491.992.598.963.883.1
Iso-C71.773.584.888.078.784.696.765.280.4
Iso-CTS71.673.684.187.173.080.995.465.178.8
WUDI68.972.785.195.094.995.199.367.284.8
AdaMerging*67.871.284.192.387.593.098.266.982.6
LUMEN70.972.891.296.996.096.598.871.486.8 ± 0.1

20 tasks

SUN397CarsRESISCEuroSATSVHNGTSRBMNISTDTDFlowersPCamFERPetsSTL10C100C10FoodF-MNISTEMNISTKMNISTSST2Avg
fine-tuned (ceiling)74.978.595.199.197.398.999.679.788.688.071.692.497.588.497.688.494.795.698.271.389.8
task arithmetic20.412.225.625.630.929.878.022.321.153.234.342.471.029.564.115.167.017.015.451.236.3
TIES51.036.247.845.158.157.792.140.644.866.947.373.289.951.386.350.176.521.019.755.955.6
TSV-M64.353.274.087.081.984.997.958.461.879.463.884.996.067.693.373.484.738.749.170.373.2
Iso-C65.543.870.877.477.182.297.256.159.777.162.182.495.370.893.869.683.036.437.169.670.3
Iso-CTS68.059.276.981.873.980.896.558.572.783.564.686.396.973.995.076.883.235.337.269.873.5
WUDI55.144.859.378.479.782.998.150.349.382.058.577.793.459.990.353.183.935.440.069.067.1
AdaMerging*66.467.279.992.379.187.091.059.772.753.265.485.296.369.790.879.770.215.810.050.669.1
LUMEN70.571.289.895.993.494.097.769.065.584.767.888.096.478.994.785.090.591.591.471.484.4 ± 0.2

We thank Reviewers R4cN, wgfF, and c8wv for their careful and constructive feedback, which has helped us substantially strengthen the paper.

A central question raised by all three reviewers is whether LUMEN's improvements come from more effective specialist mixing or simply from fine-tuning after merging. Our new component ablations show that the gain comes from learning the mixture together with the correction, not from fine-tuning a fixed merge. With the same correction at N=6, a fixed merge retains only 21.3%, learned scalars 68.6%, and learned gates (LUMEN) 71.1%. Both components matter, and only their combination gives the best result.

In particular, when merging six specialists, the gate alone achieves a retention score of 66.7, compared to 21.3 for correction alone and 18.6 for LoRA fine-tuning of the base model. Adding the correction to the gate (LUMEN) further improves retention to 71.1, outperforming scalar mixing with the same correction mechanism (68.6). These results demonstrate that the improvements cannot be attributed to post-merge fine-tuning alone and highlight the benefit of learned specialist mixing.

To systematically isolate these contributions, we evaluated LUMEN's gating and correction components under the same data and training recipe.

We ablated the mixing and the correction components of LUMEN separately, leading to the methods:

  • LUMEN
  • Gate only
  • Task Arithmetic + correction: Correction only
  • Scalar mixing + correction
  • Scalar mixing per-tensor, no correction: aTLAS
  • No mixing, LoRA correction: rank 2 LoRA on base model

Table 1: component ablation (Eq. 6, original ladder; seed 1, except ± cells: mean ± sd over three seeds; aTLAS at its selected learning rate)

N=2N=3N=4N=5N=6
Scalar mixing only (aTLAS)98.786.156.846.833.6
correction only100.880.647.827.821.3
LoRA on the base, 4.3M params17.519.318.420.618.6
gate only101.996.384.375.766.7
scalar mixing + correction102.898.287.1 ± 0.779.368.6 ± 0.7
LUMEN103.1 ± 0.598.4 ± 0.488.2 ± 0.381.5 ± 0.571.1 ± 0.3

The ablation also reveals that the benefits of learned gating become more pronounced as the number of specialists increases. LUMEN matches scalar mixing with correction at N=2–3 and leads it by 1.1–2.5 points from N=4 on. This suggests that adaptive specialist mixing becomes increasingly important as the merging task becomes more challenging.

Beyond these ablations, we have expanded our experimental evaluation with new results across different modalities, where LUMEN outperforms every method we ran, further demonstrating LUMEN's robustness. These results, together with the clarified analysis, will be included in the revised manuscript (uploaded soon).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.