- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether model merging preserves the factual information acquired during specialist fine-tuning, arguing that standard task benchmarks can obscure substantial losses in factual recall. The authors introduce ten factual datasets and evaluate retention through the merged model’s accuracy normalized by each specialist’s pre-merge accuracy. Their experiments show substantial recall degradation, worsening retention as more specialists are merged, and asymmetric losses across specialists.
The paper also proposes LUMEN, which combines low-rank-parameterized, coordinate-wise mixing of specialist updates with a learned low-rank residual correction. Both components are optimized using a small calibration subset and materialized into a single checkpoint. LUMEN substantially improves normalized factual accuracy over the evaluated baselines, particularly for larger merges, and achieves the highest aggregate MergeBench score among the methods evaluated in the paper.
Critical strengths
-
The paper identifies a practically meaningful evaluation gap. Distinguishing broad task performance from recall of specialist-taught information is useful for assessing whether merging preserves the value of specialization.
-
LUMEN achieves substantial empirical improvements without increasing inference cost. The gains over the strongest reported baseline become large at higher merge sizes: at six specialists, normalized factual accuracy increases from 30.5 to 60.1.
Critical weaknesses
- The experiments do not isolate the contribution of the proposed merging mechanism from residual fine-tuning. LUMEN introduces both coordinate-wise gates and an additive low-rank correction, but the paper does not report gate-only, correction-only, or scalar-mixing-plus-correction ablations.
2.The aggregation of per-specialist retention in Figure 4/Table 5 and Table 2 needs clarification. For Open Library + ClinicalTrials, the displayed values average to 69.2%, whereas Table 2 reports 70.7%; for OpenAlex + ClinicalTrials-PI, they average to 56.6%, versus 58.0%. Please specify whether Table 2 uses the macro-average defined in Eq. (6) or another aggregation rule, and reconcile these values.
- section 3.2 describes validation-based hyperparameter selection, whereas section 5.1 explicitly reports per-cell oracle choices for dataless baselines and best-of-two learning-rate results for aTLAS. appendix A.4 does not identify the split used to maximize retention. Please specify the selection protocol for each experiment/table and clarify which choices use reporting/test performance.
Questions for authors
-
Component attribution. Can you provide results for gates without correction, scalar mixing with the same correction, and correction-only adaptation of a conventional merge?
-
Please specify LUMEN’s training objective: is it supervised cross-entropy on gold answers, distillation from the specialists, or another loss? If cross-entropy is used, is it computed only over answer tokens or over the full sequence? For the N-way experiments, how many candidate checkpoints are evaluated using the 125 selection probes per specialist, and are these probes also used for hyperparameter selection?
-
Which specialists constitute the nested sequence in Table 3, and how were they selected from the eight specialists mentioned in Section 5.1?
6 comments
W3. Selection protocol. “Please specify the selection protocol for each experiment/table and clarify which choices use reporting/test performance.”
We agree the submission mixed two protocols: the data-free baselines and aTLAS's learning rate were chosen per cell on the reporting probes, and LUMEN's checkpoint on the 125 selection probes. In the revised paper, no choice uses test data, and every learned method is evaluated at its final step.
| Benchmark | What is tuned | Chosen on |
|---|---|---|
| Factual ladder | Baselines' coefficients and learning rates (aTLAS per N); LUMEN's gate learning rate, once at N=4 | 125 selection facts per specialist, excluded from every reported number |
| MergeBench | Every method's coefficients and learning rates | Held-out loss on 200 calibration rows per task |
| FusionBench | LUMEN's correction learning rate and steps; baselines at FusionBench defaults | A held-out part of LUMEN's ≤ 5% training slice |
| Pairwise merges (Table 2) | Data-free baselines' merging coefficient | Each baseline's best coefficient per pair on the reporting probes, as in the submission. Table 2 contains only data-free baselines, so each is shown at its strongest and the loss we measure is conservative. |
aTLAS is tuned at every N while LUMEN is tuned once, which favours the baseline.
Q2. Training objective and selection probes. “Please specify LUMEN’s training objective […] how many candidate checkpoints are evaluated using the 125 selection probes per specialist, and are these probes also used for hyperparameter selection?”*
LUMEN minimizes cross-entropy on the gold answer tokens only. The loss covers the tokens after
<ans>; the prompt tokens are masked.
For each calibration fact entity > relation > <ans> value, the loss is the cross-entropy of value given everything up to <ans>, averaged over the answer tokens. The targets are the gold answers, not the specialists’ outputs, so there is no distillation. On MergeBench, the loss is likewise computed only on response tokens. The gates, the correction and the scalar coefficients are trained jointly on this single loss. We have added this to Section 4 of the revised paper (uploaded soon).
In the submission, LUMEN’s step was chosen among 6 evaluations (every 200 steps) on the 125 selection probes. Now no checkpoint is chosen: every learned method is evaluated at its final step. Yes, the same selection facts are used for hyperparameter selection, and they never enter a reported number: data-free baselines compare 4–16 settings on them, aTLAS four or five learning rates per merge size, and aTLAS×K nine (lr, weight decay) settings at N=2 and N=4, reused at every other N.
Q3. The ladder’s specialists. “Which specialists constitute the nested sequence in Table 3, and how were they selected from the eight specialists mentioned in Section 5.1?”
The ladder adds specialists in this order: NIH RePORTER, Wikidata, OpenAlex, ClinicalTrials, UniProt, Caselaw. The order was randomly selected before any ladder method was run. The other two (Open Library, ClinicalTrials-PI) enter in the B and C orderings below. Further, we also tested two more orderings, and found that LUMEN’s lead was consistent across all orderings we tested.
Across all 3 orderings, LUMEN leads the best prior method at every merge size, by 22–33 points at N ≥ 4. The 2 ordering results (selected methods) are shown below: Ordering B (OpenAlex, Caselaw, NIH RePORTER, UniProt, Open Library, Wikidata):
| Method | N=2 | N=3 | N=4 | N=5 | N=6 |
|---|---|---|---|---|---|
| Task Arithmetic | 61.7 | 26.4 | 8.3 | 5.2 | 4.4 |
| RegMean | 64.9 | 31.1 | 17.9 | 13.2 | 9.8 |
| Iso-C | 75.0 | 61.1 | 52.1 | 41.4 | 30.6 |
| Iso-CTS | 72.0 | 59.1 | 49.0 | 42.9 | 32.2 |
| aTLAS | 92.9 | 74.8 | 57.6 | 44.2 | 34.3 |
| aTLAS×K | 66.5 | 65.6 | 51.6 | 50.7 | 41.9 |
| LUMEN | 100.2 | 94.4 | 88.2 | 79.5 | 70.0 |
Ordering C (ClinicalTrials, Wikidata, UniProt, NIH RePORTER, ClinicalTrials-PI, Caselaw):
| Method | N=2 | N=3 | N=4 | N=5 | N=6 |
|---|---|---|---|---|---|
| Task Arithmetic | 72.5 | 25.1 | 10.1 | 7.2 | 4.3 |
| RegMean | 64.5 | 33.4 | 18.9 | 14.3 | 11.4 |
| Iso-C | 88.4 | 74.6 | 53.9 | 30.8 | 23.6 |
| Iso-CTS | 76.5 | 70.9 | 52.3 | 33.5 | 25.0 |
| aTLAS | 96.4 | 79.2 | 56.2 | 33.9 | 26.1 |
| aTLAS×K | 85.1 | 71.0 | 62.7 | 43.8 | 34.6 |
| LUMEN | 101.8 | 95.5 | 87.7 | 65.8 | 56.9 |
Absolute retention does depend on which specialists are merged (ordering C is the hardest, 56.9% at N=6), but LUMEN leads the best prior method at every merge size in all three orderings, by 22–33 points at N ≥ 4.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. However, a few follow-up questions remain.
-
Why are the scores in Table W1 significantly higher than those in Table 3 of the original paper? (For example, when N=6, the score for LUMEN increases from 60.1 to 71.1, and for aTLAS from 26.8 to 33.6.) Where does this improvement come from?
-
There are "MATH" and "SYNTAX" columns in Table 4 of the original paper. Why are there not these two columns in the Rebuttal W1 table?
We thank reviewer c8wv for a careful and precise review! The reviewer raises 3 concerns. First, whether LUMEN's gain comes from its merging mechanism or from the correction. Second, why the per-specialist values in Figure 4 / Table 5 do not average to the aggregates in Table 2. Third, which hyperparameter and checkpoint choices used the reporting data. The reviewer also asks for LUMEN's training objective, how the selection probes are used, and how the ladder's specialists were chosen. All questions have helped improve the precision and strength of the paper, and have been answered below.
W1 / Q1. Component attribution. “The experiments do not isolate the contribution of the proposed merging mechanism from residual fine-tuning. […]”
On our factual retention ladder, when merging N=6 specialists, the gate-only variant (without correction) achieves 66.7% retention. Scalar mixing + correction improves retention to 68.6%, but still falls short of the full LUMEN method, which achieves the highest retention of 71.1%.
Correction only (LUMEN’s rank-1 correction trained on top of task arithmetic, with the merge coefficients frozen at task arithmetic’s tuned α) keeps 100.8% at N=2 but falls to 21.3% at N=6, below even scalar mixing (aTLAS, 33.6%). A correction on top of a fixed merge cannot recover what the merge lost once several specialists are merged, so LUMEN’s gain does not come from fine-tuning a fixed merge.
Retention (%), macro-averaged as in Eq. 6. ± is the standard deviation over 3 seeds, where shown; other cells seed 1. aTLAS at its selected learning rate.
On MergeBench, the components do different jobs: the gate alone gives the best GSM8K score (56.6), and the gate with the correction (LUMEN) raises safety from 45.0 to 67.6. Together, LUMEN has the highest average (46.9 ± 1.7, vs 46.1 ± 1.8 for scalar mixing with the same correction). On MergeBench, correction only reaches an average of 43.3 against LUMEN’s 46.9. It keeps less safety (58.3 vs 67.6) and fewer Aya facts (77.8% vs 95.4%).
So the gain comes from learning the mixture together with the correction, not from fine-tuning a fixed merge. With the same correction at N=6, a fixed merge retains 21.3%, learned scalars 68.6%, and learned gates 71.1%. The gate's edge over scalars grows with N, to 1.1–2.5 points from N=4 on.
W2. Aggregation in Table 2 vs. Figure 4 / Table 5
Open Library + ClinicalTrials has a second cause: its per-specialist values in Figure 4 / Table 5 (62.9 / 75.4) were copied from Task Arithmetic’s training run with hyperparameter λ = 0.45, not the selected λ = 0.55. The correct values are 63.8 / 77.1. The aggregation rule changes five of the six cross-dataset rows, not only the two that were identified:
OpenAlex + ClinicalTrials is unchanged (74.3). Every change is under 1.5 points, and none affects a conclusion. On the new full test sets, the two aggregation rules differ by at most 0.05 points.