acceptodds
Under review as a conference paper at ICLR 2027

MOSAIC: Continual LoRA Merging via Hypernetwork Structure

12 views

Abstract

Hypernetworks offer an efficient alternative to LoRA fine-tuning by directly generating adapters from conditioning inputs, such as document chunks or task descriptions. However, in streaming settings, retaining every generated LoRAs increases storage and leaves information distributed across separate LoRAs. Existing weight-space merging approaches consolidate these adapters without exploiting their shared hypernetwork, whose decoder relates generating coordinates to adapter behavior. To address this, we propose MOSAIC, a continual merging method that uses the hypernetwork structure to merge generated LoRAs in hypernetwork coordinates. MOSAIC measures adapter differences through their layer-output contributions on frozen target-model activation probes and pulls the resulting activation-induced metric back to hypernetwork coordinates through the frozen decoder's Jacobian, yielding a matrix-weighted barycenter formulation with additive sufficient statistics. This formulation enables one-pass updates and order-invariant accumulation with memory independent of stream length, producing a single deployable LoRA after each arrival. MOSAIC further applies covariance shrinkage to stabilize the accumulated functional metric. Experiments across multiple hypernetwork-based LoRA generation families and target LLMs demonstrate the effectiveness of MOSAIC in continually merging generated LoRAs.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, continual model merging, hypernetworks, low-rank adaptation, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mtDAOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

MOSAIC continually merges hypernetwork-generated LoRAs in their shared coordinate space. It approximates functional discrepancies with a quadratic objective (Q_t), differentiates it, and solves the resulting linear system for the minimum-error coordinate. After stabilizing the system with covariance shrinkage, it decodes the merged coordinate into a single fixed-rank LoRA. MOSAIC achieves the best document-merging scores across all three tested chunk sizes on Qwen3-4B and the highest final task averages on both Mistral-7B and Qwen3-8B, although some gains over strong baselines are small.

Critical strengths
  1. The mathematical treatment is complete and coherent, with no obvious errors in the main derivation. The reduction to a matrix-weighted quadratic problem is elegant.

  2. The paper is clearly written, and the connection between the objective, derivation, and algorithm is easy to follow.

  3. The method is practical for streaming updates: storage does not grow with the number of arrivals, and deployment requires only one fixed-rank LoRA.

Critical weaknesses
  1. The objective treats each layer separately and keeps its input activations fixed. In the deployed model, however, changes in earlier layers alter the inputs to later layers. Matching LoRA contributions on frozen activations therefore misses this interaction. How much does the layerwise objective reflect the behavior of the full merged model?

  2. The paper bounds the linearization error but does not measure it directly. A useful control would optimize the original accumulated objective in Eq. 7, without the Jacobian approximation, either over hypernetwork coordinates or directly over fixed-rank LoRA factors. How far is MOSAIC’s solution from the directly optimized solution in both functional error and downstream performance? The paper should report this gap alongside the optimization cost.

Questions for authors

No

6 comments

Thank you for your careful review and for the thoughtful questions about MOSAIC's objective and its approximation. We conducted additional measurements to address them and respond to each point below, with the supporting tables at the end.

Q1. How much does the layerwise objective reflect the behavior of the full merged model?

A1. Thank you for raising this point. MOSAIC’s frozen-input objective measures how closely the merged LoRA reproduces each source LoRA’s contribution to a projection’s output when both receive the same inputs from the frozen base model. During a full forward pass, upstream adapters can change the inputs to later projections. We measured the size of this drift and tested how well the frozen-input discrepancy tracks changes in full-model predictions.

Frozen versus deployed inputs. In the document-conditioned setting on Qwen3-4B at 1024 tokens, we compared the inputs to down_proj in the frozen base model and in MOSAIC with all merged LoRAs active. We used identical content-token positions held out from fitting the merge statistics. For each document, we measured the ℓ2\ell_2 difference relative to the norm of the frozen base inputs. Across QASPER, 2WikiMultihopQA, and MultiFieldQA, the median drift ranged from 30.67% to 35.44% at layer 12, 45.39% to 49.38% at layer 24, and 43.29% to 50.18% at layer 35. The largest median observed is 99.27% at layer 16 on 2WikiMultihopQA. These measurements show that the layer inputs used by the frozen-input objective differ from those in the deployed model by roughly a third to a half in relative norm at the layers we report (Table R1).

Propagation after a single-layer swap. The drift above compares the base model with all adapters active. To isolate how a change at one layer propagates, we next swapped a single layer's LoRA. At 512, 1024, and 2048 tokens on the same three datasets, we replaced the merged LoRA at layer XX with one source chunk’s LoRA and kept all other layers fixed. We measured the change at down_proj in a later layer YY for (X,Y)=(0,12),(12,24),(24,35)(X,Y)=(0,12),(12,24),(24,35). Relative to the unmodified MOSAIC forward pass, the median relative ℓ2\ell_2 changes ranged from 2.55% to 6.87% at the downstream input and from 2.31% to 5.37% at its output.

Alignment with next-token predictions. For each swap, ΔF\Delta F is the after-minus-before change in the frozen-input discrepancy to the selected source chunk, summed across adapted layers. The swap removes layer XX’s contribution to this discrepancy, so ΔF≤0\Delta F\leq 0. On identical token prefixes, we compute next-token DKL(psource∥pcandidate)D_{\mathrm{KL}}(p_{\mathrm{source}}\|p_{\mathrm{candidate}}). The reference uses the source chunk’s LoRA at every adapted layer, and the candidate is MOSAIC before or after the swap. A negative ΔKL\Delta\mathrm{KL} means that the swap reduced this divergence.

At matched chunk sizes, layer 24 swaps produced median absolute KL changes 13.5–30.5 times as large as those from layer 0 swaps. Spearman correlations between ΔF\Delta F and ΔKL\Delta\mathrm{KL} are 0.63–0.68 at layer 24 and 0.24–0.31 at layer 12. At layer 0, the correlation intervals included zero at all three chunk sizes (Table R2). KL decreases in about half of the non-tied layer 0 swaps and in over 97% of the non-tied layer 24 swaps (Table R3). A local improvement on frozen inputs can propagate through later layers, which determines its effect on the next-token distribution. The relationship is weak at the earliest layer tested and much stronger at the deepest layer tested, where the swaps also produced the largest KL changes.

Downstream QA. In the functional weighting ablation, Full weighting improves average normalized ROUGE-L F1 over Identity by 3.10, 0.15, and 2.63 points at 512, 1024, and 2048 tokens, respectively (Table R4).

In summary, the frozen-input objective is a projection-level criterion and does not model cross-layer propagation. It is most predictive at the depth where swaps affect next-token predictions most. It should therefore be read as a tractable proxy, and the evidence that it yields good merges is the downstream performance in Section 6 in the manuscript. We will state this in the paper and add Tables R1–R4.

Q2. How large is the linearization error? How does MOSAIC compare with direct optimization of the original objective in functional error, downstream performance, and cost?

A2 We first measure the linearization error, and then we compare MOSAIC with direct Adam optimization of the original frozen-probe objective FtF_t over the hypernetwork coordinates on Mistral-7B across ten tasks and five task orders in task-conditioned merging.

Direct measurement of linearization error. At the final MOSAIC latent, we compared the exact decoder-induced change with its first-order prediction around each source task latent, using the same frozen probes for both. Across all five orders, the exact objective is 4190.98 per probe token, and the surrogate without OAS shrinkage is 2822.63 per probe token. The quadratic underestimated the exact objective by 1368.35 per probe token (32.65%), and the relative Taylor residual is 0.79. To measure the consequence for the merged adapter, we compare MOSAIC's solution with direct optimization of FtF_t below. (Table R5)

Functional error and downstream performance. For direct optimization, at each optimized prefix, Adam ran for 1000 updates per initialization at a learning rate of 0.03. We initialized from the mean of the task latents seen so far and from the first task’s latent, and retained the iterate with the lowest FtF_t across both starts. Adam's final FtF_t per probe token is 17.2% lower than MOSAIC's final FtF_t. This gap reflects both the first-order surrogate and OAS shrinkage. The downstream performance difference is +0.06 ± 0.12 Macro points (Direct minus MOSAIC). Both merges are order-invariant, so the five orders yield the same final FtF_t, and this spread reflects evaluation variability. (Table R6)

Cost. With cached probes, constructing the merge at all ten prefixes takes 55.02 s for MOSAIC and 2320 s for Direct Adam on average, giving a 42.17× ratio of mean construction times at the tested budget. (Table R7)

Budget sensitivity. At 800 updates per initialization, Adam is within 0.10% of its 1000-update objective at every prefix and within 0.02% at the final prefix, so a longer run would not change the 17.2% gap materially. At 250 updates, it is already within 1.11%, so the cost gap would remain large under a much smaller budget. The 42× ratio is measured on the timed 1000-update runs. (Table R8)

In summary, MOSAIC's first-order surrogate with shrinkage costs about 17% in FtF_t relative to direct optimization, with no measurable downstream difference and a 42× lower construction time.

Supporting tables

Table R1. Median relative down_proj input drift from the frozen base model to fully adapted MOSAIC at 1024 tokens. Layers 12, 24, and 35 are the downstream measurement layers in the swap experiment. Layer 16 contains the largest median drift observed on 2WikiMultihopQA.

LayerQASPER2WikiMultihopQAMultiFieldQA
00.00%0.00%0.00%
1235.44%30.67%30.77%
1655.48%99.27%52.80%
2449.38%48.24%45.39%
3544.82%50.18%43.29%

Table R2. Magnitude and rank correlation of full-model KL changes after single-layer swaps. At each chunk size, we pooled swaps from 87 unique documents across the three datasets. The 95% confidence intervals use 2000 bootstrap replicates that resample documents within each dataset and retain all swaps from each sampled document.

Chunk size (tokens)Swapped layerMedian ∣ΔKL∣\lvert\Delta\mathrm{KL}\rvert per token [95% CI]Spearman ρ(ΔF,ΔKL)\rho(\Delta F,\Delta\mathrm{KL}) [95% CI]
51200.001745 [0.001608, 0.001893]−0.035 [−0.083, 0.011]
512120.002170 [0.001980, 0.002420]0.306 [0.259, 0.352]
512240.053270 [0.048430, 0.058940]0.684 [0.636, 0.730]
102400.001279 [0.001189, 0.001460]−0.021 [−0.089, 0.042]
1024120.001641 [0.001505, 0.001864]0.298 [0.223, 0.375]
1024240.029770 [0.026070, 0.033470]0.627 [0.569, 0.678]
204800.001096 [0.000945, 0.001256]0.063 [−0.030, 0.162]
2048120.001260 [0.001075, 0.001381]0.238 [0.150, 0.328]
2048240.014810 [0.013350, 0.018660]0.626 [0.558, 0.683]

Table R3. Fraction of non-tied swaps that reduced full-model next-token KL. Each column identifies the swapped layer and the downstream layer where we measure activation changes. We pool swaps across the three datasets and exclude ties in either metric.

Chunk size (tokens)Swap 0, measure at 12Swap 12, measure at 24Swap 24, measure at 35
51250.1% (695/1387)66.4% (921/1388)98.8% (1371/1388)
102450.2% (359/715)66.3% (474/715)98.0% (701/715)
204852.0% (197/379)63.9% (242/379)97.6% (370/379)

Table R4. Downstream QA for complete merges on Qwen3-4B. Scores are macro-average normalized ROUGE-L F1 (%) across QASPER, 2WikiMultihopQA, and MultiFieldQA.

Chunk size (tokens)IdentityDiagonal without OASFull without OASFull − Identity (points)
51240.0840.6843.18+3.10
102442.3840.4242.53+0.15
204843.0942.9345.72+2.63

Table R5. Measure linearization error. Mistral-7B across 10 tasks on five orders.

OrderExact Ft/tokenRaw Qt/tokenSigned (Ft − Qt)/tokenRelative Taylor residual
04190.982822.631368.350.794819
14190.982822.631368.350.794819
24190.982822.631368.350.794819
34190.982822.631368.350.794819
44190.982822.631368.350.794819
Mean ± SD4190.98 ± 0.002822.63 ± 0.001368.35 ± 0.000.794819 ± 0.00

Table R6. MOSAIC versus direct latent Adam. Mistral-7B across 10 tasks on five orders.

OrderMOSAIC FtF_t/token ↓Direct FtF_t/token ↓MOSAIC MacroDirect MacroDirect − MOSAIC (points)
04190.983469.1367.4267.36-0.06
14190.983469.1367.3167.51+0.20
24190.983469.1367.4567.63+0.18
34190.983469.1367.5167.51-0.00
44190.983469.1367.4967.49+0.00
Mean ± SD4190.98 ± 0.003469.13 ± 0.0067.44 ± 0.0867.50 ± 0.100.06 ± 0.12

Table R7. Construction time Mistral-7B across 10 tasks on five orders.

OrderMOSAIC (s)Direct Adam (s)Direct / MOSAIC
058.242641.3645.35×
157.742173.6037.64×
252.112191.6142.06×
353.212300.5643.24×
453.812294.9842.65×
Mean55.022320.4242.17×

Table R8. Sensitivity of direct Adam to its step budget (Mistral-7B, ten tasks). For each budget, we take the best logged FtF_t across both starts and report its percentage excess over the best logged value at 1000 updates per start.

Updates per startMedian excess loss, all prefixesWorst excess loss, all prefixesWorst excess loss, prefix 10
2500.6088%1.1059%0.8214%
5000.1357%0.3449%0.2451%
7500.0248%0.1219%0.0334%
8000.0165%0.0985%0.0210%
9000.0053%0.0492%0.0068%
10000.0000%0.0000%0.0000%
Reviewer_RsS6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MOSAIC continually merges hypernetwork-generated LoRAs in the hypernetwork’s coordinate space using an activation-based metric derived through the decoder Jacobian. Its quadratic approximation enables incremental accumulation without retaining previous adapters, with experiments covering document- and task-conditioned generation.

Critical strengths

The paper presents a coherent way to exploit the shared hypernetwork for continual LoRA merging and evaluates it across multiple models and adaptation settings.

Critical weaknesses
  1. The incremental contribution needs clearer positioning. The weighted quadratic objective and additive sufficient statistics closely resemble the structure underlying Fisher-weighted averaging [1] (which is not discussed) and RegMean [2]. The main distinction therefore lies in the hypernetwork coordinate space and decoder-derived metric, whose contributions should be separated from the established merging formulation. Moreover, order invariance follows directly from the additive accumulation rule and is shared by standard running averaging and RegMean. [1] Matena M S, Raffel C. Merging models with Fisher-weighted averaging. NeurIPS, 2022. [2] Jin X, Ren X, Preotiuc-Pietro D, et al. Dataless knowledge fusion by merging weights of language models. ICLR, 2023.

  2. The benefit of the proposed functional weighting is not consistently established. In Fig. 3b, identity weighting (i.e., simple coordinate averaging) outperforms the full metric on QASPER and 2WikiMultihopQA. Without including simple coordinate averaging in the main comparisons, it remains unclear whether the gains stem primarily from merging in coordinate space or from the proposed metric.

  3. Evidence for practical scalability remains limited. Task merging is evaluated on only ten tasks, leaving long-term retention under increasingly diverse inputs unclear. Moreover, construction takes approximately 1.8 times as long as OPCM, while costs relative to simpler averaging methods are not reported, making the performance–cost trade-off difficult to assess.

Questions for authors
  1. Could the authors correct the “Jiang et al., 2023” citation for Mistral 7B? The current bibliography entry points to an unrelated paper on 6G non-terrestrial networks (arXiv:2311.09047).

  2. Please correct the extraneous ∇\nabla symbols in Appendix A.3, the missing verb in “LoCoMo results 1024 tokens” in Appendix B.4, and the missing punctuation in “Tab 12” and at the end of Appendix B.5.

4 comments

Thank you for your careful review and for suggesting clarifying MOSAIC’s contributions and strengthening its evaluation. We respond to your review below and provide the supporting tables at the end.

Q1. Clarify MOSAIC’s contributions to Fisher-weighted averaging and RegMean.

A1. We agree that weighted quadratic objectives and additive sufficient statistics are established ideas in model merging. Order invariance follows from the accumulation rule and is shared by running averaging and RegMean with equal weighting across adapters. We revised the manuscript to cite Matena and Raffel and to clarify MOSAIC’s contributions. MOSAIC merges hypernetwork-generated adapters in the shared hypernetwork’s coordinate space, weighting differences based on how strongly they affect adapter outputs on frozen activations.

The frozen-activation discrepancy that MOSAIC starts from is the same quantity RegMean minimizes over the weights. MOSAIC differs in where it solves and how it weights. RegMean and Fisher-weighted averaging solve for model weights, and their solutions are generally not adapters the hypernetwork can generate. MOSAIC solves for a single hypernetwork coordinate, so the merged adapter stays in the decoder's range at the generator's rank. MOSAIC pulls the activation-induced discrepancy back to hypernetwork coordinates through the frozen decoder's Jacobian. The resulting weighting is a full matrix over coordinates and has no counterpart in weight-space merging. We also include shrinkage of the accumulated metric, which stabilizes the solve on long streams.

In our manuscript, section 6 already compares MOSAIC with RegMean, which uses activation statistics to merge adapter updates in their weight space. MOSAIC performs the merging process in coordinates defined by the shared hypernetwork and weights discrepancies by their effect on adapter outputs on frozen activations. Here, we add a Fisher-weighted averaging baseline that uses estimated Fisher information to weight the adapter weights. Across three base models and three chunk sizes across three datasets, MOSAIC achieves higher average normalized ROUGE-L F1 than Fisher-weighted averaging in 8 of 9 settings, with gains of 0.62–4.05 points, while Fisher leads by 0.16 points on Gemma-2-2B at 1024 tokens (Table R1).

Q2. The benefit of the proposed functional weighting.

A2. Thank you for suggesting a more comprehensive analysis. In Fig. 3b, we provided 1024 chunk size comparison without OAS, and Identity weighting (coordinate averaging) outperforms the full metric on QASPER and 2WikiMultihopQA. To establish where the gains come from, we extend the ablation to five chunk sizes and two hypernetwork families, and we separated the effect of the coordinate space from the effect of the metric.

(1) In document-conditioned merging on Qwen3-4B, Full without OAS has the highest average at all five chunk sizes, with gains of 0.15–3.84 normalized ROUGE-L F1 points over Identity. Across datasets, Full weighting leads in 12 of 15 cases and chunk-size settings. Identity leads in the other three: QASPER and 2WikiMultihopQA at 1024 tokens, which is the setting shown in Fig. 3b, and 2WikiMultihopQA at 256 tokens (Table R2). The 1024-token setting is therefore where the benefit is smallest.

(2) In task-conditioned merging on ten tasks with Qwen3-8B, Full without OAS improves the final macro average over Identity from 79.16 to 83.96, while Diagonal leads on WinoGrande (Table R3). On the longer 21-task stream, the benefit depends on shrinkage. Full without OAS is close to Identity in macro average (75.41 vs. 75.26) and higher in micro average (82.37 vs. 79.86), and both are below weight-space S-Average (78.23). Full with OAS reaches 79.43, which is 3.70 points above Diagonal with OAS and the highest among all methods (Tables R4–R5). On long streams, the gain therefore comes from the full metric together with shrinkage, not from the coordinate space or from either component alone.

We will add coordinate averaging (Identity) as a baseline in the main comparison tables and include these ablations in the revised manuscript.

Q3. Evaluate task-conditioned merging on longer task sequences.

A3. We extended the task-conditioned merging evaluation on Qwen3-8B to 21 unseen tasks over five random task orders. MOSAIC achieves the highest final macro average among the evaluated methods at 79.43, compared with 78.97 for S-TIES, 77.10 for OPCM, and 77.04 for RegMean. MOSAIC also has the highest mean backward transfer (BWT) at +1.01 points. Table R4 summarizes the final macro average and BWT. We report each task's results in Table 14 in the revised manuscript.

Q4. Construction time comparison and GPU cost.

A4. Thank you for asking us to clarify the trade-off among methods. Fig. 3(c) reports time per arrival and peak additional GPU memory: MOSAIC uses 2402.6 ms/30.72 GiB, versus OPCM’s 1318.7 ms/43.36 GiB. We also re-measured S-Average on Qwen3-4B at 1024 tokens using MOSAIC’s per-chunk schedule: 854.9 ms/3.44 GiB. Thus, MOSAIC improves downstream performance at higher computational cost than S-Average, since it runs frozen target model and updates its per-layer statistics, whereas S-Average only updates a running mean. (Table R6)

Q5. Correct the Mistral reference.

A5. Thank you for your careful reading. We corrected the Mistral model reference in the revised manuscript.

Q6. Correct the notation and proofreading issues.

A6. Thank you for your careful reading. We corrected the notation and fixed the missing verb and punctuation identified.

Supporting tables

Table R1. MOSAIC versus Fisher-weighted averaging on three datasets. Macro-average normalized ROUGE-L F1 (%) across QASPER, 2WikiMultihopQA, and MultiFieldQA.

ModelChunk sizeMOSAICFisherMOSAIC − Fisher
Qwen3-4B204845.6541.60+4.05
Qwen3-4B102442.9241.85+1.07
Qwen3-4B51243.2241.54+1.68
Mistral-7B204876.2675.64+0.62
Mistral-7B102472.3870.65+1.73
Mistral-7B51271.8668.98+2.88
Gemma-2-2B204862.8961.80+1.09
Gemma-2-2B102463.0463.20−0.16
Gemma-2-2B51262.6760.10+2.57

Table R2. Functional weighting for document-conditioned merging on Qwen3-4B. Average normalized ROUGE-L F1 (%) across the three datasets. Diagonal and Full here are without OAS.

Chunk sizeWeightingQASPER2WikiMultihopQAMultiFieldQAAverage
2048Identity50.2626.2752.7443.09
2048Diagonal48.8027.6252.3742.93
2048Full52.3830.4554.3345.72
1024Identity48.7028.0850.3742.38
1024Diagonal44.3226.9949.9640.42
1024Full47.0827.2353.2742.53
512Identity44.5428.9646.7440.08
512Diagonal45.3728.6648.0040.68
512Full46.3829.4553.7143.18
256Identity41.0834.9945.4440.50
256Diagonal40.2134.3845.6840.09
256Full41.9733.1353.7242.94
128Identity37.9035.6143.9139.14
128Diagonal39.0235.1343.9539.37
128Full41.8535.8851.2242.98

Table R3. Functional weighting for task-conditioned merging on ten tasks. Results cover five task orders on Qwen3-8B. Diagonal and Full use no OAS.

WeightingARC-CARC-EBoolQGSM8KHellaSwagOBQAPIQAWinoGrandeHumanEvalMBPPMacroMicro
Identity89.1195.2886.3166.7580.3285.2085.4868.3263.2971.5379.16 ± 0.0882.02 ± 0.03
Diagonal89.6295.1386.1566.5480.6284.8885.3068.8163.1771.0879.13 ± 0.0682.12 ± 0.03
Full92.0096.7287.0083.2683.9687.5286.3566.4279.2777.0983.96 ± 0.0485.27 ± 0.04

Table R4. Qwen3-8B on 21 unseen tasks. Merging methods report over five task orders. Completed task-level results are shown in Table 14 in the revised manuscript.

MethodFinal macro avg.BWT
Individual T2L76.26—
OPCM77.10 ± 0.38−0.22 ± 0.59
RegMean77.04 ± 0.06+0.25 ± 0.60
DOP74.51 ± 0.81−1.16 ± 1.21
S-Average78.23 ± 0.27+0.39 ± 0.45
S-Task Arithmetic14.29 ± 1.60−20.33 ± 1.88
S-TIES78.97 ± 0.15−0.37 ± 0.46
S-KnOTS-TIES54.66 ± 1.74−11.31 ± 2.20
MOSAIC79.43 ± 0.02+1.01 ± 1.05

Table R5. Functional weighting and OAS on 21-task evaluation. The results are over five task orders. Completed task-level results are shown in the revised manuscript Table 15.

WeightingFinal macro avg.Final micro avg.
Identity75.26 ± 0.0779.86 ± 0.02
Diagonal without OAS75.80 ± 0.0780.32 ± 0.04
Diagonal with OAS75.73 ± 0.1680.35 ± 0.02
Full without OAS75.41 ± 0.0182.37 ± 0.02
Full with OAS (MOSAIC)79.43 ± 0.0283.65 ± 0.01

Table R6. Construction time and GPU cost.

MethodConstruction time (ms/arrival)Peak additional GPU memory (GiB)Time relative to OPCMMemory relative to OPCM
OPCM1318.743.361.00×1.00×
MOSAIC2402.630.721.82×0.71×
S-Average854.93.440.65×0.08×

Thank you for the detailed response and the additional experiments. They clarify the points I raised. Please make sure the manuscript reflects these changes.

Dear Reviewer RsS6,

Thank you for your engagement and for confirming that our response clarifies the points you raised. We have uploaded a revised manuscript that incorporates these changes, marked in blue:

  • Discussion of Fisher-weighted averaging (Matena & Raffel) and a clarified positioning relative to RegMean (lines 126-131 in Section 2.2, lines 288-297 in Section 4.3)
  • The Fisher-weighted averaging baseline (Tables 1, 7-9)
  • Coordinate averaging (Identity) in the functional-weighting comparison (Tables 3, 13, 16)
  • The 21-task evaluation (Tables 15-16)
  • Construction time and memory comparison including S-Average under the same chunk-by-chunk encoding schedule as MOSAIC (Figure 3(c))
  • The corrected Mistral reference and proofreading fixes

If these revisions resolve your concerns, we would be grateful if you would consider updating your score. If any concerns remain, we would be glad to address them during the remaining discussion period.

Best regards,

Authors of Submission 50740

Reviewer_82KrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MOSAIC, a continual merging method for hypernetwork-generated LoRA adapters. Instead of merging adapters directly in weight space, MOSAIC operates in the hypernetwork's latent space and constructs a functional metric using the decoder Jacobian and frozen-model activations. By approximating the functional discrepancy with a quadratic objective, the method enables online merging through additive sufficient statistics while maintaining a fixed-rank LoRA. Experiments across document- and task-conditioned settings demonstrate competitive performance against existing merging methods.

Critical strengths
  1. Interesting problem setting and motivation. The paper studies continual merging of hypernetwork-generated LoRAs, a meaningful setting that has received relatively limited attention. Exploiting the shared hypernetwork structure provides an interesting perspective on LoRA merging.
  2. Well-structured formulation. The proposed functional metric leads to a quadratic objective with additive sufficient statistics, enabling efficient online accumulation without retaining previous adapters. The theoretical derivations are generally clear.
Critical weaknesses
  1. The ablation study does not adequately isolate the contributions of latent-space merging, activation-based functional matching, and Jacobian-based weighting. In particular, the full metric does not consistently outperform simple latent-space averaging in Figure 3(b). The effectiveness and necessity of the key components therefore remain insufficiently demonstrated.
  2. MOSAIC relies on first-order approximations around individual latent coordinates, but the merged coordinate may be far from these points. Although Section 5.2 provides an approximation error bound, the bound contains cubic and quartic terms in ri(z)=∣z−zi∣2r_i(z)=|z-z_i|_2, which may become very loose when merging diverse adapters. Moreover, the authors do not compare MOSAIC against direct numerical optimization of the original functional objective FtF_t, making it unclear how much performance is sacrificed by the quadratic approximation.
Questions for authors
  1. Can the authors provide more comprehensive ablations to isolate the contributions of the proposed method?
  2. How large are the distances ri(zm)=∣zm−zi∣2r_i(z_m)=|z_m-z_i|_2 defined in Section 5.2 in practice, and how do they change as more adapters are merged? Given that the theoretical error bound grows cubically and quartically with these distances, can the authors report the actual linearization errors? Additionally, could the authors compare MOSAIC with direct numerical optimization of the original functional objective FtF_t to assess the impact of the quadratic approximation?
  3. The authors state that directly optimizing FtF_t requires repeatedly evaluating previous adapters on their probes, motivating the quadratic approximation. However, in my opinion, the original objective FtF_t can also be expressed using additive sufficient statistics, similar to RegMean, without revisiting previous adapters or probes. Could the authors clarify why direct optimization of FtF_t is considered incompatible with continual merging?
3 comments

Thank you for your careful review and for suggesting experiments that clarify the contributions of MOSAIC. We address each point below and put the supporting tables at the end.

Q1. The ablations should distinguish the effects of the activation-based matching, latent merge space, and Jacobian weighting.

A1. We separate the three factors with the following comparisons.

Activation weighting. In Fig. 3b, we provided 1024 chunk size comparison without OAS, and Identity weighting (coordinate averaging) outperforms the full metric on QASPER and 2WikiMultihopQA. To establish where the gains come from, we extend the ablation to five chunk sizes and two hypernetwork families, and we separated the effect of the coordinate space from the effect of the metric. (1) In document-conditioned merging on Qwen3-4B, Full without OAS had the highest average at all five chunk sizes, with gains of 0.15–3.84 normalized ROUGE-L F1 points over Identity. Across datasets, Full weighting leads in 12 of 15 datasets and chunk-size settings. Identity leads in the other three: QASPER and 2WikiMultihopQA at 1024 tokens, which is the setting shown in Fig. 3b, and 2WikiMultihopQA at 256 tokens (Table A1). The 1024-token setting is therefore where the benefit is smallest. (2) In task-conditioned merging on ten tasks with Qwen3-8B, Full without OAS improves the final macro average over Identity from 79.16 to 83.96, while Diagonal leads on WinoGrande (Table A2). The gains therefore come from the full structure of the metric.

Merge space. We compare weight-space averaging with latent-space Identity on Qwen3-4B. Their averages are within 0.38 points at every chunk size, and each leads in some settings (Table A3). Simply merging in latent space alone therefore does not account for the gains.

Jacobian weighting. In MOSAIC, the Full metric is Mi=Ji⊤JiM_i=J_i^\top J_i, where JiJ_i combines the decoder Jacobian with the frozen activations, and setting Mi=IM_i=I gives the arithmetic latent mean, Identity. The comparison between Identity and Full (Tables A1–A2) therefore ablates the two jointly. RegMean, which applies the activation weighting in weight space without the decoder Jacobian, isolates activation weighting alone, and we use γ=0.9\gamma=0.9 as its paper (Section 6).

Q2. How large are ∣zm−zi∣2|z_m-z_i|_2, and how they change as more adapters are merged?

A2. In document-conditioned merging, we measure the Frobenius distance from the merged latent to each source latent included at every prefix. Across the nine Qwen3-4B settings, the reported medians range from 4.00 to 6.01. They peak at prefixes of two to four chunks and then decrease through eight chunks in every setting. Longer prefixes include only documents with enough chunks, so this is a descriptive trend across the available documents at each prefix. The distances in Section 5.2 are defined in latent space (Table A4).

Q3. How large is linearization error?

A3. We agree that the bound in Section 5.2 can be loose when the merged coordinate is far from the source coordinates, so we measured the linearization error directly in MOSAIC on Mistral-7B across ten tasks and five task orders in task-conditioned merging (Table A5). At the final MOSAIC latent, we compare the exact decoder-induced change with its first-order prediction around each source task latent, using the same frozen probes for both. Across all five orders, the exact objective is 4190.98 per probe token, and the surrogate without OAS shrinkage is 2822.63 per probe token. The quadratic underestimated the exact objective by 1368.35 per probe token (32.65%), and the relative Taylor residual is 0.79. These values are identical across the five task orders because the merge is order-invariant. To measure the consequence for the merged adapter, we compare MOSAIC's solution with direct optimization of FtF_t in A4.

Q4. How does MOSAIC compare with direct optimization of the original objective in functional error, downstream performance, and cost?

A4. We compare MOSAIC with direct Adam optimization of the original frozen-probe objective FtF_t over the hypernetwork coordinates on Mistral-7B across ten tasks and five task orders. For direct optimization, at each optimized prefix, Adam ran for 1000 updates per initialization at a learning rate of 0.03. We initialized from the mean of the task latents seen so far and from the first task’s latent, and retained the iterate with the lowest FtF_t across both starts.

Functional error and downstream performance. Adam's final FtF_t per probe token is 17.2% lower than MOSAIC's final FtF_t. This gap reflects both the first-order surrogate and OAS shrinkage. The downstream performance difference is +0.06 ± 0.12 Macro points (Direct Adam minus MOSAIC). Both merges are order-invariant, so the five orders yield the same final FtF_t, and this spread reflects evaluation variability. (Table A6)

Cost. With cached probes, constructing the merge at all ten prefixes takes 55.02 s for MOSAIC and 2320 s for Direct Adam on average, giving a 42.17× ratio of mean construction times at the tested budget. (Table A7)

Budget sensitivity. At 800 updates per initialization, Adam is within 0.10% of its 1000-update objective at every prefix and within 0.02% at the final prefix, so a longer run would not change the 17.2% gap materially. At 250 updates, it is already within 1.11%, so the cost gap would remain large under a much smaller budget. The 42× ratio is measured on the timed 1000-update runs. (Table A8)

In summary, MOSAIC's first-order surrogate with shrinkage costs about 17% in FtF_t relative to direct optimization, with no measurable downstream difference and a 42× lower construction time.

Q5. The original objective can also be maintained online with sufficient statistics.

A5. Thank you for pointing this out. We agree that direct optimization of FtF_t is compatible with continual merging, and we have corrected the statement in Section 4.2. With frozen layer inputs, running activation Gram matrices and adapter–activation cross-moments determine FtF_t up to an additive constant, and both can be updated once per arrival.

Minimizing FtF_t over the weights has a RegMean-style closed form, but RegMean solves for model weights and its solutions are generally not adapters the hypernetwork can generate, and we compare against RegMean, followed its paper using (γ=0.9\gamma=0.9) in Section 6. Minimizing over hypernetwork coordinates keeps the solution in the decoder’s range. Because the decoded weight updates generally depend nonlinearly on those coordinates, this formulation generally requires a numerical solve at each readout. The MOSAIC surrogate gives a quadratic problem in the hypernetwork coordinates with a closed-form solution from the accumulated statistics.

We have quantified this trade-off in A4. Compared to direct Adam optimization of FtF_t over the hypernetwork coordinates, MOSAIC showed no measurable downstream difference (Table A6), while Adam takes 42× as long to construct the adapter (Table A7).

Supporting tables

Table A1. Activation weighting in document-conditioned merging. Qwen3-4B, normalized ROUGE-L F1 (%). Diagonal and Full here are without OAS.

Chunk sizeWeightingQASPER2WikiMultihopQAMultiFieldQAAverage
2048Identity50.2626.2752.7443.09
2048Diagonal48.8027.6252.3742.93
2048Full52.3830.4554.3345.72
1024Identity48.7028.0850.3742.38
1024Diagonal44.3226.9949.9640.42
1024Full47.0827.2353.2742.53
512Identity44.5428.9646.7440.08
512Diagonal45.3728.6648.0040.68
512Full46.3829.4553.7143.18
256Identity41.0834.9945.4440.50
256Diagonal40.2134.3845.6840.09
256Full41.9733.1353.7242.94
128Identity37.9035.6143.9139.14
128Diagonal39.0235.1343.9539.37
128Full41.8535.8851.2242.98

Table A2. Activation weighting in task-conditioned merging. Qwen3-8B over five orders of ten tasks. Diagonal and Full here are without OAS.

WeightingARC-CARC-EBoolQGSM8KHellaSwagOBQAPIQAWinoGrandeHumanEvalMBPPMacroMicro
Identity89.1195.2886.3166.7580.3285.2085.4868.3263.2971.5379.16 ± 0.0882.02 ± 0.03
Diagonal89.6295.1386.1566.5480.6284.8885.3068.8163.1771.0879.13 ± 0.0682.12 ± 0.03
Full92.0096.7287.0083.2683.9687.5286.3566.4279.2777.0983.96 ± 0.0485.27 ± 0.04

Table A3. Weight-space S-Average and Latent-space Identity (Mi=IM_i=I). Qwen3-4B, normalized ROUGE-L F1 (%).

Chunk sizeMethodQASPER2WikiMultihopQAMultiFieldQAAverage
2048Weight-space49.8327.1151.8542.93
2048Latent-space50.2626.2752.7443.09
2048Weight − latent-0.430.84-0.89-0.16
1024Weight-space48.9028.4150.4742.59
1024Latent-space48.7028.0850.3742.38
1024Weight − latent0.200.330.100.21
512Weight-space42.2931.8846.0940.09
512Latent-space44.5428.9646.7440.08
512Weight − latent-2.252.92-0.650.01
256Weight-space40.6135.3746.1240.70
256Latent-space41.0834.9945.4440.50
256Weight − latent-0.470.380.680.20
128Weight-space38.4535.5542.2738.76
128Latent-space37.9035.6143.9139.14
128Weight − latent0.55-0.06-1.64-0.38

Table A4. Median MOSAIC latent distance by number of merged chunks.

DatasetChunk size2345678
QASPER5125.3095.9306.0145.9145.7715.5745.418
QASPER10245.7186.0045.8345.6225.3305.1054.803
QASPER20485.4925.5155.1214.7834.4824.2044.005
2WikiMultihopQA5125.0895.7885.9085.8525.7145.5535.398
2WikiMultihopQA10245.6405.9005.7185.4695.2225.0024.805
2WikiMultihopQA20485.3845.3365.0454.7044.4134.1914.001
MultiFieldQA5125.0175.7375.8155.7585.5825.4195.249
MultiFieldQA10245.5365.7695.6135.3655.1684.9754.770
MultiFieldQA20485.2225.2384.9504.5484.3344.2354.093

Table A5. Measure linearization error. Mistral-7B across 10 tasks on five orders.

OrderExact Ft/tokenRaw Qt/tokenSigned (Ft − Qt)/tokenRelative Taylor residual
04190.982822.631368.350.794819
14190.982822.631368.350.794819
24190.982822.631368.350.794819
34190.982822.631368.350.794819
44190.982822.631368.350.794819
Mean ± SD4190.98 ± 0.002822.63 ± 0.001368.35 ± 0.000.794819 ± 0.00

Table A6. MOSAIC versus direct latent Adam. Mistral-7B across 10 tasks on five orders.

OrderMOSAIC FtF_t/token ↓Direct FtF_t/token ↓MOSAIC MacroDirect MacroDirect − MOSAIC (points)
04190.983469.1367.4267.36-0.06
14190.983469.1367.3167.51+0.20
24190.983469.1367.4567.63+0.18
34190.983469.1367.5167.51-0.00
44190.983469.1367.4967.49+0.00
Mean ± SD4190.98 ± 0.003469.13 ± 0.0067.44 ± 0.0867.50 ± 0.100.06 ± 0.12

Table A7. Construction time Mistral-7B across 10 tasks on five orders.

OrderMOSAIC (s)Direct Adam (s)Direct / MOSAIC
058.242641.3645.35×
157.742173.6037.64×
252.112191.6142.06×
353.212300.5643.24×
453.812294.9842.65×
Mean55.022320.4242.17×

Table A8. Sensitivity of direct Adam to its step budget (Mistral-7B, ten tasks). For each budget, we take the best logged FtF_t across both starts and report its percentage excess over the best logged value at 1000 updates per start.

Updates per startMedian excess loss, all prefixesWorst excess loss, all prefixesWorst excess loss, prefix 10
2500.6088%1.1059%0.8214%
5000.1357%0.3449%0.2451%
7500.0248%0.1219%0.0334%
8000.0165%0.0985%0.0210%
9000.0053%0.0492%0.0068%
10000.0000%0.0000%0.0000%

Comments on the paper

5 comments
~Phileas_Sprecher1OpenReview ↗

MOSAIC’s main claims are not yet sufficiently supported

Dear Area Chairs, Reviewers, and Authors,

Thank you for an interesting paper, merging in the hypernetwork’s own coordinates is a natural idea. Complementing Reviewers 82Kr, RsS6, and mtDA, I would like to raise some concerns that the current evidence does not establish the necessity of the surrogate, the contribution of the functional metric, or a practical advantage over simpler alternatives.

1. FtF_t is computable from RegMean's running statistics, contrary to the stated motivation (cf. 82Kr Q3, mtDA W2)

Sec. 4.2 (Line 204) motivates the surrogate because directly optimizing FtF_t "would require repeatedly evaluating previous adapters on their probes". However, with Hl=∑iXi,lXi,l⊤H_l=\sum_i X_{i,l}X_{i,l}^{\top} and Rl=∑iΔWl(zi)Xi,lXi,l⊤R_l=\sum_i\Delta W_l(z_i)X_{i,l}X_{i,l}^{\top}, Eqs. 6–7 give

Ft(z)=12∑l∥(ΔWl(z)−RlHl†)Hl1/2∥F2+const.F_t(z)=\frac{1}{2}\sum_l\left\|\left(\Delta W_l(z)-R_lH_l^\dagger\right)H_l^{1/2}\right\|_F^2+\mathrm{const}.

HlH_l and RlR_l are exactly the RegMean statistics of Eq. 58 with γ=1\gamma=1 and MOSAIC's probes, which App. B.1 itself notes "are retained between arrivals, so earlier adapters and activation batches are not used for later updates". Hence FtF_t and its gradient can be evaluated without revisiting earlier adapters or probes, which contradicts the reason given at Line 204, and RegMean (γ=1\gamma=1) gives an unconstrained layer-wise minimizer of FtF_t. Please report its downstream performance and the achieved FtF_t of each method. If RegMean(γ=1\gamma=1) outperforms MOSAIC, solving MOSAIC's own objective exactly beats the constrained surrogate solution; if not, MOSAIC's advantage over this unconstrained minimizer is not explained by FtF_t, and it would help to identify what explains it.

2. The contribution of the functional metric is not yet established

In the metric ablation (Fig. 3b, Qwen3-4B, 1024 tokens), the mean of the three LongBench bars is 42.38 for identity and 42.53 for the full metric: a gain of about 0.15 points, that still leaves it below S-Average (42.59, Table 7). The ranking also flips across datasets: identity is better on QASPER and 2Wiki, the full metric on MultiFieldQA and LoCoMo.

Fig. 3b also appears inconsistent with Table 7: its Full variant scores 27.23 on 2Wiki and 42.53 on average, whereas Table 7 reports 29.10 and 42.92 for MOSAIC, and this mean appears to match the w/o OAS curve in Fig. 3a. Was OAS disabled in this ablation? Could you please clarify the configuration and report uncertainty estimates?

3. Empirical gains over simpler baselines are limited and inconsistent

The conclusion states that MOSAIC "remains competitive with the strongest weight-space baselines". In Table 1, MOSAIC is not the best method in 4 of 9 settings: RegMean (γ=0.9\gamma=0.9) beats it on Gemma-2-2B at 512 and 1024 tokens and on Mistral-7B at 1024, and S-Average beats it on Mistral-7B at 1024 and 2048. Outside Qwen3-4B, it is best in 2 of 6 settings, yet all document-setting analyses (Figs. 2a, 3a–c) use Qwen3-4B.

On Mistral-7B, MOSAIC exceeds OPCM and S-Average by only 0.16 and 0.23 points, compared with their reported standard deviations of 0.15 and 0.31 (Table 2). Without paired comparisons or confidence intervals for these differences, the reliability of these small gains remains unclear. The Base model of App. B.1, described as "a reference for assessing the effect of adapter generation and merging", appears in none of Tables 2 and 10–12.

4. The practical advantage needs clarification

The abstract motivates MOSAIC by noting that retaining every generated LoRA "increases storage", yet the paper reports neither pp (Eq. 1) nor the size of the state kept between arrivals, St∈Rp×pS_t\in\mathbb{R}^{p\times p} (Fig. 3c reports peak, not persistent, memory). For scale, retaining all 22 adapters of the longest LongBench document at 1024-token chunks (Table 3) takes about 0.16 GB in bf16 (0.31 GB in fp32) for the released Qwen3-4B checkpoint (rank 8 on down_proj). If zz stacks that checkpoint’s per-layer, per-rank latents (pp = 36 (layers) × 8 (#latents per layer) × 512(latent dim) = 147,456), a dense StS_t takes about 87 GB in fp32, and about 2.4 GB even keeping only its per-layer blocks, several times the adapters it replaces. Could you please report pp and MOSAIC's persistent state.

The abstract's key properties (one-pass updates, order invariance, memory independent of stream length) also hold for S-Average (Eq. 47). A running mean of zz (coordinate averaging, as 82Kr and RsS6 request) would also give a single deployable LoRA, with O(p)O(p) state and no probes or Jacobians; if this is the identity variant of Fig. 3b, it is within 0.15 points of the full metric. What does MOSAIC add in practice?

Thank you for your interest in our work and for the detailed comment. Most points overlap with the reviews, so we refer to those responses and address the new items here.

1. FtF_t is computable from RegMean-style statistics. We agree and have corrected Section 4.2 (responses to Reviewer 82Kr, A5). Direct optimization of FtF_t is compatible with continual merging. RegMean gives the unconstrained layer-wise minimizer over the weights, which is generally not an adapter the hypernetwork can generate, and we followed the RegMean paper setting using γ=0.9\gamma=0.9 when evaluating it in Table 1. We will add γ=1.0\gamma=1.0 for RegMean. On whether a lower FtF_t explains downstream performance: it does not fully. In task-conditioned merging, direct Adam optimization reached a 17.2% lower FtF_t than MOSAIC with no measurable downstream difference (+0.06 ± 0.12 Macro points) at 42× the construction time (responses to Reviewer 82Kr, A4, and Reviewer mtDA, A2). We therefore describe FtF_t as a tractable proxy, which is consistent with the mixed ranking of RegMean and MOSAIC in Table 1.

2. Contribution of the functional metric, and the Fig. 3b averages. Fig. 3b disables OAS in order to isolate the effect of the metric. This is why its Full variant (42.53 average, 27.23 on 2WikiMultihopQA) differs from MOSAIC in Table 7 (42.92 and 29.10) and matches the w/o OAS curve in Fig. 3a. We have clarified this in the caption of Fig. 3b. With OAS, MOSAIC's 42.92 is above S-Average's 42.59 in the 1024-token setting. We want to mention that the 1024-token setting is also where the metric's benefit is smallest: across five chunk sizes on Qwen3-4B, Full without OAS has the highest average at every size, with gains of 0.15–3.84 points over Identity, and leads in 12 of 15 dataset and chunk-size settings. In task-conditioned merging on Qwen3-8B, it improves the macro average over Identity by 4.80 points (responses to Reviewer 82Kr, A1, and Reviewer RsS6, A2). On the 21-task stream, the full metric’s gain requires shrinkage (Response to RsS6, A2).

3. Gains over simpler baselines. We agree that MOSAIC is not the best method in every setting of Table 1, and that the Mistral-7B comparisons with OPCM and S-Average involve small differences in macro average. We will report paired differences with confidence intervals and include the Base model in Tables 2 and 10–12. Two additions broaden the evidence beyond Qwen3-4B: a Fisher-weighted averaging baseline across three base models, which MOSAIC outperformed in eight of nine settings, and a 21-task stream on Qwen3-8B, where MOSAIC reached 79.43 ± 0.02 against 78.97 ± 0.15 for S-TIES and 78.23 ± 0.27 for S-Average (response to Reviewer RsS6, A1 and A3).

4. Persistent state and practical cost. We agree that running averaging also supports one-pass accumulation with state independent of stream length. What MOSAIC contributes is the decoder-derived metric, with the gains over Identity given in point 2. For Qwen3-4B in the document-conditioned hypernetwork structure, MOSAIC accumulates its statistics in its specific shared generator core, where they reduce to two 512×512512\times512 FP64 statistics per adapted layer. These total about 145 MiB, including rank scales and diagnostic tensors, excluding the frozen hypernetwork and current deployable adapter. This is far below the dense and block-diagonal estimates (87 GB and 2.4 GB). It is comparable to the storage quoted for 22 BF16 source adapters and stays fixed as the stream grows, so its storage advantage appears for longer streams. On cost, at 1024 tokens, MOSAIC took 2402.6 ms per arrival with 30.72 GiB of peak additional GPU memory, against 854.9 ms and 3.44 GiB for S-Average (response to Reviewer RsS6, A4, and Reviewer mtDA). With matched schedules, S-Average is lighter in both time and memory, and we have revised the peak-memory statement accordingly. MOSAIC costs more because at each arrival it runs the frozen target model on the chunk, updates its per-layer statistics, and solves for and decodes the merged coordinate, whereas S-Average only updates a running mean. We will report persistent state alongside peak construction memory and state the scope of each benchmark.

~Phileas_Sprecher1OpenReview ↗

Follow-up: Main claims and contributions remain insufficiently supported after the revision

Thanks for answering my questions.

The authors acknowledge several key points of my earlier comment, but the revision rewrites the motivation, concerns still stand. FtF_t is computable from RegMean-style statistics → reason for the surrogate is withdrawn and FtF_t is now "a tractable proxy" (rep to mtDA, A1). Fig 3b was run w/o OAS, and the storage advantage "appears for longer streams" only (rep to my comment, point 4). With the original justification withdrawn, the claimed value of the decoder-derived metric rests on the experiments, yet the reviewed submission gives it little direct support: its only ablation isolating the metric (Fig 3b) shows a 0.15-point gain over coordinate averaging on LongBench, and the paper concludes that "the benefit of coordinate interactions varies across datasets" (L469–470, original, removed on Nov 18). Larger gains (up to 3.84 in TabA1, 82Kr; 4.80 in Tab12) appear only in the discussion.

Other concerns:

I. The 21-task gain hinges on an unexplained Full × OAS interaction. Without OAS, the full metric trails S-Average by 2.82 Macro (leading by 0.12 Micro) and is 0.15 Macro (2.51 Micro) above coordinate averaging (Tab14–15). OAS barely changes the diagonal metric (75.80 → 75.73) but adds 4.02 Macro to the full metric, 3.16 of it from four tasks (Tab 15). On lol 706, OAS recovers 28.65 points, yet MOSAIC still scores 10.52 below both coordinate averaging and S-Avg (63.16 vs. 73.68). The response names "the decoder-derived metric" as "What MOSAIC contributes" (rep to my comment, point 4), but on this stream the metric alone trails S-Avg on Macro, whereas on ten tasks, and at 512 and 2048 tokens in the document setting, it beats S-Average without OAS (Tab 2, 12; Tab A1, 82Kr; Tab 1). The paper presents OAS as a step "to improve numerical conditioning" (L 84) and does not explain this dependence, and Section 6.4, unchanged in the revision, still reports similar performance with and without OAS (L 463–466), which does not hold on this stream.

II. Gains over simple baselines are modest and concentrated. On 21 tasks, 1.04 of MOSAIC's 1.20 Macro lead over S-Average comes from MBPP, HumanEval and GSM8K; the other 18 tasks average +0.19 (Tab14). Margins over S-TIES are +0.35 and +0.46 Macro (+0.80 and +0.89 Micro; Tab11, 14). The across-order standard deviations do not quantify uncertainty in paired differences, since MOSAIC's adapter is identical across orders (L452–453). The missing Base row matters because every merging method except S-Task Arithmetic and S-KnOTS-TIES beats Individual T2L on GSM8K by 6–14 points (Tab 11, 14).

III. Results hard to reconcile with the paper’s own descriptions.

  • Eq. 48 makes the final S-Average adapter order-independent, yet its MBPP score varies by ±5.45 across orders on 21 tasks (Tab 14), against 74.19–74.69 on ten tasks (Tab 13); MBPP is also MOSAIC's second-largest lead over S-Average (+7.67). Were adapters, seeds and decoding identical across orders?
  • Individual T2L uses "the frozen generated adapter for each task" (Tab 14), yet HumanEval moves from 70.73 (Tab 11) to 69.51 (Tab 14) and GSM8K from 72.33 to 71.87. Were the same adapters, examples, decoding and scoring used?

IV. Related work. GenerativeAdapter (Chen et al, ICLR25), not cited, already builds a single adapter from context that "can arrive in chunks sequentially", with a fixed-size partial sum that "acts as the memory of history context chunks". Its generator is trained for this, whereas MOSAIC merges outputs of a frozen hypernetwork, so the methods differ, but the one-pass, bounded-state properties the abstract emphasizes are not novel in isolation.

V. Cost. MOSAIC is about 3× slower per arrival than S-Average (Tab R6, RsS6). MOSAIC’s gains are measured against coordinate averaging (Identity; rep to my comment, point 4), which also yields a fixed-rank adapter but keeps only a running mean of coordinates, with no probes, Jacobians or 145 MiB of statistics. A running average needs only the merged and incoming updates, yet S-Average reportedly peaks 22.73 GiB above MOSAIC (53.45 vs. 30.72 GiB), which also runs probes and Jacobians. The gap likely reflects implementation, not the merging rules.

Overall. The metric’s larger gains (up to 3.84 in TabA1, 82Kr; 4.80 in Tab12) appear only in results added during the discussion, as do all unexplained baseline results in points III and V, in comparisons MOSAIC is reported to win. This raises concerns about the reliability of these new results and of the baseline implementations. The current evidence does not establish that the decoder-derived metric combined with OAS provides a reliable performance advantage over simpler merging methods sufficient to justify its substantially higher cost.

Line and table numbers in this comment refer to the Nov 15 revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.