- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
MOSAIC continually merges hypernetwork-generated LoRAs in their shared coordinate space. It approximates functional discrepancies with a quadratic objective (Q_t), differentiates it, and solves the resulting linear system for the minimum-error coordinate. After stabilizing the system with covariance shrinkage, it decodes the merged coordinate into a single fixed-rank LoRA. MOSAIC achieves the best document-merging scores across all three tested chunk sizes on Qwen3-4B and the highest final task averages on both Mistral-7B and Qwen3-8B, although some gains over strong baselines are small.
Critical strengths
-
The mathematical treatment is complete and coherent, with no obvious errors in the main derivation. The reduction to a matrix-weighted quadratic problem is elegant.
-
The paper is clearly written, and the connection between the objective, derivation, and algorithm is easy to follow.
-
The method is practical for streaming updates: storage does not grow with the number of arrivals, and deployment requires only one fixed-rank LoRA.
Critical weaknesses
-
The objective treats each layer separately and keeps its input activations fixed. In the deployed model, however, changes in earlier layers alter the inputs to later layers. Matching LoRA contributions on frozen activations therefore misses this interaction. How much does the layerwise objective reflect the behavior of the full merged model?
-
The paper bounds the linearization error but does not measure it directly. A useful control would optimize the original accumulated objective in Eq. 7, without the Jacobian approximation, either over hypernetwork coordinates or directly over fixed-rank LoRA factors. How far is MOSAIC’s solution from the directly optimized solution in both functional error and downstream performance? The paper should report this gap alongside the optimization cost.
Questions for authors
No
6 comments
Q2. How large is the linearization error? How does MOSAIC compare with direct optimization of the original objective in functional error, downstream performance, and cost?
A2 We first measure the linearization error, and then we compare MOSAIC with direct Adam optimization of the original frozen-probe objective over the hypernetwork coordinates on Mistral-7B across ten tasks and five task orders in task-conditioned merging.
Direct measurement of linearization error. At the final MOSAIC latent, we compared the exact decoder-induced change with its first-order prediction around each source task latent, using the same frozen probes for both. Across all five orders, the exact objective is 4190.98 per probe token, and the surrogate without OAS shrinkage is 2822.63 per probe token. The quadratic underestimated the exact objective by 1368.35 per probe token (32.65%), and the relative Taylor residual is 0.79. To measure the consequence for the merged adapter, we compare MOSAIC's solution with direct optimization of below. (Table R5)
Functional error and downstream performance. For direct optimization, at each optimized prefix, Adam ran for 1000 updates per initialization at a learning rate of 0.03. We initialized from the mean of the task latents seen so far and from the first task’s latent, and retained the iterate with the lowest across both starts. Adam's final per probe token is 17.2% lower than MOSAIC's final . This gap reflects both the first-order surrogate and OAS shrinkage. The downstream performance difference is +0.06 ± 0.12 Macro points (Direct minus MOSAIC). Both merges are order-invariant, so the five orders yield the same final , and this spread reflects evaluation variability. (Table R6)
Cost. With cached probes, constructing the merge at all ten prefixes takes 55.02 s for MOSAIC and 2320 s for Direct Adam on average, giving a 42.17× ratio of mean construction times at the tested budget. (Table R7)
Budget sensitivity. At 800 updates per initialization, Adam is within 0.10% of its 1000-update objective at every prefix and within 0.02% at the final prefix, so a longer run would not change the 17.2% gap materially. At 250 updates, it is already within 1.11%, so the cost gap would remain large under a much smaller budget. The 42× ratio is measured on the timed 1000-update runs. (Table R8)
In summary, MOSAIC's first-order surrogate with shrinkage costs about 17% in relative to direct optimization, with no measurable downstream difference and a 42× lower construction time.
Supporting tables
Table R1. Median relative down_proj input drift from the frozen base model to fully adapted MOSAIC at 1024 tokens. Layers 12, 24, and 35 are the downstream measurement layers in the swap experiment. Layer 16 contains the largest median drift observed on 2WikiMultihopQA.
| Layer | QASPER | 2WikiMultihopQA | MultiFieldQA |
|---|---|---|---|
| 0 | 0.00% | 0.00% | 0.00% |
| 12 | 35.44% | 30.67% | 30.77% |
| 16 | 55.48% | 99.27% | 52.80% |
| 24 | 49.38% | 48.24% | 45.39% |
| 35 | 44.82% | 50.18% | 43.29% |
Table R2. Magnitude and rank correlation of full-model KL changes after single-layer swaps. At each chunk size, we pooled swaps from 87 unique documents across the three datasets. The 95% confidence intervals use 2000 bootstrap replicates that resample documents within each dataset and retain all swaps from each sampled document.
| Chunk size (tokens) | Swapped layer | Median per token [95% CI] | Spearman [95% CI] |
|---|---|---|---|
| 512 | 0 | 0.001745 [0.001608, 0.001893] | −0.035 [−0.083, 0.011] |
| 512 | 12 | 0.002170 [0.001980, 0.002420] | 0.306 [0.259, 0.352] |
| 512 | 24 | 0.053270 [0.048430, 0.058940] | 0.684 [0.636, 0.730] |
| 1024 | 0 | 0.001279 [0.001189, 0.001460] | −0.021 [−0.089, 0.042] |
| 1024 | 12 | 0.001641 [0.001505, 0.001864] | 0.298 [0.223, 0.375] |
| 1024 | 24 | 0.029770 [0.026070, 0.033470] | 0.627 [0.569, 0.678] |
| 2048 | 0 | 0.001096 [0.000945, 0.001256] | 0.063 [−0.030, 0.162] |
| 2048 | 12 | 0.001260 [0.001075, 0.001381] | 0.238 [0.150, 0.328] |
| 2048 | 24 | 0.014810 [0.013350, 0.018660] | 0.626 [0.558, 0.683] |
Table R3. Fraction of non-tied swaps that reduced full-model next-token KL. Each column identifies the swapped layer and the downstream layer where we measure activation changes. We pool swaps across the three datasets and exclude ties in either metric.
| Chunk size (tokens) | Swap 0, measure at 12 | Swap 12, measure at 24 | Swap 24, measure at 35 |
|---|---|---|---|
| 512 | 50.1% (695/1387) | 66.4% (921/1388) | 98.8% (1371/1388) |
| 1024 | 50.2% (359/715) | 66.3% (474/715) | 98.0% (701/715) |
| 2048 | 52.0% (197/379) | 63.9% (242/379) | 97.6% (370/379) |
Table R4. Downstream QA for complete merges on Qwen3-4B. Scores are macro-average normalized ROUGE-L F1 (%) across QASPER, 2WikiMultihopQA, and MultiFieldQA.
| Chunk size (tokens) | Identity | Diagonal without OAS | Full without OAS | Full − Identity (points) |
|---|---|---|---|---|
| 512 | 40.08 | 40.68 | 43.18 | +3.10 |
| 1024 | 42.38 | 40.42 | 42.53 | +0.15 |
| 2048 | 43.09 | 42.93 | 45.72 | +2.63 |
Table R5. Measure linearization error. Mistral-7B across 10 tasks on five orders.
| Order | Exact Ft/token | Raw Qt/token | Signed (Ft − Qt)/token | Relative Taylor residual |
|---|---|---|---|---|
| 0 | 4190.98 | 2822.63 | 1368.35 | 0.794819 |
| 1 | 4190.98 | 2822.63 | 1368.35 | 0.794819 |
| 2 | 4190.98 | 2822.63 | 1368.35 | 0.794819 |
| 3 | 4190.98 | 2822.63 | 1368.35 | 0.794819 |
| 4 | 4190.98 | 2822.63 | 1368.35 | 0.794819 |
| Mean ± SD | 4190.98 ± 0.00 | 2822.63 ± 0.00 | 1368.35 ± 0.00 | 0.794819 ± 0.00 |
Table R6. MOSAIC versus direct latent Adam. Mistral-7B across 10 tasks on five orders.
| Order | MOSAIC /token ↓ | Direct /token ↓ | MOSAIC Macro | Direct Macro | Direct − MOSAIC (points) |
|---|---|---|---|---|---|
| 0 | 4190.98 | 3469.13 | 67.42 | 67.36 | -0.06 |
| 1 | 4190.98 | 3469.13 | 67.31 | 67.51 | +0.20 |
| 2 | 4190.98 | 3469.13 | 67.45 | 67.63 | +0.18 |
| 3 | 4190.98 | 3469.13 | 67.51 | 67.51 | -0.00 |
| 4 | 4190.98 | 3469.13 | 67.49 | 67.49 | +0.00 |
| Mean ± SD | 4190.98 ± 0.00 | 3469.13 ± 0.00 | 67.44 ± 0.08 | 67.50 ± 0.10 | 0.06 ± 0.12 |
Table R7. Construction time Mistral-7B across 10 tasks on five orders.
| Order | MOSAIC (s) | Direct Adam (s) | Direct / MOSAIC |
|---|---|---|---|
| 0 | 58.24 | 2641.36 | 45.35× |
| 1 | 57.74 | 2173.60 | 37.64× |
| 2 | 52.11 | 2191.61 | 42.06× |
| 3 | 53.21 | 2300.56 | 43.24× |
| 4 | 53.81 | 2294.98 | 42.65× |
| Mean | 55.02 | 2320.42 | 42.17× |
Table R8. Sensitivity of direct Adam to its step budget (Mistral-7B, ten tasks). For each budget, we take the best logged across both starts and report its percentage excess over the best logged value at 1000 updates per start.
| Updates per start | Median excess loss, all prefixes | Worst excess loss, all prefixes | Worst excess loss, prefix 10 |
|---|---|---|---|
| 250 | 0.6088% | 1.1059% | 0.8214% |
| 500 | 0.1357% | 0.3449% | 0.2451% |
| 750 | 0.0248% | 0.1219% | 0.0334% |
| 800 | 0.0165% | 0.0985% | 0.0210% |
| 900 | 0.0053% | 0.0492% | 0.0068% |
| 1000 | 0.0000% | 0.0000% | 0.0000% |
Thank you for your careful review and for the thoughtful questions about MOSAIC's objective and its approximation. We conducted additional measurements to address them and respond to each point below, with the supporting tables at the end.
Q1. How much does the layerwise objective reflect the behavior of the full merged model?
A1. Thank you for raising this point. MOSAIC’s frozen-input objective measures how closely the merged LoRA reproduces each source LoRA’s contribution to a projection’s output when both receive the same inputs from the frozen base model. During a full forward pass, upstream adapters can change the inputs to later projections. We measured the size of this drift and tested how well the frozen-input discrepancy tracks changes in full-model predictions.
Frozen versus deployed inputs. In the document-conditioned setting on Qwen3-4B at 1024 tokens, we compared the inputs to down_proj in the frozen base model and in MOSAIC with all merged LoRAs active. We used identical content-token positions held out from fitting the merge statistics. For each document, we measured the ℓ2 difference relative to the norm of the frozen base inputs. Across QASPER, 2WikiMultihopQA, and MultiFieldQA, the median drift ranged from 30.67% to 35.44% at layer 12, 45.39% to 49.38% at layer 24, and 43.29% to 50.18% at layer 35. The largest median observed is 99.27% at layer 16 on 2WikiMultihopQA. These measurements show that the layer inputs used by the frozen-input objective differ from those in the deployed model by roughly a third to a half in relative norm at the layers we report (Table R1).
Propagation after a single-layer swap. The drift above compares the base model with all adapters active. To isolate how a change at one layer propagates, we next swapped a single layer's LoRA. At 512, 1024, and 2048 tokens on the same three datasets, we replaced the merged LoRA at layer X with one source chunk’s LoRA and kept all other layers fixed. We measured the change at down_proj in a later layer Y for (X,Y)=(0,12),(12,24),(24,35). Relative to the unmodified MOSAIC forward pass, the median relative ℓ2 changes ranged from 2.55% to 6.87% at the downstream input and from 2.31% to 5.37% at its output.
Alignment with next-token predictions. For each swap, ΔF is the after-minus-before change in the frozen-input discrepancy to the selected source chunk, summed across adapted layers. The swap removes layer X’s contribution to this discrepancy, so ΔF≤0. On identical token prefixes, we compute next-token DKL(psource∥pcandidate). The reference uses the source chunk’s LoRA at every adapted layer, and the candidate is MOSAIC before or after the swap. A negative ΔKL means that the swap reduced this divergence.
At matched chunk sizes, layer 24 swaps produced median absolute KL changes 13.5–30.5 times as large as those from layer 0 swaps. Spearman correlations between ΔF and ΔKL are 0.63–0.68 at layer 24 and 0.24–0.31 at layer 12. At layer 0, the correlation intervals included zero at all three chunk sizes (Table R2). KL decreases in about half of the non-tied layer 0 swaps and in over 97% of the non-tied layer 24 swaps (Table R3). A local improvement on frozen inputs can propagate through later layers, which determines its effect on the next-token distribution. The relationship is weak at the earliest layer tested and much stronger at the deepest layer tested, where the swaps also produced the largest KL changes.
Downstream QA. In the functional weighting ablation, Full weighting improves average normalized ROUGE-L F1 over Identity by 3.10, 0.15, and 2.63 points at 512, 1024, and 2048 tokens, respectively (Table R4).
In summary, the frozen-input objective is a projection-level criterion and does not model cross-layer propagation. It is most predictive at the depth where swaps affect next-token predictions most. It should therefore be read as a tractable proxy, and the evidence that it yields good merges is the downstream performance in Section 6 in the manuscript. We will state this in the paper and add Tables R1–R4.