- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes FML, which consolidates independently trained LoRA adapters into a shared basis through projection, norm matching, and functional selection using CPQR. The resulting memory supports adapter reuse, initialization, and description-conditioned coefficient generation. The approach is interesting, but experimental comparability and attribution of downstream gains remain insufficiently established.
Critical strengths
1.The motivation for scale correction is reasonable. The paper distinguishes between the loss of useful directions and the reduction in update magnitude caused by projection. The motivation experiments support correcting the update scale before deciding whether additional directions are necessary.
2.The 440-adapter construction experiment provides useful evidence. The weight-ranked variant shares the norm matching and admission criterion with FML, offering a meaningful controlled comparison. The reported reduction of approximately 29.8% in basis directions, while maintaining strong behavioral recovery, is among the paper’s more convincing findings.
3.The stored representations have potential practical value. Representing historical adapters through shared bases and task coefficients enables subsequent generator training without revisiting the original task data or full adapters. The appendix also provides substantial algorithmic details and sensitivity analyses.
Critical weaknesses
-
The main language comparison appears to use different adapter streams.Appendices C.1 and D describe a 22-adapter stream for FML, whereas SHARE consolidates hundreds of adapters and uses the 22 tasks for evaluation. Figure 4 therefore appears to compare different compression workloads. The reported ROUGE-L advantage (74.80 vs. 67.47) and smaller basis width require validation under identical adapter streams and evaluation procedures.
-
Protocol mismatches and structural preservation weaken the continual-learning evidence.Vision adapters are independently trained with task-specific heads, while Table 1 includes baselines with different learning interfaces and an unmatched Caltech split. Moreover, zero post-admission forgetting follows from preserving existing basis directions and task representations, rather than demonstrating the benefit of functional selection. Initial compression error and subsequent forgetting should be evaluated separately under matched protocols and storage budgets.
-
The functional-ranking mechanism needs stronger validation.Raw-logit sensitivities need not align with KL-based admission: common shifts across all logits leave softmax probabilities unchanged. It is also unclear whether local teacher sensitivities reliably predict the effects of removing directions. Comparison with functional-sketch column-norm ranking is needed to isolate the benefit of CPQR’s redundancy reduction.
-
Initialization gains are confounded by rank allocation.Uniform-rank FML-Init matches UWSH at 72.0 average ROUGE-L; the improvement to 73.4 appears with global rank allocation. Without applying the same allocation to UWSH and weight-ranked bases, Table 2 cannot distinguish the contribution of functional selection from capacity redistribution.
-
The advantage over task-specific LoRAs lacks a matched baseline.Table 4 compares adapted FML (85.0 average accuracy) with Oracle LoRAs borrowed from Text-to-LoRA (79.2), without establishing matched data, optimization budgets, and validation procedures. Retrained task-specific LoRAs, alternative shared bases, and different coefficient initializations under the same protocol are needed to attribute these gains.
-
Resource accounting is incomplete.Basis width omits task coefficients, scales, and classifier heads. Construction also requires functional probes and repeated admission evaluations, yet runtime and peak memory are unreported. Table 3’s 10k-versus-1M step comparison covers generator training only. Total persistent storage and end-to-end compute are needed to substantiate the compactness and efficiency claims.
Questions for authors
1.How many adapters are actually consolidated by FML and SHARE in Figure 4? Please distinguish the construction stream from the evaluation set and provide a direct comparison using identical adapter arrivals and evaluation procedures.
2.Is the ground-truth task identity available during vision inference? How are the task coefficients and classifier heads selected, and are the prediction space, data splits, and parameter-counting conventions consistent across Table 1?
3.How much does CPQR improve over ranking candidates by functional-sketch column norm under the same storage budget?
4.Does FML retain its initialization advantage when UWSH and weight-ranked bases use the same global rank allocation? Does the advantage in Table 4 persist when task-specific LoRAs are retrained using the same training and validation protocol?
5.What are the total persistent storage, construction time, and peak accelerator-memory requirements, including all task representations, functional probing, admission evaluations, and generator training?
5 comments
Response to Reviewer xJAD - Question 2
Is the ground-truth task identity available during vision inference? How are the task coefficients and classifier heads selected, and are the prediction space, data splits, and parameter-counting conventions consistent across Table 1?
We followed SHARE's benchmark design and reporting practice to make the comparison as fair and directly comparable as possible. We use its benchmark task structure and ViT-B/16 backbone, and report accuracy and forgetting alongside its published baselines.
To clarify the exact evaluation protocol in Table 1:
- Task identity and prediction space: Yes, ground-truth task identity is used at inference (matching SHARE's task-incremental evaluation). It selects the stored coefficients, scale, and 10-way classifier head for task . Both the original LoRA and its FML reconstruction share the exact same head.
- Data splits: We use standard CIFAR-100 and official Flowers splits. Because SHARE's exact Caltech split was not publicly released, we used a fixed stratified 70/30 split (noted in the appendix).
- Parameter accounting: Imported baselines follow their published parameter conventions. For FML, the 0.44M reported in Table 1 on Caltech already includes the shared bases, task coefficients, and all classifier heads (Table 1 below).
Table 1: Parameter count breakdown for FML on Caltech.
| Quantity | Parameters |
|---|---|
| Shared bases | 336,384 |
| Task coefficients | 28,661 |
| All classifier heads | 76,900 |
| Total | 441,945 |
Moreover, zero post-admission forgetting follows from preserving existing basis directions and task representations, rather than demonstrating the benefit of functional selection. Initial compression error and subsequent forgetting should be evaluated separately under matched protocols and storage budgets.
Table 1 of the manuscript already separates these two quantities: initial compression error is shown directly by comparing FML against the Original LoRA row, while subsequent forgetting after admission is guaranteed to be zero (0.0%) because FML freezes admitted basis directions and task representations (Appendix B.2). By contrast, methods like SHARE re-estimate their shared subspace as new adapters arrive, suffering both initial compression loss and subsequent forgetting as earlier directions are overwritten.
Response to Reviewer xJAD - Question 3
How much does CPQR improve over ranking candidates by functional-sketch column norm under the same storage budget?
Thanks for the suggestion. We ran the functional column-norm experiment across all 440 tasks under identical adaptive admission settings, which further confirms the effectiveness of the functional sketch for selecting basis directions (Table 1).
Table 1: Comparison of functional selection strategies vs. weight ranking across 440 tasks.
| Metric | Functional CPQR | Functional column norm | Weight ranking |
|---|---|---|---|
| Avg. ROUGE-L | 67.82 | 68.45 | 66.27 |
| Total basis directions | 61,264 | 60,176 | 88,480 |
| Retained storage (GiB) | 1.166 | 1.141 | 1.859 |
Across all 440 tasks under identical adaptive admission settings, functional column-norm performs on par with, and even slightly better on average than, CPQR. Both functional selectors achieve nearly identical compression and accuracy, while requiring ~31% fewer columns and ~38% less storage than weight ranking. This confirms that FML's advantage comes from the functional sketch itself rather than the specific pivoting rule, and both CPQR and functional column-norm are both valid strategies for selecting functional directions. We will add and present functional column-norm alongside CPQR in the manuscript as an alternative function-based selection strategy.
Raw-logit sensitivities need not align with KL-based admission: common shifts across all logits leave softmax probabilities unchanged. It is also unclear whether local teacher sensitivities reliably predict the effects of removing directions.
Both direction selection and the admission criterion are configurable in FML. While a raw-logit-based criterion can also be used for admission, we use KL in our language experiments because it has been thoroughly tested and works well empirically, and our vision experiments already demonstrate the use of a different loss. Raw-logit sensitivities can also be used in place of KL-based loss.
To directly test whether lower functional capture corresponds to worse performance on unseen examples when directions are removed, we ran a basis-pruning experiment across 32 tasks from the 440-task stream (figure added to the revised appendix). For each task, we froze its post-admission representation and progressively removed stored basis columns in a fixed nested random order, retaining 100%, 95%, 90%, 75%, or 50% of columns, while rescaling the update norm to match the full stored update without retraining.
Across all retention levels, removing basis directions monotonically reduces functional capture on the calibration inputs and increases the ROUGE-L drop on held-out examples:
- 100% columns retained: 98.99% functional capture → 1.68-point ROUGE-L drop.
- 75% columns retained: 96.47% functional capture → 5.46-point ROUGE-L drop.
- 50% columns retained: 89.27% functional capture → 12.18-point ROUGE-L drop.
Within individual tasks, KL divergence correlates consistently with held-out ROUGE-L loss (mean Spearman , 95% CI ), confirming that functional capture reliably tracks downstream performance degradation as basis capacity is removed. We will add both the complete analysis and the figure to the revised manuscript.
Response to Reviewer xJAD - Question 1
We thank the reviewer for the detailed examination of the experimental protocols and attribution of the results.
In Figure 4 of the manuscript, FML consolidates the 22 evaluation adapters sequentially from an empty basis, whereas SHARE's numbers are taken from its reported setting using 500 adapters, with 50 arriving per time step, and evaluates on those 22 tasks. We used SHARE's reported setup because it gives SHARE a richer initial subspace, whereas FML builds its basis strictly from the 22 evaluation adapters.
To provide the exact apples-to-apples comparison requested, we ran a new controlled experiment where both FML and SHARE start from an empty memory and sequentially consolidate the exact same 22 rank-16 adapters in the same arrival order, with only the incoming adapter available at each step. Across all 7,182 test examples using identical prompts, decoding, and scoring:
Table 1: Per-task ROUGE-L after consolidating the same 22 adapters in the same arrival order.
When restricted to the same 22-adapter sequential stream and matched basis width, SHARE degrades sharply because repeatedly re-estimating and truncating the shared subspace as new adapters arrive overwrites directions needed by earlier tasks. Because FML expands its basis using functional selection while preserving previously admitted directions (Appendix B.2), it maintains high performance across the entire stream.
We will add Table 1 and clarify the stream sizes in Figure 4 in the revised manuscript.