acceptodds
Under review as a conference paper at ICLR 2027

Continually Learning a Functional Memory from Task Experience

Abstract

Trained low-rank adapters (LoRAs) record what a model has learned from task experience. We ask how a stream of these adaptations can form a shared memory that preserves past solutions and supports new ones. The core question is which weight directions each adapter should contribute. We find that projecting an adapter into existing directions can reduce its update magnitude, and restoring that magnitude often recovers much of the lost task performance. A small number of task-dependent residual additions can recover further behaviour. Retained weight energy alone therefore does not determine behavioural recovery. These findings motivate Functional Memory of Low-rank subspaces (FML), which uses adapter predictions on unlabelled task inputs to guide the growth of a shared basis after correcting update scale. The admitted task is stored through coefficients and a global scale, allowing the original adapter and task inputs to be discarded. As the shared basis expands, previously admitted reconstructed updates remain unchanged. We evaluate continual retention and basis growth over a large SNI adapter stream and several ViT image-classification benchmarks, and further demonstrate downstream reuse through target-data-free LoRA initialization for subsequent fine-tuning and description-conditioned LoRA generation and adaptation within the functional basis, with evaluations on unseen SNI tasks and seven language benchmarks. Our findings demonstrate how structure retained from completed learning can support subsequent adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Model Subspaces, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xJADOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FML, which consolidates independently trained LoRA adapters into a shared basis through projection, norm matching, and functional selection using CPQR. The resulting memory supports adapter reuse, initialization, and description-conditioned coefficient generation. The approach is interesting, but experimental comparability and attribution of downstream gains remain insufficiently established.

Critical strengths

1.The motivation for scale correction is reasonable. The paper distinguishes between the loss of useful directions and the reduction in update magnitude caused by projection. The motivation experiments support correcting the update scale before deciding whether additional directions are necessary.

2.The 440-adapter construction experiment provides useful evidence. The weight-ranked variant shares the norm matching and admission criterion with FML, offering a meaningful controlled comparison. The reported reduction of approximately 29.8% in basis directions, while maintaining strong behavioral recovery, is among the paper’s more convincing findings.

3.The stored representations have potential practical value. Representing historical adapters through shared bases and task coefficients enables subsequent generator training without revisiting the original task data or full adapters. The appendix also provides substantial algorithmic details and sensitivity analyses.

Critical weaknesses
  1. The main language comparison appears to use different adapter streams.Appendices C.1 and D describe a 22-adapter stream for FML, whereas SHARE consolidates hundreds of adapters and uses the 22 tasks for evaluation. Figure 4 therefore appears to compare different compression workloads. The reported ROUGE-L advantage (74.80 vs. 67.47) and smaller basis width require validation under identical adapter streams and evaluation procedures.

  2. Protocol mismatches and structural preservation weaken the continual-learning evidence.Vision adapters are independently trained with task-specific heads, while Table 1 includes baselines with different learning interfaces and an unmatched Caltech split. Moreover, zero post-admission forgetting follows from preserving existing basis directions and task representations, rather than demonstrating the benefit of functional selection. Initial compression error and subsequent forgetting should be evaluated separately under matched protocols and storage budgets.

  3. The functional-ranking mechanism needs stronger validation.Raw-logit sensitivities need not align with KL-based admission: common shifts across all logits leave softmax probabilities unchanged. It is also unclear whether local teacher sensitivities reliably predict the effects of removing directions. Comparison with functional-sketch column-norm ranking is needed to isolate the benefit of CPQR’s redundancy reduction.

  4. Initialization gains are confounded by rank allocation.Uniform-rank FML-Init matches UWSH at 72.0 average ROUGE-L; the improvement to 73.4 appears with global rank allocation. Without applying the same allocation to UWSH and weight-ranked bases, Table 2 cannot distinguish the contribution of functional selection from capacity redistribution.

  5. The advantage over task-specific LoRAs lacks a matched baseline.Table 4 compares adapted FML (85.0 average accuracy) with Oracle LoRAs borrowed from Text-to-LoRA (79.2), without establishing matched data, optimization budgets, and validation procedures. Retrained task-specific LoRAs, alternative shared bases, and different coefficient initializations under the same protocol are needed to attribute these gains.

  6. Resource accounting is incomplete.Basis width omits task coefficients, scales, and classifier heads. Construction also requires functional probes and repeated admission evaluations, yet runtime and peak memory are unreported. Table 3’s 10k-versus-1M step comparison covers generator training only. Total persistent storage and end-to-end compute are needed to substantiate the compactness and efficiency claims.

Questions for authors

1.How many adapters are actually consolidated by FML and SHARE in Figure 4? Please distinguish the construction stream from the evaluation set and provide a direct comparison using identical adapter arrivals and evaluation procedures.

2.Is the ground-truth task identity available during vision inference? How are the task coefficients and classifier heads selected, and are the prediction space, data splits, and parameter-counting conventions consistent across Table 1?

3.How much does CPQR improve over ranking candidates by functional-sketch column norm under the same storage budget?

4.Does FML retain its initialization advantage when UWSH and weight-ranked bases use the same global rank allocation? Does the advantage in Table 4 persist when task-specific LoRAs are retrained using the same training and validation protocol?

5.What are the total persistent storage, construction time, and peak accelerator-memory requirements, including all task representations, functional probing, admission evaluations, and generator training?

5 comments
AuthorsOpenReview ↗

Response to Reviewer xJAD - Question 1

We thank the reviewer for the detailed examination of the experimental protocols and attribution of the results.

How many adapters are actually consolidated by FML and SHARE in Figure 4? Please distinguish the construction stream from the evaluation set and provide a direct comparison using identical adapter arrivals and evaluation procedures.

In Figure 4 of the manuscript, FML consolidates the 22 evaluation adapters sequentially from an empty basis, whereas SHARE's numbers are taken from its reported setting using 500 adapters, with 50 arriving per time step, and evaluates on those 22 tasks. We used SHARE's reported setup because it gives SHARE a richer initial subspace, whereas FML builds its basis strictly from the 22 evaluation adapters.

The reported ROUGE-L advantage (74.80 vs. 67.47) and smaller basis width require validation under identical adapter streams and evaluation procedures.

To provide the exact apples-to-apples comparison requested, we ran a new controlled experiment where both FML and SHARE start from an empty memory and sequentially consolidate the exact same 22 rank-16 adapters in the same arrival order, with only the incoming adapter available at each step. Across all 7,182 test examples using identical prompts, decoding, and scoring:

  • FML achieves 73.76 ROUGE-L versus 50.04 for SHARE (+23.73 points), outperforming SHARE on 20 of 22 tasks.
  • Basis capacity is tightly matched: 3,488 total columns for FML versus 3,456 for SHARE.

Table 1: Per-task ROUGE-L after consolidating the same 22 adapters in the same arrival order.

TaskFMLSHARE
76994.4651.96
144886.6763.07
36296.6692.32
61534.7827.55
116882.7279.10
170397.6489.20
6589.1990.68
28516.5915.32
16159.7261.43
160690.5342.12
69446.157.60
129492.0021.14
147980.1935.90
148985.0919.42
29696.8366.86
36367.9851.38
12359.1214.26
12970.5946.91
59081.2763.38
1453100.0082.51
70483.3374.29
140011.294.41
Average73.7650.04

When restricted to the same 22-adapter sequential stream and matched basis width, SHARE degrades sharply because repeatedly re-estimating and truncating the shared subspace as new adapters arrive overwrites directions needed by earlier tasks. Because FML expands its basis using functional selection while preserving previously admitted directions (Appendix B.2), it maintains high performance across the entire stream.

We will add Table 1 and clarify the stream sizes in Figure 4 in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer xJAD - Question 2

Is the ground-truth task identity available during vision inference? How are the task coefficients and classifier heads selected, and are the prediction space, data splits, and parameter-counting conventions consistent across Table 1?

We followed SHARE's benchmark design and reporting practice to make the comparison as fair and directly comparable as possible. We use its benchmark task structure and ViT-B/16 backbone, and report accuracy and forgetting alongside its published baselines.

To clarify the exact evaluation protocol in Table 1:

  • Task identity and prediction space: Yes, ground-truth task identity tt is used at inference (matching SHARE's task-incremental evaluation). It selects the stored coefficients, scale, and 10-way classifier head for task tt. Both the original LoRA and its FML reconstruction share the exact same head.
  • Data splits: We use standard CIFAR-100 and official Flowers splits. Because SHARE's exact Caltech split was not publicly released, we used a fixed stratified 70/30 split (noted in the appendix).
  • Parameter accounting: Imported baselines follow their published parameter conventions. For FML, the 0.44M reported in Table 1 on Caltech already includes the shared bases, task coefficients, and all classifier heads (Table 1 below).

Table 1: Parameter count breakdown for FML on Caltech.

QuantityParameters
Shared bases336,384
Task coefficients28,661
All classifier heads76,900
Total441,945

Moreover, zero post-admission forgetting follows from preserving existing basis directions and task representations, rather than demonstrating the benefit of functional selection. Initial compression error and subsequent forgetting should be evaluated separately under matched protocols and storage budgets.

Table 1 of the manuscript already separates these two quantities: initial compression error is shown directly by comparing FML against the Original LoRA row, while subsequent forgetting after admission is guaranteed to be zero (0.0%) because FML freezes admitted basis directions and task representations (Appendix B.2). By contrast, methods like SHARE re-estimate their shared subspace as new adapters arrive, suffering both initial compression loss and subsequent forgetting as earlier directions are overwritten.

AuthorsOpenReview ↗

Response to Reviewer xJAD - Question 3

How much does CPQR improve over ranking candidates by functional-sketch column norm under the same storage budget?

Thanks for the suggestion. We ran the functional column-norm experiment across all 440 tasks under identical adaptive admission settings, which further confirms the effectiveness of the functional sketch for selecting basis directions (Table 1).

Table 1: Comparison of functional selection strategies vs. weight ranking across 440 tasks.

MetricFunctional CPQRFunctional column normWeight ranking
Avg. ROUGE-L67.8268.4566.27
Total basis directions61,26460,17688,480
Retained storage (GiB)1.1661.1411.859

Across all 440 tasks under identical adaptive admission settings, functional column-norm performs on par with, and even slightly better on average than, CPQR. Both functional selectors achieve nearly identical compression and accuracy, while requiring ~31% fewer columns and ~38% less storage than weight ranking. This confirms that FML's advantage comes from the functional sketch itself rather than the specific pivoting rule, and both CPQR and functional column-norm are both valid strategies for selecting functional directions. We will add and present functional column-norm alongside CPQR in the manuscript as an alternative function-based selection strategy.

Raw-logit sensitivities need not align with KL-based admission: common shifts across all logits leave softmax probabilities unchanged. It is also unclear whether local teacher sensitivities reliably predict the effects of removing directions.

Both direction selection and the admission criterion are configurable in FML. While a raw-logit-based criterion can also be used for admission, we use KL in our language experiments because it has been thoroughly tested and works well empirically, and our vision experiments already demonstrate the use of a different loss. Raw-logit sensitivities can also be used in place of KL-based loss.

To directly test whether lower functional capture corresponds to worse performance on unseen examples when directions are removed, we ran a basis-pruning experiment across 32 tasks from the 440-task stream (figure added to the revised appendix). For each task, we froze its post-admission representation and progressively removed stored basis columns in a fixed nested random order, retaining 100%, 95%, 90%, 75%, or 50% of columns, while rescaling the update norm to match the full stored update without retraining.

Across all retention levels, removing basis directions monotonically reduces functional capture on the calibration inputs and increases the ROUGE-L drop on held-out examples:

  • 100% columns retained: 98.99% functional capture → 1.68-point ROUGE-L drop.
  • 75% columns retained: 96.47% functional capture → 5.46-point ROUGE-L drop.
  • 50% columns retained: 89.27% functional capture → 12.18-point ROUGE-L drop.

Within individual tasks, KL divergence correlates consistently with held-out ROUGE-L loss (mean Spearman ρ=0.645\rho = 0.645, 95% CI [0.492,0.778][0.492, 0.778]), confirming that functional capture reliably tracks downstream performance degradation as basis capacity is removed. We will add both the complete analysis and the figure to the revised manuscript.

Reviewer_8jH2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Functional Memory of Low-rank subspaces (FML) to consolidate sequentially arriving, independently trained task LoRAs into a reusable shared memory. FML combines global norm matching, functional ranking of residual directions, and a functional admission criterion to expand the shared basis as needed, while storing task-specific coefficients and scales. Experiments on a large adapter stream, language and vision task retention, and new-task initialization, coefficient generation, and adaptation demonstrate its effectiveness in compression, retention, and reuse.

Critical strengths
  1. Clear motivation and a coherent method. The paper combines scale correction, function-guided direction selection, and a functional admission criterion so that construction of the shared representation directly serves the preservation of task behaviour. Append-only basis expansion also provides a clear mechanism for retaining previously admitted tasks.

  2. The experiments cover both the core mechanism and downstream reuse. The controlled comparison in Figure 3 shows that functional ranking reduces the number of basis directions by 29.8% relative to weight-ranked admission while maintaining high recovery of task performance. The language and vision retention experiments, together with the initialization and coefficient adaptation experiments, further demonstrate the method’s practical value.

Critical weaknesses
  1. The sources of the downstream gains are not yet sufficiently isolated. In Table 2, uniform-rank FML and UWSH both achieve an average score of 72.0, whereas the score of 73.4 comes from the FML variant that additionally uses global rank allocation. Table 4 also lacks a comparison applying the same coefficient adaptation procedure to other shared bases. The independent contribution of functional direction selection to gains on unseen tasks therefore remains unclear, limiting the assessment of the benefits of reusing the learned memory.

  2. The practical resource benefits are not yet fully quantified. The 440-task experiment measures compactness primarily through the number of basis directions, without reporting total storage including all historical task coefficients and scales, or the construction cost of functional probes and repeated full-model evaluations for functional admission. Without these measurements, it is difficult to determine whether fewer basis directions translate into practical storage savings over weight-based methods and how much additional computation is required.

Questions for authors
  1. Two-stage processing versus direct parameter learning. Section 3.2 takes trained LoRAs as input, whereas Table 4 demonstrates direct coefficient adaptation within a frozen basis. When the training process for a new task can be controlled, what are the main trade-offs between first training a LoRA and then consolidating it, and directly learning coefficients and a scale in the basis available at that time? Under the paper’s assumed input conditions, have you tried fixing the existing basis and briefly optimizing the current task’s coefficients and scale before deciding whether to expand the basis, to distinguish insufficient subspace capacity from incomplete use of the existing basis by analytic projection?

  2. Controlled comparisons for downstream contributions. In Table 2, how does UWSH perform when it uses the same per-module rank allocation as FML? In Table 4, does FML retain its advantage when the same coefficient adaptation procedure and training budget are applied to a capacity-matched weight-based basis?

  3. Complete resource costs. Could you report the total storage of the shared bases and all task coefficients and scales, together with per-task construction time and peak GPU memory, for the 440-task stream? At the same level of task-performance recovery, how do these measurements compare with weight-ranked admission?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8jH2 - Question 1

Thank you for this suggestion.

When the training process for a new task can be controlled, what are the main trade-offs between first training a LoRA and then consolidating it, and directly learning coefficients and a scale in the basis available at that time?

We designed FML for cases where a trained LoRA is already available, for example on Hugging Face, while its original training data are usually unavailable due to privacy or intellectual property concerns. The LoRA carries experience acquired across its full training set. We reconstruct its full weight update and use a small unlabelled calibration set to check how much of its behaviour we preserve. Tuning on that handful of inputs could improve the fit there while changing behaviour elsewhere.

When we control training and have representative labelled data, we can learn coefficients and a scale directly in the frozen basis. This adds no directions and works when the existing basis has sufficient capacity. Training a full LoRA first gives the task room to learn outside the basis. Consolidation can then add the directions needed to preserve that behaviour.

Under the paper's assumed input conditions, have you tried fixing the existing basis and briefly optimizing the current task's coefficients and scale before deciding whether to expand the basis, to distinguish insufficient subspace capacity from incomplete use of the existing basis by analytic projection?

We deliberately make coefficient reconstruction optimisation-free. We project the original LoRA into the basis and correct its scale directly, so we avoid iterative updates and per-task tuning of the learning rate, step budget, regularisation, and stopping rule. The original LoRA carries experience from its full training set. If we change the coefficients to improve the fit on a handful of calibration inputs, we also change the coefficients used on every other input. That can undo behaviour the reconstruction had already preserved. We use the calibration inputs to check the reconstruction and decide whether to expand the basis.

In Section 4.3, we consider a new task with labelled training data and no trained LoRA to preserve. We learn new task behaviour by training coefficients and a scale within the existing basis. This keeps the basis fixed and limits the update to the directions already available. Training a full LoRA first gives the task room to learn outside the basis, and consolidation can then add directions as needed.

AuthorsOpenReview ↗

Response to Reviewer 8jH2 - Question 2 (1/2)

R1. Attribution of downstream improvements (1/2)

In Table 2, how does UWSH perform when it uses the same per-module rank allocation as FML?

FML uses past task experience collected dynamically over the basis construction to decide where to allocate rank. UWSH, on the other hand, cannot tell us how strongly each past task used a direction. UWSH ranks directions separately within each module, so it only supports local allocation and not global allocation.

In FML, we use the stored coefficients and scales to measure how strongly past tasks used the directions in each module (Appendix B.5). We then rank those directions across the model. A module receives more rank when more of its directions rank highly. We make this allocation before seeing any target-task inputs. With the same total rank budget of 384 and the same fine-tuning budget, we raise FML's average from 72.0 to 73.4.

To directly isolate the benefit of functional selection from global capacity redistribution, we applied the same global rank allocation and fine-tuning protocol to the weight-ranked basis across the five unseen evaluation tasks (Table 1).

Table 1: Unseen-task initialisation (ROUGE-L) comparing Functional FML and the Weight-Ranked basis over 3 seeds.

TaskFunctional FML (Paper)Weight-rankedFML − Weight
Task 3969.5068.80 ± 2.58+0.70
Task 29090.9061.43 ± 51.58+29.47
Task 39191.2086.51 ± 1.76+4.69
Task 44265.5066.03 ± 0.95−0.53
Task 159849.8049.31 ± 0.82+0.49
All-task average73.3866.42+6.96
Average excl. Task 29069.0067.66+1.34

Under global rank allocation, Functional FML outperforms the weight-ranked basis by +6.96 ROUGE-L overall (73.38 vs. 66.42) while using a 30.8% smaller basis.

AuthorsOpenReview ↗

Response to Reviewer 8jH2 - Question 2 (2/2)

R1. Attribution of downstream improvements (2/2)

In Table 4, does FML retain its advantage when the same coefficient adaptation procedure and training budget are applied to a capacity-matched weight-based basis?

We already compare functional and weight-based ranking directly in the 440-task consolidation experiment (Section 4.1, Figure 3). With the same adapters and admission procedure, functional ranking retains more task performance (67.82 vs. 66.27 ROUGE-L) and uses 37.3% less persistent adapter storage.

We use Table 4 to establish that FML can adapt to unseen tasks without expanding its basis. FML-CoeffGen initialises the coefficients from a task description, and fine-tuning those coefficients raises mean accuracy from 73.5% to 85.0% across seven benchmarks. A capacity-matched weight-based control would answer a separate question about how much of that downstream gain is attributable to functional ranking.

For adaptation to unseen tasks, we also compare the full FML-CoeffGen procedure with fresh rank-16 LoRAs under the same target-data and optimisation budget. At 512 updates, FML-CoeffGen achieves 84.18% vs. 76.21% mean accuracy across seven tasks and three seeds. Using the same validation protocol to select checkpoints, FML-CoeffGen achieves 85.32% vs. 81.55% (Table 2).

Table 2: Held-out accuracy (%) comparing FML-CoeffGen with freshly trained rank-16 LoRAs using identical target training examples, 512 optimiser updates, and three matched seeds.

TaskFML (512)LoRA (512)FML (validation)LoRA (validation)
ARC-Challenge76.8273.8376.9673.78
ARC-Easy90.4986.9490.4088.06
BoolQ89.4871.0387.7986.74
HellaSwag90.7886.1990.7886.56
OpenBookQA85.2779.9384.4080.47
PIQA85.9371.7386.6384.00
WinoGrande70.5163.8580.2771.24
Average84.1876.2185.3281.55

Validation columns report best checkpoint selection using a common validation set across 11 checkpoints.

Reviewer_vEvxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FML, a method for consolidating independently trained LoRA adapters into a shared low-rank basis. The interesting idea is that directions which are important in weight space aren't necessarily important for model behaviour, so rather than relying purely on weight-space decompositions, the authors use functional information to decide which directions to retain.
They show that substantial parts of the LoRA weight updates can be discarded while preserving most of the original task performance, particularly after correcting for the reduction in update magnitude caused by projection. On a stream of 440 language-task adapters, their functionally guided approach requires around 30% fewer basis directions than a weight-based alternative. They also demonstrate good performance retention on language and vision benchmarks, and some evidence that the resulting basis can help with adaptation to unseen tasks.
I think the approach is interesting, particularly the amount of functional redundancy they find across independently trained adapters. The idea of identifying shared, functionally important directions could also have applications beyond continual learning, for example in model merging or more efficient task adaptation.

Critical strengths

1. The functionally motivated approach with convincing experimental results: The idea of selecting subspace directions based on their impact on model behaviour, rather than just their magnitude in weight space, is motivated well. The experiments provide good evidence that this helps in practice: FML achieves comparable functional reconstruction using around 30% fewer basis directions than a weight-based alternative. I also found it interesting how much performance could be recovered after projection simply by correcting the scale of the remaining update.
2. The comprehensive evaluation demonstrating both performance retention and potential reuse: The authors evaluate the method across a large stream of 440 language adapters, as well as separate language and vision continual-learning benchmarks. The results show that the shared basis can preserve most of the original adapters' task performance while reducing redundancy. I think the additional experiments on unseen-task initialisation and coefficient-based adaptation are also valuable, as they suggest the learned basis may be useful beyond simply compressing existing adapters.

Critical weaknesses

I find the proposed approach interesting and the experimental results promising, particularly the use of functional information to identify a more compact shared basis for LoRA adapters. However, I'm not yet convinced that the evaluation establishes the broader significance of the method. The strongest evidence of functional reuse comes from a collection of related instruction-following tasks, and it's unclear how reliably the KL-based functional capture criterion preserves task-relevant behaviour at the individual-task level. I'd like to see these questions addressed to better understand how general and practically meaningful the reported improvements are. More specifically, the paper would be strengthened by going deeper into the following:

1. Evaluation of generalisability across more diverse tasks. The larger consolidation experiment uses 440 adapters from the Super-NaturalInstructions collection, which may share considerable underlying structure. While the results demonstrate effective reuse within this task family, it's less clear whether the same benefits would hold across more distinct language-model capabilities, such as code generation or mathematical reasoning. Testing how much the basis needs to expand when encountering these different tasks would help establish whether the method identifies broadly transferable functional structure or primarily exploits redundancy among related tasks.
2. Reliability of the functional capture criterion. The method relies on KL divergence over a relatively small number of unlabelled examples to determine whether an adapter has been adequately reconstructed. While the aggregate task-performance results are good, it's unclear how consistently this criterion predicts performance retention across individual tasks. For example, high average functional capture might still obscure degradation on rare or challenging examples. A more direct analysis of the relationship between functional capture and downstream task performance would help validate this central assumption.

Questions for authors
  1. How reliably does the KL-based functional capture criterion predict downstream task-performance retention across individual adapters? Are there cases where an adapter achieves high functional capture but still experiences a meaningful drop in task accuracy, perhaps on rare or challenging examples?
  2. The 440-adapter experiment shows substantial functional reuse across SNI tasks. Have you considered evaluating how well the learned basis generalises to more distinct capabilities, such as code generation or mathematical reasoning? Would these tasks require substantial basis expansion, or could they largely be represented using the existing directions?
  3. How does the computational cost of functional sensitivity estimation, CPQR ranking, and repeated KL evaluations compare with weight-based alternatives? Does the reduction in basis size also translate into savings in overall memory or computational cost?
5 comments
AuthorsOpenReview ↗

Response to Reviewer vEvx - Question 1

We thank you for this insightful question.

How reliably does the KL-based functional capture criterion predict downstream task-performance retention across individual adapters?

We find that a small unlabelled calibration set is enough to guide reconstruction for most adapters in our 440-task run. We specifically target public LoRAs (e.g. Hugging Face), where only a handful of unlabelled inputs are available. Using at most eight inputs per adapter, 299 of the 328 adapters (91.2%) that met both admission thresholds directly lost fewer than 10 held-out ROUGE-L points relative to the original LoRA. Across the full run, 396 of 440 tasks (90%) stayed below that loss threshold, including 110 tasks whose scores exceed the original LoRAs.

The per-task performance breakdown is provided below.

ROUGE-L relative to original LoRATasksPercentage
Improved11025.00%
Unchanged or lose up to 2 points16637.73%
Lose over 2 and up to 5 points7116.14%
Lose over 5 but under 10 points4911.14%
Lose 10 points or more4410.00%
Total440100.00%

Table 1: Per-task ROUGE-L performance relative to the original LoRA.

Even on the remaining 44 tasks, we found that FML still preserved most of the original LoRAs' functionalities. While the base model averaged only 10.63 ROUGE-L on these tasks, FML reached 65.57, retaining 78% of the gain delivered by the original LoRAs (81.11), while only storing a fraction of the original LoRA's weights. Across all 440 tasks, FML reduced the number of stored directions by ~21.9x.

To test this relationship directly, we progressively pruned stored basis columns (from 100% down to 50%) across 32 tasks while keeping the update norm fixed. As columns are removed, functional capture drops from 98.99% to 89.27% and held-out ROUGE-L loss rises monotonically from 1.68 to 12.18 points (mean within-task Spearman ρ=0.645\rho = 0.645 between KL and ROUGE-L loss). The corresponding figure, including 95% bootstrap confidence intervals, is provided in the revised appendix (Appendix E., Figure 15)

Are there cases where an adapter achieves high functional capture but still experiences a meaningful drop in task accuracy, perhaps on rare or challenging examples?

We found that FML generalises beyond the small unlabelled calibration set. As seen by Table 1, most tasks are able to preserve the original LoRA's functionality using FML with only small unlabelled calibration set. On task 1168, for example, none of the eight admission inputs called for the tag NOUN, yet FML correctly tagged held-out NOUN examples.

ExampleGround-TruthOriginal LoRAFMLExample KL
... how a team will suddenly start hitting ...NOUNNOUNNOUN0.00002
... a base from which such needs can be filled.NOUNNOUNNOUN0.00183
Mr. Stone suffered fractured ribs ...NOUNNOUNPRON0.89118
... my little adventure in Marshall Field & Co.CONJCONJX0.37149

Table 2: Held-out examples from task 1168.

While we also observed some exceptions on a small number of challenging inputs, where the KL divergence of the held-out example deviates from the calibration set, a fresh calibration sample would have flagged this reconstruction, suggesting that broader calibration coverage could help detect such cases before admission.

We believe the functional sketch and admission criteria provide a practical way to identify which directions matter to a LoRA's behaviour and assess how well the resulting reconstruction preserves it.

AuthorsOpenReview ↗

Response to Reviewer vEvx - Question 2

The 440-adapter experiment shows substantial functional reuse across SNI tasks. Have you considered evaluating how well the learned basis generalises to more distinct capabilities, such as code generation or mathematical reasoning? Would these tasks require substantial basis expansion, or could they largely be represented using the existing directions?

The 440-adapter SNI stream spans many task families. Figure 5 from the paper shows this, with nearly half of the tasks (196/440 tasks) falling outside of the main categories, many categories containing only a single task.

We tested cross-family generalisation directly by rebuilding the basis after excluding 94 tasks associated with coding and mathematics datasets. We then select each of 14 held-out mathematics and coding adapters independently at each checkpoint.

We found that the basis learned from other task families consistently improved retention and reduced the need for new directions. Compared with an empty basis, the 346-task basis increased mean held-out ROUGE-L from 51.69 to 54.46 while adding 21% fewer columns per adapter. It also brought 13 of 14 adapters within five points of their original LoRA, up from 7 of 14 with an empty basis (Table 1).

Table 1: Independent admission of 14 held-out SNI adapters into bases of different sizes.

Source tasksMean held-out ROUGE-LWithin 5 points of original LoRAMean new basis directions added
051.697/14283.43
2552.8810/14270.86
5053.0410/14250.29
10054.4311/14257.14
20054.2010/14219.43
34654.4613/14224.00

The 346-task basis (which excludes all math and coding tasks) also transfers effectively to mathematical reasoning outside SNI. On GSM8K, functional capture before adding any target directions rises steadily from 68.52% with 25 source tasks to 90.83% with 346 (Table 2). While the model still adds 208 columns (vs. 224 from an empty basis) to capture the remaining task-specific math skills, starting from our learned basis boosts held-out accuracy from 47.66% to 51.56%. In short, the existing basis already covers 91% of the adapter's behaviour and delivers a nearly 4-point accuracy gain while requiring fewer new directions.

Table 2: GSM8K admission from different source bases.

Source tasksPre-admission captureHeld-out accuracyNew basis columns
00.00%47.66%224
2568.52%49.22%224
5071.17%47.66%224
10076.98%48.44%224
20085.17%47.66%224
34690.83%51.56%208

We also provided a practical way of adapting external tasks into our collection within the basis in our manuscript. As shown in Table 4 of the manuscript, FML-CoeffGen uses a hypernetwork to predict initial coefficients from a task description. Across seven external benchmarks, refining only those coefficients raised mean accuracy from 73.5% to 85.0%, with no new basis directions. This shows that the learned directions can support new task solutions as well as reconstruct existing adapters.

AuthorsOpenReview ↗

Response to Reviewer vEvx - Question 3

How does the computational cost of functional sensitivity estimation, CPQR ranking, and repeated KL evaluations compare with weight-based alternatives? Does the reduction in basis size also translate into savings in overall memory or computational cost?

We compared functional CPQR and weight-energy ranking on the same 440 adapters and the same NVIDIA RTX PRO 6000 Blackwell GPU, using identical calibration inputs, norm correction, and KL admission criteria.

Compared with weight ranking, functional ranking takes longer to build the basis, but we retain more task performance with less persistent adapter storage and lower peak GPU memory. The extra time comes mainly from estimating functional sensitivity (Tables 1–2).

Table 1: 440-adapter FML Functional CPQR and weight-based ranking breakdown.

MeasurementFunctional CPQRWeight ranking
Construction time (h)3.9221.103
Mean construction time/task (s)32.099.03
Candidate-prefix KL evaluations3,9075,589
Peak allocated GPU memory (GiB)21.3435.29

Table 2: Mean FML construction cost per task across the 440-task stream.

Construction stageTime per task (s)
Initial KL checks1.172
Projection and residual computation0.216
Functional sensitivity estimation26.374
CPQR ranking0.026
Candidate-prefix KL evaluations3.247
Other construction work1.052
Total32.086

Table 3: Persistent adapter storage after 440 tasks.

MeasurementFunctional CPQRWeight ranking
Total basis columns61,26488,480
Shared basis (GiB)0.7261.177
All 440 task coefficients and scales (GiB)0.4400.682
Total tensor storage (GiB)1.1661.859
Serialized basis and task files (GiB)1.1901.882

Generator training and storage.

Unlike T2L-L's supervised fine-tuning, we train FML-CoeffGen on task descriptions and stored coefficients, without loading the base model or replaying task examples [1]. This also lets us train the generator without keeping a copy of the original LoRAs. The 440-task functional memory uses 1.166 GiB of tensors, compared with 7.734 GiB for the original LoRAs, while also consuming significantly less computational resources (generator training uses 7.04 min vs ~10 hrs for T2L, 0.95 GiB peak GPU memory vs 66.45 GiB for T2L)

Table 4: Generator training resources.

MeasurementFML-CoeffGenT2L-L (SFT)
Maximum optimiser updates10,0001,000,000
Early stopping3,420N/A
Generator wall time7.04 min~10 h
Peak allocated GPU memory0.951 GiB66.447 GiB
FML-CoeffGen training dataset
Reuse existing tasks1.196 GiB–
Task description embeddings3.343 GiB–

We report maximum update budgets for both methods. Our timed experiment used cached embeddings and targets and included validation and checkpoint saving. T2L's timing comes from its training-scheme ablation. The paper does not report wall time for the main SFT run with the one-million-update budget; we estimated this based on profiling on the same machine as our other experiments [1].

Reference

[1] Charakorn et al. (2025). Text-to-LoRA: Instant Transformer Adaption. ICML 2025, PMLR 267, 7485–7514.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.