acceptodds
Under review as a conference paper at ICLR 2027

STAGE: Scaling Task-Specific LoRA Generation from Few-Shot Examples

Abstract

Parameter-efficient fine-tuning reduces the cost of adapting pretrained models, but still requires a separate optimization process for every new task. We study whether previously optimized task adapters can serve as supervision for generating a new adapter from few-shot examples, and how generation changes as this supervision grows. We introduce STAGE, a conditional Flow Matching framework that generates LoRA adapters directly from support examples without gradient-based adapter fitting at deployment. For each training task, we pair a small labeled support set with an independently optimized adapter. We represent the adapter through SVD-based factors of its effective weight update to reduce ambiguity in the original LoRA factors. For image classification, we construct up to 100,000 fitted adapters per dataset-backbone pair, and compare bank sizes of 1K, 10K, and 100K. At a fixed generation budget, support-selected accuracy increases with bank size in all six dataset-backbone settings. With only a few support images, generated adapters improve official-split accuracy over the frozen classifier by up to approximately 13.4 and 9.9 percentage points on CIFAR-100 and Tiny-ImageNet, respectively. We also evaluate a text generator on 14 language-generation datasets, with accuracy gains of 31.7 and 21.9 percentage points on e-SNLI and WikiSQL, respectively. These results show that scaling task-adapter supervision can improve few-shot adaptation without gradient-based adapter fitting at deployment.

Primary area:
generative models
Keywords:
Parameter Generation, LoRA, Flow Matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dnfAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STAGE, a conditional Flow Matching framework that generates LoRA parameters from few-shot task examples without test-time gradient descent. To address the non-uniqueness of LoRA parameters, STAGE uses SVD-based normalization and conditions generation on support-set representations. The authors construct a large-scale adapter bank covering vision and text tasks to train and evaluate the framework. Experiments show that scaling adapter supervision improves generated LoRA quality and enables effective gradient-free adaptation without exactly recovering the original adapter weights.

Critical strengths
  1. The paper explores gradient-free few-shot adaptation by leveraging LoRA adapters from previous tasks. Replacing per-task optimization with conditional parameter generation offers a promising direction for efficient and rapid adaptation.

  2. STAGE uses SVD-based normalization to address LoRA parameter non-uniqueness and conditional Flow Matching to generate task-specific adapters. Selecting candidates based on support-set performance also makes the framework practical without relying on query labels.

  3. The authors construct a large-scale adapter bank and evaluate STAGE across vision and text tasks. Comparisons with existing methods, adapter-bank scaling studies, and weight-space analyses provide useful evidence of its effectiveness.

Critical weaknesses
  1. Most evaluation tasks reuse seen classes or datasets. Without tests on unseen classes and out-of-distribution tasks, it is unclear whether STAGE learns genuine task adaptation or relies on training priors.

  2. STAGE mainly combines existing SVD normalization and Flow Matching techniques. Key baselines are missing, and the benefits of scaling adapter supervision lack deeper analysis.

  3. Inconsistencies between the main text and appendix, along with missing multi-seed results, weaken experimental reliability. Adapter generation also appears more costly than direct LoRA fine-tuning, questioning its practical efficiency.

Questions for authors
  1. Text-task results may reflect dataset-level priors rather than task-specific adaptation. Could the authors provide support-shuffling, unconditional generation, or dataset-identity baselines to clarify this distinction?

  2. Vision tasks mainly recombine seen classes. How does STAGE generalize to genuinely unseen classes or distributions, particularly the excluded classes in Tiny-ImageNet seen-180?

  3. The main text and appendix report conflicting GSM8K and MATH results, with some evaluations still pending re-decoding. The target-LoRA training protocols also appear inconsistent. Which results and settings are authoritative?

  4. The experiments do not distinguish the effects of task diversity, repeated optimization, and training randomness. What evidence shows that adapter-bank size drives improvement, especially when gains are small or saturate?

11 comments
AuthorsOpenReview ↗

Response to Seen classes and datasets; no unseen-class / OOD tests and Combination of existing techniques; missing baselines; scaling analysis

Dear Reviewer dnfA,

Thank you for your careful reading and thoughtful comments. We provide detailed responses below.

W1: Most evaluation tasks reuse seen classes or datasets. Without tests on unseen classes and out-of-distribution tasks, it is unclear whether STAGE learns genuine task adaptation or relies on training priors.

A1: Thanks for the question. We would like to clarify that unseen tasks and unseen classes are two distinct forms of generalization. Our image experiments evaluate new adaptation tasks within a pretrained classifier's label space, using unseen class combinations and held-out images. Generalization to excluded class identities is a stronger setting that requires additional effort in algorithm design (e.g., a different training loss) and would be an interesting direction for future work. Nevertheless, we evaluated this setting as the reviewer suggests, alongside transfer to held-out text datasets.

Additional evaluations. We evaluated STAGE and all six compared generators on the 20 held-out Tiny-ImageNet classes under the same seen-180 protocol. None improves over the frozen classifier in this setting, whereas STAGE retains its gains on unseen combinations of the 180 training classes (Table 28, Appendix E.5 of the revised paper). We also evaluated the text generator, without retraining, on five held-out datasets, where it improves the average accuracy from 74.02% to 75.21% (Table 29, Appendix E.6).

W2: STAGE mainly combines existing SVD normalization and Flow Matching techniques. Key baselines are missing, and the benefits of scaling adapter supervision lack deeper analysis.

A2: Thanks for the comments. We answer the three parts in turn.

(i) Novelty. We do not claim new primitives. The contribution is the combination of (a) scale: 100K independently optimized adapters per bank, where prior weight generators use a few hundred checkpoints or about a thousand tasks, together with a measurement of how generation quality scales from 1K to 100K targets; (b) canonicalization: on the same effective updates, removing the LoRA gauge by SVD is worth 2.6-12.0 pp, and without it the generator stays at the frozen classifier (Table 22, Appendix E.2 of the revised paper); and (c) stochastic generation with support-only candidate selection, which a deterministic hypernetwork cannot use: the matched deterministic model trails STAGE by 0.3-5.1 pp (Table 22). We revised the contribution list in the introduction accordingly.

(ii) Baselines. Table 3 already compares against adaptations of p-diff, RPG, DeepWeightFlow, and DnD on the same bank and episodes, and reports DnD ahead of STAGE on three of six configurations. We have added FLoWN-style and HyperTuning-style baselines on all six vision configurations, using the same setting (same bank, split, episodes, and candidate selection). Table 3 (Section 6) reports them as columns FLoWN and HyperTune, and its caption describes how we implemented each (FLoWN has no public code, so we re-implemented it). Support-selected accuracy (%), 100 official episodes, full-class scoring:

ConfigFrozenFLoWN-styleHyperTuning-style (matched deterministic)STAGE
CIFAR-100 EffNet-V2-S86.9189.3090.2394.39
CIFAR-100 ConvNeXt-V2-T81.0588.0591.3693.74
CIFAR-100 ViT-S/1681.6480.1293.7894.79
Tiny-ImageNet EffNet-V2-S79.5281.0781.6182.15
Tiny-ImageNet ConvNeXt-V2-T81.1783.9584.8085.13
Tiny-ImageNet ViT-S/1680.1275.8284.2389.32

Both matched baselines trail STAGE on every configuration: the FLoWN-style generator by 1.1-14.7 pp (it falls below the frozen classifier on both ViT-S/16 configurations), and the matched deterministic hypernetwork by 0.3-5.1 pp. The deterministic model also emits a single adapter per task, so it has no best-of-16 headroom.

(iii) Scaling analysis. Increasing the number of sampled training tasks improves adaptation across all six image configurations at a fixed generation budget (Figure 3 and Appendix E.1 of the revised paper). The smaller marginal gains in some settings are observed under a fixed generator architecture and do not establish an intrinsic limit of task scaling; scaling task supervision and the generator jointly is an important direction for future work.

Our Q4 answer clarifies what the bank-size study separates: target quality is constant across bank sizes, seed variance is tenfold smaller than the size effect, and task count and optimization runs vary together by design. Thus, the paper's claim concerns bank size, not either factor alone. A control (Table 20, Appendix E.1) separates them on CIFAR-100 ViT: 100K adapters trained on 10K tasks score 94.22, versus 89.12 for the 10K bank and 95.08 for the 100K bank, showing that both more fitted adapters and more distinct tasks help.

AuthorsOpenReview ↗

Response to Inconsistencies, missing multi-seed results, cost

We thank the reviewer for careful reading and thoughtful comments. We provide detailed responses below.

W3: Inconsistencies between the main text and appendix, along with missing multi-seed results, weaken experimental reliability. Adapter generation also appears more costly than direct LoRA fine-tuning, questioning its practical efficiency.

A3: We address the three points in turn.

  • Inconsistencies. They all stem from our early text experiments, reported in Tables 12, 14, and 25 of the submission (Tables 12, 14, and 37 in the revised paper). These tables differ from Table 2 in two ways: a limited decoding budget (448 new tokens) that truncates long solutions, and target LoRAs trained on the reference solutions rather than by rejection-sampling fine-tuning (RFT). Table 2 is authoritative: it uses the final budgets (640 new tokens for GSM8K, 1536 for MATH) and RFT targets. In addition, we labeled Best-of-6 as Best-of-16 for MetaMathQA and Python. The revision specifies a single protocol for each text table and replaces the three appendix tables (details under Q3).

  • Multi-seed. The adapted prior-method rows in Table 3 are means across three training seeds; we reported the means without the spread and will add it (for example, the DnD adaptation: 95.84 +/- 0.06 on CIFAR-100 ViT, 93.92 +/- 0.71 on Tiny-ImageNet ViT). STAGE itself had one seed, and we are adding generator seeds. Three seeds of the STAGE generator now exist across all six configurations. Support-selected accuracy (%):

  • | Config | Official episodes | Random episodes | |---|---|---| | CIFAR-100 EfficientNet | 94.36 +/- 0.15 | 94.53 +/- 0.05 | | CIFAR-100 ConvNeXt | 93.58 +/- 0.20 | 93.59 +/- 0.40 | | CIFAR-100 ViT | 94.91 +/- 0.11 | 95.14 +/- 0.17 | | Tiny-ImageNet EfficientNet | 82.16 +/- 0.05 | 82.89 +/- 0.03 | | Tiny-ImageNet ConvNeXt | 85.11 +/- 0.05 | 85.07 +/- 0.03 | | Tiny-ImageNet ViT | 89.19 +/- 0.13 | 89.38 +/- 0.39 |

The generator seed improves support-selected accuracy by 0.1-0.4 pp on the official episodes (largest range: CIFAR-100 ConvNeXt, 0.39 pp) and by up to 0.8 pp on the random episodes (Tiny-ImageNet ViT, where STAGE relies most on selecting among diverse candidates). Episode-level 95% bootstrap intervals are already reported for every paired comparison.

  • Cost. Appendix F reports the cost of our default of 16 candidates x 96 Euler steps. On a single H200, a single Euler step is as accurate as 96 on five of six configurations. At that setting, STAGE (16 candidates plus support-based selection) takes 9-13 ms per task, compared with 12-47 ms for a tuned SGD LoRA fit on the same support set. We state in the same place that STAGE ties the gradient fit on two configurations and is 1-6 pp behind on four, and that it still uses more FLOPs than the fit: the benefit is gradient-free, fixed-latency adaptation, not higher accuracy or lower total compute.
AuthorsOpenReview ↗

Response to Text: dataset-level priors vs task-specific adaptation

We thank the reviewer for careful reading and thoughtful comments. We provide detailed responses below.

Q1: Text-task results may reflect dataset-level priors rather than task-specific adaptation. Could the authors provide support-shuffling, unconditional generation, or dataset-identity baselines to clarify this distinction?

A4: Thank you for raising this question. We ran the requested controls for text: support shuffling, unconditional generation, and a dataset-identity baseline (Appendix E.4, Table 26 of the revised paper). We first clarify that in our formulation each sampled task has its own fitting, support, and query examples, with an independently fitted target LoRA. These tasks simulate repeated few-shot adaptation settings and can share an underlying task distribution. Support from another task of the same dataset can therefore remain informative for adaptation, consistent with our task construction. All controls use the same generator, test tasks and query questions, with six candidates per task, and report the mean over candidates (no selection).

Commonsense (50 tasks per dataset)ARTPIQAECQAe-SNLICommonGenAverage
Frozen model67.6470.0865.5255.7615.3454.87
STAGE, unconditional (no support set)65.0769.5664.5767.6318.3557.04
STAGE, support from a different dataset33.4374.8864.6362.7320.2451.18
STAGE, support from another task of the same dataset70.7173.6768.3388.0220.0964.16
STAGE, own support set70.7073.7268.3888.0820.0664.19
Dataset LoRA (one LoRA per dataset, 2,000 training examples)72.8474.7066.5482.5023.3663.99

Reading. On the commonsense benchmarks, STAGE achieves an average score of 64.19 with its own support and 64.16 with support from another task within the same dataset, compared with 57.04 without conditioning, 51.18 with support from a different dataset, and 54.87 for the frozen model. These results support the role of conditioning in identifying task-relevant information shared across sampled tasks: for text, the support set identifies the dataset-level task, as stated in the revised paper (Appendix E.4). The dataset-specific LoRA baseline achieves a comparable 63.99, consistent with substantial shared adaptation needs within each dataset; unlike STAGE, it requires a dataset label and training on that dataset. In vision, where each task is a different set of classes, the same exchange of support sets removes the gain (matched advantage of +3.6 to +52.4 pp; Table 24, Appendix E.3). Crucially, STAGE also transfers beyond the training datasets. Without retraining or dataset identifiers, the same generator improves over the frozen model on all five datasets excluded from bank and generator training, averaging 75.16 versus 74.02 for the frozen model and 72.92 for unconditional generation (Table 29, Appendix E.6). Together, these results support our goal of learning transferable adaptation from sampled task-adapter pairs and generating useful adapters from representative support examples, including on unseen datasets.

Reviewer_rxVuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents STAGE, a framework for generating task-specific LoRA adapters from few-shot examples without gradient-based optimization at deployment. The authors construct large banks of independently optimized task adapters, canonicalize their effective weight updates using SVD, and train conditional Flow Matching models to generate LoRA parameters from labeled support examples. The approach is evaluated on episodic image classification using three backbones and two datasets, as well as text generation using Qwen3-1.7B across 14 datasets. The authors also investigate how increasing the size of the task–adapter bank affects downstream adaptation performance. The paper addresses an interesting direction in amortized adaptation and weight-space generative modeling. However, several issues concerning experimental consistency, generalization, methodological validation, and computational efficiency limit the strength of its conclusions.

Critical strengths
  1. Learning to generate task-specific adapters from a small number of examples is a promising alternative to repeatedly optimizing LoRA parameters. Treating previously optimized adapters as reusable supervision is a meaningful research direction.
  2. The construction of up to 100,000 adapters per vision dataset–backbone pair, together with experiments on 14 text datasets, represents a substantial engineering effort. The bank-size experiments provide useful empirical observations about scaling task–adapter supervision.
Critical weaknesses
  1. The main-paper results and appendices contain substantial discrepancies, particularly for mathematical reasoning tasks. Importantly, Appendix Table 25 explicitly indicates that some evaluation results are pending re-decoding with a corrected output budget. These discrepancies may result from different decoding or evaluation configurations rather than incorrect experiments. Nevertheless, the submission does not provide a sufficiently clear explanation to establish which results are comparable. Since the claimed improvements depend on these evaluations, a consistent and fully specified evaluation protocol is essential before the empirical conclusions can be considered reliable.
  2. In the image experiments, the proposed method operates on pretrained classifiers whose output spaces already cover the relevant dataset classes. The primary "novel-task" setting excludes previously sampled ten-class combinations, but the individual classes remain seen during training (Appendix D.4.1). The paper also constructs a Tiny-ImageNet seen-180 adapter bank, excluding 20 classes during adapter-bank and generator training. However, a clear evaluation of the resulting generator on the excluded classes is not provided. Such an experiment would help distinguish generalization to unseen class combinations from generalization to unseen class identities. Similarly, the text generator is trained and evaluated on tasks sampled from the same 14 datasets, without a held-out-dataset evaluation. Thus, the results convincingly demonstrate adaptation to new episodes within familiar task distributions, but provide weaker evidence for adaptation to genuinely unseen task families.
Questions for authors
  1. Evaluation consistency. Could the authors reconcile the substantial discrepancies among Tables 2, 12, 14, and 25, particularly on GSM8K and MATH? Were all methods evaluated using identical decoding budgets, prompt formats, and correctness metrics? Which results correspond to the final corrected evaluation protocol?
  2. How does the generator trained with the Tiny-ImageNet seen-180 bank perform on tasks containing the 20 excluded classes? Can the authors provide or clarify evidence of generalization beyond previously observed classes or dataset-specific task distributions?
  3. What performance differences arise when SVD canonicalization is removed, Flow Matching is replaced with deterministic parameter prediction, or support conditions are shuffled between tasks from the same dataset?
  4. Can the authors clarify the candidate counts used across all text datasets, provide matched-budget comparisons where necessary, and report end-to-end adaptation costs under comparable hardware and data budgets?
9 comments
AuthorsOpenReview ↗

Response to Discrepancies between main paper and appendices; pending re-decoding

Dear Reviewer rxVu,

We thank you for careful reading and thoughtful comments. We provide the detailed responses below.

W1: The main-paper results and appendices contain substantial discrepancies, particularly for mathematical reasoning tasks. Importantly, Appendix Table 25 explicitly indicates that some evaluation results are pending re-decoding with a corrected output budget. These discrepancies may result from different decoding or evaluation configurations rather than incorrect experiments. Nevertheless, the submission does not provide a sufficiently clear explanation to establish which results are comparable. Since the claimed improvements depend on these evaluations, a consistent and fully specified evaluation protocol is essential before the empirical conclusions can be considered reliable.

A1: We apologize for the inconsistent numbers in the appendix. Table 2 reports the final results, decoded with the following budgets:

CommonsenseGSM8KMATHMetaMathQASQLPython
Max new tokens6464015361024256640

The discrepancies come from our early experiments, reported in Tables 12, 14 and 25 of the submission (Tables 12, 14 and 37 in the revised paper), which differ from Table 2 in two ways: a limited decoding budget (448 new tokens), and target LoRAs trained by supervised fine-tuning (SFT) on the reference solutions instead of rejection-sampling fine-tuning (RFT; Yuan et al., 2023) on the model's own correct solutions. The submission describes only the first recipe (Appendix B.2, D.2) and states none of the budgets; both omissions are ours. The table below separates the two effects on GSM8K and MATH.

DatasetSourceMax new tokensTarget recipeNFrozenGen. meanSupport-sel.Best-of-NTarget LoRA
GSM8KTable 25448SFT on reference675.4760.4860.8065.3150.74
GSM8KRe-decoded640SFT on reference1676.4760.5860.5365.5750.99
GSM8KTable 2640RFT1676.4776.5776.8680.4476.70
MATHTable 25448SFT on reference637.8032.7033.1737.2524.14
MATHRe-decoded1536SFT on reference1662.1733.0633.9638.0324.54
MATHTable 21536RFT1662.1765.1565.6168.9965.81

The final budget restores the frozen model (MATH: 37.80% to 62.17%) but not the adapters trained on reference solutions, which stay far below it (MATH generated mean 33.06%). With RFT targets, both the target LoRA and STAGE's support-selected adapter are above the frozen model (GSM8K: 76.70% and 76.86% vs. 76.47%; MATH: 65.81% and 65.61% vs. 62.17%). The same two changes explain the full fine-tuning results of Table 12; the corrected setting is shown below (Test = official test split).

DatasetSourceSettingBase lossBase metricTrain lossTrain metricVal lossVal metricTest lossTest metric
GSM8KTable 12448, SFT on ref.2.61374.870.26677.950.40267.890.43264.62
GSM8KCorrected640, RFT2.61377.220.05292.250.19884.490.18880.64
MATHTable 12448, SFT on ref.1.93236.820.49641.750.63738.020.62935.15
MATHCorrected1536, RFT1.93261.100.06797.300.37972.050.40871.21

Evaluation protocol. Every method (the frozen model, all STAGE candidates, the target LoRA, Full SFT and the adapted baselines) is evaluated with Qwen3-1.7B, thinking disabled, greedy decoding, the budgets above, the same prompt template per dataset and the metrics of Appendix B.2. The deployed STAGE adapter is selected by support-set loss only; Best-of-N uses query labels and is reported only as a diagnostic ceiling. Table 2 uses N = 16 candidates for the commonsense, SQL, GSM8K, and MATH datasets and N = 6 for MetaMathQA and Python, which the submitted caption misstated. The revised paper states this protocol in one place (Appendix C.4), corrects the caption of Table 2, and replaces the GSM8K and MATH entries of Tables 12, 14 and 25 (now Table 37) with numbers obtained under it.

References

  • Yuan et al. (2023). Scaling Relationship on Learning Mathematical Reasoning with Large Language Models. arXiv:2308.01825. PDF
AuthorsOpenReview ↗

Response to Seen classes; no evaluation on the Tiny-ImageNet held-out 20 classes; no held-out text dataset

We thank the reviewer for careful reading and thoughtful comments. We provide the detailed responses below.

W2: In the image experiments, the proposed method operates on pretrained classifiers whose output spaces already cover the relevant dataset classes. The primary "novel-task" setting excludes previously sampled ten-class combinations, but the individual classes remain seen during training (Appendix D.4.1). The paper also constructs a Tiny-ImageNet seen-180 adapter bank, excluding 20 classes during adapter-bank and generator training. However, a clear evaluation of the resulting generator on the excluded classes is not provided. Such an experiment would help distinguish generalization to unseen class combinations from generalization to unseen class identities. Similarly, the text generator is trained and evaluated on tasks sampled from the same 14 datasets, without a held-out-dataset evaluation. Thus, the results convincingly demonstrate adaptation to new episodes within familiar task distributions, but provide weaker evidence for adaptation to genuinely unseen task families.

A2: Thanks for the question. We would like to clarify that unseen tasks and unseen classes are two distinct forms of generalization. Our image experiments evaluate new adaptation tasks within a pretrained classifier's label space, using unseen class combinations and held-out images. Generalization to excluded class identities is an additional, stronger setting, which requires additional effort in algorithm design (e.g., a different training loss) and would be an interesting direction for future work. Nevertheless, we evaluated this setting as the reviewer suggests, alongside transfer to held-out text datasets.

Additional evaluations. We evaluated STAGE and all six compared generators (p-diff, RPG, DeepWeightFlow, DnD, FLoWN, 2025; HyperTuning) on the 20 held-out Tiny-ImageNet classes under the same seen-180 protocol (100 ten-way tasks). None improves over the frozen classifier in this setting (STAGE: 75.52 vs 75.82 on EfficientNet, 76.54 vs 76.84 on ConvNeXt and 52.45 vs 77.69 on ViT), whereas STAGE retains its gains on unseen combinations of the 180 training classes (+2.9 / +3.5 / +11.8 pp). Support-selected query accuracy in %, with the change against the frozen classifier in parentheses (Table 28 in Appendix E.5 of the revised paper)

MethodEffNet held-outEffNet seenConvNeXt held-outConvNeXt seenViT held-outViT seen
Frozen classifier75.8279.6976.8481.9577.6979.71
p-diff75.64 (-0.2)79.67 (-0.0)76.72 (-0.1)82.20 (+0.2)40.49 (-37.2)56.00 (-23.7)
RPG (LSTM)28.85 (-47.0)36.37 (-43.3)76.77 (-0.1)82.09 (+0.1)50.37 (-27.3)58.93 (-20.8)
DeepWeightFlow74.55 (-1.3)78.01 (-1.7)76.59 (-0.3)82.23 (+0.3)70.82 (-6.9)77.75 (-2.0)
DnD75.67 (-0.2)82.00 (+2.3)76.61 (-0.2)85.00 (+3.1)71.47 (-6.2)94.62 (+14.9)
FLoWN-style75.55 (-0.3)81.34 (+1.6)76.75 (-0.1)84.63 (+2.7)77.37 (-0.3)77.26 (-2.4)
HyperTuning-style75.79 (-0.0)82.12 (+2.4)76.66 (-0.2)85.18 (+3.2)66.41 (-11.3)85.07 (+5.4)
STAGE (ours)75.52 (-0.3)82.63 (+2.9)76.54 (-0.3)85.47 (+3.5)52.45 (-25.2)91.47 (+11.8)

We also evaluated the text generator, without retraining, on five held-out datasets (ARC-Easy, ARC-Challenge, OpenBookQA, SciQ and HellaSwag; 50 tasks per dataset, 10 support examples and 6 generated adapters per task). STAGE improves over the frozen model on all five datasets and raises the average support-selected accuracy from 74.02% to 75.21% (Table 29 in Appendix E.6 of the revised paper)

Method (unseen datasets; accuracy %)ARC-EasyARC-ChallengeOpenBookQASciQHellaSwagAverage
Frozen Qwen3-1.7B88.7876.1067.1288.2849.8074.02
Support LoRA: gradient fit on the 10 support questions86.7273.9662.3486.0648.3671.49
Target LoRA: gradient fit on 100 task questions (bank recipe)89.0876.8870.7088.5655.9876.24
STAGE without the support set (unconditional), generated mean87.8874.8164.9287.8549.1572.92
STAGE, generated mean89.5877.2867.9889.0951.8775.16
STAGE, support-selected89.9077.4467.8889.1651.6675.21
STAGE, best-of-6 (uses query labels; diagnostic only)90.7479.1269.7490.2054.5876.88
AuthorsOpenReview ↗

Response to Q1 Evaluation consistency (Tables 2, 12, 14, 25) and Q2 Generator trained on seen-180 evaluated on the 20 excluded classes

We thank the reviewer for careful reading and thoughtful comments. We provide detailed responses below.

Q1: Evaluation consistency. Could the authors reconcile the substantial discrepancies among Tables 2, 12, 14, and 25, particularly on GSM8K and MATH? Were all methods evaluated using identical decoding budgets, prompt formats, and correctness metrics? Which results correspond to the final corrected evaluation protocol?

A3: The reconciliation is as follows.

  • Final results: Table 2. It uses the final decoding budgets (640 new tokens for GSM8K, 1536 for MATH) and target LoRAs trained by rejection-sampling fine-tuning (RFT).
  • Early experiments: Table 25 (Table 37 in the revised paper), Table 12 and Table 14 used a 448-token budget that truncates long solutions, and target LoRAs trained on the reference solutions. They are not comparable to Table 2 and have been replaced in the revised paper. The table in our answer to W1 separates the two effects.
  • Consistency: Within each table, all methods were evaluated with the same decoding budget, prompt format, and correctness metric. Across tables, the budget and the target recipe differed as described above. The number of candidates differs by dataset: N = 16 for the commonsense, SQL, GSM8K, and MATH datasets, and N = 6 for MetaMathQA and Python (the submitted caption of Table 2 was wrong on this point and is corrected).
  • Final protocol: 64 / 640 / 1536 / 1024 / 256 / 640 new tokens for commonsense / GSM8K / MATH / MetaMathQA / SQL / Python, with identical prompts and metrics for the frozen model, STAGE, the target LoRA and Full SFT. The full specification is in our answer to W1.

Q2: How does the generator trained with the Tiny-ImageNet seen-180 bank perform on tasks containing the 20 excluded classes? Can the authors provide or clarify evidence of generalization beyond previously observed classes or dataset-specific task distributions?

A4: Thanks for the question. As clarified in our response to W2, unseen tasks and unseen classes are two distinct forms of generalization. Our image experiments evaluate new adaptation tasks within a pretrained classifier's label space, using unseen class combinations and held-out images, whereas generalization to excluded class identities is an additional, stronger setting that would be an interesting direction for future work. Nevertheless, we evaluated the seen-180 generator on the 20 excluded classes, as the reviewer asks.

Evaluation on the 20 excluded classes. Every method below is trained on the same Tiny-ImageNet seen-180 bank (no bank task contains any of the 20 held-out classes) and scored on the same episodes: 100 ten-way tasks drawn from the 20 held-out classes and 100 drawn from the 180 seen classes (class combinations absent from the bank). Full-class scoring, support-selected query accuracy in %, change against the frozen classifier in parentheses (Table 28 in Appendix E.5 of the revised paper):

MethodEffNet held-outEffNet seenConvNeXt held-outConvNeXt seenViT held-outViT seen
Frozen classifier75.8279.6976.8481.9577.6979.71
p-diff75.64 (-0.2)79.67 (-0.0)76.72 (-0.1)82.20 (+0.2)40.49 (-37.2)56.00 (-23.7)
RPG (LSTM)28.85 (-47.0)36.37 (-43.3)76.77 (-0.1)82.09 (+0.1)50.37 (-27.3)58.93 (-20.8)
DeepWeightFlow74.55 (-1.3)78.01 (-1.7)76.59 (-0.3)82.23 (+0.3)70.82 (-6.9)77.75 (-2.0)
DnD75.67 (-0.2)82.00 (+2.3)76.61 (-0.2)85.00 (+3.1)71.47 (-6.2)94.62 (+14.9)
FLoWN-style75.55 (-0.3)81.34 (+1.6)76.75 (-0.1)84.63 (+2.7)77.37 (-0.3)77.26 (-2.4)
HyperTuning-style75.79 (-0.0)82.12 (+2.4)76.66 (-0.2)85.18 (+3.2)66.41 (-11.3)85.07 (+5.4)
STAGE (ours)75.52 (-0.3)82.63 (+2.9)76.54 (-0.3)85.47 (+3.5)52.45 (-25.2)91.47 (+11.8)

None of the compared generators, including STAGE, improves over the frozen classifier on the held-out classes, so this limitation is not specific to STAGE. On unseen combinations of seen classes, STAGE retains its gains (+2.9 / +3.5 / +11.8 pp on EfficientNet / ConvNeXt / ViT), and there the adaptation is task-specific rather than a dataset prior (support-shuffling control, Q3). Beyond the training distribution, the text generator improves on five held-out datasets without retraining (74.02% to 75.21% on average; W2 and Table 29). The revised paper includes this table, states the scope explicitly, and lists transfer to unseen classes (e.g., adapters on backbone layers rather than on fixed class rows) as future work.

Reviewer_17YUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose STAGE, a conditional flow matching framework that generates task-specific LoRA adapters from few-shot examples without gradient-based fitting at deployment. The framework learns from offline task–adapter pairs, uses SVD-based canonical representations, and selects generated candidates using support examples, while studying how adapter-bank size affects adaptation.

Critical strengths
  • The paper provides a systematic study of how scaling task–adapter supervision affects few-shot adapter generation, offering useful insights into the benefits and limitations of larger adapter banks.
  • The proposed approach is evaluated on a variety of tasks and demonstrates competitive performance against the baselines.
  • The paper focuses on reducing the need to optimize a new adapter for each task, and generating adapters directly from few-shot examples is meaningful.
  • The code is attached, making the method reproducible.
Critical weaknesses
  • The baseline comparisons omit several closely related approaches discussed in the related work. In particular, HyperTuning and FLoWN are relevant to few-shot parameter generation and support-conditioned flow matching, respectively.

  • Appendix F reports that STAGE requires more adaptation time and FLOPs than direct LoRA fitting under the default configuration, leaving its practical computational benefits unclear.

  • Several key ablations appear to be missing. Comparisons between raw and SVD-canonicalized factors, deterministic hypernetworks and stochastic flow generation, and alternative support encoders would help clarify the contribution of each component.

  • Some frozen baselines use early checkpoints to leave room for adaptation (Appendix D.1). The large gains may therefore partly reflect incomplete baseline training. Results with fully trained baselines are needed.

  • STAGE follows a fairly standard conditional generation pipeline. Could the authors clarify its methodological novelty over existing approaches?

Questions for authors

I am somewhat confused by the inconsistent GSM8K results. The tab.2 reports frozen and support-selected accuracies of 76.47% and 76.86%, whereas tab.25 reports 75.47% and 60.80%. Could the authors explain the reasons for these discrepancies?

6 comments
AuthorsOpenReview ↗

Response to Missing baselines (HyperTuning, FLoWN)

Dear Reviewer 17YU,

We thank you for careful reading and thoughtful comments. We provide detailed responses below.

W1: The baseline comparisons omit several closely related approaches discussed in the related work. In particular, HyperTuning and FLoWN are relevant to few-shot parameter generation and support-conditioned flow matching, respectively.

A1: We thank the reviewer for the suggestion. We added both baselines across all six vision configurations using the same settings. We report them in Table 3 (Section 6) of the revised paper as columns FLoWN and HyperTune, and the caption describes how each was implemented; we also discuss the comparison between the deterministic model and STAGE in Appendix E.2 (Component Ablations). Below is the summary.

  • FLoWN-style (Saragih et al., 2025). FLoWN has no public code, so we re-implemented its few-shot recipe on our bank: a convolutional VAE over the adapter vectors, latent conditional flow matching, and a CAML-style support-set encoder. The one deliberate deviation is that the encoder reads the base model's features, so that every method sees identical inputs.- HyperTuning-style (Phang et al., 2023). HyperTuning maps the few-shot examples to adapter parameters with a deterministic hypernetwork that is trained end-to-end through the frozen model on the task loss. In our setting, the supervision is a bank of fitted adapters, so our matched baseline keeps the deterministic map but trains it by regression (MSE) to the canonicalized adapter: same support encoder, same network and same data as STAGE. It therefore isolates deterministic prediction against stochastic generation. Support-selected accuracy (%), 100 official episodes, full-class scoring:
ConfigFrozenFLoWN-styleHyperTuning-style (matched)STAGE
CIFAR-100 EffNet-V2-S86.9189.3090.2394.39
CIFAR-100 ConvNeXt-V2-T81.0588.0591.3693.74
CIFAR-100 ViT-S/1681.6480.1293.7894.79
Tiny-ImageNet EffNet-V2-S79.5281.0781.6182.15
Tiny-ImageNet ConvNeXt-V2-T81.1783.9584.8085.13
Tiny-ImageNet ViT-S/1680.1275.8284.2389.32

(The STAGE column is the seed-matched reference run used for all new comparisons; it is within 0.8 pp of the paper's Table 3 values, 94.28 / 93.43 / 95.08 / 82.21 / 85.15 / 90.04.)

Both matched baselines trail STAGE on every configuration: the FLoWN-style generator by 1.1-14.7 pp (it falls below the frozen classifier on both ViT-S/16 configurations), and the matched deterministic hypernetwork by 0.3-5.1 pp. The deterministic model also emits a single adapter per task, so it has none of the best-of-16 headroom (STAGE's best candidate reaches 95.72 / 95.76 / 96.49 / 83.33 / 85.64 / 91.36).

References

  • Saragih et al. (2025). Flow to Learn: Flow Matching on Neural Network Parameters. ICLR 2025 Workshop on Neural Network Weights as a New Data Modality. Paper
  • Phang et al. (2023). HyperTuning: Toward Adapting Large Language Models without Back-propagation. ICML 2023. Paper
AuthorsOpenReview ↗

Response to Computational benefit unclear (Appendix F)

We thank the reviewer for careful reading and thoughtful comments. We provide detailed responses below.

W2: Appendix F reports that STAGE requires more adaptation time and FLOPs than direct LoRA fitting under the default configuration, leaving its practical computational benefits unclear.

A2: Thanks for raising this important question. We would like to clarify that Appendix F priced 16 candidates x 96 Euler steps, and at that setting STAGE is slower than fitting a LoRA. That default was a choice rather than a property of the method. We have since measured the full accuracy-latency frontier on all six configurations, timing every method on the same H200 GPU with cached features. One Euler step is as accurate as 96 on five of six configurations (within 0.3 pp, sometimes better); on the sixth (CIFAR-100 EfficientNet) four steps close the gap (94.09 at 29.5 ms). At one step, generating 16 candidates and selecting by support loss costs 9-13 ms per task, versus 12-47 ms for a tuned SGD LoRA fit on the same support set and 553-983 ms for our old default.

Accuracy in % (median per-task latency in ms), 100 official episodes, full-class scoring. K = candidates, L = Euler steps. Support LoRA (the Support LoRA column of Table 3, named Target LoRA in the submission) = SGD LoRA fit on the same one-shot support set, with learning rate and steps tuned on validation episodes. Frozen = the frozen classifier without any adapter (no adaptation cost).

ConfigFrozenSupport LoRASTAGE K=16, L=96 (old default)STAGE K=16, L=1STAGE K=1, L=1
CIFAR-100 EffNet-V2-S86.9193.95 (47.0)94.28 (597.7)92.71 (10.8)92.15 (4.0)
CIFAR-100 ConvNeXt-V2-T81.0594.70 (12.2)93.43 (567.7)93.71 (9.4)93.06 (3.8)
CIFAR-100 ViT-S/1681.6495.47 (47.0)95.08 (983.1)95.25 (12.8)94.65 (5.4)
Tiny-ImageNet EffNet-V2-S79.5284.04 (46.1)82.21 (600.5)82.13 (11.1)82.09 (4.0)
Tiny-ImageNet ConvNeXt-V2-T81.1791.17 (12.3)85.15 (571.1)84.87 (10.0)84.81 (3.9)
Tiny-ImageNet ViT-S/1680.1294.34 (46.9)90.04 (552.7)90.32 (8.6)86.45 (3.7)

At L=1, STAGE is 1.2–5.5× faster per task than support-set LoRA fitting, offering a practical accuracy–latency trade-off for gradient-free adaptation despite higher FLOPs. The accuracy–latency frontier is added to the revised paper in Appendix F.1, "Matched-Hardware Accuracy–Latency Comparison": Table 34 and Figure 8.

AuthorsOpenReview ↗

Response to Missing ablations and Early-checkpoint frozen baseline (Appendix D.1)

We thank the reviewer for careful reading and thoughtful comments. We provide detailed responses below.

W3: Several key ablations appear to be missing. Comparisons between raw and SVD-canonicalized factors, deterministic hypernetworks and stochastic flow generation, and alternative support encoders would help clarify the contribution of each component.

A3: Thanks for the suggestion. We added all three ablations under matched conditions; they are reported in Appendix E.2 (Component Ablations) of the revised paper, in Table 22 (canonicalization and deterministic prediction) and Table 23 (support encoders). Two are complete on all six configurations.

Support-selected accuracy (%), 100 official episodes, full-class scoring:

ConfigFrozenSTAGERotated factors (gauge not canonicalized)Deterministic (regression)STAGE best-of-16
CIFAR-100 EffNet-V2-S86.9194.3987.9390.2395.72
CIFAR-100 ConvNeXt-V2-T81.0593.7482.5491.3695.76
CIFAR-100 ViT-S/1681.6494.7982.7693.7896.49
Tiny-ImageNet EffNet-V2-S79.5282.1579.5381.6183.33
Tiny-ImageNet ConvNeXt-V2-T81.1785.1381.2884.8085.64
Tiny-ImageNet ViT-S/1680.1289.3278.7484.2391.36

Setting. Every ablation keeps the STAGE reference run fixed, and changes exactly one component. (1) Rotated factors: the generator is trained on the same effective updates, but each task's LoRA factors are given a random rank-32 rotation instead of the SVD-canonical form (Delta W is unchanged; only the (A, B) representation differs). (2) Deterministic (regression): the same network and support encoder are trained by regression (MSE) instead of flow matching, so the model outputs one adapter per task. (3) Alternative support encoders: a DeepSets (mean-pooling) encoder replaces the set transformer, or frozen pretrained DINOv2 ViT-B/14 features replace the base model's own features (Table 23 of the revised paper).

Results. Our default configuration achieves the best result in every row of the table above, which shows that each default component is needed. Without SVD canonicalization, the generator loses 2.6-12.0 pp and ends at roughly the frozen classifier, so canonicalization is what makes the adapter bank learnable. Replacing flow matching with deterministic regression costs 0.3-5.1 pp, and a single prediction does not support the best-of-16.

For the support encoder, the DeepSets encoder costs 0.2-5.3 pp on five configurations and matches STAGE on one, while DINOv2 features perform on par with the base model's own features (within 0.7 pp on all six configurations): the attention across support examples in the set transformer matters, whereas the source of the image features does not.

W4: Some frozen baselines use early checkpoints to leave room for adaptation (Appendix D.1). The large gains may therefore partly reflect incomplete baseline training. Results with fully trained baselines are needed.

A4: Thanks for raising this insightful question. Below we clarify our motivation for choosing the early checkpoints, and, for completeness, we ran the experiment the reviewer asks for.

Why early checkpoints. STAGE is meant for the common situation in which a frozen pretrained model is reasonable but not specialized to the task at hand, so that a task-specific adapter has something to add.

Fully trained base results (%, dataset-wide scoring; gain = support-selected minus frozen; oracle gain = query accuracy of the per-task fitted LoRA minus that of the frozen model, averaged over the 100K bank tasks, i.e. the headroom available to any head adapter): (continued)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussions

Dear Area Chair and Reviewers,

We thank the Area Chair and the reviewers for handling our paper. STAGE generates LoRA adapters from a few support examples without gradient-based fitting, and shows that their accuracy increases with the number of fitted adapters in the training bank.

We thank the reviewers for recognizing the direction as meaningful (17YU, rxVu, dnfA), the bank-size study as insightful (17YU, rxVu, dnfA), and the adapter bank and evaluation as a substantial effort (rxVu, dnfA). In response to the reviews, the revised paper adds:

  • Generalization (rxVu, dnfA): held-out-dataset and unseen-class evaluations.
  • Scaling (dnfA): controls showing that both bank size and task diversity contribute.
  • Cost (17YU, rxVu, dnfA): in text, one-step generation is faster and more accurate than LoRA fitting.
  • Baselines and ablations (17YU, rxVu, dnfA): FLoWN, HyperTuning, component ablations and fully trained classifiers.
  • Consistency (17YU, rxVu, dnfA): one evaluation protocol for all text results.

After the discussion, Reviewer rxVu raised the score, and Reviewer dnfA would not oppose acceptance. We will incorporate the remaining clarifications into the final version and release the LoRA banks with the code.

We sincerely thank the Area Chair and the reviewers again for their time and constructive feedback.

Sincerely,

All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.