- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes STAGE, a conditional Flow Matching framework that generates LoRA parameters from few-shot task examples without test-time gradient descent. To address the non-uniqueness of LoRA parameters, STAGE uses SVD-based normalization and conditions generation on support-set representations. The authors construct a large-scale adapter bank covering vision and text tasks to train and evaluate the framework. Experiments show that scaling adapter supervision improves generated LoRA quality and enables effective gradient-free adaptation without exactly recovering the original adapter weights.
Critical strengths
-
The paper explores gradient-free few-shot adaptation by leveraging LoRA adapters from previous tasks. Replacing per-task optimization with conditional parameter generation offers a promising direction for efficient and rapid adaptation.
-
STAGE uses SVD-based normalization to address LoRA parameter non-uniqueness and conditional Flow Matching to generate task-specific adapters. Selecting candidates based on support-set performance also makes the framework practical without relying on query labels.
-
The authors construct a large-scale adapter bank and evaluate STAGE across vision and text tasks. Comparisons with existing methods, adapter-bank scaling studies, and weight-space analyses provide useful evidence of its effectiveness.
Critical weaknesses
-
Most evaluation tasks reuse seen classes or datasets. Without tests on unseen classes and out-of-distribution tasks, it is unclear whether STAGE learns genuine task adaptation or relies on training priors.
-
STAGE mainly combines existing SVD normalization and Flow Matching techniques. Key baselines are missing, and the benefits of scaling adapter supervision lack deeper analysis.
-
Inconsistencies between the main text and appendix, along with missing multi-seed results, weaken experimental reliability. Adapter generation also appears more costly than direct LoRA fine-tuning, questioning its practical efficiency.
Questions for authors
-
Text-task results may reflect dataset-level priors rather than task-specific adaptation. Could the authors provide support-shuffling, unconditional generation, or dataset-identity baselines to clarify this distinction?
-
Vision tasks mainly recombine seen classes. How does STAGE generalize to genuinely unseen classes or distributions, particularly the excluded classes in Tiny-ImageNet seen-180?
-
The main text and appendix report conflicting GSM8K and MATH results, with some evaluations still pending re-decoding. The target-LoRA training protocols also appear inconsistent. Which results and settings are authoritative?
-
The experiments do not distinguish the effects of task diversity, repeated optimization, and training randomness. What evidence shows that adapter-bank size drives improvement, especially when gains are small or saturate?
11 comments
Response to Inconsistencies, missing multi-seed results, cost
We thank the reviewer for careful reading and thoughtful comments. We provide detailed responses below.
W3: Inconsistencies between the main text and appendix, along with missing multi-seed results, weaken experimental reliability. Adapter generation also appears more costly than direct LoRA fine-tuning, questioning its practical efficiency.
A3: We address the three points in turn.
-
Inconsistencies. They all stem from our early text experiments, reported in Tables 12, 14, and 25 of the submission (Tables 12, 14, and 37 in the revised paper). These tables differ from Table 2 in two ways: a limited decoding budget (448 new tokens) that truncates long solutions, and target LoRAs trained on the reference solutions rather than by rejection-sampling fine-tuning (RFT). Table 2 is authoritative: it uses the final budgets (640 new tokens for GSM8K, 1536 for MATH) and RFT targets. In addition, we labeled Best-of-6 as Best-of-16 for MetaMathQA and Python. The revision specifies a single protocol for each text table and replaces the three appendix tables (details under Q3).
-
Multi-seed. The adapted prior-method rows in Table 3 are means across three training seeds; we reported the means without the spread and will add it (for example, the DnD adaptation: 95.84 +/- 0.06 on CIFAR-100 ViT, 93.92 +/- 0.71 on Tiny-ImageNet ViT). STAGE itself had one seed, and we are adding generator seeds. Three seeds of the STAGE generator now exist across all six configurations. Support-selected accuracy (%):
-
| Config | Official episodes | Random episodes | |---|---|---| | CIFAR-100 EfficientNet | 94.36 +/- 0.15 | 94.53 +/- 0.05 | | CIFAR-100 ConvNeXt | 93.58 +/- 0.20 | 93.59 +/- 0.40 | | CIFAR-100 ViT | 94.91 +/- 0.11 | 95.14 +/- 0.17 | | Tiny-ImageNet EfficientNet | 82.16 +/- 0.05 | 82.89 +/- 0.03 | | Tiny-ImageNet ConvNeXt | 85.11 +/- 0.05 | 85.07 +/- 0.03 | | Tiny-ImageNet ViT | 89.19 +/- 0.13 | 89.38 +/- 0.39 |
The generator seed improves support-selected accuracy by 0.1-0.4 pp on the official episodes (largest range: CIFAR-100 ConvNeXt, 0.39 pp) and by up to 0.8 pp on the random episodes (Tiny-ImageNet ViT, where STAGE relies most on selecting among diverse candidates). Episode-level 95% bootstrap intervals are already reported for every paired comparison.
- Cost. Appendix F reports the cost of our default of 16 candidates x 96 Euler steps. On a single H200, a single Euler step is as accurate as 96 on five of six configurations. At that setting, STAGE (16 candidates plus support-based selection) takes 9-13 ms per task, compared with 12-47 ms for a tuned SGD LoRA fit on the same support set. We state in the same place that STAGE ties the gradient fit on two configurations and is 1-6 pp behind on four, and that it still uses more FLOPs than the fit: the benefit is gradient-free, fixed-latency adaptation, not higher accuracy or lower total compute.
Response to Text: dataset-level priors vs task-specific adaptation
We thank the reviewer for careful reading and thoughtful comments. We provide detailed responses below.
Q1: Text-task results may reflect dataset-level priors rather than task-specific adaptation. Could the authors provide support-shuffling, unconditional generation, or dataset-identity baselines to clarify this distinction?
A4: Thank you for raising this question. We ran the requested controls for text: support shuffling, unconditional generation, and a dataset-identity baseline (Appendix E.4, Table 26 of the revised paper). We first clarify that in our formulation each sampled task has its own fitting, support, and query examples, with an independently fitted target LoRA. These tasks simulate repeated few-shot adaptation settings and can share an underlying task distribution. Support from another task of the same dataset can therefore remain informative for adaptation, consistent with our task construction. All controls use the same generator, test tasks and query questions, with six candidates per task, and report the mean over candidates (no selection).
| Commonsense (50 tasks per dataset) | ART | PIQA | ECQA | e-SNLI | CommonGen | Average |
|---|---|---|---|---|---|---|
| Frozen model | 67.64 | 70.08 | 65.52 | 55.76 | 15.34 | 54.87 |
| STAGE, unconditional (no support set) | 65.07 | 69.56 | 64.57 | 67.63 | 18.35 | 57.04 |
| STAGE, support from a different dataset | 33.43 | 74.88 | 64.63 | 62.73 | 20.24 | 51.18 |
| STAGE, support from another task of the same dataset | 70.71 | 73.67 | 68.33 | 88.02 | 20.09 | 64.16 |
| STAGE, own support set | 70.70 | 73.72 | 68.38 | 88.08 | 20.06 | 64.19 |
| Dataset LoRA (one LoRA per dataset, 2,000 training examples) | 72.84 | 74.70 | 66.54 | 82.50 | 23.36 | 63.99 |
Reading. On the commonsense benchmarks, STAGE achieves an average score of 64.19 with its own support and 64.16 with support from another task within the same dataset, compared with 57.04 without conditioning, 51.18 with support from a different dataset, and 54.87 for the frozen model. These results support the role of conditioning in identifying task-relevant information shared across sampled tasks: for text, the support set identifies the dataset-level task, as stated in the revised paper (Appendix E.4). The dataset-specific LoRA baseline achieves a comparable 63.99, consistent with substantial shared adaptation needs within each dataset; unlike STAGE, it requires a dataset label and training on that dataset. In vision, where each task is a different set of classes, the same exchange of support sets removes the gain (matched advantage of +3.6 to +52.4 pp; Table 24, Appendix E.3). Crucially, STAGE also transfers beyond the training datasets. Without retraining or dataset identifiers, the same generator improves over the frozen model on all five datasets excluded from bank and generator training, averaging 75.16 versus 74.02 for the frozen model and 72.92 for unconditional generation (Table 29, Appendix E.6). Together, these results support our goal of learning transferable adaptation from sampled task-adapter pairs and generating useful adapters from representative support examples, including on unseen datasets.
Response to Seen classes and datasets; no unseen-class / OOD tests and Combination of existing techniques; missing baselines; scaling analysis
Dear Reviewer dnfA,
Thank you for your careful reading and thoughtful comments. We provide detailed responses below.
W1: Most evaluation tasks reuse seen classes or datasets. Without tests on unseen classes and out-of-distribution tasks, it is unclear whether STAGE learns genuine task adaptation or relies on training priors.
A1: Thanks for the question. We would like to clarify that unseen tasks and unseen classes are two distinct forms of generalization. Our image experiments evaluate new adaptation tasks within a pretrained classifier's label space, using unseen class combinations and held-out images. Generalization to excluded class identities is a stronger setting that requires additional effort in algorithm design (e.g., a different training loss) and would be an interesting direction for future work. Nevertheless, we evaluated this setting as the reviewer suggests, alongside transfer to held-out text datasets.
Additional evaluations. We evaluated STAGE and all six compared generators on the 20 held-out Tiny-ImageNet classes under the same seen-180 protocol. None improves over the frozen classifier in this setting, whereas STAGE retains its gains on unseen combinations of the 180 training classes (Table 28, Appendix E.5 of the revised paper). We also evaluated the text generator, without retraining, on five held-out datasets, where it improves the average accuracy from 74.02% to 75.21% (Table 29, Appendix E.6).
W2: STAGE mainly combines existing SVD normalization and Flow Matching techniques. Key baselines are missing, and the benefits of scaling adapter supervision lack deeper analysis.
A2: Thanks for the comments. We answer the three parts in turn.
(i) Novelty. We do not claim new primitives. The contribution is the combination of (a) scale: 100K independently optimized adapters per bank, where prior weight generators use a few hundred checkpoints or about a thousand tasks, together with a measurement of how generation quality scales from 1K to 100K targets; (b) canonicalization: on the same effective updates, removing the LoRA gauge by SVD is worth 2.6-12.0 pp, and without it the generator stays at the frozen classifier (Table 22, Appendix E.2 of the revised paper); and (c) stochastic generation with support-only candidate selection, which a deterministic hypernetwork cannot use: the matched deterministic model trails STAGE by 0.3-5.1 pp (Table 22). We revised the contribution list in the introduction accordingly.
(ii) Baselines. Table 3 already compares against adaptations of p-diff, RPG, DeepWeightFlow, and DnD on the same bank and episodes, and reports DnD ahead of STAGE on three of six configurations. We have added FLoWN-style and HyperTuning-style baselines on all six vision configurations, using the same setting (same bank, split, episodes, and candidate selection). Table 3 (Section 6) reports them as columns FLoWN and HyperTune, and its caption describes how we implemented each (FLoWN has no public code, so we re-implemented it). Support-selected accuracy (%), 100 official episodes, full-class scoring:
Both matched baselines trail STAGE on every configuration: the FLoWN-style generator by 1.1-14.7 pp (it falls below the frozen classifier on both ViT-S/16 configurations), and the matched deterministic hypernetwork by 0.3-5.1 pp. The deterministic model also emits a single adapter per task, so it has no best-of-16 headroom.
(iii) Scaling analysis. Increasing the number of sampled training tasks improves adaptation across all six image configurations at a fixed generation budget (Figure 3 and Appendix E.1 of the revised paper). The smaller marginal gains in some settings are observed under a fixed generator architecture and do not establish an intrinsic limit of task scaling; scaling task supervision and the generator jointly is an important direction for future work.
Our Q4 answer clarifies what the bank-size study separates: target quality is constant across bank sizes, seed variance is tenfold smaller than the size effect, and task count and optimization runs vary together by design. Thus, the paper's claim concerns bank size, not either factor alone. A control (Table 20, Appendix E.1) separates them on CIFAR-100 ViT: 100K adapters trained on 10K tasks score 94.22, versus 89.12 for the 10K bank and 95.08 for the 100K bank, showing that both more fitted adapters and more distinct tasks help.