- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose an idea of using distributions of LoRA parameters for LLM adaptation. This is implemented using hypernetworks which predict the LoRA weight updates. In order to predict distributions of LoRA parameters, authors using reparameterisation trick together with hypernetworks which predict mean and variance of the Gaussian distribution of LoRA parameters. On top of that, the authors also propose to model mixture over reference LoRA parameters and they predict the mixing probabilities.
The results in the paper are quite strong and consistent -- using this distributional idea seems to improve empirical performance and it also proposes a new axis of scaling.
Critical strengths
Strong idea of using distributional hypernetworks either with Gaussian or mixing distributions.
Overall, easy to follow and understand the idea.
The experiments demonstrate consistent results and improvements in practice.
Critical weaknesses
Not a big weakness but I have a few questions / concerns:
-
How do we come up with reference LoRA parameters for mixing distribution? Are these trained or pretrained? If they are trained, how do you ensure there is diversity?
-
Is "reg" represents Gaussian and "mix" mixing? If yes, maybe "Gaussian" is a better name than "reg"?
-
When you train Gaussian and "mixing", you basically maximize the likelihood of the data under your model. But typically, when working with distributions, we are interested in the posterior distribution over the parameters. In practice, it would involve you adding a prior to the objective. Have you explored it?
-
Can you analyse the rank of the learned variance via your procedure?
-
Have you explored using maximum-a-posteriori estimate in your regime?
-
I am not sure I understand why I am not sure I fully understand what is "detereministic" version of your idea. Is it that you just turn off the noise? Could you please explain?
-
It is not fully clear why MoL in many cases does not outperform LoRA. Your mixing results suggest that mixing should generally help. How can you explain it? Do you use the same reference LoRA for MoL and your method?
Questions for authors
See above.
Summary of Rebuttal and Reviewer Feedback
We thank reviewers iv1j, TVJc, and jUjh for their thoughtful feedback. The reviewers highlighted several strengths: weight-space test-time scaling as a novel direction (iv1j, TVJc), using the query alone as an adaptation signal (iv1j, TVJc), predicting distributions over LoRA updates (jUjh), and the best-of-K analysis across multiple models and benchmarks (TVJc, iv1j, jUjh). Below, we summarize the main concerns and our revisions.
We use free-form generation with a maximum of 512 new tokens and count outputs without a valid extracted answer as incorrect. We also report the unadapted Qwen3-1.7B reference: 36.5 under greedy decoding, 38.2 at K=1,T=0.8, and 43.0 at K=32.
Under matched greedy decoding at K=1, the sampled regression and mixing hypernetworks achieve 50.4 and 55.1, compared with 46.0 for the strongest standard baseline. We also clarify that greedy decoding removes token-level randomness but not weight sampling: 48.4 evaluates the predicted mean LoRA, whereas 50.4 evaluates one sampled LoRA followed by greedy decoding. We added these protocols and distinctions to the paper and will report method-level invalid-answer rates and complete matched-decoding curves in the camera-ready version.
Before test-time sampling, distributional training improves the regression hypernetwork’s mean from 41.3 to 48.4 under the same architecture and data. At K=32, learned weight sampling achieves 59.9, compared with 58.0 for random-direction noise and 56.7 for token sampling. The random perturbation is rescaled per matrix to match the learned perturbation norm, so only its direction is randomized.
We also clarify that weight sampling is not uniformly superior. On Qwen3-4B at K=32, deterministic token sampling is slightly stronger for regression (68.3 vs. 66.9) and mixing (72.4 vs. 71.1). We therefore narrow the claim: weight sampling is strongest at small K and on smaller models, while remaining competitive on larger models. We will add standard-LoRA noise with validation-selected scales and multiple seeds in the camera-ready version.
Figure 5 is a separate transfer experiment over ARC-Challenge, GSM8K, MedQA, MMLU-STEM, SciQ, and MBPP. It uses 200 fixed held-out examples and disjoint construction pools of 16, 64, or 256 prompts. The hypernet processes only the construction prompts; the resulting LoRA is frozen and evaluated exclusively on the held-out set.
For regression hypernet, factor averaging is a rank-preserving summary rather than the exact mean update, whose rank can grow to Nr. For the mixing hypernet, coefficient averaging remains exact. We now state the benchmark set, protocol, and intended interpretation explicitly.
MoL uses 54 rank-8 LoRAs trained togther, it does not use the 50 frozen reference LoRAs. For the mixing hypernet, the 50 reference LoRAs are trained on disjoint data shards and then frozen before training the mixer.
Because mixing receives additional training exposure, we do not treat its comparison against standard baselines as budget-matched. Our controlled comparisons are within each family, deterministic versus distributional regression and deterministic versus distributional mixing, where architecture, data, and training schedules are shared. We corrected the MoL and training descriptions accordingly.
Unlike HyperTuning and Text-to-LoRA, our method requires no demonstrations, instructions, or task descriptions. Unlike HyperDecoders, it predicts a distribution rather than one point adaptation and uses it for test-time scaling. Weight-space scaling is complementary to token-space scaling, and the two can be combined.
The learned distribution is a task-optimized search distribution, not a Bayesian posterior. Its mean is the MAP estimate for the direct Gaussian variant, while the deterministic hypernetwork is trained separately as a point predictor. The hypernetwork requires approximately 0.58–0.79 GB in BF16, each sampled LoRA requires 8–21 MB, and LoRA projections add less than approximately 1% to the MLP linear-layer FLOPs.
We replaced “4.6–11.7%” with “up to 11.8 percentage points,” changed “consistently outperforms” to “generally outperforms,” clarified the different Figure 3/5 protocols, and updated Neural Thickets to its ICML 2026 publication.
We hope the new evidence and revisions address the remaining concerns of iv1j and TVJc. If so, we respectfully ask them to reconsider their assessments and update their scores as well. We remain happy to address any further questions.