acceptodds
Under review as a conference paper at ICLR 2027

Learning to Predict Distributions over Weight Updates for Test-Time Adaptation

Abstract

Hypernetworks have recently shown success in dynamically adapting the parameters of Large Language Models (LLMs) at runtime based on signals such as task descriptions or additional demostrations. Here we ask: how much adaptation signal can be obtained using only the input query to an LLM?. To answer this, we study query-conditioned Hypernetworks for LoRA estimation. Further, we introduce distributional Hypernetworks, able to produce not only point estimates of parameter adaptors, but also a distribution over possible LoRAs. For this we propose a simple end-to-end loss using a differentiable Monte Carlo approximation and explore multiple distribution parametrizations including regression and convex combination variants. Results show that even using the mean of the learned distribution can outperform deterministic hypernetworks. Crucially, the learned distribution enables a different form of test-time scaling: instead of spending additional compute only by sampling more token sequences from a fixed model, we sample weight updates, yielding multiple adapted models for the same query. Performance improves as more weight samples are considered and remains stronger than corresponding token-sampling adaptation baselines. Finally, we find that generated updates can transfer across queries, suggesting that the hypernetwork learns reusable structure in how the model should adapt. Together, these results show that query-conditioned distributions over weight updates can support both adaptation and test-time scaling.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Gradient-free Updates, Probabilistic Modeling, Test-time Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jUjhOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose an idea of using distributions of LoRA parameters for LLM adaptation. This is implemented using hypernetworks which predict the LoRA weight updates. In order to predict distributions of LoRA parameters, authors using reparameterisation trick together with hypernetworks which predict mean and variance of the Gaussian distribution of LoRA parameters. On top of that, the authors also propose to model mixture over reference LoRA parameters and they predict the mixing probabilities.

The results in the paper are quite strong and consistent -- using this distributional idea seems to improve empirical performance and it also proposes a new axis of scaling.

Critical strengths

Strong idea of using distributional hypernetworks either with Gaussian or mixing distributions.

Overall, easy to follow and understand the idea.

The experiments demonstrate consistent results and improvements in practice.

Critical weaknesses

Not a big weakness but I have a few questions / concerns:

  1. How do we come up with reference LoRA parameters for mixing distribution? Are these trained or pretrained? If they are trained, how do you ensure there is diversity?

  2. Is "reg" represents Gaussian and "mix" mixing? If yes, maybe "Gaussian" is a better name than "reg"?

  3. When you train Gaussian and "mixing", you basically maximize the likelihood of the data under your model. But typically, when working with distributions, we are interested in the posterior distribution over the parameters. In practice, it would involve you adding a prior to the objective. Have you explored it?

  4. Can you analyse the rank of the learned variance via your procedure?

  5. Have you explored using maximum-a-posteriori estimate in your regime?

  6. I am not sure I understand why I am not sure I fully understand what is "detereministic" version of your idea. Is it that you just turn off the noise? Could you please explain?

  7. It is not fully clear why MoL in many cases does not outperform LoRA. Your mixing results suggest that mixing should generally help. How can you explain it? Do you use the same reference LoRA for MoL and your method?

Questions for authors

See above.

Reviewer_TVJcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper trains a hypernetwork that generates a LoRA from the input query alone, without task descriptions or examples. It also predicts a distribution over LoRAs, so different LoRAs can be sampled at test time and combined by majority voting. Experiments on five small models and seven benchmarks show the generated LoRAs match or beat standard LoRA and full fine-tuning, and that sampling weights is competitive with sampling tokens.

Critical strengths

(1) The idea of spending test-time compute in weight space is new and interesting. Sampling different LoRAs per query, instead of only sampling more tokens, gives a different way to get diverse answers. If the results hold, this could be a useful direction for test-time scaling.

(2) The setup only needs the input query, with no task description or examples. This makes the method easy to use in practice. The analysis of training objectives is also helpful, since it shows why best-of-K keeps the distribution from collapsing.

Critical weaknesses

(1) The evaluation looks broken. Many multiple-choice scores in Table 4 are well below random guessing (25% for four options), for example 9.2 on MedQA and 10.6 on GPQA for Gemma-3-1B, and 17.2 on MMLU-STEM for Llama-3.2-1B. Most likely the answers were not extracted correctly. If so, the gaps between methods may mostly reflect which model follows the output format better. The base model without any adaptation is also not reported, so there is no reference point. Since all main claims rely on these numbers, this is my biggest concern.

(2) Figure 3 and Figure 5 do not agree. Full FT on Qwen3-4B gets 64.8 in Figure 3 but around 78 in Figure 5, and other models show similar jumps of over 10 points. Full FT is the same model in both experiments, so the evaluation setup must have changed, but the paper never explains this.

(3) Several claims in the text are stronger than the tables. The intro says 9 benchmarks, Section 5 uses 7, and Appendix F.1 says 8. The claimed 4.6–11.7% gain of the distribution mean over the deterministic hypernet does not match Table 4, where the gap ranges from -1.8 to +11.8. Finding 2 says “consistently outperforms,” yet the deterministic version wins for the mixing hypernet on Qwen3-4B (67.1 vs. 65.3) and Llama-3.2-3B (56.6 vs. 56.0).

(4) Is the gain coming from the learned distribution, or just from more parameters and added noise? The hypernetworks have about 200–400M parameters while LoRA has about 5M, and the distributional version has around 100M more than the deterministic one (295M vs. 194M on Qwen3-1.7B). A simple baseline that adds random noise to a trained LoRA and then votes, as in Neural Thickets [29], is missing. Also, at K=32 in Table 5, weight sampling is below its deterministic counterpart in several cases, such as 66.9 vs. 68.3 and 71.1 vs. 72.4 on Qwen3-4B.

(5) Section 5.4 has its own problems. The hypernet sees prompts from the test benchmark while the baselines do not, so the comparison is not fair. It also averages the A and B factors separately, which Appendix C itself says introduces untrained cross terms. And if one averaged adapter works this well, it weakens the case for generating a new LoRA for every query.

Questions for authors

(1) Why are many multiple-choice scores below random guessing? Please explain how answers are extracted and how failed extractions are counted. Scores for the base model without adaptation would also help.

(2) Full FT on Qwen3-4B is 64.8 in Figure 3 but around 78 in Figure 5. What changed in the evaluation between these two experiments?

(3) I would like to see a baseline that adds random Gaussian noise to a trained LoRA and uses majority voting, at the same K. Does the learned distribution still win?

(4) How many epochs were Full FT, LoRA, and MoL trained for? Were they matched to the hypernetwork?

Reviewer_iv1jOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores an interesting question: how much adaptation signal can be obtained from the input query alone for an LLM? To study this, the authors use hypernetworks to estimate LoRA. Building on this, they propose distributional hypernetworks, which predict not only point estimates of LoRA weights but also a distribution over them. To train these, they use a differentiable Monte Carlo approximation. The learned distribution enables a different form of test-time scaling: instead of spending additional compute solely on sampling more token sequences, the authors sample weight updates that yield multiple adapted models for the same query. They also find that the generated updates can transfer across queries, suggesting that the hypernetwork learns a reusable structure for how the model should adapt.

Critical strengths
  • I liked the way the paper proposes to update model parameters through hypernetwork-generated LoRAs. I acknowledge that the idea is not entirely new, since hypernetworks have already been used to generate adapters, but its application here is well motivated.

  • It is good to see the method demonstrated across multiple benchmarks.

  • In my opinion, the extraction of the adaptation signal from a single prompt is a more significant contribution than the rest, and it is genuinely innovative.

  • Finding 3 deserves more emphasis. To my knowledge, scaling test-time compute by sampling in weight space is a novel direction, as existing methods mostly scale by generating more reasoning or output tokens from fixed weights.

Critical weaknesses
  • The "recent evidence" cited around line 176 rests on an unpublished preprint, which makes me question its credibility, especially since the motivation for the method leans on this hypothesis.

  • Figure 4 is interesting, as sampling weights clearly helps accuracy. However, I wonder why accuracy is already much higher than the baselines with very few samples. Is this because the method can effectively infer the adaptation signal, or is it due to other factors, such as differences in the decoding setup?

  • Despite the new and quite surprising results, I would like more detail on the experiment behind Figure 5, where updates generated from one set of queries transfer to held-out queries, and a technique akin to weight sharing works well for this method.

  • Regarding the hypernetwork, I agree that it comes with its own cost, as the authors state. I would like to know the specifics of this cost and the main advantages over conventional methods of scaling test-time compute.

Questions for authors
  • How does the method differ conceptually and empirically from existing instance-level or hypernetwork adaptation methods such as HyperDecoders, Text-to-LoRA, and HyperTuning? Can you show that the gain comes from the query-only conditioning, for example, against a constant or shuffled query? Is Figure 5 evidence for this?
  • In Figure 4, why is accuracy already high at K = 1? Please add a control in which baselines also use greedy decoding, and report how much of the gap comes from distributional training vs. the decoding protocol.
  • What are the latency, memory, and FLOPs of weight sampling vs token sampling at the same K? Weight sampling requires a separate, adapted model for each sample, so it cannot share the prompt's prefill cache.
  • Does the method depend on the claim that many diverse task-improving specialists exist near the pretrained weights (reference )?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Reviewer Feedback

We thank reviewers iv1j, TVJc, and jUjh for their thoughtful feedback. The reviewers highlighted several strengths: weight-space test-time scaling as a novel direction (iv1j, TVJc), using the query alone as an adaptation signal (iv1j, TVJc), predicting distributions over LoRA updates (jUjh), and the best-of-KK analysis across multiple models and benchmarks (TVJc, iv1j, jUjh). Below, we summarize the main concerns and our revisions.

Concern C1 (TVJc, iv1j): Evaluation validity and decoding differences.

We use free-form generation with a maximum of 512 new tokens and count outputs without a valid extracted answer as incorrect. We also report the unadapted Qwen3-1.7B reference: 36.5 under greedy decoding, 38.2 at K=1,T=0.8K=1,T=0.8, and 43.0 at K=32K=32.

Under matched greedy decoding at K=1K=1, the sampled regression and mixing hypernetworks achieve 50.4 and 55.1, compared with 46.0 for the strongest standard baseline. We also clarify that greedy decoding removes token-level randomness but not weight sampling: 48.4 evaluates the predicted mean LoRA, whereas 50.4 evaluates one sampled LoRA followed by greedy decoding. We added these protocols and distinctions to the paper and will report method-level invalid-answer rates and complete matched-decoding curves in the camera-ready version.

Concern C2 (TVJc, iv1j): Whether the gains come from learned distributions or arbitrary noise.

Before test-time sampling, distributional training improves the regression hypernetwork’s mean from 41.3 to 48.4 under the same architecture and data. At K=32K=32, learned weight sampling achieves 59.9, compared with 58.0 for random-direction noise and 56.7 for token sampling. The random perturbation is rescaled per matrix to match the learned perturbation norm, so only its direction is randomized.

We also clarify that weight sampling is not uniformly superior. On Qwen3-4B at K=32K=32, deterministic token sampling is slightly stronger for regression (68.3 vs. 66.9) and mixing (72.4 vs. 71.1). We therefore narrow the claim: weight sampling is strongest at small KK and on smaller models, while remaining competitive on larger models. We will add standard-LoRA noise with validation-selected scales and multiple seeds in the camera-ready version.

Concern C3 (iv1j, TVJc): Interpretation of Figure 5.

Figure 5 is a separate transfer experiment over ARC-Challenge, GSM8K, MedQA, MMLU-STEM, SciQ, and MBPP. It uses 200 fixed held-out examples and disjoint construction pools of 16, 64, or 256 prompts. The hypernet processes only the construction prompts; the resulting LoRA is frozen and evaluated exclusively on the held-out set.

For regression hypernet, factor averaging is a rank-preserving summary rather than the exact mean update, whose rank can grow to NrNr. For the mixing hypernet, coefficient averaging remains exact. We now state the benchmark set, protocol, and intended interpretation explicitly.

Concern C4 (TVJc, jUjh): Training budgets, reference LoRAs, and MoL.

MoL uses 54 rank-8 LoRAs trained togther, it does not use the 50 frozen reference LoRAs. For the mixing hypernet, the 50 reference LoRAs are trained on disjoint data shards and then frozen before training the mixer.

Because mixing receives additional training exposure, we do not treat its comparison against standard baselines as budget-matched. Our controlled comparisons are within each family, deterministic versus distributional regression and deterministic versus distributional mixing, where architecture, data, and training schedules are shared. We corrected the MoL and training descriptions accordingly.

Concern C5 (iv1j, jUjh): Positioning, interpretation, and cost.

Unlike HyperTuning and Text-to-LoRA, our method requires no demonstrations, instructions, or task descriptions. Unlike HyperDecoders, it predicts a distribution rather than one point adaptation and uses it for test-time scaling. Weight-space scaling is complementary to token-space scaling, and the two can be combined.

The learned distribution is a task-optimized search distribution, not a Bayesian posterior. Its mean is the MAP estimate for the direct Gaussian variant, while the deterministic hypernetwork is trained separately as a point predictor. The hypernetwork requires approximately 0.58–0.79 GB in BF16, each sampled LoRA requires 8–21 MB, and LoRA projections add less than approximately 1% to the MLP linear-layer FLOPs.

Concern C6 (TVJc, iv1j): Claims and citation status.

We replaced “4.6–11.7%” with “up to 11.8 percentage points,” changed “consistently outperforms” to “generally outperforms,” clarified the different Figure 3/5 protocols, and updated Neural Thickets to its ICML 2026 publication.

We hope the new evidence and revisions address the remaining concerns of iv1j and TVJc. If so, we respectfully ask them to reconsider their assessments and update their scores as well. We remain happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.