acceptodds
Under review as a conference paper at ICLR 2027

Deployment-Time Mixture Optimization for Large Language Models

Abstract

Domain-specific fine-tuning produces language models with complementary strengths, yet an unseen deployment distribution may not be well matched by any single model. We introduce Hierarchical Routing, a mixture optimization method that uses an unlabeled batch to identify the Top-\(H\) most relevant sources and optimizes a weight-space combination of their selected models using autoregressive loss. The hierarchy reduces optimization from the full model bank to a low-dimensional combination of relevant candidates, while preserving complementary specializations. We instantiate the bank with models spanning source domains and robustness levels and evaluate on nine Pile domains and deployment streams whose distributions change across batches. The experiments show that routing is beneficial under heterogeneous and changing conditions, despite higher deployment cost, and that applying test-time adaptation after routing further improves performance. The results show that adaptation to unseen distributions can be solved as navigation and composition in a precomputed model space.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
model routing, test-time adaptation, distribution shift, model composition, weight-space interpolation, language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AEeNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces “deployment-time robustness routing,” which really is a mixture of techniques: (1) a way to produce a “model bank” of finetuned, specialized models from a single model, spanning two dimensions: specialization and “robustness”, (2) a way to combine models from this model bank at test time to make better predictions than using just the original single model checkpoint.

Critical strengths

It is interesting how Hierarchical Routing+ demonstrates complementary benefits of routing and local test-time adaptation.

Critical weaknesses

Model bank construction: A fair bit of the paper (and appendix) are devoted to discussing the “robustness” parameter lambda and the two-dimensional filter bank (specialization and robustness). However, this model bank is not compared to other approaches.

Questions for authors

Why this particular model bank design? Did you consider alternatives? Why do we need the robustness dimension? Do you have baselines without it?

Small detail: Please fix the Figure 2 legend - it is all the same shape.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer AEeN

We appreciate the reviewer's recognition of the complementary benefits of routing and test-time adaptation. Our model-bank design is motivated by the observation that different deployment distributions favor different robustness levels, as illustrated in Table 1. This justifies considering robustness alongside domain specialization. We agree that comparing our ERM/KL-DRO bank with an ERM-only bank would help quantify the contribution of the robustness dimension, and will consider this ablation in future work. We will also correct the Figure 2 legend.

Thank you for your reply. I will be interested to see that explored more in-depth in future work. As it stands, I maintain my rating.

Reviewer_BimKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether a language model can pick its specialization and its level of robustness at deployment time, after it sees unlabeled data, instead of fixing them during training. The setting is language modeling on Pile domains. The deployment data arrive in batches, and their domain may be unknown or may change over time.

The idea is to build a bank of fine-tuned checkpoints offline. Each checkpoint comes from one source domain and one robustness level. The robustness level is set by a KL-DRO loss with a temperature lambda, which puts more weight on high-loss training samples when lambda is small. At deployment, the method scores bank models by their next-token loss on the incoming batch. For each source it keeps the best lambda, keeps the top-H sources, and then searches for weights that average these checkpoints (plus the pretrained model) in weight space. The search uses exponentiated-gradient steps on the batch loss, with several random starts. A cheaper variant, Hierarchical Routing+, routes once on the first batch and then runs online Tent on LayerNorm parameters.

Experiments use GPT-2 small and medium with a bank of three sources (PubMed Central, ArXiv, PG-19) and ten lambda values each. The paper evaluates on six held-out Pile targets in a stationary "episodic" setting (Table 3, Table 7) and on one non-stationary stream of 24 mixed batches (Table 2). Appendices add a leave-one-source-out study, an EG convergence check, and a small top-H ablation. The main finding is that Hierarchical Routing+ gives the lowest loss among the deployable methods, while plain Hierarchical Routing is safe but often gives little gain and is slow.

Critical strengths

S1. A clear and practical framing. The paper treats deployment-time adaptation as search in a fixed bank of models, indexed by domain and robustness level. Nothing in the bank is retrained, so stored specializations are not forgotten. The motivating analysis supports the framing. Figures 1 and 2 show that no single fine-tuned model is best everywhere, and that token-level KL only partly predicts transfer. Table 1 suggests that the best robustness level changes with the shift. The promised release of the full bank could also be useful to others working on routing and merging.

S2. Careful reporting of cost and of where the method does not help. The paper reports wall-clock cost per batch for every method (Tables 2, 3, 4). It openly states that plain Hierarchical Routing is expensive and often returns the pretrained model when the bank does not cover the target. It also flags targets where the oracle is degenerate (FreeLaw, OWT2) instead of hiding them. The EG convergence check in Appendix B.2 and the leave-one-source-out study in Appendix B.7 show some effort to understand when routing works and when it does not.

Critical weaknesses

W1. The evaluation does not isolate the value of routing, and the headline result is mostly explained by test-time adaptation on the evaluation batch. This is my biggest concern.

  • The same batch is used to choose or fit the model and to report its loss (Section 5.2, Appendix B.4). Hard Routing picks the bank model with the lowest loss on the batch, and Hierarchical Routing fits continuous weights to that same loss with several restarts. Since the pretrained model is in the bank, these methods cannot do worse than Pretrained on the reported number by construction. The "no worse than the best candidate" guarantee in Section 4 is also only a statement about the training objective on B. With 16 sequences per batch, some of the reported gain may be fitting to the batch itself. A held-out evaluation (route on one batch, score on the next batch or on a held-out half) is needed to show that routing finds a better model and not just a better fit to the batch.
  • The best method, Hierarchical Routing+, routes only once on the first batch with a truncated search, and then runs online Tent for the rest of the stream (Appendix B.4). Most of its gain over plain routing therefore comes from Tent. The Tent baselines in Tables 2, 3, and 7 all start from Best Single FT, which is the worst starting point in almost every column. The natural control, online Tent started from the pretrained model, is described in Appendix B.1 but never reported. This matters because Hard Routing and Static Hierarchical Routing return the pretrained model on most episodic targets in Table 3. So it is likely that Hierarchical Routing+ starts from or near the pretrained model on those targets. If online Tent from Pretrained matches it, the bank adds little.
  • Plain Hierarchical Routing, which is the method in the title and Section 4, gives very small or zero gains over Pretrained on most episodic targets in Table 3. GitHub is the clear exception. It costs over a hundred times more than the static baselines.
  • On the non-stationary stream (Table 2), the per-batch spread is larger than the gaps between the routing variants. The text says Routing+ is "statistically indistinguishable" from Hierarchical Routing, but no test is reported. All results use one seed (seed 42), and the bank itself is trained once. The lambda differences in Table 1 are also small, so claims such as "ERM is never optimal" need repeated training runs to be reliable.
  • In the non-stationary stream, Routing+ does not reroute after the first batch (Appendix B.4). So its result does not support the claim that the deployed model adapts by routing as the distribution changes.

W2. The experiments do not test the method that is described, and several claims go beyond the evidence.

  • The bank has S = 3 sources and the method uses H = 3 (Section 5.3). So the Top-H screening step, which is the "hierarchy" that is supposed to reduce the search, removes nothing. The method is then the per-source lambda pick followed by EG over all sources. Also, scoring needs the loss of every bank model on the batch, so the hierarchy does not save loss evaluations either. The H ablation in Appendix B.8 uses H = 4 with only three sources, which is unclear, and covers only one target where routing already does nothing.
  • The contributions list "weight interpolation filling in intermediate robustness levels without extra fine-tuning" (Section 1). I could not find any experiment on this. Appendix A says it gives "the proof connecting an estimated deployment shift to the corresponding robustness level", but no such proof appears.
  • The DRO setup is not consistent with the math. Section 2 and Appendix A.2 say that small lambda gives the strongest reweighting and large lambda approaches ERM. Yet the grid in Table 5 labels lambda = 0 as ERM, and the GPT-2 small grid goes down to 1e-5, where the loss is close to training on the single worst sequence in each mini-batch. Appendix B.4 also mentions a "KL-DRO mixed loss" with gamma = 1.0, which is not defined anywhere. The reader cannot tell what objective each checkpoint was trained with.
  • The title and abstract refer to large language models, but all results use GPT-2 small and medium. The medium bank is smaller than the small bank (Appendix B.6), and the leave-one-source-out results are reported as incomplete ("6 of 10 targets", "eight of the ten completed so far", Appendix B.7).
  • The "Why Tent is competitive" paragraph (Section 5.3) refers to a non-stationary bank with four sources and three targets. This does not match the setup, which states the same three-source bank and six targets.
  • On novelty, choosing merge coefficients from unlabeled test data is already done by AdaMerging (Yang et al., 2024) and Local Mixtures of Experts (Bertolissi et al., 2025), and LoraHub (Huang et al., 2024) learns composition weights over a library of adapters from few examples. The new part here is mainly the robustness axis of the bank. None of these methods is used as a baseline, so the paper does not show that its routing is better than existing test-time merging.
Questions for authors

Clarifications:

Q1. Held-out evaluation. Can you report results where the routing weights (and Tent updates) are fitted on one batch and the loss is measured on a different batch from the same target, or on a held-out half of each batch? If plain Hierarchical Routing and Routing+ keep a clear margin over Pretrained and over the Tent baselines in that setting, across a few seeds, my main concern would be largely resolved.

Q2. Tent from the pretrained model. Please report online Tent started from the pretrained model with the same hyperparameters as Routing+, on Tables 2, 3, and 7. Also please report which bank model or weights Routing+ starts from on each target. This would show how much the bank adds beyond Tent.

Q3. Hierarchy. With S = 3 and H = 3, the Top-H step is not active. Can you show results with a larger bank (for example the 9-source bank of Appendix B.7) and H smaller than S, and compare cost and loss against flat EG over all sources? How was H = 4 run in Appendix B.8 with only three sources?

Q4. DRO objective. What exactly is the "KL-DRO mixed loss" with gamma = 1.0? Is lambda = 0 trained with plain cross-entropy as a special case? At lambda = 1e-5, is training effectively on the worst sequence per mini-batch, and is that stable?

Q5. Variance. How many training seeds were used for the bank and for the deployment runs? Can you add standard errors or paired tests for Table 1, Table 3, and the comparison between Hierarchical Routing and Routing+ in Table 2?

Q6. Merging baselines. Can you compare with an existing test-time merging method, such as AdaMerging-style coefficient learning on the same bank, or a LoraHub-style search? This would show whether the hierarchical routing itself improves on known approaches.

Minor:

Q7. The contribution about filling intermediate robustness levels by weight interpolation, and the proof promised at the start of Appendix A, do not appear in the paper. Please add them or remove the claims. Please also complete the leave-one-source-out runs.

Q8. Small issues: "btach" (Figure 3 caption), "uncoming" (Appendix B.1), "Degreee" (Table 4), "considerable lower cost" (Section 5.3), "gpt-small" (Appendix B.7). Table 7 uses "Episodic TENT" and "New Order" without definition. Appendix B.4 lists an A100 with 128GB of memory, which seems incorrect. Mixed-source FT appears in Table 2 but not in Table 3.

Q9. The paper says the bank and code will be released. Please confirm that the deployment streams, the batch order, and the routing code will be released too.

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer BimK

We thank the reviewer for their thorough analysis and for raising important methodological questions. We would like to clarify that our Primary Episodic protocol already includes held-out evaluation through Static Hierarchical Routing and Static Tent. Both methods adapt only on the first batch and evaluate the fixed model on all 50 batches, 49 of which are unseen during adaptation. Regarding Routing+ initialization, on the first GitHub batch, the selected model combines three bank checkpoints: Pretrained (40.43%), PG-19 (λ=10−5\lambda=10^{-5}, 31.77%), and ArXiv (λ=10−3\lambda=10^{-3}, 27.80%). We also clarify that Pretrained is included in the Top-H selection. With three source domains, there are four candidates in total. Setting H=3H=3 retains the three best candidates, whereas H=4H=4 retains all four. Furthermore, our leave-one-source-out experiments in Appendix B.7 use a larger bank with S=9S=9 and H=3H=3, where Top-H selection is substantially more restrictive. Regarding the KL-DRO objective, our bank's KL-DRO checkpoints are trained using the mixed loss:

Lmix=(1−γ)LERM+γLDROL_{mix} = (1-\gamma)L_{ERM} + \gamma L_{DRO}

with γ=0.7\gamma=0.7. In contrast, the ERM checkpoints are trained separately using standard cross-entropy. We denote these checkpoints by λ=0\lambda=0 solely as a convention, not as the mathematical limit of the KL-DRO objective. For λ>0\lambda>0, smaller values concentrate more weight on high-loss sequences, whereas larger values approach ERM. Our training runs converged successfully for λ=10−5\lambda=10^{-5}. Finally, while existing model-merging methods are relevant, our focus is on deployment-time adaptation using a bank of specialized models. We therefore consider BTM and TTMM more directly relevant baselines for our setting. We will also correct the identified manuscript inconsistencies.

Reviewer_PapPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies deployment-time domain adaptation for autoregressive language modeling, where a pretrained LLM (GPT-2 in the experiments) performs next-token prediction on text from unknown or changing domains. The authors construct a checkpoint bank spanning source domains and KL-DRO robustness levels. Given an unlabeled deployment batch, Hierarchical Routing selects relevant candidates and optimizes their weight-space combination using the batch’s autoregressive loss; Routing+ additionally applies LayerNorm test-time adaptation. Experiments on nine Pile domains with GPT-2 Small and Medium cover stationary episodic targets and a non-stationary stream, using the same batch for adaptation and evaluation. Plain Hierarchical Routing provides modest gains at substantial computational cost. Routing+ achieves the strongest deployable results, including the lowest loss on all six Small episodic targets and a 4.57% average loss reduction over the pretrained model on the non-stationary stream.

Critical strengths

A simple and well-motivated deployment strategy. The method uses the incoming batch’s autoregressive loss as a self-supervised routing signal, selects relevant domain–robustness checkpoints, and optimizes a low-dimensional weight-space combination. It is intuitive and does not require training an auxiliary router.

Some evidence of generalization to unseen deployment distributions. Experiments cover heterogeneous and non-stationary Pile streams, two GPT-2 scales, and leave-one-source-out settings where the target domain is absent from the bank. The results suggest that the method can reuse related specializations, although its effectiveness remains dependent on model-bank coverage.

Critical weaknesses

Substantial computational and storage costs, with unclear scalability to larger LLMs. The approach requires training and storing many full checkpoints, scoring candidates, and performing repeated forward-backward passes during routing. Full Hierarchical Routing takes over 130 seconds per batch on GPT-2 Small, compared with roughly one second for ordinary inference. Extending this framework to billion- or trillion-parameter models could be prohibitively expensive without parameter-efficient checkpoints or a substantially cheaper routing mechanism.

The transductive evaluation does not demonstrate generalization to future batches. Each deployment batch is used for both adaptation and evaluation, so the results primarily show that the method reduces loss on the batch it has already observed. Moreover, evaluation is limited to GPT-2 and language-modeling loss, without downstream tasks or experiments on modern instruction-tuned LLMs.

Questions for authors
  1. Scalability to larger models. The experiments are limited to GPT-2 Small and Medium, with a reduced model bank at the Medium scale. Have the authors evaluated substantially larger models, or analyzed how offline training, checkpoint storage, and deployment-time routing costs scale with model size and bank size?

  2. Why full checkpoints instead of LoRA adapters? Why does the framework require (N) full fine-tuned models rather than a shared pretrained backbone with (N) domain- and robustness-specific LoRA adapters? Have the authors compared these designs, and would parameter-efficient checkpoints preserve the benefits of weight-space composition while reducing storage and routing costs?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer PapP

We appreciate the reviewer's encouraging assessment of our approach. We would like to clarify that our Primary Episodic protocol already includes held-out evaluation through Static Hierarchical Routing and Static Tent, with 49 batches unseen during adaptation. Regarding scalability, we constructed model banks for all four GPT-2 scales: Small, Medium, Large, and XL. The routing experiments reported in the paper focus on Small and Medium, while a systematic evaluation across all four scales remains future work. Finally, we considered LoRA adapters but focused on full fine-tuned checkpoints sharing a common pretrained initialization, allowing direct weight-space interpolation. We acknowledge the potential storage benefits of parameter-efficient approaches.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.