acceptodds
Withdrawn from ICLR 2027

Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Low-rank adaptation (LoRA) applies the same learned update to every input, even when an adapter must perform multiple tasks. Queryable LoRA is a parameter-efficient fine-tuning (PEFT) approach that adapts its low-rank update to each input by retrieving a sparse mixture of rank-space atoms from a shared memory. A block uses its current representation and a summary of earlier blocks to select the atoms. In the Instruction-Queryable LoRA (IQ) approach, an instruction embedding also guides the query and defines a prior over atoms. We show that the router solves a KL-regularized retrieval problem. We also demonstrate that its updates stay norm-bounded across routing switches and that the blockwise gradients factorize exactly. To test transfer, we trained one adapter on a mixture of source tasks and chose its checkpoint on validation data. We then evaluated it on held-out tasks with unseen instructions. Across model sizes, Queryable LoRA matched or exceeded LoRA in most paired runs. In the main comparison, both variants exceeded every baseline in average accuracy at a training cost close to LoRA's.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, LoRA, low-rank adaptation, adapters, routing, large language models, instruction tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fQYBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Queryable LoRA. The idea is to keep a global bank of small r×r matrices ("atoms") shared across all layers, and have a router mix a few of them once per block of layers. The mixture acts inside the LoRA bottleneck, i.e. ΔW = (α/r)B(I + gS_b)A with a learned gate g. The router looks at the current rank-space state and summaries of earlier blocks; the IQ variant also adds an instruction embedding to the query and uses it as a prior over atoms. The paper gives a variational characterization of the router, norm bounds, and gradient identities. For validation, one adapter is trained on five source tasks and tested on held-out MMLU/MMLU-Pro/GPQA with unseen instructions, on Qwen2.5 models from 1.5B to 14B. Reported gains over LoRA and dynamic baselines are about 1-2 points on average, at roughly 1.2x LoRA's training cost.

Critical strengths
  1. Cheap. Only +0.29% parameters over LoRA at 7B and 1.2x training time, while MoE-LoRA and Lily take 4.3x.

  2. The evaluation against LoRA is more careful than most PEFT papers: held-out tasks and instructions, several seeds, bootstrap intervals, and a full-size re-evaluation. The reporting is honest (e.g. seed-level losses on GPQA are disclosed) and the paper is easy to read.

Critical weaknesses
  1. Limited novelty, and the actual contribution is never isolated. Each component has a close precedent: a trainable r×r matrix inside the LoRA bottleneck is MoSLoRA's mixing matrix, shared banks are VB-LoRA/ShareLoRA, input-dependent routing is the MoE-LoRA line of work, cross-layer coupling is Lily, and instruction conditioning is Text-to-LoRA/qa-FLoRA. What is new is the specific combination. Worse, the paper's own ablation shows routing without the depth summary does worse than plain LoRA (0.752 vs 0.759, Table 5), so routing by itself is not what helps. There is also no static control (shared atoms with a fixed uniform mixture, no router), which seems like the obvious baseline to run. Since the instruction channel also does nothing at 7B (IQ is basically identical to Queryable), the isolated contribution boils down to passing a depth summary to the router.

  2. The theory is decorative and the evaluation sidesteps the motivating problem. Theorem 4.1 is the standard softmax/KL duality, Theorem 4.2 follows from convexity, Theorem 4.3 is the chain rule. None of this explains why depth conditioning helps. The method is motivated by multi-task interference but is only evaluated on transfer from 1,500 training examples to multiple-choice benchmarks, never on the interference setting itself. The hyperparameters also drift across settings (atom init std 0.30/0.15/0.05, gate init changed at 14B, three different checkpoint-selection rules), which makes the gains hard to attribute. Finally, "runtime close to LoRA" only holds for training and prefill: decode throughput is about 29% lower than LoRA (33.8 vs 47.5 tok/s).

Questions for authors
  1. What happens if the atom bank is used statically, i.e. a fixed uniform mixture with the router removed, at matched parameter count? If that matches the routed model, the routing mechanism is unnecessary. Please also spell out the concrete delta to MoSLoRA, VB-LoRA, and Lily.

  2. The depth summary seems to be the only component that matters. What does it carry that the block-entry state does not? Would summaries shuffled across blocks do equally well?

  3. Do the main results survive if atom init, gate init, and the checkpoint-selection rule are held fixed across all settings?

3 comments

Thank you for your time and review.

Regarding your points and questions, we reran our main 7B study with a single fixed setup. In this setup, we trained each approach and control method on the same 32 seeds.

[W2, Q3] Specification of fixed settings. Every run of this 32 seed study uses the settings in the table below.

SettingValue
TrainingQwen2.5-7B-Instruct, 3 epochs, learning rate 2e-4 (Lily 3e-4, as in its paper)
Checkpointepoch with the best validation score on the training tasks
Atom and gate initializationatom standard deviation 0.15, gate parameter η\eta = 1.0, for our method and every control
Router and atom learning rate25 times the base rate (2 times in the paper), for our method and its controls, set before the 32-seed study started
BaselinesMoSLoRA, VB-LoRA, Lily from their authors' code, with the respective hyperparameters of their papers
Seedsthe same 32 seeds for every method, paired in every table

As is common for top-k routers, Queryable and IQ add the load-balancing loss of Switch Transformer (Fedus et al., 2022) in these runs, with the weight 0.01 chosen beforehand on 11 separate development subjects and categories. The loss function acts on the top-k router; therefore, it applies only to Queryable and IQ.

Under this setup, we observe that Queryable and IQ are significantly ahead of LoRA and VB-LoRA (Table B).

[W1] Isolating the contribution. We see that Queryable has a higher mean accuracy than LoRA, MoSLoRA, VB-LoRA, and Lily (Table B). We ran the static control you asked for. Here, we trained the same kind of atom bank with the router removed and equal weights on all atoms (the shared bank in our tables). On 31 of 32 seeds where training converged, it has the highest mean accuracy in Table A and is significantly ahead of LoRA, MoSLoRA, VB-LoRA, and Lily. However, when we include the single non-converged run, the shared bank’s 32-seed mean accuracy falls below LoRA (Table A). Additionally, this variant merges into the weights like LoRA, thus, adding no cost at inference.

Further, Queryable and IQ use the same kind of bank, and their routes vary with the input and the task. Routing without the depth summary also outperforms LoRA (Table B), unlike Table 5 of the paper (MMLU-12 only, 3 seeds, router learning rate 2 times the base rate). The revision will add these controls and update Table 5 accordingly.

Main results. For the main results, we tested on the held-out MMLU-12 subjects and MMLU-Pro categories with unseen instructions. One run of the shared bank did not converge. Its training loss stayed near 0.45, and the other 31 runs of this variant reached about 0.01. We showcase that variant both with and without this run.

Table A. Accuracy (%), mean over 32 seeds (31 in the converged-seeds row). Mean is the average of MMLU-12 and MMLU-Pro.

MethodMeanMMLU-12MMLU-Pro
LoRA54.7467.3042.18
MoSLoRA55.2267.5242.92
VB-LoRA54.5366.9142.15
Lily55.2967.7542.82
Queryable55.4967.8543.13
IQ55.4767.8543.09
Shared bank, 31 converged seeds55.7167.9443.48
Shared bank, all 32 seeds54.5266.5642.48
Learned static mixture55.4567.8143.08

Table B. Accuracy (%) and paired difference, with the 95% confidence interval and the p-value of a paired sign-flip (permutation) test. Shared bank rows and their baselines use its 31 converged seeds.

MethodBaselineSeedsBaseline accuracyMethod accuracyMethod − baselinep
QueryableLoRA3254.7455.49+0.75 [+0.25, +1.25]0.003
QueryableMoSLoRA3255.2255.49+0.27 [−0.09, +0.62]0.13
QueryableVB-LoRA3254.5355.49+0.96 [+0.57, +1.34]< 0.001
QueryableLily3255.2955.49+0.20 [−0.19, +0.60]0.29
IQLoRA3254.7455.47+0.73 [+0.37, +1.08]< 0.001
IQMoSLoRA3255.2255.47+0.25 [−0.08, +0.58]0.13
IQVB-LoRA3254.5355.47+0.94 [+0.64, +1.24]< 0.001
IQLily3255.2955.47+0.18 [−0.12, +0.49]0.23
Shared bankLoRA31 converged54.7355.71+0.98 [+0.69, +1.28]< 0.001
Shared bankMoSLoRA31 converged55.2655.71+0.45 [+0.14, +0.75]0.005
Shared bankVB-LoRA31 converged54.5255.71+1.19 [+0.93, +1.45]< 0.001
Shared bankLily31 converged55.2655.71+0.45 [+0.10, +0.80]0.01
Queryable without depth summaryLoRA3254.7455.39+0.64 [+0.31, +0.98]< 0.001
IQ without depth summaryLoRA3254.7455.50+0.76 [+0.42, +1.10]< 0.001

[W2] The interference setting. We trained one adapter on the six tasks at once. The task set includes GSM8K with step-by-step answers, ARC, OpenBookQA, BoolQ, RACE, and HellaSwag, with 300 training examples each. Each task is scored on its own test set. GSM8K uses all 1319 test problems and requires open-ended answers.

Table C. Interference setting, accuracy (%), mean over 16 seeds, and paired differences with 95% CI and sign-flip p.

MethodGSM8KSix task mean
LoRA75.5386.99
MoSLoRA74.6886.89
VB-LoRA75.3486.61
Lily74.6286.73
Queryable76.0087.15
IQ76.0987.06
Shared bank76.6187.18
MethodBaselineSeedsSix task mean, method − baselinepGSM8K, method − baselinep
QueryableLoRA16+0.16 [−0.14, +0.45]0.28+0.47 [−0.76, +1.71]0.46
QueryableMoSLoRA16+0.25 [+0.01, +0.50]0.04+1.32 [+0.31, +2.33]0.01
QueryableVB-LoRA16+0.54 [+0.20, +0.88]0.007+0.67 [−0.55, +1.89]0.27
QueryableLily16+0.42 [+0.12, +0.72]0.006+1.39 [+0.11, +2.67]0.04
IQLoRA16+0.07 [−0.25, +0.38]0.65+0.56 [−0.62, +1.74]0.34
IQMoSLoRA16+0.17 [−0.05, +0.39]0.13+1.41 [+0.41, +2.41]0.010
IQVB-LoRA16+0.45 [+0.14, +0.77]0.007+0.76 [−0.24, +1.75]0.13
IQLily16+0.33 [+0.08, +0.59]0.02+1.48 [+0.31, +2.64]0.02
Shared bankLoRA16+0.18 [−0.03, +0.40]0.09+1.08 [+0.01, +2.15]0.05
Shared bankMoSLoRA16+0.28 [−0.01, +0.57]0.06+1.93 [+0.88, +2.98]0.003
Shared bankVB-LoRA16+0.57 [+0.19, +0.95]0.007+1.27 [+0.01, +2.54]0.05
Shared bankLily16+0.45 [+0.13, +0.77]0.009+1.99 [+0.96, +3.03]0.002

In the six-task setting, Queryable and IQ significantly outperform VB-LoRA and Lily on the six-task mean and outperform MoSLoRA and Lily on GSM8K; the shared bank does as well. The shared bank has the best accuracy for GSM8K of all methods in the table.

[W1, Q1] Static bank. Each control in Table D trains its own bank of atoms, like Queryable's. They differ in how they mix the atoms. The shared bank (i.e., the fixed mixture) weights every atom equally. Therefore, all layers share one r×rr \times r matrix. For each block, the learned static mixture during training learns an input-independent atom subset as well as a mixture-weight vector. It then reuses these for every input. Random routing selects the atoms uniformly at random in each block for every input.

Table D. Learned router removed, accuracy (%) and paired difference. The shared bank and random routing each had one run that did not converge, so they appear with and without it. In the rows with 31 converged seeds, both models are averaged over the same 31 seeds.

RoutedControlSeedsControl accuracyRouted accuracyRouted − controlp
QueryableShared bank31 converged55.7155.52−0.19 [−0.60, +0.22]0.35
QueryableShared bank3254.5255.49+0.97 [−1.43, +3.37]0.83
QueryableLearned static mixture3255.4555.49+0.04 [−0.36, +0.45]0.82
QueryableRandom routing31 converged55.4555.50+0.05 [−0.37, +0.48]0.80
QueryableRandom routing3254.2655.49+1.23 [−1.21, +3.68]0.40
IQShared bank31 converged55.7155.50−0.21 [−0.49, +0.08]0.14
IQShared bank3254.5255.47+0.95 [−1.43, +3.33]0.93
IQLearned static mixture3255.4555.47+0.03 [−0.23, +0.28]0.84
IQRandom routing31 converged55.4555.47+0.02 [−0.31, +0.35]0.90
IQRandom routing3254.2655.47+1.21 [−1.24, +3.67]0.45

While the routed models reach accuracy similar to these controls on converged runs (Table D), Queryable and IQ's learned routers vary their atom selection with the input. IQ uses 16 distinct top-4 atom sets per 100 validation prompts. Its atom selection differs between tasks.

Note that Table B puts the shared bank significantly ahead of MoSLoRA on the converged seeds.

[W1, Q1] Prior work. Each approach changes a different part of the low-rank update.

MethodWhat is sharedDependence on inputWhere the mixing acts
MoSLoRAnothing across layersnothing (one trainable r×rr \times r mixing matrix per projection, merged after training)inside the bottleneck
VB-LoRAone vector bank for all LoRA factorsnothing (learned top-k mixtures of bank vectors, fixed after training)the factors themselves
ShareLoRAthe LoRA factors A, B or both, across layersnothingthe factors themselves
MoE-LoRAnothing across layersthe mix of LoRA experts, per layerwhole LoRA experts, each with its own A and B
Lilyhigh-dimensional up-projections across layersthe mix of up-projections, per layerthe full-width up-projection
Shared bank (fixed mixture)one r×rr \times r matrix for all layers, the mean of the banknothinginside the bottleneck
Queryableone bank of r×rr \times r atoms across all layersthe atom mixture, once per block, from the block-entry state and a summary of earlier blocksinside the bottleneck

MoSLoRA mixes inside the bottleneck with a separate matrix for each projection. The shared bank uses a single matrix across all layers. VB-LoRA shares parameters with a fixed selection. Lily routes over full-width experts at every layer. ShareLoRA shares the LoRA factors themselves across layers.

Text-to-LoRA generates adapter parameters from a task description. qa-FLoRA uses the input query to fuse separately trained LoRAs. IQ trains one adapter and uses the instruction embedding only to steer the choice among the shared atoms.

Queryable adds around 58K parameters over LoRA at 7B. Almost all of these are in the router, which the shared bank removes. Tables A to C compare MoSLoRA, VB-LoRA, Lily with Queryable under one setup.

[Q2] Depth summary. In an experiment, we removed the depth summary. In a second control, each input received the depth summaries of another input in the same batch; hence, the router sees a summary of the right depth but from the wrong input.

Table E. Depth controls, accuracy (%) and paired difference. One run with summaries of another input did not converge, so this variant appears with and without that run.

ModelVariantSeedsVariant accuracyModel accuracyModel − variantp
QueryableNo depth summary3255.3955.49+0.10 [−0.26, +0.47]0.57
IQNo depth summary3255.5055.47−0.03 [−0.28, +0.21]0.78
IQSummaries of another input31 converged55.5155.46−0.06 [−0.34, +0.22]0.68
IQSummaries of another input3254.3455.47+1.13 [−1.31, +3.56]0.66

We see that the model without the depth summary is also significantly ahead of LoRA (see [W1] in part 1), so the gain over LoRA holds without the depth summary.

[W2] Theory. The three theorems (Theorems 4.1 to 4.3) are stated for the shared, blockwise routing of Queryable and IQ. Theorem 4.1 derives IQ's additive log-prior routing logits from KL-regularized retrieval, while Theorems 4.2 and 4.3 formalize norm stability and gradient accumulation under shared blockwise routing.

Under the stated norm assumptions, Theorem 4.2 guarantees that each layer's effective update norm stays uniformly bounded even when the router selects different atoms across inputs. By the blockwise gradient identity, the gradients of all routed blocks accumulate on the shared atoms. The revision will present them as properties of the method, separate from the accuracy results.

[W2] Decode speed. In Table F, the routes are fixed after prefill (the paper refreshes them once at the first decoding step). Hence, the routed update can be merged into the weights once per request. Decoding then runs almost as fast as a merged LoRA (Table F). Without a router, the shared bank is fixed after training and merges into the weights like LoRA.

Table F. Decode throughput, Qwen2.5-7B-Instruct on one H100 with fp32 adapters, batch 1, 256-token prompt, 128 new tokens, tokens per second (median of 5 runs). Table 11 used bf16 (not fp32) adapters, and the absolute values differ.

SettingTokens/s
LoRA, merged into the weights70.8
LoRA, kept as a separate adapter35.9
IQ, kept as a separate adapter28.5
IQ, routed r×rr \times r matrix combined with the LoRA factors after prefill39.7
IQ, update merged into the weights once per request (merge 0.11 s)67.4

References:

Fedus, William, Barret Zoph, and Noam Shazeer. "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity." Journal of Machine Learning Research 23.120 (2022): 1-39.

Reviewer_8pgGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Queryable LoRA, a parameter-efficient fine-tuning method that dynamically selects low-rank update atoms from shared memory using input states and depth summaries. Its instruction-conditioned variant (IQ) further incorporates instruction embeddings to guide routing. Experiments across multiple model sizes and benchmarks demonstrate modest improvements over standard LoRA, supported by theoretical analysis and ablation studies.

Critical strengths
  1. Interesting dynamic adaptation design: The shared low-rank atom memory and depth-aware routing provide a flexible extension to standard LoRA with relatively few additional trainable parameters.
  2. Reasonable experimental coverage: The paper evaluates multiple model sizes and compares against several static and dynamic PEFT baselines, showing consistent but modest improvements.
Critical weaknesses
  1. Insufficient isolation of instruction conditioning: The main Queryable and IQ configurations differ in both instruction conditioning and depth pooling. Although Appendix F provides controlled experiments on a 0.5B model, the independent contribution of instruction conditioning remains insufficiently demonstrated in the main 7B setting.
  2. Limited gains relative to architectural complexity: The accuracy improvements over standard LoRA and competitive baselines are relatively modest. Without stronger static and routing-control baselines, it remains unclear whether the additional Top-k router, shared memory, and depth conditioning are necessary.
Questions for authors
  1. Instruction Contribution: Could the authors conduct a controlled ablation on the main 7B configuration, enabling and disabling instruction conditioning while keeping depth pooling and all other components fixed?
  2. Practical Benefits: Given the relatively modest accuracy gains, how do the authors justify the additional architectural complexity and inference overhead, particularly the reported generation throughput reduction?
  3. Routing Necessity: Could the authors compare against fixed routing weights, random routing weights, and parameter- or FLOPs-matched static LoRA variants? These comparisons would help establish whether dynamic routing provides benefits beyond a simpler static rank-space transformation.
3 comments

Thank you for taking the time for this review.

We wanted to clarify that the new tables (relevant to your points) are in our replies to Reviewers fQYB and ytMM. Our main result is stated in the first part of the reply to the Reviewer fQYB. Over the 32 seeds, Queryable has a higher mean accuracy than LoRA, MoSLoRA, VB-LoRA and Lily. The differences in this metric to LoRA and VB-LoRA are significant.

[W1, Q1] Instruction contribution. Table I in our response to Reviewer ytMM switches the router's instruction embedding on and off, keeping depth attention and all other parts fixed. IQ has similar accuracy with and without it (Table I). We note that the instruction embedding makes the choice of atoms depend more on the task (this is Table J in the same response). Further, IQ keeps its accuracy when the router gets a paraphrased or mismatched instruction at test time (last two rows of Table I).

Separately, the base Queryable variant omits a separate instruction embedding by design (using uniform depth averaging instead of IQ's depth attention) and reaches similar accuracy (55.49% vs. 55.47%, Table A). We will add the Table I ablation in the revision.

[W2, Q2] Practical benefits. Queryable adds a total of around 58K parameters to LoRA at 7B. In Table 4 of the paper, a complete run takes around 1.2 times the time required for LoRA. Meanwhile, MoE-LoRA and Lily take 4.3 times as long as LoRA (Table 4). In Table F of our reply to Reviewer fQYB, merging the routed update into the weights once per request removes most of the decode cost of routing.

On 31 of 32 seeds where training converged, the shared bank (the same kind of atom bank trained with equal weights and no router) significantly outperforms all four baselines in this study (Table B in that reply). In the six-task interference setting (one adapter trained jointly on six tasks; Table C in the same reply), the shared bank also achieves the highest GSM8K accuracy among all methods. Its update merges into the weights like LoRA, and thus, it adds no cost at inference.

[W2, Q3] Routing necessity. The fixed and random routing options asked can be found in Table D (in our response to Reviewer fQYB). Both these routing options use the same kind of atom bank as Queryable, and they reach accuracy similar to the routed models on converged runs. The fixed routing either weights every atom equally (the shared bank rows), or it uses one learned static mixture per block.

LoRA with rr = 8 is already closely matched in parameters to Queryable (20.19M against 20.24M). MoSLoRA adds one trainable r×rr \times r mixing matrix to LoRA. It is the simpler static rank-space transformation you mention. We note that the shared bank is significantly ahead of MoSLoRA on its 31 converged seeds (Table B in our response to Reviewer fQYB). Tables A to C are relevant here for comparison.

Thanks for detailed response and additional experiments. I think the new experimental results show the effectiveness of the Shared Atom Bank. However, static routing and random routing have similar performance, and instruction conditioning provides a limited improvement in accuracy. So we hope authors can clarify what contributions from Queryable Routing and Instruction Conditioning, particularly in terms of cross-task generalization.

Thank you for following up.

For the cross-task generalization, the gain comes from the atom bank. We had fixed this reading before running the 32 seeds, as the one to report in case no routed variant performed significantly better than the shared bank, and none of them did. All the accuracies in our main study are cross-task results, since the adapters are trained on the five source tasks and then tested on held-out MMLU-12 subjects and MMLU-Pro categories with unseen instructions. Here, the routed models are close in accuracy to the static mixture and to random routing on the converged runs (Table D). IQ also has similar accuracy as that of the variant without the instruction embedding (Table I). We note that this holds for MMLU-12 and MMLU-Pro separately. In the 6 task setting, the shared bank is on par with the routed models (as seen in Table C).

Routing and instruction conditioning changes the usage of the atom bank on the source tasks. W.r.t. routing, the atom choice changes with the input. For every 100 validation prompts, Queryable uses around 7 distinct top-4 atom sets and IQ around 16. The learned static mixture uses one set for every input. Further, the instruction embedding makes the atom choice more task-specific (Table J). On unseen tasks, this specialization does not raise accuracy. So what carries over to the new tasks is the part of the update that is common to all inputs. Our guess for the instruction embedding is that it adds little here since the router already gets the instruction text through the block-entry state.

In the revision, we will state this clearly as the result and credit the cross-task performance gains to the atom bank, while routing and instruction conditioning will be described by their measured effects on atom use. For cross-task transfer, this makes the atom bank with a fixed mixture the simplest form of the method, which does not require a router and merges into the weights similar to LoRA.

Reviewer_ytMMOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adds a small shared bank of rank-space matrices to LoRA. A blockwise router uses current activations, summaries of earlier blocks, and optionally an instruction embedding to select a sparse mixture. It adds only about 58K trainable parameters, and the main experiments show improvements over LoRA with checkpoints selected on source-task validation.

Critical strengths
  1. The design adds little trainable capacity. Sharing small rank-space atoms across layers avoids a separate set of LoRA factors for every expert. At 7B, trainable parameters increase from 20.185M to 20.243M, about 0.29%, and blockwise reuse limits the number of routing calls (Table 6). This is an appealing way to make adaptation input-dependent without a large adapter bank.

  2. The experiments give a useful picture of both benefits and costs. There are multiple model sizes, static and dynamic baselines, repeated seeds, depth-summary ablations, small-model tasks, and inference measurements. Reporting the roughly 29% decode-throughput reduction alongside the favorable long-prompt prefill results helps readers assess the trade-off. The theoretical identities also help explain the implementation, within their stated local or conditional scope.

Critical weaknesses

The main quality results do not yet cover realistic long-form generation. The 7B/14B experiments use multiple-choice tasks, inputs of at most 1,024 tokens, and up to eight generated tokens (Table 9). Small-model open-ended results are useful, but leave the main-scale generation question open. Routes are fixed after the first decoding step, with one refresh for most blocks, yet Queryable's throughput drops from 47.5 to 33.8 tokens/s (Table 11). The small long-prompt prefill overhead alone does not establish the end-to-end quality–latency trade-off.

Questions for authors
  1. Open-ended generation. Could you compare quality and latency on open-ended tasks at the main model scale, testing fixed versus refreshed routes against an efficiently merged static LoRA? This would help establish whether the generation benefit justifies the throughput penalty.

  2. Uncertainty against strong baselines. Could you report paired differences across training seeds and task categories, including intervals against the strongest dynamic baselines? I would like to understand how stable the small accuracy margins are beyond the average ranks.

  3. The independent value of instructions. I'm curious how much IQ gains from language conditioning when depth pooling is held fixed. Could you run a matched 7B transfer ablation with query-only, prior-only, and paraphrased or mismatched instructions?

2 comments

Thank you for the review.

Tables A to F are in our reply to Reviewer fQYB. Over 32 seeds, Queryable has a higher mean accuracy than LoRA, MoSLoRA, VB-LoRA and Lily, and the differences to LoRA and VB-LoRA are significant. Further, the same kind of atom bank trained with equal weights and no router (the shared bank) is significantly ahead of all four baselines on the 31 seeds where the training converged.

[W1, Q1] Open-ended generation. Table C in our reply to Reviewer fQYB includes GSM8K with step-by-step answers on all the 1319 test problems, and the adapter is trained on GSM8K together with five other tasks.

In Table G, we take three trained seeds per method of this six-task adapter from our preliminary H100 runs (trained without the load-balancing loss described in our reply to Reviewer fQYB) and evaluate each checkpoint without retraining, comparing fixed routes after prefill against routes recomputed at every generated token.

Table G. GSM8K accuracy (%) with routes fixed after prefill and recomputed at each token, checkpoints of the six task adapter from our preliminary H100 runs, evaluated on one A100, mean over 3 seeds, with the paired difference, 95% CI and sign-flip p. We measure the time (in minutes) to answer all 1319 problems at batch 8.

MethodRoutes fixedRoutes recomputedRecomputed − fixedpMinutes, fixed to recomputed
Queryable75.1675.26+0.10 [−0.93, +1.13]0.759.4 to 11.6 (+23%)
IQ75.4675.54+0.08 [−0.91, +1.06]1.009.1 to 12.0 (+32%)

Recomputing the routes at every token gives similar accuracy and takes longer (Table G). In Table F of our reply to Reviewer fQYB, merging the routed update into the weights once per request brings decoding close to the speed of a merged LoRA.

[Q2] Uncertainty against strong baselines. Table B in our reply to Reviewer fQYB gives paired differences with 95% intervals over 32 seeds against LoRA, MoSLoRA, VB-LoRA and Lily, the strongest baseline in Table A. Table H splits the comparison with Lily by category.

Table H. Paired difference against Lily by category, in accuracy points over 32 seeds, with 95% CI. Bold marks the category interval that excludes zero.

CategoryQueryable − LilyIQ − Lily
MMLU-12+0.11 [−0.19, +0.41]+0.10 [−0.10, +0.31]
MMLU-Pro+0.30 [−0.28, +0.88]+0.27 [−0.23, +0.76]
MMLU astronomy+0.60 [−0.05, +1.24]+0.08 [−0.65, +0.82]
MMLU clinical knowledge−0.19 [−0.77, +0.39]−0.12 [−0.66, +0.42]
MMLU college physics−0.58 [−2.07, +0.91]−0.43 [−1.75, +0.89]
MMLU econometrics−0.22 [−1.37, +0.93]−0.30 [−1.32, +0.72]
MMLU formal logic−0.30 [−1.67, +1.08]−0.99 [−2.16, +0.18]
MMLU high school psychology+0.02 [−0.31, +0.36]−0.03 [−0.40, +0.33]
MMLU machine learning−0.17 [−1.63, +1.30]−0.59 [−1.98, +0.81]
MMLU marketing−0.11 [−0.53, +0.32]+0.13 [−0.31, +0.58]
MMLU philosophy−0.24 [−0.90, +0.42]−0.39 [−0.96, +0.17]
MMLU professional law+0.34 [−0.02, +0.71]+0.50 [+0.14, +0.86]
MMLU US foreign policy+0.81 [−0.08, +1.71]+0.34 [−0.48, +1.17]
MMLU world religions−0.11 [−0.60, +0.38]−0.09 [−0.50, +0.32]
MMLU-Pro biology+0.34 [−0.24, +0.93]+0.32 [−0.12, +0.76]
MMLU-Pro chemistry+0.69 [−0.18, +1.55]+0.31 [−0.56, +1.19]
MMLU-Pro economics+0.36 [−0.09, +0.81]+0.38 [−0.11, +0.87]
MMLU-Pro engineering−0.26 [−1.54, +1.02]+0.53 [−0.53, +1.58]
MMLU-Pro history−0.02 [−0.86, +0.81]+0.05 [−0.70, +0.80]
MMLU-Pro law+0.28 [−0.35, +0.91]+0.33 [−0.09, +0.76]
MMLU-Pro math+0.51 [−0.25, +1.27]+0.38 [−0.31, +1.07]
MMLU-Pro philosophy+0.21 [−0.72, +1.13]−0.05 [−0.78, +0.68]
MMLU-Pro physics+0.46 [−0.27, +1.18]+0.29 [−0.49, +1.08]
MMLU-Pro psychology−0.03 [−0.62, +0.56]−0.31 [−0.80, +0.19]

Here, we observe that no category interval favours Lily. The one category interval that excludes zero favours IQ.

[Q3] Independent value of instructions. We note that within each comparison, all the variants share the same depth attention and differ only in the router's use of the instruction embedding. The instruction text itself stays in every prompt and for every approach.

The last two rows evaluate IQ checkpoints of the main 7B setting from our preliminary H100 runs (these are trained without the load-balancing loss) and change only the instruction embedding given to the router during test time.

Table I. Instruction ablation at 7B, accuracy (%) and paired difference.

VariantSeedsVariant accuracyIQ accuracyIQ − variantp
No instruction embedding3255.4555.47+0.02 [−0.33, +0.37]0.91
Instruction in the query only3255.3155.47+0.16 [−0.08, +0.41]0.19
Instruction in the prior only31 converged55.3155.47+0.17 [−0.13, +0.47]0.26
Instruction in the prior only3254.1755.47+1.30 [−1.03, +3.64]0.13
Paraphrased instruction for the router at test, preliminary H100 run655.4455.43−0.01 [−0.09, +0.07]0.91
Mismatched instruction for the router at test, preliminary H100 run655.4455.43−0.02 [−0.06, +0.03]0.41

We see that IQ has similar accuracy without the instruction embedding (first row of Table I) and when the router gets a paraphrased or mismatched instruction at test time (last two rows). A likely reason is that the router already receives the instruction text through the prompt.

The instruction embedding makes the choice of atoms depend more on the task. Fewer prompts share one dominant atom set. The atom use differs more between the tasks (Table J).

Table J. Routing on validation prompts of the training tasks, 32 seeds, paired difference and 95% CI. The second row reports the mean pairwise L1 distance between tasks' per-atom top-4 selection rates (which sum to 4 per task), ranging from 0 (identical selection rates) to 8 (disjoint top-4 atom sets).

MeasureIQ without instruction embeddingIQIQ − variantp
Prompts on the most common atom set (%)4722−25 [−36, −14]< 0.001
Difference in atom use between tasks (0 to 8)1.343.23+1.89 [+1.54, +2.25]< 0.001

Comments on the paper

2 comments
Area_Chair_vyTzOpenReview ↗

Discussion phrase will end soon

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

Dear AC,

We are encouraged that the reviewers find our design "an appealing way to make adaptation input-dependent without a large adapter bank" (ytMM) and our evaluation against LoRA "more careful than most PEFT papers" (fQYB). We have addressed all the reviewer comments in our rebuttal. Reviewer 8pgG then noted that the new results show the effectiveness of the shared atom bank, while Reviewers fQYB and ytMM have not yet responded. We do hope that our clarifications and new results are considered in the evaluation.

Based on reviewer comments, we have:

  • Rerun our main 7B study with a single fixed setup and 32 seeds. (fQYB)
  • Added controls without the learned router, including a shared bank with equal weights on all atoms. (fQYB, 8pgG)
  • Added comparisons with MoSLoRA, VB-LoRA and Lily, using their authors' code. (fQYB, ytMM)
  • Added depth-summary controls and a 7B instruction ablation. (fQYB, 8pgG, ytMM)
  • Added a six-task interference setting and measured decode speed. (fQYB, ytMM, 8pgG)
  • Clarified the difference from MoSLoRA, VB-LoRA and Lily, and the role of the theory. (fQYB)

In the 32 seed study, Queryable and IQ significantly outperform LoRA and VB-LoRA. The shared bank significantly outperforms all four baselines on the 31 seeds where training converged. The revised paper adds these results in blue (Table 5 and Appendix K) and corrects an error in Appendix F.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.