- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes Queryable LoRA. The idea is to keep a global bank of small r×r matrices ("atoms") shared across all layers, and have a router mix a few of them once per block of layers. The mixture acts inside the LoRA bottleneck, i.e. ΔW = (α/r)B(I + gS_b)A with a learned gate g. The router looks at the current rank-space state and summaries of earlier blocks; the IQ variant also adds an instruction embedding to the query and uses it as a prior over atoms. The paper gives a variational characterization of the router, norm bounds, and gradient identities. For validation, one adapter is trained on five source tasks and tested on held-out MMLU/MMLU-Pro/GPQA with unseen instructions, on Qwen2.5 models from 1.5B to 14B. Reported gains over LoRA and dynamic baselines are about 1-2 points on average, at roughly 1.2x LoRA's training cost.
Critical strengths
-
Cheap. Only +0.29% parameters over LoRA at 7B and 1.2x training time, while MoE-LoRA and Lily take 4.3x.
-
The evaluation against LoRA is more careful than most PEFT papers: held-out tasks and instructions, several seeds, bootstrap intervals, and a full-size re-evaluation. The reporting is honest (e.g. seed-level losses on GPQA are disclosed) and the paper is easy to read.
Critical weaknesses
-
Limited novelty, and the actual contribution is never isolated. Each component has a close precedent: a trainable r×r matrix inside the LoRA bottleneck is MoSLoRA's mixing matrix, shared banks are VB-LoRA/ShareLoRA, input-dependent routing is the MoE-LoRA line of work, cross-layer coupling is Lily, and instruction conditioning is Text-to-LoRA/qa-FLoRA. What is new is the specific combination. Worse, the paper's own ablation shows routing without the depth summary does worse than plain LoRA (0.752 vs 0.759, Table 5), so routing by itself is not what helps. There is also no static control (shared atoms with a fixed uniform mixture, no router), which seems like the obvious baseline to run. Since the instruction channel also does nothing at 7B (IQ is basically identical to Queryable), the isolated contribution boils down to passing a depth summary to the router.
-
The theory is decorative and the evaluation sidesteps the motivating problem. Theorem 4.1 is the standard softmax/KL duality, Theorem 4.2 follows from convexity, Theorem 4.3 is the chain rule. None of this explains why depth conditioning helps. The method is motivated by multi-task interference but is only evaluated on transfer from 1,500 training examples to multiple-choice benchmarks, never on the interference setting itself. The hyperparameters also drift across settings (atom init std 0.30/0.15/0.05, gate init changed at 14B, three different checkpoint-selection rules), which makes the gains hard to attribute. Finally, "runtime close to LoRA" only holds for training and prefill: decode throughput is about 29% lower than LoRA (33.8 vs 47.5 tok/s).
Questions for authors
-
What happens if the atom bank is used statically, i.e. a fixed uniform mixture with the router removed, at matched parameter count? If that matches the routed model, the routing mechanism is unnecessary. Please also spell out the concrete delta to MoSLoRA, VB-LoRA, and Lily.
-
The depth summary seems to be the only component that matters. What does it carry that the block-entry state does not? Would summaries shuffled across blocks do equally well?
-
Do the main results survive if atom init, gate init, and the checkpoint-selection rule are held fixed across all settings?
3 comments
Part 2
[W2] The interference setting. We trained one adapter on the six tasks at once. The task set includes GSM8K with step-by-step answers, ARC, OpenBookQA, BoolQ, RACE, and HellaSwag, with 300 training examples each. Each task is scored on its own test set. GSM8K uses all 1319 test problems and requires open-ended answers.
Table C. Interference setting, accuracy (%), mean over 16 seeds, and paired differences with 95% CI and sign-flip p.
| Method | GSM8K | Six task mean |
|---|---|---|
| LoRA | 75.53 | 86.99 |
| MoSLoRA | 74.68 | 86.89 |
| VB-LoRA | 75.34 | 86.61 |
| Lily | 74.62 | 86.73 |
| Queryable | 76.00 | 87.15 |
| IQ | 76.09 | 87.06 |
| Shared bank | 76.61 | 87.18 |
| Method | Baseline | Seeds | Six task mean, method − baseline | p | GSM8K, method − baseline | p |
|---|---|---|---|---|---|---|
| Queryable | LoRA | 16 | +0.16 [−0.14, +0.45] | 0.28 | +0.47 [−0.76, +1.71] | 0.46 |
| Queryable | MoSLoRA | 16 | +0.25 [+0.01, +0.50] | 0.04 | +1.32 [+0.31, +2.33] | 0.01 |
| Queryable | VB-LoRA | 16 | +0.54 [+0.20, +0.88] | 0.007 | +0.67 [−0.55, +1.89] | 0.27 |
| Queryable | Lily | 16 | +0.42 [+0.12, +0.72] | 0.006 | +1.39 [+0.11, +2.67] | 0.04 |
| IQ | LoRA | 16 | +0.07 [−0.25, +0.38] | 0.65 | +0.56 [−0.62, +1.74] | 0.34 |
| IQ | MoSLoRA | 16 | +0.17 [−0.05, +0.39] | 0.13 | +1.41 [+0.41, +2.41] | 0.010 |
| IQ | VB-LoRA | 16 | +0.45 [+0.14, +0.77] | 0.007 | +0.76 [−0.24, +1.75] | 0.13 |
| IQ | Lily | 16 | +0.33 [+0.08, +0.59] | 0.02 | +1.48 [+0.31, +2.64] | 0.02 |
| Shared bank | LoRA | 16 | +0.18 [−0.03, +0.40] | 0.09 | +1.08 [+0.01, +2.15] | 0.05 |
| Shared bank | MoSLoRA | 16 | +0.28 [−0.01, +0.57] | 0.06 | +1.93 [+0.88, +2.98] | 0.003 |
| Shared bank | VB-LoRA | 16 | +0.57 [+0.19, +0.95] | 0.007 | +1.27 [+0.01, +2.54] | 0.05 |
| Shared bank | Lily | 16 | +0.45 [+0.13, +0.77] | 0.009 | +1.99 [+0.96, +3.03] | 0.002 |
In the six-task setting, Queryable and IQ significantly outperform VB-LoRA and Lily on the six-task mean and outperform MoSLoRA and Lily on GSM8K; the shared bank does as well. The shared bank has the best accuracy for GSM8K of all methods in the table.
[W1, Q1] Static bank. Each control in Table D trains its own bank of atoms, like Queryable's. They differ in how they mix the atoms. The shared bank (i.e., the fixed mixture) weights every atom equally. Therefore, all layers share one matrix. For each block, the learned static mixture during training learns an input-independent atom subset as well as a mixture-weight vector. It then reuses these for every input. Random routing selects the atoms uniformly at random in each block for every input.
Table D. Learned router removed, accuracy (%) and paired difference. The shared bank and random routing each had one run that did not converge, so they appear with and without it. In the rows with 31 converged seeds, both models are averaged over the same 31 seeds.
| Routed | Control | Seeds | Control accuracy | Routed accuracy | Routed − control | p |
|---|---|---|---|---|---|---|
| Queryable | Shared bank | 31 converged | 55.71 | 55.52 | −0.19 [−0.60, +0.22] | 0.35 |
| Queryable | Shared bank | 32 | 54.52 | 55.49 | +0.97 [−1.43, +3.37] | 0.83 |
| Queryable | Learned static mixture | 32 | 55.45 | 55.49 | +0.04 [−0.36, +0.45] | 0.82 |
| Queryable | Random routing | 31 converged | 55.45 | 55.50 | +0.05 [−0.37, +0.48] | 0.80 |
| Queryable | Random routing | 32 | 54.26 | 55.49 | +1.23 [−1.21, +3.68] | 0.40 |
| IQ | Shared bank | 31 converged | 55.71 | 55.50 | −0.21 [−0.49, +0.08] | 0.14 |
| IQ | Shared bank | 32 | 54.52 | 55.47 | +0.95 [−1.43, +3.33] | 0.93 |
| IQ | Learned static mixture | 32 | 55.45 | 55.47 | +0.03 [−0.23, +0.28] | 0.84 |
| IQ | Random routing | 31 converged | 55.45 | 55.47 | +0.02 [−0.31, +0.35] | 0.90 |
| IQ | Random routing | 32 | 54.26 | 55.47 | +1.21 [−1.24, +3.67] | 0.45 |
While the routed models reach accuracy similar to these controls on converged runs (Table D), Queryable and IQ's learned routers vary their atom selection with the input. IQ uses 16 distinct top-4 atom sets per 100 validation prompts. Its atom selection differs between tasks.
Note that Table B puts the shared bank significantly ahead of MoSLoRA on the converged seeds.
Part 3
[W1, Q1] Prior work. Each approach changes a different part of the low-rank update.
| Method | What is shared | Dependence on input | Where the mixing acts |
|---|---|---|---|
| MoSLoRA | nothing across layers | nothing (one trainable mixing matrix per projection, merged after training) | inside the bottleneck |
| VB-LoRA | one vector bank for all LoRA factors | nothing (learned top-k mixtures of bank vectors, fixed after training) | the factors themselves |
| ShareLoRA | the LoRA factors A, B or both, across layers | nothing | the factors themselves |
| MoE-LoRA | nothing across layers | the mix of LoRA experts, per layer | whole LoRA experts, each with its own A and B |
| Lily | high-dimensional up-projections across layers | the mix of up-projections, per layer | the full-width up-projection |
| Shared bank (fixed mixture) | one matrix for all layers, the mean of the bank | nothing | inside the bottleneck |
| Queryable | one bank of atoms across all layers | the atom mixture, once per block, from the block-entry state and a summary of earlier blocks | inside the bottleneck |
MoSLoRA mixes inside the bottleneck with a separate matrix for each projection. The shared bank uses a single matrix across all layers. VB-LoRA shares parameters with a fixed selection. Lily routes over full-width experts at every layer. ShareLoRA shares the LoRA factors themselves across layers.
Text-to-LoRA generates adapter parameters from a task description. qa-FLoRA uses the input query to fuse separately trained LoRAs. IQ trains one adapter and uses the instruction embedding only to steer the choice among the shared atoms.
Queryable adds around 58K parameters over LoRA at 7B. Almost all of these are in the router, which the shared bank removes. Tables A to C compare MoSLoRA, VB-LoRA, Lily with Queryable under one setup.
[Q2] Depth summary. In an experiment, we removed the depth summary. In a second control, each input received the depth summaries of another input in the same batch; hence, the router sees a summary of the right depth but from the wrong input.
Table E. Depth controls, accuracy (%) and paired difference. One run with summaries of another input did not converge, so this variant appears with and without that run.
| Model | Variant | Seeds | Variant accuracy | Model accuracy | Model − variant | p |
|---|---|---|---|---|---|---|
| Queryable | No depth summary | 32 | 55.39 | 55.49 | +0.10 [−0.26, +0.47] | 0.57 |
| IQ | No depth summary | 32 | 55.50 | 55.47 | −0.03 [−0.28, +0.21] | 0.78 |
| IQ | Summaries of another input | 31 converged | 55.51 | 55.46 | −0.06 [−0.34, +0.22] | 0.68 |
| IQ | Summaries of another input | 32 | 54.34 | 55.47 | +1.13 [−1.31, +3.56] | 0.66 |
We see that the model without the depth summary is also significantly ahead of LoRA (see [W1] in part 1), so the gain over LoRA holds without the depth summary.
[W2] Theory. The three theorems (Theorems 4.1 to 4.3) are stated for the shared, blockwise routing of Queryable and IQ. Theorem 4.1 derives IQ's additive log-prior routing logits from KL-regularized retrieval, while Theorems 4.2 and 4.3 formalize norm stability and gradient accumulation under shared blockwise routing.
Under the stated norm assumptions, Theorem 4.2 guarantees that each layer's effective update norm stays uniformly bounded even when the router selects different atoms across inputs. By the blockwise gradient identity, the gradients of all routed blocks accumulate on the shared atoms. The revision will present them as properties of the method, separate from the accuracy results.
[W2] Decode speed. In Table F, the routes are fixed after prefill (the paper refreshes them once at the first decoding step). Hence, the routed update can be merged into the weights once per request. Decoding then runs almost as fast as a merged LoRA (Table F). Without a router, the shared bank is fixed after training and merges into the weights like LoRA.
Table F. Decode throughput, Qwen2.5-7B-Instruct on one H100 with fp32 adapters, batch 1, 256-token prompt, 128 new tokens, tokens per second (median of 5 runs). Table 11 used bf16 (not fp32) adapters, and the absolute values differ.
| Setting | Tokens/s |
|---|---|
| LoRA, merged into the weights | 70.8 |
| LoRA, kept as a separate adapter | 35.9 |
| IQ, kept as a separate adapter | 28.5 |
| IQ, routed matrix combined with the LoRA factors after prefill | 39.7 |
| IQ, update merged into the weights once per request (merge 0.11 s) | 67.4 |
References:
Fedus, William, Barret Zoph, and Noam Shazeer. "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity." Journal of Machine Learning Research 23.120 (2022): 1-39.
Part 1
Thank you for your time and review.
Regarding your points and questions, we reran our main 7B study with a single fixed setup. In this setup, we trained each approach and control method on the same 32 seeds.
[W2, Q3] Specification of fixed settings. Every run of this 32 seed study uses the settings in the table below.
As is common for top-k routers, Queryable and IQ add the load-balancing loss of Switch Transformer (Fedus et al., 2022) in these runs, with the weight 0.01 chosen beforehand on 11 separate development subjects and categories. The loss function acts on the top-k router; therefore, it applies only to Queryable and IQ.
Under this setup, we observe that Queryable and IQ are significantly ahead of LoRA and VB-LoRA (Table B).
[W1] Isolating the contribution. We see that Queryable has a higher mean accuracy than LoRA, MoSLoRA, VB-LoRA, and Lily (Table B). We ran the static control you asked for. Here, we trained the same kind of atom bank with the router removed and equal weights on all atoms (the shared bank in our tables). On 31 of 32 seeds where training converged, it has the highest mean accuracy in Table A and is significantly ahead of LoRA, MoSLoRA, VB-LoRA, and Lily. However, when we include the single non-converged run, the shared bank’s 32-seed mean accuracy falls below LoRA (Table A). Additionally, this variant merges into the weights like LoRA, thus, adding no cost at inference.
Further, Queryable and IQ use the same kind of bank, and their routes vary with the input and the task. Routing without the depth summary also outperforms LoRA (Table B), unlike Table 5 of the paper (MMLU-12 only, 3 seeds, router learning rate 2 times the base rate). The revision will add these controls and update Table 5 accordingly.
Main results. For the main results, we tested on the held-out MMLU-12 subjects and MMLU-Pro categories with unseen instructions. One run of the shared bank did not converge. Its training loss stayed near 0.45, and the other 31 runs of this variant reached about 0.01. We showcase that variant both with and without this run.
Table A. Accuracy (%), mean over 32 seeds (31 in the converged-seeds row). Mean is the average of MMLU-12 and MMLU-Pro.
Table B. Accuracy (%) and paired difference, with the 95% confidence interval and the p-value of a paired sign-flip (permutation) test. Shared bank rows and their baselines use its 31 converged seeds.