- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies routing over a large pool of LoRA adapters sharing the same backbone. PAIR builds compact fingerprints for adapters, clusters them, and uses hierarchical routing to select and combine a small set of adapters for each input. It does not require task labels or a trained router. Experiments across several backbones and datasets show better performance than existing routing baselines while reducing the number of adapter evaluations.
Critical strengths
- Scoring every adapter becomes expensive as the pool grows, so this is a useful problem to study.
- Fixing the composition strategy while changing only the router gives a cleaner comparison.
Critical weaknesses
- Eq. (4) worries me because changes with the set of adapters scored together. Level-1 and Level-2 use different sets, but the paper still reuses and compares these scores. I do not see why they should be comparable across stages.
- I am also not convinced by the JL argument. The projection is only applied to one side of , while the standard JL lemma assumes a full random projection. The paper should clarify what is actually preserved and what assumptions on are needed.
Questions for authors
- Table 1 reports around 190 adapter-score evaluations per input, while the last row of Table 3 reports 92 for what seems to be the same setting. What causes this difference?
- How were the routing hyperparameters chosen, eg. , top-20 composition, , and ? The paper says no evaluation examples are used for tuning, so it would be useful to clarify what data was used.
4 comments
Comment 2
Q1. Why 190 in Table 1 but 92 in Table 3?
We traced the difference to the Level-1 counter of Table 3, which counted 80 clusters instead of 178 representatives and therefore undercounts by 98. The complete count has median 190 and range 185–200. A replay of all 60,619 inputs with unchanged routed sets confirms it. The count includes repeated scores of the same adapter across stages; the number of distinct adapters has median 179 (Update 3).
Q2. Hyperparameter selection
The configuration was developed on LLaMA with BBH sweeps and a fixed subset of up to 128 examples per task. Both overlap the evaluation suite, which updates the submission's statement that no evaluation examples informed tuning (Update 1). The composition cap of 20 matches LoGo. The stored development runs vary , , the per-cluster quota, , and the selection and scoring rules on these sets, including the rule that expands from four to five clusters. Six scoring variants were also compared on the full suite, and the final configuration had the highest macro.
Qwen and DeepSeek use the LLaMA configuration without further tuning. After excluding all BBH tasks and the development subset, 57,562 examples remain across 18 tasks. On their four-family macro, a post-hoc analysis gives a matched LLaMA selection difference of +0.91 [0.76, 1.06] (Update 1).
Summary. This reply gives the stage-specific scoring definitions, a direct standardisation ablation, and the row-wise argument for the raw-sketch guarantee. Together with the updated count and the fixed-composition comparison that your review recognises, they are the evidence for PAIR's selection procedure. We would appreciate your reassessment in light of these additions and of the configuration-selection protocol in Q2. Thank you again for your time and for the precise questions.
Thank you for the detailed response. The stage-specific scoring explanation and the clarification of the raw-sketch guarantee address much of my technical concern, and the evaluation-count discrepancy is now explained. However, the tuning protocol raises a more substantial concern: scoring variants were compared on the full evaluation suite, and the best-performing configuration was selected. Removing the development subset afterwards does not provide an independent evaluation. I would need results on an untouched test set with the configuration fixed to be convinced of the reported gains. I am therefore keeping my rating at 2.
Thank you for confirming that the scoring, sketch, and count questions are resolved.
We evaluated the unchanged submitted configuration on 6,799 previously unused validation examples from 14 non-BBH tasks. These examples were not used for development or for any reported evaluation. The arms, the primary comparison and both scoring rules were fixed in writing before the run. LoGo and LoRARetriever used their Table 1 settings and the same generation budgets as PAIR.
With ARC answers credited as either option letters or option text, and all other scoring unchanged, the four-family macro differences are:
| Contrast | Difference, 95% bootstrap interval |
|---|---|
| PAIR − Base | +1.79 [1.26, 2.34] |
| PAIR − Flat, matched adapter count | +0.91 [0.52, 1.31] |
| PAIR − LoGo | +1.63 [1.23, 2.05] |
| PAIR − LoRARetriever | +1.16 [0.68, 1.65] |
| PAIR − Fixed combination from development inputs | +0.05 [−0.16, 0.26] |
The fixed combination applies PAIR's average weights on the development subset to every input, without routing. Under the paper scorer the five differences are +1.51, +2.27, +1.52, +0.96 and −0.19; the first four intervals exclude zero.
The prespecified hierarchy−flat comparison, with uniform weights in both arms, gives +1.73 [1.23, 2.23] under the paper scorer and +0.28 [−0.09, 0.65] under the predefined ARC content-scoring check. On a separate evaluation of 19 BBH tasks outside the suite, exact match is below 5% in every arm because the outputs are mostly option text; scored by option content on the 17 multiple-choice tasks, a rule defined after those outputs were seen, the same comparison gives −0.99 [−2.37, 0.40].
The requested independent evaluation is now available: full PAIR retains positive gains over Base, Flat, LoGo, and LoRARetriever on previously unused examples. We distinguish these full-method gains from the more limited evidence for hierarchy alone or dynamic routing over fixed composition. We would appreciate your reassessment in light of this new evidence.
Comment 1
Thank you for your review. It recognises the practical routing problem and the comparison that fixes the composition and changes only the router. Your two technical concerns lead to the clarifications below: we state the stage-specific scoring rule, test a shared-reference alternative, and give the row-wise argument for the one-sided sketch. The Update are listed in the general response.
W1. Comparing scores across stages
The two stages make separate rankings with their own reference sets. Level-1 standardised scores are not reused in the Level-2 ranking. We denote by sm(x;S) the response of adapter m to input x, standardised within the reference set S. Eq. (4) then reads:
s_m^{(2,k)}(x)=s_m(x;\mathcal M_k).$$ Here $\mathcal R$ contains all representatives and $\mathcal M_k$ the candidates of cluster $k$. Level 1 compares clusters within $\mathcal R$. Level 2 ranks the candidates of a selected cluster within $\mathcal M_k$. The final weight is $$\pi_m(x)=\widetilde s_{k(m)}(x)\cdot\nu_m\cdot s_m^{(2,k(m))}(x),$$ where $\widetilde s_{k(m)}$ is the cluster score of adapter $m$ and $\nu_m$ its centrality. The three factors are relative; they are not calibrated to a common absolute scale. On a fixed routed set, the product adds **+0.30 [0.03, 0.56]** over uniform weights. **We tested the shared-reference alternative that your concern suggests.** The ablation standardises Level-2 responses with the Level-1 reference statistics, so that both stages share one reference set. The routed set stays the same for 99.5% of 5,637 inputs. The weights concentrate: the effective number of adapters $1/\sum_m w_m^2$, with $w_m=\pi_m/\sum_j\pi_j$, falls from 7.5 to 3.8. The macro falls by **5.61 [4.97, 6.27]**. This supports local standardisation for the tested weighting rule. **The matched control does not depend on the weight product.** It uses uniform weights. It evaluates the complete selection procedure, which includes the stage-specific standardisation. Algorithm 1 states that Level 2 reuses cached Level-1 outputs; this does not describe the implementation (Update 3). ### W2. What the one-sided projection preserves **A full random projection of the vectorised update is not required. A one-sided Gaussian projection suffices for the Frobenius-distance guarantee on the raw sketch.** Right multiplication by the sketch matrix $\Xi$ applies a Gaussian projection to every row of the update. The argument has three steps: Gaussian concentration for each row, a union bound over rows and adapter pairs, and the sum over squared row norms. These steps bound the pairwise Frobenius distances within distortion $\varepsilon$, with failure probability $\delta$, for sketch width $N=O(\varepsilon^{-2}\log(d_LM/\delta))$. Here $d_L$ is the number of rows and $M$ the pool size. For a fixed nonzero difference $D$ of two updates and independent entries $\Xi_{ij}\sim\mathcal N(0,1/N)$, $$\mathbb E\lVert D\Xi\rVert_F^2=\lVert D\rVert_F^2,\qquad \frac{\operatorname{Var}\lVert D\Xi\rVert_F^2}{\lVert D\rVert_F^4} =\frac{2\lVert D^\top D\rVert_F^2}{N\lVert D\rVert_F^4}\le\frac2N.$$ The second relation gives the scale of the distortion. The argument assumes a fixed finite pool that is independent of the shared $\Xi$. It requires no rank or incoherence condition. **The formal guarantee applies to the raw sketch.** Normalisation and PCA are later representation stages, and the Appendix B sentence should not have extended the guarantee to them. At width $N=64$, we drew five independent sketches of all 33,670 LLaMA adapter pairs. The ratio of sketched to exact distances has pooled 1st–99th percentiles of **0.864–1.090**, and the mean rank correlation is **0.995**.