- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is a clean, well-motivated idea: route samples to frozen per-task LoRA adapters using a closed-form incremental LDA over intermediate CLIP features, with no gradient training in the router. The core observation (intermediate CLIP features separate tasks well) is real and the write-up is easy to follow. The statistical appendix is also, to my relief, mostly correct.
Critical strengths
- Taking router features from a frozen backbone is the right constraint. It makes the "no forgetting in the router" argument essentially trivial, and it is the honest way to build a task-agnostic router.
- Refusing to over-claim from the Fisher trace is good practice. The three-task counterexample in Appendix C (Eq. 34) is the right observation to make, and more papers should make it.
Critical weaknesses
The released code does not implement the LoRA described in Eq. 10; The 5-shot Transfer row is identical to the full-shot Transfer row, digit for digit;The router layer and the adapter layer are hard-wired together, and "intermediate feature" has two different definitions in the same codebase;In the 5-shot setting, checkpoint selection uses the full validation set;
Questions for authors
- Which of Eq. 8 and
lda.py:194-200produced Tables 1-3? If the Mahalanobis form, please correct Eq. 8. - Can you regenerate the 5-shot Transfer block (M2), and confirm the beta used in each of Tables 1 and 2 (M9)?
- Was Table 5's ILDA-CL-679 row measured, or copied from ILDA-CL (M4)?
4 comments
Part 2/3: Verified Transfer Results, Beta, and Five-Shot Model Selection
4. Q2: Identical Transfer results and beta.
We have re-evaluated the five-shot Transfer block under the original experimental setting. The regenerated results exactly match the original per-dataset entries in Table 2, including the aggregate Transfer accuracy of 65.9%. These entries also coincide with the full-shot Transfer results in Table 1.
This agreement follows from the evaluation pipeline for unseen-task samples:
-
Frozen zero-shot gate. At each stage, the frozen CLIP model predicts over the global candidate vocabulary. If the predicted class belongs to the seen-class set, the sample enters the seen-task branch; otherwise, it enters the OOD branch.
-
OOD branch. The sample is classified using the frozen CLIP model with the evaluation dataset's class vocabulary. Neither the task-specific adapters nor the ILDA statistics participate in this prediction.
-
Seen-task branch. An unseen-task sample assigned to a seen-task expert is counted as incorrect under the evaluation rule, regardless of which expert ILDA selects.
Thus, for an unseen-task sample , its contribution to Transfer accuracy is where denotes the indicator function, which equals 1 if the enclosed condition holds and 0 otherwise. Here, is the frozen zero-shot gate at stage , and is the frozen classifier evaluated over the dataset's candidate classes.
With the test data, preprocessing, candidate vocabularies, prompt templates, and task order fixed, both factors remain unchanged between full-shot and five-shot training. Consequently, reducing the adapter training samples or changing the estimated ILDA statistics does not change this Transfer metric. The identical scores are therefore consistent with the evaluation procedure, and the repeated evaluation confirms the reported entries.
Part 3/3: Verified Provenance and Detailed Comparison of ILDA-CL-679
6. Q3: Was the ILDA-CL-679 row measured or copied?
The ILDA-CL-679 row was independently measured; it was not copied from the full-dimensional ILDA-CL results. We have now completed an independent re-evaluation of this configuration, retaining the first 679 coordinates of the original 768-dimensional CLIP feature.
The table below reports the per-dataset differences between ILDA-CL-679 and full-dimensional ILDA-CL for Transfer, Average, and Last.
Table R1. Accuracy differences between the 679-dimensional and 768-dimensional configurations.
We define Δ = accuracy of ILDA-CL-679 - accuracy of full-dimensional ILDA-CL. All entries are absolute accuracy differences in percentage points (pp). Positive values favor the 679-dimensional configuration, and negative values favor the 768-dimensional configuration.
| Dataset | Δ Transfer (pp) | Δ Average (pp) | Δ Last (pp) |
|---|---|---|---|
| Aircraft | — | 0.0000 | 0.0000 |
| Caltech101 | 0.0000 | +0.0173 | 0.0000 |
| CIFAR100 | 0.0000 | 0.0000 | 0.0000 |
| DTD | 0.0000 | -0.0904 | -0.0532 |
| EuroSAT | 0.0000 | 0.0000 | 0.0000 |
| Flowers102 | 0.0000 | +0.0065 | +0.0163 |
| Food101 | 0.0000 | -0.0059 | -0.0079 |
| MNIST | 0.0000 | 0.0000 | 0.0000 |
| Oxford Pets | 0.0000 | -0.0245 | -0.1090 |
| Stanford Cars | 0.0000 | -0.0087 | -0.0871 |
| Avg. | 0.0000 | -0.0106 | -0.0241 |
"—" denotes an inapplicable Transfer entry for the first task in the sequence.
Transfer remains unchanged across all applicable datasets, consistent with the evaluation pipeline explained above. The mean differences are -0.0106 pp in Average accuracy and -0.0241 pp in Last accuracy. The largest absolute per-dataset difference is 0.1090 pp, observed for Last accuracy on Oxford Pets. These results show a small measured effect of reducing the input-feature dimensionality from 768 to 679 on the evaluated benchmarks.
We will update Table 5 using the verified results, provide the corresponding evaluation records and higher-precision details in the supplementary material.
We thank the reviewer for their careful examination of our manuscript and code and for their constructive feedback. We hope our rebuttal has clarified the implementation details and addressed their questions about the reported results. We would welcome any remaining questions and would be happy to provide further clarification or supporting evidence.
Part 1/3: Adapter Implementation and Routing Consistency
Thank you for the careful code-level review and for recognizing the motivation for frozen-feature routing and the scope of our statistical analysis. We agree that the implementation, equations, and evaluation protocol must be consistent. We address your questions and implementation concerns below.
1. Adapter implementation and correction to Eq. (10).
You are correct that Eq. (10) does not accurately describe the implemented adapter. The implementation uses a nonlinear bottleneck adapter, whereas the original equation describes a linear LoRA weight update. We will correct both the equation and the associated terminology.
The implemented mapping is Mt(m)(h)=M0(m)(h)+Bt(m)(Dropout0.1(ReLU(At(m)(h)))).(10) Here, h is the post-attention hidden representation, and M0(m) and Mt(m) denote the frozen and adapted residual FFN mappings, respectively. At(m) and Bt(m) are affine down- and up-projections, including biases, with the fixed scaling absorbed into Bt(m). Dropout is disabled at inference.
We will consistently describe the implemented module as a task-specific nonlinear bottleneck adapter. This correction makes the manuscript accurately describe the architecture used in the reported experiments.
2. Q1: Which routing rule produced Tables 1-3?
Tables 1-3 were generated using the Mahalanobis distance implementation in lda.py, lines 194-200. We will replace the Euclidean expression in Eq. (8) with the implemented routing rule.
For s≥2, define vt,s(x)=Ws⊤[g(l∗)(x)−μ^t], Cs=N≤s−sWs⊤SW,sWs+ϵIks. Then dt,s2(x)=vt,s(x)⊤Cs−1vt,s(x),t^s(x)=argmin1≤t≤sdt,s2(x).(8) Here, SW,s denotes the unnormalized weighted matrix stored as self.Sw, N≤s=∑j=1sNj, and ϵ corresponds to reg_param. We will also align the surrounding definitions and normalization conventions with the implementation, so that the revised equations consistently describe the router used for the reported results.
3. Router layer, adapter placement, and feature definitions.
The motivation for coupling the router layer and adapter start layer was to permit reuse of the frozen prefix. The intended shared boundary is the input to the first adapted block, equivalently the output of the preceding frozen block. Its CLS representation supports routing, while the full token sequence can be retained for the selected adapter branch.
The released interfaces use different conventions: the routing hook captures a block output, whereas the auxiliary interface returns an adapter-start block input. We have checked these conventions and will explicitly document the feature-extraction boundary and adapter start layer used in the reported experiments, with consistent block indexing and interface definitions.
The coupling is an implementation choice rather than a requirement of ILDA. We will also distinguish the intended prefix-reuse strategy from the execution path implemented in the released code.