- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates the problem of identifying the most suitable transfer layer to improve the performance of transfer learning by reusing the knowledge in certain layers of the network that benefit the most the target task. They propose a CKA-based method for layer selection, after which a bridge network is trained at the optimal layer to enable target prediction. The authors investigate this method in several experiments and demonstrate its effectiveness against training from scratch.
Critical strengths
The problem of identifying crucial layers to maximize the performance when applying transfer learning is an important problem. The authors motivate this clearly and propose a method to this end. They evaluated the proposed method against baselines including from scratch training to measure the effectiveness of the proposed method.
Critical weaknesses
The paper's motivation for identifying the optimal layer is interesting but I have concerns about the practicality and novelty of the proposed CKA based method. First of all, it requires training an additional split-local target probe as an additional component. Secondly, in certain cases, for example in Table 55, I can see that Random depth is performing surprisingly well. What makes it practically appealing to use the proposed layer selection strategy? Further, can this exact method be used for modern backbones such as CNNs and ViTs to solve other relevant transfer learning tasks, such as adapting the backbone itself for image classification ? The paper considers primarily the scenarios in which the network head is retained. In many practical transfer learning scenarios the target task is different and requires training a new head. I believe this is major limitation of current experiments.
Questions for authors
See above
4 comments
Reviewer feedback request [by AC]
Dear Reviewer,
Please kindly go through the authors' rebuttal and provide timely feedback. The discussion phase ends in 24 hours (11/18 23:59 AoE). If you have further concerns and need additional clarifications, please specify them and take the limited time the authors have into account. Thank you.
Best,
AC
To Reviewer BwZe: Follow-up on our response
Dear Reviewer BwZe,
Thank you for your constructive feedback. We have tried to address all your concerns and have updated the paper as you suggested, with changes marked in blue. As the discussion window is still open, please let us know if you have any further thoughts.
Best regards,
Authors
Comment
Thanks to the authors for responding to my questions. I maintain my original score.
Response to Reviewer BwZe
We thank the reviewer for finding the paper sound and the problem important. Your summary showed us that our framing put too much weight on layer selection, and we have revised it. We uploaded a revised PDF. All changes are in blue.
W1: the extra target probe. The probe solves a problem specific to our setting. CKA compares two sets of activations row by row, which normally requires the same samples in both networks. Our source and target share no samples and no features, so their rows cannot be matched directly. The probe gives each target example a task score. We sort source and target examples by task score and place both on a common quantile grid. Rows at the same quantile then correspond by task outcome, so a pre-fit compatibility score needs no paired data.
The ordering audit shows the probe is what makes this work (Appendix G.5, Table 40). With probe ordering, CKA is 0.72 to 0.77 across depths. With random target ordering, it collapses to 0.01 to 0.04. A probe trained on shuffled labels gives intermediate values (0.25 to 0.55).
The probe also makes depth selection cheaper, not more expensive:
Revisions. Section 4.2 and the Figure 1 caption.
W2 / Q1: why random depth works well, and the appeal of layer selection. Random depth looks strong on Census only because every Census depth is good: all three are within about 1 BA point of each other (76.7 to 77.8, Table 51), so any pick lands near the best. This does not hold in general. On HCV → ILPD, depths span 58.7 to 64.5 BA. There, random depth averages 61.9, while validation-selected depth reaches 63.2 (+1.3) and stays 4.5 points above the worst depth. On WDBC, depth also changes accuracy materially (Figure 3(b)).
The right case is unknown in advance, so selection works as insurance: it matches random depth when depth does not matter, and beats it when it does. On Census, CKA gives this at 2.5x lower cost than validation (Table 62).
Revisions. Sections 4.2 and 7.7 now present depth selection this way.
W3 / Q2: CNNs and ViTs. ADLG extends naturally beyond MLP sources. Many vision backbones end in fully connected layers. VGG, for example, is f=Ln∘⋯∘Lm+1∘φ, with φ the convolutional extractor and Lm+1,…,Ln fully connected layers. ADLG attaches at any of these layers. For depth k≥m, a bridge G(k) maps target features into the input of Lk+1, and Fk=Ln∘⋯∘Lk+1 stays frozen. The formulation, training and Theorem 1 apply unchanged, since they depend only on the frozen tail Fk.
In HAM10000 → Derm7pt, both domains include frozen image embeddings. ADLG improves BA by +2.99 points over the target multimodal model (5–20% labels).
Revisions. Section 8 now describes the extension to CNN backbones.
W4: retained output head. ADLG targets the common case where source and target predict the same outcome. Reusing the source head is then an advantage: it keeps the number of parameters learned from scarce labels small. In HCV → ILPD, class 1 means liver disease in both datasets. For a different prediction target, a small trainable head can replace the frozen head on top of the frozen tail.
Revisions. Section 8 now describes this scope and the new-head extension.
Additional: comparison with model stitching. We also compared ADLG directly with stitching (Bansal et al., 2021), which assumes a shared input space and keeps a separately trained target front network. ADLG maps raw target features through a compact bridge into the frozen tail. ADLG minus each of six stitching variants, low-label average (BA points; * significant after Holm correction):
ADLG wins every comparison and trains about 20x fewer parameters on HCV → ILPD. On METABRIC → TCGA-BRCA, the direction holds: ADLG has higher mean BA than every variant at every depth (Appendix H).
Revisions. Section 7.6 (Table 3) and Appendix H.
Thank you again for your careful reading and valuable suggestions. We hope our responses and revisions address your concerns, and we welcome any further questions.