acceptodds
Under review as a conference paper at ICLR 2027

Adaptive-Depth Layer Grafting: Representation-Level Transfer Across Heterogeneous Tabular Feature Spaces

Abstract

Transfer learning is especially challenging when a label-scarce target comes from a disparate domain and shares the same prediction endpoint as a source model but uses a different feature space, with neither paired samples nor known feature correspondence. We ask whether knowledge encoded in a pretrained source hierarchy can still be leveraged under this schema mismatch. To that effect, we introduce Adaptive-Depth Layer Grafting (ADLG), which maps target inputs through a lightweight bridge into a selected internal source representation while keeping the retained source tail fixed. ADLG-Frozen trains only the bridge and the interface selection is handled separately, with quantile-aligned CKA used as one optional pre-fit screening method. On Census-Income→Adult, ADLG improves balanced accuracy over a model trained from scratch by 1.52, 2.56, and 2.89 percentage points with 0.5%, 2%, and 10% of the target labels, respectively, and by 2.36 points across the prespecified 0.5–20% label regime. It also gains 1.88 points over a tuned target-only MLP and 3.41 points in minority recall over the model trained from scratch. Additional studies span heterogeneous tabular settings, including structured multimodal and inter-modal representations, as well as multiclass and regression endpoints. Controlled audits show that attachment depth is consequential and that source-label information is most discernible at shallow interfaces. Overall, ADLG provides a compact mechanism for leveraging knowledge in a pretrained source hierarchy when schemas differ and target labels are scarce.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
heterogeneous transfer learning, representation transfer, label-efficient learning, representation similarity, tabular deep learning, multimodal transfer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BwZeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the problem of identifying the most suitable transfer layer to improve the performance of transfer learning by reusing the knowledge in certain layers of the network that benefit the most the target task. They propose a CKA-based method for layer selection, after which a bridge network is trained at the optimal layer to enable target prediction. The authors investigate this method in several experiments and demonstrate its effectiveness against training from scratch.

Critical strengths

The problem of identifying crucial layers to maximize the performance when applying transfer learning is an important problem. The authors motivate this clearly and propose a method to this end. They evaluated the proposed method against baselines including from scratch training to measure the effectiveness of the proposed method.

Critical weaknesses

The paper's motivation for identifying the optimal layer is interesting but I have concerns about the practicality and novelty of the proposed CKA based method. First of all, it requires training an additional split-local target probe as an additional component. Secondly, in certain cases, for example in Table 55, I can see that Random depth is performing surprisingly well. What makes it practically appealing to use the proposed layer selection strategy? Further, can this exact method be used for modern backbones such as CNNs and ViTs to solve other relevant transfer learning tasks, such as adapting the backbone itself for image classification ? The paper considers primarily the scenarios in which the network head is retained. In many practical transfer learning scenarios the target task is different and requires training a new head. I believe this is major limitation of current experiments.

Questions for authors

See above

4 comments
AuthorsOpenReview ↗

Response to Reviewer BwZe

We thank the reviewer for finding the paper sound and the problem important. Your summary showed us that our framing put too much weight on layer selection, and we have revised it. We uploaded a revised PDF. All changes are in blue.

W1: the extra target probe. The probe solves a problem specific to our setting. CKA compares two sets of activations row by row, which normally requires the same samples in both networks. Our source and target share no samples and no features, so their rows cannot be matched directly. The probe gives each target example a task score. We sort source and target examples by task score and place both on a common quantile grid. Rows at the same quantile then correspond by task outcome, so a pre-fit compatibility score needs no paired data.

The ordering audit shows the probe is what makes this work (Appendix G.5, Table 40). With probe ordering, CKA is 0.72 to 0.77 across depths. With random target ordering, it collapses to 0.01 to 0.04. A probe trained on shuffled labels gives intermediate values (0.25 to 0.55).

The probe also makes depth selection cheaper, not more expensive:

  • Compared with the alternative, inner validation, CKA with the probe is 2.5x faster (1.04 vs. 2.63 s median) and needs half the neural fits (2 vs. 4), with the same accuracy (79.309 vs. 79.313 BA, Table 62).
  • It is not part of the final model. The probe is used only during training to choose the depth, then discarded. The deployed model is just the bridge and the frozen tail, so it is no larger and no slower.

Revisions. Section 4.2 and the Figure 1 caption.

W2 / Q1: why random depth works well, and the appeal of layer selection. Random depth looks strong on Census only because every Census depth is good: all three are within about 1 BA point of each other (76.7 to 77.8, Table 51), so any pick lands near the best. This does not hold in general. On HCV → ILPD, depths span 58.7 to 64.5 BA. There, random depth averages 61.9, while validation-selected depth reaches 63.2 (+1.3) and stays 4.5 points above the worst depth. On WDBC, depth also changes accuracy materially (Figure 3(b)).

The right case is unknown in advance, so selection works as insurance: it matches random depth when depth does not matter, and beats it when it does. On Census, CKA gives this at 2.5x lower cost than validation (Table 62).

Revisions. Sections 4.2 and 7.7 now present depth selection this way.

W3 / Q2: CNNs and ViTs. ADLG extends naturally beyond MLP sources. Many vision backbones end in fully connected layers. VGG, for example, is f=Ln∘⋯∘Lm+1∘φf = L_n \circ \cdots \circ L_{m+1} \circ \varphi, with φ\varphi the convolutional extractor and Lm+1,…,LnL_{m+1}, \ldots, L_n fully connected layers. ADLG attaches at any of these layers. For depth k≥mk \ge m, a bridge G(k)G^{(k)} maps target features into the input of Lk+1L_{k+1}, and Fk=Ln∘⋯∘Lk+1F_k = L_n \circ \cdots \circ L_{k+1} stays frozen. The formulation, training and Theorem 1 apply unchanged, since they depend only on the frozen tail FkF_k.

In HAM10000 → Derm7pt, both domains include frozen image embeddings. ADLG improves BA by +2.99 points over the target multimodal model (5–20% labels).

Revisions. Section 8 now describes the extension to CNN backbones.

W4: retained output head. ADLG targets the common case where source and target predict the same outcome. Reusing the source head is then an advantage: it keeps the number of parameters learned from scarce labels small. In HCV → ILPD, class 1 means liver disease in both datasets. For a different prediction target, a small trainable head can replace the frozen head on top of the frozen tail.

Revisions. Section 8 now describes this scope and the new-head extension.

Additional: comparison with model stitching. We also compared ADLG directly with stitching (Bansal et al., 2021), which assumes a shared input space and keeps a separately trained target front network. ADLG maps raw target features through a compact bridge into the frozen tail. ADLG minus each of six stitching variants, low-label average (BA points; * significant after Holm correction):

Stitching variantCensus → AdultHCV → ILPD
Frozen front, linear+1.72*+2.06
Frozen front, ReLU+1.74*+2.04
Fine-tuned front, linear+4.00*+3.31
Fine-tuned front, ReLU+3.91*+3.79*
Ridge closed-form+6.18*+4.64*
Procrustes closed-form+19.25*+2.44

ADLG wins every comparison and trains about 20x fewer parameters on HCV → ILPD. On METABRIC → TCGA-BRCA, the direction holds: ADLG has higher mean BA than every variant at every depth (Appendix H).

Revisions. Section 7.6 (Table 3) and Appendix H.

Thank you again for your careful reading and valuable suggestions. We hope our responses and revisions address your concerns, and we welcome any further questions.

Area_Chair_zUAhOpenReview ↗

Reviewer feedback request [by AC]

Dear Reviewer,

Please kindly go through the authors' rebuttal and provide timely feedback. The discussion phase ends in 24 hours (11/18 23:59 AoE). If you have further concerns and need additional clarifications, please specify them and take the limited time the authors have into account. Thank you.

Best,

AC

AuthorsOpenReview ↗

To Reviewer BwZe: Follow-up on our response

Dear Reviewer BwZe,

Thank you for your constructive feedback. We have tried to address all your concerns and have updated the paper as you suggested, with changes marked in blue. As the discussion window is still open, please let us know if you have any further thoughts.

Best regards,

Authors

Reviewer_BwZeOpenReview ↗

Comment

Thanks to the authors for responding to my questions. I maintain my original score.

Reviewer_eUpGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors have proposed Adaptive Depth Layer Grafting (ADLG) for heterogeneous transfer learning between source and taregt domains. This learns a lightweight target to hidden bridge that connects target features to an intermediate layer of a pretrained source network while kjeeping remaining source layers and o/p head frozen. They also have used optional quantile-aligned CKA criterion which selects the attachment depth using representations from a supervised target probe. ADLG achieves favorable low-label results on cesnsus-income -> Adult, but its relative performance varies substantially across other transfer pairs.

Critical strengths

This paper addresses a meaningful heterogeneous-transfer setting in which source and taregt feature spaces are incompatible and explicit feature correspondence is unavailable. But this proposed method provides a a straightforward and lightweight way to reuse portions of an existing pretrained network. This also covers extensive experimental and diagnostic analysis while reporting the limitations and negative results.

Critical weaknesses

This paper has limited methodological novelty and insufficient evidence for th proposed mechanism. The central architetcurak component is a learned mapping from target i/p into an internal representation of a pretrained source network, followed by a frozen source network tail - which is conceptually very close to established mnodel-stitching and cross-modal adaptation frameworks (1,2). The experiments do not convincingly demonstrate that this adaptive-depth component provides a meaningful predictive advantage. [Appendix Table 55] the cka selection achieves a mean balanced acc of 0.793091, compared with 0.793127 for validation-based selection.Its differences relative to fixed-depth and random selection are also small, with confidence intervals including zero. This paper is motivated by the hypothesis that useful predictive knowleedge can be transferred from a pretrained source hierarchy through a learned bridge. But the source conditioned experiments reveal a substantial tension with this interpretation.

  1. Bansal, Yamini, Preetum Nakkiran, and Boaz Barak. "Revisiting model stitching to compare neural representations." Advances in neural information processing systems 34 (2021): 225-236.
  2. Shen, J., Li, L., Dery, L.M., Staten, C., Khodak, M., Neubig, G. & Talwalkar, A.. (2023). Cross-Modal Fine-Tuning: Align then Refine. <i>Proceedings of the 40th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 202:31030-31056 Available from https://proceedings.mlr.press/v202/shen23e.html.
Questions for authors
  1. What evidence establishes that the performance improvements primarily arise from pretrained source knowledge rather than architectural inductive bias?
  2. What is the measurable benefit of adaptive depth selection compared with simpler alternatives?
  3. Can the authors verify the shared prediction endpoint in HCV→ILPD?
3 comments
AuthorsOpenReview ↗

Response to Reviewer eUpG

We thank the reviewer for a constructive review and for finding ADLG a lightweight way to reuse a pretrained network. We uploaded a revised PDF. All changes are in blue.

W1: novelty relative to stitching and cross-modal adaptation. ADLG differs from both in setup and method.

  • Setup. Stitching (Bansal et al., 2021) assumes both networks take the same kind of input and pairs layers on shared inputs. ORCA (Shen et al., 2023) aligns a new modality at the input level. Our source and target have different features and no paired samples.
  • Method. Stitching keeps a separately trained target front network. ORCA fine-tunes the whole model. ADLG has neither. It maps raw target features through a compact bridge directly into the frozen source tail.

We compared ADLG with six stitching variants on identical splits. ADLG minus each variant, low-label average (BA points; * significant after Holm correction):

Stitching variantCensus → AdultHCV → ILPD
Frozen front, linear+1.72*+2.06
Frozen front, ReLU+1.74*+2.04
Fine-tuned front, linear+4.00*+3.31
Fine-tuned front, ReLU+3.91*+3.79*
Ridge closed-form+6.18*+4.64*
Procrustes closed-form+19.25*+2.44

ADLG wins every comparison, and trains about 20x fewer parameters on HCV → ILPD. On METABRIC → TCGA-BRCA, the direction holds: ADLG has higher mean BA than every variant at every depth (Appendix H).

Revisions. Section 2, Table 1, Section 7.6 (Table 3) and Appendix H.

W2 / Q2: measurable benefit of adaptive depth. The benefit is robustness at low cost.

  • When depth matters, selection pays off. On HCV → ILPD, accuracy spans about 6 points across depths (58.7% to 64.5%). Validation-selected depth reaches 63.2%, which is +1.3 over random depth.
  • When depth matters little, nothing is lost. On Census, CKA matches validation (79.309 vs. 79.313 BA, Table 62) at about 40% of its cost.
  • Random depth is a lower bound, and validation-based selection was above it on both pairs.

In practice, a user can take a fixed depth or run CKA as a cheap screen. No expensive depth search is needed.

Revisions. Sections 4.2 and 7.7 now present depth selection this way.

W3 / Q1: source knowledge vs. architectural bias. The gains come mainly from the architecture. This is a strength: ADLG does not need a closely matched pretrained source. We let each source condition use its own best depth, chosen without test data. A random frozen tail performs as well as, or better than, a pretrained one. Even then, ADLG beats a tuned target-only MLP on Census by +2.3 BA points. Pretraining still carries useful signal: a pretrained tail beats a shuffled-label tail by +0.98 points [0.79, 1.17] at Census depth 1 and by +0.83 points [0.43, 1.24] on WDBC. The main driver is the compact graft, which learns only a small bridge from the scarce labels.

Revisions. The abstract, Section 1 and Section 7.7 now state this.

Q3: HCV → ILPD endpoint. Yes, we verified it against the official UCI files.

  • HCV: class 1 is exactly the 75 hepatitis, fibrosis and cirrhosis cases. Class 0 is the 533 blood donors. The 7 suspect donors are excluded.
  • ILPD: class 1 is exactly the 416 liver patients.
  • Every label matches the raw data. Class counts match the saved records in all 40 split–budget settings.

Class 1 therefore means liver disease in both datasets. The endpoints are related but not identical: disease stage in HCV, liver-patient status in ILPD.

Revisions. Appendix C.4 now states this mapping.

Thank you again for the insightful reviews. Please let us know if you have any further comments or questions. We look forward to a fruitful discussion.

Area_Chair_zUAhOpenReview ↗

Reviewer feedback request [by AC]

Dear Reviewer,

Please kindly go through the authors' rebuttal and provide timely feedback. The discussion phase ends in 24 hours (11/18 23:59 AoE). If you have further concerns and need additional clarifications, please specify them and take the limited time the authors have into account. Thank you.

Best,

AC

AuthorsOpenReview ↗

To Reviewer eUpG: Follow-up on our response

Dear Reviewer eUpG,

Thank you for your constructive feedback. We have tried to address all your concerns and have updated the paper as you suggested, with changes marked in blue. As the discussion window is still open, please let us know if you have any further thoughts.

Best regards,

Authors

Reviewer_2HryOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes ADLG for transfer learning across heterogeneous tabular feature spaces. It learns a lightweight bridge from target inputs to an intermediate layer of a pretrained source model while keeping the remaining layers frozen. Quantile-aligned CKA is used as an optional depth selection strategy. Experiments across tabular, multimodal, and regression tasks show improvements in some low-label settings, although the benefits vary across datasets.

Critical strengths
  1. The paper addresses a practical transfer learning setting where source and target features differ without explicit correspondence. The proposed framework is simple and easy to implement.

  2. The experiments cover multiple datasets, label budgets, and prediction tasks. The authors also report negative transfer cases and conduct several controlled analyses.

Critical weaknesses
  1. Learning a bridge into a pretrained network resembles existing model stitching and adapter-based transfer approaches. The fundamental distinction is not sufficiently established.

  2. ADLG shows clear gains on only two of the five main transfer pairs, while WDBC favors the target-only MLP.

  3. The effectiveness of adaptive depth selection is not convincing. CKA performs similarly to validation-based selection, and several experiments consistently select the same depth.

  4. The source-condition analysis shows that correctly pretrained tails do not consistently outperform randomly initialized frozen tails.

  5. The practical scope remains limited to pretrained MLPs with shared prediction endpoints. Moreover, simple linear and tree-based methods outperform ADLG on the main Census benchmark.

Questions for authors
  1. Clarify the novelty relative to model stitching.

  2. Explain the inconsistent transfer gains across datasets.

  3. Provide stronger evidence for adaptive depth selection.

  4. Clarify the contribution of source pretraining over random frozen networks.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2Hry (Part 1/2)

(Part 1/2)

We thank the reviewer for a careful review and for recognizing our controlled analyses and our reporting of negative cases. We ran new experiments for each concern and uploaded a revised PDF. All changes are in blue.

W1 / Q1: novelty relative to model stitching and adapters. ADLG differs from stitching in both setup and method, and a new direct comparison confirms it.

  • Setup. Stitching (Lenc & Vedaldi, 2015; Bansal et al., 2021; Csiszárik et al., 2021) assumes both networks take the same kind of input. It pairs layers using inputs passed through both. Zero-shot variants also need matched anchors (Moschella et al., 2023; Maiorca et al., 2023). Our source and target have different features, no feature correspondence and no paired samples.
  • Method. Stitching keeps a separately trained target front network and learns a stitch from it into the source. ADLG has no front network. It maps raw target features through a compact bridge directly into the frozen source tail. On HCV → ILPD this means 704 target-trained parameters, against 4,864 to 25,664 for stitching. LoRA (Hu et al., 2022) alone does not handle incompatible input schemas.

We compared ADLG with six stitching variants on identical splits, with the method set fixed in advance. ADLG minus each variant, low-label average (BA points, paired 95% CI; * significant after Holm correction):

Stitching variantCensus → AdultHCV → ILPD
Frozen front, linear+1.72* [1.55, 1.87]+2.06 [0.35, 3.87]
Frozen front, ReLU+1.74* [1.58, 1.90]+2.04 [0.41, 3.82]
Fine-tuned front, linear+4.00* [3.84, 4.16]+3.31 [1.52, 5.15]
Fine-tuned front, ReLU+3.91* [3.85, 3.99]+3.79* [2.02, 5.54]
Ridge closed-form+6.18* [5.83, 6.53]+4.64* [2.31, 6.76]
Procrustes closed-form+19.25* [18.92, 19.58]+2.44 [0.40, 4.15]

ADLG wins every comparison. All six are significant on Census, and the gap grows to about +3 points at 1–2% labels. On METABRIC → TCGA-BRCA, the direction holds: ADLG has higher mean BA than every variant at every depth (Appendix H).

The results also show why stitching struggles here. On Census, the stitched model is 0.2 to 2.5 points worse than its own front network used alone. The front network adds many parameters to learn from few labels, and adds no useful structure.

Revisions. Section 2 and Table 1 now separate grafting from stitching. Section 7.6 and Table 3 add the comparison. Appendix H gives the protocol, per-depth results, confidence intervals and parameter counts.

Continued in Part 2/2 below.

AuthorsOpenReview ↗

Response to Reviewer 2Hry (Part 2/2)

(Part 2/2)

W2 / Q2: transfer gains across datasets. The pattern matches the design. ADLG's advantage is largest where labels are scarcest. On Census → Adult, it beats frozen-front stitching by +2.7 to +3.2 BA points with 1–2% of labels. ADLG learns only a compact bridge, so it pays off most when few target labels exist. When the target task is simple, a target-only model also does well. WDBC has only five target features, so it marks the edge of ADLG's useful range.

Revisions. Sections 1 and 8 now state when ADLG is most useful.

W3 / Q3: evidence for adaptive depth selection. Adaptive selection costs nothing when depth does not matter and helps when it does. Low-label mean BA (%):

Depth choiceHCV → ILPDCensus → Adult
Worst depth58.776.7
Random depth (average)61.977.4
Validation-selected depth63.277.7
Best depth (oracle)64.577.8
  • When depth matters (HCV), accuracy spans about 6 points. Selection gains +1.3 over random depth and +4.5 over the worst depth.
  • When depth matters little (Census), all choices are close. CKA and validation give 79.309 vs. 79.313 BA (Table 62).
  • Random depth acts as a lower bound. Validation-based selection was above it on both pairs.
  • The best depth is unknown in advance, so validation-based selection is a safe default. On Census, CKA matches it at about 40% of the cost.

Revisions. Sections 4.2 and 7.7 now present depth selection this way.

W4 / Q4: source pretraining vs. random frozen tails. This analysis shows a strength of ADLG: it does not need a closely matched pretrained source. We let each source condition use its own best depth, chosen without test data. A random frozen tail performs as well as, or better than, a pretrained one. Even then, ADLG beats a tuned target-only MLP on Census by +2.3 BA points. Pretraining still carries useful signal: a pretrained tail beats a shuffled-label tail by +0.98 points [0.79, 1.17] at Census depth 1 and by +0.83 points [0.43, 1.24] on WDBC. But the main driver is the compact graft. Only a small bridge is learned from the scarce labels. This matches the estimation term of Theorem 1.

Revisions. The abstract, Section 1 (contributions), Section 5.1 and Section 7.7 now state this.

W5: scope limited to MLPs with shared endpoints. ADLG targets tabular transfer where target features differ from source features in number and meaning. Here the target data does not fit the source input layer, so standard fine-tuning does not apply. This is a common setting, for example the same disease measured with different tests at different hospitals.

On Census, tuned linear and tree models are strong target-only baselines. ADLG addresses a different goal: reusing an existing pretrained network when target labels are scarce. Among methods that reuse a pretrained network, ADLG is the strongest, ahead of all six stitching variants (W1 / Q1).

Revisions. Sections 1 and 8 now state this scope.

Thank you again for the insightful reviews. Please let us know if you have any further comments or questions. We look forward to a fruitful discussion.

Thank you for the response. I have revised my review to reflect your clarifications and the additional results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.