- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether cross-model transferability reliably reflects target-side attribute decodability through orthogonal alignment and invertible rotation techniques. Orthogonal alignment preserves attribute information while adjusting shadow representations’ coordinate directions to better match target representations. Under identical preprocessing, alignment improves transfer performance, suggesting that poor transfer may partly arise from coordinate mismatch rather than difficulty in decoding target attributes.
Critical strengths
- The paper evaluates the gap between target-side decodability and cross-model transfer in graph–language models.
- This results show weak cross-model transfer can coexist with strong target-side attribute decodability.
Critical weaknesses
- Experiments are limited to two models (SI and Llama). It is interesting to include more recent models.
Questions for authors
- Can authors use other models in the experiments?
5 comments
Respectful Reply to the Reviewer: Revision Response and Request for Feedback
Dear Reviewer,
Thank you for reviewing our paper and sharing your feedback. We have addressed your comments in the discussion and provided additional experimental results. If you have a chance, we would appreciate hearing your thoughts on our response. Your feedback would help us identify what needs further clarification and present our findings more clearly in the revised manuscript.
If any questions remain, we would be happy to clarify. Thank you again for your time and consideration.
Please respond to the authors' rebuttal
Hello,
The discussion period ends soon. Please respond to the authors' last comments and note whether they changed your evaluation.
Thanks,
Your AC
Additional Validation on GLM-4-9B: Decodability–Transfer Gap and Alignment Gains
Thank you for suggesting that we evaluate our findings on additional models. We had also been considering this extension and had begun preparing the experiments. Your suggestion encouraged us to complete the evaluation on GLM-4-9B, testing whether both the decodability–transfer gap and the improvement from alignment also occur on another model. The results show that GLM-4-9B also exhibits a decodability–transfer gap and that alignment improves transfer, further supporting our main findings.
The experiments now cover three model families—S1, Llama, and GLM—within the graph-encoder–projector–language-decoder design studied in the paper. On GLM-4-9B, we independently trained the graph encoder and projector and evaluated the Frozen setting on NCI1 and BACE. Results are averaged over five model seeds, with five classifier initializations per model seed for MLP.
Here, T and S denote representations from the target and shadow models. The arrow means “probe training source → probe test source”: T→T measures decoding on held-out target representations, S→T measures transfer from shadow to target, and S→S measures decoding on held-out shadow representations.
1. Decoding and direct transfer
We compare Raw representations (standardized, without dimensionality reduction) with PCA-64 representations (standardized and reduced to 64 dimensions). For each path, the preprocessing is learned from the probe training data and then applied to its validation and test data:
In both spaces, target-trained probes perform better than probes transferred from the shadow model. For example, the Raw MLP reaches 0.710 on NCI1 and 0.924 on BACE for T→T, compared with 0.530 and 0.582 for S→T. The decoding–transfer gap therefore also appears on GLM.
2. Transfer after alignment
For alignment, we use a shared PCA-64 space built from paired validation representations (247 pairs for NCI1; 91 for BACE). We then compare S→T with and without Procrustes alignment under this same preprocessing. The probe is trained on shadow representations and evaluated on target representations in both cases.
Alignment improves mean transfer in all four comparisons, by 0.083–0.249 AUROC. All four pass Holm correction across this four-test family. The intervals use 10,000 paired bootstrap resamples of the five model seeds. Gains are calculated before rounding.
We also repeated the LR random-rotation controls on both datasets and all five model seeds, keeping PCA-64 fixed. Rotating the target training and test representations together leaves mean T→T AUROC unchanged. Applying the same rotation to both models changes mean S→T AUROC by less than 0.000002. On BACE, rotating only the target reduces S→T from 0.609 to 0.417, while independent rotations give 0.493. Relative to the common-rotation control, the changes and pointwise 95% intervals are −0.191 [−0.268, −0.114] and −0.115 [−0.200, −0.065], respectively. On NCI1, the corresponding changes are smaller and their pointwise intervals include zero. These controls show that changing coordinate directions can change transfer while preserving target-side linear decoding.
These experiments extend the findings to GLM-4-9B in the Frozen setting: target-trained probes decode attributes more accurately than transferred probes, alignment improves transfer, and random rotations can change transfer while preserving target-side linear decoding. We will include these experiments in the manuscript after completing the rebuttal experiments, together with detailed results for each of the five model seeds in the appendix.
Thank you again for this constructive suggestion. It helped us test our findings on another model and strengthen the evidence for distinguishing decodability from transfer.