- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper examines how geometric similarity relates to the linguistic information encoded in sentence representations. The main experiments compare the [CLS] embedding, mean-pooled token embeddings, and a randomly selected token embedding from four pretrained encoders. The evaluation combines cosine similarity analyses across six languages, 216 linguistic probing tasks in FlashHOLMES, and structural probing on the CN and SSR datasets. The main finding is that representations with low cosine similarity can support similar probing performance. In some settings, a structural encoder-decoder trained on one representation also transfers to another without retraining, with particularly strong results for BERT and ELECTRA. The authors interpret these findings as evidence that sentence embeddings have internal structure that raw cosine similarity does not adequately capture.
Critical strengths
The fixed-probe transfer in Section 5 is the most interesting observation. A structural encoder-decoder trained on one representation can work on another without retraining, even when their direct cosine similarity is low. The BERT and ELECTRA results in Figure 5 and Appendix Table 7 provide stronger evidence than similar scores from separately trained probes. They show that some of the tested linguistic information is accessible to the same learned mapping across representation types.
Critical weaknesses
1 The central explanation is not established by the experiments The Abstract, Introduction, and Conclusion attribute the apparent shortcomings of sentence embeddings to shallow similarity measures and argue for a particular account of their internal organization. These are central contribution claims. Section 5 defines encoding in a similar manner through transfer of a fixed probe, and the experiment does address that operational definition. The later inference about where information is encoded, and the broader explanation of why cosine gives a distorted picture, require evidence beyond successful transfer. A shared decoder can recover the same labels from inputs whose information is organized differently. For example, two representations could share a task-relevant linear subspace while differing in large background components orthogonal to it. Their direct cosine could then be low, while the same projection recovers the shared information. This is a simple alternative explanation that the experiments do not rule out. The t-SNE plots cannot resolve the issue either: they show the output of a learned compression, not the locations of features in the original embeddings. Appendix B.3 makes this distinction particularly important. On STS-B sentence pairs, the [CLS] and mean-pooled cosine scores have Spearman correlations of 0.70–0.81 for DeBERTa and 0.80–0.83 for ELECTRA. This suggests related geometric information despite low direct cross-representation cosine. The paper needs to test whether centering, standardization, dominant-component removal, or linear alignment explains the observations. The Euclidean-distance plots in Appendix B.4 do not isolate these possibilities. The broader limitation of raw similarity measures is also not new: Timkey and van Schijndel (2021), already cited in the paper, show how dominant dimensions can obscure representational quality. The present experiments would need to establish what they reveal beyond such effects. They currently support a narrower observation about shared accessibility of linguistic information, without establishing the proposed explanation or localization claim. This gap affects the main scientific contribution, rather than only the wording of the conclusions.
2 The remaining empirical contribution is insufficiently established The structural-probing method has direct precedents in the cited work of Nastase and Merlo (2023, 2024). The 2023 study already examines two-dimensional sentence embeddings, and the 2024 study uses the 32 × 24 CNN/VAE-style probe with a five-dimensional latent representation and contrastive structural examples. Section 5 appropriately acknowledges reuse of the earlier setup and data. The present comparison of [CLS], mean-pooled, and random-token representations, especially fixed-probe transfer, extends those studies. Its contribution therefore depends on what new and well-supported conclusion this comparison establishes. The narrower transfer finding still needs stronger validation. DeBERTa's SSR F1 scores in Figure 5 are 0.13–0.15, including when training and testing use the same representation type, despite its stronger morphology and syntax results in Section 4. The tasks differ, so this does not identify the cause. Section 5.2 suggests a mismatch between DeBERTa and the probe, but does not test it. Reshaping 768 coordinates into a 32 × 24 array imposes a particular neighborhood structure for the CNN. A simpler MLP-based encoder-decoder would help assess how much the results depend on that choice. The near-perfect BERT results on CN in Appendix Table 7, including transfer involving randomly selected tokens, also need a clearer generalization test. Contextualized tokens can contain sentence-level information, so high scores are plausible. However, the random split does not establish transfer to unseen lexical items or surface templates. The target structural categories are intentionally shared across splits; the concern is overlap in how those categories are instantiated. Without controlling that overlap, the results do not show how far the transfer extends beyond this dataset construction. The checkpoint selection further limits comparisons across models. Section 3 combines multilingual BERT and XLM-RoBERTa with English-pretrained DeBERTa and ELECTRA in the six-language analysis. Differences in language coverage, tokenization, and pretraining data can affect these comparisons. The discussion should separate these factors from effects attributed to architecture or pretraining objective. The model descriptions also need to match the checkpoints. Footnote 1 identifies microsoft/deberta-v3-base, but Section 3.1 describes the original DeBERTa training setup when contrasting it with ELECTRA. DeBERTa-v3 uses replaced-token detection rather than the original masked-language-modeling objective. On p. 3, the next-sentence-prediction discussion also cites Liu et al. (2019), although RoBERTa removes that objective, as the manuscript itself notes on p. 4. Please clarify which objectives apply to each model and use those distinctions when interpreting the representation differences.
Questions for authors
- Central explanation. What evidence distinguishes the proposed account of internal organization from representation-specific offsets, dominant components, or a shared linear task subspace? How do the results change after controls such as centering and standardization? Please clarify whether the location claim concerns the original embeddings or only the learned latent space, and what evidence supports it.
- Transfer validity. Does the cross-representation transfer persist with a simpler probe and splits that control lexical and surface-template overlap? What explains DeBERTa's low SSR performance, and which conclusions remain supported across models? These checks are needed to establish the scope of the transfer finding.
- Incremental contribution. Please distinguish the new findings from the structural information already recovered in Nastase and Merlo (2023, 2024). Once the conclusions are limited to what the current experiments establish, what additional scientific claim is supported by the cross-representation comparison?
5 comments
Thanks for the response. The rebuttal clarifies that the author's intended claim is about raw embeddings and that Section 5 is meant as evidence of shared accessibility of structural information. However, it still does not establish the stronger claim about where/how information is organized in the representations, nor rule out simpler geometric explanations. It also does not adequately address robustness to probe architecture, the DeBERTa anomaly, novelty beyond prior probing work, or the checkpoint/objective-description issues. I would maintain the score.
Thank you for considering our answer. We respectfully disagree with some of the assertions:
robustness to probe architecture -- we disagree that the syntactic-semantic structure we probe for should be accessible through any probe. We show that it is accessible through a probe that relies on finding repetitive patterns. The fact that DeBERTa does not encode information in the same manner as the other three models does not invalidate our study and conclusions, but provides a starting point in understanding the impact of disentangling position and token embeddings within the architecture.
where/how the information is encoded -- our results show conclusively that the structural information we mine for is encoded the same way across variations. This is the important finding that in conjunction with the shallow (i.e. dimension-level) analysis shows the complex nature of sentence embeddings. This is an important step towards distilling, or making more overt, symbolic structures encoded in the continuous sentence embeddings. Localizing this type of information does not have a bearing on this particular investigation, although work on this specific topic has been done, but we suppress the citation for anonymity reasons.
Central explanation, results across languages and contribution.
Thank you for the comments. We will address the questions in order below, and will add a few clarifications related to issues raised in the "Critical weaknesses" section, which we include with the question they are related to, or at the end:
1a. (central explanation) We use as input raw sentence embeddings obtained from the last layer of each model. We do not normalize or recenter them on purpose, as we aim to understand how and what information is encoded in these raw representations. The use the CNN architecture is specifically to go beyond linear probing to more complex patterns. A FFNN has a maximum field, so to speak, allowing for all combinations of dimensions, whereas the CNN is more restrictive, and rewards repetitive patterns. The latent clusters are presented as an added proof that structure was distilled from sentence embeddings. None of the experiment scores (F1s) are based on the latents. The F1 scores are computed based on the choices at the output of the VAE, among the out+ and the sentences in Out-
1b. (decoder recovering labels) The decoder does not recover a label. The output of the decoder is a sentence representation, that is then compared to the candidates. Through the use of contrastive examples, we assumed – and the experiments confirm it – that by using as a positive outcome a sentence that has the same pattern as the input sentence but is not the same, and use negative examples that have different patterns, will force the latent space to encode the shared information, i.e. the patterns.
2a. (Transfer validity) The aim in using the CNN architecture was to show that there are repetitive patterns in the sentence embeddings, which the high results confirm. Whether encoder&decoders using only FFNNs will lead to higher results or not does not invalidate the results obtained with the more constraining architecture. We would like to reiterate that pattern information is not provided to the system, which only manipulates sentence embeddings, both as input, and as the output produced by the decoder and the candidate answers.
2b: (DeBERTa) DeBERTa treats token and position embeddings differently than the other three models, and this may explain the difference in performance. It’s low performance though is useful, because it shows the results are not due to a very powerful probe – all experiments are done with a VAE that has the same architecture and the same hyperparameters. Probing how DeBERTa embeddings differ is the subject of a different investigation.
Additional explanations to issues raised in the "Critical weaknesses" section:
(recentering/filtering of embeddings) Our aim was to study the raw embeddings, without any kind of transformation, to understand what the model encodes in them and how they are similar or different. The progression of experiments and the methods used serve to show that, and to counter some alternative explanations (e.g. using both cosine and euclid to investigate the relative positions of the embedding variations in the space, using tasks grouped by the type of information they probe for).
(the tSNE plots vs location of features) The cross-representation testing does show that the same information is encoded in the same way in the different embedding variations. A model learned on one representation type is applied on all the others (each row on the heatmaps shows the results obtained by using a model trained on representations indicated on the y-axis and tested on the others with no modifications). Should the information not be encoded in the same manner across representation types, either or both the indicators – the F1 scores and the latent projection – would show different results. But that is not the case.
additional results: Section 5, a sentence that appears in the input in the training data does not appear as input in an instance in the test data. The results on a similar dataset for French shows the same profile as for English, with the F1 lower by approx 0.02. We have used gemma-4-31b-it to generate similar datasets in English and French with wider range of lexical variation. Unofficially (manual check of the datasets is ongoing), the F1 results profiles are similar (for French for Electra, all F1 are between 0.91 and 0.96 for all cross-testing set-ups, for English they are slightly lower 0.83-0.86), and further support our hypothesis.