acceptodds
Under review as a conference paper at ICLR 2027

Probing Transformer Representations Layer by Layer, Module by Module

Abstract

Which representation should be probed in a transformer-based foundation model? Recent work shows that the standard practice of probing the final layer is often suboptimal, as intermediate layers can provide better features. Yet, representations are still largely studied along depth alone, treating each transformer block as an atomic unit. In this work, we analyze transformer representations *layer by layer, module by module*. Through more than linear probing evaluations of vision foundation models (VFMs) and large language models (LLMs), we show that layer and module choices should be considered jointly: the best layer-module pair yields substantially larger gains over the final-layer residual stream than the best layer with the residual stream fixed. Moreover, module performance can vary substantially across layers, showing that useful representations cannot be characterized by depth alone. Building on these findings, we evaluate simple fixed probing strategies: the final-layer feedforward activation () for vision and the second normalization layer at % depth () for language. These strategies improve over the probing baseline in a majority of evaluations and recover, on average, % and % of the gains from probing the best layer-module pair for vision and language, respectively. For VFMs, these gains increase with the transfer difficulty between pretraining and downstream data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Foundation Models, Large Language Models, Linear Probing, Transformer Representations, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UK1TOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work conducts a comprehensive empirical study on the probing quality of different layers and modules of vision transformers and LLMs. The conclusion is, in general, the optimal layer id and activation varies a lot across models and tasks, but most ViT prefers the last layer residual stream and the second normalization layer at 60% of LLMs are fairly good choice.

Critical strengths

The empirical study has a board and comprehensive sweep on vision transformer. And this paper also touched the LLM as well. For ViT, the linear probing is an effective way to evaluate the feature and representation learning, which is an important topic requiring detailed understanding.

Critical weaknesses
  1. The analysis on LLM is not comprehensive enough. Especially authors only focus on very toy multi lingual text classification task.
  2. Authors didn't provide enough in-depth analysis on the reason why the certain position is a better feature for the task. This makes this study fully empirical so that the conclusion hard to generalize.
Questions for authors

See weakness In addition, what do you think about the necessity of studying the probing potential of LLMs? Also, does the conclusions above relevant to model architecture (e.g. prenorm vs postnorm) and training objective?

4 comments
  • Extend LLM experiments

    W1: The analysis on LLM is not comprehensive enough. Especially authors only focus on very toy multi lingual text classification task.

We thank the reviewer for this feedback. Our LLM experiments are conducted on the MTEB benchmark, which is commonly used in the literature, whether in research papers such as "Layer by Layer: Uncovering Hidden Representations in Language Models" by Skean et al. (ICML 2025), "KV-embedding: Training-free text embedding via internal KV re-routing in decoder-only LLMs" by Tang et al. (ACL 2026), "Your mixture-of-experts LLM is secretly an embedding model for free" by Li et al. (ICLR 2025), or to evaluate industrial embedding models, such as Gemini Embedding and EmbeddingGemma. For consistency with the vision experiments, we focused on 24 classification tasks spanning 31 languages and 14 domains. To address the reviewer's concerns, we provide additional experiments beyond classification on 3 new types of tasks (29 additional tasks in total): STS (Semantic Textual Similarity), reranking, and clustering. We observe that our conclusions hold: the fixed strategy improves performance on STS (+2.9 for Qwen and +4.6 for Llama), reranking (+1.5 and +7.8), and to a lesser extent for clustering (+0.5 and +0.4). These new results are in the revised version section 3.3, Tables 3, 4 and 5 and below. Due to the time constraints of the rebuttal period, we were not able to run the oracle experiment but our current results highlight the interest of an effective and efficient probing strategy. If the reviewer believes this if of importance, we will add the oracle performance in the final version of the paper.

ModelStrategyCIFAR-10SketchQuickDraw
CLIP-LBaseline97.274.959.6
CLIP-LAct@100+0.5+2.9+4.9
StrategyGemma3 1BGemma3 4BGemma3 12BGemma4 E2B
Baseline59.160.961.556.4
LN2@6061.1 (+2.0)63.1 (+2.3)64.2 (+2.6)62.2 (+5.7)
ModelStrategySTSClusteringReranking
Qwen3-1.7BBaseline39.443.029.2
LN2@60+2.9+0.5+1.5
Llama-3.2-1BBaseline40.746.326.1
LN2@60+4.6+0.4+7.8
  • Explanation of why our strategy are trustable

    W2: Authors didn't provide enough in-depth analysis on the reason why the certain position is a better feature for the task. This makes this study fully empirical so that the conclusion hard to generalize.

We thank the reviewer for this question. We first note that the additional experiments above show that our conclusions generalize to held-out models and tasks. We further discuss the reason why a certain position is a better feature for the task in the two parts below.

Why Act for VFMs. Act is the output of the nonlinearity in the expanded feed-forward block (dimension 4d). The same location was singled out by Ghiasi et al. (2022), who visualize ViT features and find that the outputs of the GELU layers in the feed-forward block are “significantly more interpretable than other layers”. They hypothesize that “the network exploits these high-dimensional spaces to store relatively disentangled representations”, whereas “compressing the features into a lower dimensional space may result in the jumbling of features”. Our results are consistent with this view: Act outperforms FC1, which has the same dimension but lies before the nonlinearity (see discussion in Appendix, Section C.1.7 and Table 23). This suggests that the nonlinearity, rather than the dimension alone, makes Act a better representation to probe. When averaged over patches, these activations summarize which features are present across the image.

Why LN2 at intermediate depth for LLMs. On depth, our finding is consistent with Skean et al. (2025) and with Lad et al. (2025), who describe the final layers of LLMs as specialized for next-token prediction, whereas intermediate layers build more general features. On the module, LN2 is a per-token normalized version of RC1. To test whether this normalization explains LN2’s advantage, we applied the same per-token normalization before pooling to all modules (Appendix C.2.3, Table 26 and below). Normalization improves most other representations, but no representation module outperforms LN2. Per-token normalization therefore explains part of LN2’s advantage, but not all of it, and the remaining part is an open question that we discuss in Appendix C.2.3.

ModuleQwen3-1.7BLlama-3.2-1B
RawNormalizedRawNormalized
LN154.454.657.057.2
MHA54.454.753.857.0
RC156.257.656.358.4
LN257.657.658.658.8
FC156.656.658.258.0
ACT56.256.456.558.6
FC255.255.554.758.0
RC256.157.555.557.7
  • Impact of training strategies

    Q2: Also, does the conclusions above relevant to model architecture (e.g. prenorm vs postnorm) and training objective?

    The literature mainly relies on pre-norm architectures, which is why we chose to focus on them. While we agree that we cannot claim our results extend to post-norm models, such a study is out of the scope of this paper. Regarding the training objective, however, our comprehensive experiments cover different training strategies: supervised (ViT), self-supervised (DINOv2/v3), autoregressive (AIMv2) and contrastive (CLIP). This suggests that the benefits of a good probing strategy hold consistently across training methods.

  • Utility of LLM probing

    Q1: In addition, what do you think about the necessity of studying the probing potential of LLMs?

We thank the reviewer for this question. Linear probing is a protocol for evaluating how reusable frozen embeddings are: a linear classifier on top of frozen features measures how much task-relevant information the representation exposes without any finetuning. Studying where to probe is therefore, more broadly, studying where a model’s most reusable embeddings lie. This question is actively studied for LLMs, as shown by Skean et al. (ICML 2025), who analyse representation quality layer by layer. Our work identifies a simple fixed strategy, LN2@60, that improves embedding quality across a wide range of tasks and models without any retraining. To show that this goes beyond linear probing, we added MTEB experiments on STS, reranking, and clustering tasks, which involve no trained classifier at all: they rely directly on cosine similarity and k-means over the embeddings. The fixed strategy improves performance on STS, reranking, and clustering, as stated earlier. The choice of location thus also matters when embeddings are reused as they are; beyond linear probing, probing strategies transfer effectively to these settings

Thanks for the reply. Regarding the LLM benchmark selection, I understand that you followed the previous work, but this does not imply that is correct.

  1. Previous work might be not perfect as well
  2. The "proper" eval shifts with time. GenAI research is moving fast and crazy. For example, in LLM, frontier labs hill climb different benchmarks every year (or even every month),

Just want to share my view, the new eval looks better and helped to improve this paper, so I raised my score to weak accept.

AuthorsOpenReview ↗

Thank you!

We thank the reviewer for their insightful feedback and interesting perspective. We agree that appropriate evaluation evolves quickly for LLMs. We are glad that the additional evaluation helped improve the paper, and we thank the reviewer for the increased score.

Reviewer_oqe7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors investigate the usefulness of representations from transformer foundation models across layers and modules. In their experiments, the authors systematically probe all module-layer combinations across a wide range of foundation models from the vision and language domain. Based on these empirical findings, the authors construct two probing strategies (one for vision and one for language) that provide a decent approximation of the oracle best probing performance.

Critical strengths
  1. The core idea of the paper, to study the effect of the layer and module position of a representation jointly, is novel and highly relevant. While several works have shown the value of intermediate-layer representations, the module position (to my knowledge) has been assumed to be fixed.
  2. The experiments cover a wide range of commonly used models and benchmarks, spanning both the vision and language domains. This broad coverage gives the empirical observations a good grounding.
  3. The paper is clearly structured and follows a consistent storyline.
Critical weaknesses
  1. Some of the differences between the different probing locations (as e.g. shown in Table 1) are relatively small. As the authors do not provide results across multiple seeds, it is difficult to tell how much of their effects is actually relevant and how much is intra-seed noise. I understand that, with the large number of probing experiments running, working across multiple seeds is very challenging. However, I encourage the authors to conduct an additional study in which they test multiple seeds for a subset of model-benchmark combinations.
  2. The authors develop fixed probing strategies based on the empirical results. While this makes sense, the evaluation of these probing strategies should occur on held-out tasks that were not used to develop them.
  3. In Figure 6a, the authors correlate the oracle gain with the mean probing error, using the mean probing error as a proxy for how much out-of-domain a downstream task is. This might be problematic, as it is generally easier to improve a model from a lower performance level than from, e.g., a near-saturated performance level. Measuring the distance using, e.g., a similarity measure would mitigate this confounder.
  4. In Table 1, it would help if the datasets were structured into in-domain and out-of-domain datasets. This would make it easier for the reader to assess the conclusion drawn, e.g., in line 269, where the authors cite "WILDS" as evidence of larger improvements on OOD tasks. As Table 1 includes only the two subtasks of the WILDS dataset (iWildCam and RxRx1), this can cause confusion.
Questions for authors
  1. Would the observations on the vision foundation models hold when probing the CLS token instead of the AP representation?
7 comments
  • Experiments over multiple seeds

    W1: Some of the differences between the different probing locations (as e.g. shown in Table 1) are relatively small. As the authors do not provide results across multiple seeds, it is difficult to tell how much of their effects is actually relevant and how much is intra-seed noise. I understand that, with the large number of probing experiments running, working across multiple seeds is very challenging. However, I encourage the authors to conduct an additional study in which they test multiple seeds for a subset of model-benchmark combinations.

    We thank the reviewer for raising the question of variability across runs. Nevertheless, all our linear probing experiments use a logistic regression head trained with the L-BFGS solver, which is deterministic. We are not training a linear layer with SGD as it can be done in some other studies. In addition, the train/test splits are fixed: they are either provided by the datasets themselves or, when no official split exists, generated once for all. Running the experiments with different seeds would therefore give identical results, which is why we do not report any variance.

  • Generalization over new tasks

    W2:The authors develop fixed probing strategies based on the empirical results. While this makes sense, the evaluation of these probing strategies should occur on held-out tasks that were not used to develop them.

    We thank the reviewer for this very relevant point. Our fixed strategies were derived only from classification tasks, and we agree that they should also be evaluated on held-out settings. Following your remark, we have added two such settings, in which the strategies were applied as they are, without any new tuning. First, for each modality, we added one new model that was not in the pool used to choose the strategy, and for both modalities the fixed strategy improves performance. Second, we evaluated Llama and Qwen on 3 MTEB types of tasks that were not used to choose the strategy (29 additional tasks in total): STS (Semantic Textual Similarity), reranking and clustering. The fixed strategy improves performance on STS (+2.9 for Qwen and +4.6 for Llama), reranking (+1.5 and +7.8), and to a lesser extent for clustering (+0.5 and +0.4). These new results are in the revised version section 3.3, Tables 3, 4 and 5 and below. Due to the time constraints of the rebuttal period, we were not able to run the oracle experiment but our current results highlight the interest of an effective and efficient probing strategy. If the reviewer believes this if of importance, we will add the oracle performance in the final version of the paper.

ModelStrategyCIFAR-10SketchQuickDraw
CLIP-LBaseline97.274.959.6
CLIP-LAct@100+0.5+2.9+4.9
StrategyGemma3 1BGemma3 4BGemma3 12BGemma4 E2B
Baseline59.160.961.556.4
LN2@6061.1 (+2.0)63.1 (+2.3)64.2 (+2.6)62.2 (+5.7)
ModelStrategySTSClusteringReranking
Qwen3-1.7BBaseline39.443.029.2
LN2@60+2.9+0.5+1.5
Llama-3.2-1BBaseline40.746.326.1
LN2@60+4.6+0.4+7.8
  • Out-of-distribution measure

    W3: In Figure 6a, the authors correlate the oracle gain with the mean probing error, using the mean probing error as a proxy for how much out-of-domain a downstream task is. This might be problematic, as it is generally easier to improve a model from a lower performance level than from, e.g., a near-saturated performance level. Measuring the distance using, e.g., a similarity measure would mitigate this confounder.

    W4: In Table 1, it would help if the datasets were structured into in-domain and out-of-domain datasets. This would make it easier for the reader to assess the conclusion drawn, e.g., in line 269, where the authors cite "WILDS" as evidence of larger improvements on OOD tasks. As Table 1 includes only the two subtasks of the WILDS dataset (iWildCam and RxRx1), this can cause confusion.

    We agree that a measure of how out-of-distribution a dataset is would be the holy grail, but computing a direct metric is out of reach in our case, for two reasons. First, existing datasest divergences do not scale to large datasets without subsampling, which would make them imprecise. For instance, the Optimal Transport Dataset Distance (OTDD, Alvarez-Melis & Fusi, 2020), that can be seen as the reference divergence between labelled datasets, is described by its authors as "prohibitive for all but toy datasets". Second, assuming the computation of such a metric would be tractable on large-scale datasets, there is no suitable distribution to compare against: the full pre-training data is often not publicly available and ImageNet, one realistic proxy, covers only a fraction of it. Finally, note also that Alvarez-Melis & Fusi (2020) validate OTDD precisely through its correlation with transfer performance (the exact quantity we monitor in Fig 6a): downstream performance thus remains the standard against which such distances are assessed.

  • Token probing strategie

    Q1:Would the observations on the vision foundation models hold when probing the CLS token instead of the AP representation?

    This is a good point. We initially chose to focus on the mean of the patch tokens for probing since AIMv2 has no CLS token, but the question remains relevant. We therefore evaluated our strategy with CLS token probing, and the results are in the revised version in Section C.1.5. Our claims still hold: Act@100 remains the best fixed strategy on the three selected datasets as shown in the new Table 19 and Figure 15 and below. We thank the reviewer for pointing this out. This experiment was needed, and it strengthens the paper.

StrategyCifar10SketchQuickdraw
Avg. patchCLSAvg. patchCLSAvg. patchCLS
ViT
Baseline94.994.545.144.543.543.0
Act@100+0.8-0.1+3.4+1.0+0.8-0.6
Oracle+1.2+0.7+7.3+5.0+4.1+2.4
DINOv2
Baseline99.099.173.975.059.658.2
Act@100+0.10.0+2.8+1.7+3.5+2.8
Oracle+0.1+0.1+2.8+1.7+8.4+6.0
DINOv3
Baseline98.898.977.478.761.061.8
Act@100+0.1-0.1+1.4+0.3+2.3+1.2
Oracle+0.1+0.1+1.4+0.3+7.0+1.2

Thank you for the clear rebuttal and the additional experiments. The clarifications and additional analyses address most of my concerns. Two points remain:

Table 1. As the table structure remains unchanged in the revision, my original concern remains.

Figure 6a. I agree that estimating distance to the true pretraining distribution is difficult when the training data are unavailable. My concern is that baseline probing error also sets the maximum possible accuracy gain, so the correlation may partly reflect greater headroom on harder tasks. I would scope the conclusion from Figure 6a to empirical transfer difficulty. For the distribution-shift claim, the existing CIFAR-10-C experiment offers a useful check. Does the trend with corruption severity persist when gains are expressed as the fraction of baseline errors removed?

Reviewer_V5sXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper conducts a large-scale linear probing study across different layers and modules within Transformer blocks in vision foundation models (VFMs) and large language models (LLMs). By comparing the classification performance of representations extracted from the models, it proposes Act@100 for VFMs and LN2@60 for LLMs as fixed probing strategies that are useful across multiple evaluations. For VFMs, it further examines how the gains from selecting probing locations vary with pretraining scale, transfer difficulty, and distribution shift. These extensive experiments provide useful empirical evidence for subsequent research that uses the internal representations of VFMs and LLMs.

Critical strengths
  • The paper systematically analyzes performance differences across probing layers and modules through extensive experiments on diverse models and datasets. It provides useful insights for subsequent research by identifying probing locations.

  • The paper demonstrates that the choice of module within a Transformer block matters alongside layer depth. In particular, it shows the effectiveness of using the output of the second normalization module (LN2) at approximately 60% depth in LLMs, providing empirical support for considering layer and module choices jointly.

Critical weaknesses
  • The evaluation is limited to classification using conventional linear probing, and the paper offers limited novelty in both methodology and perspective. Although a broad range of models and datasets is evaluated, it remains unclear whether the same observations hold for other types of models and downstream tasks.

  • Although the paper includes supplementary analyses of distribution shift, representations before and after finetuning, and feature dimensionality, its analysis of the factors underlying performance differences across probing locations is insufficient. In particular, it does not adequately explain why VFMs and LLMs favor different modules and depths. Furthermore, the usefulness of the proposed fixed strategies has not been directly validated on models outside the evaluation, making it difficult to assess the generalizability of the findings.

Questions for authors
  • The paper presents Act@100 for VFMs and LN2@60 for LLMs as useful fixed probing strategies. Have the authors conducted additional experiments to investigate why the two model types favor different modules and depths? If not, what hypotheses do the authors have to explain these differences?

  • Do similar performance trends hold for downstream tasks such as object detection and segmentation in VFMs? For LLMs, do the observations regarding layer and module selection extend beyond text classification to next-token prediction or downstream tasks involving generation?

3 comments
  • Generalization to new model and new downstream tasks:

W1: The evaluation is limited to classification using conventional linear probing, and the paper offers limited novelty in both methodology and perspective. Although a broad range of models and datasets is evaluated, it remains unclear whether the same observations hold for other types of models and downstream tasks.

Q2: Do similar performance trends hold for downstream tasks such as object detection and segmentation in VFMs? For LLMs, do the observations regarding layer and module selection extend beyond text classification to next-token prediction or downstream tasks involving generation?

We thank the reviewer for this very relevant point. Our fixed strategies were derived only from classification tasks, and we agree that they should also be evaluated on held-out settings. Following your remark, we have added two such settings, in which the strategies were applied as they are, without any new tuning. First, for each modality, we added one new model that was not in the pool used to choose the strategy, and for both modalities the fixed strategy improves performance. Second, we evaluated Llama and Qwen on 3 MTEB types of tasks that were not used to choose the strategy (29 additional tasks in total): STS (Semantic Textual Similarity), reranking and clustering. The fixed strategy improves performance on STS (+2.9 for Qwen and +4.6 for Llama), reranking (+1.5 and +7.8), and to a lesser extent for clustering (+0.5 and +0.4). These new results are in the revised version section 3.3, Tables 3, 4 and 5 and show below. Due to the time constraints of the rebuttal period, we were not able to run the oracle experiment but our current results highlight the interest of an effective and efficient probing strategy. If the reviewer believes this if of importance, we will add the oracle performance in the final version of the paper.

We agree that our study on vision models focuses on classification tasks, but we strongly believe that this choice allows us to study the task thoroughly, across many models and datasets. Adding a study on dense tasks would require additional computation time and careful design, and is beyond the scope of our paper.

ModelStrategyCIFAR-10SketchQuickDraw
CLIP-LBaseline97.274.959.6
CLIP-LAct@100+0.5+2.9+4.9
StrategyGemma3 1BGemma3 4BGemma3 12BGemma4 E2B
Baseline59.160.961.556.4
LN2@6061.1 (+2.0)63.1 (+2.3)64.2 (+2.6)62.2 (+5.7)
ModelStrategySTSClusteringReranking
Qwen3-1.7BBaseline39.443.029.2
LN2@60+2.9+0.5+1.5
Llama-3.2-1BBaseline40.746.326.1
LN2@60+4.6+0.4+7.8
  • Explanation of why our strategy are trustable

    W2: Although the paper includes supplementary analyses of distribution shift, representations before and after finetuning, and feature dimensionality, its analysis of the factors underlying performance differences across probing locations is insufficient. In particular, it does not adequately explain why VFMs and LLMs favor different modules and depths. Furthermore, the usefulness of the proposed fixed strategies has not been directly validated on models outside the evaluation, making it difficult to assess the generalizability of the findings.

    Q1: The paper presents Act@100 for VFMs and LN2@60 for LLMs as useful fixed probing strategies. Have the authors conducted additional experiments to investigate why the two model types favor different modules and depths? If not, what hypotheses do the authors have to explain these differences?

    We thank the reviewer for this question. We address the two parts below.

    Why Act for VFMs. Act is the output of the nonlinearity in the expanded feed-forward block (dimension 4d). The same location was singled out by Ghiasi et al. (2022), who visualize ViT features and find that the outputs of the GELU layers in the feed-forward block are “significantly more interpretable than other layers”. They hypothesize that “the network exploits these high-dimensional spaces to store relatively disentangled representations”, whereas “compressing the features into a lower dimensional space may result in the jumbling of features”. Our results are consistent with this view: Act outperforms FC1, which has the same dimension but lies before the nonlinearity (see discussion in Appendix, Section C.1.7 and Table 23). This suggests that the nonlinearity, rather than the dimension alone, makes Act a better representation to probe. When averaged over patches, these activations summarize which features are present across the image.

    Why LN2 at intermediate depth for LLMs. On depth, our finding is consistent with Skean et al. (2025) and with Lad et al. (2025), who describe the final layers of LLMs as specialized for next-token prediction, whereas intermediate layers build more general features. On the module, LN2 is a per-token normalized version of RC1. To test whether this normalization explains LN2’s advantage, we applied the same per-token normalization before pooling to all modules (Appendix C.2.3, Table 26 and below). Normalization improves most other representations but no representation module outperforms LN2. Per-token normalization therefore explains part of LN2’s advantage but not all of it, and the remaining part is an open question that we discuss in Appendix C.2.3.

ModuleQwen3-1.7BLlama-3.2-1B
RawNormalizedRawNormalized
LN154.454.657.057.2
MHA54.454.753.857.0
RC156.257.656.358.4
LN257.657.658.658.8
FC156.656.658.258.0
ACT56.256.456.558.6
FC255.255.554.758.0
RC256.157.555.557.7

Dear reviewer V5sX,

As the discussion period comes to an end, we would greatly appreciate your feedback on whether our response addresses your concerns. We are also happy to answer any further questions or requests.

The Authors

Comments on the paper

1 comment

We thank all the reviewers for their careful reading and insightful feedback. We are glad that they found our study systematic and extensive, our module-level analysis novel and relevant, and our paper clearly structured.

Their constructive comments led to additional experiments that strengthen the paper:

  • Generalization beyond the models and tasks used to derive the rules: we add 2 new models, for VFMs and LLMs, and 3 new types of MTEB tasks (29 additional tasks in total) to show the transferability of our strategies.
  • Insight into why the rules work: in LLMs, we show that RC1, normalized per token without learned weights, nearly matches the performance of LN2. This suggests that the gain of LN2 comes mostly from normalizing the residual stream, to which logistic regression is sensitive. In VFMs, we relate the success of Act to the expansion and non-linearity of the MLP, which projects low-level patch features into semantically richer representations.
  • Study on CLS token probing: Extending our study to the CLS token shows that the conclusions remain the same.

We will add all new results to the final version, and we have uploaded a revised version with all additional experiments and changes in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.