acceptodds
Under review as a conference paper at ICLR 2027

On the Persistent Effects of Lexicality in Large Language Models

Abstract

Representations extracted from large language models (LLMs) play an important role in many downstream applications. However, the structure of these representations is often influenced by lexical overlap rather than semantic content. Our understanding of the relationship between this lexical influence and semantic content, and its implications for downstream tasks, remains limited. In this work, we investigate representations using adversarial semantic stress tests to quantify the effect of lexical overlap relative to semantic content. We find that lexical influence extends across model depth and persists across training regimes and objective functions, including models trained for semantic similarity. Moreover, in autoregressive models we observe an intermediate regime in which semantic discrimination weakens alongside reduced linear accessibility of surface-form information, indicating that reduced lexical accessibility does not imply stronger semantic discrimination. We relate this regime to changes in representation geometry and find that its semantic weakness coincides with a plateau or partial contraction in intrinsic dimension. We further demonstrate the effect of lexical influence on downstream uses of LLMs using summarization and model editing as case studies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, language model interpretability, text embeddings, lexical semantics, semantic similarity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1a77OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the influence of lexical similarity on the semantic representations learned by large language models. The authors analyze how this influence varies across network layers, model architectures, and training regimes, using several benchmarks and probing techniques. They also investigate possible connections between lexical sensitivity, intrinsic dimensionality, and semantic performance, and explore the practical implications for summarization evaluation and model editing.

Critical strengths

The paper addresses an important research question: understanding how lexical similarity influences semantic representations in language models. This topic is relevant to a broad audience and has potentially important practical implications.

Some of the results, particularly those concerning the effects of lexical similarity on summarization evaluation and model editing, are potentially interesting and deserve further investigation.

Critical weaknesses

Weakness 1: Lack of a coherent scientific narrative. The paper presents a large number of experiments addressing different aspects of lexical and semantic representations, but their motivations and connections are often unclear. The analyses do not appear to be organized around a well-defined research hypotheses, and results are presented without explaining what specific conclusions can be drawn from them. As a consequence, the manuscript reads more like a collection of loosely connected observations than a systematic investigation.

Weakness 2: Insufficient evidence supporting the main conclusions. Several interpretations appear stronger than the numerical evidence. In particular, the proposed connection between lexical influence, semantic discrimination, and intrinsic dimensionality remains largely correlational, and the depth-dependent profiles are not always consistent across models and benchmarks. Moreover, failures in semantic discrimination are frequently interpreted as evidence of lexical influence, although other explanations are possible. Overall, the paper does not convincingly demonstrate the underlying mechanisms of the phenomena it describes.

Questions for authors

Could the authors formulate more explicitly the main scientific hypothesis of the paper, and explain which experiments provide the strongest evidence supporting it? In particular, what is the central new conclusion that emerges from combining the different analyses?

How can the authors distinguish failures caused specifically by lexical overlap from more general failures of semantic discrimination? Would it be possible to introduce controls in which lexical and semantic similarity are varied independently?

2 comments
AuthorsOpenReview ↗

Response to reviewer. (1)

Weakness 1: Lack of a coherent scientific narrative. The paper presents a large number of experiments addressing different aspects of lexical and semantic representations, but their motivations and connections are often unclear. The analyses do not appear to be organized around a well-defined research hypotheses, and results are presented without explaining what specific conclusions can be drawn from them. As a consequence, the manuscript reads more like a collection of loosely connected observations than a systematic investigation.

Could the authors formulate more explicitly the main scientific hypothesis of the paper, and explain which experiments provide the strongest evidence supporting it? In particular, what is the central new conclusion that emerges from combining the different analyses?

Our objective is to systematically characterize how lexical influence persists and evolves across model depth and training regimes, along with its representational characteristics and downstream consequences.

Our findings show that lexical influence persists even when semantic discrimination improves through adaptation or learned readouts. The intermediate regime further reveals that reduced accessibility of token identity can coincide with weakening semantic discrimination, demonstrating that reduced lexical accessibility does not necessarily lead to semantic robustness. While [1] and [2] investigate linguistic abstraction and representation quality across depth, our work focuses on a distinct problem by systematically characterizing failures in semantic similarity arising from lexicality, their relationship with representation geometry, and their consequences for downstream applications.

The strongest evidence comes from § 3 and § 4, which establish the persistence of lexical influence across training regimes and readouts, and its relationship with lexical decodability and semantic fidelity. § 5 extends this investigation to representation geometry, identifying associated changes in intrinsic dimension and demonstrating that spectral interventions can modestly improve semantic discrimination. § 6 examines the downstream consequences of lexical influence in summarization evaluation and model editing locality.

The combined analyses establish that improvements in semantic representation quality do not necessarily translate into robustness to misleading lexical cues. The manuscript already develops these connections and their conclusions within the respective sections. We will further make the research questions and main findings clearer in the Introduction.

[1] Emergence of a High-Dimensional Abstraction Phase in Language Transformers

[2] Layer by Layer: Uncovering Hidden Representations in Language Models

AuthorsOpenReview ↗

Response to reviewer. (2)

Weakness 2: Insufficient evidence supporting the main conclusions. Several interpretations appear stronger than the numerical evidence. In particular, the proposed connection between lexical influence, semantic discrimination, and intrinsic dimensionality remains largely correlational, and the depth-dependent profiles are not always consistent across models and benchmarks. Moreover, failures in semantic discrimination are frequently interpreted as evidence of lexical influence, although other explanations are possible. Overall, the paper does not convincingly demonstrate the underlying mechanisms of the phenomena it describes.

How can the authors distinguish failures caused specifically by lexical overlap from more general failures of semantic discrimination? Would it be possible to introduce controls in which lexical and semantic similarity are varied independently?

The lexical influence experiments in § 3 are explicitly designed to examine how lexical similarity affects semantic discrimination. Each triplet contains an anchor, a meaning preserving paraphrase, and a semantically different distractor with substantial lexical overlap. Success requires the paraphrase pair to be closer than either pair involving the distractor. Thus, the evaluation specifically tests whether lexical cues distort semantic similarity rankings rather than interpreting low semantic task performance as lexical influence. Separately, § 4.2 evaluates broader semantic fidelity through MTEB, where several tasks also exhibit intermediate weakening. Appendix H.2 further examines the role of lexical overlap in MTEB retrieval, showing that it affects absolute performance while largely preserving the depth dependent trajectory. These evaluations therefore examine distinct but related aspects of representation quality.

To further isolate the effect of lexical overlap, we construct a controlled subset from 5,000 CounterFact records. For each anchor, we select two distinct paraphrases preserving the anchor entity and two neighborhood distractors referring to the same different entity. We retain records where both the paraphrase (P) and distractor (D) have High and Low lexical overlap variants, guaranteeing that P High has greater overlap than D Low and D High has greater overlap than P Low. These filtering criteria yield 1,929 eligible instances. This design preserves the semantic relationships within each pair while varying their lexical similarity to the anchor. The results for Gemma instruct are provided in the table below. Complete results across Gemma and Llama models are provided in a new Appendix § M.

Table: Success Rate under independently controlled paraphrase (P) and distractor (D) lexical overlap.

Layer(P High, D Low)(P Low, D Low)(P High, D High)(P Low, D High)
1100.098.499.289.8
499.797.098.785.5
899.898.098.887.8
1299.695.895.675.4
1698.189.689.166.7
2096.985.686.266.0
2395.283.482.761.3
2696.385.884.363.9
3099.395.693.982.3
3699.899.098.893.8
4099.899.499.596.9
4499.899.099.396.5
4899.698.198.494.2

The results show a consistent effect of lexical overlap across depth, particularly at intermediate layers. At layer 23, success is highest (95.2%) when paraphrase overlap is high and distractor overlap is low, and lowest (61.3%) when these conditions are reversed. The two remaining conditions yield similar success rates of approximately 83%.

Differences in depth-dependent profiles are expected given variations in model architectures, training objectives, representation readouts, and benchmark characteristics. Our conclusions do not require identical trajectories across these settings. Rather, we establish the persistence of lexical influence across models and adaptation regimes, while identifying a recurring intermediate regime in the autoregressive models studied. In this regime, discrimination weakens despite progression through model depth, before recovering in later layers. This pattern is not confined to the adversarial stress tests. Several MTEB tasks exhibit similar intermediate weakening or stagnation (§ 4.2), while OLMo checkpoints show that the regime emerges early during pretraining and persists through subsequent training (§ 3). This provides evidence for a recurring depth-dependent phenomenon rather than an isolated benchmark effect. The manuscript explicitly documents differences across models, architectures, and benchmarks (§ 3, Appendices B and D).

Regarding geometry, we acknowledge that the correspondence between this regime and stalled or partially contracting intrinsic dimension is correlational, as already reported in § 5. The spectral interventions provide further evidence that representation geometry plays a role in semantic discrimination.

Reviewer_yrUKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how lexical overlap affects sentence representations across LLM layers, mainly using a triplet stress test (anchor, paraphrase, lexical distractor) on CounterFact and SugarCrepe++ (SCPP). A triplet succeeds if the anchor-paraphrase pair is closer than both distractor pairs. This lexical sensitivity appears at all depths, including instruction-tuned and embedding models. With mean pooling, autoregressive models show a mid-depth "valley", which the authors link to lower token-identity probe accuracy and a stall in intrinsic dimension (appearing early in OLMo 3 pretraining). Learned pooling and linear/nonlinear readouts improve scores, however the depth curve shape stays the same. For downstream tasks, BERTScore prefers lexical non-summaries in 64% of XSum cases, and AlphaEdit changes lexically similar locality prompts more than dissimilar ones.

Critical strengths
  1. Broad coverage and careful control setup. The study tests 6 main models, 3 extra autoregressive models, 2 encoders, and 17 OLMo checkpoints from random init to RLVR. Also, readout controls are trained on external NLI/STS/paraphrase data with Jaccard-matched pairs rather than the test sets, giving a useful layer-by-layer map for people using hidden states as embeddings.

  2. Practical takeaway for evaluation and model editing. The paper shows that "deeper means more semantic" is not a safe assumption, so lexically similar controls are necessary when evaluating text metrics and knowledge editing. The proposed LS/LD editing split (Jaccard 0.41 vs 0.10) is also a useful benchmark.

Critical weaknesses
  1. The mid-depth valley is not shown to be a lexical effect. (a) With mean pooling, layer 0 (probably the input embeddings) is just a bag of token vectors, so it is the most lexical layer. Yet it is often best or near-best: both Llama 3.2 3B models score highest at layer 0 on CounterFact (Fig. 1a), and DeBERTa drops from ~0.88 to ~0.51 on CounterFact after its first layers (Fig. 11). So mid-depth failures are worse than pure lexical matching, which points to another cause. (b) On ArguAna (App. H.2), the depth curve is almost the same with and without lexical conflict (dCor 0.989 for Gemma, 0.986 for Llama). (c) The 2 benchmarks are confounded in opposite ways. SCPP swap distractors have almost the same bag of words as the anchor, so they mainly test word order, while in the CounterFact example of Table 3 the paraphrase shares more words with the anchor than the distractor does (Jaccard 0.43 vs 0.25), so a bag-of-words baseline already passes. An easy-negative control and a test that varies overlap at fixed meaning are needed.

  2. A known alternative explanation is not ruled out. (a) In many LLMs, massive activations on BOS and delimiter tokens appear in early layers, stay in middle layers and fade at the end (Sun et al., 2024). A few rogue dimensions can dominate raw cosine, and standardization removes this effect (Timkey & van Schijndel, 2021). This could explain much of the mid-depth drop. (b) The spectral intervention in App. I.2 is only a partial check: it adds the PCA mean back, uses α <= 0.75 and clips gains to [0.1, 10], so it never fully whitens the space. If the valley disappears after standardization or after removing BOS/delimiter tokens, the main story of Sec. 3-5 changes.

Questions for authors

Clarifications:

  1. Is 0% depth in Fig. 1 the input embedding layer? Does mean pooling include BOS, chat templates, or punctuation tokens, and what happens if they are excluded?
  2. Does the mid-depth valley still remain after simple per-dimension z-score standardization, or after removing the top massive dimensions?
  3. Does the valley also appear on easy-negative triplets (no lexical overlap)?
  4. In Fig. 1d, the flat curve at ~0.18 seems to be step 0 (random init). On SCPP, it stays above all later checkpoints at almost every depth. Is this correct, and how do you explain it?
  5. In Sec. 6.2, are editing metrics computed after all 2,000 sequential edits? Can you provide single-edit results with confidence intervals?

Suggestion:

  1. Please add a simple Jaccard/BM25 baseline and the 1/3 random chance line to Fig. 1.
3 comments
AuthorsOpenReview ↗

Response to reviewer. (1)

A known alternative explanation is not ruled out. (a) In many LLMs, massive activations on BOS and delimiter tokens appear in early layers, stay in middle layers and fade at the end (Sun et al., 2024). A few rogue dimensions can dominate raw cosine, and standardization removes this effect (Timkey van Schijndel, 2021). This could explain much of the mid-depth drop. (b) The spectral intervention in App. I.2 is only a partial check: it adds the PCA mean back, uses a<=0.75a <= 0.75 and clips gains to [0.1,10][0.1, 10], so it never fully whitens the space. If the valley disappears after standardization or after removing BOS/delimiter tokens, the main story of Sec. 3−53-5 changes.

Is 0% depth in Fig. 1 the input embedding layer? Does mean pooling include BOS, chat templates, or punctuation tokens, and what happens if they are excluded?

Does the mid-depth valley still remain after simple per-dimension z-score standardization, or after removing the top massive dimensions?

The manuscript already examines sensitivity to representation dimensions through learned per-dimension scaling and full-dimensional linear and nonlinear transformations (§ 3, Appendix C, Figures 9 and 10). To directly address the concern about dominant activation dimensions, we additionally evaluate z-score standardization and removal of the highest-RMS dimension. As shown in the table below, performance remains largely similar to the baseline under both interventions, with highly consistent layerwise trajectories. Relative to the baseline, z-score standardization yields dCor values of 0.966 and 0.941 for Gemma PT and IT, while removing the highest-RMS dimension yields 0.960 and 0.987, respectively. We observe similarly high correlations for Llama PT and IT, ranging from 0.941 to 0.999 across these two interventions. Complete layerwise results with additional controls are added in a new Appendix § N.

Table: Success rate under z-score standardization and removal of the highest-RMS dimension.

LayerGemma PT BaseGemma PT Z-scoreGemma PT Top 1Gemma IT BaseGemma IT Z-scoreGemma IT Top 1
183.791.186.281.091.286.4
681.792.190.290.395.091.2
1273.274.077.981.178.781.4
1872.873.773.778.578.078.9
2364.463.464.770.871.171.6
2659.758.060.367.066.667.5
3682.085.682.591.994.291.8
4084.687.384.793.495.394.0
4483.386.383.992.994.493.2
4884.384.485.391.292.291.5

For the concerns regarding BOS and other token positions, the original evaluation uses all tokens, including BOS, for pretrained models and content-only pooling with native chat templates for instruction-tuned models. We additionally evaluate four pooling configurations: content tokens only, content with BOS, all attended tokens, and content without punctuation. The layerwise success profiles remain highly consistent across these configurations, with distance correlations (dCor) ranging from 0.904 to 0.999 relative to the original settings. Specifically, excluding BOS from pretrained model pooling yields dCor values of 0.920 for Gemma PT and 0.924 for Llama PT, while including all attended tokens in instruction-tuned models yields 0.963 for Gemma IT and 0.973 for Llama IT. The results show that the depth-dependent pattern is robust to token selection. Complete layerwise results are added in a new Appendix § O.

Table: Success rate for Gemma-3-12B IT under different token-pooling configurations.

LayerContent only+BOSAll tokensNo punct.
181.084.183.883.1
690.383.683.890.8
1281.276.372.581.3
1878.575.871.378.0
2371.069.461.870.3
2667.064.659.065.6
3691.990.489.491.3
4093.592.892.393.0
4492.992.591.992.6
4891.390.590.491.1
AuthorsOpenReview ↗

Response to reviewer. (2)

The mid-depth valley is not shown to be a lexical effect. (a) With mean pooling, layer 0 (probably the input embeddings) is just a bag of token vectors, so it is the most lexical layer. Yet it is often best or near-best: both Llama 3.2 3B models score highest at layer 0 on CounterFact (Fig. 1a), and DeBERTa drops from ~0.88 to ~0.51 on CounterFact after its first layers (Fig. 11). So mid-depth failures are worse than pure lexical matching, which points to another cause. (b) On ArguAna (App. H.2), the depth curve is almost the same with and without lexical conflict (dCor 0.989 for Gemma, 0.986 for Llama). (c) The 2 benchmarks are confounded in opposite ways. SCPP swap distractors have almost the same bag of words as the anchor, so they mainly test word order, while in the CounterFact example of Table 3 the paraphrase shares more words with the anchor than the distractor does (Jaccard 0.43 vs 0.25), so a bag-of-words baseline already passes. An easy-negative control and a test that varies overlap at fixed meaning are needed.

Does the valley also appear on easy-negative triplets (no lexical overlap)?

We would like to clarify that our characterization of the intermediate regime does not attribute its behavior solely to lexical influence. Rather, we identify this regime through several observations, including weaker performance on lexical influence stress tests (§ 3), stagnation or degradation on several MTEB tasks (§ 4.2), and reduced linear recoverability of token identity (§ 4.1). These findings characterize related properties of intermediate representations without claiming that lexical overlap alone explains the observed depth pattern.

Regarding strong early layer performance, input embeddings are not equivalent to naive bag-of-words. Although token embeddings are context-independent, they are learned through language modeling and encode distributional semantic information. Strong early performance is therefore consistent with our findings. Several MTEB tasks also exhibit relatively strong early performance before intermediate weakening. The ArguAna analysis (Appendix H.2) further shows that lexical overlap affects absolute retrieval performance while the depth trajectories remain largely unchanged.

Regarding benchmark construction, SCPP swap distractors deliberately preserve lexical content while changing semantic relationships, testing whether representations capture compositional meaning beyond shared words. For CounterFact, the example in Table 3 was randomly selected from the original dataset rather than the filtered evaluation set. As described in Appendix A.1, we select the five neighborhood prompts with the highest Jaccard similarity to the anchor. A better representative "What is the twin city of Prague? It is" (anchor), "Prague shares a twinning relationship with" (paraphrase), and "What is the twin city of Palma? It is" (distractor) illustrate cases where lexical similarity strongly favors the distractor.

For the easy negative control, we filter CounterFact for triplets where the distractor has no lexical overlap with either the anchor or paraphrase after excluding named entities. This filtering yields 2,323 triplets. The table below shows substantially higher success rates, particularly at intermediate depths, compared with Figure 1(a). At layer 23, Gemma PT and IT achieve 78.9% and 79.9%, compared with approximately 60% to 65% in the original evaluation. This is consistent with our ArguAna analysis (Appendix H.2), where lexical conflict lowers absolute performance while the depth trajectories remain highly similar. Complete layerwise results are added in a new Appendix § P.

Table: Success Rate under zero overlap of distractor with anchor and paraphrase

LayerOriginal PTOriginal ITEasy negatives PTEasy negatives IT
283.283.897.799.1
681.783.497.697.9
1273.372.697.996.6
1872.971.589.385.1
2364.662.178.979.9
2864.065.785.090.1
3682.189.495.999.0
4084.792.196.999.2
4483.491.998.299.2
4884.690.398.498.1

In Fig. 1d, the flat curve at ~0.18 seems to be step 0 (random init). On SCPP, it stays above all later checkpoints at almost every depth. Is this correct, and how do you explain it?

At initialization, similarity rankings arise from randomly initialized representations rather than any learned linguistic structure. The higher SCPP success rate therefore does not necessarily indicate stronger semantic representations, as the metric reflects relative distances between embeddings. The nearly uniform trajectory across depth is consistent with the absence of learned layerwise specialization.

AuthorsOpenReview ↗

Response to reviewer. (3)

In Sec. 6.2, are editing metrics computed after all 2,000 sequential edits? Can you provide single-edit results with confidence intervals?

The results in § 6.2 evaluate the effects of sequential editing with 2,000 edit requests processed in batches of 100, following the default AlphaEdit settings. To examine the effects of individual edits without cumulative interference, we additionally conduct 500 independent single-edit experiments. In each experiment, a single edit is applied and the resulting changes in model behavior are measured immediately, without carrying the update into subsequent experiments.

Table: Model Editing results for LS: Lexically Similar and LD: Lexically Dissimilar on 500 single edits.*

MetricTop-K-20 LSTop-K-20 LDTop-K-20 Diff. [95% CI]Top-K-50 LSTop-K-50 LDTop-K-50 Diff. [95% CI]Top-K-100 LSTop-K-100 LDTop-K-100 Diff. [95% CI]
JSD0.00880.0055+0.0033 [-0.0003,0.0072]0.00900.0056+0.0034 [-0.0001,0.0073]0.00920.0057+0.0035 [0.00004,0.0074]
TV0.05020.0363+0.0139 [0.0072,0.0211]0.05160.0375+0.0142 [0.0075,0.0214]0.05240.0381+0.0142 [0.0077,0.0211]
Kendall τ0.82830.8573-0.0290 [-0.0403,-0.0182]0.82870.8580-0.0293 [-0.0388,-0.0204]0.83120.8588-0.0276 [-0.0362,-0.0195]
ETP0.00200.0000+0.0020 [0.0000,0.0060]0.00200.0000+0.0020 [0.0000,0.0060]0.00800.00800.0000 [-0.0100,0.0100]

Top-1 Flip Rate: LS = 0.0420 [0.0260, 0.0600], LD = 0.0400 [0.0240, 0.0580], LS$-$LD = +0.0020 [-0.0220, 0.0260].

The single edit results show greater unintended changes for LS prompts than for lexically dissimilar (LD) prompts. TV and Kendall's τ\tau show consistent differences across all Top-K settings, with paired confidence intervals excluding zero. JSD also shows a statistically significant difference at Top-K-100, while Top-K-20 and Top-K-50 exhibit the same directional trend. ETP is higher for LS at Top-K-20 and Top-K-50, and Top-1 Flip Rate is slightly higher on average. Overall, these results indicate that lexically similar prompts experience greater unintended behavioral changes after individual edits, without cumulative interference from sequential editing.

Reviewer_8MYZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how lexical overlap influences LLM representations across model layers and training stages. Through semantic stress tests, lexical and semantic probing, and representation geometry analysis, the authors find that lexical influence persists even after semantic adaptation, and that intermediate layers of autoregressive models show reduced lexical decodability without a corresponding improvement in semantic discrimination. The paper further examines the implications of these findings for summarization evaluation and model editing.

Critical strengths
  1. The paper provides a broad empirical investigation across multiple model families, training stages, and representation extraction methods. The learned pooling and transformation experiments help examine whether the findings depend on specific representation readouts.
  2. The observation that reduced lexical decodability does not necessarily correspond to improved semantic discrimination is interesting. The combined analysis of lexical probing, semantic performance, and representation geometry provides useful insights into how LLM representations evolve across layers.
  3. The paper also explores how these patterns emerge throughout training and their potential implications for downstream applications, including summarization evaluation and model editing.
Critical weaknesses
  1. The overall research narrative could be clearer, particularly the relationships between lexical influence, lexical decodability, semantic fidelity, and representation geometry. A clearer distinction between retaining lexical information and relying on lexical overlap would help connect the experiments. Some figures are also difficult to interpret due to overlapping curves and similar colors.
  2. Autoregressive LLMs are not explicitly trained to organize semantic similarity under cosine distance in their embedding spaces, so lower triplet success rates may reflect limitations of the representation readout rather than weaker semantic understanding. Moreover, triplet failures do not necessarily isolate lexical influence from other difficulties, such as understanding relations or sentence structure. Although the learned readout and MTEB experiments partially address this concern, more controlled comparisons that isolate lexical overlap from semantic difficulty, or direct semantic judgments on the same triplets, would strengthen the conclusions.
  3. The training-stage and geometry analyses provide interesting observations, but the mechanisms behind the intermediate-layer weakness remain unclear. For example, the spectral interventions improve performance across layers without removing the intermediate-layer pattern, leaving its relationship with representation geometry open to further investigation.
  4. The downstream experiments could be more clearly connected to the earlier findings. In summarization, the main text describes lexical overlap with the reference, while the appendix constructs distractors using overlap with the source, which is also used for BERTScore evaluation. Clarifying this setup and independently validating distractor quality would strengthen the experiment. For model editing, evaluating whether the observed distribution changes lead to incorrect factual predictions would further strengthen the practical implications.
Questions for authors

see above.

2 comments
AuthorsOpenReview ↗

Response to reviewer. (1)

The overall research narrative could be clearer, particularly the relationships between lexical influence, lexical decodability, semantic fidelity, and representation geometry. A clearer distinction between retaining lexical information and relying on lexical overlap would help connect the experiments. Some figures are also difficult to interpret due to overlapping curves and similar colors.

Lexical influence (§ 3) measures whether lexical overlap distorts semantic similarity judgments, whereas lexical decodability (§ 4.1) measures how readily exact token identity can be recovered from hidden states. Retaining lexical information therefore does not necessarily mean that representations rely on lexical overlap when comparing inputs. Our results illustrate this distinction in intermediate layers, where token identity becomes less linearly accessible while semantic discrimination weakens. The MTEB evaluations (§ 4.2) further show that semantic fidelity on several tasks weakens or stagnates over this region. We then examine the associated representation geometry (§ 5), finding that intrinsic dimensional growth stalls or partially contracts, while spectral interventions yield modest improvements in semantic discrimination. We will make these relationships more explicit in the Introduction and section transitions, and improve figure readability using more distinguishable colors, line styles, and markers.

Autoregressive LLMs are not explicitly trained to organize semantic similarity under cosine distance in their embedding spaces, so lower triplet success rates may reflect limitations of the representation readout rather than weaker semantic understanding. Moreover, triplet failures do not necessarily isolate lexical influence from other difficulties, such as understanding relations or sentence structure. Although the learned readout and MTEB experiments partially address this concern, more controlled comparisons that isolate lexical overlap from semantic difficulty, or direct semantic judgments on the same triplets, would strengthen the conclusions.

For this concern, the paper evaluates learned pooling, per-dimension scaling, and full linear and nonlinear readouts trained with independent semantic supervision, explicitly optimizing the extracted representations for semantic discrimination while keeping the underlying models frozen. These transformations improve triplet success but do not eliminate lexical influence or the intermediate depth pattern. We also evaluate models explicitly trained for semantic embeddings. Furthermore, we would highlight that evaluating semantic similarity using representations extracted from autoregressive models has been explored in prior work [1], [2], while [3] further motivates examining representation quality across model depth.

To provide a more controlled comparison of lexical overlap, we conduct an additional experiment using CounterFact. For each anchor, we select two distinct paraphrases preserving the anchor entity and two neighborhood distractors referring to the same but different entity. We retain records where both the paraphrase (P) and distractor (D) have High and Low lexical overlap variants, guaranteeing that P High has greater overlap than D Low and D High has greater overlap than P Low. These filtering criteria yield 1,929 eligible instances. This design preserves the semantic relationships within each pair while varying their lexical similarity to the anchor. The results for Gemma instruct are provided in the table below. Complete results across Gemma and Llama models are provided in a new Appendix § M.

Table: Success Rate under independently controlled paraphrase (P) and distractor (D) lexical overlap.

Layer(P High, D Low)(P Low, D Low)(P High, D High)(P Low, D High)
1100.098.499.289.8
499.797.098.785.5
899.898.098.887.8
1299.695.895.675.4
1698.189.689.166.7
2096.985.686.266.0
2395.283.482.761.3
2696.385.884.363.9
3099.395.693.982.3
3699.899.098.893.8
4099.899.499.596.9
4499.899.099.396.5
4899.698.198.494.2

[1] Scaling Sentence Embeddings with Large Language Models

[2] Emergence of a High-Dimensional Abstraction Phase in Language Transformers

[3] Layer by Layer: Uncovering Hidden Representations in Language Models

AuthorsOpenReview ↗

Response to reviewer. (2)

The training-stage and geometry analyses provide interesting observations, but the mechanisms behind the intermediate-layer weakness remain unclear. For example, the spectral interventions improve performance across layers without removing the intermediate-layer pattern, leaving its relationship with representation geometry open to further investigation.

The intermediate regime involves several aspects of model representations, which we examine through training stages, representation geometry, and Transformer component analyses. Each analysis addresses a different aspect of the observed behavior, from its emergence during training to its geometric characteristics and the contributions of individual Transformer components. The spectral interventions test whether changes to the variance spectrum affect semantic discrimination. The modest improvements suggest that representation geometry plays some role, although the intermediate depth pattern remains.

The downstream experiments could be more clearly connected to the earlier findings. In summarization, the main text describes lexical overlap with the reference, while the appendix constructs distractors using overlap with the source, which is also used for BERTScore evaluation. Clarifying this setup and independently validating distractor quality would strengthen the experiment. For model editing, evaluating whether the observed distribution changes lead to incorrect factual predictions would further strengthen the practical implications.

We apologize for the confusion. The term “reference” in §6.1 was intended to refer to the source document, not the gold summary. The distractors are generated to have high lexical overlap with the source document while differing in semantic content.

For model editing, we conduct an additional evaluation to see whether distribution changes also affect factual predictions. For this, we rely on the Qwen 32B model as a judge. After 2,000 sequential edits, 67.42% of lexically similar (LS) responses and 64.71% of lexically dissimilar (LD) responses are judged incorrect. This overall performance degradation after repeated editing is consistent with prior work [4]. To examine the effect without cumulative editing, we also conduct 500 independent single edits, where the incorrect answer rates are 4.35% for LS and 3.81% for LD. Both settings show higher error rates for LS, consistent with the larger distributional and ranking changes reported in our experiments.

[4] Model Editing at Scale Leads to Gradual and Catastrophic Forgetting

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.