acceptodds
Under review as a conference paper at ICLR 2027

Multi-scale intrinsic dimension estimators for text embeddings

Abstract

Persistent homology dimension (PHDim) characterizes the geometry of text embeddings, but its single value combines structures at different scales. We introduce scale-resolved PHDim based on quantile-trimmed minimum spanning tree energies and prove consistent dimension recovery for every fixed quantile interval under regular sampling assumptions. In contextual token embeddings, we identify an approximate cell–skeleton geometry: repeated token types form local cells connected by a type-level skeleton. Building on this structure, we propose a three-band estimator with a fixed sampling budget for stable, informative comparisons between texts. Controlled interventions and synthetic experiments link its responses to contextual variability, lexical diversity, and the organization of longer-range geometry. Scale separation reveals a more consistent signal for AI-text detection: classical PHDim can be either higher or lower for generated texts, whereas removing the lexically dominated lower band yields a consistently nonnegative AI–human difference across the tested generators. Beyond AI-text detection, scale separation provides a more informative characterization of text complexity and defects. Different defects produce distinct three-band profiles: recognition errors predominantly raise fine and middle dimensions, repetitive boilerplate lowers middle and coarse dimensions, and disrupted semantic coherence yields a different pattern from phrase repetition. Together with scale-dependent differences in writing proficiency, these profiles enable a more granular analysis of text collections than a single dimension estimate.

Primary area:
interpretability and explainable AI
Keywords:
Intrinsic dimension, Topological Data Analysis, LLM interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Pbj1OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes scale-resolved PHDim, an intrinsic dimension estimator based on quantile-trimmed minimum spanning tree (MST) energies. The authors introduce a token cell–skeleton geometric framework to interpret the multi-scale structure of contextual token embeddings and investigate the relationships between intrinsic dimensions at different scales and textual properties such as contextual variability, lexical composition, and text organization. The proposed method is further applied to controlled text transformations, AI-generated text detection, language proficiency analysis, and text error analysis, demonstrating its potential value across several text analysis tasks.

Critical strengths

(1) A novel and mathematically motivated method. The paper extends traditional single-valued PHDim into a scale-resolved intrinsic dimension estimator by selecting different quantile intervals of MST edge lengths. The method has natural connections to persistent homology, graph theory, and asymptotic statistics. Theorem 1 in Section 3.3 provides an asymptotic dimension recovery guarantee for regular distributions, offering a meaningful theoretical foundation for this new perspective on text representation geometry.

(2) An intuitive geometric interpretation and broad experimental exploration. Beyond proposing a new statistic, the authors identify how traditional PHDim may mix statistical signals from different geometric scales and demonstrate the potential of scale separation to distinguish these signals. The cell–skeleton framework, controlled interventions, and experiments across multiple practical tasks provide useful insights. Overall, the paper presents a reasonably comprehensive framework combining geometric intuition, theoretical analysis, and empirical investigation.

Overall, I believe the paper makes a meaningful methodological contribution through its mathematically motivated scale-resolved estimator and broad empirical exploration. While the concerns discussed below limit the strength and generalizability of some interpretations, I consider the paper's novelty and existing contributions sufficient to outweigh these limitations.

Critical weaknesses

(1) A remaining theoretical gap between asymptotic guarantees and the central finite-sample observations. Theorem 1 in Section 3.3 shows that, for sampling distributions satisfying its regularity assumptions, different fixed quantile intervals of positive width recover the same asymptotic intrinsic dimension. However, the main geometric interpretations and experimental conclusions rely on differences in effective dimensions across quantile intervals at finite sample sizes. Although Section 4.2 and Appendix E discuss sampling dependence and cell–skeleton energy mixtures, these analyses do not yet fully establish the connection between finite-sample multi-scale behavior and the asymptotic theory, particularly regarding the stability and geometric meaning of the resulting dimension profiles.

(2) Some linguistic interpretations lack precise definitions and sufficiently independent validation. The paper associates dimensions at different scales with contextual variability, lexical richness, and broader text organization. However, some of these concepts are not clearly operationalized. For example, Section 5.2 suggests that Coarse PHDim may capture diversity organized within coherent text, but the current experiments do not sufficiently distinguish the respective contributions of semantic diversity and discourse coherence.

Furthermore, Appendices V and Y use LLM-generated annotations of textual defects. Although the authors verify whether the quoted text spans actually appear in the original documents, this check does not establish whether the assigned defect categories and severity scores are correct. Without independent human annotation checks or human–LLM agreement analysis, the evidence provided by these annotations for the proposed linguistic interpretations remains limited.

(3) The robustness and representativeness of some experimental results require further investigation. Section 5 applies 26 interventions to 50 source texts and mainly reports median paired changes in the scale-resolved PHDim estimates of their token embeddings for each intervention. The main correlation and regression analyses in Appendix M are then based on aggregated changes from 24 of these interventions. These statistics can reveal meaningful overall trends, but they do not fully characterize the consistency and variability of changes in scale-resolved PHDim across different texts under the same intervention. If substantial cross-text heterogeneity exists, it may limit the strength of interpreting these aggregate trends as general geometric mechanisms.

In addition, the AI-generated text detection experiments in Section 6 compute PHDim using contextual embeddings extracted from only the first 128 tokens of each continuation (Appendix Q.2). This relatively short window, taken from a fixed position, may not adequately represent the embedding geometry of the full generated text.

Questions for authors

(1) Theoretical connection: Could the authors further discuss or propose a preliminary theoretical framework connecting the dependence of finite-sample dimension estimates on quantile interval selection with the interval-independent asymptotic dimension in Theorem 1? For example, would it be possible to establish additional theoretical results under certain structured distribution assumptions, or provide stability guarantees for finite-sample dimension profiles?

(2) Linguistic interpretation: Could the authors use existing computational linguistics metrics, such as sentence-embedding-based semantic diversity measures and discourse coherence measures based on Entity Grid or adjacent-sentence semantic similarity, to quantify diversity and coherence separately? Investigating their relationships with Coarse PHDim and its changes under text interventions could provide more direct evidence for the interpretation proposed in Section 5.2.

Additionally, for the experiments involving automatic LLM annotations, could the authors provide a complete case study showing the original text, the corresponding LLM annotations, and the multi-scale intrinsic dimension estimates, to help readers understand their relationship?

(3) Experimental robustness: For the text interventions in Section 5, could the authors report additional statistics, such as the proportion of source texts exhibiting the same direction of change in their embedding-based scale-resolved PHDim estimates under each intervention, as well as the interquartile range of these changes, to better assess cross-text robustness? For Section 6, could the authors compare results across different token window lengths and positions (beginning, middle, and end) on a representative subset of texts to assess the sensitivity of the main findings to the sampling protocol?

Reviewer_tUakOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce scale-resolved PHDim based on quantile-trimmed minimum spanning tree energies. Building on cell-skeleton geometry, the authors proposed a three-band estimator with a stable, informative comparisons between texts. The experiments examine the AI-generated text, writing proficency, and natural occuring errors.

Critical strengths
  1. The authors finds that conventional PHDim strongly follows lexical diversity, which helps explain why AI texts sometimes have higher dimension than human texts and sometimes lower.

  2. The paper shows why a single dimension estimate can be vague. The repetition, vocabulary variation, and contextual organization can affect it differently. And the scale separation provides an informative characterization of text complexity and defects.

  3. The 26 transformation to 50 English source text (thirteen mechanical operations and thirteen LLM-generated rewrites) helps connect geometric changes to specific text modifications.

Critical weaknesses
  1. Finding the average difference between AI and human texts does not show that the methods can reliably classify each individual text. The scores may overlap, resulting some human texts to be labelled as AI and some AI texts to be labelled as human.

  2. The paper's experiment is focused on English text, so its unclear whether the same geometric patterns appear in other language.

  3. The results depend on the selected sample sizes and edge bands, so a sensitivity analysis is needed to show that the main findings remain stable under other choices.

Questions for authors
  1. Provide confidence intervals for the main correlation and for the AI detection results (figure 4).

  2. Adding a key takeaway to each figure would help readers interpret the results more easily.

  3. Most detailed experiments use ModernBERT and the results across other models (BERT, RoBERTa, DeBERTa, NeoBERT) can provide a broader evidence.

Reviewer_T3j9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This article presents a new study on geometrical properties of embedding methods, including those derived from large language models. Part of the results is about the estimation of a quantity referred to as intrinsic dimension (PHDim) of a collection of point clouds. The second part of the results is about the geometric explanation of the embeddings generation process. The last part is regarding the interpretation of text generation, including errors and proficiency. The last part of the article contains some numerical experiments to illustrate the main claims.

Critical strengths

The topic of geometrical description of text generation, from the embeddings standpoint can potentially be a source of understanding of LLMs.

Critical weaknesses

This article is very difficult to read; and it is unclear what scientific problem is to be solved. The assumptions under which the results apply (model generation, embedding techniques) are not clearly stated either. There should be a clear repositioning of the article starting from the natural starting point: what is the type of investigation of problem solved here? How does it compare with the existing literature. This should be clearly developed in the introduction and first sections, which are poorly written for now. The application to text embeddings and AI text detection can be interesting (watermarking and/or detection), but again the article does not set the right frame and problem studied in the first place to be publishable at this point.

Questions for authors

I do not have any question at this point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.