acceptodds
Under review as a conference paper at ICLR 2027

Beyond Accuracy: Stable Predictions, Unstable Topology in Transformer Representations

Abstract

Predictive accuracy remains the dominant metric for evaluating robustness in modern natural language processing (NLP) systems, often suggesting strong resilience even under adversarial or corrupted inputs. We argue that this view can obscure failure modes that emerge in the induced local topological structure of learned representations. Focusing on BERT-based classifiers, we study the stability of local structure induced by k-nearest neighbors (kNN) under semantically-preserving transformations, including adversarial obfuscations. Despite near-perfect classification performance across all conditions, we find that kNN neighborhood structure shows substantial changes, where local relationships between embeddings are rewired even when predictions and semantic content remain unchanged. We further introduce a normalization-based sensitivity measure that relates topological change to embedding perturbation magnitude and show that neighborhood topology changes disproportionately relative to embedding displacement across all conditions, with cosine displacement remaining near-zero while Rank Stability increases monotonically with perturbation strength. This effect is consistent across neighborhood scales and occurs without loss of label consistency, indicating that predictive decisions remain stable while representation structure varies. We quantify these effects using topology-based stability metrics and observe large, statistically significant differences between clean and corrupted inputs. Overall, we find a systematic mismatch between predictive performance and the stability of the induced neighborhood topology, suggesting that accuracy alone is insufficient to characterize robustness and that topology-based measures provide a complementary diagnostic for hidden vulnerabilities.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
representation stability, neighborhood topology, adversarial robustness, interpretability, transformer representations, k-nearest neighbors

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ciWjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether high classification accuracy can coexist with instability in the local neighborhood structure of BERT representations under synthetic input perturbations. It introduces several kNN-based metrics to characterize representation-level instability. While the general research question is potentially interesting, I have serious concerns about the validity of the experimental design, and whether the evidence supports the central claims. Several issues are fundamental rather than matters of presentation or minor experimental omissions. The work reads more like a preliminary empirical observation than a sufficiently developed methodological or scientific contribution for ICLR.

Critical strengths

As a preliminary empirical investigation or pilot study, this work has some merit.

Critical weaknesses
  1. The assumption is very strong. Specifically, the central observation that stable model predictions do not necessarily imply stable internal representations is neither theoretically surprising nor, by itself, a sufficiently substantial contribution. Classification decisions depend on whether representations remain on the same side of a decision boundary, whereas nearest-neighbor relationships depend on their relative positions in the embedding space. We can't assume that they are on the same page, actually we can't expect that one does not imply the other.

  2. The experiments rely on synthetically generated, template-based data and several handcrafted perturbation categories. Such a controlled setting can be useful for preliminary analysis, but it is insufficient for establishing general conclusions about transformer representation robustness. The reported trends may primarily reflect properties of the synthetic data construction rather than a general phenomenon in transformer representations.

  3. This paper lacks the basic baseline, which should be carefully designed. The authors can easily attribute substantial neighborhood changes to intrinsic sensitivity of learned representations. However, it does not compare the results against basic reference conditions that would help support this conclusion.

  4. I don't really understand what new insight is proposed in this manuscript. There are many benchamrks are related to this field, I can't find something new. The authors should clearly articulate the key novelty, scientific insights, and contributions of this work relative to the state of the art.

  5. My greatest concern is the substantial gap between the experimental evidence and the conclusions. The experimental results are largely limited to descriptive observations of kNN neighborhood changes under synthetic perturbations. These observations alone do not establish that the measured changes reflect meaningful robustness failures.

  6. The manuscript suffers from substantial formatting inconsistencies, and apparent template-related issues. The overall presentation gives the impression that the submission has not undergone adequate preparation or quality control.

Questions for authors

Pls check the weakness

Reviewer_L5c4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether robust predictive accuracy under semantically preserving transformations implies stability in the local topological structure of a model's representation space. To test this, the authors fine-tune BERT-family classifiers on a 40,000-sample synthetic dataset extracting final-layer [CLS] embeddings for evaluation. They quantify structural changes between original and normalized inputs using four proposed metrics: Rank Stability (RS) to measure kNN set overlap, Normalized RS to scale overlap by cosine displacement, Neighborhood Label Conflict (NLC), and a composite Latent Robustness Gap (LRG). The experiments reveal that despite near-perfect predictive accuracy (~99.4%), near-zero NLC, and minimal cosine displacement, RS increases substantially from 0.05 on clean inputs to 0.78 on obfuscated ones. Supported by statistical testing and multi-scale kNN analysis, the authors conclude that local representation topology is disproportionately disrupted by perturbations, arguing that topology-based measures expose hidden vulnerabilities missed by standard accuracy evaluations.

Critical strengths
  1. Probing whether prediction invariance guarantees geometric stability in latent space is an interesting research direction. A lightweight, model-agnostic metric capturing local neighbor perturbations could serve as a valuable diagnostic tool if rigorously validated.
  2. Rank Stability (RS) is simple to define and implement, and the authors explicitly note that it is an empirical neighbor-overlap proxy rather than a formal topological invariant.
  3. Restricting metric computation to correctly classified samples ensures that misclassifications do not directly confound the neighbor overlap calculations. The authors explore multiple neighborhood scales and replicate the core metrics across three BERT-family architectures.
  4. The authors explicitly note the synthetic setting and the confinement of their experiments to the BERT family in the limitations section.
Critical weaknesses
  1. Fundamental Experimental Design Confound: According to Appendix A and Appendix B, the classifier is explicitly trained on a 4-class sequence classification task where the four classes are: clean, prefix, suffix, and obfuscated. Consequently, the model is trained as an artifact detector, not on an underlying semantic task.
  2. Unsupported "Disproportionate to Displacement" Claim: The authors repeatedly state that cosine displacement remains "near-zero" while RS increases. However, displacement increases across conditions spanning six orders of magnitude.
  3. Tokenizer Interactions and Perturbation Realism: Standard bert-base-uncased preprocessing strips zero-width Unicode characters during tokenization. If stripped before subword splitting, these perturbations are exact no-ops for the model, explaining why certain perturbed inputs exhibit near-zero displacement while others change radically. There are no token-level ablations or analysis of edit distance versus RS to isolate which surface corruptions drive embedding movement.
  4. Lack of Downstream Impact or Practical Utility: The authors describe the topological instability as a "hidden vulnerability," but show no downstream consequences. If NLC is essentially zero across all conditions, a downstream kNN classifier or retriever operating on these embeddings would perform identically. Without showing that RS correlates with practical failure, the label "vulnerability" is unwarranted.
  5. Missing Comparative Baselines and Literature Context: The paper mentions linear CKA (0.972) in passing, but does not benchmark against established local geometric metrics such as Representation Topology Divergence (RTD), local CKA, or Procrustes distance. The distinction from Mashaido & Das (2026), which examines latent embedding collapse under obfuscation attacks on similar models, is not clearly articulated.
Questions for authors
  1. Task Formulation: What are the ground-truth target classes in your fine-tuning setup? If they are indeed clean, prefix, suffix, and obfuscated, what does "prediction invariance" mean given that the normalization removes the defining features of the corruption class? Could you report the per-sample prediction agreement?
  2. Clean Condition Noise Floor: Why is RS equal to 0.049 in the clean condition where the normalization is a no-op? How are identical-distance neighbor ties broken, is the query point excluded from its own neighborhood, and is exact or approximate kNN search used?
  3. Matched-Noise Control Baseline: What happens to RS when you add random isotropic Gaussian noise of matched magnitude to clean embeddings? Does RS increase by a similar amount?
  4. Downstream Impact: In what concrete setting does an increase in RS lead to downstream performance degradation if NLC remains near zero?
Reviewer_YwotOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether predictive accuracy captures the stability of local neighborhoods in transformer representations. Using a synthetic dataset with four corruption categories and three BERT-family classifiers, it compares cosine-based nearest-neighbor sets before and after text normalization. The authors introduce Rank Stability, a displacement-normalized variant, and a composite Latent Robustness Gap. They report substantial neighborhood turnover despite high classification accuracy, low neighborhood label conflict, and high global representation similarity. Experiments across several neighborhood sizes and model capacities motivate local structural diagnostics as complementary robustness measures. However, the task definition and missing controls limit the interpretation.

Critical strengths
  1. The distinction between predictive performance and local representation stability is worth investigating, and the neighborhood overlap formulation provides a simple, interpretable diagnostic for this purpose.
  2. The evaluation includes three BERT-family models and four neighborhood sizes, providing useful initial evidence that the reported neighborhood turnover persists across these specific experimental choices.
Critical weaknesses
  1. Appendices A–B define prediction targets as corruption types, rather than semantic labels. High accuracy therefore demonstrates corruption recognition, not prediction invariance under semantic-preserving transformations; similarly, low NLC indicates corruption-type clustering rather than preserved semantic organization.
  2. Dividing RS by cosine displacement does not establish instability independent of displacement. Discrete neighbor sets can change under small movements near distance ties; without matched-noise controls and neighbor-margin analysis, the claimed intrinsic vulnerability remains unsupported.
  3. Table 3 reports clean RS = 0.049 although normalization supposedly leaves clean inputs unchanged. Under deterministic embeddings and an unchanged reference set, RS should be zero; this unexplained baseline raises concerns about numerical or implementation artifacts.
Questions for authors

See the weaknesses part.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment (to all reviewers)

We uploaded a revised PDF. The revision answers the reviewers’ requested fix: same question, same metric, with the label confound removed by using semantic labels (AG News, four topic classes; BERT-base with three seeds, DistilBERT; Sec. 4.1). The original synthetic study is in App. A.

Other changes: RS is rebuilt on one fixed clean pool, so clean RS is 0 (was 0.045–0.048; Sec. 3.2). We added random-direction noise of the same cosine displacement, neighbor margins, CKA, Procrustes, clean-twin retrieval and bootstrap intervals (Secs. 3.3, 4.2–4.6).

Headline: perturbations that keep accuracy at 97–99% replace 26–53% of an input’s ten nearest neighbors, 3.9–4.9× the random-direction noise value, while CKA stays at 0.97–0.99. Replies to each review follow.

AuthorsOpenReview ↗

Note for reviewers in light of upcoming deadline

A brief pointer for reviewers: the revised PDF answers each review (Sec. 4.2 for the matched-noise comparison, Sec. 4.6 for neighbor margins, Sec. 4.3 for CKA and Procrustes). We are glad to answer any follow-up questions before the deadline tomorrow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.