acceptodds
Under review as a conference paper at ICLR 2027

Invariant Atoms: Sparse Coordinates of Local Semantic Geometry in Language Model Representations

Abstract

Large language models often preserve meaning despite substantial changes in wording, style, and syntax, while small semantic edits can systematically alter their hidden representations. This suggests that semantic variation may be organized along recurring local directions. We propose the Invariant Atom Hypothesis: local semantic motion admits preferred sparse coordinates along directions that remain stable under meaning-preserving transformations. We learn a shared semantic frame and sparse coordinates that reconstruct semantic displacements while suppressing nuisance variation, with anchor-dependent diagonal modulation adjusting atom strengths without sample-specific rotations. Empirically, the atoms exhibit strong semantic–nuisance separation, sparse reconstruction, reproducible directions, and causal effects on model predictions. The learned geometry generalizes to unseen semantic neighborhoods and nuisance families, while local reweighting improves semantic selectivity and preserves a consistent global-to-local structure. Atom signatures also remain stable under model modification. These findings support reusable invariant directions as a sparse coordinate system for local semantic geometry in language models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation learning, language models, invariant representations, representation geometry

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hv3wOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes invariant atoms, a framework for capturing semantic variation in LLMs that remains stable under meaning-preserving transformations.

Critical strengths
  • The concept of meaning-preserving transformations is novel and interesting to the best of my knowledge.
  • The schematic figures are instructive.
  • The experiments are well done and extensive (across seven model families).
  • The dataset construction is overall well documented.
Critical weaknesses
  • The invariant semantic subspaces proposed by Dasgupta et al. (2026) seem closely related. The paper is cited, but a more detailed comparison, possibly including experiments, would be good.
  • Excessive white space removal makes the paper very dense and hard to read. The paper’s clarity would benefit from streamlined writing.
  • The manual quality checks of the dataset could be described in more detail, e.g., the sample size and error rate.
Questions for authors

When do finite SP/SC displacements approximate the proposed nuisance and semantic tangent components well (Sec. 3.1)? Some more details would be good.

3 comments

We thank the reviewer for the encouraging assessment and constructive suggestions. We appreciate the opportunity for constructive scientific discussion. We have expanded the comparison with invariant semantic subspaces, clarified the geometric interpretation of finite SP/SC displacements, and provided additional details on dataset verification.

Question:

1. When do finite SP/SC displacements approximate tangent directions?

We agree that this distinction warrants clarification. Our formulation concerns local tangent geometry, whereas the experiments observe finite differences between discrete text inputs.

We have added a conditional first-order analysis. Suppose a smooth representation trajectory γ:[0,ϵ]→M\gamma:[0,\epsilon]\rightarrow\mathcal{M} connects two representations and satisfies ∥γ¨(t)∥2≤M\|\ddot{\gamma}(t)\|_2\leq M. Taylor's theorem gives

\qquad \|r_\epsilon\|_2\leq \frac{M\epsilon^2}{2}.$$ Consequently, $$\left\|\frac{\Delta z}{\epsilon}-\dot{\gamma}(0)\right\|_2 \leq \frac{M\epsilon}{2}.$$ Thus, finite displacements provide accurate first-order tangent approximations when the connecting representation trajectory is sufficiently local and its curvature-induced error is small relative to its first-order motion. Interpreting these directions as nuisance or semantic tangent components additionally requires that the corresponding trajectories follow the appropriate semantic equivalence or semantic-changing variations. Importantly, **discrete sentence pairs do not uniquely define smooth trajectories**, and an SC edit may simultaneously change semantic content and surface form. We therefore treat finite SP/SC displacements as *empirical, transformation-labeled probes* of local geometry, rather than exact tangent vectors. The mathematical analysis identifies sufficient approximation conditions, not assumptions verified for every sentence pair. **Manuscript Change:** Section 3.1–3.2; Appendix J, *When do finite displacements approximate tangent directions?* ### Weakness: **1. Relationship to invariant semantic subspaces (Dasgupta et al., 2026)** We agree that invariant subspaces and invariant atoms are closely related. However, they characterize **different levels of semantic geometry**. An invariant subspace identifies *where* semantic variation occurs, but does not specify preferred directions within that subspace: orthogonal rotations preserve the subspace. Our hypothesis goes further by proposing that semantic motion admits **preferred sparse coordinates**, allowing individual changes to be represented efficiently through recurring directions. To distinguish these properties empirically, we added the generalized-eigenvector invariant-subspace baseline (InvSub) using the same SP/SC supervision as our method. | Method | S/T ↑ | SC reconstruction cosine ↑ | |---|---:|---:| | InvSub ($K=256$) | 8.03 | 0.136 | | cPCA ($\lambda=1$) | 2.61 | 0.521 | | Global atoms | 1.88 | 0.670 | | Local atoms | 1.98 | 0.679 | On LLaMA, InvSub achieves stronger semantic selectivity but substantially lower reconstruction fidelity. Our method instead balances selectivity with sparse preservation of semantic displacement. More directly, we tested **whether the learned atom directions matter beyond their span**. Randomly rotating the learned frame preserves its subspace but reduces top-32 reconstruction cosine from 0.675 to 0.421. The rotated frame requires approximately 128 coordinates to match the original top-32 fidelity. This provides direct evidence for preferred sparse coordinates beyond subspace membership, without implying a uniquely identifiable basis. **Manuscript Change:** Section 3.1, comparison with invariant semantic subspaces; Section 4.2, *Contrastive subspaces do not establish preferred sparse coordinates*; Appendix K, *Contrastive Subspaces and Preferred Sparse Coordinates*, Tables 34–35.

2. Dataset quality verification

We have clarified the extent of manual inspection. All 62,693 selected SP/SC variants, across 1,400 anchor-centered neighborhoods, underwent manual quality inspection—not merely a random subset. SP variants were checked for preservation of the anchor's meaning, while SC variants were checked for the intended semantic change without substantial unrelated modifications.

The retained variants were judged to satisfy these criteria during inspection. However, we do not equate manual inspection with guaranteed annotation correctness: residual errors cannot be ruled out, and no separately measured annotation error rate or independent inter-annotator agreement is available. We have made this distinction explicit.

Manuscript Change: Appendix A, Manual quality verification.

3. Presentation density and readability

We appreciate the reviewer's observation. Our intention in using compact spacing was to provide sufficient detail to clearly explain and evaluate the proposed Invariant Atom Hypothesis. However, the reviewer's feedback helped us recognize that the aggressive spacing may have placed an unnecessary burden on readers. We have therefore used the additional pages permitted during the discussion stage to improve the layout and readability. We have also carefully reviewed the manuscript sentence by sentence to improve clarity, precision, and conciseness.

We hope these revisions make the paper easier to follow while preserving the technical details needed to evaluate our contributions. Should any questions or concerns remain, we would be grateful for the opportunity to clarify them promptly during the discussion period.

Thanks to the authors for the detailed response, which addressed my questions. While still very dense, the revision has improved the readability of the paper. I maintain my positive score.

Reviewer_6Ey2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Invariant Atom Hypothesis, positing that local semantic variations in large language model (LLM) representations are organized along a preferred, near-orthogonal sparse coordinate frame that reconstructs semantic-changing (SC) displacements while suppressing meaning-preserving (SP) nuisance variations. To learn these coordinates, the authors introduce a two-stage global-to-local optimization framework that first learns a globally shared semantic frame (A0A_0) with a Top-kk sparse encoder and Stiefel orthogonality regularization, and subsequently adapts atom magnitudes locally via anchor-conditioned diagonal modulation (A(x)=A0diag⁡(1+r(x))A(x) = A_0 \operatorname{diag}(1+r(x))). The Across evaluate the learned frames using seven open-weight LLM families and a custom synthetic dataset of 1,400 anchor-centered SP/SC neighborhoods.

Critical strengths

1.The synthesis for SAE non-identifiability and nuisance sensitivity is interesting: The paper addresses an important challenge in mechanistic interpretability by combining top-kk sparse dictionary learning, local coordinate coding, and transformation-defined invariance to separate semantic displacements from surface-form paraphrasing.

2.Global-to-Local parameterization: Restricting local adaptation to anchor-conditioned diagonal scaling preserves a globally shared directional coordinate system while reweighting atom relevance per neighborhood. This avoids the severe reconstruction collapse demonstrated by unconstrained or QR-reprojected low-rank directional adaptations.

3.Comprehensive empirical evaluation: The empirical suite spans seven diverse LLM families and systematically probes multiple facets of the learned representations, demonstrating consistent gains in semantic selectivity over uncontrasted baselines, nontrivial cross-seed atom alignment from a shared initialization, and monotonic dose-response effects in bidirectional causal interventions.

Critical weaknesses

1.Omission of important baselines: All evaluated baselines (PCA, Random, and Recon-Only) are completely blind to the semantic-preserving (SP) nuisance displacements during training, which confounds the benefit of sparse atom coordinates with the benefit of contrastive SP/SC supervision. The authors should consider comparing against contrastive linear subspace and nuisance-erasure methods that utilize the exact same SP/SC pairs, as well as some established causal steering baselines.

2.Evaluation-Set Leakage: The experimental protocol selects model checkpoints every 10 epochs directly on the held-out odd-indexed evaluation split without a separate validation split, and estimates per-dimension standardization statistics (μj,σj\mu_j, \sigma_j) transductively across the entire 64,093-sentence corpus including held-out test neighborhoods.

3.Reliance on Synthetic Perturbations and Generator-Distribution Sensitivity: All training and evaluation sets rely exclusively on synthetic single-sentence LLM perturbations thus exhibiting strong directional asymmetry in cross-generator transfer (Qwen →\to Mistral vs. Mistral →\to Qwen) and could be strengthened by evaluating on human-validated adversarial paraphrase benchmarks with high lexical overlap such as PAWS.

Questions for authors

I would like the authors to resolve the above highlighted weaknesses: do more rigorous baselining, clarify the leakage of evaluation set and evaluate on human-validated adversarial paraphrase benchmarks.

4 comments

We thank the reviewer for identifying three important issues concerning baseline fairness, evaluation validity, and generalization beyond synthetic perturbations. We appreciate the opportunity for constructive scientific discussion. We conducted additional experiments addressing each concern and revised the manuscript accordingly.

R1. SP/SC-Supervised Contrastive Baselines

We agree that PCA and Recon-Only do not control for the benefit of SP/SC supervision. We therefore added contrastive PCA (cPCA) and the generalized-eigenvector invariant-subspace baseline (InvSub), both trained using the same SP/SC displacements as invariant atoms.

The comparison on LLaMA (layer 16, K=256K=256, top-32 reconstruction) shows:

MethodS/T ↑SC reconstruction cosine ↑
PCA0.830.654
cPCA (λ=0.5\lambda=0.5)1.580.608
cPCA (λ=1\lambda=1)2.610.521
InvSub8.030.136
Global (ours)1.880.670
Local (ours)1.980.679

The contrastive baselines can achieve higher S/T, but with substantially reduced reconstruction fidelity. Our method offers a more favorable balance between semantic selectivity and preservation of semantic displacement, rather than maximizing either metric independently.

More importantly, we directly tested whether the learned atom directions provide preferred sparse coordinates beyond a subspace. Orthogonally rotating the learned atom frame preserves its span but reduces top-32 reconstruction cosine from 0.675 to 0.421. The rotated frame requires approximately 128 coordinates to match the original top-32 fidelity. This provides evidence for a non-arbitrary sparse directional organization, rather than merely the benefit of contrastive supervision.

We also added causal comparisons against PCA, Recon-Only, cPCA, and InvSub under natural, norm-matched, and approximately reconstruction-matched interventions. Invariant atoms exhibit higher causal semantic selectivity than PCA in all 18 comparisons and Recon-Only in 17 of 18 comparisons. These comparisons extend the original causal controls, although we do not claim to have exhaustively evaluated all established steering methods.

Manuscript Change: Section 4.2, Contrastive subspaces do not establish preferred sparse coordinates; Appendix K, Tables 34–35; and Appendix M, Controlled Causal Selectivity, Table 38.

R2. Evaluation-Set Leakage

We acknowledge both issues identified by the reviewer: the original odd-indexed evaluation split was reused for checkpoint selection, and feature standardization included held-out representations. We have explicitly clarified the original procedure and conducted two leakage-free evaluations.

First, we retain the original within-neighborhood setting but separate validation and test examples, estimate normalization statistics from training data only, and compare the original and corrected protocols on identical test examples.

MethodOriginal S/TLeakage-free S/T
Global2.022.00
Local2.112.14
Recon-Only0.991.00
PCA0.930.93

The correction produces negligible changes in semantic selectivity or reconstruction fidelity.

Second, we evaluate a stricter group-disjoint 70/15/15 train/validation/test split (979/210/211 anchor neighborhoods), fitting normalization and PCA initialization using training groups only and selecting checkpoints exclusively on validation groups.

MethodGroup-disjoint S/T ↑Reconstruction cosine ↑
Global1.720.653
Local1.750.664
Recon-Only0.830.679
PCA0.810.651

Although selectivity decreases under the more difficult group-disjoint protocol, both atom models retain a clear advantage over PCA and Recon-Only. These results indicate that the original evaluation choices do not explain the principal observed advantage, while confirming that performance is sensitive to neighborhood-level generalization difficulty.

Manuscript: Sections 4.1 and 4.4, Leakage-free validation; Appendix L, Clean within-neighborhood evaluation and Clean group-disjoint evaluation, Tables 36–37.

R3. Synthetic Perturbations, PAWS, and Cross-Generator Asymmetry

We agree that the original controlled perturbation dataset cannot establish distribution-independent semantic invariance. To examine this limitation, we evaluated PAWS-Wiki, a human-labeled benchmark containing high-overlap paraphrase and non-paraphrase pairs, under three settings: zero-shot transfer, PAWS-specific training/adaptation, and validation-selected representation layers.

On the official disjoint PAWS test groups (1,408 anchors), the results are:

MethodMedian S/T ↑Reconstruction cosine ↑
Raw representation1.289—
Global (zero-shot)1.3140.217
Global (PAWS-trained)1.3000.622
Local (PAWS-trained)1.2840.629
PCA (PAWS-trained)1.2540.638

The result is informative but qualified. PAWS-specific learning substantially improves semantic-displacement reconstruction, but provides only modest selectivity beyond the raw representation. The zero-shot evaluation also shows limited coverage of PAWS displacement energy. Furthermore, selecting among layers 8, 16, 24, and 32 using validation data does not materially resolve this limitation.

PAWS presents a particularly challenging perturbation distribution because semantic-changing and semantic-preserving pairs can exhibit similarly high lexical overlap, with subtle word-order and syntactic differences determining their semantic relationships. This may increase the overlap between semantic and nuisance variation in representation space, making selective recovery more difficult even when PAWS-specific training improves displacement reconstruction. Although this provides a plausible explanation for the observed results, we have not directly isolated this mechanism.

We therefore interpret PAWS as an empirical generalization boundary of the current learned atom frames, rather than evidence of universal transfer. Our central contribution is the hypothesis that local semantic variation admits preferred sparse coordinates, together with a framework for recovering them from SP/SC supervision. The controlled perturbation dataset provides one empirical approximation of this geometry, rather than an exhaustive characterization of semantic and nuisance variation. Consequently, the effectiveness of the recovered atoms depends on the coverage of the training perturbations. The PAWS results highlight this distinction: the underlying geometric hypothesis is not restricted to a particular perturbation dataset, but its current empirical realization does not establish distribution-independent invariance. We explicitly clarify this scope in Appendix J.

We also investigated the reported cross-generator asymmetry. The Mistral and Qwen perturbation sets have similar SC-type distributions and share 1,374 anchors, but differ in paraphrase length, lexical overlap, and representation-space statistics. Their SP displacement norms (66.97 versus 59.91) are measured under source-specific normalization and therefore are not directly comparable as absolute magnitudes. A separate diagnostic confirms asymmetric transfer ratios of 1.23 (Mistral→Qwen) versus 0.30 (Qwen→Mistral).

These findings are consistent with differences in nuisance coverage contributing to directional transfer asymmetry. However, the analysis is observational, and we do not claim that nuisance magnitude alone explains the effect.

Manuscript Change: Appendix O, External Generalization on Human-Validated PAWS Perturbations, Tables 42–43 and Sensitivity to representation-layer selection; Appendix D.4, Diagnosing Cross-Generator Transfer Asymmetry, Table 20; and Appendix J, Scope and Limitations of the Invariant Atom Hypothesis.

Together, we hope these additions strengthen the controlled evidence for preferred sparse semantic coordinates, establish that the main results persist after leakage correction, and clarify the generalization boundary of the proposed framework. Should any questions or concerns remain, we would greatly appreciate hearing from you at your earliest convenience, so that we may provide timely clarification within the remaining discussion period.

Reviewer_6Ey2OpenReview ↗

Reply to authors

I thank the authors for the detailed response. My concerns have been addressed and I have revised my scores. I would like the authors to improve the readability of the draft, specifically figures 1, 2, and 3, whose captions seem to be merged with the main text.

We appreciate your feedback and constructive suggestions. We have uploaded a revised manuscript with improved spacing around the captions of Figures 1–3 to enhance readability while keeping the main paper within the 10-page limit. Thank you again for your helpful comments.

Reviewer_NkRzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes invariant atoms as sparse coordinates for local semantic changes in the LLM representations. Using frozen LLM activations, the method learns a shared dictionary that reconstructs semantic-changing displacements while suppressing responses to semantic-preserving transformations.

Critical strengths
  • The experiments provide complementary evidence for the usefulness of the learned dictionary.: Beyond reconstruction and selectivity, Section 4.3 shows bidirectional effects on model predictions with inactive and shuffled controls. Section 4.4 evaluates transfer beyond the standard within-neighbourhood split, supporting reuse beyond the training examples.
  • The reconstruction-only control helps isolate the contribution of SP supervision.: Using the same dictionary and sparse encoder, Recon-Only achieves competitive reconstruction but substantially weaker semantic selectivity. This supports the value of explicitly suppressing SP variation rather than relying on sparse reconstruction alone.
Critical weaknesses
  • The role of near-orthogonality in the semantic hypothesis remains insufficiently established.: Sparse and reusable semantic directions need not be approximately orthogonal. Near-orthogonality is motivated as a useful regularizer for reducing redundancy, but this does not establish it as a property of the underlying semantic geometry.

  • The causal and generalisation results do not fully distinguish a useful transformation-supervised dictionary from preferred semantic coordinates.: Section 4.3 establishes that reconstructed SC displacements influence predictions, but does not compare these interventions with PCA or Recon-Only reconstructions. Consequently, it remains unclear whether the effects specifically support the invariant-atom interpretation beyond successful displacement reconstruction.

Questions for authors

Would PCA and Recon-Only reconstructions produce similar causal effects under the same intervention protocol, with intervention norm and reconstruction quality controlled for or reported? Can individual atoms also show consistent semantic effects across lexically and syntactically different contexts, and recur under independent initialisations?

2 comments

We thank the reviewer for the constructive questions and appreciate the communication opportunity. We address the reviewer's questions in three parts, supported by additional experiments that distinguish the causal effects of invariant atoms from generic displacement reconstruction, examine semantic consistency across contexts, and assess atom recurrence under independent initializations.

1. Do PCA and Recon-Only produce comparable causal effects under controlled interventions?

We extended the causal evaluation to LLaMA, Mistral, and Qwen, comparing invariant atoms with PCA, Recon-Only, cPCA, and invariant-subspace baselines. We evaluate deletion and injection for both SC and SP displacements under three conditions: natural interventions, matched intervention norms, and approximately matched reconstruction fidelity with matched norms.

We measure causal semantic selectivity as S=RSC−RSPS = R_{\mathrm{SC}} - R_{\mathrm{SP}}, where larger values indicate stronger responses to semantic changes relative to nuisance variation. The table reports ΔS=Sbaseline−Satoms\Delta S = S_{\mathrm{baseline}} - S_{\mathrm{atoms}} under the reconstruction-matched condition; negative values favor atoms.

ModelPCA deletionPCA injectionRecon-Only deletionRecon-Only injection
LLaMA−0.0534−0.0502−0.0361−0.0286
Mistral−0.0295−0.0325−0.0068+0.0020
Qwen−0.0389−0.0443−0.0308−0.0288

Across the three intervention conditions, invariant atoms achieve higher causal semantic selectivity than PCA in 18/18 comparisons and Recon-Only in 17/18 comparisons. The improvement over Recon-Only is smaller and varies across intervention settings. Importantly, PCA and Recon-Only also produce causal effects, but generally exhibit greater sensitivity to SP displacements. cPCA can achieve higher selectivity, although its SC causal effectiveness and reconstruction fidelity are substantially lower.

These results support the value of transformation-aware atom learning beyond reconstructing semantic displacements alone, while acknowledging that causal selectivity is not uniquely achievable by our method.

Manuscript Change: Section 4.3, Causal Validation of Invariant Atoms, Appendix M, especially Table 38 and the paragraphs on intervention matching and semantic effectiveness.

2. Do atom directions exhibit consistent semantic organization across different contexts?

We evaluated 13,997 held-out SC examples across 13 domains and seven semantic-change types, using five domain-disjoint partitions. Class centroids learned from one set of domains are evaluated on the other, testing whether coordinate-level semantic associations persist across different linguistic content.

RepresentationSigned-coordinate accuracy
Global atom projection0.749 ± 0.009
Rotated atom projection0.721 ± 0.016
Raw representation0.734 ± 0.010
PCA0.676 ± 0.011
Recon-Only0.705 ± 0.011

Direct projections onto learned atom directions achieve the highest cross-domain accuracy. Rotating the frame reduces performance despite preserving its subspace, providing evidence that semantic-change information is organized non-arbitrarily among the learned directions. We also observe recurring atom–SC-type associations across domain partitions.

These experiments establish cross-context consistency of semantic associations, although they do not establish context-independent causal effects for every individual atom.

Manuscript Change: Section 4.2, Atom coordinates are reproducible beyond subspace stability; Appendix N, Semantic consistency across disjoint contexts and Recurrence of semantic-change-associated atoms, especially Table 41.

3. Do individual atoms recur under independent initializations?

We additionally trained three LLaMA atom frames from independent random orthogonal initializations, rather than sharing the PCA initialization used in our original seed-reproducibility experiment.

The independently initialized models retain S/T values of 1.80–1.88, compared with 1.88 for the original PCA-initialized frame. After Hungarian atom matching, their mean absolute cosine correspondence with the original frame ranges from 0.205 to 0.250, substantially exceeding the random-frame reference of 0.045.

Among the 256 original atoms, 27 directions recur with matched cosine above 0.5 across all three independently initialized runs; 16 exceed 0.6 and five exceed 0.7.

Thus, independent initialization preserves comparable semantic selectivity and recovers a nontrivial subset of recurring directions, but does not yield a unique atom frame. This is consistent with our hypothesis of preferred sparse coordinates without requiring every atom to be individually identifiable.

Manuscript Change: Section 4.2 and Appendix N, Recurrence under independent initialization, especially Tables 39–40. The identifiability limitation is further clarified in Appendix J, Reproducibility and identifiability.

Clarification Regarding Near-Orthogonality

We agree that near-orthogonality is not a necessary intrinsic property of semantic geometry. In our formulation, it serves as a regularization principle for reducing directional redundancy and improving coordinate conditioning. Consistently, removing the orthogonality penalty changes S/T only marginally in the tested ablations (Mistral: 1.27 to 1.28; Qwen: unchanged at 1.63). We therefore distinguish the central geometric hypothesis—preferred sparse semantic directions—from the approximately orthogonal parameterization used to learn them.

Manuscript Change: Section 3.1, Near-Stiefel structure and non-redundant coordinates; Appendix B.6, Table 13; and Appendix J, Reproducibility and identifiability.

We hope these additional experiments strengthen the evidence that invariant atoms provide semantically organized and functionally selective sparse coordinates beyond generic displacement reconstruction, while clarifying the limits of individual-atom identifiability. We would welcome any further questions or concerns during the remaining discussion period and would be glad to provide timely clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions During the Discussion Stage

We sincerely thank all reviewers for their careful evaluations and constructive feedback. We have substantially revised the manuscript and conducted additional experiments to address concerns about the distinction between invariant atoms and invariant subspaces, evaluation rigor, causal validity, reproducibility, and generalization. The revised manuscript is uploaded, and includes six new appendix sections (J–O), extensions to existing appendices, and targeted clarifications in the main paper.

New Experimental Evidence

  • Appendix K Sparse Coordinates vs. Invariant Subspaces: Introduces SP/SC-supervised contrastive baselines (cPCA and invariant subspaces) and orthogonal-rotation controls, distinguishing preferred sparse coordinates from subspace-level invariance.
  • Appendix L Evaluation Rigor: Establishes that the principal findings persist under leakage-free validation and group-disjoint evaluation.
  • Appendix M Causal Validity: Adds causal comparisons with PCA, Recon-Only, and contrastive baselines under natural, norm-matched, and reconstruction-matched interventions.
  • Appendix N Semantic Consistency and Reproducibility: Examines semantic consistency across disjoint linguistic domains and atom recurrence under independent initializations.
  • Appendix O External Generalization: Evaluates generalization on human-labeled PAWS perturbations, including zero-shot transfer, PAWS-specific training, and representation-layer sensitivity.

We further extend Appendix D with an analysis of cross-generator transfer asymmetry and Appendix A with clarification of the complete manual inspection procedure.

Scientific Scope and Limitations

We particularly appreciate the reviewers' questions about what the Invariant Atom Hypothesis establishes. As with most scientific investigation, it is important to clearly define the scope of the proposed hypothesis and acknowledge the limitations of the supporting evidence. We have therefore added Appendix J, Scope and Limitations of the Invariant Atom Hypothesis, to explicitly distinguish the proposed geometric structure from the particular learned dictionary and clarify the boundaries of our theoretical and empirical claims.

Our hypothesis concerns the local organization of semantic variation into preferred sparse directions, rather than the existence of a unique, universally transferable semantic basis. The learned atom frame is a finite approximation estimated from sampled semantic-preserving and semantic-changing perturbations; its effectiveness depends on their coverage.

We also clarify that near-orthogonality is a regularization principle, while finite SP/SC displacements serve as empirical probes of local tangent geometry under stated approximation conditions. These qualifications refine the interpretation of our evidence without changing the central scientific question.

Presentation and Readability

We also appreciate the feedback concerning aggressive spacing. Our original intention was to provide sufficient detail for evaluating a new geometric hypothesis, but we recognize that the resulting density made the paper harder to read. We have used the additional pages permitted during the discussion stage to improve spacing and layout, carefully reviewed the manuscript for clarity and precision, and organized the expanded evidence into dedicated appendix sections.

Overall, we hope the revisions provide stronger evidence that invariant atoms capture a non-arbitrary sparse directional organization beyond invariant subspaces, with functional relevance and reproducible structure under controlled conditions. Equally importantly, they clarify the limits of the current framework and the distinction between empirical support for a local geometric hypothesis and claims of universal semantic invariance.

We hope the revised manuscript and the detailed point-by-point responses address the reviewers' principal concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.