- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes invariant atoms, a framework for capturing semantic variation in LLMs that remains stable under meaning-preserving transformations.
Critical strengths
- The concept of meaning-preserving transformations is novel and interesting to the best of my knowledge.
- The schematic figures are instructive.
- The experiments are well done and extensive (across seven model families).
- The dataset construction is overall well documented.
Critical weaknesses
- The invariant semantic subspaces proposed by Dasgupta et al. (2026) seem closely related. The paper is cited, but a more detailed comparison, possibly including experiments, would be good.
- Excessive white space removal makes the paper very dense and hard to read. The paper’s clarity would benefit from streamlined writing.
- The manual quality checks of the dataset could be described in more detail, e.g., the sample size and error rate.
Questions for authors
When do finite SP/SC displacements approximate the proposed nuisance and semantic tangent components well (Sec. 3.1)? Some more details would be good.
3 comments
2. Dataset quality verification
We have clarified the extent of manual inspection. All 62,693 selected SP/SC variants, across 1,400 anchor-centered neighborhoods, underwent manual quality inspection—not merely a random subset. SP variants were checked for preservation of the anchor's meaning, while SC variants were checked for the intended semantic change without substantial unrelated modifications.
The retained variants were judged to satisfy these criteria during inspection. However, we do not equate manual inspection with guaranteed annotation correctness: residual errors cannot be ruled out, and no separately measured annotation error rate or independent inter-annotator agreement is available. We have made this distinction explicit.
Manuscript Change: Appendix A, Manual quality verification.
3. Presentation density and readability
We appreciate the reviewer's observation. Our intention in using compact spacing was to provide sufficient detail to clearly explain and evaluate the proposed Invariant Atom Hypothesis. However, the reviewer's feedback helped us recognize that the aggressive spacing may have placed an unnecessary burden on readers. We have therefore used the additional pages permitted during the discussion stage to improve the layout and readability. We have also carefully reviewed the manuscript sentence by sentence to improve clarity, precision, and conciseness.
We hope these revisions make the paper easier to follow while preserving the technical details needed to evaluate our contributions. Should any questions or concerns remain, we would be grateful for the opportunity to clarify them promptly during the discussion period.
Thanks to the authors for the detailed response, which addressed my questions. While still very dense, the revision has improved the readability of the paper. I maintain my positive score.
We thank the reviewer for the encouraging assessment and constructive suggestions. We appreciate the opportunity for constructive scientific discussion. We have expanded the comparison with invariant semantic subspaces, clarified the geometric interpretation of finite SP/SC displacements, and provided additional details on dataset verification.
Question:
1. When do finite SP/SC displacements approximate tangent directions?
We agree that this distinction warrants clarification. Our formulation concerns local tangent geometry, whereas the experiments observe finite differences between discrete text inputs.
We have added a conditional first-order analysis. Suppose a smooth representation trajectory γ:[0,ϵ]→M connects two representations and satisfies ∥γ¨(t)∥2≤M. Taylor's theorem gives
\qquad \|r_\epsilon\|_2\leq \frac{M\epsilon^2}{2}.$$ Consequently, $$\left\|\frac{\Delta z}{\epsilon}-\dot{\gamma}(0)\right\|_2 \leq \frac{M\epsilon}{2}.$$ Thus, finite displacements provide accurate first-order tangent approximations when the connecting representation trajectory is sufficiently local and its curvature-induced error is small relative to its first-order motion. Interpreting these directions as nuisance or semantic tangent components additionally requires that the corresponding trajectories follow the appropriate semantic equivalence or semantic-changing variations. Importantly, **discrete sentence pairs do not uniquely define smooth trajectories**, and an SC edit may simultaneously change semantic content and surface form. We therefore treat finite SP/SC displacements as *empirical, transformation-labeled probes* of local geometry, rather than exact tangent vectors. The mathematical analysis identifies sufficient approximation conditions, not assumptions verified for every sentence pair. **Manuscript Change:** Section 3.1–3.2; Appendix J, *When do finite displacements approximate tangent directions?* ### Weakness: **1. Relationship to invariant semantic subspaces (Dasgupta et al., 2026)** We agree that invariant subspaces and invariant atoms are closely related. However, they characterize **different levels of semantic geometry**. An invariant subspace identifies *where* semantic variation occurs, but does not specify preferred directions within that subspace: orthogonal rotations preserve the subspace. Our hypothesis goes further by proposing that semantic motion admits **preferred sparse coordinates**, allowing individual changes to be represented efficiently through recurring directions. To distinguish these properties empirically, we added the generalized-eigenvector invariant-subspace baseline (InvSub) using the same SP/SC supervision as our method. | Method | S/T ↑ | SC reconstruction cosine ↑ | |---|---:|---:| | InvSub ($K=256$) | 8.03 | 0.136 | | cPCA ($\lambda=1$) | 2.61 | 0.521 | | Global atoms | 1.88 | 0.670 | | Local atoms | 1.98 | 0.679 | On LLaMA, InvSub achieves stronger semantic selectivity but substantially lower reconstruction fidelity. Our method instead balances selectivity with sparse preservation of semantic displacement. More directly, we tested **whether the learned atom directions matter beyond their span**. Randomly rotating the learned frame preserves its subspace but reduces top-32 reconstruction cosine from 0.675 to 0.421. The rotated frame requires approximately 128 coordinates to match the original top-32 fidelity. This provides direct evidence for preferred sparse coordinates beyond subspace membership, without implying a uniquely identifiable basis. **Manuscript Change:** Section 3.1, comparison with invariant semantic subspaces; Section 4.2, *Contrastive subspaces do not establish preferred sparse coordinates*; Appendix K, *Contrastive Subspaces and Preferred Sparse Coordinates*, Tables 34–35.