acceptodds
Under review as a conference paper at ICLR 2027

Representation Symmetries and Architectural Constraints on Probe Identifiability

Abstract

A central question in neural representation analysis is when observations of hidden representations uniquely determine an associated symbolic assignment. A symbolic label may be predicted accurately from a representation without being uniquely determined by the observed information. Architectural information can restrict hidden-coordinate changes that preserve a neural model and its observations, but invariance under known symmetries does not by itself imply identifiability. We distinguish observational identifiability, symmetry invariance, and agreement of fixed linear sign-probe predictions. A general observation-factorization criterion characterizes exact extractability, while full-space and boundary-local probe criteria identify the positive-ray condition for exact agreement. We then study finite representation calibration, where allowed orthogonal transformations fix specified hidden states pointwise. For an explicit norm-and-inner-product observation protocol on a known linear subspace, we classify every observational fibre and characterize both global and fibrewise probe identifiability. Under Gaussian evaluation, the largest disagreement over all calibration-preserving transformations has a closed form determined by the uncalibrated component of the effective probe and is attained by a Householder reflection. Independent Gaussian calibration gives an exact finite-sample law governed by the support dimension. Architectural stabilizers provide upper bounds, not automatic completeness certificates. Controlled FP64 experiments agree with the finite-calibration predictions. Calibration-preserving weight changes in Pythia-1.4B and Pythia-2.8B still change fixed-probe predictions, while complete-forward checks identify a reproducible numerical deviation under one joint intervention.

Primary area:
interpretability and explainable AI
Keywords:
representation identifiability, neural network symmetries, linear probing

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_zQbxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper uses concepts and tools from abstract algebra to formalize symbolic equivalence and observational equivalence of hidden representations, and to study when the latter implies the former.

Critical strengths
  • The definitions of the observation map, symbolic assignment, etc., build a bridge from abstract algebra to the problem studied, and this framing is potentially useful for further studies.
    • This is balanced by the fact that Theorem 1 and some other early results are mostly "translations" of textbook math results, e.g., Theorem 3.5 in [1]; Dummit & Foote should also contain this result. This is not a weakness in itself, though, as long as the imported math can lead to interesting results in ML — but I think it does not; see below.

[1] Blyth, Thomas Scott. Module theory: an approach to linear algebra. University of St Andrews, 1990.

Critical weaknesses
  • It is not clear how Proposition 2 and the analysis of fixed probes under a single hidden representation (Theorems 2 and 3) or under multiple hidden representations (Theorems 4 and 5; termed calibration in the paper) shed light on representation learning.
    • What is the practical significance of the sign-probe as a symbolic assignment? In particular, how do we choose the state vector ww, and what does it mean for a problem setting? When can we further obtain those multiple calibration states? What is their practical interpretation?

      • The paper admits that the results do not "assign logical semantics to probe labels." But I think you should show some examples in which the probe labels have semantics, as evidence that your math does mean something in practice.
    • For example, Theorem 2 characterizes "when the labels produced by a fixed linear probe are exactly extractable under the remaining representation transformations." But why does this question matter?

    • Similarly, Proposition 2 establishes if-and-only-if conditions between symbolic equivalence and observational equivalence, which is interesting math, but it needs examples built on real DNNs to show that it is interesting for representation learning.

  • The exposition writing requires much more human effort than it is now. Some example issues (ordered roughly by importance):
    • Is the gauge known from the problem setting? Examples?

    • Section 5 as a whole is unclear:

      • Line 270 puts requirements on the global linear symmetry group. This is necessary for your math to be well-defined. But again, how are these requirements relevant to practical NNs?
      • "…upper bounds on the actual observation-preserving group rather than classifications of observational fibres." This nearly identical sentence appears in both the Introduction and Section 5, with no further explanation. What is the "upper bound" (I guess it is related to Equation 1, which appears later and is itself unclear; see below), and what is the "classification"?
      • Equation 1 and the surrounding discussion are unclear; e.g., what is an "architectural datum"? When in practical NNs does it happen that it "takes values in a space with a specified left group action"? "Stabilizer" is mentioned here for the first time since the Abstract ("architectural stabilizers"), and it is undefined!
    • What are the "allowed transformations", and how do we see from the statement of Theorem 5 that they "have zero disagreement"?

    • Lemma 1 says that GOG_O is a subgroup, but it appears after you have already claimed this on the line above! Neither Lemma 1 nor Corollary 2 is used later, and they indeed interrupt the flow, so they should be moved to the Appendix.

    • It is odd that the term "covector" appears in the paper; this is a highly specialized term, but the paper uses it only to distinguish row and column vectors! Possibly a sign of AI slop.

  • The Introduction (two paragraphs) and Related Work (one short paragraph) are far too thin to convey the significance and novelty of the work.
    • No contributions are listed.
    • Some related work is missing: [1] the mathematical formulation seems related to Higgins et al. (2018); [2] (possibly parallel), which gives identifiability result via concrete analysis of model symmetry of a nonlinear class, is arguably more related to your paper than Godfrey et al. (2022) or Grigsby et al. (2023), since the latter two do not seem to study identifiability.

[1] Irina Higgins, David Amos, David Pfau, Sébastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. In arXiv preprint arXiv:1812.02230, 2018.

[2] Pengzhou Wu. "Beyond ICA: Identifiability by Symmetry Breaking." NeurIPS 2026.

Questions for authors
  • My questions are interwoven with the Weaknesses.
  • Given the current LLMs, it is challenging but not impossible that the paper could be improved close to the acceptance bar, and I will take a look it the revised paper.
Reviewer_V4YiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies when a fixed linear sign probe’s label is determined by observations of hidden states, given that output-preserving linear reparameterizations exist. It shows that a fixed probe is invariant exactly when every symmetry maps the probe’s direction to a positive multiple of itself, and it derives upper bounds on the symmetry group from the architecture. For a finite calibration protocol, it characterizes when the probe label is identifiable, gives the worst-case disagreement under Gaussian inputs, and gives its exact distribution under random calibration. Numerical checks and GPT-2/Pythia interventions confirm that fixed probes flip under such reparameterizations, while transported probes do not.

Critical strengths

The theory is rigorous and correct as far as I checked (Thms. 4–5, Cor. 6). It cleanly separates identifiability, invariance, and fixed-probe agreement, and the limitations are stated candidly. Theorem 5 / Corollary 6 give an attained worst-case bound with an explicit Householder witness and an exact finite-sample law

Critical weaknesses

Weak motivation. A probe is a covector, so a frozen probe is expected to change under reparameterization. In standard practice probes are refit, which gives the transported probe, and the paper’s own results show transported probes are invariant. Most experimental outcomes (0%, 100%, about 50% disagreement, and roughly 10−1510^{-15} errors) are therefore forced by algebra. The paper names no real workflow that applies a frozen probe to a reparameterized model. Limited novelty and weak link to the experiments. Much of the framework restates standard quotient arguments, and the architectural bounds have no criterion for when they are tight. In the GPT-2 calibration study, the reflection is built from the probe itself, and the Gaussian theory is not claimed for real features, so the study exhibits an example rather than testing the theory. Closely related work (Park et al., 2024; Moschella et al., 2023) is not discussed.

Questions for authors
  1. In what real workflow is a probe fit in one basis and applied unchanged to a reparameterized model, e.g., transfer across checkpoints or fine-tunes? Can you demonstrate the framework there?
  2. Is there any realistic protocol, other than section 6, for which orbit separation or stabilizer completeness can be verified?
Reviewer_Wc6ROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies disentangling observational identifiability from invariance under representation symmetries.

Critical strengths

from how it is written, i don't understand the problem domain, the gap in the literature, and the solution. there might be strengths, but I cannot assess from the submitted paper.

Critical weaknesses

it is poorly written and for someone who has not worked in the field of neural representation analysis, this paper is not decodable. writing has to be improved significantly. from the introduction, i do not understand the problem and contributions at all.

Questions for authors

N/A

Reviewer_metROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Probes are linear classifiers that predict attributes such as sentiment from a model's hidden vectors. Prior work in neural representation analysis considers changing these vectors and adjusting the downstream weights to keep model outputs unchanged. Reusing a probe trained on the original representations can then produce different predictions.

The paper adds a new constraint to these weight changes: it selects a set of reference hidden vectors and requires each to remain unchanged. Under this restriction, it asks to what extent the same probe is guaranteed to preserve its predictions on other inputs.

Within this setting, the paper derives conditions for prediction invariance and constructs a reflection that fixes the reference vectors while changing probe predictions on other inputs. To quantify this effect, the authors assume Gaussian test vectors and calculate the largest possible fraction of predictions that can change. They then assume that the reference vectors are also independent Gaussian samples and analyze how their number affects this fraction. In GPT-2 experiments, the authors implement the reflection through weight changes and verify that the original probe's predictions on other inputs can change substantially while model outputs and reference vectors remain unchanged.

Critical strengths

The authors implement the reflection in the attention weights of a full pretrained GPT-2 model and adjust the subsequent projection to cancel its effect on model outputs. They then rerun the full model, verify that the outputs and reference vectors are preserved, and confirm substantial changes in the original probe's predictions on other samples. These experiments demonstrate that the construction can be implemented in a real network while satisfying both requirements: preserving model outputs and keeping the reference vectors fixed.

Critical weaknesses

1. Fixing reference vectors adds limited value beyond the existing setting.

Existing analyses of representation reparameterization already consider rotations of hidden representations that preserve model outputs. This paper additionally requires a finite set of reference vectors to remain unchanged. This constraint fixes their span but leaves the original freedom to rotate or reflect in the orthogonal complement. The counterexample therefore uses the same geometric mechanism, with changes confined to the remaining directions. The paper does not explain which important model interpretation or probe transfer problem calls for this additional constraint, or show how the resulting analysis advances our understanding of practical alignment or probe adaptation. Relative to the existing setting, this extension offers limited new insight and limited guidance for practice.

2. The core theory quantifying prediction changes under reflection provides limited new insight.

Once the paper's setting is accepted, the worst-case disagreement in Theorem 5 follows from a direct geometric argument. The proof separates the probe into components within and orthogonal to the span of the reference vectors. Reversing the latter maximizes the angle between the original and transformed classification directions. Under the paper's Gaussian assumption, disagreement is proportional to this angle, so the reflection also attains the worst-case disagreement. Corollary 6 then applies the standard distribution of a random subspace projection to obtain the relationship between the number of random reference vectors and disagreement. The new derivations therefore mainly apply standard geometric and probabilistic results to a setting with fixed reference vectors, providing a limited theoretical advance.

Questions for authors
  1. Compared with the setting without fixed reference vectors, what important new problem is addressed by fixing a finite set of references? What need in model interpretation or probe transfer does this constraint capture, and how do the results change our understanding of representation alignment or probe adaptation?
  2. What key new insights do the exact results in Theorem 5 and Corollary 6 provide about the stability of a fixed probe? Please explain their importance and how they go beyond the phenomenon already demonstrated by the reflection counterexample.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.