acceptodds
Under review as a conference paper at ICLR 2027

Representation Without Capability, Then Capability Without Abstraction: Parity in GPT-2

Abstract

Probing has become a standard instrument for characterizing what a language model represents, and it underlies a family of intervention methods, among them directional ablation and activation steering, that manipulate behavior aling probe-derived directions. These methods operationalize a rarely tested assumption: that the direction along which a concept is linearly decodable (representation) is the direction the model reads and uses (capability). Evaluating this presupposition is non-trivial for semantically rich concepts because a robust test requires an independent ground truth about what the model actually computes. We therefore construct a setting in which that ground truth is available and the rule the model implements can be identified from behaviour. We use the pretrained GPT-2 small model, whose representations were formed by ordinary language modeling, a natural setting. We observed that pretrained GPT-2 cannot perform parity detection above chance in any prompt format we administered, while a hyperplane in its embedding space separates numerals by parity at 0.994 on held-out tokens: Representation Without Capability. This representation the model cannot act upon is selective, as divisibility by three performs at 0.687. We then fine-tune it on a three-operand arithmetic verification task adapted from an intracranial-EEG protocol for human subjects kalinova2026temporal: with false candidates at . We fine-tune on both the easy and difficult variants, which differ in operand and result digit-width: single-digit operands with results in , versus a two-digit with results in . Arithmetic is not an intrinsic capability at this model scale and remains heuristic at far larger ones. We observed GPT-2 constructing a chained-parity solution: the fine-tuned model attains 97.9–100% accuracy by comparing the candidate result's parity against . Neither constituent of this solution originates in fine-tuning: parity is already separable in the embeddings, and the pretrained model already computes XOR at 0.91–1.00 over other feature-disjoint pairs. Fine-tuning enables the connection between parity identity and XOR, turning on the chained-parity detection capacity that solves the task. But the Capability it creates is not Abstract: behavioral tests suggest that the model stores a parity bit per token-position pair but does not compute parity from the numeral. A token never seen in contributes always even, and an odd three-digit and the word “cat” become indistinguishable, both at 0.985 and mean 8.25. XOR composition was inherited from pretraining. Causal ablation confirms parity is used upon fine-tuning in the arabic easy task, as removing it costs 6.5 accuracy points at rank 40 against 0.2 for an equally decodable property of the same token and 0.0 for random directions. In the difficult condition, however, parity and the matched control damage the model equally at rank 80 (0.9554 vs 0.9582, against a 0.9826 baseline) while random directions do not (0.9813), suggesting that parity over a ninety-token operand occupies far higher rank than over a ten-token one. Interestingly, learnings from the easy condition show that removing parity leaves the model confident and wrong while random ablation leaves it hesitant and right. The same signature appears when we break a token slot's surface format, an intervention applied at the input that touches no activations. Making parity usable also dispersed it: an identical 40-dimensional cut reduces its decodability to 0.4916 in the pretrained model—chance—but only to 0.8183 after fine-tuning. Our findings have practical implications for questioning how the representation-capability relation is tested. Standard procedures of directional ablation, computing the direction on a forward pass and projecting it out finds nothing: accuracy stays at 1.000, failing because every block writes parity back into the residual stream along fresh directions. Altogether, this paper suggests that representation does not entail capability, and capability does not guarantee abstraction. These observations have consequences for the literatures of probing and benchmarking, and for current views on concept erasure and approaches for unlearning. Beyond, we connect our findings with safety discussions, since a procedure that activates dormant knowledge is powerful when intended and hazardous when it arises as an unanticipated side effect of postraining.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation, capability, abstraction, safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_87ZbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether information that can be linearly decoded is actually used by the model. It shows that parity can be decoded from pretrained embeddings even though the model cannot answer parity questions. After fine-tuning on an arithmetic verification task, the model mainly learns a parity shortcut instead of arithmetic. The later experiments show that this behavior is brittle and depends on specific tokens and formats rather than an abstract parity rule.

Critical strengths
  • Control setups are used so shortcut can be identified clearly.
  • The parity-neutral and out-of-distribution tests make it easy to see what the fine-tuned model is doing.
  • The paper includes several diagnostic and causal controls instead of relying only on probe accuracy.
Critical weaknesses
  • The paper is a technical report written by AI that is mostly unreadable. There are massive paragraphs with dense numerical performance numbers, repetitive language, and little to no structure in the paper.
  • The easy and difficult settings vary in arithmetic complexity, but both preserve the same perfect parity shortcut. As a result, the main finding still depends strongly on the way the training data is constructed
  • The paper overinterprets probe accuracy as evidence of a parity representation. Its own analysis shows that parity is mostly recoverable from final-digit identity, which is a lexical property. Prior work has already shown that lexical information can dominate representations [1], probe accuracy does not necessarily imply task relevance [2], and decodability does not establish causal use [3].
  • There are no insights in the paper that are not already well known from probing to shortcut learning.

[1] On the Persistent Effects of Lexicality in Large Language Models

[2] Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?

[3] Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

Questions for authors

N/A

Reviewer_jd2dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates the relationship between representation, capability, and abstraction using parity in GPT-2. Specifically, it distinguishes between the direction along which a concept is linearly decodable (representation) and the direction the model reads and uses (capability). The authors find that numeral parity is highly decodable from pretrained embeddings, whereas the model performs near chance on the tested parity-judgment prompts. They then fine-tune the model on an arithmetic verification task in which incorrect answers differ from correct answers by ±1, making parity a sufficient shortcut. Counterfactual evaluations demonstrate that the resulting high accuracy largely reflects parity checking rather than arithmetic verification. Further experiments reveal poor generalization to unfamiliar numeral formats and token–position combinations, which the authors interpret as evidence for position-dependent parity lookup. Layerwise probes and directional ablations examine the availability and behavioral relevance of parity information before and after fine-tuning. The paper argues that decodability, successful task performance, and abstraction require separate assessment, and suggests that fine-tuning connects pre-existing representations with computational components.

Critical strengths
  • The experimental setting is controlled and interpretable. The choice of parity as a precisely defined property, together with an arithmetic verification task that admits a clear parity-based shortcut, makes the distinction between decodable information and observed behavior concrete.

  • The behavioral evidence for shortcut learning is convincing. Counterfactual answer sweeps and parity-neutral evaluations distinguish parity checking from arithmetic verification, demonstrating why near-perfect accuracy on the original task can give a misleading impression of the capability acquired through fine-tuning.

  • The generalization tests are informative. The paper examines changes in arithmetic content, numeral format, and operand position, including interventions that change only one slot’s format at a time.

  • The experimental controls are meaningful. Equation-level splitting, answer-only baselines, and shuffled-label probes address relevant confounds. The ablation experiments also include both random directions and a matched, equally decodable property.

Critical weaknesses
  • I am not convinced that the central claim about reusing pretrained machinery is established. The abstract states that XOR composition was inherited from pretraining, but the main text explicitly acknowledges that demonstrating reuse of a pretrained XOR circuit requires further causal experiments (lines 404–406 and Appendix A.5). It is unclear to me how observing decodable XOR information before and after fine-tuning establishes that the same computational mechanism was reused.

  • The token–position lookup mechanism is not uniquely supported by the experiments. Several generalization tests change both token familiarity and numeral format, making their respective effects difficult to disentangle. The observed failures are consistent with position-dependent lookup, but could they also arise from format-dependent access to more general parity information?

  • The paper’s specific contribution is not sufficiently clear, particularly in relation to existing causal-probing methods. As far as I understand, the distinction between decodable information and behavioral use is already central to amnesic probing. The paper should clarify what its adaptive ablation adds beyond these established ideas and methods.

  • The robustness of the mechanistic conclusions is insufficiently demonstrated. The analyses focus on selected first-peak checkpoints from training runs that repeatedly alternate between high accuracy and collapse. It remains unclear whether the proposed mechanisms persist across independent training runs and later checkpoints.

  • The presentation needs substantial tightening. The abstract is unusually long—approximately 670 words—and extends onto the second page. It includes detailed experimental results that make the main contribution harder to identify. More importantly, the manuscript should consistently distinguish demonstrated findings from proposed explanations, particularly regarding pretrained circuit reuse and the internal implementation of parity checking.

  • The reference at lines 558–559 appears to be incorrectly attributed (hallucinated author?): it duplicates the paper cited at lines 507–509 but lists different authors.

  • Amnesic probing is discussed, but the relevant paper is not directly cited. The reference to Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals should be added.

Questions for authors
  1. What evidence supports the claim that fine-tuning reuses pretrained XOR circuitry? How should this claim in the abstract be reconciled with lines 404-406 and Appendix A.5, which acknowledge that causal verification remains outstanding?

  2. Does the proposed token-position lookup mechanism still explain the results when token novelty is varied independently of numeral format? -> for example, by withholding both odd and even numerals within the trained digit-width and operand position? Could the observed failures instead reflect format-dependent access to more general parity information?

Reviewer_JbuMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the distinction between representation, capability, and abstraction in language models using parity detection and arithmetic verification tasks. The authors show that parity information is linearly decodable from pretrained GPT-2 Small representations, even though the model cannot reliably perform parity judgments. After fine-tuning, the model achieves high accuracy on an arithmetic verification task by exploiting a parity-based shortcut rather than performing arithmetic. The authors further demonstrate that this behavior does not generalize reliably to unseen tokens or input formats.

Critical strengths

The paper provides a relatively controlled experimental setting, and the counterfactual evaluations help identify the shortcut strategy used by the fine-tuned model.

Critical weaknesses
  1. Limited novelty in both the research question and the conclusions.

The central observation that linear decodability does not imply causal relevance or behavioral capability is already well established in the probing and mechanistic interpretability literature. Similarly, the distinction between benchmark performance and genuine abstraction or systematic generalization has been studied extensively. The paper largely reproduces these established insights in a narrowly constructed parity task, without offering a substantially new theoretical perspective, methodology, or empirical finding.

  1. Extremely limited model coverage and questionable contemporary relevance.

All experiments are conducted using GPT-2 Small (124M parameters), a model introduced in 2019. Despite being submitted to ICLR 2027, the paper does not evaluate a single contemporary language model or even a second model architecture. The different fine-tuning conditions do not address this limitation, as they all use the same pretrained model.

Although small models can be useful for controlled mechanistic studies, the authors make broad claims about language model representations, capabilities, abstraction, and even safety. Such claims are difficult to justify based on experiments involving only one small, outdated architecture. The complete absence of cross-model validation substantially limits the generality and significance of the findings.

  1. The experimental task is overly artificial, and the primary result is largely expected from its construction.

The arithmetic verification task is designed such that every incorrect answer differs from the correct result by exactly ±1. Consequently, parity checking is sufficient to solve the entire task without performing arithmetic. The finding that the fine-tuned model exploits this shortcut, and consequently fails when incorrect answers differ by ±2, is therefore not particularly surprising.

This experimental design demonstrates a familiar shortcut-learning phenomenon rather than establishing a fundamentally new insight into language model reasoning or abstraction. It is unclear how much the conclusions extend beyond this carefully constructed setting.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.