acceptodds
Withdrawn from ICLR 2027

When Is a Steerable Concept Representation Real? Measurement Confounds in a Cross-Family Audit of Neuroscience Parallels in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) are increasingly reported to exhibit human-like neural and cognitive signatures, including concept cells, mental number lines, and cognitive maps. These claims often rely on linear probing and activation steering applied to a single model, yet both methods are highly sensitive to measurement choices. A reported parallel may therefore reflect the model, the measurement procedure, or both. We audit four representative neuroscience-inspired paradigms across 17 models from five families, spanning B to B parameters. Our main experiment examines the causal steerability of concept directions. With raw activation units and a fixed layer and coefficient, steerability appears to increase with model scale, resembling an emergent capability. However, this pattern is produced by an uncalibrated pipeline rather than by a claim established in the steering literature. The trend depends jointly on raw units, the readout metric, and the operating point; correcting any one of these removes it. With residual-norm-comparable interventions and held-out operating-point selection, concept steering remains significant at every scale, but shows no significant trend across the Qwen3 series, although the confidence interval does not rule out a moderate positive slope. The remaining results are mixed. A linear geographic world map is consistently decodable in every tested checkpoint up to B. Number magnitude is strongly encoded, but whether individual neurons appear bell-shaped or monotonic depends on the selection criterion. Language-specific structure is localizable, but the direction of the cross-lingual asymmetry reverses under a different attribution method. These results suggest that the main constraint on AI neuroscience is not a lack of phenomena, but a lack of comparable measurements and adequate controls. We release the protocol, stimuli, and code.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Alignment, Neuroscience

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3RrZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript explores four types of neuroscience analogies in LLM. It emphasizes that the focus of the development of AI neuroscience lies not in the absence of phenomena, but in the insufficiency of comparable measurements and control.

Critical strengths
  1. The article points out that the stability of the world map's decoding lies in its pure decoding process, absence of single neuron screening, and no intervention operation points. The article considers "where to believe and where to be skeptical", but does not directly provide a very definite answer. This approach takes into account the different thinking groups.

  2. The manuscript has made the JSON files related to it public, enhancing the credibility of the article's methods. The manuscript focused on "how to measure correctly". Subsequently, some methods can be applied to the new model for experiments.

Critical weaknesses
  1. The manuscript takes a new perspective, but fails to set a determination threshold for the establishment criteria of the neuroscience parallel. This may cause readers to have uncertain judgments. It is recommended to clearly define the criteria.

  2. The manuscript assumes that the fixed c is a functionally comparable dose across different models. However, the increase in residual norm does not imply that the semantic space, attention dynamics, and inter-layer information flow are comparable. The article uses the inverted-U and layer dependence to illustrate the sensitivity of the operation point, but it fails to conduct a "cross-model equivalence verification under the same c".

  3. The author acknowledges that direction is not threshold-dependent but proportion is threshold-dependent. However, using the "The number is n" prompt within the range of 1-40 might also induce logarithmic/segmented structures. The article should also provide stability data for different numerical ranges and different prompt templates.

  4. The article does not explain the reasons for choosing these four specific paradigms, nor does it systematically describe which type of neuroscience parallel they represent, nor does it clarify whether they cover the main types of representation.

  5. The charts mentioned in the text description (such as Figure 2 and Figure 4) lack detailed annotations. It is recommended that all charts be clearly labeled in the main text with explanations of the meanings of the horizontal and vertical axes, legends, etc. Additionally, it is suggested that the icons be given more detailed descriptions to highlight the experimental results.

Questions for authors

The manuscript reveals the ubiquity of artifact measurement in artificial intelligence neuroscience, preventing the misidentification of methodological flaws as cognitive parallels. Secondly, the experiments span five model families, 17 checkpoints, with scales ranging from 0.6B to 72B. The experiments are relatively comprehensive, but revisions are needed to address the aforementioned weaknesses.

Reviewer_awaEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a method to test the results of methods like linear probing and activation steering to make claims about whether LLMS exhibit human-like neural and cognitive signatures. Specifically, they test four neuroscience-inspired paradigms, including concept cells, mental number lines, and cognitive maps. The main hypothesis is that the reported results that support these paradigms may reflect the model or the measurement procedure chosen.

To check this, they audit these 4 paradigms across 17 models from five families with a method where they standardise different choices made for those methods (R1-R4). Their main result is that in causal steerability (whose general claim was that it appears to increase with model scale), there is a pattern produced by an uncalibrated pipeline. In the other paradigms, results are mixed, and only the linear world map survives the controls.

Critical strengths

The paper is easy to read, and the research questions are well explained and justified. The strongest part is the method proposed, the extended model grid they use, and the fact that the protocol, stimuli, and code are publicly available.

Critical weaknesses
  • (W1) The net contribution of the main claim is limited for three reasons: (1) the paper itself admits that the emergence of steerability is not an established claim; (2) the fact that interpretability conclusions in each method depend on the method selected is already established (Hewitt & Liang 2019; Schaeffer et al. 2023); (3) the novelty is incremental, and the closest prior work on the same cross-model comparability problem is neither cited nor discussed: Huang et al. (ACL 2025)

Cross-model Transferability among Large Language Models on the Platonic Representations of Concepts (Huang et al., ACL 2025)

  • (W2) In terms of rigour, citations of new related work are missing, such as Huang et al., ACL 2025. Also, one citation shows up as (?) (line 103). Finally, in the AI disclosure, there are what look like placeholders from an LLM response, ironically placed right after the sentence where the authors claim to have reviewed all AI-generated content.
Questions for authors
  • (W1) Could you better position your novelty against the literature that already establishes the results' dependency on the method decisions?

  • (W1) Could you position the paper against other analyses like Huang et al., ACL 2025? Their open problem could help redefine R2's scope. They study cross-model transferability learning: how to project one steer vector from one model to another. Your research question is different but highly connected: both tackle the fact that the activation space is different in each model.

  • Could you expand on the limitations of how the models, data, numbers os stimuli can affect results? (for example, only two languages, 45 cities for world map probing...)

  • You admit the trend test is underpowered. What would an adequately powered version of the scaling analysis require? Which of these could you run within the rebuttal period, and how much confidence would you place in the “no trend” claim?

Reviewer_q4kEOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits four neuroscience-inspired findings across 17 open-weight language models: concept steering, numerical tuning, language localization, and geographic decoding. Its main experiment examines how steering results change with intervention normalization, output scoring, and layer/strength selection. The authors find positive steering effects throughout the tested scale range but no detectable trend across five Qwen3 checkpoints. The other analyses report selection-dependent numerical tuning, attribution-dependent language selectivity, and consistently decodable geographic coordinates.

Critical strengths

Selecting the intervention layer and strength on disjoint concepts is a useful improvement over comparing models at an arbitrary operating point. I also value the cross-family replication results. In particular, geographic decoding remains above the shuffled-label baseline across all 17 LLMs, and the numerical analysis includes a digit-to-word test rather than treating digit selectivity alone as evidence of magnitude coding. These measurements are worth reporting independently of the broader audit verdicts.

Critical weaknesses

The main experiment does not establish which choices manufacture the apparent scaling trend. Figure 6 fixes the readout to the specificity contrast, and even its raw-unit, fixed-operating-point condition has no significant slope. This cannot show that normalization alone or operating-point selection alone removes the trend in Figure 2b, which uses a different readout. Nor does a nonsignificant corrected slope establish that the slope changed. The authors appropriately acknowledge the remaining uncertainty, but the stronger claim that correcting any one of the three choices removes an artifact needs a matched comparison that changes one factor at a time.

The language-localization analysis has a related problem with its statistical interpretation. Section 4.4 calls English selectivity indistinguishable from zero in 12/17 models, yet Figure 4 shows several intervals entirely below zero, most clearly Llama-70B near −4.2. Those are negative effects, not null effects. This does not change the count of positive double dissociations, but it does change the interpretation. Likewise, finding significant Chinese and English selectivity in 7/9 models under activation ranking does not establish that their asymmetry reverses. That claim requires a paired comparison of the signed effects under the two attribution methods.

Questions for authors
  1. On the same concepts and prompts, what happens to the scaling slope when normalization, readout, and operating-point selection are changed separately? Please report uncertainty on the change in slope, not only on each slope.
  2. Can you reconcile the English significance counts with Figure 4 and provide the paired language-asymmetry comparison for the nine models evaluated under both attribution methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.