acceptodds
Under review as a conference paper at ICLR 2027

A Polanyi-Like Paradox in Large Language Models: The Gap Between Spatial Representation and Decision

Abstract

Polanyi’s paradox—that we can know more than we can tell—motivates a question for spatial decision-making in large language models (LLMs): can spatial information recovered from their internal representations support more accurate spatial predictions than the models' final choices? We investigate this question in a route-choice task requiring the integration of separate travel experiences to select the next move yielding the shortest route to a destination. Using 115,362 task inputs, we trace how spatial information recovered from hidden states relates to model choices across layers, and use causal interventions to test how intermediate computations influence output preferences. The results show that spatial connectivity and distance are partially recoverable from hidden states, with more accurate recovery of start-centered distances than global pairwise distances under the respective probing setups. Route-comparison accuracy exhibits a sharp increase at intermediate layers, suggesting an emergence-like transition in the accessibility of goal-relevant spatial information, consistent with the integration of fragmented spatial experiences. Route preferences obtained from hidden states are more accurate than the final choices in every evaluated model, revealing a discrepancy between internally recoverable spatial information and behavior. Causal interventions suggest that opposing component effects and their downstream changes may contribute to output-preference instability across layers. Together, these findings reveal a Polanyi-like representation–decision gap: spatial information recoverable through supervised readouts is not fully reflected in the models’ choices. Our results distinguish the emergence of decodable spatial information from its reliable behavioral expression and provide a framework for diagnosing failures in LLM spatial decision-making.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Spatial Decision-Making, In-Context Representation Learning, Representation–Behavior Gap, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_td3yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether spatial information recoverable from LLM hidden states is reflected in the models' final choices. It uses a synthetic route-choice task. Models must integrate textual travel experiences and choose the next move that gives the shorter path to a destination. Linear probes recover spatial distances and route-cost differences across layers. These probe predictions are compared with Logit Lens preferences and with the final answers. Activation patching is then used to study how later computation changes these preferences. Across six Qwen3.5 and Gemma 3 models, route-comparison probes outperform final-answer accuracy by 5.4 to 16.6 points. The paper interprets this difference as a representation–decision gap.

Critical strengths
  1. The task design is carefully controlled. Matched variants change the history, destination, and option order while preserving much of the surface form. This enables clean counterfactual comparisons and supports the intervention experiments.

  2. The experimental execution is strong. The results are consistent across six models from two families, and the probing and intervention analyses are applied systematically across layers.

Critical weaknesses
  1. The gap between the supervised probe and model behavior is difficult to interpret. The route probe is trained on thousands of labeled examples with direct supervision on the target quantity, while the model answers the task zero-shot. A supervised linear readout outperforming zero-shot behavior does not by itself show that the model has computed task-relevant information that it fails to use. The near-chance probe accuracy in early layers suggests that the probe is not simply decoding the input. Even so, this observation does not establish that the decodable signal is information the model itself relies on, rather than structure that the supervised readout extracts from intermediate features. Additional controls would help separate these explanations. Examples include shuffled-label probes, randomly initialized models, and probes over simple input-derived features. This concern is especially relevant for the start-centered probes, which already perform substantially better than the random baseline from the earliest layers and change relatively little with depth.

  2. Prior work already shows that information decoded from hidden states can be more accurate than model outputs, and the paper positions itself as extending this observation to spatial decision-making. The intervention results show that some hidden states and components influence output preferences. However, they do not explain the observed probe–behavior gap of 5.4 to 16.6 points. The paper also does not test whether the decoded signal can actually improve final decisions, or whether the gap persists under stronger reasoning settings such as thinking mode.

Questions for authors
  1. How do the route-comparison probes perform with shuffled labels, with randomly initialized models, or against simple input-feature baselines?

  2. Why are start-centered distances already strongly recoverable in the earliest layers, and how much of this can be explained by shallow properties of the input?

  3. Does the probe–behavior gap persist when Qwen3.5 is evaluated in thinking mode or with explicit reasoning?

  4. Can the probe signal or the identified components be used to improve final-answer accuracy?

3 comments

Overall response

Thank you for your constructive feedback. Your comments motivated additional probe controls, analyses of shallow input cues, tests of whether interventions improve final answers, and a matched thinking-mode comparison. We have uploaded a revised PDF, with major changes highlighted in blue. We address your concerns and questions below.

Responses to Weaknesses

W1 — Probe supervision, controls, and shallow input cues

We added intercept-only baselines, shuffled-label controls, input-feature baselines, and target-exclusion analyses. The results show that final-layer Δ probes outperform shuffled-label and basic-feature controls. We therefore interpret the probe results as evidence that task-related information is linearly recoverable. Detailed methods and results are provided in Q1 and Q2.

W2 — Explaining the gap and improving decisions

We added answer-label bias calibration, competing-component mean ablation, Δ-direction interventions, and a matched thinking-mode comparison. Calibration reduces but does not eliminate the gap, and ablating some competing components improves final-answer accuracy. Δ-direction interventions do not establish a consistent accuracy improvement. Thinking eliminates the gap on the successfully completed matched inputs across all three Qwen3.5 models. Detailed methods and results are provided in Q3 and Q4.

Responses to Questions

Q1 — Shuffled labels, random initialization, and input-feature baselines

We added shuffled-label controls and a linear input-feature baseline. For the shuffled-label controls, we kept hidden states and data splits fixed, shuffled targets within partitions while preserving matched A/B swaps, and retrained probes using three random seeds. The input-feature baseline uses location mention counts, first and last occurrence positions, and indicators identifying the start, destination, option A, and option B.

ModelTrue-label Δ probeShuffled-label controlBasic input features
Qwen3.5-4B81.00%50.29%49.92%
Qwen3.5-9B83.72%50.16%49.92%
Qwen3.5-27B90.50%49.63%49.92%
Gemma3-4B78.84%50.25%49.92%
Gemma3-12B83.73%50.21%49.92%
Gemma3-27B84.09%49.68%49.92%

Final-layer Δ probes outperform both controls in all six models. Revised Figure 4 (line226) includes the full layerwise comparisons, with methodological details in the appendix.

Q2 — Early-layer start-centred distance recovery

The apparent flatness partly reflects the original plotting scale: including the substantially higher errors of the random-sampling baseline expands the vertical range, making changes in the probe curves less visible. Revised Figure 2 (line 172) makes the early-layer decrease in global-distance MAE and increase in adjacency Jaccard more visible.

Some early distance recoverability can also be explained by input statistics and query-role constraints. However, after excluding the two candidates and the destination from both training and evaluation and retraining the probes, start-centered probes retain an advantage over global probes on identical targets, with maximum relative MAE reductions of 11.1–14.6% (revised Figure 3 (line 187)).

Q3 — Thinking mode and explicit reasoning

Given the computational cost of generating long reasoning traces in thinking mode, we used the same 960 inputs as in the intervention experiments to compare Qwen3.5-4B, 9B, and 27B in thinking and non-thinking modes(line399). We excluded 44 truncated generations for 4B and nine for 9B from both modes in the paired comparison. For completed responses that did not begin with A/B (208 for 4B, one for 9B, and none for 27B), we scored the explicitly stated choice in the full final response.

At the position sampled after thinking, Δ information is already highly recoverable in early layers, with first-layer direction accuracy ranging from 88.18% to 91.16%. At the final layer, both probe accuracy and final-answer accuracy reach 100% in all three models on the completed inputs, reducing the probe–behavior gap to zero(Figure7-line 432).

Thinking provides additional reasoning steps before the final choice. As illustrated by the examples in the appendix F (line 1735), models explicitly enumerate candidate routes, count and compare their lengths, and then select an answer. These step-by-step calculations provide intermediate results that can support the final decision, offering a plausible explanation for the improved accuracy and the disappearance of the gap on the retained matched inputs.

Q4 — Improving final-answer accuracy with probes or components

We tested three approaches: answer-label bias calibration, mean ablation of identified competing components, and interventions along the Δ-probe direction.

  1. Answer-label bias calibration

    We fitted an additive bias to the final A/B logit margin on the training folds and evaluated it on the corresponding test folds, keeping model parameters and Δ probes unchanged. Calibration improves final-answer accuracy in all six models, although gaps of 4.72–9.38 percentage points remain relative to the final-layer Δ probe.

    ModelOriginal accuracyCalibrated accuracyΔ-probe accuracyRemaining gap (pp)
    Qwen3.5-4B72.14%74.31%81.00%6.68
    Qwen3.5-9B78.27%79.00%83.72%4.72
    Qwen3.5-27B73.90%82.20%90.50%8.30
    Gemma3-4B67.29%70.34%78.84%8.50
    Gemma3-12B74.39%75.29%83.73%8.43
    Gemma3-27B70.12%74.71%84.09%9.38
  2. Mean ablation of competing components

    For each identified competing component, we replaced its residual contribution at the final input token with its mean over the original training families. Components were ablated individually on the same 960 intervention inputs per model. Observed accuracy increased for nine of the 12 competing components:

    ModelCompeting component 1Accuracy change (pp)Competing component 2Accuracy change (pp)
    Qwen3.5-4BL31 MLP+3.48L31 Attention+2.04
    Qwen3.5-9BL31 MLP+0.61L28 MLP+0.10
    Qwen3.5-27BL63 MLP+2.52L63 Attention+1.15
    Gemma3-4BL33 MLP+0.78L31 Attention−0.21
    Gemma3-12BL47 MLP−0.40L44 MLP−0.13
    Gemma3-27BL53 MLP+0.08L60 MLP+0.50
  3. Interventions along the Δ-probe direction

    We also modified hidden states along the frozen Δ-probe direction, setting the decoded value to +2 or −2. We tested single interventions at the rise and final layers, as well as repeated interventions from the rise layer through the final layer. These interventions did not meaningfully improve final-answer accuracy under the tested settings(appendix E.7 line1651).

Label-bias calibration improves accuracy in all six models, and mean ablation of selected competing components improves accuracy on the evaluated intervention inputs. Interventions along the Δ-probe direction do not yield a consistent accuracy improvement.

Dear Reviewer td3y:

We would like to thank you for your thoughtful review and helpful suggestions.

Please let us know if you have any remaining questions about our responses. We truly appreciate this opportunity to further clarify our work and would be grateful for any additional feedback you may have.

Best regards,

Authors

Reviewer_YAowOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether spatial information linearly decodable from LLM hidden states can support more accurate route choices than the models' own answers, framed as a "Polanyi-like" gap between representation and decision.

Each input lists 3 to 8 textual travel "experiences" (paths through three locations) that jointly define a connected graph with 6 to 16 locations. The model must decide which of the start's two neighbours (options A and B) lies on the shorter route to a destination that never co-occurs with the start in a single experience. An eight-way matched design yields 115,362 inputs: a destination-name swap that reverses the correct answer while preserving topology, degree, and mention counts; two goals; and two option orders. Six frozen chat models (Qwen3.5 4B/9B/27B in non-thinking mode, Gemma 3 IT 4B/12B/27B) answer with a single token.

At the decision token, ridge probes with nested, topology-grouped cross-validation decode pairwise distances, adjacency, start-centred distances, and the signed route-cost difference Δ\Delta. Logit Lens tracks layerwise A/B margins. Natural counterfactual interventions replace either the whole decision-token state at one late layer (134,336 directed configurations per model) or single attention/MLP outputs selected by direct logit attribution (960 inputs, 4 components per model).

Reported findings: partial, size-dependent recovery of adjacency and distance (Qwen3.5 stronger than Gemma 3), with lower error for start-centred distances; an abrupt rise of Δ\Delta decodability at roughly 42 to 47 percent of network depth; final-layer probe accuracy above behavioural accuracy in all six models, by 5.4 to 16.6 percentage points; later (68 to 84 percent depth), non-monotonic and incomplete probe-output agreement (final 71.2 to 83.9 percent); transplanted-state effects that decay downstream in Qwen3.5 but grow in Gemma 3; and screened "supporting" and "competing" components whose swap effects have opposite signs.

Critical strengths

1. Well-controlled task and unusually careful evaluation protocol. The counterfactual history construction exchanges two destination names while preserving unlabelled topology, degree, and mention counts, and reverses the optimal move; together with goal and option-order variants this yields natural minimal pairs that are later reused for interchange interventions. Probing uses nested cross-validation grouped by structural graph hashes so that related histories and their variants never straddle train and test splits, with explicit weighting and group bootstrap intervals. The activation-collection, Logit Lens, and patching pipelines are numerically validated against native outputs, and interventions use natural counterfactual donors rather than arbitrary vectors. This is consequential because the paper's contribution is fundamentally metrological: the measurements can be trusted, and the dataset (19,364 histories, 115,362 inputs) and the probe-plus-lens-plus-intervention pipeline are directly reusable for other in-context reasoning tasks.

2. A consistent two-stage temporal dissociation, plus an unexplained family contrast, across two model families and three scales. Route-comparison information becomes linearly decodable at the decision token abruptly at roughly 42 to 47 percent of depth in all six models, while the native output readout aligns with it only at roughly 68 to 84 percent of depth, non-monotonically and incompletely. Whole-state transplants at a late layer influence the output in all models, but the effect decays downstream in Qwen3.5 (for example, 1.075 to 0.096 in the 9B model) and grows in Gemma 3 (1.238 to 5.701 in the 27B model). This "decodable early, expressed late and lossily" pattern is a crisp empirical anchor that mechanistic follow-up work can build on, and the family contrast is a genuinely puzzling observation worth explaining.

Critical weaknesses

1. The headline gap is close to expected by construction, and the analyses needed to interpret it are missing. The model's A/B decision is itself (up to a scalar normalisation) a fixed linear readout of the same final hidden state that the probe reads. A ridge probe fitted on tens of thousands of labelled in-distribution examples is therefore expected to match or exceed the model's own readout on held-out data; "probe beats behaviour in 6 of 6 models" is weak evidence that the model "knows" more than it uses. The informative objects are the size and the source of the gap, and several mundane sources are not examined: invalid outputs are scored as errors but their rate is not reported; letter bias and option-order consistency are not reported even though both option orders and the A/B logits were stored; and late-layer binding of the chosen location to its letter label is a known separate failure stage in multiple-choice settings. Most importantly, no experiment shows that the decodable information can be made to improve the model's own choices (for example, steering along the probe's Δ\Delta direction, ablating "competing" components, or debiasing the margin). The conclusion defers this to future work, but it is precisely the evidence the title claim requires. Finally, the framing analogy is inverted: Polanyi's tacit knowledge is visible in performance but resists articulation, whereas here an externally trained decoder exceeds the system's own performance. The closer and more area-appropriate frame is the decoder-exceeds-behaviour (suboptimal readout) dissociation familiar from systems neuroscience. Consequence: the central conceptual contribution currently reduces to a known and partly methodological phenomenon.

2. The spatial and mechanistic interpretations outrun the evidence; simpler explanations are not excluded. (a) The distance-recovery claims may be shallower than presented. The random baseline draws independent samples from the marginal distribution; a per-head intercept-only predictor would be considerably stronger, and each pair-specific or location-specific head can learn its own mean. The reported global-distance MAEs sit at roughly 0.65 of the sampling baseline (0.563 versus approximately 0.85 at 6 locations, and 1.523 versus approximately 2.3 at 16 locations, baseline values read approximately from Fig. 2a); the six models' distance curves nearly coincide while their adjacency curves differ clearly; Gemma 3 adjacency at 16 locations (0.148 to 0.155) sits barely above the random baseline, yet Gemma's Δ\Delta accuracy is comparable to Qwen3.5's, suggesting the decision need not rely on a map-like representation at this position at all; and the Fig. 3 distance profiles are essentially flat from the first blocks onward, unlike the abrupt mid-depth rise of Δ\Delta, which is consistent with surface statistics present from the embedding rather than a map computed through depth. (b) The start-centred advantage is confounded by the query format: the two options are exactly the start's neighbours (distance 1 by construction) and the destination is 2 to 4 steps away, so a role-based linear feature ("this name is an option", "this name is the destination") is available to per-location start-centred heads but not to fixed-pair global heads. A back-of-envelope estimate of this leakage is of the same order as the observed MAE difference, so the link to egocentric coding (Kunz et al., 2021) is premature without a comparison that excludes options and destination. (c) Integration is not demonstrated. Whenever the start-to-goal distance is 2, the correct option is necessarily adjacent to the destination and therefore co-occurs with it in a single experience, so a one-step lexical lookup solves all such queries; this rule answers all eight variants of the paper's own example (Table 2), and the name swap preserves mention counts but not co-occurrence. Without distance-stratified results and heuristic baselines, the mid-depth rise may mark when answer-relevant tokens are routed into the decision position rather than "cognitive emergence through integration of fragmented experiences". (d) The component results are partly entailed by selection: components were screened by the sign of their direct contribution relative to the probe preference, so sign-consistent swap effects at the intervened layer are expected by construction; in five of the six models the top-ranked competing component is the final-layer MLP, whose effect on the output is purely direct, and damping of the leading prediction by last-layer components is a generic calibration pattern in language models rather than something specific to spatial decision-making. Effects are reported in raw logits without normalisation or flip rates, whole-state patching uses a single exploratorily selected layer per model, and some analyses described in App. D.2 and D.5 (for example, downstream changes in Δ^\hat{\Delta}, strict preference flips) are not reported. Relatedly, Qwen3.5 is not a pure transformer: it interleaves Gated DeltaNet linear-recurrence layers with full-attention layers in a 3:1 pattern, and every Qwen3.5 attention component in Table 5 sits at a full-attention layer (indices 3 mod 4). The paper does not discuss how this bears on which layers produce the Δ\Delta rise or on the decay-versus-amplification contrast with Gemma 3. Consequence: the cognitive-science interpretations that motivate the framing and the venue are not yet supported.

Questions for authors

Analyses whose answers could materially change my assessment:

  1. Sources of the gap. Please report (a) invalid-output rates and A/B-restricted accuracy; (b) P(answer = A) and the consistency of the chosen physical location across the two option orders of each matched pair; (c) a calibrated behavioural baseline, for example the final-layer A/B margin with a single bias term fitted on the training folds; (d) the probe-correct by model-correct contingency, in particular the fraction of model errors that the final-layer probe gets right. How much of the 5.4 to 16.6 percentage-point gap remains after (a) to (c)?
  2. Can the decoded information be used? For example, add the Δ\Delta-probe direction to the hidden state at the decodability-rise layers or at late layers, mean-ablate the "competing" components, or apply label debiasing, and report the change in the model's own answer accuracy. A positive result would substantially strengthen the central claim; a null result would also be informative.
  3. Heuristics versus integration. What fraction of queries have start-to-goal distance 2, 3, and 4 at each environment size? Please stratify behavioural accuracy, Δ\Delta-probe accuracy, and probe-output agreement by distance and size, and report the accuracy of simple lexical rules (the option that co-occurs with the destination, or is written adjacent to it). Does the mid-depth rise occur at the same layers for distance-2 and distance-4 queries?
  4. Probe controls. Please add per-head intercept-only baselines and control-task (selectivity) results with randomised labels for all probes. Please repeat the start-centred versus global comparison excluding the two options and the destination from the targets. Why are the Fig. 3 profiles essentially flat from the first blocks onward, if distance information is computed rather than read off the input?
  5. Interventions. (a) Are the 960 component-intervention inputs disjoint from the 80 percent screening partition? (b) Please report normalised effects (for example, as a fraction of the donor-recipient margin difference) and flip rates, oriented also by ground truth and by the recipient's own output preference, together with a layer sweep for whole-state patching rather than a single layer per model. (c) Do the abrupt Δ\Delta rises coincide with Qwen3.5's full-attention layers (every fourth layer), and could the hybrid Gated DeltaNet architecture explain the decay-versus-amplification contrast in Table 6?
  6. Scope. Does the gap persist when Qwen3.5's thinking mode is enabled? If it closes, the phenomenon is better described as a limitation of single-pass answering than as tacit knowledge. Separately, behavioural accuracy is non-monotonic in scale within both families and the largest model shows the largest gap (16.6 points for Qwen3.5-27B): is the gap itself scale-dependent?

Minor (not affecting my score):

  • The local effect reported for Qwen3.5-27B L62 MLP (+2.272, Sec. 4.2) equals the Gemma3-12B first-downstream value in Table 6 to three decimals; please double-check.
  • Fig. 2 reports the best layer per model and size, a selection not reflected in the intervals.

Suggestion for future work (not affecting my score): a matched human behavioural experiment (accuracy as a function of route length and of the number of experiences that must be integrated) would make the cognitive-science contribution concrete, and would fit this area well.

13 comments

Overall response

Thank you for your constructive feedback. Your comments motivated additional analyses of label bias, shallow input cues, probe selectivity, intervention effectiveness, architecture, and thinking mode. The revised main text now includes the core probe controls, matched target comparisons, distance-stratified curves, label calibration, layerwise intervention effects, component mean ablation, and thinking results. We have uploaded a revised PDF, with major changes highlighted in blue. We address your concerns and questions below.

Responses to Weaknesses

W1 — Sources and interpretation of the gap

(a) To examine the sources of the gap, we added invalid-output statistics, analyses of label bias and option-order consistency, a calibrated behavioral baseline, and probe–answer correctness contingency tables. Invalid-output rates are 0% in all six models. Label-bias calibration improves answer accuracy in every model but leaves a gap of 4.72–9.38 percentage points relative to the Δ probe. Detailed methods and results are provided in Q1.

(b) To test whether final decisions can be improved, we evaluated answer-label bias calibration and mean ablation of identified competing components. Calibration improves accuracy in all six models, while mean ablation increases observed accuracy for nine of the twelve competing components. Detailed methods and results are provided in Q2.

(c) Thank you for pointing out this conceptual distinction. Our intended analogy treats spatial information that is recoverable by probes but not fully reflected in the model’s final choices as an analogue of tacit knowledge. The failure to translate this information into a correct final answer is, in turn, analogous to a limitation in expressing that knowledge through the model’s response. We agree that our setting differs from Polanyi’s original distinction between knowing and articulating. Our original use of “Polanyi-like” was intended to mark this as an analogy, rather than equate the two phenomena. Following the reviewer’s suggestion, we have added a brief discussion of decoder-exceeds-behavior dissociations and suboptimal-readout accounts (line 459).

W2 — Sources and interpretation of the gap

(a) We added intercept-only baselines, shuffled-label controls, and input-feature comparisons. The apparent flatness partly reflects the original plotting scale, which accommodated the substantially higher errors of the random-sampling baseline. Revised Figure 2(line 172) makes the early-layer decrease in global-distance MAE and increase in adjacency Jaccard more visible. Some early recoverability can be explained by input statistics and query-role constraints, but the start-centered advantage remains after excluding candidate and destination targets and retraining the probes (Figure3 line 187). Detailed methods and results are provided in Q4.

(b) To assess the contribution of query-role constraints, we compared start-centered and global probes on identical start-to-location targets after excluding the two candidates and the destination. The maximum relative MAE reduction across layers decreases from 26.4–28.8% with all targets to 11.1–14.6% after exclusion and retraining. Thus, query-role targets contribute to the observed advantage, but the advantage persists on the remaining distances. Revised Figure 3 (line 187) presents these comparisons, with detailed methods and results provided in Q4.

(c) We added distance-stratified analyses and co-occurrence and adjacency-rule baselines. The adjacency rule solves all distance-2 queries. For distance-4 queries, however, both rules achieve 50% accuracy, while final-layer Δ-probe accuracy, averaged across environment sizes 8–16, ranges from 56.43% to 74.53% across models. Thus, these single-experience lookup rules do not fully explain the probe’s performance. Revised Figure 4(line 226) also shows the mid-depth rise separately for distances 2, 3, and 4. Detailed methods and results are provided in Q3.

(d) We added normalized effects, answer flip rates, a full layer sweep for whole-state patching, and analyses of downstream component effects(figure6 line 335). The results show that some effects persist to final choices, while others weaken, and that ablating selected competing components improves accuracy. Detailed intervention methods and results are provided in Q2 and Q5.

Responses to Questions

Q1 — Sources of the probe–behaviour gap

(a) All models have a 0% invalid-output rate. Restricting the final choice to A/B leaves all predictions and accuracies unchanged. As noted in the manuscript footnote, output-format compliance also motivated our choice of Gemma 3: preliminary checks showed that Gemma 4 did not consistently produce A or B as the first generated token under the same answer-format instruction.

(b) We anticipated that model choices could be sensitive to answer labels and option order. This motivated our inclusion of option-swapped pairs in the dataset, keeping the experiences, start, goal, and candidate locations fixed while reversing their A/B assignments. The table below reports the proportion of answers selecting A and the consistency of the chosen physical location across the two option orders.

ModelP(answer = A)Physical-choice consistency
Qwen3.5-4B64.59%68.70%
Qwen3.5-9B42.82%80.17%
Qwen3.5-27B71.29%57.38%
Gemma3-4B72.47%54.95%
Gemma3-12B60.48%74.87%
Gemma3-27B69.62%58.87%

(c) Following the reviewer’s suggestion, we added a bias term to the model’s final-layer A/B logit margin to calibrate its overall answer-label preference. Using the original five-fold splits, we fitted a separate bias for each model, environment size, and training fold to maximize weighted training accuracy, then fixed the bias and evaluated it on the corresponding test fold. Model parameters and Δ probes remained unchanged. Calibration improves answer accuracy in all six models, reducing the original 5.4–16.6 percentage-point gap to 4.72–9.38 percentage points relative to the final-layer Δ probe.

ModelOriginal answer accuracyCalibrated accuracyRemaining gap (pp)
Qwen3.5-4B72.14%74.31%6.68
Qwen3.5-9B78.27%79.00%4.72
Qwen3.5-27B73.90%82.20%8.30
Gemma3-4B67.29%70.34%8.50
Gemma3-12B74.39%75.29%8.43
Gemma3-27B70.12%74.71%9.38

(d) We cross-tabulate the correctness of the original model answers and final-layer Δ-probe predictions. The four joint-outcome columns show weighted percentages of all inputs; the last column reports the percentage of model errors for which the Δ probe is correct.

ModelModel correct: Probe correctModel correct: Probe incorrectModel incorrect: Probe correctModel incorrect: Probe incorrectProbe correct among model errors
Qwen3.5-4B65.48%6.66%15.52%12.34%55.70%
Qwen3.5-9B72.70%5.57%11.02%10.71%50.71%
Qwen3.5-27B70.62%3.28%19.89%6.22%76.18%
Gemma3-4B58.57%8.73%20.27%12.43%61.98%
Gemma3-12B68.18%6.20%15.55%10.07%60.69%
Gemma3-27B63.70%6.41%20.38%9.50%68.20%

As reported in (a)–(c), invalid outputs do not contribute to the gap, and label-bias calibration reduces the original 5.4–16.6 percentage-point gap to 4.72–9.38 percentage points.

Q2 — Improving final-answer accuracy

We evaluated three approaches: answer-label bias calibration, mean ablation of identified competing components, and interventions along the Δ-probe direction.

(a) As reported in Q1(c), we fitted an additive bias to the final A/B logit margin on the training folds and evaluated it on the corresponding test folds. Calibration improves final-answer accuracy in all six models, although gaps of 4.72–9.38 percentage points remain relative to the Δ probe.

(b) For each identified competing component, we replaced its residual contribution at the final input token with its mean over the original training families, weighting families equally. Components were ablated individually on the same 960 intervention inputs per model. Observed accuracy increases for nine of the twelve competing components. These results show that ablating selected competing components can improve final-answer accuracy on the evaluated inputs.

ModelCompeting component 1Accuracy change (pp)Competing component 2Accuracy change (pp)
Qwen3.5-4BL31 MLP+3.48L31 Attention+2.04
Qwen3.5-9BL31 MLP+0.61L28 MLP+0.10
Qwen3.5-27BL63 MLP+2.52L63 Attention+1.15
Gemma3-4BL33 MLP+0.78L31 Attention−0.21
Gemma3-12BL47 MLP−0.40L44 MLP−0.13
Gemma3-27BL53 MLP+0.08L60 MLP+0.50

(c) We directly modified hidden states along the frozen Δ-probe direction, setting the decoded value to +2 or −2, favoring A or B, respectively. On the same 960 inputs per model, we tested a single intervention at the probe’s rise layer, a single intervention at the final layer, and repeated interventions from the rise layer through the final layer using each layer’s own probe. Accuracy changes are small and vary across models, intervention schedules, and target directions, without a consistent improvement. The table reports changes in final-answer accuracy, with each cell ordered as target +2 / target −2.

ModelRise-layer interventionFinal-layer interventionRepeated interventions
Qwen3.5-4B+0.22 / +0.73+0.42 / −0.10+0.02 / +0.19
Qwen3.5-9B+0.94 / +0.730.00 / +0.10+0.31 / +0.31
Qwen3.5-27B−0.10 / +0.420.00 / 0.00−0.10 / −0.09
Gemma3-4B−0.21 / −0.420.00 / −0.21−0.62 / −0.31
Gemma3-12B−0.21 / +0.09−0.10 / −0.10−0.14 / 0.00
Gemma3-27B−0.13 / +0.10−0.13 / −0.13−0.23 / −0.13
Reviewer_6RXwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether LLMs' internal representations contain linearly decodable spatial information in a synthetic route-choice task, and whether route preferences decoded from hidden states outperform the models' final choices. The model must pick which of two neighbouring locations is the first move on the shortest route to a goal, given short two-step experiences over unicyclic graphs of 6 to 16 locations (with matched variants). On Qwen3.5 and Gemma 3 (4B to 27B) they train linear probes on final-token hidden states for: 1. pairwise distances and adjacency, 2. start-centered distances, 3. route-cost differences between the two options. They compare what the model knows (probe preference) with what it says (Logit-Lens preference per layer, and final answer), and patch whole states and individual attention/MLP outputs. Contributions: global spatial structure is partially recoverable, start-centered distances are easier to recover than global ones, the probe is more accurate than the final answers in every model (the "Polanyi-like representation-decision gap"), and patching shows components pushing the A/B logits in opposite directions, with effects that often weaken or reverse in later layers. Validation relies on grouped nested cross-validation, group-level bootstrap CIs, random baselines, and numerical checks of the intervention pipeline. Overall, this is a solid work that needs polishing.

Critical strengths
  • The paper compares what a probe can recover from hidden states at each layer (route preference, distances) with what the model actually outputs at the last layer, and tracks where the two diverge across depth. This turns "does the model know it?" into a measurable question, and the finding that decodable route information appears at ~42-47% depth while agreement with output preferences builds up later is a useful way to localize where the gap emerges. It matters because it goes beyond "probe > output" and provides a template for diagnosing failures of use rather than failures of representation

  • Matched counterfactual variants (history, goal, option order) with a unique correct answer, grouped cross-validation to avoid leakage, and group-level bootstrap CIs make the probing results make the probing results more credible, and the appendix is transparent about exploratory choices.

Critical weaknesses
  • Logit Lens is reliable mainly in the last layers, yet it is used as a verbalization signal across depth (agreement curves, Fig. 5-6, and the intermediate margins in Eq. 4). The fluctuations in output preferences (from line 457) are therefore not a reliable indication: the probe could be right and the lens wrong, or the probe could be poorly trained, and the paper cannot tell which. A tuned lens or a sanity check would be needed.

  • Fig. 2 is hard to read and the differences are not distinguishable; Fig. 3 is mentioned as a comparison with Fig. 2 but is not comparable to it; averaged curves (especially Fig. 3) should report standard deviations; Fig. 6 is unreadable. Since the main claims rest on layerwise curves, this weakens the evidence.

Questions for authors
  • Lines 257-264: is the recovery drop at larger environments related to hidden dimension or probe capacity? Qwen-4B and Gemma-4B share d=2560 but differ strongly in adjacency Jaccard (0.240 vs ~0.15), and Gemma-27B has larger d than Qwen-27B but much lower Jaccard.

  • Lines 306-307: are the two candidates always one step from the start by construction? Please state it in the main text. What does CE in Eq. 4 stand for?

  • Do the fluctuations in output preferences survive a tuned lens, or a probe trained directly on the output margin?

  • The related work frames the setting as in-context (representation) learning, but the task has no demonstrations and uses short contexts (3 to 8 two-step experiences). In what sense is this in-context learning rather than retrieval/reasoning over the context? Would the conclusions change under the second interpretation?

2 comments

Overall response

Thank you for your positive assessment and constructive feedback. We have added Tuned Lens calibration, stronger probe controls, and whole-state intervention sweeps across layers, and redrawn the relevant figures. We have uploaded a revised PDF, with major changes highlighted in blue. We address your concerns and questions below.

Responses to Weaknesses

W1 — Logit Lens reliability

Thank you for pointing out this limitation. We agree that uncalibrated Logit Lens readouts alone are insufficient to establish reliable fluctuations in intermediate output preferences. We therefore added Tuned Lens calibration and additional probe controls. Calibration reduced, but did not eliminate, repeated A/B preference reversals or disagreement with the Δ probe. The Tuned Lens training procedure and detailed results are provided in Q4.

W2 — Figure readability, comparability, and uncertainty

We have redrawn Figures 2, 3, and 6(line172, line 187 and line335) to improve readability and comparability. Figure 3 now directly compares start-centered and global probes on identical prediction targets. Figures 2 and 3 include pointwise 95% family-bootstrap intervals to show uncertainty, while Figure 6 separates normalized effects and flip rates into distinct panels.

Responses to Questions

Q1 — Hidden dimension and probe capacity

Hidden dimensionality alone cannot explain these differences. As the reviewer notes, the two 4B models share the same hidden dimension, while Gemma3-27B has a larger hidden dimension than Qwen3.5-27B but weaker adjacency recovery. We train probes independently for each model, environment size, and layer. Each global-distance or adjacency target also has its own independently fitted linear head. Increasing the environment size therefore increases the number of prediction heads, rather than requiring a shared head to accommodate more targets. The revised Figure 2 presents the complete layerwise results under the same probing protocol. Qwen3.5 generally exhibits lower global-distance errors and stronger early-layer adjacency recovery than Gemma 3, indicating model-family differences in the linear recoverability of spatial information.

Q2 — Candidate adjacency to the start

Yes. Both candidates are guaranteed by construction to be one step from the start. We have clarified the task definition by explicitly referring to them as “the two neighbors of the starting location.”(line125)

Q3 — Definition of CE

CE stood for “causal effect” and measured the intervention-induced change in the A/B logit margin, oriented by the donor’s original final-layer Δ preference. To communicate intervention magnitudes and their effects on choices more clearly, the revised main text reports normalized effects (NE), relative to the natural donor–recipient margin difference, together with answer flip rates. We no longer use the abbreviation CE.

Q4 — Output-preference fluctuations after Tuned Lens calibration

Yes. Repeated A/B preference reversals persist after Tuned Lens calibration, although their frequency decreases. Following your suggestion, we trained layerwise affine translators using the Tuned Lens approach, minimizing KL divergence to match the frozen model’s final output distribution. Training lasted five epochs, with family-disjoint training, validation, and test sets. To quantify the remaining fluctuations, we measured the proportion of inputs with at least two A/B preference switches within the analyzed late-layer window. We also measured how often the calibrated readout favored the opposite option to the same-layer Δ probe, to assess whether calibration resolved their disagreement.

ModelRepeated reversals: Logit LensRepeated reversals: Tuned LensTuned Lens–Δ opposition
Qwen3.5-4B53.39%23.87%21.86%
Qwen3.5-9B27.33%3.59%17.36%
Qwen3.5-27B42.08%8.04%22.32%
Gemma3-4B17.74%4.92%29.97%
Gemma3-12B24.80%5.95%22.29%
Gemma3-27B46.06%14.00%27.69%

Calibration reduces repeated reversals in all six models, but 3.59–23.87% of inputs still exhibit them. The calibrated readouts also remain opposed to the same-layer Δ probes at rates of 17.36–29.97%, averaged over the analyzed nonfinal layers. Thus, calibration reduces the observed fluctuations but eliminates neither repeated preference reversals nor disagreement with the Δ probe.

Q5 — In-context representation learning

We use “in-context representation learning” following the perspective of Park et al. (2025), who provide models with random-walk traces and examine how internal representations reflect the graph structure specified in context. Here, “learning” concerns the formation of context-specific structural representations without parameter updates and does not require question–answer demonstrations. Similarly, we provide frozen models with 3–8 interconnected spatial experiences specifying the current environment’s connectivity. We use distance, adjacency, and route-comparison probes to assess the decodability of this relational information from hidden states. We further examine the extent to which this context-derived spatial information is reflected in the model’s final goal-directed decisions. Our terminology therefore emphasizes context-dependent internal representations and their relationship to decisions, extending the focus beyond task-level retrieval or reasoning.

Interpreting the task as retrieval/reasoning over the context would not change our central empirical conclusion: under direct-answer conditions, linearly decoded route preferences are more accurate than the model’s final choices. This finding concerns the relationship between decodable information and decisions and does not depend on classifying the task as in-context learning.

References

[1] Park, C. F., Lee, A., Lubana, E. S., Yang, Y., Okawa, M., Nishi, K., Wattenberg, M., & Tanaka, H. (2025). ICLR: In-context learning of representations. In The Thirteenth International Conference on Learning Representations.

Comments on the paper

1 comment

Dear reviewers,

The authors have now responded to your comments/critiques. As we are nearing the end of the discussion period, please read through the authors' rebuttal and respond if there are points that remain unclear (or to let them know they have addressed your concerns).

Sincerely,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.