- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether spatial information recoverable from LLM hidden states is reflected in the models' final choices. It uses a synthetic route-choice task. Models must integrate textual travel experiences and choose the next move that gives the shorter path to a destination. Linear probes recover spatial distances and route-cost differences across layers. These probe predictions are compared with Logit Lens preferences and with the final answers. Activation patching is then used to study how later computation changes these preferences. Across six Qwen3.5 and Gemma 3 models, route-comparison probes outperform final-answer accuracy by 5.4 to 16.6 points. The paper interprets this difference as a representation–decision gap.
Critical strengths
-
The task design is carefully controlled. Matched variants change the history, destination, and option order while preserving much of the surface form. This enables clean counterfactual comparisons and supports the intervention experiments.
-
The experimental execution is strong. The results are consistent across six models from two families, and the probing and intervention analyses are applied systematically across layers.
Critical weaknesses
-
The gap between the supervised probe and model behavior is difficult to interpret. The route probe is trained on thousands of labeled examples with direct supervision on the target quantity, while the model answers the task zero-shot. A supervised linear readout outperforming zero-shot behavior does not by itself show that the model has computed task-relevant information that it fails to use. The near-chance probe accuracy in early layers suggests that the probe is not simply decoding the input. Even so, this observation does not establish that the decodable signal is information the model itself relies on, rather than structure that the supervised readout extracts from intermediate features. Additional controls would help separate these explanations. Examples include shuffled-label probes, randomly initialized models, and probes over simple input-derived features. This concern is especially relevant for the start-centered probes, which already perform substantially better than the random baseline from the earliest layers and change relatively little with depth.
-
Prior work already shows that information decoded from hidden states can be more accurate than model outputs, and the paper positions itself as extending this observation to spatial decision-making. The intervention results show that some hidden states and components influence output preferences. However, they do not explain the observed probe–behavior gap of 5.4 to 16.6 points. The paper also does not test whether the decoded signal can actually improve final decisions, or whether the gap persists under stronger reasoning settings such as thinking mode.
Questions for authors
-
How do the route-comparison probes perform with shuffled labels, with randomly initialized models, or against simple input-feature baselines?
-
Why are start-centered distances already strongly recoverable in the earliest layers, and how much of this can be explained by shallow properties of the input?
-
Does the probe–behavior gap persist when Qwen3.5 is evaluated in thinking mode or with explicit reasoning?
-
Can the probe signal or the identified components be used to improve final-answer accuracy?
3 comments
Q4 — Improving final-answer accuracy with probes or components
We tested three approaches: answer-label bias calibration, mean ablation of identified competing components, and interventions along the Δ-probe direction.
-
Answer-label bias calibration
We fitted an additive bias to the final A/B logit margin on the training folds and evaluated it on the corresponding test folds, keeping model parameters and Δ probes unchanged. Calibration improves final-answer accuracy in all six models, although gaps of 4.72–9.38 percentage points remain relative to the final-layer Δ probe.
Model Original accuracy Calibrated accuracy Δ-probe accuracy Remaining gap (pp) Qwen3.5-4B 72.14% 74.31% 81.00% 6.68 Qwen3.5-9B 78.27% 79.00% 83.72% 4.72 Qwen3.5-27B 73.90% 82.20% 90.50% 8.30 Gemma3-4B 67.29% 70.34% 78.84% 8.50 Gemma3-12B 74.39% 75.29% 83.73% 8.43 Gemma3-27B 70.12% 74.71% 84.09% 9.38 -
Mean ablation of competing components
For each identified competing component, we replaced its residual contribution at the final input token with its mean over the original training families. Components were ablated individually on the same 960 intervention inputs per model. Observed accuracy increased for nine of the 12 competing components:
Model Competing component 1 Accuracy change (pp) Competing component 2 Accuracy change (pp) Qwen3.5-4B L31 MLP +3.48 L31 Attention +2.04 Qwen3.5-9B L31 MLP +0.61 L28 MLP +0.10 Qwen3.5-27B L63 MLP +2.52 L63 Attention +1.15 Gemma3-4B L33 MLP +0.78 L31 Attention −0.21 Gemma3-12B L47 MLP −0.40 L44 MLP −0.13 Gemma3-27B L53 MLP +0.08 L60 MLP +0.50 -
Interventions along the Δ-probe direction
We also modified hidden states along the frozen Δ-probe direction, setting the decoded value to +2 or −2. We tested single interventions at the rise and final layers, as well as repeated interventions from the rise layer through the final layer. These interventions did not meaningfully improve final-answer accuracy under the tested settings(appendix E.7 line1651).
Label-bias calibration improves accuracy in all six models, and mean ablation of selected competing components improves accuracy on the evaluated intervention inputs. Interventions along the Δ-probe direction do not yield a consistent accuracy improvement.
Dear Reviewer td3y:
We would like to thank you for your thoughtful review and helpful suggestions.
Please let us know if you have any remaining questions about our responses. We truly appreciate this opportunity to further clarify our work and would be grateful for any additional feedback you may have.
Best regards,
Authors
Overall response
Thank you for your constructive feedback. Your comments motivated additional probe controls, analyses of shallow input cues, tests of whether interventions improve final answers, and a matched thinking-mode comparison. We have uploaded a revised PDF, with major changes highlighted in blue. We address your concerns and questions below.
Responses to Weaknesses
W1 — Probe supervision, controls, and shallow input cues
We added intercept-only baselines, shuffled-label controls, input-feature baselines, and target-exclusion analyses. The results show that final-layer Δ probes outperform shuffled-label and basic-feature controls. We therefore interpret the probe results as evidence that task-related information is linearly recoverable. Detailed methods and results are provided in Q1 and Q2.
W2 — Explaining the gap and improving decisions
We added answer-label bias calibration, competing-component mean ablation, Δ-direction interventions, and a matched thinking-mode comparison. Calibration reduces but does not eliminate the gap, and ablating some competing components improves final-answer accuracy. Δ-direction interventions do not establish a consistent accuracy improvement. Thinking eliminates the gap on the successfully completed matched inputs across all three Qwen3.5 models. Detailed methods and results are provided in Q3 and Q4.
Responses to Questions
Q1 — Shuffled labels, random initialization, and input-feature baselines
We added shuffled-label controls and a linear input-feature baseline. For the shuffled-label controls, we kept hidden states and data splits fixed, shuffled targets within partitions while preserving matched A/B swaps, and retrained probes using three random seeds. The input-feature baseline uses location mention counts, first and last occurrence positions, and indicators identifying the start, destination, option A, and option B.
Final-layer Δ probes outperform both controls in all six models. Revised Figure 4 (line226) includes the full layerwise comparisons, with methodological details in the appendix.
Q2 — Early-layer start-centred distance recovery
The apparent flatness partly reflects the original plotting scale: including the substantially higher errors of the random-sampling baseline expands the vertical range, making changes in the probe curves less visible. Revised Figure 2 (line 172) makes the early-layer decrease in global-distance MAE and increase in adjacency Jaccard more visible.
Some early distance recoverability can also be explained by input statistics and query-role constraints. However, after excluding the two candidates and the destination from both training and evaluation and retraining the probes, start-centered probes retain an advantage over global probes on identical targets, with maximum relative MAE reductions of 11.1–14.6% (revised Figure 3 (line 187)).
Q3 — Thinking mode and explicit reasoning
Given the computational cost of generating long reasoning traces in thinking mode, we used the same 960 inputs as in the intervention experiments to compare Qwen3.5-4B, 9B, and 27B in thinking and non-thinking modes(line399). We excluded 44 truncated generations for 4B and nine for 9B from both modes in the paired comparison. For completed responses that did not begin with A/B (208 for 4B, one for 9B, and none for 27B), we scored the explicitly stated choice in the full final response.
At the position sampled after thinking, Δ information is already highly recoverable in early layers, with first-layer direction accuracy ranging from 88.18% to 91.16%. At the final layer, both probe accuracy and final-answer accuracy reach 100% in all three models on the completed inputs, reducing the probe–behavior gap to zero(Figure7-line 432).
Thinking provides additional reasoning steps before the final choice. As illustrated by the examples in the appendix F (line 1735), models explicitly enumerate candidate routes, count and compare their lengths, and then select an answer. These step-by-step calculations provide intermediate results that can support the final decision, offering a plausible explanation for the improved accuracy and the disappearance of the gap on the retained matched inputs.