- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether language models represent properties of a system more reliably internally than in their output. Specifically, they study a setting with 1,200 Python request handlers that a model is given, and it is asked to find capacity bottlenecks. They find that a simple linear probe on the model's internal activations relatively reliably predicts the bottleneck, while when checking the model outputs, the model gets it right at a much lower rate. In more detail, they find that the middle layers predict this best. Furthermore, steering along the probe direction is not stable.
Critical strengths
The setup of the Python request handlers is well-designed to elicit whether a model internally represents a property that has to be deduced in multi-step reasoning (i.e., it has to be worked out based on multiple numbers the code specifies).
Critical weaknesses
Generally, I believe the paper does not offer significant new insights. Generating a new probing task, comparing probes across scales and families, and comparing internal decodability with the model's output are things that have been done relatively often in and I do not see how this work offers significant insight beyond that.
Questions for authors
- It seems to me like there is a potential issue that the probe is trained on labeled instances from the distribution we are testing on, while the direct output test is zero-shot. Do I understand this correctly? Is this an issue? Have you thought about testing few-shot prompting?
- Furthermore, the probe might just directly be doing work that the model struggles with internally. How do you view this?
2 comments
Remaining questions on the new controls
Dear Reviewer 7PYj, if you have a moment before the discussion closes, we would be glad to clarify any of the new experiments: the read-outs of the answer with the probe's supervision, whose gap to the probe widens to 29.6 points at 7B (Table 4), or the random-weight, bag-of-tokens and cost-shuffled controls, which show that the trained network, not the probe, does the computation (Table 7, Figure 2b). Thank you again for your time.
Response to Reviewer 7PYj
Thank you for the constructive review and questions.
Novelty. Most work showing that internal states know more than outputs studies targets the model may have stored: truthfulness, factual knowledge and hallucination signals (Burns et al., 2023; Kadavath et al., 2022; Liu et al., 2023; Orgad et al., 2025; Gekhman et al., 2025). The closest studies use numbers in the input: probes recover the sum of three-digit additions whether or not the answer is right (Sun et al., 2025) and rank two numerals better than the model says (Yuchi et al., 2026). All are now cited and discussed. Our target composes several printed quantities by an exact law, a product per stage and then a maximum over the stages that hold a server, and this allows three analyses that the revision now reports:
csno cost-free rule or linear model exceeds 48.3%, yet the 7B answer-token probe reaches 67.9% and randomly initialized networks at most 40.8%, so a probe above that level must use the printed costs.Q1 (supervision of the two read-outs). Yes, the submission had this asymmetry, and the revision removes it in three ways (Tables 4 and 9): three-shot prompting with solved training instances gives 21.2% at 7B on
cs, a logistic read-out of the answer logits trained on the same 720 labels as the probe gives 38.3%, and LoRA fine-tuning on those 720 instances gives 92.1%. The first two leave the model unchanged and stay far below the probe at the same token (67.9%). LoRA retrains the model, so it is not a read-out of the pretrained representation; it shows that the answer format can express the label, so the gap is not an artifact of the read-out. Below 7B, an adapter trained onorigreaches only 34.6–50.4% oncs, so fine-tuning, too, takes the cost-free shortcut when the training data allow it.Q2 (whether the probe does the computation). Four controls address this. (1) The same probes on the same architectures with random weights reach 34.6–40.8% on
csat the answer token (Table 7), below the best cost-free baseline (48.3%), against 56.2% at 1.5B and 67.9% at 7B for the trained networks, and Figure 2b shows the random 7B network below that baseline at every layer; the trained network computes the composition. (2) Cost-free rules and linear models reach at most 48.3% oncs, and paired with the best of them on the same 240 instances the answer-token probe is ahead at 1.5B, 3B, 7B and 14B (exact McNemar test, one-sided p=0.02, 0.03, 2×10−7 and 2×10−4), as is the probing-prompt probe of DeepSeek-Coder and StarCoder2 (p=0.04 and 0.006). (3) The probe trained on shuffled labels reaches 16.2–20.4% across the Qwen scales (18.3% at 7B), against a majority class of 22.5%. (4) The bag-of-tokens probe, trained on the mean input embedding of the code tokens, sees every token but no contextual computation and reaches only 31.2% oncs. Finally, adding the class-mean direction at the answer token moves the answer toward the chosen stage in eight of ten model–set pairs, so this direction of the probed representation is causally connected to the answer.We would be glad to answer further questions during the discussion.