- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a probe trained on a language model's hidden states can be reused as a decision signal in an agent loop. In a file workflow, an external controller thresholds a coverage score to choose TEST or PUBLISH. In a prespecified Qwen3.8-27B confirmation, adding a successful test of another file lowers a feedback-trained probe's pairwise accuracy from 0.984 to 0.428, while the model's output score keeps 0.988. Feedback training repairs repeated-test failures in Qwen and Gemma-4-31B-IT, and activation edits along probe directions move output scores but rarely flip choices.
Critical strengths
- Ranking and decisions are separated clearly. The model output score keeps 0.988 pairwise accuracy under the suffix yet classifies only 53.9 percent of inputs correctly at its old threshold, a useful reminder that ranking metrics alone do not validate a decision rule.
Critical weaknesses
- The central result is expected. Appending another file's test changes the probe's input, and a fixed classifier may or may not keep its ranking, just as results change with layer or reader. The paper's own results vary this way without a rule: a block-63 refit keeps 0.912, the +History-controls centroid keeps 0.914, and on write-order histories Qwen's output beats its probes while Gemma's probes beat its output.
- Why this is worth studying is not shown. The paper names no agent that chooses actions from a probe score, and says the task "does not establish their frequency in real software projects" and that "the tracker makes internal activations unnecessary in this task". Both models' own generated actions also complete all 64 live cases.
- Novelty is limited. Prior work already shows that probe accuracy does not imply the model uses the information (Amnesic Probing, https://arxiv.org/abs/2006.00995), that probes fail to generalize and can disagree with model outputs (Levinstein and Herrmann, https://arxiv.org/abs/2307.00175; Orgad et al., https://arxiv.org/abs/2410.02707), and how probe monitors behave in realistic settings (Goldowsky-Dill et al., https://arxiv.org/abs/2502.03407). None of these is discussed.
Questions for authors
- Is there an agent system in which a probe score chooses actions, and how would these results change its design?
- Why was the confirmation run on the +Feedback probe, already known to fail under this suffix, rather than on the +History-controls readers?
- How do the results change across a representative set of layers, reader types and both models?
Thanks the authors for detailed rebuttal, it solves some of my concerns. I’ll raise my score from 1 to 2.