acceptodds
Under review as a conference paper at ICLR 2027

Can Probes Reliably Guide Agent Decisions? Evidence from Controlled File Workflows

Abstract

Can a probe trained on a language model's internal state reliably guide an agent's next action? Not necessarily, even when the task state remains readable. We study a file task: test the current contents, or publish them if a successful test already applies. This setting lets us change the visible history without changing the correct decision. In a confirmation on 32 new file-roster families, testing another file reduces a fixed, feedback-trained Qwen probe's pairwise accuracy from 0.984 to 0.428, while the model's output scores retain 0.988. Both prespecified tests pass Holm correction. An exploratory centroid trained only on initial histories achieves 0.996 at the same layer and token position: the distinction remains linearly accessible there. Ranking is not enough, however: with the added test, the output score classifies only 53.9 percent of inputs correctly at its old threshold. Workflow experiments in Qwen and Gemma show why the reuse question matters: feedback training repairs repeated decisions, but the Qwen confirmation exposes a limit of that repair. Readable task state need not yield a reusable decision rule. The controlled task separates failures of ranking, thresholds, and feedback handling; it does not establish their frequency in real software projects.

Primary area:
interpretability and explainable AI
Keywords:
probing, interpretability, state representations, transfer, linear probes, Jacobian lens, world models, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jeDFOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a probe trained on a language model's hidden states can be reused as a decision signal in an agent loop. In a file workflow, an external controller thresholds a coverage score to choose TEST or PUBLISH. In a prespecified Qwen3.8-27B confirmation, adding a successful test of another file lowers a feedback-trained probe's pairwise accuracy from 0.984 to 0.428, while the model's output score keeps 0.988. Feedback training repairs repeated-test failures in Qwen and Gemma-4-31B-IT, and activation edits along probe directions move output scores but rarely flip choices.

Critical strengths
  1. Ranking and decisions are separated clearly. The model output score keeps 0.988 pairwise accuracy under the suffix yet classifies only 53.9 percent of inputs correctly at its old threshold, a useful reminder that ranking metrics alone do not validate a decision rule.
Critical weaknesses
  1. The central result is expected. Appending another file's test changes the probe's input, and a fixed classifier may or may not keep its ranking, just as results change with layer or reader. The paper's own results vary this way without a rule: a block-63 refit keeps 0.912, the +History-controls centroid keeps 0.914, and on write-order histories Qwen's output beats its probes while Gemma's probes beat its output.
  2. Why this is worth studying is not shown. The paper names no agent that chooses actions from a probe score, and says the task "does not establish their frequency in real software projects" and that "the tracker makes internal activations unnecessary in this task". Both models' own generated actions also complete all 64 live cases.
  3. Novelty is limited. Prior work already shows that probe accuracy does not imply the model uses the information (Amnesic Probing, https://arxiv.org/abs/2006.00995), that probes fail to generalize and can disagree with model outputs (Levinstein and Herrmann, https://arxiv.org/abs/2307.00175; Orgad et al., https://arxiv.org/abs/2410.02707), and how probe monitors behave in realistic settings (Goldowsky-Dill et al., https://arxiv.org/abs/2502.03407). None of these is discussed.
Questions for authors
  1. Is there an agent system in which a probe score chooses actions, and how would these results change its design?
  2. Why was the confirmation run on the +Feedback probe, already known to fail under this suffix, rather than on the +History-controls readers?
  3. How do the results change across a representative set of layers, reader types and both models?
1 comment

Thanks the authors for detailed rebuttal, it solves some of my concerns. I’ll raise my score from 1 to 2.

Reviewer_EWjmOpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies whether probes trained on frozen LLM hidden states can reliably guide decisions to publish a file or continue to test it based on edit-test iterative context. It finds that training on feedback histories fixes repeated-testing failures, but the context of an unrelated file test can substantially disrupt a Qwen probe. However, the disruption is not significant on the Gemma probe.

Critical strengths
  1. The controlled task makes the central comparison clear. Exact file contents and successful-test records determine the correct answer, allowing the authors to change irrelevant history while keeping the correct answer unchanged.

  2. The evaluation separates several meanings of reliability. Comparing pairwise ranking, accuracy at a fixed threshold, and workflow completion reveals failures that a single prediction-accuracy metric would conceal.

Critical weaknesses
  1. The practical relevance remains limited. The task excludes real-world test file generation, file system exploration, iterative failed-test repair, and dependencies between files. Moreover, the small direct-generation experiment shows no repeated-testing failures, leaving unclear how much the probe system’s difficulties matter for ordinary LLM agents.

  2. The strongest negative result has narrow scope. The main confirmation concerns one Qwen model, a simple one-layer feedback-trained logistic probe, and a specific irrelevant-history change, whereas Gemma preserves pairwise ordering in the earlier experiment and an alternative Qwen readout performs strongly. These results establish a specific failure case but do not establish how widespread or avoidable it is across reasonable probe configurations.

  3. The experiments identify the failure more clearly than its cause. Readers learn that feedback training can help and later fail, but receive limited guidance on why it fails or how to build a reliably better probe.

Questions for authors
  1. The authors should better clarify their motivation and significance (impact) of the paper in the Introduction section. For example, why should we use the probe to make decisions instead of directly using the LLM output logit(yes)-logit(no)? The experiment in the paper already shows LLM direct output is more robust to irrelevant context. This makes the readers feel that the major finding of the paper is that probe is not a good classifier compared to direct LLM output, making the meaning of using a probe as the classifier unclear.
  2. The paper trains the probe to make decisions, and reports that the trained probe performs badly when label-preserving, unrelated-file additions are introduced. But the paper does not explicitly report whether the probe will become more robust to irrelevant history if the probe is trained on history with the label-preserving, unrelated-file additions. The failure to make decisions with unrelated context may be because the probe did not see such cases in the training data, which makes such tasks out of the distribution. Please explicitly train the probe on contexts with label-preserving, unrelated-file additions and report the resulted decision accuracy; if the authors have already done this experiment, please clarify.
  3. The probe performs worse than directly using the LLM ‘yes’ or ‘no’ output, possibly because the probe structure is too simple. The probe only accepts the last token output of the LLM and goes through one liner layer. Therefore, apart from explicitly adding file-editing history of irrelevant files into the training data, I suggest increasing the complexity of the probe structure such as increasing input features and increasing probe layers, and also increasing the training set size accordingly. More complex network structure with more training data would improve the probe performance and even change the conclusion.
Reviewer_FTbqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether a probe that decodes a language model's internal state remains a reliable decision rule when task histories change. A controlled four-file workflow makes coverage depend on whether a successful test receipt matches the current bytes. In a Qwen confirmation on 32 new file-roster families, an unrelated file's TEST reduces a fixed feedback-trained probe's pairwise accuracy from 0.984 to 0.428, while output-score ranking remains 0.988. An exploratory centroid preserves the distinction at the same site. Separate workflow experiments show that feedback training of the readout repairs repeated score-based decisions in Qwen and Gemma. The paper distinguishes ranking, fixed-threshold accuracy, and completion, and qualifies its limited applicability to generated agents.

Critical strengths
  1. The target-preserving perturbation is well controlled: another file's TEST leaves the queried bytes and receipts unchanged, while execution checks and a deterministic tracker independently establish the labels. The frozen-head confirmation has a large ranking effect, and the model-output comparator plus same-site centroid prevent conflating projection failure with absence of state information (Section 3; Appendices O-P).
  2. The paper carefully respects dependence and evidence levels. It resamples complete families, separates the two prespecified ranking endpoints from exploratory analyses, and distinguishes cached replay from live execution. It also reports negative controls and qualifications, including the Jacobian numerical failure and corrected BF16 norm matching. Comparing feedback training with deterministic recovery is especially useful for separating readout quality from controller design.
Critical weaknesses
  1. The confirmation expands file rosters while retaining one language template and a small event grammar; the central suffix ranking failure is confirmed only for Qwen. Gemma preserves ranking in discovery. The live check uses different prompts and omits the suffix; after the disclosed parser amendment, both models complete all 64 cases without repeated successful TESTs. These acknowledged boundaries limit the result's breadth and its direct implications for agent behavior.
  2. The study establishes failure more strongly than conditions for reliable reuse. Readout form, training mixture, site, and threshold substantially change results, but no training/validation-only criterion is shown to select a robust combination. The headline native-threshold failure also uses a rule limiting unnecessary TESTs and already has only 67.9% accuracy without the suffix. Balanced calibration performs much better in some discovery cells (Table 43). This does not explain away the probe's rank reversal, but limits practical interpretation of the threshold results.
Questions for authors
  1. Does the fixed Qwen probe show the same ranking collapse under a held-out history rendering, rather than only new rosters within the current template?
  2. Can original-validation-only selection across regularization, training mixtures, and readout forms identify a robust predictor before evaluating the suffix? Which results distinguish training-data effects from readout-form effects?
  3. What are the balanced-rule outcomes on the 32-family confirmation, and do suffix-induced errors affect cost or completion under a matched live/recovery controller?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.