- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents an approach to determining when from a live coding agent's hidden activations counts as evidence of agent state. It looks at three properties: D - the task variable can be linearly decoded; C - a readout fit where the authoritative value is also the last-presented one still tracks the authoritative value when stale evidence comes last; U - the agent's action follows the authoritative value even if there is a conflict.
Activations are captured online from six models acting on real Python repositories. The state is strongly decodable. The agreement-trained readout falls to chance under conflict at its frozen threshold, while still ranking the authoritative value well. Behavior follows the authoritative value about 99% of the time.
Critical strengths
The paper shows how to build a probe that actually tracks the current state. A probe trained where the current and stale values coincide can't distinguish them. Training with those values varied independently produces a probe that keeps tracking the current value when they conflict. This holds across models, task families, presentation orders, and when the prompt never says which source is current.
The paper shows that where an intervention is applied determines whether it changes behavior. An intervention changes the model's internal answer score when applied at the end of the prompt, but leaves the answer unchanged. When it's applied where the model actually chooses its action, it does change the answer, and the matched controls change nothing.
Critical weaknesses
In the training data the current and stale values are perfectly correlated, so the probe has no way to tell them apart, and failure under conflict is basically inevitable.
In the main setup the prompt explicitly marks one block as current and the other as an earlier snapshot. The agent following the marked block shows that it obeys the label, not that it tracks state. In the version without labels, too few replies can be parsed to support a conclusion.
Questions for authors
-
Is the probe failure specific to stale evidence? If you add an irrelevant block of similar length in the same position, does the agreement-trained probe also lose calibration? That would separate a stale-evidence failure from a general sensitivity to any change in context.
-
Does the agent still follow the current value without labels? In the setting where the prompt doesn't say which source is current, can you raise the parse rate, for example with a longer generation budget, enough to support a conclusion?
-
Why does the agreement-trained probe score differently in two analyses? It scores only moderately in the same-order evaluation (Appendix C.2) but much higher on held-out data in the crossed-fitting comparison (Appendix C.5). Are these different data, positions or captures?
Response to Reviewer bthH: Addressing the Concerns with Additional Experiments, New Evidence, and Clarifications
Thank you for your thoughtful review and helpful questions. We sincerely appreciate your recognition of our current-state probing approach and causal intervention experiments. Your questions have been particularly valuable in helping us examine the sources of probe failure, improve the behavioral evaluation, and clarify our experimental results. We have conducted additional experiments and revised the manuscript accordingly. We have also revised the presentation to make the research question and the limits of each measurement easier to follow. Below, we address your concerns and questions.
W1. Probe training and sensitivity to context (Q1)
We agree that agreement-only training cannot distinguish current from stale values. This is precisely the ambiguity we aim to examine. Importantly, probe failure under conflict is not inevitable: the original readout passes our currentness criterion in 31 of the 42 model–state combinations. The contribution is therefore not the drop in agreement-probe accuracy itself, but separating a readout's failure to transfer from the recoverability of current-state information, and showing how independently varied evidence can resolve that ambiguity.
To answer your first question, we added the control you suggested, replacing the stale evidence with an unrelated code block of similar length at the same position (Section 5.2, Table 1). For Qwen3.6-35B, the agreement-trained probe's accuracy decreases from 0.968 to 0.692 with unrelated code, and to 0.521 with conflicting stale evidence. Your concern is therefore partly supported: the probe is sensitive to general context changes, although conflicting evidence causes a larger decrease. Interestingly, Qwen3-Coder-Next shows a different pattern. Its accuracy remains at 0.947 with unrelated code, compared with 0.957 under agreement, but drops to 0.543 under conflict.
These results suggest that the sources of probe failure differ across models. We also find that crossed-fit probes maintain high accuracy under both tested changes. We have updated our interpretation to reflect these findings, with further details in Appendices C.10–C.11.
W2. Behavior without explicit authority labels (Q2)
We agree that following explicitly marked authority labels alone is insufficient to establish whether an agent can identify the current state without them. We also acknowledge that the original response coverage was too low to support a reliable conclusion.
To address this, we repeated the experiment with a 2,048-token generation budget, retained the raw responses, and corrected the parser to evaluate only the final answer. In the primary conflict setting, 84/96 responses produced valid action choices. Without explicit authority labels, the new all-sample accuracy is 64.6% (95% CI: 58.3%–71.0%), compared with 99.0% in the paired labeled condition. The results differ between initial states, with 48/48 correct choices on violated files and 14/48 on already-satisfied files. We now report these results and the response categories in Section 5.3 and Appendix C.7. This asymmetry also informs our revised claims that behavior without labels is measurable, but substantially less reliable than in the labeled setting.
We also added experiments in which all six models execute actual coding commands and observe their results. Without explicit authority labels, the primary model passes the final requirement test in 95/96 runs, including 47/48 initially violated cases. These experiments provide additional behavioral evidence beyond constrained text responses (Section 5.3, Table 2).
Q3. Different scores in Appendices C.2 and C.5
Thank you for pointing out this difference. We apologize that our original presentation did not make the distinction between the two analyses clear enough.
The 0.716 value in Appendix C.2 is the highest accuracy in a layer sweep evaluated on conflict cases where the current observation appears last. Although the two sources disagree, the current value is also the most recently presented one. This is different from the 0.927 value in Appendix C.5, which measures accuracy on held-out agreement cases. The two analyses also use different activation captures and evaluation procedures. Appendix C.2 uses the original vLLM captures, while C.5 uses later Hugging Face captures with repository-disjoint evaluation. For a direct comparison, C.5 evaluates both probes on the same held-out conflict cases. The agreement-trained probe achieves 0.500/0.542 accuracy, while crossed fitting improves this to 0.917/0.948. We have clarified these differences in Sections 5.1–5.2 and Appendix C.2.
We sincerely appreciate the questions that helped us strengthen this work. We hope the new experiments and clarifications address your concerns, and we would be grateful if you would reconsider the revised manuscript and share any further feedback.