acceptodds
Under review as a conference paper at ICLR 2027

When Does Decodable Task Information Constitute Agent State? Dissociating Information, Currentness, and Behavioral Use in Live Coding Agents

Abstract

Probing is widely used to study task-relevant information in a model's internal representations, including information about the state of an ongoing task. For an acting agent, however, treating such information as state raises two further questions. Does it track the task as it changes, and does it shape the actions the agent takes? We study these questions in coding agents working on real software repositories, recording model activations and using task variables whose values can be determined directly from the environment. To separate current state from recency, we create conflicts in which an observation from the environment provides the current value, while stale evidence presented later asserts a different one. Our results empirically dissociate decodable information, authoritative currentness, and behavioral use. In our primary setting, where the prompt explicitly identifies the authoritative source, the current task value remains strongly decodable and continues to guide the agent's natural action. Yet a readout trained on ordinary cases fails at the frozen decision threshold when stale evidence is presented last. When current and stale values vary independently during fitting, however, the resulting readout transfers to held-out conflict data in every eligible case we test. In a separate construction, authority must be inferred from provenance rather than explicit labels, and we vary presentation order. The readout of the current value remains strong regardless of whether current or stale evidence appears first. We complement these measurements with causal interventions that separate changes in an internal score, in the decision process, in the emitted action, and in the final action. Effects depend on where the intervention enters the computation, and an action changed at one decision point may still be restored later in the trajectory. Together, these results show that evidence for agent state extends beyond representation to how task information enters decisions and persists as the task evolves.

Primary area:
interpretability and explainable AI
Keywords:
state tracking, probing, model interpretability, coding agents, causal interventions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bthHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents an approach to determining when from a live coding agent's hidden activations counts as evidence of agent state. It looks at three properties: D - the task variable can be linearly decoded; C - a readout fit where the authoritative value is also the last-presented one still tracks the authoritative value when stale evidence comes last; U - the agent's action follows the authoritative value even if there is a conflict.

Activations are captured online from six models acting on real Python repositories. The state is strongly decodable. The agreement-trained readout falls to chance under conflict at its frozen threshold, while still ranking the authoritative value well. Behavior follows the authoritative value about 99% of the time.

Critical strengths

The paper shows how to build a probe that actually tracks the current state. A probe trained where the current and stale values coincide can't distinguish them. Training with those values varied independently produces a probe that keeps tracking the current value when they conflict. This holds across models, task families, presentation orders, and when the prompt never says which source is current.

The paper shows that where an intervention is applied determines whether it changes behavior. An intervention changes the model's internal answer score when applied at the end of the prompt, but leaves the answer unchanged. When it's applied where the model actually chooses its action, it does change the answer, and the matched controls change nothing.

Critical weaknesses

In the training data the current and stale values are perfectly correlated, so the probe has no way to tell them apart, and failure under conflict is basically inevitable.

In the main setup the prompt explicitly marks one block as current and the other as an earlier snapshot. The agent following the marked block shows that it obeys the label, not that it tracks state. In the version without labels, too few replies can be parsed to support a conclusion.

Questions for authors
  1. Is the probe failure specific to stale evidence? If you add an irrelevant block of similar length in the same position, does the agreement-trained probe also lose calibration? That would separate a stale-evidence failure from a general sensitivity to any change in context.

  2. Does the agent still follow the current value without labels? In the setting where the prompt doesn't say which source is current, can you raise the parse rate, for example with a longer generation budget, enough to support a conclusion?

  3. Why does the agreement-trained probe score differently in two analyses? It scores only moderately in the same-order evaluation (Appendix C.2) but much higher on held-out data in the crossed-fitting comparison (Appendix C.5). Are these different data, positions or captures?

1 comment
AuthorsOpenReview ↗

Response to Reviewer bthH: Addressing the Concerns with Additional Experiments, New Evidence, and Clarifications

Thank you for your thoughtful review and helpful questions. We sincerely appreciate your recognition of our current-state probing approach and causal intervention experiments. Your questions have been particularly valuable in helping us examine the sources of probe failure, improve the behavioral evaluation, and clarify our experimental results. We have conducted additional experiments and revised the manuscript accordingly. We have also revised the presentation to make the research question and the limits of each measurement easier to follow. Below, we address your concerns and questions.

W1. Probe training and sensitivity to context (Q1)

We agree that agreement-only training cannot distinguish current from stale values. This is precisely the ambiguity we aim to examine. Importantly, probe failure under conflict is not inevitable: the original readout passes our currentness criterion in 31 of the 42 model–state combinations. The contribution is therefore not the drop in agreement-probe accuracy itself, but separating a readout's failure to transfer from the recoverability of current-state information, and showing how independently varied evidence can resolve that ambiguity.

To answer your first question, we added the control you suggested, replacing the stale evidence with an unrelated code block of similar length at the same position (Section 5.2, Table 1). For Qwen3.6-35B, the agreement-trained probe's accuracy decreases from 0.968 to 0.692 with unrelated code, and to 0.521 with conflicting stale evidence. Your concern is therefore partly supported: the probe is sensitive to general context changes, although conflicting evidence causes a larger decrease. Interestingly, Qwen3-Coder-Next shows a different pattern. Its accuracy remains at 0.947 with unrelated code, compared with 0.957 under agreement, but drops to 0.543 under conflict.

These results suggest that the sources of probe failure differ across models. We also find that crossed-fit probes maintain high accuracy under both tested changes. We have updated our interpretation to reflect these findings, with further details in Appendices C.10–C.11.

W2. Behavior without explicit authority labels (Q2)

We agree that following explicitly marked authority labels alone is insufficient to establish whether an agent can identify the current state without them. We also acknowledge that the original response coverage was too low to support a reliable conclusion.

To address this, we repeated the experiment with a 2,048-token generation budget, retained the raw responses, and corrected the parser to evaluate only the final answer. In the primary conflict setting, 84/96 responses produced valid action choices. Without explicit authority labels, the new all-sample accuracy is 64.6% (95% CI: 58.3%–71.0%), compared with 99.0% in the paired labeled condition. The results differ between initial states, with 48/48 correct choices on violated files and 14/48 on already-satisfied files. We now report these results and the response categories in Section 5.3 and Appendix C.7. This asymmetry also informs our revised claims that behavior without labels is measurable, but substantially less reliable than in the labeled setting.

We also added experiments in which all six models execute actual coding commands and observe their results. Without explicit authority labels, the primary model passes the final requirement test in 95/96 runs, including 47/48 initially violated cases. These experiments provide additional behavioral evidence beyond constrained text responses (Section 5.3, Table 2).

Q3. Different scores in Appendices C.2 and C.5

Thank you for pointing out this difference. We apologize that our original presentation did not make the distinction between the two analyses clear enough.

The 0.716 value in Appendix C.2 is the highest accuracy in a layer sweep evaluated on conflict cases where the current observation appears last. Although the two sources disagree, the current value is also the most recently presented one. This is different from the 0.927 value in Appendix C.5, which measures accuracy on held-out agreement cases. The two analyses also use different activation captures and evaluation procedures. Appendix C.2 uses the original vLLM captures, while C.5 uses later Hugging Face captures with repository-disjoint evaluation. For a direct comparison, C.5 evaluates both probes on the same held-out conflict cases. The agreement-trained probe achieves 0.500/0.542 accuracy, while crossed fitting improves this to 0.917/0.948. We have clarified these differences in Sections 5.1–5.2 and Appendix C.2.

We sincerely appreciate the questions that helped us strengthen this work. We hope the new experiments and clarifications address your concerns, and we would be grateful if you would reconsider the revised manuscript and share any further feedback.

Reviewer_FQiZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses conflicts between current and stale evidence to distinguish decodability, current-value readout, and behavioral use in coding agents. It also uses activation interventions to examine whether immediate changes persist to the final action.

Critical strengths
  • The experimental design directly addresses the research question. Current information is often also the most recently presented information, making the two difficult to distinguish with ordinary probes. By controlling evidence content and presentation order, the authors demonstrate this confound and its consequences.

  • The findings offer practical guidance. Varying current and stale values independently during training improves conflict balanced accuracy from approximately 0.50 to 0.92–0.95 in the primary experiment. Controls involving presentation order, removal of explicit authority labels, and text baselines further support this finding.

  • The causal analysis considers subsequent behavior. Changes in internal scores do not necessarily change actions, and immediate action changes can be reversed by later decisions. This highlights the need to evaluate local intervention effects and final behavior separately.

Critical weaknesses
  • The interpretation of C may be too broad. C directly measures whether a fixed probe transfers to conflict conditions. In the primary experiment, probe failure is largely calibration-limited, while the model’s actions remain 98.9% correct. The authors should clarify what a C− result establishes about the agent’s own state-tracking ability.

  • Behavioral evidence under more natural conditions is limited. The primary experiment explicitly identifies the authoritative source and constrains the action responses. Removing authority labels reduces the parse rate to 56.3%, below the authors’ 90% threshold. Accuracy among parseable responses therefore cannot establish overall behavioral reliability.

  • State-specific causal control is not fully established. Full-vector transplantation introduces substantial nonspecific disruption. The experiment separating state from answer identity shifts the decision margin but does not change the final answer. The causal claims should remain limited to the effects actually observed.

  • Some statistics require reconciliation. Appendix D.4 states that only the C criterion changes, yet the number of U+ cases increases from 31 to 32 within the same 36 cells. Some cells also lack recoverable continuous records. A complete per-cell table of statistics and classification changes would help resolve these concerns.

Questions for authors
  1. How should readers distinguish a fixed probe’s transfer failure from a failure of the agent’s own state tracking?
  2. What are the main types of unparseable responses after explicit authority labels are removed?
  3. Please explain the change in the U+ count in Appendix D.4 and provide the corresponding per-cell results.
2 comments
AuthorsOpenReview ↗

Response to Reviewer FQiZ: Addressing the Concerns with Clarifications, New Experiments, and Corrections

Thank you for your thoughtful review. We are grateful for the care and attention you have given our work. We appreciate your recognition of our experimental design, crossed fitting, and analysis of later decisions. Your comments helped us clarify our interpretation, strengthen the behavioral evaluation, and correct the numerical reporting. We have revised the manuscript and added experiments to address your concerns. We respond to each point below.

W1. Interpretation of currentness (Q1)

We agree that the failure of a fixed probe should not be interpreted as a failure of the agent to track its state. We apologize that this distinction was not sufficiently clear in the original manuscript.

We now clarify in Section 3 that C measures whether an agreement-trained readout can still identify the authoritative current value under conflicting evidence. In the primary experiment, the probe still distinguishes the states (AUROC 0.861), even though its original decision threshold fails. Recalibrating the readout improves accuracy, while training with current and stale values varied independently achieves even stronger results (Sections 5.1–5.2).

Therefore, C− does not mean the agent has lost its current-state information. Rather, it shows that a probe trained on ordinary cases may not reliably identify the current state when evidence conflicts. We evaluate the agent's behavior separately through action choices and actual command execution.

W2. Behavior without explicit authority labels (Q2)

We agree that the original 56.3% response coverage was insufficient to establish overall behavioral reliability. To address this, we repeated the experiment with a larger generation budget, retained the raw responses, and corrected the parser to evaluate only the final answer rather than the reasoning text.

Without authority labels, the new all-sample accuracy is 64.6% (95% CI: 58.3%–71.0%), compared with 99.0% in the paired labeled condition. The results also differ considerably between the two initial states: the model makes correct choices in 14/48 already-satisfied cases, compared with 48/48 violated cases. We now report these limitations directly in Section 5.3.

The original raw replies were not retained, so we could not determine the exact causes of the earlier parsing failures. In the new experiment, most unparseable responses were unfinished reasoning or outputs that did not follow the requested format. We provide the full breakdown in Appendix C.7.

We also added a separate experiment in which all six models execute actual coding commands and observe their results, providing further behavioral evidence beyond constrained text responses (Section 5.3, Table 2).

W3. State-specific causal claims

We agree that full-vector transplantation can introduce changes unrelated to task state, and that changing an internal score does not necessarily change the final action. We have clarified these limitations in Section 6.

At the same time, our causal experiments provide further evidence about how internal changes affect decisions. In GLM-4-32B, the same intervention changes 13/26 final actions when applied at the action-producing step, compared with 0/26 at the end of the prompt. We also observe that some agents return to their original choices during later decisions, even after repeated interventions.

We have retained the negative results from the experiment separating state from answer identity, where interventions change action-token logits but do not reverse final actions at the tested strength. We therefore do not claim that these experiments establish state-specific causal control or identify the underlying state-updating mechanism. Instead, they help explain when internal changes affect actions and whether those effects persist through later decisions.

W4. Statistical consistency and quantitative results (Q3)

Thank you for identifying the inconsistency in Appendix D.4. We apologize for this error. After rechecking the original per-cell results, we found that the discrepancy came from errors in the summary counts, rather than changes to the behavioral classifications.

We have corrected Appendix D.4. Across the same 36 cells, the U+ count is now consistently 30 under all three currentness criteria (primary threshold, AUROC-C, and Intercept-C), as expected when only the C criterion changes. We have also substantially improved the numerical reporting. Tables 5–7 now provide the available per-cell accuracies, confidence intervals, response coverage, and additional analyses for the 42 model–state combinations. Missing records are explicitly identified, and early experimental results are reported separately where the original continuous records are unavailable. We also clarify which archived classifications fall below the stated response-coverage requirement.

We hope these revisions address your concerns and make our findings and their limitations clearer. Thank you again for your constructive feedback.

Thank you for running the additional experiments and for clarifying how to interpret the probe results. I found the breakdown by task state especially helpful in understanding where the model struggled without explicit labels. I’ll keep my original score.

Reviewer_cpKUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper distinguishes decodable task information, authoritative currentness, and behavioral use in coding agents. Experiments cover six models and seven state families, with provenance/order controls and activation interventions separating score changes, emitted actions, and final answers.

Critical strengths
  1. The manuscript is well-organized and readable.
  2. The causal analysis carefully distinguishes margin influence, behavioral reversal, and persistence. Norm-matched controls, held-out replication, and transparent reporting of null results strengthen the evidence.
Critical weaknesses
  1. suspected AI-generated paper. parts of the paper read as AI-generated.
  2. Limited novelty. Prior work already shows that a model can encode information without using it. This paper adds experiments with current and outdated information, but provides little new explanation of how agents update and use task state.
  3. The main experiment explicitly maps states to “RUN TESTS” or “EDIT FILE,” without executing either action. The reported self-repair occurs within text generation, leaving its relevance to actual coding behavior unclear.
  4. Missing comprehensive quantitative results. The paper lacks a single table reporting numerical results and confidence intervals for all 42 model–state combinations. Instead, it presents categorical labels and scattered numerical results, making the main findings difficult to compare and independently verify.
Questions for authors

refer to weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer cpKU: Clarifications and Improvements in the Revised Manuscript

Thank you for your thorough review and helpful comments. Your feedback has been valuable in helping us improve the manuscript. We appreciate your positive comments on the paper’s organization and the careful design of the causal experiments. We have taken your concerns seriously and carefully revised the paper, added new experiments, and improved the presentation. Below, we respond to each of the four points you raised.

W1. AI assistance and readability

We acknowledge that we used generative AI tools to assist with writing the manuscript, as clearly disclosed in our AI Use Statement. We believe that the open and responsible use of AI can make a positive contribution to scientific research. Nevertheless, we agree that some parts of the original manuscript could have been clearer and may have made our work difficult to follow. We have taken this concern seriously and carefully reviewed every section of the manuscript to improve the overall readability wherever possible. We sincerely hope you will consider reading the revised version and sharing any further feedback on its clarity and presentation.

W2. Novelty and contribution

We agree with the reviewer that previous studies have made important contributions to the distinction between decodable information and actual use. We apologize that our original manuscript did not explain our research motivation and contributions clearly enough.

The central question of our work is whether acting coding agents tend to maintain an internal representation of the current state as they interact with a changing environment. To investigate this, we introduce a separate currentness measurement to examine whether a readout can identify the authoritative current value, particularly when current observations conflict with stale evidence. Our experiments provide some encouraging evidence. By varying current and stale information independently, we can recover reliable current-state readouts across several models and state families.

More importantly, we also use causal interventions to investigate how internal representations affect the agent's decisions. We find that the same intervention can change the final action when applied at the action-producing step, but not at the end of the prompt (13/26 versus 0/26 cases). We further observe that some agents can return to their original choices after repeated interventions, suggesting that changes to an immediate action do not necessarily persist through later decisions. These experiments help us better understand how internal changes influence actions and how agents revisit their decisions, although the underlying state-updating mechanism remains an open question.

We believe this question is important because reliable current-state information could help agents make better decisions in changing environments, reduce repeated processing of execution histories, and potentially support code world models. We have expanded this discussion in Section 7 and hope the revised manuscript better communicates the motivation and contributions of our work.

W3. Actual coding behavior

We agree that selecting between RUN TESTS and EDIT FILE does not demonstrate actual coding behavior.

To address this, we added experiments in which all six models execute real commands in repository environments. The evaluation covers 48 episodes from 47 repositories, with 192 execution runs per model. Agents can read files, make edits, run tests, and observe the results before deciding what to do next. The results are now presented in Section 5.3 and Table 2. Without explicit authority labels, the primary model achieves 87.5% first-decision accuracy and passes the final requirement test in 95/96 runs, including 47/48 initially violated cases. Other models show different behaviors, which we also report.

This new experiment allows us to evaluate actual coding actions and outcomes rather than relying only on textual choices. We have also clarified that our causal intervention experiments concern generated actions, not executed commands.

W4. Comprehensive quantitative results

We apologize that the original manuscript did not provide enough numerical results and tables in the main text, making it difficult to examine and compare our findings.

In the revised manuscript, we have added two new tables to the main text. Table 1 presents the results of our matched context-control experiments, while Table 2 summarizes the actual coding behavior across six models. The original cross-model and cross-family results are retained in Table 3.

We have also substantially improved the numerical reporting in the appendix. Tables 5–7 now provide a more complete overview of the 42 model–state combinations, including available accuracies, confidence intervals, sample sizes, and additional analyses. We have also clearly indicated where some early experimental records are unavailable.

We hope these changes make our results easier to compare and verify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.