- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLMs driving coding agents linearly encode properties of the program they iteratively edit. It collects mini-swe-agent trajectories from two open-weight models on SWE-Bench-Verified and SWE-Bench-Pro. Each intermediate code version is labeled by execution for well-formedness, full correctness, partial correctness, and regression. Logistic-regression probes on residual-stream activations reach up to about 0.83 AUC, peak in middle layers, and transfer across benchmarks. Probes trained to predict labels k steps ahead stay above chance up to k = 25, which the authors interpret as a “latent programming horizon” analogous to look-ahead in game-playing networks.
Critical strengths
[S1] Timely, practically relevant setting. It moves correctness probing from single-shot snippet generation to multi-step agentic repair on real repositories, where program state changes over time and is only partially observed.
[S2] Execution-grounded labels at scale: every intermediate version is checked out and run, across 22.7k trajectories and 1231 tasks. This pipeline could be a useful resource if released.
[S3] Partial correctness and regression are labels defined relative to the initial repository state. They are meaningful only in the agentic setting and are rarely studied.
[S4] Task-level splits, a layer-wise analysis that is consistent across models, and cross-benchmark transfer give some robustness to the k = 0 results.
[S5] Potential downstream value for cheap agent monitoring and early failure detection, and as a starting point for steering or causal work.
Critical weaknesses
[W1] Limited methodological novelty. Linear probes with a shuffled control, correctness probing, and future-state probing are all established in existing literature. The paper mainly applies them to a new setting without new analysis tools or causal interventions, so its value only rests on the strength of its empirical results.
[W2] No controls for surface information. The agent’s context directly contains test outputs (e.g., “1 failed” / “3 passed”), so probes may simply read recent tool output rather than a latent program representation. A token-level shuffled-label control is very weak. Needed baselines include classifiers on recent tool-output text, step index, and task identity, plus task-level permutation.
[W3] The “programming horizon” is likely confounded. Labels are piecewise-constant, with a median of only about 2 edits per trajectory, so the label at t+k usually equals the current label. Near k = 50, under the 100-step cap and >= 50-step filter, the target is essentially the final task outcome, which task difficulty can predict. Maybe consider to evaluate only on positions where the label flips between t and t+k, and compare against current-label and task-prior baselines. Without this, claims of planning or look-ahead are overstated.
[W4] Evaluation and reporting issues. AUC is computed over highly correlated token positions, so long trajectories dominate. No confidence intervals, seed variance, or per-edit and per-trajectory metrics are given. The practical utility of a ~0.8-AUC probe is also not discussed.
Questions for authors
What are the concrete implications or potential use cases of the empirical findings demonstrated in this paper? The paper needs to answer why do we need to know these results and how to leverage them to do something.
5 comments
Thank you for your reply! I think the rebuttal responses address many previous concerns. Though I still have the following questions and concerns about this work:
- For decoding current-state properties, how do the probes compare with simpler alternatives, e.g., directly asking the model at a given step whether it believes the current version correctly resolves the issue? This would clarify how much the probes capture beyond what the model can verbalize, to what extent they are consistent, or which one is better.
- I understand that the paper is positioned as an empirical study for better understanding, and I acknowledge the value of such contributions. Still, I believe even one or two setups evaluating whether the findings are useful in practice would make the work more complete and convincing. An honest analysis would be valuable even if the gains turn out to be limited. As a reader, it is not intuitive to understand how strong the signals provided in this paper are.
Thanks for the reply! We're happy to answer:
1. For each test trajectory, we sampled 5 steps evenly spaced over the trajectory plus the last step of the trajectory. At each sampled step, we appended the question "Stop and answer with one word, yes or no: does the current code in the repository correctly resolve the issue?" to the agent's context and computed P(yes)/(P(yes)+P(no)) from the next-token distribution. This results in 16,035 labelled states with both trained full correctness probes and a prompt comparison. The table below compares the AUC from both methods with 95% CI.
| Run | Asking | Probe | Delta (Probe - Asking) | Spearman |
|---|---|---|---|---|
| Laguna/Verified | 0.88 | 0.86 | −0.022 [−0.035, −0.009] | 0.81 |
| Laguna/Pro | 0.87 | 0.83 | −0.042 [−0.068, −0.016] | 0.78 |
| Qwen/Verified | 0.86 | 0.89 | +0.030 [+0.012, +0.048] | 0.77 |
| Qwen/Pro | 0.83 | 0.87 | +0.036 [+0.002, +0.075] | 0.66 |
The results show that both scores are correlated and have similar AUCs. We believe these results show the same information being decoded in different ways. In our probes, the linear projections decode the probability directly from the hidden states. In the asking prompt mode, the entire model's forward pass does a similar job first building up the representation in the residual stream and then constructing the answer (as token logits).
2. While the main goal of our study is to improve the understanding of the internal representations of LLMs when operating as coding agents, we believe that, as the community develops better understanding of these representations and develops better probing mechanisms, these could be useful, for example, in monitoring progress of agents.
As a pilot experiment, we tested whether the probe helps to choose among attempts. In our test dataset each task has ~10 atempts each. For each attempt, we computed the probe score at its last step and compared it with the ground-truth evaluation.
- Best-of-n. In this scenario, only one solution can be chosen out of n attempts. With all attempts, a random pick resolves 31.8% and the ground-truth oracle 45.2%. The gain over a random pick is +1.1 points [−0.9, +3.1] for the probe and +1.6 [+0.7, +2.6] for the baseline "last test run passed". With 4 attempts, the gains are +2.2 [+1.2, +3.3] and +1.5 [+0.9, +2.3].
- Ranking attempts on the same task. The AUC is 0.67 for the probe, 0.60 for the test-run rule, and 0.48 for a classifier on the state before the first action. The probe score therefore depends on the attempt and not only on the task.
We thank the reviewer for their time and useful feedback. We agree that the horizon was confounded, and we followed the suggested evaluation (see the general response). This evaluation still shows information beyond persistence and trajectory phase, and, as the reviewer suspected, a large part of it reflects the eventual outcome. We will release the code and data after the review period.
W1: Limited methodological novelty
We partly agree. The probing methodology is standard. We see the contribution being novel from three different angles:
W2: Controls for surface information
We added a trajectory-level permutation, step-index and relative-position baselines, language-only and repository-only baselines, and an evaluation on held-out repositories (Section 4.1, Appendix A.4).
On test outputs in context: right after an edit that changes a label, before any tool runs, the probes mostly report the old label (App. A.6). This suggests that the hidden states do not yet fully reflect the effect of an edit before tools such as tests run, which we now state in the Limitations.
Please see the general response.
W3: The horizon is confounded
We agree, and we followed the suggestion.
These results show that a large part of the signal in the hidden states is linked to the eventual outcome of the trajectory. Please see the general response.
W4: Evaluation and reporting
Q: Concrete implications and use cases
The main usefulness of our study is in improving the understanding of what is represented about programs internally in models while they act as coding agents. We believe this is a necessary step towards building more explainable and safer agents.
Beyond that, we see two concrete use cases that could use our results: