- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper discusses how modifications to an LLM agent's harness can affect its tool-use behavior while keeping the underlying model weights fixed. The authors propose distinguishing between the operation an agent performs (e.g., search, read, edit) and the interface through which the operation is executed (shell commands or API-style tool calls). Empirical analysis based on gpt-oss-20b with six harnesses on SWE-bench Verified, combines behavioral analysis, hidden-state probing, cross-harness transfer experiments, matched-file comparisons, controlled context rewrites, and activation interventions. The main findings suggest that hidden states contain information about upcoming operations beyond a structured call-history baseline, while interface selection is largely predictable from previous tool calls. The authors also observe that adding tools changes interface usage considerably more than the distribution of coarse operations. Controlled interventions further show that rewriting the form of previous tool calls primarily affects interface selection, whereas changing search results primarily affects subsequent operation choices.
Critical strengths
- understanding how harness modifications influence fixed-weight agents is important for the development and evaluation of agentic systems.
- The paper discusses the problem from multiple complementary perspectives, including behavioral comparisons, hidden-state probing, cross-harness transfer, and controlled interventions.
- The finding that an operation decoder trained using one invocation interface can retain substantial predictive performance when applied to another interface is interesting.
Critical weaknesses
-
several central findings reinforce observations that are already reasonably well established in the literature. I would be important that the authors bring more discussion on what new understanding of agent decision-making emerges from their results/analyses that was not already available through prior behavioral studies?
-
the paper does not convincingly establish why hidden-state analysis is necessary to address this problem.
-
in the experiments, some interventions substantially change the frequency of unregistered tool calls. Since these calls are excluded from the principal rewrite comparisons, the treatment of invalid and unclassifiable outcomes deserves more attention.
Questions for authors
-
How would the operation decoder perform relative to a stronger baseline that includes actual tool outputs and the preceding transcript? Is there a concrete scenario in which the decoder enables a better decision than observable behavioral information alone?
-
Can the authors clarify the extent to which their results demonstrate independent versus preferential effects? How sensitive are the rewrite results to invalid or unclassifiable tool calls, and would crossed interventions on the same contexts strengthen the interpretation?
2 comments
thank you for the responses
The response restates the findings but does not position them relative to prior behavioral studies of LLM agents. It is still unclear which conclusion about agent decision-making could not be reached from behavioral traces alone.
The authors confirm three limits: the +11.8-point gain is measured only against structured call history, no full-transcript baseline was evaluated, and the practical application is only proposed (Appendix H.5).
The all-continuation rates show that the direction of interface adoption persists. However, the operation-TV comparison, which supports the claim that form rewrites leave operations largely unchanged, still excludes unregistered calls.
The response does not address the readability concern. A clear roadmap linking each research question to its experiment and main result would help readers.
W1. Our contribution is joint behavioral and representational evidence that invocation history and task feedback preferentially affect different components of tool use. Large interface changes under content-preserving history rewrites coexist with comparatively small changes in the operation readout, whereas replacing search hits with no matches produces larger changes in both operation choice and its pre-call readout (Tables 21–22). Together with cross-interface transfer, this characterizes how operation information is represented and responds to different harness-context edits in a fixed model.
W2/Q1. Internal-state analysis establishes that operation information can be read across invocation interfaces before the call is written. For read/search classification, the decoder retains 87% of its within-interface normalized gain across interfaces, with tasks held out (Table 15). In the rewrite experiments, the frozen decoder’s mean posterior shifts by 18.1 points of TV under hits → no matches, compared with 2.2 and 0.8 points under form rewrites (Tables 21–22). Emitted calls alone do not establish this transfer and differential sensitivity of the pre-call representation.
The +11.8-point increment is specifically over structured call history; a full-transcript predictor would provide a further comparison of predictive performance. Our demonstrated contribution is diagnostic: the readout supplies a common pre-call measure for checking whether increased interface adoption accompanies changes in operation information. Using this diagnostic to guide automated harness selection is a proposed application in Appendix H.5.
W3/Q2. Our interventions establish preferential effects, with cross-effects allowed. The interpretation does not require operation and interface to be statistically independent. Applying both rewrites to the same contexts would further characterize their interaction, as discussed in Appendix H.5.
Registration validity is reported as an outcome of the form rewrites. The principal comparison of operation TV and API-share change uses classifiable registered calls, while Table 21 separately reports unregistered calls changing from 0 to 34 and from 54 to 15.
The counts and shares in Table 21 also imply that, over all 360 continuations per version, registered, classifiable API-call rates change from approximately 2.5% to 25.3% under shell → API rewrites and from 44.2% to 21.9% under API → shell rewrites. Thus, the direction of interface adoption persists when all continuations are retained in the denominator. These descriptive rates complement the conditional operation/interface comparison and the separately reported registration outcomes.
For the result rewrite, Table 22 reports a +42.2-point change in search share minus read share over all continuations, including those without a call (p = 7.2 × 10⁻⁷). Separately, the frozen decoder is evaluated on every context version, regardless of whether a continuation produces a classifiable registered call.
We will present these denominators and validity outcomes more explicitly, distinguishing the conditional call statistics, the all-continuation behavioral measures, and the all-context representation analysis.