acceptodds
Under review as a conference paper at ICLR 2027

From Operation to Interface: Where Harness Changes Enter a Fixed Model’s Tool Calls

Abstract

Procedures that improve a coding agent without touching its weights edit its prompts, tools and middleware and keep the edits that raise task success, but task success alone does not reveal which aspects of tool use an edit changes. We study gpt-oss-20b under six harnesses on SWE-bench Verified and separate each classifiable tool-call attempt into an operation category (such as search or read) and an invocation interface (shell or API-style). In TOOLS, the layer-20 state at the first generated-token position adds +11.8 accuracy points for the operation but +1.0 for the interface beyond a structured call-history baseline, and an operation decoder transfers across most harnesses and across interfaces. At file-read-matched anchors, TOOLS and SEED differ by +45.8 points in API share, compared with 4.4 points of total variation in the operation distribution, which does not exceed its permutation null. Rewriting past reads between shell and API forms shifts API share by +40.8 and −36.2 points, with smaller operation shifts of 8.6 and 2.5 points, while replacing search hits with no matches shifts the operation by 29.0 points and API share by −6.5. The frozen operation decoder responds most strongly to this result rewrite, but pushing the state along the decoder’s direction changes the decoder readout without establishing targeted control of the next operation. These findings separate changes in operation choice from history-sensitive changes in invocation interface.

Primary area:
interpretability and explainable AI
Keywords:
LLM agents, coding agents, agent harnesses, tool use, tool interfaces, internal states, representation probing, SWE-bench

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_73vZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper discusses how modifications to an LLM agent's harness can affect its tool-use behavior while keeping the underlying model weights fixed. The authors propose distinguishing between the operation an agent performs (e.g., search, read, edit) and the interface through which the operation is executed (shell commands or API-style tool calls). Empirical analysis based on gpt-oss-20b with six harnesses on SWE-bench Verified, combines behavioral analysis, hidden-state probing, cross-harness transfer experiments, matched-file comparisons, controlled context rewrites, and activation interventions. The main findings suggest that hidden states contain information about upcoming operations beyond a structured call-history baseline, while interface selection is largely predictable from previous tool calls. The authors also observe that adding tools changes interface usage considerably more than the distribution of coarse operations. Controlled interventions further show that rewriting the form of previous tool calls primarily affects interface selection, whereas changing search results primarily affects subsequent operation choices.

Critical strengths
  • understanding how harness modifications influence fixed-weight agents is important for the development and evaluation of agentic systems.
  • The paper discusses the problem from multiple complementary perspectives, including behavioral comparisons, hidden-state probing, cross-harness transfer, and controlled interventions.
  • The finding that an operation decoder trained using one invocation interface can retain substantial predictive performance when applied to another interface is interesting.
Critical weaknesses
  • several central findings reinforce observations that are already reasonably well established in the literature. I would be important that the authors bring more discussion on what new understanding of agent decision-making emerges from their results/analyses that was not already available through prior behavioral studies?

  • the paper does not convincingly establish why hidden-state analysis is necessary to address this problem.

  • in the experiments, some interventions substantially change the frequency of unregistered tool calls. Since these calls are excluded from the principal rewrite comparisons, the treatment of invalid and unclassifiable outcomes deserves more attention.

Questions for authors
  1. How would the operation decoder perform relative to a stronger baseline that includes actual tool outputs and the preceding transcript? Is there a concrete scenario in which the decoder enables a better decision than observable behavioral information alone?

  2. Can the authors clarify the extent to which their results demonstrate independent versus preferential effects? How sensitive are the rewrite results to invalid or unclassifiable tool calls, and would crossed interventions on the same contexts strengthen the interpretation?

2 comments

W1. Our contribution is joint behavioral and representational evidence that invocation history and task feedback preferentially affect different components of tool use. Large interface changes under content-preserving history rewrites coexist with comparatively small changes in the operation readout, whereas replacing search hits with no matches produces larger changes in both operation choice and its pre-call readout (Tables 21–22). Together with cross-interface transfer, this characterizes how operation information is represented and responds to different harness-context edits in a fixed model.

W2/Q1. Internal-state analysis establishes that operation information can be read across invocation interfaces before the call is written. For read/search classification, the decoder retains 87% of its within-interface normalized gain across interfaces, with tasks held out (Table 15). In the rewrite experiments, the frozen decoder’s mean posterior shifts by 18.1 points of TV under hits → no matches, compared with 2.2 and 0.8 points under form rewrites (Tables 21–22). Emitted calls alone do not establish this transfer and differential sensitivity of the pre-call representation.

The +11.8-point increment is specifically over structured call history; a full-transcript predictor would provide a further comparison of predictive performance. Our demonstrated contribution is diagnostic: the readout supplies a common pre-call measure for checking whether increased interface adoption accompanies changes in operation information. Using this diagnostic to guide automated harness selection is a proposed application in Appendix H.5.

W3/Q2. Our interventions establish preferential effects, with cross-effects allowed. The interpretation does not require operation and interface to be statistically independent. Applying both rewrites to the same contexts would further characterize their interaction, as discussed in Appendix H.5.

Registration validity is reported as an outcome of the form rewrites. The principal comparison of operation TV and API-share change uses classifiable registered calls, while Table 21 separately reports unregistered calls changing from 0 to 34 and from 54 to 15.

The counts and shares in Table 21 also imply that, over all 360 continuations per version, registered, classifiable API-call rates change from approximately 2.5% to 25.3% under shell → API rewrites and from 44.2% to 21.9% under API → shell rewrites. Thus, the direction of interface adoption persists when all continuations are retained in the denominator. These descriptive rates complement the conditional operation/interface comparison and the separately reported registration outcomes.

For the result rewrite, Table 22 reports a +42.2-point change in search share minus read share over all continuations, including those without a call (p = 7.2 × 10⁻⁷). Separately, the frozen decoder is evaluated on every context version, regardless of whether a continuation produces a classifiable registered call.

We will present these denominators and validity outcomes more explicitly, distinguishing the conditional call statistics, the all-continuation behavioral measures, and the all-context representation analysis.

Reviewer_73vZOpenReview ↗

thank you for the responses

The response restates the findings but does not position them relative to prior behavioral studies of LLM agents. It is still unclear which conclusion about agent decision-making could not be reached from behavioral traces alone.

The authors confirm three limits: the +11.8-point gain is measured only against structured call history, no full-transcript baseline was evaluated, and the practical application is only proposed (Appendix H.5).

The all-continuation rates show that the direction of interface adoption persists. However, the operation-TV comparison, which supports the claim that form rewrites leave operations largely unchanged, still excludes unregistered calls.

The response does not address the readability concern. A clear roadmap linking each research question to its experiment and main result would help readers.

Reviewer_3E6ROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether changes to a coding agent’s instructions, tools and tool responses (its harness) change its actions or mainly how it calls tools. It runs fixed gpt-oss-20b under six setups on small SWE-bench Verified subsets. SEED offers a shell; TOOLS adds five dedicated tools, including read_file, while keeping the shell. A file read through either route has the same action category (operation) but a different call form (interface). A separate classifier predicts the next operation from the model’s internal state at the start of a reply. The authors compare setups and edit earlier messages or internal states to measure how subsequent calls change.

Critical strengths
  1. Changing earlier call forms changes later tool use. Rewriting earlier shell reads as read_file calls, with returned file contents unchanged, raises subsequent dedicated-tool use from 3.8% to 44.6%. The reverse edit lowers it from 64.9% to 28.7%.
Critical weaknesses
  1. Limited novelty and unclear additional insight: Comparing coding interfaces with similar capabilities and grouping shell commands by action are established (https://arxiv.org/pdf/2608.11386). Predicting tool need from internal states before an action is also established (https://arxiv.org/pdf/2605.06890). This paper adds a comparison of action category with call form and controlled history edits that preserve returned contents. Neither cited work establishes those exact experiments. However, the resulting explanation and consequences remains limited: the tool-addition comparison starts from different situations (2 below), and the state’s added insight beyond visible context is unclear (3 below). The consequences of these observations for understanding or designing agents remain unclear, like what's the impact of this steering behaviour for practical deployments/scenarios?
  2. Tool addition is not tested from an identical starting point. The “same file” comparison groups reads from different runs without matching earlier messages, file contents or read ranges (Appendix F.1). Similar action counts do not establish preserved decisions or file targets. The authors propose comparing both setups from the same conversation and repository snapshot, but do not run this test..
  3. What the classifier adds beyond the visible context is unclear. The history baseline knows the last action and recent dedicated-tool use, but lacks corresponding summaries of earlier action categories and the actual text returned by tools. The internal state has processed all that context. Its better action prediction could partly reflect information omitted from the baseline. The measured gain is valid for the chosen summary; a richer baseline is needed to assess the extra insight from inspecting internal states.
  4. The central experiments need clearer explanations. The abstract uses SEED and TOOLS before defining them, leaves the two rewrite directions unlabeled, and describes “the decoder’s direction” without explaining prediction versus actual action. Details appear later, but readers must reconstruct the central experiments across sections and appendices. Definitions at first use and worked before/after examples are needed.
Questions for authors
  1. What does inspecting internal states establish about tool changes that the emitted calls and history-edit experiments alone would miss? Please give a concrete finding and its implication.
  2. From the same conversation and repository snapshot, does changing only the available tools still change call form much more than action choice?
  3. Does the pattern persist with longer generations?
  4. Which observation changes how we should understand or design a coding agent? Please give one concrete consequence and evidence supporting it. What do internal-state measurements add beyond analyzing emitted calls?
1 comment

W1/Q4. Our new finding is that invocation conventions and task feedback preferentially affect different components of tool use, with corresponding differences in a transferable pre-call operation readout. Content-preserving history rewrites mainly change interface use, whereas replacing search hits with no matches produces larger changes in both operation choice and its internal readout (§§5, 8). This connects prior behavioral architecture comparisons and pre-action tool-need/risk analyses to the question of how operation information is represented across invocation interfaces (§2).

A concrete evaluation consequence is to assess operation choice, interface adoption, and registration validity separately when judging a harness edit. Increased API adoption can coexist with much smaller changes in the operation readout. For diagnosing tool underuse, applicability also matters: search adoption is 38.9% overall versus 73.7% among applicable calls (§7). These findings identify specific checks for distinguishing invocation changes from changes in operation choice.

W2/Q2. Sections 6 and 8 address complementary questions. Section 6 compares separately developed trajectories at shared-file anchors, while Section 8 isolates the effect of invocation history with the tool menu and returned contents fixed. The paired history rewrites shift API share by +40.8/−36.2 points, compared with operation TV of 8.6/2.5 points (Table 1).

The requested menu-only intervention from an identical conversation and repository snapshot is not reported. Our controlled rewrites establish sensitivity to invocation history; the matched-trajectory analysis provides complementary evidence about differences between harness configurations. We will make this distinction explicit throughout the presentation.

W3/Q1. Internal measurements show that operation information transfers across invocation interfaces before the call is written. In the read/search comparison, the decoder retains 87% of its within-interface normalized gain across interfaces, with tasks held out (§5; Table 15). In the rewrite experiments, the frozen decoder’s mean posterior shifts by only 2.2/0.8 points of TV under form rewrites, versus 18.1 points when search hits become no matches (Tables 21–22). It is evaluated on every context version, including those whose continuations produce no classifiable call.

The +11.8-point accuracy increment is specifically over the structured-history baseline. A full-transcript predictor would provide a further comparison of predictive performance. The transfer and rewrite analyses address a distinct question: how operation information is represented across interfaces and responds to controlled context edits. Emitted calls alone do not establish these properties of the pre-call representation.

Q3. The representation and transfer analyses use original agent trajectories; the 256-token budget applies specifically to the local rewrite experiments. Seed-family requests allowed up to 32,000 output tokens (Appendix A.1). Whether the local rewrite effects persist under longer continuation budgets is an interesting question for further investigation.

W4. We will define the harnesses and illustrate the central interventions at first use. SEED is shell-only; TOOLS retains shell access and adds five dedicated tools. Worked examples will make the controls explicit: sed → read_file with identical returned contents, and search hits → no matches with an unchanged search call. We will also distinguish decoder predictions from emitted operations throughout.

Reviewer_bYQ1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks how a fixed model’s tool calls change when its harness changes. It splits each call into two parts, the operation (search, read, edit, exec, list) and the interface (API style or shell). Across six harnesses and SWE-bench tasks, adding tools or rewriting the form of earlier reads mainly moves the interface, while rewriting the last search result mainly moves the operation. A readout from the model’s layer-20 state predicts the coming operation beyond the call history, adding about 11.8 points of accuracy, and this readout transfers across most harnesses and across interfaces. The state adds little for the interface. The authors conclude that harness edits should be checked for which half of a call they move before they are scored. Pushing the state along the decoder direction changes the readout but does not establish targeted control of the next operation.

Critical strengths
  1. The paper studies a very interesting problem that try to answer how model call difference tools with similar functionalities. This gives a more closed look at the model behaviors and may reveal, to some extent, the insight that helps us understand how model call tools.
  2. The paper splits a tool call into two simple parts: what the model wants to do and how it writes the call. This makes harness changes measurable, and its matched file-read comparisons and small context rewrites give stronger evidence than task scores alone.
Critical weaknesses
  1. I find the writing is a bit hard to follow, e.g., in the abstract, the authors mention "Seed" and "TOOLs", and they are all harness. The authors should clearly indicate what they are in case some audiances are not familiar with these harness systems.
  2. The paper’s causal analysis focuses mainly on read and search, so it is unclear whether its conclusion that operation and interface can be separated also holds for edit, exec, and list, or extends to other tool functions.
  3. The paper tests only one model, gpt-oss-20b, at a single layer on a single benchmark, so it is unclear whether the findings generalize to other models, layers, or benchmarks.
  4. The evaluated harnesses are mostly self-built or adapted versions rather than widely used agent frameworks, so it is unclear whether the findings transfer to common real-world harnesses.
Questions for authors
  1. Could the authors clarify which harness-design recommendations in the paper are explicitly supported by their results and which are extrapolations from them? The paper reports several implications, such as checking an added tool as an interface change, measuring adoption on applicable calls, and registering names the model tends to write, but it does not present a systematic design guide. Could the authors summarize the actionable design principles, state the evidence supporting each, and indicate whether these principles are expected to hold for edit, exec, and list as well as for read and search? 2.Why do you choose layer 20 as the example?
  2. How about other open-weight models and Harness?
2 comments

W1. SEED is our shell-only baseline, while TOOLS retains shell access and adds five dedicated tools from ModularRSI: read_file, grep_search, glob_files, edit_file, and write_file (§3; Appendix A). Reading the same lines through sed and read_file illustrates our distinction: the operation is read, while the invocation interface differs. We will introduce these definitions at first use.

W2/Q1. Our results support four concrete diagnostic principles for evaluating harness changes.

  1. Separate operation choice from interface adoption. At file-read-matched anchors, API share differs by 45.8 points, while operation TV is 4.4 points, within its permutation null (§6).
  2. Measure adoption on applicable calls. Search-tool adoption is 38.9% overall but 73.7% among searches that one dedicated call can serve (§7).
  3. Account for preceding call forms. Content-preserving history rewrites change subsequent API share by +40.8 and −36.2 points (§8).
  4. Check names and argument conventions. Among unregistered names, 95.6% resemble an added tool; 42 of 60 inspected cases use named parameters despite the positional schema (§7).

These results directly support the diagnostic principles above. Early tool demonstrations and registering common aliases are further design recommendations motivated by these observations.

Our behavioral and decoding analyses cover search, read, edit, exec, and list across 8,172 classified calls. The controlled rewrite evidence primarily concerns read and search, for which invocation form and returned content can be varied separately. The decomposition can also be applied to other operations wherever comparable invocation routes exist.

W3/Q2. We choose layer 20 because our previously released J64 frame provides a fixed measurement basis at this layer. Its 64-dimensional basis is used unchanged across harnesses. The agent lens and random projections reproduce the operation–interface predictive asymmetry at the same layer (Appendix E.1), showing that this result is robust to these alternative readouts. Examining how the distinction develops across layers is a natural extension.

W3/W4/Q3. Our evidence spans six harnesses, including adapted Codex and configurations incorporating AHE and ModularRSI outputs. The operation-prediction increment beyond structured history is positive in all six harnesses, including +14.6 points in Codex; all 20 transfer cells among Harmony-based harnesses exceed their permutation nulls (§5; Appendix E).

Holding the model fixed allows us to compare harness configurations without conflating their effects with differences in model weights. These results establish transfer across the evaluated harnesses, while extending the diagnostics across model families and task domains would further characterize their generality.

Reviewer_bYQ1OpenReview ↗

React to the author's response

I appreciate the author's response. However, the responses do not address my concerns, so I maintain my scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.