acceptodds
Under review as a conference paper at ICLR 2027

Calling Is Not Using: Do Post-Training Gains Increase Dependence on Tool Returns?

Abstract

Tool-using language models are judged by task success, which does not separate using a tool from using what the tool returns: a model can score higher by calling more often, executing more reliably, and stopping in a scorable format, none of which requires reading a return. We trace interactions through five stages: routing, execution, acquisition, integration, and termination. Only integration leaves no trace, since a response can agree with a return without being caused by it; we measure it by intervention instead. Replaying each episode twice from the same history and first action, Natural replay preserves tool returns while matched-null replay replaces every executed return with a length-matched uninformative observation. Their paired accuracy difference measures the return’s contribution to task success, and its change relative to the frozen base model is what post-training did to that dependence. Across our training grid, returns are worth most on the interface a model trained against: elsewhere accuracy rises, but most of the gain survives deleting them. Transfer does occur: among the single-source settings, supervised fine-tuning on mathematics yields the only confirmed increase on a debugging interface where every policy starts from the same failing program. But acquiring a return is not using it: on a diagnostic answerable only from the retrieved passage, all three Search-SFT runs retrieve the answer in every example yet answer only 7.3% of items correctly; given it in context, they are almost always right. Scoring well, acquiring information, and using it are three different things; tool-use transfer should be tested by the contribution of returns to success, not inferred from task performance.

Primary area:
reinforcement learning
Keywords:
language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1D4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether post-training gains in tool-using language models reflect greater dependence on tool returns. It distinguishes routing, execution, acquisition, integration, and termination, and introduces paired replay from the same recorded history and first action. Natural replay preserves ordinary returns, while matched-null replay replaces every executed return with a token-length-matched uninformative observation. The accuracy difference measures intervention-specific return dependence, and its change relative to the frozen base model measures the training-induced change.

The primary experiment crosses Code, Math, and Search training with SFT, GRPO, and on-policy distillation, using three training runs per cell. Additional experiments include fixed-initial-opportunity CodeDebug evaluation and a private-fact diagnostic. The results show that endpoint accuracy, acquisition, and return dependence can move separately.

Critical strengths

1.The paper addresses an important ambiguity with an explicit intervention rather than inferring information use from tool-call counts or answer agreement. Replaying a common first action and replacing subsequent returns prevents simple recovery through additional calls. The reported contrast is reproducible and has a clearly stated operational interpretation.

2.The supporting experiments go substantially beyond a single return-ablation result. CodeDebug equalizes the initial program, action, and diagnostic across policies, while the private-fact diagnostic shows that Search-SFT retrieves the answer on every item but answers only 7.3% correctly, versus 96.8% under direct placement. Three training runs per primary cell, additional SFT runs on CodeDebug, and comparator sensitivity analyses strengthen the distinction between receiving information and benefiting from it.

Critical weaknesses

1.The interpretation remains broader than semantic integration. The repeated-token null is visibly abnormal and changes subsequent behavior: the retrospective Search-to-Math audit reports 2.06 queries per item under null versus 1.46 under Natural. Because all subsequent returns are replaced and unscorable endings receive zero, the accuracy contrast includes changes in later calling and termination. Furthermore, the positive Search-SFT-to-Math effect is concentrated on Base-unrouted items, while the Base-routed estimate is approximately zero. The paper acknowledges these boundaries, but stronger separation of information use from intervention-induced behavioral disruption would improve the central interpretation.

2.The generality and statistical resolution are limited. The primary student is Qwen2.5-3B, evaluation panels contain approximately 200 items, and confirmatory tests use only three training-run effects with two degrees of freedom. The Math-SFT-to-CodeDebug adjusted p-value of .0489 should therefore be interpreted cautiously, although the additional SFT runs support the direction. Training procedures also differ in tokens, compute, and visited-state diversity, while the family and teacher extensions change coupled factors. These experiments support conditional observations rather than a general ranking of post-training methods.

Questions for authors

1.Can Help/Harm outcomes be cross-tabulated with changes in whether a scorable answer is submitted? This would clarify how much of the measured dependence is associated with termination changes.

2.Is a schema-preserving, uninformative comparator available for CodeDebug under the same frozen evaluation contract? Its effect on the Math-SFT result would materially inform the interpretation of the fixed-opportunity finding.

3.Can the Math-SFT-to-CodeDebug effect be reported separately for runtime-error and wrong-output diagnostics? This would help determine whether the result is concentrated in one feedback type.

1 comment

Thank you for your detailed review and specific questions. We run each task twice from the same starting point: once with the real tool outputs, and once with every output replaced by filler (one token repeated to the same length; the "matched null" in our paper). We call these two runs "replays". If accuracy drops, the model was relying on the tool outputs. T is the trained model's accuracy drop minus the base model's drop.

W1: The measure is broader than integration. U, the accuracy drop under filler, includes effects on later calls and on whether the model answers (Section 3.2). For SFT and GRPO models trained on Math or Search, T stays positive in every training run on their own tool when filler is replaced by real outputs from other items. In the original filler comparison, most of T comes from items where both replays answer. On CodeDebug, Math-SFT's T stays positive under the format control (see Q2 below).

W2: Generality, statistical power and compute.

  • Generality. Without further training by us, Qwen2.5-14B and Qwen2.5-72B also rely heavily on tool outputs: filler lowers their Math accuracy by 51.5 and 31.0 points. We also train students from three model families (Section 4.5).
  • Statistical power. With 500 items per task instead of 200, all eight tested comparisons keep their direction, and on the training tool all intervals stay above zero. For the CodeDebug result (adjusted p = .0489), Math-SFT is positive in all five training runs (Appendix E.1) and under the format control (see Q2 below).
  • Compute. Our methods use the same number of prompts, trajectories and updates (Appendix B). We compare each method as a whole, including its token use and compute. The model-family and teacher comparisons show how results vary across settings (Section 4.5). We will state this in Section 3.4.

Q1: Help/Harm by whether a program is submitted. The table covers the 199 CodeDebug items. Columns group items by which replay submits a program. Each cell gives the number of items, then Help / Harm (Help: correct with the real diagnostic, wrong with filler; Harm: the reverse). For example, "76; 29/0" means that on 76 items only the real-output replay submits, and its program is correct on 29 of them.

ModelBoth submitOnly real-output replay submitsOnly filler replay submitsNeither submits
Base4; 1/049; 8/09; 0/1137; 0/0
Math-SFT run 158; 8/176; 29/013; 0/052; 0/0
Math-SFT run 245; 8/183; 36/012; 0/059; 0/0
Math-SFT run 347; 5/286; 37/09; 0/057; 0/0

Math-SFT's T on CodeDebug is 15.9 points (average of three training runs): 13.6 points come from items where only one replay submits, and 2.3 from items where both do. So with filler, Math-SFT submits a program less often. When we score the last program written, submitted or not, T stays positive in all three runs. The appendix will give these groups for every Math and Search cell.

Q2: An uninformative output that keeps the diagnostic format. Our format control replaces only the first output. It keeps the diagnostic's layout (output blocks, field names, success flag and Traceback header) and replaces its text with filler of about the same length; later outputs stay real. Math-SFT's T is +9.0 points (interval [3.2, 15.1]) and positive in all five training runs. So T stays positive when the first output keeps the diagnostic format.

Q3: Runtime errors and wrong outputs. T by diagnostic type, from the original filler comparison:

DiagnosticItemsT95% interval
Runtime error81+26.7[+13.2, +40.7]
Wrong output118+8.5[+3.1, +14.4]

T is positive in all three training runs for both types and about three times larger for runtime errors.

Reviewer_HvYcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether post-training improves models’ use of tool returns or merely their task accuracy. It proposes paired replay with natural versus length-matched uninformative returns, measuring the accuracy gap and its change from the base model. Across training settings, these quantities diverge. The motivation is strong, but the method establishes intervention-specific task utility more directly than semantic integration.

Critical strengths
  1. The paper tackles an important evaluation gap: calling a tool and retrieving useful information do not necessarily mean using it. The nonce experiment makes this distinction especially clear.
  2. Replaying from the same history and first action gives a well-controlled comparison. The dependence metric and its decomposition of accuracy gains are mathematically sound.
  3. The controls are thoughtful. CodeDebug gives models the same initial feedback, while alternative returns and direct-context experiments help rule out some competing explanations.
Critical weaknesses

W1. The accuracy gap measures how much returns help, but does not isolate content integration. Repeated-token replacements can also trigger extra calls or failures to finish. Could the authors extend the plausible-return controls and report termination changes separately to clarify what drives the gap?

W2. Transfer remains entangled with routing. Search-SFT’s positive Math effect comes almost entirely from items where the base model never calls, giving limited evidence of better integration under equal access. Authors are encouraged to add a Math or Search comparison with the same initial action and return across policies, as in CodeDebug.

W3. Most comparisons rely on three training seeds and roughly 200 items, making significance sensitive to estimated seed variance. The extra SFT runs help, but broader patterns remain uncertain.

The motivation and evaluation framework are valuable, but the evidence does not yet cleanly separate improved integration from routing and reactions to return replacement. I'm looking forward to see more direct evidence in rebuttal phase.

Questions for authors

please see weaknesses

1 comment

Thank you for your careful review and concrete suggestions. We run each task twice from the same starting point. One run receives the real tool outputs. In the other, every output is replaced by filler: one token repeated to match the output's length. We call this the "matched null" in our paper. If accuracy drops, the model was relying on the tool outputs. We call the two runs "replays". U is the accuracy drop between them, and T is how much larger U is for a trained model than for the base model. Tool outputs can change later calls and whether the model gives an answer, and U includes their effects on final accuracy (Section 3.2). Section 4.4 reports the call changes. A "training run" is one training seed.

W1: Realistic replacement outputs and stopping.

Realistic outputs. Here each tool output is replaced by a real output from another item, with the same token length and without the answer. In all four SFT and GRPO settings, T stays positive in every training run and is at least as large on average as with filler (Table R1).

Table R1. T on the trained tool, with filler vs. with other items' real outputs (percentage points, 3 training runs). The interval is for the other-item column.

Training → testWith fillerWith other items' real outputs95% interval
Math-SFT → Math+21.7+26.8[+20.0, +34.0]
Math-GRPO → Math+17.5+27.0[+19.3, +34.7]
Search-SFT → Search+15.0+18.8[+11.5, +26.2]
Search-GRPO → Search+13.3+15.8[+5.5, +25.5]

So the increase on the trained tool does not depend on using filler.

Stopping. In the original filler comparison, most of T in these four settings comes from items where the model answers in both replays (all four answer groups will be in the revised appendix).

Extra calls. Math-GRPO run 2 never makes more calls with filler than with real outputs, and Search-GRPO run 1 does so on only 3 of 200 items. Both runs still show a clear increase, so extra calls cannot explain the increase in these runs.

W2: The same first action and output across models. On Math, Base and five Search-SFT training runs start from the same history, first call and first output. Only that first output is replaced by another item's output; later outputs stay real.

On the 186 items with a non-empty first output, Search-SFT's T is −5.1 points and negative in all five training runs. So, given the same call and output, Search-SFT shows no clear increase in reliance over Base. This agrees with Figure 3, where Search-SFT's Math gain comes mostly from items on which the base model does not call the tool. Together, these results point to more frequent calling as the main source of Search-SFT's Math gain.

W3: Few training runs and small test sets. Table R2 uses all 500 items of the fixed Math and Search validation sets. The 300 added items overlap with neither the original 200 items nor the training data.

Table R2. T with filler on 500 items (as in Table R1; percentage points, 3 training runs; intervals resample runs and items).

Training → test500 items95% interval
Math-SFT → Math+21.2[+16.6, +25.7]
Math-GRPO → Math+17.2[+11.4, +23.8]
Search-SFT → Math+10.6[+6.2, +14.9]
Search-GRPO → Math−1.5[−10.8, +6.4]
Math-SFT → Search−1.5[−5.7, +2.6]
Math-GRPO → Search−13.4[−16.7, −10.2]
Search-SFT → Search+13.6[+8.9, +18.3]
Search-GRPO → Search+12.4[+6.3, +18.3]

Every row has the same sign as on the original 200 items, and on the trained tool all four intervals stay above zero. On the original 200 items, the main settings also have five training runs: Search-SFT → Math is positive in all five, and Math-GRPO → Search is negative in all five.

Reviewer_P7vDOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to study whether an LLM agent effectively uses the results returned by the tool. The authors conduct controlled experiments by inserting a special string in tool results or putting the special string in context. The results show that only 7.3 correctly answer questions based on this special string when it's put in the tool call while 96.8 when it's put in the agent context.

Critical strengths

None

Critical weaknesses

This paper is targeting an interesting question, yet I have major concerns about the motivation

  1. The paper says "a model can score higher by calling tools more often....". Why is that the case? Many training use the outcome correctness as reward signals for training agentic LLMs. Does this setup reflect a training bias in the authors' reward signal setup?
  2. I don't understand why the authors need to train models to demonstrate the study? For example, if the goal is to study whether the LLM can effectively use tool results, why not use existing LLMs and do the controlled experiments? Why is training required? Is this a generally meaningful finding for most LLMs or does it only hold for the models trained by the authors?
  3. What is the difference between 'tool results' and 'contexts' in an agent? My understanding was that an agent's context is the history of tool calls and results.
  4. I have difficulty understanding the flow of the paper. The presentation is a bit hard to follow. For example, this sentence is not clear to me "Replaying each episode twice from the same history and first action, Natural replay preserves tool returns while matched-null replay replaces every executed return with a length-matched uninformative observation. Their paired accuracy difference measures the return’s contribution to task success, and its change relative to the frozen base model is what post-training did to that dependence."
Questions for authors

Questions:

  1. Many agents are trained by outcome reward. Why do the authors say a model can score higher by calling tools more often?
  2. Why is training models required in this study? Why not studying existing LLMs? Does the conclusion generalize to other models?
  3. In Fig1, what do these stages mean? How is it related to the study?
  4. What is the difference between a string put in tool call results and contexts?
1 comment

Thank you for your review and questions. We run each task twice from the same starting point. One run receives the real tool outputs. In the other, every output is replaced by filler: one token repeated to match the output's length. We call this the "matched null" in our paper. If accuracy drops, the model was relying on the tool outputs.

W1/Q1: Outcome reward and more calls. We use final-answer correctness as the GRPO reward, and SFT copies teacher solutions that reached a correct answer. A reward for correct answers checks the result, not how the model got there. Accuracy can rise because the model calls the tool more often, reasons better on its own, or formats its answer correctly more often, and none of these requires better use of the tool output. For example, Math-SFT raises Search accuracy from 11.5% to 28.2%, but its accuracy drop under filler is no larger than the base model's (Table 12). That is why we measure accuracy and reliance on tool outputs separately. We will make the abstract sentence clearer.

W2/Q2: Why train models? What about existing LLMs? We train models because our question is what post-training changes; the test itself needs no training. It also applies directly to existing models. We tested Qwen2.5-14B-Instruct and Qwen2.5-72B-Instruct without further training by us, on 200 items per task. The table gives accuracy with real outputs, accuracy with filler, and their difference, U. Five smaller models will be in the appendix.

ModelTaskAccuracy with real outputsAccuracy with fillerDrop (U, points)
Qwen2.5-3B (our base)Math14.5%2.0%12.5
Qwen2.5-14BMath55.0%3.5%51.5
Qwen2.5-72BMath60.0%29.0%31.0
Qwen2.5-3B (our base)Search11.5%0.5%11.0
Qwen2.5-14BSearch38.5%18.0%20.5
Qwen2.5-72BSearch46.0%27.0%19.0

The test works on models we did not train, and these models also rely heavily on tool outputs. It also shows how they differ: on Math, most of the 14B drop comes from items where the model answers in only one of the two runs, while most of the 72B drop comes from items where it answers in both runs. For training effects across model families, see Section 4.5 and Tables 5–6.

W3/Q4: Tool outputs and context. Tool outputs are part of the context. In the private-fact test, the natural condition requires the model to retrieve the key, and retrieval can fail. In the oracle condition, we supply the key inside a tool output. We will revise the phrase "directly in context" to say this.

Q3: The stages in Figure 1. Using a tool has five stages: Route: does the model call a tool? Execute: does the call run? Acquire: does the output contain the needed information? Integrate: does the model use that information? Stop: does the model give an answer that can be scored?

Route, Execute and Stop can be read from the transcript. Acquire can be checked on tasks where we can tell whether the output contains the needed information. Integrate leaves no trace, so we replay each task with and without the real outputs to measure how much they contribute to success (see W4). For example, Search-SFT's Math gain is concentrated on items where the base model does not call the tool (Route). When we give Base and Search-SFT the same first call and the same output, Search-SFT shows no clear increase in reliance over Base.

W4: A simpler explanation of replay. Suppose a model runs a Python calculation. We save the question, the history and that call, then continue twice: once with the real output, and once with filler in place of this and every later output. U is the difference in final accuracy between the two replays, and T is how much U changes after training, compared with Base. We will add this example before the notation and use it in the abstract.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.