acceptodds
Under review as a conference paper at ICLR 2027

Action-Space Shaping For LLM Agents: Measuring and Mitigating Tool-Schema Bias

Abstract

Large Language Models (LLMs) have shown strong performance on tool-use agentic tasks when given a fixed tool schema. Yet a tool schema is not the action space of an agent; it is merely one interface representation of it. The same executable action can be exposed through many different, functionally equivalent tool definitions, and an agent that has truly learned a task should behave consistently across them. We show that current agents often do not, a phenomenon we term schema bias. To study this systematically, we introduce an executable transformation framework that rewrites a native tool schema using nine operators, including merging and splitting tools, altering how a single tool is expressed, and distributing one action across several dependent calls. The tasks, executable actions, and reachable states remain fixed, so any change in success is attributable to the interface alone. Evaluating eleven LLMs, including two closed models, on up to 32 schema variants, we ask how large schema bias is, how it manifests, whether the difficulty of a schema variant can be predicted without a full evaluation, and whether training removes it. We find that schema bias is substantial even for the newest models: success rates range from complete failure to depending solely on the schema. To reliably estimate schema difficulty, it requires running a small sample of the target queries. Training repairs a schema variant only when that variant appears in the training data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Tool-use, Agent, schema, bias

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ksTZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether LLM agents remain robust when the tool schema changes but the underlying native actions and reachable environment states remain the same. It introduces nine schema transformation operators, validates transformed schemas by decoding calls back to the same native actions, and evaluates 11 LLMs across up to 32 schema variants. It also analyzes failure modes, tests methods for estimating schema difficulty, and compares prompting, constrained decoding, SFT, and GRPO as ways to reduce schema sensitivity.

Critical strengths
  1. The executable equivalence checks make the main comparison substantially more controlled. The authors keep the tasks and target native actions fixed and validate that each transformed schema can execute the same native actions and reach the same states. This is a useful design for isolating the effect of tool-interface representation from changes in task semantics.

  2. The paper provides a broad and systematic measurement of schema sensitivity. It evaluates many model–schema combinations, categorizes recurring failure modes, and tests multiple mitigation strategies. The findings that schema effects are model-dependent, and that some interface changes are much more disruptive than others, are useful for understanding tool-use evaluation.

Critical weaknesses
  1. The paper does not establish that its schema variants are comparably readable or usable. Functional equivalence does not imply equivalent readability or interaction difficulty, so some of the reported performance differences may simply reflect that certain variants are harder to understand or use. Several schema transformations also appear artificial and poorly grounded in real-world tool design. If they produce interfaces that are unusually awkward or opaque, the resulting sensitivity may offer limited guidance for improving tools or models in practice. The paper neither defines nor measures schema readability, nor assesses whether its more disruptive variants are plausible schema designs. Consequently, its premise that a capable agent should perform consistently across all tested schemas is not sufficiently justified.

  2. The paper does not make a strong case for model training as the practical solution. Training mainly helps with schema variants seen during training, generalizes poorly to new variants, and brings little overall improvement on real benchmarks. Simpler options, such as refining clearer instructions or better-designed schemas, can also help at a much lower cost, but the paper does not compare them with model training or discuss their relative costs. It therefore remains unclear why model training is necessary or worthwhile.

Questions for authors

See strengths and weaknesses

4 comments
AuthorsOpenReview ↗

W1: Readability, and whether the variants are plausible designs

Readability. We agree that functional equivalence does not imply equal readability, and we do not define a readability score. We address the concern in two ways.

  • Controls that hold the schema's size and information fixed. Three analyses in the paper vary only the representation while keeping the calls per action and the available information fixed, and a large gap survives each of them:
    • Grouping at fixed group sizes. Grouped merges with identical group counts and sizes differ only in which operations share a tool (semantic, random or anti-semantic). Semantic grouping beats anti-semantic grouping by 3.9–18.2 points for five of seven models.
    • Argument structure of identical tools. The same merged tools in flat, nested and union argument forms: the nested form costs every model 19–73 points, including Qwen3.5 models that lose under 7 points on the flat form.
    • Right action, wrong form. Executing off-schema calls as the native action they decode to recovers 69 points on fully merged for Qwen3-30B-A3B and 38 for Qwen3-4B, and on Qwen3-4B every recovered episode decodes to the gold native names. The model read the schema, planned the task correctly, and reverted to the convention it knows, which is not a readability failure.
  • Difficulty is a property of the model–schema pair, not of the schema. If the gaps reflected readability, a schema would be hard for every model. Instead, no variant is safe for every model, and a variant's difficulty for a given model is predicted by about 100 target queries run with that model, while schema-only probes, which see only the schema text, do not predict it (RQ3). A readability score is a property of the schema alone and could not explain this.

Plausibility. Each operator corresponds to a design decision that tool developers make for reasons unrelated to the agent.

  • Merge vs. split trades off retrieval and context budget. In retrieval-augmented tool use, the degree of flattening sets the load on the two sides: a split catalog has many small tools, which loads the retriever, whereas a merged catalog has few large tools, which loads the context window.
    • With large catalogs (e.g., aggregated MCP servers), tools are retrieved top-k. A recall-oriented deployment favours merged or dispatch tools, so that retrieving one tool exposes every operation on a resource.
    • A precision-oriented or least-privilege deployment favours split tools: separate read and write tools, or a separate endpoint above an approval threshold, which is exactly an interval split.
  • Per-tool representation follows how schemas are produced.
    • Schemas generated from OpenAPI specs commonly have nested request bodies, arbitrary argument order and missing descriptions.
    • Multi-server clients prefix tool names with the server name to avoid collisions.
  • Cross-call protocols are dictated by the domain.
    • Payments and transfers are opened, filled in and explicitly confirmed (transaction).
    • Records are addressed by IDs obtained from a lookup (reference resolution), as in τ²-bench's own airline and retail tools.
    • Clients increasingly load tool schemas on demand through a search tool (schema discovery).

Our premise is therefore not that every variant is a good design. It is that each is one a reasonable developer could ship, that the agent does not choose which one it receives, and that its sensitivity to this choice is a deployment risk worth measuring. On four real benchmarks the same variants are costly: fully merged costs Qwen3.5-27B 0.26–0.67 on τ²-retail, BFCL, AutomationBench and MCP-Atlas. The new Appendix C.3 (Where each operator arises in practice) names a design decision behind each of the nine operators in a table.

AuthorsOpenReview ↗

W2: Why training, compared with cheaper fixes?

We agree that training is not a general remedy for schema sensitivity, and the paper does not present it as one. It is the costliest of the four levers we compare and the only one that repairs the protocol-level variants; the cheaper levers come first. RQ4 already compares the two cheapest fixes, a one-sentence instruction and constrained decoding of the first call, side by side with SFT and GRPO on the same twelve variants (Table 3), and RQ3 covers schema choice. The methods differ in cost and in coverage, and this difference is one of our findings:

MethodCostWhat it repairs (Qwen3-4B unless noted)What it cannot repair
Schema choice (RQ3)About 100 target queries per model; no change to the modelLets a designer rank candidate schemas before deploymentApplies only when the designer controls the schema; redone per model
InstructionOne sentence per variantConventions that can be stated as a rule: fully merged +0.28, schema discovery +0.45Multi-call protocols: transaction +0.06
Constrained decodingDecoder accessInvalid tool names: fully merged +0.29; 0.07 → 0.74 on Qwen3-30B-A3BCannot enforce a protocol: transaction −0.11
SFT (mixed data)Verified demonstrations of each target variant, plus training computeWhat the cheap fixes cannot: transaction +0.23, schema discovery +0.67Taxes robust variants by 6–10 points; no transfer to unseen operators
GRPO (mixed data)The highest: on-policy rollouts, plus an executable environment that rejects invalid callsThe largest gains, without tax: transaction +0.34, schema discovery +0.68 (the same holds on Llama-3.1-8B, reply to PY6W)No transfer to unseen operators; on real benchmarks +0.03 on trained variants and +0.003 on untrained ones

We therefore recommend a ladder, not training by default:

  1. Choose or redesign the schema when you control it.
  2. Use an instruction or constrained decoding when the failure is a describable convention or an invalid call.
  3. Reserve training for protocol-level variants, and for a model that must serve many schemas it does not control.

The limited transfer of training is itself a finding: training removes the biases it targets, not schema sensitivity in general. The revision adds this cost comparison as a table in the new Appendix G.8 (Cost and coverage of each mitigation), and §8 (RQ4) now closes with the ladder above in one sentence.

Thanks to the authors for their rebuttal. I agree that agents can be sensitive to tool schemas, but I remain unsure how significant this issue is in practice, particularly when developers can address some of the observed failures through careful schema design and prompt testing. I also see the value of anti-class grouping as a stress test, but it remains unclear how often deployed tools are grouped this way. For these reasons, I maintain my original rating of 2.

Thank you for the follow-up. We would like to restate what the paper claims, because we think this is where we differ.

1. What the paper claims. Not that schema sensitivity is unfixable in deployment, but that it exists, is large and structured, and has so far gone unmeasured. A tool schema is one representation of the action space, not the task, and an agent that has learned the task should behave consistently across equivalent representations, as we expect for prompt wording. Our contribution is to bring this to attention and to investigate it in depth: a framework that verifies every variant against native actions and final states, measurement on eleven models and up to 32 variants, a failure taxonomy of how each representation breaks an agent, an estimator of schema difficulty, and a comparison of mitigations with their transfer. We see this as the core value of the paper: before a problem can be designed around, it has to be visible and understood.

2. Why careful design does not remove the issue.

  • Benchmarks and training environments commit to one schema. Rewriting the tools of four real benchmarks changes the same model's score by up to 0.67, and the most costly representation differs across model families. Benchmark scores and training gains therefore partly reflect the interface, whoever designed it.
  • Which design is safe depends on the model. No variant is safe for every model; class dispatch, the grouping a careful designer would choose, still costs Qwen3.5-27B 0.29–0.42 on three real benchmarks and gpt-6-luna 13 points. Choosing well requires measuring per model, and RQ3 provides that measurement. A one-sentence instruction (Table 3) repairs conventions a sentence can describe, not the transaction protocol (+0.06).
  • The agent developer often does not own the schema. Tools from MCP servers and aggregated catalogs arrive with their schema, and one model serves many of them. Training repairs only the changes it has seen, so robustness cannot be read off the score on one interface.

3. On anti-class grouping and how often deployed tools look like our variants. We cannot enumerate how schemas are designed in practice, and the paper does not try to. The variants are representative examples, each produced by an operator that a real design decision can produce, and their purpose is to expose a behaviour pattern: the same actions, presented differently, succeed or fail depending on the model. Anti-class grouping is one point on this spectrum; the pattern it reveals, that a merged tool's cost comes from which operations it contains, appears in the same direction at the deployed endpoints, class dispatch and fully merged. The value of the examples lies in showing that the bias exists and how it behaves, not in matching the frequency of any one design.

We thank the reviewer again for the engagement.

Reviewer_UYhpOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the sensitivity of LLM agents to tool-interface representations. While preserving the underlying native actions and execution outcomes, the authors modify tool schemas through merging, splitting, argument organization, and multi-call protocols, compare model performance across interfaces, and characterize the resulting differences as schema bias. The paper further analyzes the relationship between schema violations and task failures, investigates schema-difficulty prediction, and examines whether SFT, RL, and mixed-schema training improve generalization across interfaces. Its main value lies in systematically demonstrating that identical underlying executable capabilities do not imply identical model performance across tool interfaces, and that training gains on certain schemas do not necessarily transfer to unseen structural transformations. However, important concerns remain regarding the positioning relative to prior work, the interpretation of performance differences, and the generality of the training comparisons.

Critical strengths
  1. The paper systematically studies structural changes to tool interfaces, going beyond simple perturbations of names or descriptions. Merge, Split, and multi-call protocol transformations cover practically relevant design choices in real tool systems. The substantial interactions between models and transformation types help explain why tool-interface design affects agent performance and offer useful insights for tool-system design and model evaluation.

  2. The paper examines both failure types and transfer after training, rather than merely reporting performance drops under interface changes. The schema-form analysis helps identify failures in which a model can express the correct underlying action but does not comply with the active interface. The SFT, RL, and mixed-schema training experiments further examine whether training can mitigate these issues. In particular, the limited generalization to unseen transformations gives the study value beyond static robustness evaluation.

Critical weaknesses
  1. The omission of the directly relevant prior work PIPE affects the positioning of the core contributions. "What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?" (arXiv:2602.01611) already uses interface perturbations that preserve underlying environment semantics to investigate whether agent success reflects transferable task competence or dependence on particular interfaces. That work also discusses interface reliance after trajectory training and failure behaviors such as repeatedly invoking obsolete tool names. The present paper's core motivation, its evaluation paradigm of changing interfaces while preserving underlying semantics, and some of its observed failure behaviors are directly related to PIPE, yet the manuscript neither cites nor discusses this work. Consequently, the originality of presenting robustness to interface representations as a newly identified dimension of agent generalization is insufficiently delineated. The authors should clarify which questions have already been raised by prior work on interface reliance and which experiments or findings are new to this paper.

  2. Execution equivalence does not distinguish interface reliance from additional interaction complexity, limiting the interpretation of the results. Verifying that transformed reference trajectories decode to the same native actions and reach the same environment states establishes preservation of underlying execution semantics. It does not establish that the model faces identical demands in understanding, planning, and execution. For example, Transaction, Schema Discovery, and Reference Resolution may introduce additional calls, ordering constraints, intermediate-state tracking, or information-retrieval steps. A model may therefore fail because of these additional demands even when it correctly understands the task goal. The experiments support the claim that agent performance depends on how the action space is represented, but they do not generally establish that performance drops arise from shallow reliance on familiar interfaces, nor do they alone demonstrate a lack of transferable task understanding. The schema-form analysis offers a partial diagnosis but does not sufficiently account for the sources of failure across all transformations.

  3. The applicability of the training-method comparison across models remains insufficiently validated. The main SFT, GRPO, and ablation experiments focus on Qwen3-4B. Although the appendix includes an additional mixed-data SFT experiment on Qwen2.5-7B, that model improves on every evaluated variant, with native-interface success increasing from 0.54 to 0.78, rather than exhibiting the same SFT performance cost observed in the main experiments. Thus, the finding that Mixed RL better preserves performance on other interfaces than SFT is currently supported primarily by results on Qwen3-4B. Its applicability to different base models and initial capability levels remains unclear.

Questions for authors
  1. Please compare your work with PIPE and clarify the additional contributions concerning the research question, interface-transformation framework, failures involving obsolete interface calls, and training-related findings. After accounting for this prior work, would you revise the claims concerning the originality of interface robustness as a dimension of agent generalization?

  2. Can the authors provide more direct controlled evidence distinguishing performance drops caused by reliance on the original interface from those caused by the additional understanding and execution requirements introduced by the new interface? The current reference-trajectory validation establishes equivalence of underlying execution semantics, but transformations such as Transaction, Schema Discovery, and Reference Resolution may also increase the number of calls, ordering constraints, and state-tracking requirements. Can the existing experiments rule out or quantify the effects of these factors? If not, please clarify which results support an interpretation in terms of interface reliance and which establish only that models differ in their ability to adapt to different interaction protocols.

3 comments
AuthorsOpenReview ↗

W1 / Q1: Relation to PIPE (arXiv:2602.01611), and the originality claim

Thank you for the careful review. We accept all criticisms.

We thank the reviewer for the pointer. We revise the originality claim: PIPE already asks whether agent success reflects task competence or dependence on a particular interface, so we no longer present interface robustness as a newly identified dimension. The introduction and Related Work now credit PIPE with the question.

However, our work and PIPE differ in what are changed. PIPE perturbs only the surface form of the interface: tool names. We change the structure of the action space: how operations are grouped into tools, how arguments are organised, and how many calls an action takes. These structural changes, which PIPE does not test, produce the large losses we report, and they are the basis of the findings below. We delineate the contributions along the four axes the reviewer names.

Interface-transformation framework. PIPE perturbs the surface form only: each action identifier is replaced 1:1 by a synonym or a meaningless symbol ("we only perturb the surface forms of action/tool identifiers"), while the tools, their arguments, and the calls per action stay fixed. In our framework this is one operator out of nine (rename), and on it the two works agree: renaming costs most untrained models little. The other operators change the structure of the action space: grouping (merge, split), argument organisation (nesting), and calls per action (transaction, reference resolution, schema discovery).

More findings new to this paper.

  1. Interface bias without fine-tuning. PIPE finds untrained models robust to renaming (Δ≈0) and attributes brittleness to trajectory SFT. Structural changes break off-the-shelf models: fully merged drives Qwen3-30B-A3B to 0.07, and gpt-6-luna loses 13 points on class dispatch.
  2. Obsolete calls before any training. PIPE traces calls to legacy names to SFT. We observe them in untrained models, and decoding such calls back to the native action recovers up to 69 points, which points to conventions learned in pretraining.
  3. A method to estimate schema difficulty. No schema is safe for every model, so we propose estimating a schema's difficulty for a given model before deployment: about 100 target queries run under the schema predict it reliably, while schema-only probes do not (RQ3).
  4. Intervention. PIPE only evaluates. We compare instructions, constrained decoding, SFT and GRPO on native and on mixed schemas, and test transfer to held-out operators and real benchmarks.
AuthorsOpenReview ↗

W2 / Q2: Interface reliance vs. additional interaction demands.

We agree that execution equivalence does not establish equal demands, and the answer differs by operator. The revision states the claim separately for the two groups (§3 and §5; the text is in the reply to kviF W1):

OperatorsCalls and informationEvidenceWhat we claim
Merge/split, arguments, renaming, descriptionsMatched: same calls per action, same operations exposedSame calls with different grouping (semantic beats anti-semantic by up to 18 points); identical tools in different argument forms (19–73 points); schema-form decoding (right native action, wrong form)Interface reliance
Transaction, schema discoveryExtra calls and orderingBelowProtocol adoption only

For the cross-call operators we cannot rule out added demands, so we no longer call these losses interface reliance. We can, however, locate where the failures occur. If the added planning and state tracking were the cause, failures would concentrate on tasks that are also hard on native, and would occur after the task steps begin. Neither holds:

  • Failed tasks are no harder than average. For the models that collapse on these variants (success ≤ 0.29), 49–75% of their failed transaction tasks are tasks the same model solves on native, against a native success rate of 54–77%. For schema discovery the two numbers are 53–76% and 54–78%. The failures fall on easy and hard tasks alike.
  • Failures occur at the protocol, before any task step. In 55–96% of these transaction failures no native action is executed at all. The typical error is a protocol error, such as writing to an unopened transaction or nesting begin calls.
  • Exposure to the protocol, on the same tasks, repairs it. On Llama-3.1-8B, SFT on the same training tasks gives 0.01 on transaction with native schemas and 0.56 with mixed schemas. The tasks are identical; only the protocol seen in training differs.
  • The turn budget is not the bottleneck: doubling it from 16 to 32 turns recovers at most 1.8 points.

For these two operators we therefore claim only that models differ widely in adopting an unfamiliar but execution-equivalent protocol, not that they lack task understanding. The revision adds this analysis as Appendix E.4 (Task-level or protocol-level failures on the cross-call operators), next to the turn-budget control in Appendix E.3.

AuthorsOpenReview ↗

W3: Training comparison beyond Qwen3-4B, and initial capability.

To address reviewer's concern, we include a new base model for training-based intervention experiment. We replicated the full training comparison on Llama-3.1-8B-Instruct, with the same training set and setup as for Qwen3-4B: SFT and GRPO, each on native-only and on mixed data, evaluated on the same twelve variants. Success on the variants discussed below (full table in the new Appendix G.7):

BaseSFT native (2 seeds)SFT mixed (4 seeds)RL nativeRL mixed
Native0.660.580.780.790.81
Fully merged0.630.010.760.750.77
Class dispatch0.610.010.630.490.64
Nested args0.660.480.350.780.80
Transaction0.100.010.560.000.57
Schema discovery0.040.000.630.040.53
Mean of 12 variants0.520.300.700.590.71

The findings replicate:

  • The SFT tax. Native-only SFT collapses fully merged (0.63 → 0.01) and class dispatch (0.61 → 0.01). Mixed SFT repairs transaction (+0.46) and schema discovery (+0.59) but taxes nested args (0.66 → 0.35).
  • Mixed RL repairs without the tax. Mixed RL from the base model repairs transaction (0.10 → 0.57) and schema discovery (0.04 → 0.53) and avoids the nested-args tax (0.80).

On initial capability, the two additional models cover both directions:

  • Low initial capability on the protocols. Base Llama rarely succeeds on transaction (0.10, vs. 0.28 for Qwen3-4B) or schema discovery (0.04 vs. 0.07), yet mixed RL from the base model still learns both. A low starting score does not prevent the repair.
  • Low initial capability overall. Qwen2.5-7B starts far from its ceiling (native 0.54), so the same mixed SFT data raise every evaluated variant, as the reviewer notes. The SFT tax appears on the two models that start higher on native, where the mixed data have little to add on the robust variants.
Reviewer_kviFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether agents behave consistently when the same actions are exposed through different tool schemas. Nine rewrite operators produce 32 variants, each machine-checked to execute the same native actions and reach the same states, so any change in success comes from the interface alone. Eleven models, including two closed ones, are evaluated on a synthetic environment with real-benchmark checks. Success ranges from near zero to near perfect; merging tools hurts more than splitting and multi-call protocols hurt most; about a hundred target queries predict a variant's difficulty while schema-only probes do not; and training repairs only the variants it sees.

Critical strengths
  1. The framework is executable and verified, which is what makes the comparison meaningful. Every variant is checked to reproduce the same native actions and final states, and about fifty thousand reference-trajectory checks caught three implementation errors before any model was run. Interface-sensitivity work usually varies wording without this guarantee.
  2. The scope and the honesty are both unusual. Eleven models, up to 32 variants, a failure taxonomy with a blinded audit, real-benchmark confirmation, and negative results about transfer that a weaker paper would have left out, namely that repairs carry over to new combinations of trained changes but not to new kinds of change.
Critical weaknesses
  1. Equivalence is defined at the execution level, not at the information level. A single tool with twelve operations behind a discriminator, or a transaction protocol the model must infer from a description, is genuinely harder to use, not merely differently formatted, so part of the measured gap is task difficulty rather than arbitrary interface bias. The paper partly separates the two through its schema-form analysis and shows that reference resolution costs nothing, but the headline claim that variation is bias needs one control in which variants are matched on the number of required calls and on the information given.
  2. The headline numbers come from a synthetic environment of 2,748 templated queries over twelve domains, while the real benchmarks are used on six variants in two domains. The synthetic queries were rewritten by a language model under a grounding check, and their difficulty relative to the benchmarks is not established.
Questions for authors
  1. Can you construct two variants that require the same number of calls and give the model the same information, and measure the remaining gap? That would separate interface bias from interface difficulty.
  2. How many failures are livelocks at the turn limit, and is the turn budget identical for one-call and multi-call variants? If the budget is the same, the cross-call results partly measure the budget.
  3. How were the templated queries validated, and do the native-schema success rates agree with published numbers for the same models on the same benchmarks?
3 comments
AuthorsOpenReview ↗

W1 / Q1: Separate interface bias from interface difficulty.

Thank you for the constructive review and the specific controls you proposed.

We agree that execution equivalence does not make every variant equally demanding. In the revision we therefore separate the nine operators into two groups: the single-call operators (the first two categories of Table 1) and the cross-call operators (the third).

1. Single-call operators (merge/split, argument structure and order, renaming, descriptions). These keep the calls per native action and the exposed operations fixed, and give the model no information that the native schema does not also give. For them we have the matched comparisons the reviewer asks for:

  • Grouping at fixed group sizes (Appendix D.2). Grouped merges with identical group counts and sizes differ only in which operations share a tool (semantic, random or anti-semantic). Semantic grouping beats anti-semantic grouping by 3.9–18.2 points for five of seven models.
  • Argument structure of identical tools (Appendix D.1). The same merged tools in flat, nested and union argument forms. The nested form costs every model 19–73 points, including Qwen3.5 models that lose under 7 points on the flat form.
  • Right action, wrong form (Appendix E.1). Executing off-schema calls as the native action they decode to recovers 69 points on fully merged for Qwen3-30B-A3B and 38 for Qwen3-4B. On Qwen3-4B every recovered episode decodes to the gold native names: the model planned the task correctly but wrote the action in the familiar form.

In each comparison the calls and the information are held fixed, and the remaining gap is 4–18 points for grouping and 19–73 points for argument form. For these operators the gap is interface reliance.

2. Cross-call operators (transaction, schema discovery). These add calls and dependencies, so we no longer call their losses interface reliance. For these operators we claim only that models differ widely in how readily they adopt an unfamiliar but legitimate protocol. Two observations support this reading. The failures occur before the task's own steps begin: 55–96% of the failed episodes on these variants execute no native action at all. And a one-sentence description of the protocol recovers 0.45 on schema discovery for Qwen3-4B (Table 3), which a model that lacked the capability could not do.

Changes in the main text:

  • §3, at the end of the paragraph that introduces the three operator categories, we add:

    "This split also bounds our claim. First- and second-category operators keep the calls per native action and the operations exposed fixed, so a change in success isolates the representation of the action space. Cross-call operators add calls and dependencies, so we read their effects as how readily a model adopts an unfamiliar but execution-equivalent protocol, not as interface reliance alone."

  • §5, tool-set partitioning: "at a fixed granularity, grouping semantically related operations is consistently better …" becomes "at a fixed granularity, with the same calls and the same operations exposed, grouping semantically related operations is consistently better …".
  • §5, cross-call protocols, at the end of the paragraph, we add:

    "Because these operators add calls and dependencies, we read their losses as how readily a model adopts an unfamiliar but execution-equivalent protocol, not as interface reliance alone (§3)."

AuthorsOpenReview ↗

Q2: Livelocks and the turn budget.

We appreciate the reviewer for looking into this detail. The budget is identical, 16 turns, for every variant. Each turn may contain several parallel calls, so every protocol fits well inside it.

  • Turns used by successful episodes:
    • single-call variants: median 2 turns, 95th percentile 2–4;
    • schema discovery: 95th percentile ≤ 8;
    • reference resolution: 95th percentile ≤ 5.
  • Episodes that hit the limit are mostly loops, of three kinds:
    • repeated execution after all gold actions are already done (e.g., Gemma-4-12B on fully merged);
    • retrying a rejected non-existent name (Qwen2.5-7B/14B on fully merged and class dispatch);
    • on transaction, protocol-error loops: 87–95% of transaction livelocks carry transaction errors, such as writing to an unopened transaction or committing a missing one.

Since the reviewer's concern is sharpest for transaction, we re-ran it with twice the budget (32 turns):

TransactionSuccess, 16 → 32 turnsHit limit, 16 → 32 turnsSucceed only after turn 16
Qwen3-4B0.278 → 0.29711.6% → 6.7%1.1%
Llama-3.1-8B0.104 → 0.10626.8% → 22.9%1.8%
  • Doubling the budget recovers at most 1.8 points, against native-to-transaction gaps of 0.49 (Qwen3-4B) and 0.56 (Llama-3.1-8B).
  • Of the 319 Qwen3-4B episodes that hit 16 turns, 43% also exhaust 32 turns and only 11% then succeed.

The cross-call results therefore do not measure the budget. The revision states the budget in Appendix B.4 (Evaluation protocol) and adds this control as the new Appendix E.3 (Livelocks and the turn budget).

AuthorsOpenReview ↗

W2 / Q3: The synthetic queries and the real benchmarks

The reviewer raises three points: how the rewritten queries were validated, whether our native scores agree with published numbers, and that the real benchmarks covered only six variants in two domains, so that the difficulty of the synthetic queries relative to the benchmarks is not established. We take them in turn.

1. How the rewritten queries were validated (Q3).

  • Each query was rewritten by an LLM conditioned on its gold calls, which are never changed.
  • We then check, by string matching against the gold calls, that the argument values still appear in the rewritten text. A rewrite that loses them is discarded, and the query keeps its original wording.
  • This ensures that the text carries the information the task needs.

2. Native scores agree with published numbers (Q3). The synthetic environment has no external reference, so we compare on the real benchmarks.

  • Where the setting matches, our native scores agree with published numbers.
    • Qwen3-4B-Instruct-2507 (the checkpoint we use) on BFCL multi-turn base: 0.33 in our runs, vs. 0.265 on the BFCL leaderboard (FC mode).
    • The same model on τ²-airline: 0.26 in our runs, vs. 0.240 on its model card.
  • No published numbers exist for this model on AutomationBench and MCP-Atlas.

The revision states this comparison in Appendix B.2 (Real benchmarks).

3. Real benchmarks beyond six variants in two domains (W2). The revised paper replaces the six-variant τ² check with four real benchmarks, τ²-airline/retail, BFCL v3 multi-turn, AutomationBench and MCP-Atlas, under eleven variants, and the conclusions hold there (Full table in Appendix D.4).

  • Models: two models are evaluated on these benchmarks, Qwen3.5-27B and Qwen3-4B-Instruct-2507 (the model compared with published numbers above).
  • Setup: each model is tested under eleven of the twelve representative variants (reference resolution is not run, because its value tables come from task data), through a schema proxy that decodes every call back to the benchmark's own tool and scores it with the benchmark's own scorer.

The variants that are costly on the synthetic grid are costly here too:

  • Merging is the costliest on both models. Fully merged costs Qwen3.5-27B 0.26–0.67 on τ²-retail, BFCL, AutomationBench and MCP-Atlas, and drives Qwen3-4B from 0.17–0.33 to at most 0.08 on the same four. Class dispatch costs Qwen3.5-27B 0.29–0.42 on BFCL, AutomationBench and MCP-Atlas.
  • Schema discovery is model-dependent, as on the synthetic grid. It costs Qwen3-4B 0.12–0.31 on four benchmarks and Qwen3.5-27B at most 0.09.
  • Fully split, nested args and reorder arguments are cheap. They stay within 0.10 of native on every benchmark for both models.
  • τ²-airline (50 tasks) shows no clear pattern for either model.

A small sample of target tasks (36 per benchmark) also predicts variant difficulty on each real benchmark (RQ3, Appendix F), and the training results transfer in the same way as on the synthetic grid (Appendix G.5). Because the same variants are costly on the benchmarks themselves, our conclusions do not depend on how hard the synthetic queries are relative to them.

Reviewer_PY6WOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates tool-schema bias in LLM agents, showing that functionally equivalent tool schemas can lead to substantial differences in task success rates. The authors introduce a framework with nine schema transformation operators that preserve the underlying executable actions. They evaluate 11 LLMs across up to 32 schema variants on 2,748 tasks, with additional validation on τ²-bench and BFCL.

The results show that schema bias is substantial and model-dependent, with tool merging and multi-call protocols generally causing larger performance drops. The authors further analyze failure modes, demonstrate that small samples of target tasks can reliably estimate schema difficulty, and explore mitigation through SFT and GRPO. Training on diverse schema variants reduces bias, particularly with GRPO, but improvements generalize poorly to unseen transformation types.

Critical strengths

1.The paper introduces a structured framework with nine transformation operators to study how different tool-schema representations affect LLM agents. By preserving the underlying tasks and native actions and validating reference-trajectory equivalence, the framework provides a useful foundation for systematically investigating schema bias beyond simple prompt or formatting changes.

2.The paper evaluates 11 LLMs across up to 32 schema variants and 2,748 tasks, with additional validation on τ²-bench and BFCL. The results reveal substantial model-dependent performance differences and provide useful insights into failure modes and mitigation through SFT and RL. These findings are consequential for both agent evaluation and practical tool-schema design.

Critical weaknesses

1.Although the paper evaluates schema bias across eleven LLMs, all SFT and GRPO mitigation experiments are conducted exclusively on Qwen3-4B. No training experiments are reported on larger models or other model families. This substantially limits the evidence that the reported mitigation benefits, particularly the advantages of GRPO over SFT, generalize beyond a single relatively small model.

2.The paper defines task success by exact multiset matching of executed native actions against gold actions, which does not account for execution order. However, the benchmark includes 416 compositional tasks involving dependent operations. The stated evaluation criterion therefore does not directly verify whether these dependencies are respected. Without an explicit order aware or final state correctness check for evaluated episodes, the reliability of the reported success rates on these tasks remains uncertain. This is particularly consequential because task success is the central metric supporting the paper's empirical conclusions.

Questions for authors

1.The paper states that task success is evaluated by comparing multisets of native actions, which do not account for execution order. However, the benchmark includes 416 compositional tasks involving dependent operations. Does the evaluation also check execution order or final environment states for agent-generated trajectories? If not, could this lead to incorrect success judgments?

2.Section 3.4 describes the validation procedure for schema variants, and the Reproducibility Statement reports approximately 50,000 reference-trajectory checks. However, the benchmark contains 2,748 tasks and 31 transformed schema variants, corresponding to 85,188 possible task–variant combinations. Could the authors clarify how the reported validation checks were distributed across tasks and schema variants? Were all task–variant combinations validated? If not, how were the validation cases selected, and what coverage was achieved for each transformation operator?

3.Although RQ1 evaluates 11 models, seven belong to the Qwen family, and the reported GRPO training experiments are conducted only on Qwen3-4B. Given the substantial differences in schema bias across models, what evidence supports the generalizability of the GRPO mitigation results to other model families, such as Llama or Gemma? Should the conclusions about GRPO's effectiveness be limited to the evaluated training model?

4.The paper evaluates 11 models, but does not include other high-capability frontier models such as Claude Opus or GPT-6 Astra. Given the substantial model-dependent differences in schema bias observed in Figure 2, to what extent do the authors' findings generalize to these stronger models? In particular, would the significant performance degradation observed under certain schema transformations persist in more capable frontier models?

4 comments
AuthorsOpenReview ↗

W1 / Q3: Training only on Qwen3-4B; does GRPO generalize to other model families?

We thank the reviewer for the precise questions.

We replicated the full training comparison on Llama-3.1-8B-Instruct. It uses exactly the same training set and experiment setup as in the paper. Mean success over the twelve variants:

BaseSFT native (2 seeds)SFT mixed (4 seeds)RL nativeRL mixed
Native0.660.580.780.790.81
Fully merged0.630.010.760.750.77
Class dispatch0.610.010.630.490.64
Nested args0.660.480.350.780.80
Transaction0.100.010.560.000.57
Reference resolution0.500.430.850.560.78
Schema discovery0.040.000.630.040.53
Mean of 12 variants0.520.300.700.590.71

(Full 12-variant table: Appendix G.7 of the revision.)

In short, the diagnosis replicates and training intevention shows similar patterns as Qwen3-4B:

  • Native data does not repair the hard variants. Under SFT or RL, the repair comes from seeing the variant during training
  • Native-only SFT overfits the interface. On Llama this is even stronger than on Qwen: fully merged falls from 0.63 to 0.01 and class dispatch from 0.61 to 0.01.
  • Mixed SFT repairs the trained variants, but taxes other variants The tax concentrates on nested args, which drops from 0.66 to 0.35 (seed range 0.23–0.50). This is one of the variants that also loses accuracy on Qwen3-4B.
  • Mixed-data RL from the base model repairs transaction and schema discovery.

The full twelve-variant table, the setup and these findings are in the new Appendix G.7 (Training on Llama-3.1-8B), referenced from the training setup in Appendix G.2.

AuthorsOpenReview ↗

W2 / Q1: Multiset scoring ignores order on the 416 compositional tasks.

We thank the reviewer for pointing out our imprecise wording. The reviewer is right: the submitted results compared the multiset of executed native actions with the gold multiset, which ignores order, and "state-based scoring" overstated what we checked.

We have therefore changed the evaluation itself, not only its description. The environment now keeps the state that executed actions produce (device settings, calendar events, an active route, a playing zone, and so on): We now make sure that where order does not change the outcome (e.g., the blinds and the fan of one room), any order passes. Where it does, a wrong order fails.

Order affects 145 of the 2,748 tasks (96 of the 416 compositional tasks and 49 compound tasks): a different order of calls reaches a different final state or is rejected by a precondition. Only these tasks can change score under the new rule; for the remaining 95%, the two scorings agree by construction. We rescored every logged episode by executing its actions in the stateful environment. Models executed the order-affected tasks in a valid order most of the time: only 1% of all episodes contain a call that the precondition check rejects, and only these episodes can change from success to failure.

  • Figure 2 (11 models × 12 variants, 119 cells): success changes by −0.45 points on average and −1.1 at most. The within-model ranking of variants is unchanged in substance (Spearman ρ ≥ 0.986 against the submitted scores, and exactly 1.0 for 7 of 11 models.)
  • Training results (Table 3, SFT and RL checkpoints): at most −1.1 points per cell.
  • Appendix controls (merge encodings, class grouping): at most −0.9 points.

Therefore, No conclusion of the paper changes. The revision reports these results and describes the rule in §4 (Experimental Setup) and Appendix B.4 (Evaluation protocol); the full-coverage check is in §3.4.

AuthorsOpenReview ↗

Q2: How were the  50,000 validation checks distributed?

We thank the reviewer for spotting this omission. We have modified this scope and it now covers all task-variant pair. The revision reports the full-coverage result in §3.4 (Validating schema variants), with the per-schema breakdown in Appendix B.5 (Validation coverage).

2,748 tasks under every schema, and all 40 transformed variants, i.e. all 109,920 task–variant episodes and 345,313 oracle calls. Each oracle trajectory runs through the same executable environment used in evaluation. An episode passes only if (a) the executed native actions equal the gold actions exactly and in the gold order, (b) no call is rejected, and (c) the final state equals the gold final state. The result is 0 failures. So our framework hold solid.

We also ran negative tests on the checker itself: dropping the last call, corrupting one argument value, and using a non-existent tool name. The checker fails all 2,748 episodes under each of the three. Both the per-schema coverage table and these negative tests are in Appendix B.5.

AuthorsOpenReview ↗

Q4: Frontier models (Claude Opus, GPT-6 Astra).

We appreciate reviewer's suggestions, but our budget does not cover the largest frontier models on the full grid. Instead, we added the newest lightweight frontier model, Claude Haiku 5.5. We ran it on all twelve representative variants over a 500 subset sample. Furthermore, original paper includes results from gpt-6-luna and GLM-5.3-Flash, and in below we show the results on the same 500 subset for comparison:

nativemergedsplitnestedstrip desc.class dispatchnamespacedtransactionschema discoveryref. resolution
Claude Haiku 5.5.746.740.738.744.746.688.740.736.716.954
gpt-6-luna.762.778.766.768.758.654.718.746.736.950
GLM-5.3-Flash.770.720.758.770.734.588.646.764.662.942

The three frontier models show consistent pattern, which is similar to the pattern of the open models. Variants that change only how a single call is written cost little or nothing. Variants that change how a call is addressed cost every model, and class dispatch is the most costly variant for all three. Reference resolution scores above native for all three. We explain each part below.

  • Single-call rewrites are nearly free. For Haiku 5.5 and gpt-6-luna, merged, split, nested and strip descriptions all stay within 0.02 of native. GLM-5.3-Flash is also similar, only with losing slightly more, 5 points, on fully merged tools.
  • Addressing variants cost every model. Class dispatch is the largest drop for each model: 5.8 points for Haiku 5.5, 10.8 for gpt-6-luna and 18.2 for GLM-5.3-Flash. Schema discovery costs 3.0, 2.6 and 10.8 points, and namespaced names cost gpt-6-luna and GLM 4.4 and 12.4 points.
  • Reference resolution is above native for every model (0.94–0.95) because its lookup step corrects wording. Argument values are scored by exact match in every variant. In native, the model writes free-text values itself, and most native failures (86% for Haiku 5.5) are wording mismatches, e.g. "demo" instead of "the demo". In reference resolution, the model first calls a lookup such as resolve(query="demo"), which returns the ID of "the demo", so the executed call carries the correct value. Adding a simple normalization (articles, time formats) to native recovers about half of these failures (native 0.88–0.93).

The cost therefore shrinks with capability, but it does not vanish, and its size per variant is model-specific (fully merged costs GLM 5 points and the other two nothing). Our claim is not that every model fails. It is that success depends on the schema in a model-specific way, so an evaluation on one schema does not certify robustness on another. We will add Haiku 5.5 to Figure 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.