- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLM agents remain robust when the tool schema changes but the underlying native actions and reachable environment states remain the same. It introduces nine schema transformation operators, validates transformed schemas by decoding calls back to the same native actions, and evaluates 11 LLMs across up to 32 schema variants. It also analyzes failure modes, tests methods for estimating schema difficulty, and compares prompting, constrained decoding, SFT, and GRPO as ways to reduce schema sensitivity.
Critical strengths
-
The executable equivalence checks make the main comparison substantially more controlled. The authors keep the tasks and target native actions fixed and validate that each transformed schema can execute the same native actions and reach the same states. This is a useful design for isolating the effect of tool-interface representation from changes in task semantics.
-
The paper provides a broad and systematic measurement of schema sensitivity. It evaluates many model–schema combinations, categorizes recurring failure modes, and tests multiple mitigation strategies. The findings that schema effects are model-dependent, and that some interface changes are much more disruptive than others, are useful for understanding tool-use evaluation.
Critical weaknesses
-
The paper does not establish that its schema variants are comparably readable or usable. Functional equivalence does not imply equivalent readability or interaction difficulty, so some of the reported performance differences may simply reflect that certain variants are harder to understand or use. Several schema transformations also appear artificial and poorly grounded in real-world tool design. If they produce interfaces that are unusually awkward or opaque, the resulting sensitivity may offer limited guidance for improving tools or models in practice. The paper neither defines nor measures schema readability, nor assesses whether its more disruptive variants are plausible schema designs. Consequently, its premise that a capable agent should perform consistently across all tested schemas is not sufficiently justified.
-
The paper does not make a strong case for model training as the practical solution. Training mainly helps with schema variants seen during training, generalizes poorly to new variants, and brings little overall improvement on real benchmarks. Simpler options, such as refining clearer instructions or better-designed schemas, can also help at a much lower cost, but the paper does not compare them with model training or discuss their relative costs. It therefore remains unclear why model training is necessary or worthwhile.
Questions for authors
See strengths and weaknesses
4 comments
W2: Why training, compared with cheaper fixes?
We agree that training is not a general remedy for schema sensitivity, and the paper does not present it as one. It is the costliest of the four levers we compare and the only one that repairs the protocol-level variants; the cheaper levers come first. RQ4 already compares the two cheapest fixes, a one-sentence instruction and constrained decoding of the first call, side by side with SFT and GRPO on the same twelve variants (Table 3), and RQ3 covers schema choice. The methods differ in cost and in coverage, and this difference is one of our findings:
| Method | Cost | What it repairs (Qwen3-4B unless noted) | What it cannot repair |
|---|---|---|---|
| Schema choice (RQ3) | About 100 target queries per model; no change to the model | Lets a designer rank candidate schemas before deployment | Applies only when the designer controls the schema; redone per model |
| Instruction | One sentence per variant | Conventions that can be stated as a rule: fully merged +0.28, schema discovery +0.45 | Multi-call protocols: transaction +0.06 |
| Constrained decoding | Decoder access | Invalid tool names: fully merged +0.29; 0.07 → 0.74 on Qwen3-30B-A3B | Cannot enforce a protocol: transaction −0.11 |
| SFT (mixed data) | Verified demonstrations of each target variant, plus training compute | What the cheap fixes cannot: transaction +0.23, schema discovery +0.67 | Taxes robust variants by 6–10 points; no transfer to unseen operators |
| GRPO (mixed data) | The highest: on-policy rollouts, plus an executable environment that rejects invalid calls | The largest gains, without tax: transaction +0.34, schema discovery +0.68 (the same holds on Llama-3.1-8B, reply to PY6W) | No transfer to unseen operators; on real benchmarks +0.03 on trained variants and +0.003 on untrained ones |
We therefore recommend a ladder, not training by default:
- Choose or redesign the schema when you control it.
- Use an instruction or constrained decoding when the failure is a describable convention or an invalid call.
- Reserve training for protocol-level variants, and for a model that must serve many schemas it does not control.
The limited transfer of training is itself a finding: training removes the biases it targets, not schema sensitivity in general. The revision adds this cost comparison as a table in the new Appendix G.8 (Cost and coverage of each mitigation), and §8 (RQ4) now closes with the ladder above in one sentence.
Thanks to the authors for their rebuttal. I agree that agents can be sensitive to tool schemas, but I remain unsure how significant this issue is in practice, particularly when developers can address some of the observed failures through careful schema design and prompt testing. I also see the value of anti-class grouping as a stress test, but it remains unclear how often deployed tools are grouped this way. For these reasons, I maintain my original rating of 2.
Thank you for the follow-up. We would like to restate what the paper claims, because we think this is where we differ.
1. What the paper claims. Not that schema sensitivity is unfixable in deployment, but that it exists, is large and structured, and has so far gone unmeasured. A tool schema is one representation of the action space, not the task, and an agent that has learned the task should behave consistently across equivalent representations, as we expect for prompt wording. Our contribution is to bring this to attention and to investigate it in depth: a framework that verifies every variant against native actions and final states, measurement on eleven models and up to 32 variants, a failure taxonomy of how each representation breaks an agent, an estimator of schema difficulty, and a comparison of mitigations with their transfer. We see this as the core value of the paper: before a problem can be designed around, it has to be visible and understood.
2. Why careful design does not remove the issue.
- Benchmarks and training environments commit to one schema. Rewriting the tools of four real benchmarks changes the same model's score by up to 0.67, and the most costly representation differs across model families. Benchmark scores and training gains therefore partly reflect the interface, whoever designed it.
- Which design is safe depends on the model. No variant is safe for every model; class dispatch, the grouping a careful designer would choose, still costs Qwen3.5-27B 0.29–0.42 on three real benchmarks and gpt-6-luna 13 points. Choosing well requires measuring per model, and RQ3 provides that measurement. A one-sentence instruction (Table 3) repairs conventions a sentence can describe, not the transaction protocol (+0.06).
- The agent developer often does not own the schema. Tools from MCP servers and aggregated catalogs arrive with their schema, and one model serves many of them. Training repairs only the changes it has seen, so robustness cannot be read off the score on one interface.
3. On anti-class grouping and how often deployed tools look like our variants. We cannot enumerate how schemas are designed in practice, and the paper does not try to. The variants are representative examples, each produced by an operator that a real design decision can produce, and their purpose is to expose a behaviour pattern: the same actions, presented differently, succeed or fail depending on the model. Anti-class grouping is one point on this spectrum; the pattern it reveals, that a merged tool's cost comes from which operations it contains, appears in the same direction at the deployed endpoints, class dispatch and fully merged. The value of the examples lies in showing that the bias exists and how it behaves, not in matching the frequency of any one design.
We thank the reviewer again for the engagement.
W1: Readability, and whether the variants are plausible designs
Readability. We agree that functional equivalence does not imply equal readability, and we do not define a readability score. We address the concern in two ways.
Plausibility. Each operator corresponds to a design decision that tool developers make for reasons unrelated to the agent.
Our premise is therefore not that every variant is a good design. It is that each is one a reasonable developer could ship, that the agent does not choose which one it receives, and that its sensitivity to this choice is a deployment risk worth measuring. On four real benchmarks the same variants are costly: fully merged costs Qwen3.5-27B 0.26–0.67 on τ²-retail, BFCL, AutomationBench and MCP-Atlas. The new Appendix C.3 (Where each operator arises in practice) names a design decision behind each of the nine operators in a table.