acceptodds
Under review as a conference paper at ICLR 2027

ToolForm: A Controlled Study of Tool Formatting for Agents

Abstract

Agent frameworks show a language model its tools in one written format, such as JSON Schema, XML, YAML, a Python function signature or a natural-language description. Prior studies compare these formats as one design choice and reach different conclusions. A tool interface sets three things: the schema format the model reads, the answer protocol it writes back in, and the grading rule that scores the call. Most prior studies change the first two together and fix the third, so the cause of a difference is unclear. We introduce **ToolForm**, an open-source framework that separates the three into independently controlled variables, measures the effect of each, and turns the measurements into defaults an agent can use. Our results show that many apparent format advantages depend on the answer protocol and the grading rule, while the gain from showing the parameter schema keeps its direction under every rule and task set. **First**, we run a controlled study that changes one factor at a time, on **100 open-source models** in 17 families (0.36B to 122B) and **12 proprietary models**, with **129 schema formats**, **31 answer protocols**, **3 grading rules**, 1 to 256 tools, 5 complexity tiers, 6 task regimes and 3 pressure treatments, at 5 seeds. **Second**, we build a benchmark from three sources: a seeded generator over **24 domains** whose tools run against a world state, **66 real tool schemas** with **1,248 human-written requests**, and **6 public benchmarks** rewritten in our formats and scored by their own checkers. An audit checks every record, and two authors check **600 graded turns** by hand. **Third**, we package ToolForm as a toolkit whose advisor turns the results into a format and protocol choice for a given model and tool set. The results give four rules. *Show the parameter schema*: it raises accuracy by **57.09%** over tool names alone, the largest effect in the study. *Choose the answer protocol before the schema format*: the protocol changes accuracy **1.22 to 2.00 times** as much as the format, and the XML lead over JSON Schema under the default pairing (13.80%) falls to 2.88% over five protocols. *Prefer a structured format to natural language*: natural language is never more accurate on average than the best structured format at any model size. *Keep the tool list short and treat tool descriptions as untrusted*: going from 1 to 256 tools lowers accuracy by 41.01%, and an instruction hidden in a tool description is followed on **38.48 to 54.16%** of turns in every format that shows the description. The gap between formats narrows with model size but does not close: the median of the 20 anchor models (12 proprietary) varies by 25% across the eight core formats, against 44% for the median open-source model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tool use, function calling, tool schemas, prompt formats, language model agents, controlled study, benchmark, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RXBFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents ToolForm, a framework for studying how tool-schema formats, answer protocols, and grading rules affect agent performance. Through extensive experiments across models and benchmarks, it shows that apparent format advantages can change substantially with the output protocol and evaluator. Additional analyses examine schema information, tool-set size, constrained decoding, and prompt injection. The paper also provides a toolkit and configuration advisor.

Critical strengths
  1. The paper convincingly demonstrates that evaluation choices can change apparent format advantages. In Table 2, XML’s advantage over JSON drops from 13.80 percentage points under the main grader to 1.07 under symmetric repair. Crossing schema formats with answer protocols further helps separate effects that bundled comparisons would conflate.

  2. The empirical validation is extensive. Beyond the synthetic experiments, the paper includes public benchmarks, native tool-channel comparisons, additional seeds, family-level uncertainty estimates, and a human grading audit. Its explicit reporting of negative results and parser limitations makes the findings easier to assess.

Critical weaknesses
  1. Schema information is not fully matched across formats. Table 41 documents differences in preserved enum values, nested requiredness, and constraints. The JSON-only component ablations do not resolve these differences across renderers, so residual performance gaps cannot be attributed cleanly to notation.

  2. Some natural-language findings remain sensitive to parsing errors. The human audit reports 8.1% under-crediting by the rule parser, while Appendix H identifies unsupported dotted-path arguments. These disclosed limitations weaken the interpretation of natural-language rankings and some anchor-model format gaps.

  3. The toolkit’s advisor combines separately measured effects, but its combined recommendations are not directly evaluated . The fitted selection-rule experiment evaluates a different procedure, leaving the actual advisor’s benefit on unseen models and tool catalogues uncertain.

Questions for authors
  1. Do the main format differences persist when all renderers preserve the same schema information, with answer protocol and grading fixed?
  2. How do natural-language rankings and anchor-model gaps change after correcting parser errors and validating on held-out examples?
  3. Can the actual advisor be evaluated on held-out models and tool catalogues against fixed defaults and direct calibration?
5 comments
AuthorsOpenReview ↗

Official Response by Authors to Reviewer RXBF (1/3)

We greatly thank the Reviewer for the careful review. We are very encouraged that the Reviewer found that the paper convincingly demonstrates that evaluation choices can change apparent format advantages and that the empirical validation is extensive. We are also very happy that the Reviewer found that its explicit reporting of negative results and parser limitations makes the findings easier to assess. We also ran all three experiments listed in each of the questions by Reviewer.

We value the Reviewer's time and have made every change easy to track: all new content in the revised paper is green, new appendix sections are clearly labelled as "(Rebuttal Addition)", and each answer below cites the place in the paper where we have added new content/results in code font. We have also placed all the new appendix sections at the very start of the appendix, so the Reviewer can quickly go through and verify the changes in the revised paper. Below, we address each concern raised by the Reviewer:


Schema information is not fully matched across formats. Table 41 documents differences [...] Do the main format differences persist when all renderers preserve the same schema information, with answer protocol and grading fixed?

We thank the Reviewer for this important question. Yes for the size of the format effect, and for five of the seven formats; for two formats the position changes. The submission documented the mismatch (Table 41) and varied two of the pieces, defaults and constraints, inside JSON Schema (Table 8); in the revised paper we add the cross-format control the Reviewer asks for (Appendix Y, Tables 47-48).

We gave every renderer a matched mode in which it writes every piece JSON Schema writes (nested enum values, whether a nested field is required, defaults, constraints), and had 51 models answer the seven formats twice in one session, as submitted and matched, with a JSON object as the answer format for all (357,000 graded requests; 714,000 with a second answer format). JSON Schema and YAML have the same text in both modes and serve as a placebo. Accuracy in %:

FormatStrict: submittedStrict: matchedSymmetric: submittedSymmetric: matched
JSON Schema (placebo)51.1351.0962.9662.94
YAML (placebo)49.1549.0861.9861.88
XML50.5650.6567.9067.93
Python signature56.5355.7667.7367.46
TypeScript48.5148.2364.5463.76
Markdown table47.0950.3260.6064.21
Natural language43.7145.3864.3968.65

("Strict" grades the call as written; "symmetric" forgives a nested field written flat.)

Our main observations:

  1. The placebo rows do not change, so the two modes are comparable.
  2. The differences persist. XML, the Python signature, TypeScript, JSON Schema and YAML change by under 1%. The spread across the seven formats goes from 12.82 to 10.38% (strict) and from 7.29 to 6.77 (symmetric), and 64 to 96% of the variance between formats remains.
  3. Two positions were content. The Markdown table (+3.2 to +3.6) and natural language (+1.7 to +4.3) had left out whether a nested field is required. With it in, natural language is level with XML under symmetric grading (+0.72 [−0.83, +2.59]); graded strictly it stays last.

So on these models a gap of 7 to 10% remains between renderings that have the same information; with one renderer per format, that is as close to a notation effect as the design gets. The paper says so in Section 4.2, lines 396-401. The same run with an XML call as the answer and on naturally phrased requests gives the same result; on the hardest tool tier JSON Schema and XML are first and natural language stays 5 to 7% behind. On 17 of the 20 anchor models (12 distinct models: three hosted at up to three reasoning efforts, six rows of Claude models and the four OpenAI models through the vendors' APIs) the seven format means lie within 1.7% of each other with matched content, against 7.8% as submitted (Table 48b).

AuthorsOpenReview ↗

Official Response by Authors to Reviewer RXBF (2/3)

Some natural-language findings remain sensitive to parsing errors. [...] How do natural-language rankings and anchor-model gaps change after correcting parser errors and validating on held-out examples?

We appreciate this careful observation. The submission disclosed both limits the Reviewer cites (the 8.1% under-crediting in Appendix W.1 and the dotted paths in Appendix H). In the revised paper we fix the dotted paths and two further parser limits, then measure what is left (Appendix Z, Tables 50-52).

Correcting the parser. We fixed three limits: dotted paths (details.status: signed), several calls stated in one answer, and nested objects written in two XML styles. Every stored answer of the main grid, the format-by-protocol experiment, the tool-count sweep, the anchors and the real-tool tasks was regraded; no model was re-run. Table 2 and Table 3 now show the corrected grades, with each changed cell in green. The remaining appendix tables keep the submitted grades in the revised paper (Table 50 gives each main number both ways); we will regrade them for the camera-ready version.

  • Open-source grid: no rank changes under any of the three rules. XML moves by 1.45% and natural language by 0.25% (12 cells of Table 2).
  • Anchor models: the gaps are much smaller.
Reading of the answersNatural language weakest (of 20 anchors)Median spread across formats
Submitted parser1324.7
Fixed parser814.7
gpt-oss-120b as reader514.0
Qwen3.5-27B as reader414.0

Most of the natural-language weakness we reported for the strongest models was our parser rejecting dotted paths, and Section 4.6, lines 510-516 and the abstract give the corrected numbers. The strongest models are less sensitive to the format than we reported (15% against 45% for the median open-source model).

Validating on held-out examples. One author copied out by hand, blind to model and format, the call made in 200 natural-language answers (120 from two seeds of the open-source grid that no parser change was tuned on, 80 from the anchors), and our grader scored that transcription like any other reading. The fixed parser almost never credits a call the person does not read (0.5%) and still misses 9.0% (11.7% on the open-source answers, 5.0% on the anchors). So we also had a 27B model read every main-grid and anchor answer in every format: it disagrees with the hand transcription on 4.5% of answers, and under it natural language is fourth of eight, 6.5% below the best format (Table 53). The same check on 150 XML, Python and JSON answers found the third parser limit. A second author transcribed the same 200 natural-language answers independently: the two name the same tool on 197, and the grades of the two transcriptions agree on 195 under the symmetric rule (Cohen's kappa 0.95); against the second transcription the fixed parser credits 1.0% too many and misses 11.0% [7.4, 16.1]. The first annotator marked 52 of the 200 answers ambiguous (two tools with the same interface) and the second marked none; counting a call to such a twin as correct changes the XML lead over JSON Schema by 0.1% (Appendix Z).

One number the Reviewer cited has moved slightly, but the finding holds. The drop of the XML lead from the main rule to the symmetric rule is 12.73% before and after the fixes (13.80 to 1.07 in the submission; 15.25 to 2.52 now, Table 2). The lead that remains for XML under symmetric grading is larger than we reported: 2.5% with the corrected parser and 4.4 to 6.6% when a separate model reads the answers (Table 50). Strict-copy reader models that use none of our parsers reproduce the same drop (from 20% to 6 to 7%; Appendix AA), so the dependence on the grading rule is a property of the answers and not of our parser.

AuthorsOpenReview ↗

Official Response by Authors to Reviewer RXBF (3/3)

The toolkit's advisor combines separately measured effects, but its combined recommendations are not directly evaluated. [...] Can the actual advisor be evaluated on held-out models and tool catalogues against fixed defaults and direct calibration?

We thank the Reviewer for this excellent suggestion. Yes, and we did. The submission stated this limit (Appendix A.7) and already reported that direct calibration beats the fitted rule (Section 4.7, Table 31: 69.18 against 64.35). In the revised paper we evaluate the advisor itself, held out three ways (Appendix AC, Tables 57-58): with one model family removed from its tables at a time; on 21 models the study never ran, 13 of them from families it has no data for; and on tool catalogues it was not built from (the 66 real schemas and BFCL). The fixed defaults are always JSON Schema and always XML. Direct calibration measures the eight formats on the target, keeps the best, and scores it on other requests.

Accuracy in %, main rule (the advisor column shows its better variant in each row):

Held outAlways JSONAlways XMLAdvisorDirect calibration
One family at a time, generated tasks54.4263.3464.5864.66
21 unseen models, generated tasks41.0565.4865.4866.81
20 unseen models, real tools59.0958.5058.5064.69
Study models, real tools64.0759.0962.4066.15
Study models, BFCL82.7873.6677.7482.73

Our main observations:

  1. On generated tasks the advisor is level with the better fixed default (always XML). For a model it knows nothing about it returns XML.
  2. On real tools and on BFCL it is below always JSON Schema. Its XML default comes from the main rule, which forgives a flat nested field in XML answers only, and these catalogues are flat. Under the as-written rule it loses to JSON Schema on 16 of the 20 unseen models.
  3. Direct calibration is the best or level with the best in every row, when it is run on the tools the model will use. Rows 1 and 2 calibrate on 50 generated requests per format, rows 3 to 5 on half of the catalogue's own requests. On the unseen models with real tools it is 5.6% [0.7, 12.4] above always JSON Schema. A format picked on generated requests does not transfer to real tools (59.90 against 64.07; Table 57).

The paper now describes the toolkit accordingly (Abstract, Section 4.7, lines 520-530): it gives a starting default based on the measured effects and runs the calibration on the user's own model and tools, and the paper recommends the calibration. Table 57 reports every setting under all three grading rules, including the ones where the advisor loses, and Table 58 shows how many requests the calibration needs (ten per format already match the best fixed default).

On presentation: Section 3.1 now explains the three choices in simple words, with one worked example and a who-sees-what table, and a reader's map and a list of terms open the appendix (pages 21-22).

Once again, we thank the Reviewer for the careful and supportive feedback. Your comments have helped us identify important areas for improvement and have strengthened the clarity and impact of our work; all three requested experiments are now in the revised paper. We appreciate your valuable time and hope that the revised paper addresses these concerns. Please let us know if there are any remaining questions/concerns; we would be happy to address them!

Reviewer_KDtiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ToolForm studies how the presentation of tool information, the required answer format, and the grading rule affect tool-call accuracy. It uses generated tasks, real tool schemas, and public benchmarks. The framework uses these measurements to recommend input and output formats.

Critical strengths
  • The study treats input format, required answer format, and grading as separate experimental choices. It tests combinations of input and output formats and regrades the same answers under different rules. These controls help explain why earlier comparisons produce different rankings. They also provide a useful method for evaluating tool interfaces.
  • The evaluation covers multiple model families and several sources of tasks. The authors check sensitivity to task samples and model-family weighting, and correct for multiple comparisons. Public benchmarks retain their own checkers, and the multi-turn evaluation includes a native-call baseline (Table 25). The benefit of showing parameter schemas remains across the tested task sets and grading rules, even when individual format rankings change.
Critical weaknesses
  • The input formats do not always contain the same information, as the paper acknowledges (Table 41). Some omit allowed values for nested fields or whether those fields are required. The component tests vary information only within JSON Schema (Table 8). They therefore do not separate the effect of notation from the effect of providing more information across formats. The rankings support comparisons of these implementations, not a general ranking of notation choices.
  • The natural-language results mix the effect of input descriptions with failures in processing answers. The parser misses some correct intended calls (W.1). With a shared answer format, natural-language input trails JSON Schema by 0.6 to 2.3 percentage points on BFCL (Table 24). The larger losses under default pairings therefore cannot be attributed to the input descriptions alone because changing output handling may improve accuracy without replacing the input descriptions.
  • The practical recommendations have less support than the evaluation findings. The advisor combines measured effects, but the combined procedure was not evaluated (A.7). Under the main grader, a separate fitted selection rule scores below fixed XML and direct testing in the reported averages (Table 31). The paper discloses these limits, but they leave the benefit of automatic recommendations uncertain. The evidence supports a method for testing interfaces more clearly than a general rule for choosing one.
Questions for authors
  • On a small subset, do rankings hold when all input formats show the same information, with a shared answer format and symmetric grading?
  • How much of the natural-language accuracy loss comes from parser errors or failures to end conversations, rather than from the input descriptions?
  • Can a small held-out test establish the advisor’s benefit over simple baselines, or should its claimed role be narrowed?
6 comments
AuthorsOpenReview ↗

Official Response by Authors to Reviewer KDti (1/3)

We greatly thank the Reviewer for the careful and fair review. We are very encouraged that the Reviewer found that treating the input format, the answer format and the grading as separate choices gives a useful method for evaluating tool interfaces, that these controls help explain why earlier comparisons produce different rankings, and that the benefit of showing parameter schemas remains across the tested task sets and grading rules.

We value the Reviewer's time and have made every change easy to track: all new content in the revised paper is green, new appendix sections are clearly labelled as "(Rebuttal Addition)", and each answer below cites the place in the paper where we have added new content/results in code font. We have also placed all the new appendix sections at the very start of the appendix, so the Reviewer can quickly go through and verify the changes in the revised paper. Below, we address each concern raised by the Reviewer:


The input formats do not always contain the same information [...] (Table 41). [...] On a small subset, do rankings hold when all input formats show the same information, with a shared answer format and symmetric grading?

We thank the Reviewer for this point, which the submission raised about itself: Table 41 lists, format by format, which pieces of a nested schema each renderer keeps, and Table 8 varies two of them, defaults and constraints, inside JSON Schema. It does not vary whether a nested field is required, which is the piece that turns out to matter. It also already compared formats under a shared answer format and symmetric grading (Figure 3, Table 18, Appendix D.7). It lacked the control the Reviewer names: the same information in every format. We ran it, on 51 models and not on a small subset (Appendix Y, Tables 47-48).

Every renderer got a matched mode in which it writes, in its own notation, every piece JSON Schema writes: nested enum values, whether a nested field is required, defaults and all constraints. The check of Table 41, repeated on the matched renderers, shows no gaps (Table 47). Then 51 models answered the seven formats twice in one server session, as submitted and matched, with one answer format for all, at 8 and 32 tools and five seeds: 357,000 graded requests, and 714,000 with a second answer format. JSON Schema and YAML have identical text in both modes and serve as a placebo.

JSON-object answers, symmetric grading (the setting asked for), accuracy in %:

FormatAs submittedMatchedChange [95% CI]
JSON Schema (placebo)62.9662.94−0.02 [−0.16, +0.11]
YAML (placebo)61.9861.88−0.09 [−0.21, +0.02]
XML67.9067.93+0.03 [−0.29, +0.38]
Python signature67.7367.46−0.27 [−0.89, +0.22]
TypeScript64.5463.76−0.78 [−1.06, −0.50]
Markdown table60.6064.21+3.60 [+2.60, +4.75]
Natural language64.3968.65+4.26 [+3.28, +5.31]

Our main observations:

  1. The placebo is flat, so the two modes are comparable.
  2. The format effect is real. Best - worst goes from 7.29 to 6.77% here and from 12.82 to 10.38% graded strictly; 64 to 96% of the variance between formats remains. The other five formats move by under 1%. Under our parser's grades no pairwise contrast that excludes zero in both modes changes sign (read by a separate model, one or two do).
  3. Two positions were content, as the Reviewer suspected. The Markdown table and natural language, whose renderers did not say whether a nested field is required, gain 3.6 and 4.3%. With matched content natural language is level with XML at the top under symmetric grading (+0.72 [−0.83, +2.59]) and stays last when nesting is graded strictly.
  4. In the setting asked for, the ranking holds only in part. The submitted and matched rankings correlate at 0.89 graded strictly and at 0.54 under symmetric grading, where natural language moves from fourth to first and the Markdown table from seventh to fourth.

The same holds with an XML call as the answer format and on naturally phrased requests. On the hardest tool tier the same two formats gain, JSON Schema and XML are first, and natural language stays 5 to 7% behind. On 17 of the 20 anchor models (12 distinct models), matching removes most of what is left: the seven format means lie within 1.7% of each other with matched content, against 7.8% as submitted (1.9 against 8.6 with one row per model; Table 48b). The three rows not run are Kimi-K3, which its endpoint no longer serves, and two Claude rows at medium effort.

The paper now says this in Section 4.2, lines 396-401 and Appendix Y: what remains after matching is a spread of 7 to 10% between renderings that have the same information. With one renderer per format this is still a comparison of renderings, which is as close to a notation effect as the design gets.

AuthorsOpenReview ↗

Official Response by Authors to Reviewer KDti (2/3)

The natural-language results mix the effect of input descriptions with failures in processing answers. [...] How much of the natural-language accuracy loss comes from parser errors or failures to end conversations, rather than from the input descriptions?

We thank the Reviewer for this important question. Most of it, and the submission had the first pieces of this answer: Table 24 (which the Reviewer cites) fixes the answer format on BFCL, Appendix W.1 measures the parser against a human, and Appendix T has a second model re-read the natural-language answers of 12 of the 16 families. In the revised paper we complete the separation in four steps.

1. Answer format held fixed (Table 49). With natural-language descriptions answered as a JSON object, the English description scores +1.91 [−0.60, +4.25] against JSON Schema under symmetric grading and −6.45 [−8.75, −4.18] as written. Under symmetric grading the whole loss of natural language with its own answer format (−4.40 against JSON Schema) is therefore the answer format and its parser (−6.31; Table 50). The cost of an English description is in nesting: models that read it write nested arguments flat more often.

2. Parser fixed, then measured again on new answers (Appendix Z, Tables 50-52). We fixed three parser limits (dotted paths such as details.status, several calls in one answer, nested objects in XML answers) and regraded every stored answer used in Tables 2-3 and Figures 1-4 (the main grid, the format-by-protocol experiment, the tool-count sweep, the anchors and the real-tool tasks). Figure 5, the fixed-grammar, injection and selection-rule numbers of Sections 4.4, 4.5 and 4.7, the few numbers the text marks "(submitted grades)" and the remaining appendix tables keep the submitted grades for now and will be regraded for the camera-ready version. One author then copied out by hand, blind to model and format, the call made in 200 new answers; our grader scored that transcription like any other reading:

Reader of the answerCredits a call the person does notMisses a call the person reads
Our parser, fixed0.5%9.0% [5.8, 13.8]
Qwen3.5-27B as reader2.5%2.0%
gpt-oss-120b as reader2.0%8.0%

Measured this way our natural-language numbers are a lower bound, and the paper now has the reader models' numbers beside the parser's (Table 53). Against the first annotator's right-or-wrong verdict on the 148 answers not marked ambiguous, the parser over-credits 7.4% and under-credits 10.1%, and the 27B reader 10.1% and 2.7% (Appendix Z). A second author then transcribed the same 200 answers independently: the two name the same tool on 197 and the grades of the two transcriptions agree on 195 under the symmetric rule (Cohen's kappa 0.95); against the second transcription the parser misses 11.0% [7.4, 16.1]. The first annotator marked 52 answers ambiguous (two tools with the same interface) and the second marked none; counting a call to such a twin as correct changes the XML lead over JSON Schema by 0.1% (Appendix Z).

3. The strongest models (Table 3, Table 51). With the parser fixed, natural language is the weakest format for 8 of the 20 anchor models, not 13 (4 or 5 under the reader models). Table 3 and Section 4.6 now show the corrected values.

4. tau2-bench with description and answer format set independently (Appendix AB, Table 56): six models, two domains, four trials. Differences in reward (%):

ChangeAirlineRetail
English description instead of XML (answer: XML call)+0.3 [−2.7, +3.1]+1.8 [−0.4, +4.0]
English answer instead of XML call (description: English)−21.7 [−27.4, −16.1]−26.6 [−29.8, −23.3]
English answer, harness asks again when it cannot read a reply+15.2 [10.7, 19.7]+7.5 [5.1, 9.9]

The description has no measurable cost. The loss is in the answer format, and part of it is our harness: when the harness returns an error for a reply it cannot read, reward rises by 15% on airline and 7.5% on retail. On the original runs, compared on the same task and trial, the airline gap is concentrated in conversations that did not end (3.5% [−3.5, +10.8] on dialogues that ended under both formats); on retail and telecom it is not (23.0 and 15.0; Table 55).

The paper's claim now follows this evidence (Abstract, Section 4.4, lines 462-465, Section 4.6): natural language is never more accurate on average than the best structured format, at any model size, when it is also the answer format. A tool list described in English but answered in a structured format, with the same content, is read as well as in any other format on our standard tier when a flat nested field is forgiven, and is 5 to 10% behind when it is not.

AuthorsOpenReview ↗

Official Response by Authors to Reviewer KDti (3/3)

The practical recommendations have less support than the evaluation findings. [...] Can a small held-out test establish the advisor's benefit over simple baselines, or should its claimed role be narrowed?

We appreciate this fair assessment and agree with the Reviewer; the submission's numbers pointed the same way: Appendix A.7 states that the advisor's combination was not evaluated, and Section 4.7 with Table 31 reports that measuring eight formats on the target model (69.18%) beats every fixed rule, ours included (64.35%). Since the Reviewer asks for the direct test, we ran it, and it confirms the measurement-first reading (Appendix AC, Tables 57-58).

We scored the advisor itself (not the fitted rule of Table 31) where it had seen nothing: with one model family removed from its tables at a time; on 21 models the study never ran, 13 of them from families it has no data for (Yi 1.5, Falcon 3, EXAONE 3.5, InternLM 3 and others); and on tools it was not built from (the 66 real schemas and BFCL). The baselines are always JSON Schema, always XML, and direct testing: measure the eight formats on the target model (50 generated requests each, or half of the real catalogue's own requests), keep the best, and score it on other requests.

Unseen models, accuracy in %, main rule / as-written rule:

PolicyGenerated tasks (21 models)Real tools (20 models)
Always JSON Schema41.05 / 41.0559.09 / 59.09
Always XML65.48 / 43.2158.50 / 58.50
Advisor, model unknown65.48 / 41.3658.50 / 55.12
Measured on the target66.81 / 48.1064.69 / 64.69

Our main observations:

  1. Without the model's name the advisor is a fixed default. Under the main rule it picks XML for all 21 unseen models.
  2. On real tools it does not beat always JSON Schema. Graded strictly it loses on 16 of 20 models, and by 7.45% [3.41, 12.40] on the 13 from new families. Its XML default comes from the main rule, which forgives a flat nested field in XML answers only, and the real schemas are flat.
  3. Knowing the model family helps only for families already in the study (+3.0%; this is the optimistic case, since the family's row includes the queried model).
  4. Measuring on the tools the model will use beats the advisor in every real-tool block: 5.6% [0.7, 12.4] above always JSON Schema on the unseen models. Ten requests per format already match the best fixed default on generated tasks (Table 58). A format picked on generated requests does not transfer to real tools (59.90 against 64.07), so the measurement has to be on the user's own tools.

The paper's role for the toolkit now matches this (Abstract, Section 4.7, lines 520-530, Appendix A.7): it gives a starting default based on the measured effects and runs the direct test on the user's own model and tools, and the paper recommends the test. As the Reviewer put it, the evidence supports a method for testing tool interfaces more than a rule for choosing one. The paper claims that method as its contribution, now with the controls the Reviewer asked for.

Once again, we thank the Reviewer for the careful and constructive feedback. Your comments have helped us identify important areas for improvement and have strengthened the clarity and impact of our work; every control the Reviewer asked for is now in the revised paper. We appreciate your valuable time and hope that the revised paper addresses these concerns. Please let us know if there are any remaining questions/concerns; we would be happy to address them!

Reviewer_cqZFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I found the paper very interesting and useful. It does a deep analysis over many models and schemas and grading on how LLMs are best using their tools. The paper is super thorough and includes many experiments (which also makes us wonder how much it costs, but this is not that important), it seperates the tool use into several parts and analyzes the contribution of each one. It claims that most SOTA look at the tools as one architecture decision and do not separate them, thus struggling to understand the deep reasons for failures and might miss options for improvement. I can not validate their results e2e as it is a huge experiment, but the results sound reasonable and consistent with what I know from building agents and my experiments with enterprise agents.

Critical strengths

The experiment is super super super extensive and deep, with many models, methods, tools, and analyses. It is clear that the author invested a lot of time, money, and thought in this The decomposition of the tool use is a nice idea, and applicable to real-world use cases.

Critical weaknesses

It was very hard for me to understand what the answer protocol and the grading rule I tried to find a simple example but could not understand. As this is the core contribution of the paper, it is very bad that a reader is struggling to understand it

The paper is over 100 pages, so it is hard to find the specific things you are looking for.

Questions for authors

Without any mathematical notation, please explain in super simple words what is

  1. answer protocol
  2. grading rule

Give a super easy example for each one, with inputs, outputs, and who exactly sees what (the LLM, the tool, etc.)

5 comments
AuthorsOpenReview ↗

Official Response by Authors to Reviewer cqZF (1/2)

We greatly thank the Reviewer for the thoughtful review. We are very encouraged that the Reviewer found the work useful and super super super extensive and deep, which was exactly our intention. We are also very happy that the Reviewer found our work consistent with their experience of building enterprise agents, and called the decomposition of tool use a nice idea, and applicable to real-world use cases.

We value the Reviewer's time and have made every change easy to track: all new content in the revised paper is green, new appendix sections are clearly labelled as "(Rebuttal Addition)", and each answer below cites the place in the paper where we have added new content/results in code font. We have also placed all the new appendix sections at the very start of the appendix, so the Reviewer can quickly go through and verify the changes in the revised paper. Below, we address each concern raised by the Reviewer:


It was very hard for me to understand what the answer protocol and the grading rule [are]. [...] Without any mathematical notation, please explain in super simple words [...] Give a super easy example for each one, with inputs, outputs, and who exactly sees what (the LLM, the tool, etc.)

We thank the Reviewer for this important point, and we agree that the core idea must be clear in one reading (for anyone reading for the first time). The submission defined both terms and had a worked example, but not where a reader needs/expects them: the definitions were in notation (Section 3.2, Notation), and the example was in the last panel of Figure 1 and in Appendix S.2. In the revised paper we put a plain-word explanation, one example and a who-sees-what table at the start of the framework section, before any notation (Section 3.1, lines 175-198). Here it is:

  1. The schema format is what the model reads. The agent's tool list has to be written into the prompt as text. The same tool can be written as JSON Schema, XML tags, a Python signature, a Markdown table or an English paragraph.

  2. The answer protocol is what the model writes back. The prompt says what shape the answer must take: "reply with a JSON object", "reply with an XML <invoke> block", or "say in plain English which tool you would use". A small program (the parser) turns the reply into a call: a tool name + a dictionary of arguments. The tool never sees the model's text; it only sees the call.

  3. The grading rule is how we decide whether the call is right. We compare the call with the correct call. The model never sees the rule. Our three rules agree on everything except one case: a nested argument that the model wrote flat.

An example. The user asks: "Draft a 12 month contract with Holt & Vane worth 240000.0. Use status signed." The tool legal_draft_v3 takes counterparty, value, term_months, and a group contract_details that holds a field status. The correct call is

legal_draft_v3(counterparty="Holt & Vane", value=240000.0, term_months=12,
               contract_details={status: "signed"})

Asked for an XML block, the model writes:

<invoke name="legal_draft_v3">
  <parameter name="counterparty">Holt & Vane</parameter>
  <parameter name="value">240000.0</parameter>
  <parameter name="term_months">12</parameter>
  <parameter name="status">signed</parameter>
</invoke>

The tool is right and every value is right, but status is at the top level and not inside contract_details. Is this answer correct?

Grading ruleWhat it does with the flat statusThis XML answerThe same answer as a JSON object
mainputs it back, but only for text answers (XML, plain English)correctwrong
symmetricputs it back for every kind of answercorrectcorrect
as writtennever puts it backwrongwrong

If asked for a JSON object, the same answer would be {"name": "legal_draft_v3", "arguments": {"counterparty": "Holt & Vane", "value": 240000.0, "term_months": 12, "status": "signed"}}: the same content in another shape, and that shape is the answer protocol. Asked for plain English it would be "I will use legal_draft_v3 with counterparty = Holt & Vane, value = 240000.0, term_months = 12 and status = signed.", a third protocol. Under the main rule the same small mistake is forgiven in the XML answer and punished in the JSON one. That one asymmetry accounts for 12.73 of the 15.25% by which XML leads JSON Schema in Table 2. Graded the same way for both, the lead is 2.5% with our parser and 4 to 7% when a separate model reads the answers (Table 50); graded as written, the two formats are level. This is the paper's first finding: a format comparison means little until the answer protocol and the grading rule are fixed.

AuthorsOpenReview ↗

Official Response by Authors to Reviewer cqZF (2/2)

Who sees what in one turn (Section 3.1, second table):

Piece of one turnModelParserToolGrader
the user's requestreads it
the tool list, in one schema formatreads it
the answer instruction (the protocol)reads itset by it
the model's replywrites itreads it
the call (tool name and arguments)produces itruns itreads it
the correct call and the grading rulereads them

Which rule should a reader trust? The main rule is what our parsers do by default: XML and plain-English answers usually come back with every argument at the top level, so the parser puts fields back for them and not for JSON. It is what a deployment that uses these parsers would score, which is why it is reported first. To compare formats, read the symmetric column. The as-written rule is what a tool that accepts no repair would do. Every claim about our own tasks is checked under all three (Table 2, Figure 2, Figure 3).

Why it matters. A comparison that changes the format and the protocol together and grades with one rule cannot tell the three apart. In our data, comparing "XML tools" with "JSON tools", each with its usual answer shape, also changes how the model answers and how a misplaced field is scored. Once the three are separated, most of that XML lead turns out to depend on the grading rule and the parser, and a smaller, real difference between formats remains.

The paper is over 100 pages, so it is hard to find the specific things you are looking for.

We appreciate this practical concern, and we apologize that the Reviewer had trouble finding specific things. The appendix is very long because every number in the main text has a table and records that support it, and we wanted to keep all of it so that a study of this size can be checked/verified. With the camera-ready version we will release the full codebase and every record (Appendix X.1), together with tests that let anyone verify the results: unit tests for the renderers, the parsers and the three grading rules; a check that an answer equal to the correct call scores 100% under every format (Appendix V.5); and one command that regrades the stored replies, rebuilds every table and figure, and reports any number that differs from the paper. To help a reader find things in the appendix easily, in the revised paper we have added:

  1. A reader's map at the start of the appendix (page 22): one row per claim of the main text, with the table or figure that gives the number and the appendix section that explains it. A reader who wants to check one number goes to one row.
  2. A plain-word list of terms beside it (page 21): format, protocol, grading rule, cell, checkpoint, anchor, tier, regime, repair, strict and lenient parse.
  3. The worked example in full, in a box (page 22).
  4. A pointer from the introduction to the example and the map (Section 1, lines 85-89), which replaces a sentence of notation.
  5. Every new appendix section of experiments is labelled "(Rebuttal Addition)" in its title and in the table of contents, and the five of them come first in the appendix (pages 23-35), so nothing new is placed after the appendix of the submission. We have purposely kept the letters Y to AC and Tables 47-58 for these new sections, so that every section and table number of the submission is unchanged (the Reviewer can still easily find the old tables/figures/sections if they want to).

Note: The revised paper is longer than the submission (117 pages against 101), because the other reviewers asked for new experiments and each has its tables.

Regarding the Reviewer's question about what the study cost: we used around 24,000 GPU hours of inference on five clusters (Appendix L.3); the proprietary models add API calls for 2,400 turns each (800 for the one model that ran a single seed; Appendix H).

Once again, we thank the Reviewer for the thoughtful and encouraging feedback. Your comments have helped us identify important areas for improvement and have strengthened the clarity and impact of our work. We appreciate your valuable time and hope the explanation is now clear in one go. Please let us know if there are any remaining questions/concerns or if any part is still unclear; we would be happy to address them!

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer cqZF,

We greatly appreciate the reviewer's efforts in providing feedback for our work. We hope that, through this rebuttal, we have addressed your proposed concerns. If the concerns have been sufficiently addressed, we would appreciate if you could consider revising your scores. Or, If there are any other/further concerns, please let us know, we would be happy to address them as there is still 2 days left for rebuttal.

Thanks, ToolForm Authors!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Final Overview by the Authors (1/2)

First, we sincerely thank the reviewers for their thoughtful and constructive reviews. Every comment helped us: the questions led to new experiments in five appendix sections, corrected four statements of the submission and made the paper easier to read.

Second, we are very encouraged that all three reviewers found the separation of schema format, answer protocol and grading rule useful. Reviewer cqZF called the work "super super super extensive and deep", Reviewer KDti wrote that it gives "a useful method for evaluating tool interfaces", and Reviewer RXBF wrote that it "convincingly demonstrates that evaluation choices can change apparent format advantages". During the rebuttal we have addressed every concern raised, with new experiments, new analysis and revisions of the paper.

Third, we greatly appreciate the limited and valuable time of the area chair and the program chairs. We have made every change easy to track: all new content in the revised paper is green, new appendix sections are clearly labelled as "(Rebuttal Addition)" and placed at the very start of the appendix, and the table and appendix numbers of the submission are unchanged. Below we summarize our main contributions, the strengths highlighted by the reviewers, our responses to the main concerns and the revisions.

Our main contributions:

  1. A controlled study that separates three choices. A tool interface sets the schema format the model reads, the answer protocol it writes in and the grading rule that scores the call. Earlier comparisons changed the first two together and fixed the third. We change one at a time and score every answer under three grading rules.
  2. Scale. 100 open-source models in 17 families (0.36B to 122B) and 12 proprietary models, 1 to 256 tools, 5 complexity tiers and 6 task regimes: 571,687 graded cells and 32.1M graded requests.
  3. A benchmark from three sources. A seeded generator over 24 domains whose tools run against a world state, 1,248 human-written requests for 66 real tool schemas, and 6 public benchmarks scored by their own checkers.
  4. Findings that say what carries across settings. Six results hold across the settings we tested (showing the parameter schema adds 57.09% over tool names alone; a structured answer is more accurate than a natural-language one; the XML lead over JSON Schema comes from the grading rule), and one does not: the order among the structured formats (Table 59).
  5. An open-source toolkit. 129 format renderers, 31 protocol parsers and 3 grading rules, with a Python SDK, a command line tool, an HTTP API and a dashboard. It gives a starting default and measures the formats on the user's own model and tools.

Strengths highlighted by the reviewers:

  • Extensive and deep experiments. "The experiment is super super super extensive and deep, with many models, methods, tools, and analyses" (Reviewer cqZF); "The empirical validation is extensive" (Reviewer RXBF); "The evaluation covers multiple model families and several sources of tasks" (Reviewer KDti).
  • The separation of the three choices. "The decomposition of the tool use is a nice idea, and applicable to real-world use cases" (Reviewer cqZF); "These controls help explain why earlier comparisons produce different rankings" (Reviewer KDti); "Crossing schema formats with answer protocols further helps separate effects that bundled comparisons would conflate" (Reviewer RXBF).
  • A result that holds across settings. "The benefit of showing parameter schemas remains across the tested task sets and grading rules, even when individual format rankings change" (Reviewer KDti).
  • Open reporting of limits. "Its explicit reporting of negative results and parser limitations makes the findings easier to assess" (Reviewer RXBF); "I also appreciate that you report where the advisor falls short" (Reviewer KDti, in the discussion).
  • Agreement with practice. The results are "consistent with what I know from building agents and my experiments with enterprise agents" (Reviewer cqZF).
AuthorsOpenReview ↗

Final Overview by the Authors (2/2)

Response to the main concerns:

  1. A plain explanation of the three choices (Reviewer cqZF). Section 3.1 now explains the three in plain words with one example and a who-sees-what table, before any notation; a reader's map and a list of terms open the appendix (pages 21-22).
  2. Matched content (Reviewers KDti, RXBF). 51 models answered every format twice, as submitted and with the same information in every format (714,000 requests, with a placebo). 64 to 96% of the variance between formats remains (spread falls from 12.8 to 10.4% graded strictly), and the Markdown table and natural language gain up to 5% (Appendix Y). On 17 of the 20 anchor models, the seven format means lie within 1.7% of each other with matched content (7.8% as submitted; Table 48b).
  3. Parser errors and held-out validation (Reviewers KDti, RXBF). We fixed three limits of the parser, regraded every stored answer used in Tables 2-3 and Figures 1-4, and had separate reader models read the answers in place of our parsers. One author transcribed 350 new answers by hand (a second author re-transcribed the 200 natural-language ones; the grades agree on 195). On the open-source grid no rank changes, and natural language is the weakest format for 8 of the 20 anchor models, not 13 (Appendix Z, Appendix AA).
  4. Where the natural-language loss comes from (Reviewer KDti). A new tau2-bench run on six models sets the description and the answer format independently: the description costs nothing measurable and the answer format costs 19 to 28% (Appendix AB).
  5. The advisor, held out (Reviewers KDti, RXBF). On 21 unseen models and on real tools, the advisor does not beat the better fixed default; measuring on the tools the model will use is 5.6% more accurate, and the paper now recommends it (Appendix AC).
  6. Which results carry across settings (Reviewer KDti, in the discussion). Six results hold across the grading rules, task sources and models we tested; the order among the structured formats does not (Appendix AD, Table 59).

What the new evidence changed. The method and the first finding of the submission hold: two strict-copy reader models that use none of our parsers reproduce the dependence of the XML lead on the grading rule. Four statements are narrowed or corrected: (i) natural language is never more accurate on average than the best structured format when it is also the answer format, and an English description answered in a structured format is competitive; (ii) the median anchor model's spread across the eight formats is 15%, not 25%; (iii) the toolkit's advisor is a starting default, and measurement is the recommendation; (iv) the answer protocol outweighs the schema format under the main rule, but not when every protocol is graded alike.

A quick summary of the revisions made during the rebuttal (all in green):

Main paper:

  • Abstract, Section 1: the four rules restated with the corrected numbers.
  • Section 3.1 (new): the three choices in plain words, with one worked example.
  • Tables 2-3, Figures 1-4, Figure 18: regraded with the three parser fixes.
  • Sections 4.2, 4.3, 4.4, 4.6, 4.7: the corrected XML contrasts, matched content, the natural-language loss, tau2-bench, the anchor models and the held-out advisor.

New appendix sections, each labelled "(Rebuttal Addition)" and we placed them first in the appendix (pages 23-36; lettered Y to AD so that no section or table number of the submission changes - so that AC/Reviewer can verify everything easily):

  • Appendix Y: matched content (Tables 47-48, Table 48b).
  • Appendix Z: the parser fixes, the regrade and the held-out human check (Tables 49-52).
  • Appendix AA: reader models over every format (Tables 53-54).
  • Appendix AB: tau2-bench with description and answer format crossed (Tables 55-56).
  • Appendix AC: the advisor, held out (Tables 57-58).
  • Appendix AD: which results carry across settings (Table 59).

Final comments: We believe ToolForm gives the community a careful way to compare tool interfaces and a clear statement of which results carry across settings. We again thank all reviewers/area chair/program chairs for their valuable time, and hope our revisions show our commitment to scientific rigor and the significance of our contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.