- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents ToolForm, a framework for studying how tool-schema formats, answer protocols, and grading rules affect agent performance. Through extensive experiments across models and benchmarks, it shows that apparent format advantages can change substantially with the output protocol and evaluator. Additional analyses examine schema information, tool-set size, constrained decoding, and prompt injection. The paper also provides a toolkit and configuration advisor.
Critical strengths
-
The paper convincingly demonstrates that evaluation choices can change apparent format advantages. In Table 2, XML’s advantage over JSON drops from 13.80 percentage points under the main grader to 1.07 under symmetric repair. Crossing schema formats with answer protocols further helps separate effects that bundled comparisons would conflate.
-
The empirical validation is extensive. Beyond the synthetic experiments, the paper includes public benchmarks, native tool-channel comparisons, additional seeds, family-level uncertainty estimates, and a human grading audit. Its explicit reporting of negative results and parser limitations makes the findings easier to assess.
Critical weaknesses
-
Schema information is not fully matched across formats. Table 41 documents differences in preserved enum values, nested requiredness, and constraints. The JSON-only component ablations do not resolve these differences across renderers, so residual performance gaps cannot be attributed cleanly to notation.
-
Some natural-language findings remain sensitive to parsing errors. The human audit reports 8.1% under-crediting by the rule parser, while Appendix H identifies unsupported dotted-path arguments. These disclosed limitations weaken the interpretation of natural-language rankings and some anchor-model format gaps.
-
The toolkit’s advisor combines separately measured effects, but its combined recommendations are not directly evaluated . The fitted selection-rule experiment evaluates a different procedure, leaving the actual advisor’s benefit on unseen models and tool catalogues uncertain.
Questions for authors
- Do the main format differences persist when all renderers preserve the same schema information, with answer protocol and grading fixed?
- How do natural-language rankings and anchor-model gaps change after correcting parser errors and validating on held-out examples?
- Can the actual advisor be evaluated on held-out models and tool catalogues against fixed defaults and direct calibration?
5 comments
Official Response by Authors to Reviewer RXBF (2/3)
Some natural-language findings remain sensitive to parsing errors. [...] How do natural-language rankings and anchor-model gaps change after correcting parser errors and validating on held-out examples?
We appreciate this careful observation. The submission disclosed both limits the Reviewer cites (the 8.1% under-crediting in Appendix W.1 and the dotted paths in Appendix H). In the revised paper we fix the dotted paths and two further parser limits, then measure what is left (Appendix Z, Tables 50-52).
Correcting the parser. We fixed three limits: dotted paths (details.status: signed), several calls stated in one answer, and nested objects written in two XML styles. Every stored answer of the main grid, the format-by-protocol experiment, the tool-count sweep, the anchors and the real-tool tasks was regraded; no model was re-run. Table 2 and Table 3 now show the corrected grades, with each changed cell in green. The remaining appendix tables keep the submitted grades in the revised paper (Table 50 gives each main number both ways); we will regrade them for the camera-ready version.
- Open-source grid: no rank changes under any of the three rules. XML moves by 1.45% and natural language by 0.25% (12 cells of
Table 2). - Anchor models: the gaps are much smaller.
| Reading of the answers | Natural language weakest (of 20 anchors) | Median spread across formats |
|---|---|---|
| Submitted parser | 13 | 24.7 |
| Fixed parser | 8 | 14.7 |
| gpt-oss-120b as reader | 5 | 14.0 |
| Qwen3.5-27B as reader | 4 | 14.0 |
Most of the natural-language weakness we reported for the strongest models was our parser rejecting dotted paths, and Section 4.6, lines 510-516 and the abstract give the corrected numbers. The strongest models are less sensitive to the format than we reported (15% against 45% for the median open-source model).
Validating on held-out examples. One author copied out by hand, blind to model and format, the call made in 200 natural-language answers (120 from two seeds of the open-source grid that no parser change was tuned on, 80 from the anchors), and our grader scored that transcription like any other reading. The fixed parser almost never credits a call the person does not read (0.5%) and still misses 9.0% (11.7% on the open-source answers, 5.0% on the anchors). So we also had a 27B model read every main-grid and anchor answer in every format: it disagrees with the hand transcription on 4.5% of answers, and under it natural language is fourth of eight, 6.5% below the best format (Table 53). The same check on 150 XML, Python and JSON answers found the third parser limit. A second author transcribed the same 200 natural-language answers independently: the two name the same tool on 197, and the grades of the two transcriptions agree on 195 under the symmetric rule (Cohen's kappa 0.95); against the second transcription the fixed parser credits 1.0% too many and misses 11.0% [7.4, 16.1]. The first annotator marked 52 of the 200 answers ambiguous (two tools with the same interface) and the second marked none; counting a call to such a twin as correct changes the XML lead over JSON Schema by 0.1% (Appendix Z).
One number the Reviewer cited has moved slightly, but the finding holds. The drop of the XML lead from the main rule to the symmetric rule is 12.73% before and after the fixes (13.80 to 1.07 in the submission; 15.25 to 2.52 now, Table 2). The lead that remains for XML under symmetric grading is larger than we reported: 2.5% with the corrected parser and 4.4 to 6.6% when a separate model reads the answers (Table 50). Strict-copy reader models that use none of our parsers reproduce the same drop (from 20% to 6 to 7%; Appendix AA), so the dependence on the grading rule is a property of the answers and not of our parser.
Official Response by Authors to Reviewer RXBF (3/3)
The toolkit's advisor combines separately measured effects, but its combined recommendations are not directly evaluated. [...] Can the actual advisor be evaluated on held-out models and tool catalogues against fixed defaults and direct calibration?
We thank the Reviewer for this excellent suggestion. Yes, and we did. The submission stated this limit (Appendix A.7) and already reported that direct calibration beats the fitted rule (Section 4.7, Table 31: 69.18 against 64.35). In the revised paper we evaluate the advisor itself, held out three ways (Appendix AC, Tables 57-58): with one model family removed from its tables at a time; on 21 models the study never ran, 13 of them from families it has no data for; and on tool catalogues it was not built from (the 66 real schemas and BFCL). The fixed defaults are always JSON Schema and always XML. Direct calibration measures the eight formats on the target, keeps the best, and scores it on other requests.
Accuracy in %, main rule (the advisor column shows its better variant in each row):
| Held out | Always JSON | Always XML | Advisor | Direct calibration |
|---|---|---|---|---|
| One family at a time, generated tasks | 54.42 | 63.34 | 64.58 | 64.66 |
| 21 unseen models, generated tasks | 41.05 | 65.48 | 65.48 | 66.81 |
| 20 unseen models, real tools | 59.09 | 58.50 | 58.50 | 64.69 |
| Study models, real tools | 64.07 | 59.09 | 62.40 | 66.15 |
| Study models, BFCL | 82.78 | 73.66 | 77.74 | 82.73 |
Our main observations:
- On generated tasks the advisor is level with the better fixed default (always XML). For a model it knows nothing about it returns XML.
- On real tools and on BFCL it is below always JSON Schema. Its XML default comes from the main rule, which forgives a flat nested field in XML answers only, and these catalogues are flat. Under the as-written rule it loses to JSON Schema on 16 of the 20 unseen models.
- Direct calibration is the best or level with the best in every row, when it is run on the tools the model will use. Rows 1 and 2 calibrate on 50 generated requests per format, rows 3 to 5 on half of the catalogue's own requests. On the unseen models with real tools it is 5.6% [0.7, 12.4] above always JSON Schema. A format picked on generated requests does not transfer to real tools (59.90 against 64.07;
Table 57).
The paper now describes the toolkit accordingly (Abstract, Section 4.7, lines 520-530): it gives a starting default based on the measured effects and runs the calibration on the user's own model and tools, and the paper recommends the calibration. Table 57 reports every setting under all three grading rules, including the ones where the advisor loses, and Table 58 shows how many requests the calibration needs (ten per format already match the best fixed default).
On presentation: Section 3.1 now explains the three choices in simple words, with one worked example and a who-sees-what table, and a reader's map and a list of terms open the appendix (pages 21-22).
Once again, we thank the Reviewer for the careful and supportive feedback. Your comments have helped us identify important areas for improvement and have strengthened the clarity and impact of our work; all three requested experiments are now in the revised paper. We appreciate your valuable time and hope that the revised paper addresses these concerns. Please let us know if there are any remaining questions/concerns; we would be happy to address them!
Official Response by Authors to Reviewer RXBF (1/3)
We greatly thank the Reviewer for the careful review. We are very encouraged that the Reviewer found that the paper convincingly demonstrates that evaluation choices can change apparent format advantages and that the empirical validation is extensive. We are also very happy that the Reviewer found that its explicit reporting of negative results and parser limitations makes the findings easier to assess. We also ran all three experiments listed in each of the questions by Reviewer.
We value the Reviewer's time and have made every change easy to track: all new content in the revised paper is green, new appendix sections are clearly labelled as "(Rebuttal Addition)", and each answer below cites the place in the paper where we have added new content/results in
code font. We have also placed all the new appendix sections at the very start of the appendix, so the Reviewer can quickly go through and verify the changes in the revised paper. Below, we address each concern raised by the Reviewer:We thank the Reviewer for this important question. Yes for the size of the format effect, and for five of the seven formats; for two formats the position changes. The submission documented the mismatch (
Table 41) and varied two of the pieces, defaults and constraints, inside JSON Schema (Table 8); in the revised paper we add the cross-format control the Reviewer asks for (Appendix Y,Tables 47-48).We gave every renderer a matched mode in which it writes every piece JSON Schema writes (nested enum values, whether a nested field is required, defaults, constraints), and had 51 models answer the seven formats twice in one session, as submitted and matched, with a JSON object as the answer format for all (357,000 graded requests; 714,000 with a second answer format). JSON Schema and YAML have the same text in both modes and serve as a placebo. Accuracy in %:
("Strict" grades the call as written; "symmetric" forgives a nested field written flat.)
Our main observations:
So on these models a gap of 7 to 10% remains between renderings that have the same information; with one renderer per format, that is as close to a notation effect as the design gets. The paper says so in
Section 4.2, lines 396-401. The same run with an XML call as the answer and on naturally phrased requests gives the same result; on the hardest tool tier JSON Schema and XML are first and natural language stays 5 to 7% behind. On 17 of the 20 anchor models (12 distinct models: three hosted at up to three reasoning efforts, six rows of Claude models and the four OpenAI models through the vendors' APIs) the seven format means lie within 1.7% of each other with matched content, against 7.8% as submitted (Table 48b).