acceptodds
Withdrawn from ICLR 2027

When Does Model-Specific Tool-Call Formatting Pay Off?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Different LLMs often prefer different tool-call formats. Should each model then get its own format? Our framework, FormatGain, separates the model–format in- teraction, the oracle gain of per-model formats over the best shared format, and the selection error of a small-budget rule. Decision margins link the first two: a model gains from another format only when its interaction advantage there ex- ceeds that format’s average deficit. So one interaction can come with a large gain or none, and a gain can still be lost in selection. Eight designs, five on our suite and three external, show three regimes (post hoc): little to gain, a gain lost, a gain kept. On WorkBench, the corrected interaction share is 0.38 but the oracle gain is only 0.001, and fixed json and a rule pooling the other models’ history both had lower observed regret than every probe. On three related designs of our suite, a ten-task probe gains 0.030 to 0.105 over the hindsight best shared format. The gains transfer only in part: with new models they held on the strict-parser and 4k-token designs but not on three others, they held on new task templates, and on new BFCL task families the probe showed no detected advantage over the pooled-history rule.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, tool-call formatting, model–format interaction, harness sensitivity, algorithm selection, budget-constrained format selection, selection regret

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m9G8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors develop a framework called FormatGain to assist in selecting tool-call formats for an incoming model. Concepts such as interaction share, oracle gain, and decision margins are introduced to guide format selection.

Critical strengths

Provides a theoretical contribution to interaction analysis.

Critical weaknesses
  • The motivation for studying the interaction effect is not clearly articulated.

  • It is unclear why analyzing historical data is beneficial.

  • The paper insufficiently situates its contribution within existing literature. For instance, Section 3.1 should be connected to two-way ANOVA. Is κ\kappa a newly introduced quantity?

Questions for authors
  1. Who should care? Is the target audience a general agent R&D team, tool developers, or model providers? What is the key takeaway for each?

  2. Why is this study needed? Table (a) already appears informative on its own. Why are newly developed concepts such as oracle gain and selection error necessary? Why not using benchmark for the selection?

  3. How are μ,αm,βh,γmh\mu, \alpha_m, \beta_h, \gamma_{mh} computed? Is κ\kappa in Equation (1) the same as κ\kappa in Equation (3)?

  4. What is the format selection algorithm? How is the proposed method practically useful?

Other Comments

An important format, TOON, could be studied, as it is designed to reduce token consumption.

Reviewer_LjhEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tackles a relevant question in LLM tool development, as stated in the first sentence of the abstract: ent LLMs often prefer different tool-call formats. Should each model then get its own format?.

I like the question; the experiments covered several popular models as well and evaluated their approach on BFCL and Workbench.

Critical strengths
  • The paper tackles an important problem and is sort of using a valid experimental setup.
  • The breadth of the experiments is quite wide
Critical weaknesses
  • The paper is basically incomprehensible in the first few reads, the writing is very dense and the premise is convoluted quite a lot.
  • The work also incorporates a 140 page appendix which is basically agentic logs, and contains several things which are unacceptable to be included for a submission like AX.1 : "A reviewer asked which single experiment was run after the estimator, the parser, the null and the headline were all frozen. " There is no need of an archival record for the paper and it only adds noise.
  • The appendices before the archival record are also noisy and mostly contains pure agentic output, i am not against using AI in research and development but adding a hundered page appendix is unnecessary especially when the main text still needs considerable work.
  • Main text: After several readings i still do not understand the interaction design authors propose here?
  • The main contribution FormatGain, is mentioned twice once in abstract and once in introduction, after that we cannot find it anywhere, what happened to it?
  • The discussion and limitations text is written in a confusing manner, sentences such as "Sensitivity is not opportunity (Proposition 1), and opportunity is not realized gain (Eq. (5))." mean nothing. where are the actual discussion and limitations of the work?
  • The final takeaway " So interaction alone should not justify adaptation; held-out comparisons against strong defaults should decide." Is there a work which argues that interaction only justify adaptation? is there any work which argues against strong defaults?
Questions for authors

Please address the weaknesses

Reviewer_6jLtOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops FormatGain to investigate the effects of tool-call format for different models. It analyzes 9 designs and find that similar interaction mahy not lead to similar gain and selection error can affect the gain on some benchmarks.

Critical strengths
  1. The research question is valuable.
  2. The writing is clear and easy to follow with proper background.
Critical weaknesses
  1. For the tool-use suite created by the paper, it might be not enough since it is related to math calculations which LLM agent should be good at.
Questions for authors
  1. For existing coding agents like Codex and Claude code, could the conclusion in the paper be transferred to them?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.