acceptodds
Under review as a conference paper at ICLR 2027

Please Put Nothing Here: When Tool Documentation Helps LLMs Make the Wrong Call

Abstract

In tool interfaces, leaving a field out, sending null, sending an empty value, and sending the default can mean different things, depending on each tool's documented rules. A model can therefore send a call that is valid and executes, yet writes a state the user did not ask for. We introduce \bench, which pairs synthetic partial-update tasks that differ in one documented rule and scores the exact resulting state by deterministic execution. In aligned tasks the most direct call is correct. In conflict tasks the rule makes that call write a predictable wrong state. We test whether documenting every input form improves accuracy over stating only the rule the task depends on. Across 144 held-out task families (each an aligned and a conflict version of one task) and five model configurations, a table of equal length that gives the operation for all ten ways of writing a field raises aligned accuracy over a one-line rule but lowers conflict accuracy from 82.2% to 76.7%, below even a schema that gives no rule for omitted, null, empty, or default inputs (82.9%). The difference-in-differences (the table's effect on conflict tasks minus its effect on aligned tasks) is points and negative for all five models. Accepted calls that reach the predicted wrong state rise from 19 to 77 of 720 conflict responses, while invalid calls fall from 44 to 18 and aggregate accuracy barely moves. Four-row tables recover much of the loss. On four newer models, the table's effect is smaller on average and varies by model. In a small case series on three protocol rules that depart from JSON Merge Patch, a short table scores higher than the one-line rule. Documentation changes should therefore be evaluated on paired conflict tasks and on the states they produce, not only on call validity or aggregate accuracy.

Primary area:
datasets and benchmarks
Keywords:
tool use, function calling, LLM agents, benchmark, semantics of absence, structured output, state tracking, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JRYLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a counterintuitive failure mode in tool use: when a tool contract maps omission, null, empty values, and default values to different operations, the full table can lead a model to issue a valid, executable call that reaches the wrong final state. The authors introduce NothingBench, a paired synthetic benchmark for measuring this behavior. Within each task family, the aligned and conflict versions hold the user request, initial state, and target state fixed and differ in one rule row; responses are scored by deterministically executing the call and comparing the resulting record with the target state. Across 144 held-out families and five model configurations, relative to the one-line rule, the full table raises aligned accuracy from 81.3% to 85.6% but lowers conflict accuracy from 82.2% to 76.7%, for a difference-in-differences (DiD) of -9.9 percentage points. Across 720 conflict responses per representation, responses reaching the displaced state rise from 19 with the one-line rule to 77 with the full table, while invalid calls fall from 44 to 18. Follow-up studies show that the direction and magnitude of the effect depend on the task, model, and interface. The paper therefore argues that documentation changes should be evaluated on paired aligned and conflict tasks for the intended model and interface, using the executed final state rather than call validity or aggregate accuracy alone.

Critical strengths

The strongest part of the paper is its evaluation design. Paired aligned/conflict tasks and deterministic exact-state scoring reveal calls that are accepted and executed yet write the wrong state, an error that call validity and aggregate accuracy can hide. On conflict tasks, the full table yields 77 displaced responses versus 19 for the one-line rule, while invalid calls fall from 44 to 18; aggregate accuracy over aligned and conflict tasks changes only from 81.7% to 81.1%. This shift in error composition makes a strong case for evaluating the resulting state, and it is the main reason I lean toward acceptance.

Critical weaknesses
  1. My main concern is that the full-table-versus-one-line comparison jointly changes the number of documented mappings, sentence-versus-table format, and information density. The four-row follow-up also changes the listed mappings, their order, the header, and the filler, and none of the six exploratory isolation contrasts survives Holm correction. The evidence therefore establishes a difference between the two representations but does not identify which aspect of the representation causes it. The authors acknowledge that the causal mechanism remains open; as a result, the paper currently offers stronger guidance about how documentation changes should be evaluated than about how documentation should be designed.

  2. A second concern is external generality. The primary evidence comes from a single synthetic interface, while the source-grounded study is a descriptive case series of six selected rules from four public sources. For the three rules that depart from JSON Merge Patch, the source-status table improves exact-state accuracy by 18.3 percentage points relative to the one-line rule; for the three rules that follow Merge Patch, the difference is -1.7 points and the interval includes zero. Although the pooled gain is positive, the registered per-model success criterion was not met. In the call-interface follow-up, with raw patches, the four-row-versus-one-line DiD is -0.3 points and its interval includes zero. These results show that the failure can occur, but not yet how common it is in real tool-calling systems or when it should be expected on a new interface.

Questions for authors
  1. Each model-request pair is sampled once, and the intervals capture family variation but not rerun variation. For GPT-5.6 Terra, the new nominally identical off-mode run yields a DiD of -3.5 points, compared with -6.3 in the primary run. Could the authors report conflict accuracy and the displaced/invalid error composition for both runs alongside the already reported DiDs? Please also clarify whether claims about the effect size are intended to describe the observed frozen runs or expected behavior across repeated calls. If the former, the paper should state that scope explicitly.
Reviewer_vuPEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how tool documentation affects LLM ability when omitted fields, null, empty values, and defaults have different contract-defined meanings. The authors introduce NothingBench, which pairs aligned and conflict tasks differing in one documented rule, and evaluate five schema representations across 144 held-out task families and five primary model configurations using deterministic exact-state scoring. Compared with a length-matched one-line rule, the full ten-row table improves aligned accuracy but reduces conflict accuracy from 82.2% to 76.7% and increases accepted calls reaching the predicted wrong state. It also shows shorter tables recover much of this loss, while follow-up experiments show improvements from short tables.

Critical strengths
  1. NothingBench distinguishes valid calls from correct resulting states through deterministic execution. The executor separates an absent field, an explicit override, and an inherited default, allowing evaluation of errors that call validity alone would miss.

  2. The evaluation covers multiple model families and reports model-specific representation effects. The five primary configurations include three API models and two locally served Qwen checkpoints, while the additional cohort introduces four further models. Figure 3 and Table 13 report individual conflict-accuracy changes and DiDs, making the heterogeneous effects visible.

Critical weaknesses
  1. The FULL to ONE-LINE comparison changes information coverage and presentation simultaneously, and the follow-ups do not establish which difference explains the reported conflict loss. ONE-LINE states only the task’s rule row, whereas FULL lists all ten surface-form mappings, while not controlling the relevance or prominence. Additionally, the four-row experiment may also change “the listed mappings, their order, the header, and the filler,” while none of the six isolation-diagnostic contrasts survives Holm correction. Consequently, the experiments do not convincingly establish an explanatory advance beyond the sensitivity to documentation content and prompt format, limiting the significance of the central finding.

  2. The source-grounded case series does not establish that the ten-row FULL table’s conflict loss transfers beyond the synthetic update_record setting. The primary tasks use one synthetic interface, shared development/test templates, and JSON in text rather than native function calling. The source-grounded evaluation covers only six selected rules from four sources, implemented as local deterministic fixtures, and replaces the primary comparison with a four-row source-status table and raw JSON patches. The limited real source size and comparison setting remains insufficient to support the central observation can be generalized to real-world tasks.

Questions for authors
  1. How should the pooled representation effects be interpreted given the substantial model-specific differences and observed rerun variability? Qwen3-14B gains 11.8 points in conflict accuracy with FULL, whereas Qwen3.8-27B loses 15.3 points, the additional cohort’s pooled DiD interval includes zero. Appendix F also reports that GPT-5.6 Terra's DiD changes from −6.3 to −3.5 in a nominally equivalent rerun, while the reported intervals do not capture rerun variation. Could the conclusions distinguish effects consistent across configurations from effects driven by particular models and even particular runs?

  2. What motivates the selected documentation representations, and how would examples affect their comparison? Appendix B defines the flat operation enum and typed union, but their selection and practical representativeness are not sufficiently explained. Appendix A explicitly presents a “Complete surface-form truth table, no examples,” while Appendix I examines additional presentations without a worked-example condition.

  3. What does NothingBench establish beyond the sensitivity to tool documentation and prompt format? The paired contracts and displaced-state scoring identify a specific error pattern, but the FULL–ONE-LINE comparison also changes information coverage and presentation style, while the isolation diagnostic remains inconclusive after Holm correction. Could the authors clarify what additional conceptual or empirical insight this design provides beyond showing that documentation changes affect tool-call accuracy?

Reviewer_MdYsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces NothingBench to study how the presentation of tool documentation affects partial-update calls. Its synthetic tasks pair two contracts that differ in one rule for omitted, null, empty, or default values, then score the resulting record state. Across 144 held-out families and five models, a ten-row rule table helps when the direct call is correct, but on conflict tasks accuracy falls from 82.2% to 76.7%. Valid calls that write the predicted wrong state increase from 19 to 77 out of 720 conflict responses. Follow-ups test four more models and selected rules drawn from public API documentation.

Critical strengths
  1. The paper identifies a failure that call-validity checks can miss: an accepted update can leave a record in the wrong state. The paired contracts and exact-state oracle in Section 3 make that failure observable and tie it to a specific documented rule.
  2. The controlled finding is supported by a careful evaluation. The held-out, family-clustered analysis in Section 4 and fresh-family follow-up in Section 5 give credible evidence that documentation presentation can change the rate of valid but incorrect updates in this setting.
Critical weaknesses
  1. The paper does not establish why NothingBench is needed as a standalone benchmark. The Introduction's API examples show that omission and null can have different meanings, but not how often models make the targeted wrong-state error in real tool use. Section 2 distinguishes NothingBench through paired rule flips without showing a consequential failure that existing stateful tool evaluations miss. A benchmark need not improve downstream model scores, but it should establish what new knowledge its measurements provide. That incremental contribution remains unproven.

  2. The controlled result has uncertain practical value. Section 3 compares a sentence that preselects the exact rule needed for each task with a ten-row table that also changes layout and information density. The primary tasks use one fictional four-field interface. Section 6.1 finds only a small pooled effect across four newer models, while Section 6.2 finds that a short table helps on selected source-grounded rules under another interface. Section 7 acknowledges that deployment frequency and downstream losses are unmeasured. These results do not yet show whether the diagnosis transfers to realistic tool use or guides a practical documentation or interface decision.

  3. The writing obscures the paper's actual scope. The title, abstract, and Introduction frame a broad real-API problem without plainly stating that the main study compares a task-specific rule sentence with a fuller rule table. The reader must reach the later sections to understand this focus. The paper has no separate Conclusion, and Section 7 discusses findings and limitations without a concise synthesis of the benchmark's claimed contribution. This presentation makes the paper's central claim harder to assess.

Questions for authors
  1. What important failure does NothingBench reveal that existing final-state tool benchmarks would miss?
  2. Which existing result shows the main finding outside the fictional update_record interface?

Comments on the paper

1 comment
Area_Chair_brQMOpenReview ↗

Reviewers responses to authors

Dear reviewers,

Author-reviewer discussions end on November 18th. If possible, please respond to author comments by that time so that authors have opportunity for any additional responses.

Appreciate reviewers who already engaged in the discussion.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.