- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies a counterintuitive failure mode in tool use: when a tool contract maps omission, null, empty values, and default values to different operations, the full table can lead a model to issue a valid, executable call that reaches the wrong final state. The authors introduce NothingBench, a paired synthetic benchmark for measuring this behavior. Within each task family, the aligned and conflict versions hold the user request, initial state, and target state fixed and differ in one rule row; responses are scored by deterministically executing the call and comparing the resulting record with the target state. Across 144 held-out families and five model configurations, relative to the one-line rule, the full table raises aligned accuracy from 81.3% to 85.6% but lowers conflict accuracy from 82.2% to 76.7%, for a difference-in-differences (DiD) of -9.9 percentage points. Across 720 conflict responses per representation, responses reaching the displaced state rise from 19 with the one-line rule to 77 with the full table, while invalid calls fall from 44 to 18. Follow-up studies show that the direction and magnitude of the effect depend on the task, model, and interface. The paper therefore argues that documentation changes should be evaluated on paired aligned and conflict tasks for the intended model and interface, using the executed final state rather than call validity or aggregate accuracy alone.
Critical strengths
The strongest part of the paper is its evaluation design. Paired aligned/conflict tasks and deterministic exact-state scoring reveal calls that are accepted and executed yet write the wrong state, an error that call validity and aggregate accuracy can hide. On conflict tasks, the full table yields 77 displaced responses versus 19 for the one-line rule, while invalid calls fall from 44 to 18; aggregate accuracy over aligned and conflict tasks changes only from 81.7% to 81.1%. This shift in error composition makes a strong case for evaluating the resulting state, and it is the main reason I lean toward acceptance.
Critical weaknesses
-
My main concern is that the full-table-versus-one-line comparison jointly changes the number of documented mappings, sentence-versus-table format, and information density. The four-row follow-up also changes the listed mappings, their order, the header, and the filler, and none of the six exploratory isolation contrasts survives Holm correction. The evidence therefore establishes a difference between the two representations but does not identify which aspect of the representation causes it. The authors acknowledge that the causal mechanism remains open; as a result, the paper currently offers stronger guidance about how documentation changes should be evaluated than about how documentation should be designed.
-
A second concern is external generality. The primary evidence comes from a single synthetic interface, while the source-grounded study is a descriptive case series of six selected rules from four public sources. For the three rules that depart from JSON Merge Patch, the source-status table improves exact-state accuracy by 18.3 percentage points relative to the one-line rule; for the three rules that follow Merge Patch, the difference is -1.7 points and the interval includes zero. Although the pooled gain is positive, the registered per-model success criterion was not met. In the call-interface follow-up, with raw patches, the four-row-versus-one-line DiD is -0.3 points and its interval includes zero. These results show that the failure can occur, but not yet how common it is in real tool-calling systems or when it should be expected on a new interface.
Questions for authors
- Each model-request pair is sampled once, and the intervals capture family variation but not rerun variation. For GPT-5.6 Terra, the new nominally identical off-mode run yields a DiD of -3.5 points, compared with -6.3 in the primary run. Could the authors report conflict accuracy and the displaced/invalid error composition for both runs alongside the already reported DiDs? Please also clarify whether claims about the effect size are intended to describe the observed frozen runs or expected behavior across repeated calls. If the former, the paper should state that scope explicitly.
Reviewers responses to authors
Dear reviewers,
Author-reviewer discussions end on November 18th. If possible, please respond to author comments by that time so that authors have opportunity for any additional responses.
Appreciate reviewers who already engaged in the discussion.
Thank you.