- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LongTool-Bench introduces controlled changes to tools in 200 AppWorld tasks that require persistent state and extended sequences of actions. Its protocol verifies executable recovery requirements, changes calls at meaningful boundaries while keeping earlier state updates, and tests 16 models with unchanged tools, concealed changes, visible changes, or oracle documentation. It records completion of the original task separately from resolution of the tool-change events.
Critical strengths
- Section 3 specifies that earlier completed effects remain in place when a call is interrupted, and it verifies that recovery remains possible from that point. This creates a defined and realistic test of tool changes beyond simply injecting an error response.
- The matched evaluation reports distinct outcomes for event resolution and the original objective. Of 7,008 evolving episodes that finish their complete event chains, just 924 also pass the task assertions (Section 4.2). The separate counts make clear that handling a local change does not necessarily complete the task.
Critical weaknesses
- Across models, success without tool changes ranges from 1.33% to 18.67% (Table 3), indicating substantial difficulty in the base tasks. For each model, the authors should additionally analyze tasks solved consistently in the unchanged condition and report paired uncertainty, while retaining the results for the full task set. This would better estimate the portion of failures associated with adaptation to change.
- A change is encountered only when the agent reaches a suitable boundary. Consequently, the hidden, visible, and oracle-documentation settings may expose different collections of episodes (Sections 3.3 and 4.4). Recovery conditional on exposure and success within exposed categories do not therefore compare identical cases. Although the paper mentions this issue, its discussion still gives considerable weight to those differences. Replaying conditions from the same recorded state before an event, or estimating bounds for the selection effect, would support a more precise conclusion about the information provided.
Questions for authors
- On tasks each model solves in all three unchanged repetitions, what are the paired outcomes under tool changes? How does restricting attention to these tasks affect the observed success gap?
- For a subset of encountered changes, could all three information settings start from the same saved state immediately before the event? This would allow recovery rates to be compared for identical events.
2 comments
Response to Reviewer uCsP (2/2) - Same-state recovery comparisons
Q2 Recovery from the same saved state
Yes. We completed three-condition recovery comparisons from the same recorded first-event boundary. Three fresh processes replay each saved executable prefix and verify the database, persistent shell, controller/tool state, agent context, pending call, counters, remaining budgets and controllable RNG against the source. The endpoint is resolution of the target first event.
The separate Qwen3-8B/14B NF4 validation uses the first six tasks in frozen metadata order, each from a different family, as a nested interim scope added after data collection began. The primary scope comprises 12 tasks. The table uses source repeat0 and one branch per condition for each captured source.
| Same-state NF4 validation | Complete triads / families | Hidden resolved | Observable resolved | Oracle-Docs resolved | Observable−Hidden wins / ties / losses |
|---|---|---|---|---|---|
| Qwen3-8B, initial NF4 replay | 6 / 6 | 0/6 | 6/6 | 6/6 | 6 / 0 / 0 |
| Qwen3-14B, initial NF4 replay | 4 / 4 | 1/4 | 4/4 | 4/4 | 3 / 1 / 0 |
| Qwen3-14B, independent serializer repair | 1 / 1 | 0/1 | 1/1 | 1/1 | 1 / 0 / 0 |
Resampling the represented families gives Observable−Hidden differences of 100 pp [100,100] for 8B and 75 pp [25,100] for 14B in the initial replay, with identical Oracle-Docs−Hidden differences. These estimates have six and four families, respectively. The 8B interval is degenerate because all six paired differences agree.
The twelve original candidate sources comprise ten complete triads, one normal nonexposure and one dictionary-view serialization failure. The independent repair replays saved responses and code with zero new source inference calls. Partial databases and the pending API have directly saved comparison objects. The shell is reconstructed from saved code and RNG, and all three repaired branches match the complete reconstructed CPU boundary. Hidden exhausts eight recovery calls, while Observable and Oracle-Docs each resolve the target in one.
This validation uses NF4 weights, BF16 compute, temperature0.2 and a 16,384-token context. Hidden receives an event identifier and validator errors, differing from Section 3.3. The comparison concerns the complete implemented feedback policies, including initial observations, action guidance and retry feedback.
A historical Core-50 repeat0 sensitivity retains 148 blocks whose first planned event fires in all three conditions, spanning 36 families per model. Observable−Hidden recovery differences are 58.00 [43.75, 71.74] pp for Qwen3-32B, 82.00 [70.00, 92.31] for Gemma-3-27B-IT and 54.17 [40.82, 67.39] for Mistral-Small-24B. Oracle-Docs−Hidden matches these values for Qwen/Mistral and is 94.00 [86.54, 100.00] for Gemma. This post-treatment-conditioned common-exposure analysis complements the saved-state experiment's stronger database, shell and context controls.
Conditional LR and category-exposed OS use condition-specific fired/exposed sets. Full-episode OS and strict ECR use every evaluated episode and include routing, exposure, recovery and continuation. Strict ECR requires all planned events to fire and resolve. The controlled results support recovery differences for the evaluated feedback policies on captured encounters. The share of overall task failures attributable to adaptation requires further identification. We will incorporate the completed tables and feedback/denominator definitions into the revised manuscript, with category-level mechanisms stated as hypotheses.
Response to Reviewer uCsP (1/2) - Three-Clean-success subset
Q1 Outcomes on tasks solved in all three Clean repetitions
In the supplementary historical Core-50 evaluation, all selected tasks fail under each changed condition at the available matched repeat0. We completed Clean repeats1/2 for every task that succeeded in Clean repeat0, adding 12 episodes, all officially successful. The other 144 model/task combinations across the three historical models have a Clean repeat0 failure, determining their exclusion from the three-success intersection. Membership is therefore determined throughout each model's original 50-task scope.
Selection requires Clean success in repeats0/1/2. The paired changed-condition comparison uses repeat0. The following values apply separately to Hidden, Observable and Oracle-Docs, whose task success counts are identical in these records. Gaps are changed-condition OS minus Clean OS.
Gemma-3-27B-IT has Clean and changed successes of 0/50, yielding a full-set gap of 0 pp [0,0], an empty strict subset and undefined restricted statistics. The separate NF4 six-task scopes for Qwen3-8B/14B also have empty strict subsets because every task has a Clean failure.
Intervals use 10,000 paired scenario-family bootstrap resamples. For Q1, we sample each model's 36 original families with observed subset membership fixed. Selected-gap intervals use the 9,576 defined resamples for Mistral and 8,764 for Qwen, with 424 and 1,236 undefined selected denominators, respectively. Their zero width reflects identical losses among only three and two represented families. Selection fixes selected Clean OS at 100%, so the gap change describes the selected comparison.
The added runs use the frozen historical revisions and worker, BF16 weights, greedy temperature0 decoding, and the original generation budgets and context rules. All 12 reproduce their corresponding repeat0 output sequences, establishing execution consistency under this configuration. The historical v33 worker retains its recovery serializer system header after accepted recovery, so the measured losses include that implementation's recovery and continuation policy.