- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GroundWorks tests whether agents can handle changes during a multi-turn task, such as missing information, revised requests, or unexpected tool results. It creates pairs of scenarios from existing benchmark tasks, changing one condition between them. For each pair, it defines the evidence that matters and the response expected from the agent, such as asking for missing information or reporting a failed action. It then runs both scenarios separately and uses LLM judgments and code checks to assess the agent’s response. On average, agents failed in 47% of the perturbed scenarios whose originals they completed. About half of these failures involved targeted hallucinations. The authors also applied GroundWorks to healthcare tasks and found a similar pattern: success on the original task did not guarantee an appropriate response when conditions changed.
Critical strengths
-
GroundWorks examines aspects of agent reliability beyond task completion. It checks whether the agent's claims were supported by the evidence available at the time and distinguishes unsupported claims from other handling failures. This allows models with similar success rates to be compared by the kinds of mistakes they make.
-
The paired scenarios test how agents respond to changing conditions in a controlled multi-turn setting. Agents can ask follow-up questions and use tools to resolve the problem. The evaluation therefore examines both their ability to perform the original task and their response when information is missing or results differ from what they expected.
Critical weaknesses
-
Additional evaluation value of the integrated framework: The evaluation benefits of the integrated design are not sufficiently established. Prior work already studies multi-turn responses to incomplete information, distinguishes hallucinations from other failures, and uses operators to generate and validate perturbations (CAR-bench (https://aclanthology.org/2026.acl-long.1886.pdf); MIRAGE-Bench (https://arxiv.org/html/2507.21017v1); TrustDABench (https://arxiv.org/html/2608.24145v1)). GroundWorks brings these approaches into a single procedure that covers a range of conditions and specifies the relevant evidence and response criteria for each perturbation. However, the paper provides limited evidence that this integration better captures problems that existing evaluations struggle with or overlook. The failure breakdowns and applications to different tasks show that the framework can be used, but do not establish how much the integrated design improves coverage, consistent construction of valid scenarios, or adaptation to new tasks. The experiments show that GroundWorks can find failures, but offer limited evidence of which difficulties in existing evaluations its design resolves.
-
Reliability of failure classification: The reliability of failure classification, a core function of the tool, is not sufficiently established. GroundWorks's diagnostic value depends on accurately and consistently distinguishing targeted hallucinations from other handling failures. Its main analyses use this distinction to describe failure patterns and differences across models and conditions. Predefined criteria and code checks help verify the basis for judgments, but deciding whether a claim is supported and which failure it represents still depends on the LLM's interpretation. The authors conducted human reviews during development and a manual audit of the healthcare evaluation. However, the final-judge check in Table 9 involves one annotator reviewing 14 scenario pairs with the automated verdicts visible. This provides limited evidence that people would apply the criteria consistently or reach the same classifications without seeing the automated verdicts. The effect of misclassification on failure rates and comparisons across models and conditions also remains untested. The concern is therefore not simply that the judge may make some errors: the reliability of the failure-type diagnoses and comparisons central to the framework remains insufficiently established.
Questions for authors
-
Do human reviewers reach the same verdicts without seeing the AI's judgments? Distinguishing targeted hallucinations from other failures is central to the tool, so the accuracy of this distinction needs to be established. With the judge fixed, could multiple annotators assess cases not used during development without seeing its verdicts? Please report the models and conditions covered, agreement between annotators, how disagreements are resolved, and a confusion matrix across success, targeted hallucination, and other failure.
-
Does correcting judgment errors change the main findings? Using the same sample as Question 1, please compare failure-type proportions and the main model and condition comparisons under automated and adjudicated human labels. Report group sizes and uncertainty accounting for the sampling scheme; if the sample is insufficient, collect more labels or limit the conclusions. An inconclusive result from a small sample is not evidence of stability.
-
How often does the assessment find hallucinations in completed tasks? Finding new failures by changing test conditions differs from detecting problems that completion scoring misses. Cases where an agent completes the task but makes an unsupported claim would directly demonstrate what the tool adds to completion scoring. On feasible perturbed tasks, could the authors cross-tabulate task completion and targeted hallucinations for the same runs, and report the proportion of completed runs with hallucinations alongside examples? Since PS already requires the absence of targeted hallucinations, please assess task completion using criteria separate from the hallucination judgment.
-
How much does the perturbation increase failure? Section 5 claims that the design isolates causes, but the observed 47% failure rate does not distinguish perturbation effects from variation between runs. Could the authors repeat both versions on representative tasks and also measure failure when an unchanged original is rerun after a successful run? Since OS and PS use different criteria, direct comparisons should use tasks feasible in both versions and the same completion criteria, with handling outcomes reported separately.
-
Do other tasks fail similarly under the same condition? Some operators are evaluated on only one scenario pair from one source task (Table 11). Could the authors apply these operators to new tasks and check scenario validity and whether the failure patterns hold? Please also assess sensitivity to the existing task sample, separately from the new-task experiments.
6 comments
Author Response, part 2 of 4
W1 (Value of the integrated design): None of these works combines GroundWorks' four design decisions, each of which reveals limitations that prior evaluations miss.
Thank you for pointing us to these. We already cite CAR-bench (ACL 2026) and MIRAGE-Bench (arXiv preprint). TrustDABench is a concurrent preprint (arXiv:2608.24145v1, August 25, 2026), which we added to the revision. For each design decision, we state what the three works do instead and what it reveals.
First, targeted hallucination is formally defined (Section 3.1.1) and our evaluation distinguishes hallucination from other failures. In MIRAGE-Bench's out-of-scope setting, the lowest score groups together ignoring the query, answering with hallucinated information, and ending the task early. Our integrated design reveals that, of the 48 percentage points of failure for our submission's five agents, 26 are targeted hallucinations and 22 are other failures. For example, when the return tool is removed, Kimi K2.5 tells the user "the refund will be processed to your PayPal account within 5-7 business days" (targeted hallucination), while Claude Haiku 4.5 calls the exchange tool instead, which the user never requested (other failure). These two failure types call for different mitigations, and a single failure score would not tell them apart. Additionally, agents with similar failure rates also fail differently: 63% of failures by Kimi K2.5 are hallucinations, compared to 38% for GPT-5.6 Luna.
Second, every perturbed scenario is paired with its original, decoupling agent capability from robustness to perturbation. MIRAGE-Bench has no matched original scenario. Each test is a single snapshot with the risk already present, so a failure is never compared with the same agent's result on the unchanged task. CAR-bench and TrustDABench do not condition their hallucination or refusal measures on the same agent completing the original task (TrustDABench does so only for its robustness score). GroundWorks reports outcomes conditioned on the same agent's original success. Our integrated design reveals that 34% of all perturbed failures occur on tasks the agent also failed in the original. An unpaired evaluation would attribute them to the perturbation, and our paired analysis sets them aside.
Third, operators are defined independently of any benchmark and can expose agent failures in new domains. CAR-bench and TrustDABench each cover one domain (in-car assistance and structured-data analysis). Each of GroundWorks' 30 operators is defined by the grounding source it changes and the required response, so the same operators generate perturbations across tasks and domains. For consistent construction of valid scenarios, every candidate pair passes automatic code checks and a checker model from another model family (Appendix B.4). On PatientAgentBench (Section 4.4), 19 of the 30 operators transferred across all six families, and we used the operators to add 100 new cases from nine benchmarks (Q5). Our integrated design reveals that an agent's robustness depends on the domain. GPT-5.6 Luna and Claude Haiku 4.5 have similar failure rates in the main study (36% and 41%) but differ on healthcare tasks (31% and 60%), so a result on one benchmark does not carry over. For example, when a booking tool is removed, Claude Haiku 4.5 first says it cannot book the visit but later refers to a scheduled appointment, which does not exist. A patient could rely on a visit that was never booked.
Fourth, hallucination is also measured on feasible tasks in full multi-turn interactions, exposing completed tasks that still contain hallucinations. CAR-bench and TrustDABench measure hallucination or refusal only on requests that can no longer be satisfied. Their feasible variants (CAR-bench's disambiguation tasks and TrustDABench's answer-preserving perturbations) are not scored for hallucination. MIRAGE-Bench evaluates a single next action from a frozen snapshot, where follow-up actions "cannot be executed or observed", and does not measure task completion. GroundWorks runs full multi-turn interactions with a simulated user and scores targeted hallucination on feasible tasks, where agents must handle the change and still complete the task. It also covers world knowledge, earlier user statements, and saved memory as grounding sources (44 of the 308 pairs target world knowledge or saved memory). Our integrated design reveals that 10.4% of completed feasible tasks still contain a targeted hallucination (Q3). A completion metric would count these runs as successes.
In summary, in our integrated design, the operators define what changes, the matched original shows whether the change caused the failure, and the separate scores show what kind of failure it was. Each finding above relies on more than one of these. The 10.4% figure, for example, needs feasible perturbed tasks, a completion check, and a separate hallucination score in the same episode.
Author Response, part 3 of 4
W1, Q3 (hallucination in completed tasks): On feasible perturbed tasks, 10.4% of completed runs contain a targeted hallucination.
Thank you for proposing this test. We cross-tabulated task completion and targeted hallucination for the same runs, on feasible perturbed tasks (Appendix D.5). Completion is determined only by the completion judge's assessment of task requirements, independent of the handling verdict.
The table below shows the result per agent.
| Agent | Completed runs that contain a targeted hallucination |
|---|---|
| GPT-5.6 Luna | 7% (6/87) |
| Kimi K2.5 | 8% (7/83) |
| Claude Haiku 4.5 | 11% (8/76) |
| Ministral 3 14B | 14% (8/59) |
| Qwen3 32B | 13% (5/39) |
| Average of the five | 10% |
The average is 10.4% [6.5, 15.0]. Because GroundWorks scores completion and hallucination separately, it catches these runs.
In one example, the user asks Ministral 3 14B to change the air purifier in their order to a different size with night mode, but does not say which size. Ministral 3 14B does not ask; it picks the large variant, tells the user it "matches your requirements", and places the change. Large happens to be the size the task expected, so the run passes its task requirement, which checks that the agent makes the expected order changes. But the user never gave a size, and presenting the guess as the user's requirement is a targeted hallucination that only the separate hallucination score catches.
Q4 (run-to-run variation): The perturbation adds failure beyond run-to-run variation. We show this by rerunning the experiment.
Thank you for these concrete checks. We ran both scenarios again for a stratified sample of 106 pairs, with all five agents (Appendix D.5). The table below shows the first run beside the rerun. To directly answer your question, we report completion under the same criteria and handling separately, on the feasible tasks whose original was completed in the first run.
All p-values are above 0.05, so none of the differences between the two runs is statistically significant.
| Average of the five agents | First run | Rerun | McNemar p between runs |
|---|---|---|---|
| Original success (%) | 68 | 70 | 0.19 |
| Perturbed: handled (%) | 49 | 50 | 0.65 |
| Perturbed: targeted hallucination (%) | 27 | 27 | 0.80 |
| Perturbed: other failure (%) | 24 | 23 | 0.91 |
| Failure given original success in the same run (%) | 51 | 46 | 0.44 |
| Completed original fails when run again unchanged (%) | 9 | ||
| Same completion criteria: original completed (%) | 100 (by selection) | 93 | |
| Same completion criteria: perturbed completed (%) | 79 | 83 | 0.42 |
| Handling on those tasks: handled (%) | 68 | 69 | 1.00 |
| Handling on those tasks: targeted hallucination (%) | 15 | 17 | 0.69 |
Between the runs, every average in the table changes by less than 5 points, and the two runs give the same outcome on 86% of the originals and 80% of the perturbed scenarios. No change is significant (exact McNemar test, pooled over the five agents). To bound the change between runs, we also computed its interval: the handled rate changes by 0.9 points [-3.1, 4.8] and original success by 2.3 points [-0.6, 5.4], so on these measures run-to-run variation is at most about 5 points.
Additionally, in 143 agent–pair cases, only the perturbed scenario fails, compared to just 11 where only the unchanged original fails (exact McNemar p < 0.001 for every agent). Therefore, the perturbation has a clear effect within the same rerun. The like-for-like comparison uses the same completion criteria: in the rerun, 93% of the unchanged originals are completed again, against 83% of their perturbed scenarios. On the full set, the drop is 10.3 points [5.7, 14.8].
Counting handling as well, a completed original fails 9.3% of the time [6.0, 13.5] when it is run again unchanged, against 47.1% for its perturbed scenario in the same rerun, a difference of 37.8 points [29.6, 45.7] that is above zero for each of the five agents. Both rates are over the originals completed in the first run, while the table's 46% conditions on the rerun's own originals. These two rates use different success criteria, so the same-criteria drop is the stricter comparison. Both show that the perturbation adds failure beyond run-to-run variation (Appendix D.5).
Author Response, part 1 of 4
We thank the reviewer for recognizing that GroundWorks lets models with similar success rates be compared by the kinds of mistakes they make, in a controlled multi-turn setting where agents can ask follow-up questions and use tools.
We ran additional experiments and analyses. Please find the answers below. The revised PDF is also uploaded with changes marked in blue. We also expanded the study from 208 to 308 pairs by adding 100 pairs from nine benchmarks, and every operator now has at least five pairs. The results below are from this extended set.
W2, Q1 (blind human verdicts): Three blind annotators labelled 120 episodes, and the final human labels agree with the judge on 83.8% of them.
Thank you for laying out the study so precisely. We ran this human study.
Annotators and cases. Three annotators, all authors of the paper, labelled 120 perturbed episodes of the final run without seeing the judge's verdicts. None had seen these episodes before, and none of the cases was used in development.
Models and conditions covered. The sample has 8 episodes for each combination of the five agents and the judge's three outcomes, so 24 per agent, and it covers all six families and all 30 operators.
Agreement and disagreements. Two annotators labelled each episode independently as handled, targeted hallucination or other failure. The two annotated labels agree on 89 of the 120 episodes (inter-annotator agreement 74%, kappa 0.60). A fourth author settled the remaining 31 disagreements.
Confusion matrix. We compare the judge's verdicts with the final human labels. On the 120 sampled episodes, they agree on 74.2%. After weighting to the sampling frame, they agree on 83.8% of episodes (κ = 0.72).
The table below is the confusion matrix across handled, targeted hallucination and other failure, with the judge's verdict in rows and the final human label in columns. Counts are the 120 episodes as drawn.
The table below gives the judge's precision and recall for each label, weighted to the frame.
W2, Q2 (effect of correcting judgment errors): Conditional failure is 36.3% under human labels and 44.3% under judge labels.
We agree that this comparison is needed. We compared the judge with the final human labels on the same sample as in Q1, weighting each stratum by its size in the sampling frame.
Every failure type has a lower rate under human labels, with total failure 8.0 percentage points lower than under judge labels, suggesting that the judge is stricter than human annotators. However, even under human labels, agents still fail in more than a third of scenarios whose originals they complete.
Group sizes and uncertainty. Each agent has 24 labeled episodes, including 11–18 with successful originals, while each family has 11–30 episodes. Accounting for stratified sampling, the 95% confidence interval for the pooled failure rate is about 16 percentage points wide and includes the judge's estimate.
Comparison under the two sets of labels. Human labels preserve the judge's ranking of the five agents, except that Kimi K2.5 and Claude Haiku 4.5 swap positions, which were already very close in the paper. Under both sets of labels, Qwen3 32B and Ministral 3 14B have the highest failure rates, while GPT-5.6 Luna has the lowest. Additionally, tool capabilities and action outcomes has the highest failure rate among the three families with at least 15 labelled episodes, consistent with the judge's results.