- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies an acion integrity failure mode where agents mistake success elsewhere as succuss in current environment, with experiments on 60 adapted tasks and 13 models. Evaluations show that existing model has high false completion claim rate, and providing information on environment relatoin mitigates this failure and improves the completion rate.
Critical strengths
Identified and evaluated an action integrity failure mode in LLM agents, where agents mistake successful operations in the testing environment for the task completion in another execution environment.
Critical weaknesses
The evaluation only covers 60 unique task, with 36 from tau-bench and 24 from AppWorld, and only has single run per task for each model. Though the author emphasizes 780 runs in total, which may give a misleading impression on the statistical robustness, these evaluations are repeated on a narrow set of tasks that limit the reliability of the conclusions.
Questions for authors
Section 3.2 remains unclear why there're only 36 tasks selected from tau-bench and 24 from AppWorld, which are small numbers that limit the statistical robustness of the conclusion. Are these the only tasks that meet the selection criteria?
2 comments
Thank the authors for the clarification and additional experiments, which better support the conclusion. I am inclined to boardline accept given that these revisions has been integrated into the manuscript. A side note that it's better to highlight the revisions in the manuscript with other colors, which would enhances the readability.
Thank you for asking us to strengthen task coverage and repeatability. We now report three repeated runs on all 60 original tasks and a separate extension to 30 additional public tasks, both on six fixed deployments. Both reproduce the false-completion failure and the execution benefit of the local full join. Section references point to the revised manuscript.
1. Task coverage and statistical units
The main study has 60 source task instances (36 tau, 24 AppWorld). Across 13 deployments, each condition contains 780 task–deployment episodes; 12 conditions give 9,360 episodes. Section 3.4, Table 2 and Appendix D now distinguish these quantities explicitly. Statistical inference resamples tasks within each source, keeping each task's paired conditions and deployment measurements together rather than treating 780 rows as independent tasks. The six-deployment repetition and additional-task panels are reported separately with their own denominators.
2. Why 36 tau and 24 AppWorld tasks?
We fixed the 36-task tau and 24-task AppWorld allocation before collecting the reported model continuations, balancing source/operation coverage, native-qualification workload and API cost within the available candidate inventory. These were practical panel-size targets, not limits on task eligibility.
The tau screen identifies 690 structurally eligible instances; the selected 36 cover airline, retail and telecom and ten target APIs. The 24 AppWorld tasks cover distinct generator families across five applications. All 60 selected tasks passed independent native qualification: the reference execution succeeds, omitting the target write fails, and restoring it recovers success. Section 3.2 and Appendix A give the source-based selection rules and audit scope.
To assess whether the findings depend on this panel or a single rollout, we added 30 disjoint public tasks and three repeated runs on all 60 original tasks, both on six fixed deployments. The results below show that the false-completion failure and the benefit of the local full join persist.
3. Replication on 30 additional public tasks
We selected and natively qualified 30 disjoint tasks before collecting their new model outcomes, using fixed source-based ranking and round-robin rules: 24 new tau instances within the main study's operation types, and six AppWorld instances from six generator families absent from the main panel. All have passing native references, a necessary target operation, a state-distinguishing query and an executable recovery path. Task state, tools, inference settings, budgets and scoring are fixed across paired conditions.
Six deployments and one rollout per task–deployment–condition give 180 initially-incomplete episodes per arm:
The ordinary pre-execution control yields 2/180 false claims and 160/180 native target completions, showing that the tasks are executable without the rehearsal-success history. The local full join reduces false claims by 53.3 percentage points (source-stratified task-cluster 95% interval [46.1, 60.0]) and increases native completion by 60.0 points [53.9, 66.1]. Both outcomes improve in each source-level and deployment-level summary. Section 4.5 and Appendix B.7 (Tables 20–21) report this cohort separately: coverage spans 90 tasks, with a 13-deployment core and six-deployment extension.
4. Stability across three repeated runs
We concentrated the supplementary evaluation on six deployments to prioritize repeated runs and targeted controls while retaining coverage across developers and tool interfaces. The main study retains all 13 deployments. On all 60 main-study tasks, we ran pre-execution, distributed and local full join conditions three times on GPT-5.5, GPT-5.6-sol, DeepSeek-V4-Pro, Kimi-K2.6, grok-4 and gpt-oss-120b: 360 initially-incomplete episodes per condition per run.
The false-claim reduction is 53.3–55.3 points in every run. Pooled false claims fall from 747/1,080 to 159/1,080; native completions rise from 266/1,080 to 888/1,080. The paired reduction is 54.4 points (task-cluster 95% interval [49.3, 59.5]); resampling tasks and shared run indices gives [48.9, 60.0]. Pooled outcomes improve in each source-level and deployment-level summary. Section 4.5 and Appendix B.2 (Tables 12–13) provide the breakdowns. Together, these repetitions and the disjoint task extension strengthen robustness beyond a single rollout on the original panel.