- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EnterpriseWorld contains 200 enterprise workflows with 20 operations each and executable dependency checks. The study separates local correctness from workflow completion and uses oracle information and matched perturbations to examine error propagation.
Critical strengths
-
Dependencies are tested by corrupting upstream inputs, with additional untouched-state, damage, and skipped-operation controls in Appendix D.3. This makes the workflow construction concrete and verifiable.
-
Section 4.4 and Table 14 usefully locate many failures in upstream computation rather than information transfer alone. The paper also acknowledges ordinary error accumulation and the extra work introduced by path deepening.
Critical weaknesses
-
I would like stronger evidence of realism. Verifier sensitivity alone does not establish a plausible business requirement. Human validation helps, but annotators see model verdicts and reasons, and Table 3's workflow-level interval is wide, [71.7%, 98.9%]. Please also compare the construction directly with WorkArena++ [W25], ToolSandbox's state dependencies [TS25], and ResiliBench's stochastic workflow failures [RB26], specifying what the dependency interventions add.
-
The 14–26% perturbation break rates lack an unchanged-workflow repeat for comparison. With one attempt per cell, some failures may reflect ordinary variability. The paper acknowledges this limitation, but a repeat reference is needed to interpret the magnitude. I also could not locate the bidirectional transition counts referred to in Appendix D.
I lean toward acceptance for the executable dependency construction and process analysis. Stronger realism validation and a repeat reference would improve the conclusions.
Questions for authors
-
How many workflows were human-reviewed, how were they sampled, and how were questionable compositions handled? A short comparison with [W25, TS25, RB26] should distinguish state dependence, structural perturbations, stochastic failures, and scoring.
-
Is an unchanged-base rerun available for the perturbation subset? Please compare break rates and provide both success-to-failure and failure-to-success counts.
[W25] Boisvert, L., Thakkar, M., Gasse, M., Caccia, M., Le Sellier De Chezelles, T., Cappart, Q., Chapados, N., Lacoste, A., and Drouin, A. (2025). WorkArena++: Towards Compositional Planning and Reasoning-Based Common Knowledge Work Tasks. arXiv:2407.05291, version 2. https://arxiv.org/abs/2407.05291.
[TS25] Lu, J., Holleis, T., Zhang, Y., Aumayer, B., Nan, F., Bai, H., Ma, S., Ma, S., Bao, M., Yin, G., Wang, Z., and Pang, R. (2025). ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. Findings of the Association for Computational Linguistics: NAACL 2025, 1160–1183. https://doi.org/10.18653/v1/2025.findings-naacl.65.
[RB26] Ao, R., Min, Z., Zhu, T., Yin, W., and Wang, X. (2026). ResiliBench: Evaluating Agentic Workflow Adaptation in Stochastic Environments. International Conference on Learning Representations. https://openreview.net/forum?id=KTZ56LG7jZ.
6 comments
Hi, just a quick reminder that we’ve posted our rebuttal. Looking forward to your feedback, and thanks again for taking the time to review!
Thank you for the response and the reruns. The comparison table answers my related-work question. The response does not say whether it will be added to the paper, and the submission covers only part of this line of benchmarks, so the positioning still seems incomplete for a benchmark paper. I would encourage the authors to include the comparison.
My main concern was whether the perturbation break rates exceed rerun variability. The new counts suggest they do not. In the original run the six perturbations give 51 success-to-failure and 51 failure-to-success transitions in total, so variant success stays between 40 and 46 out of 100 against Base 43/100. The reported Base-to-Base break rates (7.7% to 20.9%) also overlap the perturbation rates. I am therefore not convinced that the 14% to 26% figures in the abstract show fragility rather than run-to-run churn, and Finding 1 would need to be restated.
The audit is described as 30% of workflows, but no count is given, and the workflow-level interval in Table 3 appears to correspond to 15 of 16 judgments. Because the perturbation result is a headline finding and the main point of difference from prior benchmarks, I have lowered my score to 2.
Response to Reviewer 7ZuK
Thanks for your insightful review. We appreciate your recognition of the value of our work!
Response to Concern of Realism
We sincerely thank the reviewer for raising this concern. AI verdicts and rationales were shown to facilitate annotation, but annotators were explicitly informed that these judgments could be incorrect, and the AI model's identity was concealed. The reported audit randomly sampled 30% of retained workflows, and full-dataset annotation is ongoing; we will update the statistics accordingly.
Our realism is grounded in plausible enterprise business processes, where calculations constrain approvals, payments, and reconciliations across shared records. Beyond verifier sensitivity, human annotation explicitly assesses workflow coherence, business justification, missing steps, and quantity mappings.
Regarding related work, WorkArena++ evaluates composed ServiceNow tasks, ToolSandbox evaluates state-dependent tool use through milestone DAGs, and ResiliBench examines workflow resilience under stochastic tool failures. EnterpriseWorld instead emphasizes counterfactually verified dependencies and matched structural interventions that deepen dependency paths or add prerequisites, enabling controlled evaluation of changing coordination requirements rather than stochastic execution failures.
Response to Concern of Perturbation Repeat
We sincerely thank the reviewer for raising this concern. We conducted two additional repetitions of the unchanged Base and all six perturbations on the same 100 workflows using GPT-5.6 Sol. Each perturbation is paired with its corresponding Base run. We report the success-to-failure (S→F) transitions across all three runs and the original failure-to-success (F→S) counts.
Unchanged Base-to-Base comparisons show 7.7–20.9% natural break rates, with both S→F and F→S transitions. Despite this variability, all six perturbations consistently break previously successful workflows across all three runs.
Response to Human Annotation
We sincerely thank the reviewer for raising this concern. The human audit reported in the paper is based on a randomly selected 30% sample of the workflows retained after data filtering. We are also continuing human annotation across the full dataset. As this annotation progresses, we will update the audit results with the expanded coverage, explicit workflow counts, and corresponding annotation statistics.
Response to Literature Comparison
We sincerely thank the reviewer for raising this concern. We clarify the differences in state dependence, workflow interventions, stochastic failures, and scoring below. Here, stochastic faults refer to injected tool failures.
EnterpriseWorld differs by combining counterfactually verified dependencies with matched structural interventions that deepen dependency paths or add prerequisites. Its node/edge and closure checks further distinguish local correctness from dependency-valid completion, enabling controlled evaluation of changing workflow coordination requirements.