The paper presents several construction stages, yet how they contribute to the dimension of capability that is intended to evaluate on is unclear. For example, the paper introduces many terms like “workstreams,” “episodes,” “occurrences,” and “Artifact Plans.” After reading the paper, I’m confused on why these different levels are needed, and how they related to the questions being tested. This makes benefit of the benchmark unclear.
The designed questions in this benchmark does not reflect useful business operations. For example, authors use pipeline to find connection between different entities and write questions that needs multi-hop reasoning (e.g., appendix D.6 case 3). In this particular example, the method need to identify an employee who recorded a decision about customer-account labels, then find what that employee wrote about a separate business issue. It is hard to understand in relatic why someone in the company need to follow this connection. Without justification on why these questions matters particularly for a company setting, the contribution of these questions is hard to be distinguished from prior work on multi-hop information retrieval.
Earlier benchmarks such as HERB[1] already ontains business document and information retrieval questions. This benchmark claims the contribution is larger and more detailed company history, yet the paper does not clearly show how the history of the a company would result in the change on the questions being asked, important gap it falls, or what additional insight it provides.
[1] Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath, Kung-Hsiang Huang, Caiming Xiong, and Chien-Sheng Wu. 2025. Benchmarking Deep Search over Heterogeneous Enterprise Data. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 501–517, Suzhou (China). Association for Computational Linguistics.