acceptodds
Under review as a conference paper at ICLR 2027

EnterpriseSaga: A Billion-Token Benchmark for Evidence Retrieval and Reasoning over Evolving Enterprise Knowledge

Abstract

Enterprise question answering requires connecting evidence across systems while distinguishing activities that share participants, terminology, and business context. We introduce EnterpriseSaga, a billion-token benchmark for evidence retrieval and reasoning built around a synthetic B2B SaaS company. It contains 966 questions spanning 18 months and 12 enterprise systems, with 1.213 billion tokens in the exported serialization. Our Coarse-to-Fine Enterprise Simulation Framework develops projects and recurring operations within a shared company history and generates artifacts grounded in their activities. To represent the repetition and overlap of real company operations in finer detail, question-conditioned augmentation enriches local business contexts with artifacts of comparable activities. Across seven evaluated systems, reranking and iterative search improve answer quality, although full-corpus performance remains substantially below the gold-evidence reference. System rankings reverse between multi-hop and synthesis questions, revealing different strengths in relational search and information coverage. Scale experiments show lower performance as the company background expands, with questions and annotated supporting files held fixed. Additional analyses examine evidence use, computational costs, reader–judge sensitivity, and question and annotation quality through human assessment.

Primary area:
datasets and benchmarks
Keywords:
Benchmarking, Information Retrieval, Multi-document Reasoning, Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wMTYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This article constructs a benchmark to test how models retrieve information and handle workflows in ultra-large-scale workflows. It conducts a detailed analysis and tests mainstream models. It is practically significant.

Critical strengths

It has practical significance, focusing on the ability of models to retrieve and reason in real work scenarios.

Critical weaknesses

The article didn't make it clear what a GPT Agent is, and it didn't mention Harness, which is commonly used in these types of work, like Codex and Claude Code.

Questions for authors

What is not clear is what a GPT agent is, is it an agent built by itself, or the Codex framework? It needs to be explained clearly. If it is an agent built by the author, it is necessary to test the capabilities of the Codex framework, because the Codex framework is widely used in mainstream work.

Reviewer_6uW1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes EnterpriseSaga, a synthetic corpus for evaluating large-scale enterprise retrieval based on a synthetic SaaS company that contains almost 1 million files and 1B tokens. The corpus is designed to mirror the real-world scale and complexity of enterprise retrieval. The experiments show some findings e.g., different system perform different on various types of questions, and as the company scale up the performance degrate.

Critical strengths
  1. The direction itself is a super interesting and exciting direction. Being able to simulate a company to bypass the difficulty of publishing real-world enterprise data for academic research in information retrieval is useful to the community.
  2. The paper at least offers some human validation, although the scale is not considered large.
Critical weaknesses
  1. The biggest concern here is still whether the simulation framework can model the real-world complexity, e.g., variance in different communication styles of employees and external people, different knowledge levels, friction in communication, and noise in written communication. Simulation is still ideal and inherits the pattern of the LLM used in the simulations, especially when the simulation is conducted from scratch to model a super long-term simulation over short-term, so that the patterns and imperfections would accumulate over rounds.

  2. Given the limitation in corpus generation, I am not fully convinced about whether such a corpus would affect the development and research of a retrieval system. Does a system perform well on the simulated corpus to work well on real-world data in reality? It would be helpful if the experiment could study the correlation of evaluating models and systems on this dataset vs existing ones and private ones.

Questions for authors

Please refer to the weaknesses.

Reviewer_PbCjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This benchmark paper presents a synthetic benchmark evaluating LLM capability for information retrieval and question answering in a company setting. The synthetic data is generated as follows: authors create a fictional company with virtual employees, projects, and events and the data spans over 18 months across different sources. Through experiments, authors show that the reranking and iterative search could improve answer quality; the performance degrades as more confounding and noisy background files are added.

Critical strengths

The paper studies an important problem of information retrieval in the enterprise setup; and the paper contributes a bigger, and more complex benchmark than prior works.

Critical weaknesses

The paper presents several construction stages, yet how they contribute to the dimension of capability that is intended to evaluate on is unclear. For example, the paper introduces many terms like “workstreams,” “episodes,” “occurrences,” and “Artifact Plans.” After reading the paper, I’m confused on why these different levels are needed, and how they related to the questions being tested. This makes benefit of the benchmark unclear. The designed questions in this benchmark does not reflect useful business operations. For example, authors use pipeline to find connection between different entities and write questions that needs multi-hop reasoning (e.g., appendix D.6 case 3). In this particular example, the method need to identify an employee who recorded a decision about customer-account labels, then find what that employee wrote about a separate business issue. It is hard to understand in relatic why someone in the company need to follow this connection. Without justification on why these questions matters particularly for a company setting, the contribution of these questions is hard to be distinguished from prior work on multi-hop information retrieval. Earlier benchmarks such as HERB[1] already ontains business document and information retrieval questions. This benchmark claims the contribution is larger and more detailed company history, yet the paper does not clearly show how the history of the a company would result in the change on the questions being asked, important gap it falls, or what additional insight it provides. [1] Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath, Kung-Hsiang Huang, Caiming Xiong, and Chien-Sheng Wu. 2025. Benchmarking Deep Search over Heterogeneous Enterprise Data. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 501–517, Suzhou (China). Association for Computational Linguistics.

Questions for authors

It would be very helpful to have some explanations and insights on the three weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.