- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This is a data synthesis paper. The authors try to model relationships among enterprise artifacts, use a graph to capture these relationships, and simulate data based on that graph in order to build a collaborative dataset.
Critical strengths
The paper conducts comprehensive experiments covering different models and different aspects of the algorithm, and it also evaluates some of the latest models such as Opus and GPT-5.6-sol. I acknowledge the authors' effort on this paper.
Critical weaknesses
(1) Lack of real-world validation. The proposed method tries to simulate data that reflects how a real company or corporation works. However, the relationships you model are not carried out only in digital form; real organizational work also involves human coordination that is not captured by files alone. I think you need to obtain data from real companies and evaluate your results by comparing this kind of ground truth with your generated output. In the current evaluation section, I did not see any real-data evaluation of this kind. At the minimum, I would expect a case study conducted with some real business within a company.
(2) Unclear grounding of the "enterprise" claim. You claim that current benchmarks are limited because enterprise data are largely private and open-source data cannot really reflect the quality of enterprise work or its dependency structure. Then where does this kind of "private" information in your benchmark and data synthesis method come from, and how did you obtain it? If you do not have access to such information, I do not think you should claim, in the introduction, that your advantage is having more information about those enterprises; or at least you should present the contribution differently.
Questions for authors
I have doubts about the incremental contribution of this paper. For example, in foundation model training, many people work on synthesizing domain data to improve model performance, especially to enable models to solve downstream tasks in a corporate setting. From my point of view, this kind of data is selected and cleaned, and it can genuinely help the model improve. Closely related to the above: I think the real uniqueness of the data produced by these benchmarks lies in the domain knowledge it contains, and this domain knowledge, including its domain-specific terminology, can only be provided by domain experts. If you are fully relying on language models to synthesize the data, then you are using an LLM with limited domain knowledge to generate data that is supposed to contain domain knowledge. I find this a little hard to accept. Could you explain it?
5 comments
Official Response to Reviewer Abxt (2/3)
(2) Unclear grounding of the "enterprise" claim. You claim that current benchmarks are limited because enterprise data are largely private and open-source data cannot really reflect the quality of enterprise work or its dependency structure. Then where does this kind of "private" information in your benchmark and data synthesis method come from, and how did you obtain it? If you do not have access to such information, I do not think you should claim, in the introduction, that your advantage is having more information about those enterprises; or at least you should present the contribution differently.
The reviewer is right in pointing out that we’re not using any private data and our use of enterprise-specific knowledge derives from explicit usage of public labor statistics data (O*Net) and AI usage statistics. We argue that this is a good estimation of what activities occur in these enterprises as they keep being updated periodically in the absence of the proprietary data. Existing synthetic works have not leveraged this statistics explicitly and our use of it should thus be seen as more explicitly grounded in the enterprises than the prior work.
How this differs from prior work. Prior synthetic environment generators condition on personas or compose subtasks (Ge et al., 2026; Jang et al., 2026; Liu et al., 2026; Bu et al., 2025), without grounding the generated work in labor-market or AI-usage statistics. Occupational data has been used to select occupations for expert-written tasks (Patwardhan et al., 2026) and to steer the generation of cross-application workflows partially (Li et al., 2026a). LineageBench instead (i) grounds the overall goals systematically such that the corresponding work activity carries labor-market weight for the occupation and is commonly done with AI, and (ii) grounds every upstream persona in the graph, not just the goal owner, in its matched O*NET occupation.
How we have revised the paper. We agree that the current introduction can be read as claiming access to enterprise information, and we have revised it as follows:
- Introduction, the "Such benchmarks remain scarce" sentence (§1). Current: "Such benchmarks remain scarce because enterprise data is largely proprietary, while public corpora does not capture the quality and dependency structure of real enterprise work." Revised: "Such benchmarks remain scarce because enterprise data is largely proprietary, while public resources provide a high-level description of the work performed by occupations."
- Contribution (1). Current: "scalable synthesis of realistic and diverse multi-persona enterprise states". Revised: "scalable synthesis of diverse multi-persona enterprise states grounded in public occupational statistics".
- Abstract. Current: "evaluates agents in realistic and diverse enterprise states". Revised: "evaluates agents in diverse enterprise states grounded in public occupational data".
Official Response to Reviewer (3/3)
I have doubts about the incremental contribution of this paper. For example, in foundation model training, many people work on synthesizing domain data to improve model performance, especially to enable models to solve downstream tasks in a corporate setting. From my point of view, this kind of data is selected and cleaned, and it can genuinely help the model improve. Closely related to the above: I think the real uniqueness of the data produced by these benchmarks lies in the domain knowledge it contains, and this domain knowledge, including its domain-specific terminology, can only be provided by domain experts. If you are fully relying on language models to synthesize the data, then you are using an LLM with limited domain knowledge to generate data that is supposed to contain domain knowledge. I find this a little hard to accept. Could you explain it?
We respectfully clarify our contributions: to the best of our knowledge, no prior work generates lineage graphs with an LLM or grounds such generation in O*NET. We would like to emphasize that our generation pipeline is explicitly and strongly grounded in O*Net in two ways (see General Response, G1). Domain knowledge enters through the O*NET occupations, tasks, and work activities that ground each goal and persona, and through these multi-stakeholder rubrics. The results show that the tasks are not solved by an LLM's general domain knowledge: under the two-judge comparison, only Grok-4.6 clearly beats the Reference artifact and Grok-4.5 roughly ties it (see Table A in Reviewer UMSe's response), and every model degrades as more intermediate steps must be reconstructed. The same pipeline can also produce training data with full provenance and graded feedback, which relates to the reviewer's point about synthesizing domain data for model training. While it is possible that training settings in frontier labs internally may have strong synthetic generation pipeline, there are little developments for knowledge work tasks in the open research community.
What the synthetic construction contributes. The main value of LineageBench is not that an LLM supplies domain knowledge, but that the lineage graph coherently turns generated content (as also highlighted by Reviewer UMSe) into hard tasks that can be checked, which hand-written or forward-simulated data does not provide on its own:
- Solvable by construction. Every target is derived from its recorded inputs, so each task provably contains the evidence it needs (see our answer to Reviewer UMSe on the input frontier).
- Controllable difficulty. The same target can be posed with inputs 1 to 5 hops upstream. Quality falls by 31% on average from 1 to 5 hops, and a controlled test shows that this is not only because there are more input files.
- Specific, checkable criteria. Because the graph records who produced each input and who consumes the output, the rubric lists the exact figures and definitions a deliverable must preserve from its inputs and what its consumers need. These are domain-specific requirements, derived from the organization's own artifacts rather than from general knowledge.
- Easy to update and extend. New organizations can be generated at any time, so evaluation sets need not go stale or leak into training data.
Official Response to Reviewer Abxt (1/3)
We thank the reviewer for the feedback and for acknowledging the breadth of our experiments. We respond to their concerns about real-world validation, the source of our enterprise grounding, and the contribution below.
We agree this is a limitation and an additional real case study would strengthen our work, but we disagree that a lack of it significantly weakens our contributions and findings. A study of the scale suggested by the reviewer, unfortunately, is not easy and involves a lot of money and effort (as shown in the General Response, G3, a large scale study amounts to over half a million dollars in annotation). We thus strongly believe that coordinating real business efforts to match the generated work deserves a full paper. Our contributions here are not real impact in industries or emulating their work but towards building scalable synthetic benchmarks that simulate and create solvable tasks from the kind of activities conducted within these organizations. To this end, we focus on the software-based activities listed by O*NET data since that aligns with the focus of agentic applications. We do not claim that it completely matches the full set of activities conducted in these organizations or that a good performance in a certain industry implies full autonomy of that work. We have highlighted this explicitly in our ethics statement and will thus soften the claims of “realistic” and replace it with “grounded”.
Why a real-company study is hard to coordinate. We do not think that a small-scale study of any sort can do justice to have any meaningful support of the claims that will be implied by such a study. Comparing against real enterprise lineage would require, for each participating company, a data-sharing agreement and a legal and privacy review before any internal documents could be examined; de-identifying confidential spreadsheets, reports, and decks; and domain experts from that company to reconstruct which files each deliverable was built from and who produced them. To be informative for a benchmark spanning 12 industry groups, this would have to be repeated across many companies in different industries. Even then, the data could not be released, so the comparison could not be reproduced or extended by others. A single case study would cover one organization in one industry and could not reliably support general claims about the benchmark, which does not justify the costs that will be involved in this.