acceptodds
Under review as a conference paper at ICLR 2027

Simulating Organizational Data Lineage for Realistic and Controllable Evaluation of Enterprise Agents

Abstract

Autonomous agents are increasingly being used to produce complex enterprise artifacts such as spreadsheets, documents, and presentations. However, generating these scenarios remains difficult since realistic benchmarks require substantial manual effort, while existing synthetic approaches offer limited control over the relevance, diversity, and complexity of generated tasks. Evaluating generated artifacts also remains a challenge since no single reference captures the requirements that are practically imposed by different organizational stakeholders. We introduce LineageBench, a synthetic framework that systematically evaluates agents in realistic and diverse enterprise states through multi-persona data lineage. Rather than simulating organizational activity forward toward a potentially misaligned emergent state, LineageBench begins with a high-level deliverable and recursively works backward to identify the upstream artifacts required to produce it, along with the organizational personas responsible for them. We instantiate LineageBench with 30 organizational graphs spanning multiple industries, grounded in the O*NET taxonomy and usage statistics. We also leverage the lineage structure to construct rubrics for evaluating each artifact by collating the expectations of its producer, data providers, and consumers. By varying the lineage distance between a target artifact and the ancestors supplied to an agent, we evaluate 12 frontier agents on more than 500 tasks with controlled compositional complexity. Results show that the strongest agents achieve nearly twice the score of the weakest, yet similarly capable agents can require up to twice as many tool calls. Performance also declines by 31% on average as lineage distance increases from one to five hops. Together, these findings expose novel gaps in the capability and efficiency of enterprise agents.

Primary area:
datasets and benchmarks
Keywords:
Synthetic Data Generation, Agents, Benchmark, Simulation, Large language models, Data lineage

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AbxtOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This is a data synthesis paper. The authors try to model relationships among enterprise artifacts, use a graph to capture these relationships, and simulate data based on that graph in order to build a collaborative dataset.

Critical strengths

The paper conducts comprehensive experiments covering different models and different aspects of the algorithm, and it also evaluates some of the latest models such as Opus and GPT-5.6-sol. I acknowledge the authors' effort on this paper.

Critical weaknesses

(1) Lack of real-world validation. The proposed method tries to simulate data that reflects how a real company or corporation works. However, the relationships you model are not carried out only in digital form; real organizational work also involves human coordination that is not captured by files alone. I think you need to obtain data from real companies and evaluate your results by comparing this kind of ground truth with your generated output. In the current evaluation section, I did not see any real-data evaluation of this kind. At the minimum, I would expect a case study conducted with some real business within a company.

(2) Unclear grounding of the "enterprise" claim. You claim that current benchmarks are limited because enterprise data are largely private and open-source data cannot really reflect the quality of enterprise work or its dependency structure. Then where does this kind of "private" information in your benchmark and data synthesis method come from, and how did you obtain it? If you do not have access to such information, I do not think you should claim, in the introduction, that your advantage is having more information about those enterprises; or at least you should present the contribution differently.

Questions for authors

I have doubts about the incremental contribution of this paper. For example, in foundation model training, many people work on synthesizing domain data to improve model performance, especially to enable models to solve downstream tasks in a corporate setting. From my point of view, this kind of data is selected and cleaned, and it can genuinely help the model improve. Closely related to the above: I think the real uniqueness of the data produced by these benchmarks lies in the domain knowledge it contains, and this domain knowledge, including its domain-specific terminology, can only be provided by domain experts. If you are fully relying on language models to synthesize the data, then you are using an LLM with limited domain knowledge to generate data that is supposed to contain domain knowledge. I find this a little hard to accept. Could you explain it?

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer Abxt (1/3)

We thank the reviewer for the feedback and for acknowledging the breadth of our experiments. We respond to their concerns about real-world validation, the source of our enterprise grounding, and the contribution below.

(1) Lack of real-world validation. The proposed method tries to simulate data that reflects how a real company or corporation works. However, the relationships you model are not carried out only in digital form; real organizational work also involves human coordination that is not captured by files alone. I think you need to obtain data from real companies and evaluate your results by comparing this kind of ground truth with your generated output. In the current evaluation section, I did not see any real-data evaluation of this kind. At the minimum, I would expect a case study conducted with some real business within a company.

We agree this is a limitation and an additional real case study would strengthen our work, but we disagree that a lack of it significantly weakens our contributions and findings. A study of the scale suggested by the reviewer, unfortunately, is not easy and involves a lot of money and effort (as shown in the General Response, G3, a large scale study amounts to over half a million dollars in annotation). We thus strongly believe that coordinating real business efforts to match the generated work deserves a full paper. Our contributions here are not real impact in industries or emulating their work but towards building scalable synthetic benchmarks that simulate and create solvable tasks from the kind of activities conducted within these organizations. To this end, we focus on the software-based activities listed by O*NET data since that aligns with the focus of agentic applications. We do not claim that it completely matches the full set of activities conducted in these organizations or that a good performance in a certain industry implies full autonomy of that work. We have highlighted this explicitly in our ethics statement and will thus soften the claims of “realistic” and replace it with “grounded”.

Why a real-company study is hard to coordinate. We do not think that a small-scale study of any sort can do justice to have any meaningful support of the claims that will be implied by such a study. Comparing against real enterprise lineage would require, for each participating company, a data-sharing agreement and a legal and privacy review before any internal documents could be examined; de-identifying confidential spreadsheets, reports, and decks; and domain experts from that company to reconstruct which files each deliverable was built from and who produced them. To be informative for a benchmark spanning 12 industry groups, this would have to be repeated across many companies in different industries. Even then, the data could not be released, so the comparison could not be reproduced or extended by others. A single case study would cover one organization in one industry and could not reliably support general claims about the benchmark, which does not justify the costs that will be involved in this.

AuthorsOpenReview ↗

Official Response to Reviewer Abxt (2/3)

(2) Unclear grounding of the "enterprise" claim. You claim that current benchmarks are limited because enterprise data are largely private and open-source data cannot really reflect the quality of enterprise work or its dependency structure. Then where does this kind of "private" information in your benchmark and data synthesis method come from, and how did you obtain it? If you do not have access to such information, I do not think you should claim, in the introduction, that your advantage is having more information about those enterprises; or at least you should present the contribution differently.

The reviewer is right in pointing out that we’re not using any private data and our use of enterprise-specific knowledge derives from explicit usage of public labor statistics data (O*Net) and AI usage statistics. We argue that this is a good estimation of what activities occur in these enterprises as they keep being updated periodically in the absence of the proprietary data. Existing synthetic works have not leveraged this statistics explicitly and our use of it should thus be seen as more explicitly grounded in the enterprises than the prior work.

How this differs from prior work. Prior synthetic environment generators condition on personas or compose subtasks (Ge et al., 2026; Jang et al., 2026; Liu et al., 2026; Bu et al., 2025), without grounding the generated work in labor-market or AI-usage statistics. Occupational data has been used to select occupations for expert-written tasks (Patwardhan et al., 2026) and to steer the generation of cross-application workflows partially (Li et al., 2026a). LineageBench instead (i) grounds the overall goals systematically such that the corresponding work activity carries labor-market weight for the occupation and is commonly done with AI, and (ii) grounds every upstream persona in the graph, not just the goal owner, in its matched O*NET occupation.

How we have revised the paper. We agree that the current introduction can be read as claiming access to enterprise information, and we have revised it as follows:

  • Introduction, the "Such benchmarks remain scarce" sentence (§1). Current: "Such benchmarks remain scarce because enterprise data is largely proprietary, while public corpora does not capture the quality and dependency structure of real enterprise work." Revised: "Such benchmarks remain scarce because enterprise data is largely proprietary, while public resources provide a high-level description of the work performed by occupations."
  • Contribution (1). Current: "scalable synthesis of realistic and diverse multi-persona enterprise states". Revised: "scalable synthesis of diverse multi-persona enterprise states grounded in public occupational statistics".
  • Abstract. Current: "evaluates agents in realistic and diverse enterprise states". Revised: "evaluates agents in diverse enterprise states grounded in public occupational data".
AuthorsOpenReview ↗

Official Response to Reviewer (3/3)

I have doubts about the incremental contribution of this paper. For example, in foundation model training, many people work on synthesizing domain data to improve model performance, especially to enable models to solve downstream tasks in a corporate setting. From my point of view, this kind of data is selected and cleaned, and it can genuinely help the model improve. Closely related to the above: I think the real uniqueness of the data produced by these benchmarks lies in the domain knowledge it contains, and this domain knowledge, including its domain-specific terminology, can only be provided by domain experts. If you are fully relying on language models to synthesize the data, then you are using an LLM with limited domain knowledge to generate data that is supposed to contain domain knowledge. I find this a little hard to accept. Could you explain it?

We respectfully clarify our contributions: to the best of our knowledge, no prior work generates lineage graphs with an LLM or grounds such generation in O*NET. We would like to emphasize that our generation pipeline is explicitly and strongly grounded in O*Net in two ways (see General Response, G1). Domain knowledge enters through the O*NET occupations, tasks, and work activities that ground each goal and persona, and through these multi-stakeholder rubrics. The results show that the tasks are not solved by an LLM's general domain knowledge: under the two-judge comparison, only Grok-4.6 clearly beats the Reference artifact and Grok-4.5 roughly ties it (see Table A in Reviewer UMSe's response), and every model degrades as more intermediate steps must be reconstructed. The same pipeline can also produce training data with full provenance and graded feedback, which relates to the reviewer's point about synthesizing domain data for model training. While it is possible that training settings in frontier labs internally may have strong synthetic generation pipeline, there are little developments for knowledge work tasks in the open research community.

What the synthetic construction contributes. The main value of LineageBench is not that an LLM supplies domain knowledge, but that the lineage graph coherently turns generated content (as also highlighted by Reviewer UMSe) into hard tasks that can be checked, which hand-written or forward-simulated data does not provide on its own:

  • Solvable by construction. Every target is derived from its recorded inputs, so each task provably contains the evidence it needs (see our answer to Reviewer UMSe on the input frontier).
  • Controllable difficulty. The same target can be posed with inputs 1 to 5 hops upstream. Quality falls by 31% on average from 1 to 5 hops, and a controlled test shows that this is not only because there are more input files.
  • Specific, checkable criteria. Because the graph records who produced each input and who consumes the output, the rubric lists the exact figures and definitions a deliverable must preserve from its inputs and what its consumers need. These are domain-specific requirements, derived from the organization's own artifacts rather than from general knowledge.
  • Easy to update and extend. New organizations can be generated at any time, so evaluation sets need not go stale or leak into training data.
Reviewer_UMSeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

LineageBench generates organizational artifact-dependency DAGs by planning backward from target deliverables and generating upstream-to-downstream files in topological order. The graph supports controlled variation in input lineage distance and owner/producer/consumer evaluation rubrics. The benchmark includes 30 organizations and 236 targets, evaluates 12 models producing editable Office artifacts, and studies lineage distance on 38 shared targets. The paper argues that this construction captures information integration and stakeholder needs in enterprise work.

Critical strengths
  1. Reusing an explicit dependency graph for generation, task construction, and evaluation is a coherent design (Sections 2–3). Backward dependency planning followed by forward content generation offers a useful way to construct related files without simply leaking completed downstream answers into upstream evidence. The appendix documents safeguards and prompts, making the construction more inspectable.

  2. The owner/producer/consumer rubric explicitly evaluates source grounding and downstream usability, in addition to output presentation. Rubrics are generated before candidate answers, and the evaluation considers both artifact images and extracted text (pp. 5 and 20). The editable Office outputs and case studies make the intended enterprise capability more concrete than text-only answer scoring.

Critical weaknesses
  1. Human validation does not establish the accuracy of the reported artifact scores. Appendix D.5 asks five annotators to distinguish relevant criteria from cross-task distractors, producing 315 comparisons and AUC 0.862; this supports criterion relevance, not scoring correctness, completeness, or professional acceptability (pp. 22–24). This distinction matters because judge choice substantially changes the leading model's rank: Grok-4.6 moves from eighth to first across judges (pp. 20–22). Averaging three judges does not independently resolve that disagreement, so small leaderboard differences and grounding conclusions need stronger validation.

  2. The lineage-distance construction needs clarification before the controlled-depth interpretation is secure. The input-frontier formula on p. 5 uses U(t)=empty for a non-source target; for a DAG with both a short source branch and a longer branch, this appears to omit needed short-branch evidence as depth increases. This may be a notation error for U(u)=empty, not an implementation bug, but the actual rule and solvability audit are necessary. Even with the corrected frontier, depth can change file count, fan-in, and input volume. The 38-target experiment does not isolate these factors from composition depth, while the appendix extension uses changing target subsets.

Questions for authors
  1. Can experts blindly score actual produced artifacts and adjudicate examples where the LLM judges disagree, especially the Grok-4.6 rank reversal? Please distinguish criterion relevance from verdict accuracy and professional acceptability.

  2. Should the p. 5 fallback condition be U(u)=empty? Please specify the implemented frontier rule for nonuniform DAGs and report whether all necessary source evidence remains available at every tested depth.

  3. Does the depth effect persist on fixed targets after controlling for input volume/fan-in and auditing evidence availability? This would separate composition difficulty from larger inputs or missing information.

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer UMSe (1/3)

We thank the reviewer for their time and effort in reviewing our work and providing useful feedback. We are glad that they recognize the coherence of reusing one lineage graph for generation, task construction, and evaluation, and for the precise questions on judge reliability and lineage depth. We address their raised concerns below.

  1. Human validation does not establish the accuracy of the reported artifact scores. Appendix D.5 asks five annotators to distinguish relevant criteria from cross-task distractors, producing 315 comparisons and AUC 0.862; this supports criterion relevance, not scoring correctness, completeness, or professional acceptability (pp. 22–24). This distinction matters because judge choice substantially changes the leading model's rank: Grok-4.6 moves from eighth to first across judges (pp. 20–22). Averaging three judges does not independently resolve that disagreement, so small leaderboard differences and grounding conclusions need stronger validation.

  2. Can experts blindly score actual produced artifacts and adjudicate examples where the LLM judges disagree, especially the Grok-4.6 rank reversal? Please distinguish criterion relevance from verdict accuracy and professional acceptability.

We further investigated the judge's disagreement the reviewer points to and have found that it comes from the absolute judgement of the rubric satisfaction, which is a harder metric to calibrate for knowledge work tasks (GDPVal, Patwardhan et al., 2025). This gets resolved by doing a comparative pairwise preference of the artifacts for a rubric instead. In particular, we added a head-to-head pairwise comparison: for every item of the rubric ii, a judge decides whether the candidate or the pipeline's Reference artifact satisfies it better and gives a score si∈s_i \in{0,12,10,\tfrac{1}{2}, 1}. Then, we consider a weighted aggregate over the relative score as S=∑wisiS = \sum w_i s_i instead of a weighted aggregate over the absolute score of rubric satisfaction. Every pair is judged in both A/B orders to cancel position bias. We ran this with two judges from different model families, Claude Opus 4.8 and GPT-6, on all 236 common targets and 12 models.

The table below measures how similarly the two judges rank the 12 models under each scoring method. It uses the same set of deliverables throughout, with 95% bootstrap intervals over targets. Measuring pairwise continuous weighted share of the rubric items on which candidate beats the Reference raises Kendall τ by +0.24 (95% CI [+0.06, +0.30], p = 0.002) and pairwise win rate raises it by +0.23 (CI [0.00, +0.29], p = 0.03).

Scoring methodKendall τ, Opus 4.8 vs GPT-6 model rankingPer-deliverable Pearson r
Absolute rubric score (paper)0.58 [0.52, 0.67]0.71
Pairwise, mean S vs Reference0.82 [0.70, 0.85]0.79
Pairwise, win rate (1N∑tI{St>0.5}\tfrac{1}{N}\sum_t \mathbb{I}\{S_t > 0.5\}) vs Reference0.81 [0.63, 0.87]–

The paper's rank change disappears. The absolute rubric scores rank Grok-4.6 7th of the 12 evaluated models (excluding reference) under Opus 4.8 but 1st under GPT-6. In the pairwise comparison, both judges rank Grok-4.6 first and Grok-4.5 second. The absolute scores mostly reflect differences in each judge's calibration (GPT-6 grades 0.15–0.20 lower) and in how harshly it treats a presentation style. Opus 4.8 gives 5.3% of Grok-4.6's sparse workbooks a rubric score below 0.2, versus 1.3% under GPT-6. Comparing against the same Reference artifact removes both effects. Thus, we propose using the win rate and Elo against the Reference as the primary metrics of the paper, where

  • A target counts as won only if both Opus 4.8 and GPT-6 judge the candidate better than the Reference; otherwise, it is lost.
  • Elo is the closed-form Bradley–Terry rating, with the Reference anchored at 1000 and 95% bootstrap intervals over targets.

This ensures one judge's lenience toward its own family can no longer turn a loss into a win. The rule does inherit the stricter judge's verdict, so we also report both single-judge tables.

AuthorsOpenReview ↗

Official Response to Reviewer UMSe (2/3)

contd. from above

*Table A. Primary leaderboard: a win requires both judges (we defer the per-judge leaderboard to the paper appendix)

AgentWin (%)Elo [95% CI]Mean S (%)Continuous margin-aware Elo [95% CI]Judges agree (%)
Grok-4.677.41213 [1166–1272]67.91130 [1118–1142]82
Grok-4.551.31009 [968–1053]57.21050 [1038–1062]69
Reference50.0100050.01000–
GPT-5.6-luna41.5941 [897–985]50.81006 [995–1018]72
GPT-5.6-sol35.6897 [854–944]50.11000 [990–1012]72
GPT-5.6-terra30.9861 [818–907]47.8985 [975–995]80
Claude Opus 4.822.5786 [731–833]46.7977 [966–989]64
Claude Sonnet 4.620.8769 [715–818]43.1952 [937–966]70
Claude Opus 4.618.6746 [687–798]43.3954 [942–966]72
Nemotron-3-Ultra8.1582 [498–656]33.5881 [867–897]84
GPT-5.44.2466 [315–550]28.4840 [826–853]95
GPT-5.54.2466 [315–550]28.2838 [825–851]95
Claude Haiku 4.52.1350 [122–466]26.3822 [807–837]94
  • Grok-4.6 is first by a wide margin under both single judges as well.
  • The bottom four (Nemotron-3-Ultra, GPT-5.4, GPT-5.5, Claude Haiku 4.5) are the same under both judges.
  • In between, the order follows the stricter judge. Because a win needs both verdicts, the GPT-5.6 models sit above the Claude models here, as in Table C. Under Opus 4.8 alone (Table B), the Claude models sit higher.
  • The two judges reach the same verdict on 64–95% of targets per model.

Expert verdict accuracy. Expert grading of verdicts remains out of reach for us due to extremely high expected costs in hiring these annotators for such long-running tasks (see General Response G3). The existing human study only uses authors as annotators and only validates that rubric criteria are relevant to the assigned tasks. Since authors are not experts in all these industries, we believe using their judgement as ground-truth will not be appropriate and will not reflect the true judgement of the artifact quality. It is thus a fair limitation and we hope that the above pairwise judgement scores help address their concerns in making the evaluation more stable.

The lineage-distance construction needs clarification before the controlled-depth interpretation is secure...

Should the p. 5 fallback condition be U(u)=empty? Please specify the implemented frontier rule for nonuniform DAGs and report whether all necessary source evidence remains available at every tested depth.

Yes, we apologize for the typo and thank the reviewer for pointing it out. Indeed, it should be U(u)=ϕ\mathcal{U}(u) = \phi and we have now updated it. Thus, all necessary source evidence remains available at every depth in case of nonuniform DAGs.

Even with the corrected frontier, depth can change file count, fan-in, and input volume. The 38-target experiment does not isolate these factors from composition depth, while the appendix extension uses changing target subsets.

Does the depth effect persist on fixed targets after controlling for input volume/fan-in and auditing evidence availability? This would separate composition difficulty from larger inputs or missing information.

We thank the reviewer for this question that lets us understand the phenomenon better. Here, we conduct a controlled ablation test by providing the information at different depths while matching the number of input files and size. We find that a significant depth effect remains and grows with depth. We kept each target fixed and gave the agent its k = 1 inputs plus randomly chosen distractor files from the same organization, until the file count matched the real k = 2 or k = 3 frontier (11.9 at k = 2, 25.7 at k = 3 on average).

  • Distractors are materialized artifacts of the same organization that are neither ancestors of the target (those would be real evidence) nor descendants (those could leak the answer). They also roughly match the real frontier's size in MB (6.7 vs 6.3 MB at k = 2, and 13.9 vs 16.7 MB at k = 3).
  • We ran 9 of the paper's generators with the paper's agent setup on 31 targets that receive all the evidence they need by construction.
  • All conditions were scored with the updated new primary pairwise comparison metric instead of an absolute metric for stability (see above): win only if both judges agree, both A/B orders.
  • The table pools all 9 generators. "Volume effect" is the drop from k = 1 to k = 1 + distractors; "depth effect" is the further drop from the distractor condition to the real frontier, which has the same file count.
AuthorsOpenReview ↗

Official Response to Reviewer UMSe (3/3)

cont. from above

Pooled results over 9 generators:

ConditionFilesInput MBWin vs Reference (both judges)Mean rubric win weighted scoreRubric score (Opus 4.8)
k = 1 (immediate parents)4.01.835%0.490.80
k = 1 + distractors, matched to k = 211.96.717%0.360.65
real k = 2 frontier11.96.310%0.300.60
k = 1 (immediate parents)3.51.143%0.520.84
k = 1 + distractors, matched to k = 325.713.924%0.370.68
real k = 3 frontier25.716.73%0.220.47

Per generator, drop in mean S (95% CI):

Generatork = 2: volume effectk = 2: depth effectk = 3: volume effectk = 3: depth effect
Grok-4.60.04 [−0.07, 0.17]0.17 [0.01, 0.29]0.05 [0.00, 0.12]0.45 [0.33, 0.58]
Grok-4.50.04 [−0.04, 0.11]0.19 [0.11, 0.28]0.04 [0.00, 0.09]0.37 [0.27, 0.46]
Claude Opus 4.80.10 [0.02, 0.17]0.10 [0.04, 0.17]0.11 [0.05, 0.16]0.18 [0.11, 0.26]
Claude Opus 4.60.07 [0.02, 0.12]0.14 [0.08, 0.20]0.07 [0.02, 0.13]0.19 [0.11, 0.29]
Claude Haiku 4.50.02 [−0.04, 0.07]0.08 [0.05, 0.11]0.10 [0.01, 0.20]0.06 [0.02, 0.09]
Claude Sonnet 4.60.18 [0.09, 0.28]−0.02 [−0.17, 0.12]0.15 [0.04, 0.27]0.15 [0.03, 0.25]
GPT-5.6-sol0.28 [0.20, 0.36]−0.08 [−0.18, 0.02]0.22 [0.14, 0.32]−0.01 [−0.12, 0.10]
GPT-5.6-luna0.27 [0.21, 0.34]−0.08 [−0.17, 0.01]0.32 [0.21, 0.43]−0.03 [−0.12, 0.06]
GPT-5.6-terra0.22 [0.14, 0.29]−0.03 [−0.09, 0.05]0.27 [0.19, 0.34]0.00 [−0.06, 0.06]
Pooled (9 generators)0.14 [0.11, 0.16]0.05 [0.02, 0.08]0.15 [0.12, 0.18]0.15 [0.11, 0.19]
  • Both factors matter, and the depth effect grows with depth. Pooled over 9 generators, input volume accounts for about 70% of the decline at k = 2 and about 50% at k = 3. The remaining depth effect, from real k-hop inputs at the same file count, is significant at both levels: mean S drops 0.05 [0.02, 0.08] at k = 2 and 0.15 [0.11, 0.19] at k = 3. Win rates show the same pattern: at k = 3 they fall from 43% to 24% with distractors, and to 3% with the real frontier.
  • Generators differ in which factor hurts them. The GPT-5.6 family is very sensitive to irrelevant files, but loses not much from real depth itself. Grok, Opus and Haiku largely ignore distractors, but drop sharply when they must reconstruct intermediate results from deeper sources. Grok-4.6, for example, wins 75% of k = 3-sized tasks with distractors but only 8% with the real k = 3 frontier. Sonnet 4.6 is affected by both.

We have therefore now refined, rather than dropped, the depth claim. Lineage distance makes tasks harder in two ways: (i) the agent must find the relevant material in a larger, noisier set of upstream files, and (ii) beyond that, it must compose intermediate results it no longer receives. The second effect is significant, grows from k = 2 to k = 3, and dominates for the strongest generators. We have added this decomposition for all generators to the paper.

Reviewer_tk1gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LineageBench, a framework for constructing realistic and controllable benchmarks for enterprise agents. It leverages O*NET occupational information to sample plausible enterprise tasks and constructs multi-persona data lineage graphs through backward dependency expansion and forward artifact generation. The resulting graphs enable controllable task complexity through lineage distance and support multi-stakeholder rubrics for evaluating artifact quality. Experiments across 30 synthetic organizations and 12 frontier agents demonstrate substantial differences in artifact generation quality and tool-use efficiency. Agent performance also declines significantly as lineage distance increases, highlighting limitations in multi-step information synthesis.

Critical strengths

Unified benchmark construction framework. The data-lineage-based design integrates realistic task generation, controllable complexity, and multi-persona evaluation into a coherent framework, providing a scalable approach to enterprise agent benchmarking.

Comprehensive experiments and useful findings. Experiments across multiple frontier agents, artifact types, and lineage depths reveal meaningful differences in artifact generation quality, multi-step information synthesis, and tool-use efficiency.

Critical weaknesses

Insufficient validation of organizational data lineage realism. The artifact dependencies and organizational roles are primarily inferred by LLMs rather than derived from real enterprise workflows. Although O*NET helps ground task selection, the paper lacks sufficient validation of whether the generated lineage graphs accurately reflect real organizational processes, raising concerns about the realism of the benchmark.

Questions for authors

1、How do the authors validate that the LLM-generated artifact dependencies and organizational roles accurately reflect real enterprise workflows?

2、How does the framework detect or mitigate unrealistic dependencies introduced during recursive backward expansion, particularly when such errors may propagate through the generated lineage graph?

6 comments
AuthorsOpenReview ↗

Official Response to Reviewer tk1g (1/3)

We thank the reviewer for their support of our paper. They have highlighted our presentation, unified scalabale benchmark construction framework, and comprehensive experiments. We address their remaining concerns below.

Insufficient validation of organizational data lineage realism. The artifact dependencies and organizational roles are primarily inferred by LLMs rather than derived from real enterprise workflows. Although O*NET helps ground task selection, the paper lacks sufficient validation of whether the generated lineage graphs accurately reflect real organizational processes, raising concerns about the realism of the benchmark.

We agree that a complete data lineage realism will strengthen our claim of organizational data lineage realism but we note that it is extremely hard to validate since real cross-team workflow data is private and scarcely available. Collecting or even validating it through experts is also very infeasible due to the prohibitive costs involved as OpenAI's GDPval, for example, had to recruit experienced industry professionals (about 14 years of experience on average) to write and review each task at over 40-200$/hour compensation, with each expert grading of a model deliverable taking about 109 minutes. That cost does not scale to the synthetic scale of 30 organizations spanning all 12 industry groups for research conducted at a small research group since it estimates to about $100k–1.5M in expert wages based on the setup (General Response, G3). To avoid overclaiming realism concerns, we have also revised the introduction to describe LineageBench as grounded in public occupational data rather than as reproducing any specific real enterprise:

We would also like to clarify that in addition to grounding the goal selection in ONet activities, we also leverage it to ground the personas by matching the LLM-sampled persona to the closest ONet occupation and including its listed activities in the persona text. This was exemplified in the prompts of Appendix F but we had missed this mechanism earlier and have now added it explicitly in Section 3.1 as follows:

During backward expansion, the generator identifies each required input and freely names the role that would own it, such as “FP&A Director” or “Regulatory Affairs Manager.” These upstream roles are not selected from a predefined organizational roster. Instead, each newly requested role is mapped later to the relevant ONet occupation. First, we normalize the requested title by lowercasing it, replacing punctuation with spaces, and collapsing whitespace. If this does not result in an exact match against an ONET’s job-title, we construct a deduplicated candidate shortlist using approximate string matching against official occupation titles, singularized titles, and alternate/reported job titles. We retrieve up to eight matches from each of the first two sources and ten from the third, using similarity cutoffs of 0.3, 0.3, and 0.6, respectively. An LLM then receives the requested role title and each candidate’s SOC code, title, and the first 90 characters of its occupational description. It selects the occupation by business function rather than superficial title overlap: for example, “FP&A Director” should map to financial planning rather than to “Art Directors.” The model may also propose a standard occupation title outside the shortlist, which is subsequently resolved against the catalog. If the model fails to return a usable mapping, the implementation falls back to the first shortlisted candidate, or to approximate title matching when the shortlist is empty.

Persona description. The matched occupation provides grounding without replacing the requested role. Each stakeholder contains a synthetic name, the original requested role, a singularized display title, a team label corresponding to the matched occupation’s SOC major group, and a list of responsibilities. The following prompt is then used for further backward expansion and materialization during generation time for grounding:

Name: <synthetic name>
Role: <requested or emergent role>
Title: <singularized requested role>
Team: <matched occupation's SOC major-group label>
Responsibilities:
  <official O*NET occupational description>
  <up to 3 sampled core tasks; supplemental tasks if no core tasks>
  <up to 3 highest-importance generalized work activities>
AuthorsOpenReview ↗

Official Response to Reviewer tk1g (2/3)

  1. How do the authors validate that the LLM-generated artifact dependencies and organizational roles accurately reflect real enterprise workflows?

Validating and mitigating the quality of LLM-generated artifact dependencies is tricky due to the lack of ground-truth that works across industries. Industrial workflows are often quite messy, dynamic, and form the major proprietary assets of companies’ organization and effectiveness. Thus, we let the LLM generate the dependencies to enhance flexibility and only ground the personas and the kind of activities they do using public sources of O*Net and labor statistics data. While this does not explicitly ground the generated dependencies, it allows us to ground both the personas and the root activity of the graph which indirectly grounds the requested dependencies to match the expected persona occupations through indirect prompting. To validate this further, we run various diagnostic tests on the generated graphs.

Two judges evaluated 600 generated edges, mixed blindly with 300 rewired and 300 owner-swapped controls. They distinguish unmodified dependencies from rewired controls with AUC 0.88–0.90, and from owner-swapped controls with AUC 0.82–0.85. We evaluate 600 generated dependencies, blindly interleaved with 300 rewired controls that replace the input artifact and 300 owner-swapped controls that replace its owner. Dependencies are rated on a five-point scale. We use Claude Opus 4.8 as the primary judge and GPT-6 as a second judge; paired results below follow that order. The judges rate 91% / 96% of unmodified dependencies as likely or standard, and rate them higher than the control set with an AUC of 0.88 / 0.90 against rewiring and 0.82 / 0.85 against owner swaps. Thus, judgments respond to both artifact content and the participating roles, rather than merely assigning uniformly favorable scores. These findings show that the generated dependencies are more plausible than their perturbed versions.

Beyond this, we compared all 30 organizations (4,052 artifacts) against O*NET fields that were withheld from the generator and against real public lineage graphs. Upstream roles are named by the expansion LLM first and only afterwards mapped to an O*NET occupation, so the mapping cannot have shaped the artifacts. Text matching here is lexical (TF-IDF), so no model judges its own outputs. "AUC" is the probability that an artifact matches its owner's occupation better than a comparison occupation (0.5 = chance).

QuestionPublic evidence the generator did not seeResult
Is each artifact the kind of work its owner's occupation does?Held-out O*NET task statements (tasks shown in the persona are excluded)Owner vs. random occupations: AUC 0.71; vs. occupations in the same industry group: 0.61
Does each occupation's overall mix of work match its O*NET profile?O*NET work-activity importance ratings (41 general activities)Positive match for 79% of 126 occupations; the occupation's own profile ranks at the 69th percentile of 894 (chance: 50th)
Are aggregating roles the ones whose decisions matter?O*NET Work Context "Impact of Decisions on Co-workers or Company Results"Spearman ρ = 0.21 (p = 0.02)
Are raw data sources owned by people who use those systems?Full O*NET software lists (the generator saw at most 4 tools)When a source names a system (CRM, ERP, HRIS, EHR, LMS, …), the owner's occupation uses it 77% of the time, vs. 65% for other roles in the organization (n = 359)
Does the graph look like real enterprise lineage?Real public dbt lineage graphs (Cal-ITP, NHS West and North London ICB), sampled the same way: 5 deliverables and their ancestorsEdges per node: 1.19 (real: 1.24–1.37). Share of nodes reused downstream: 0.13 (real: 0.15–0.16)

Taken together, the assigned work, which roles aggregate work, and the source systems agree with independent occupational statistics at rates well above chance.

AuthorsOpenReview ↗

Official Response to Reviewer tk1g (3/3)

2.How does the framework detect or mitigate unrealistic dependencies introduced during recursive backward expansion, particularly when such errors may propagate through the generated lineage graph?

As noted above, realism of dependencies is a hard problem due to the lack of a ground truth and thus, we do not explicitly detect or mitigate them during generation. Instead, we use the persona prompt and task grounding in real activities. Our additional results on validating the currently sampled dependencies above also shows that the current pipeline comes up with a good set of dependencies. While we agree that an explicit edge plausibility checks will help strengthen the pipeline, it is currently unclear on how one will model it to work across industries in the absence of any real world lineage data that is up-to-date. One can also explicitly use the ONet persona-activity based gating, but this comes with its own issues as it will limit the diversity of the work to what is presented in ONet, which is often outdated [1]. These updates are thus non-trivial and complementary to the contributions of our generation pipeline and evaluation findings of the current version.

[1] Wang, Jennifer, et al. "TaskNet: Building Occupational Task Inventories from Worker Interviews at Scale." Available at SSRN 7670419 (2026).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviews and author response

We thank all reviewers for their thoughtful feedback on our work and the area chair for coordinating the process. Reviewers found the lineage-graph design coherent, with one graph reused for generation, task construction, and evaluation (UMSe, tk1g); valued that the owner, producer, and consumer rubrics evaluate source grounding and downstream usefulness beyond presentation (UMSe); and found the experiments comprehensive (tk1g, Abxt). Below, we answer the concerns shared across reviewers and summarize our responses and revisions.

G1. Where the grounding comes from (tk1g, Abxt).

LineageBench uses no private enterprise data. It is grounded in two public sources: O*NET, the U.S. Department of Labor survey of what workers in more than 900 occupations do (tasks, work activities, work context, and software), and public AI-usage statistics. These ground generation in two places:

  1. Goals. Each root deliverable is generated for an O*NET occupation, mapped to one of its work activities, and kept only if that activity carries sufficient labor-market weight for the occupation and is commonly done with AI.
  2. Personas. Every upstream role that backward expansion names is matched to its closest O*NET occupation, whose official description, core tasks, and most important work activities form that persona's profile.

The LLM supplies organization-specific details such as file names, figures, and document structure, while who does what kind of work is anchored in O*NET. We have now described the second mechanism explicitly in §3.1 as it was not clear in the earlier version and describe LineageBench as grounded in public occupational statistics rather than as reproducing any particular real enterprise.

G2. New validation of the generated organizations (tk1g, Abxt).

Real cross-team lineage data is private, so we validate the generated graphs against public evidence the generator never saw and against perturbed controls:

  • Dependency plausibility. Two LLM judges from different model families rated 600 generated dependencies more plausible than 300 controls whose input artifact was rewired and 300 whose owner was swapped.
  • Occupational fit. The work activities in each artifact overlap 56–58% with its owner's O*NET activities, compared with 31% for a random occupation.
  • Organizational structure. When a raw source names a system such as a CRM or ERP, its owner's occupation lists that system 77% of the time, compared with 65% for other roles in the organization.
  • Graph shape. The generated graphs have similar density (1.19 edges per node, compared with 1.24–1.37) as real public lineage graphs of limited industries (dbt projects from Cal-ITP and NHS West and North London ICB) sampled the same way.

G3. Infeasibility of running a real expert study (tk1g, UMSe, Abxt).

We agree with the reviewers that expert evidence is the gold standard, but it is costly at scale and out of scope of our contributions centered on scalable synthetic generation pipeline. OpenAI's GDPval, for example, relied on industry professionals with about 14 years of experience on average to write and review each task.

We thus estimate the cost of such a study from GDPval's own measurements. An expert needs on average 109 minutes to grade one model deliverable or 404 minutes per completion, which GDPval costs around 86 dollars using median wages, or about 47 dollars per hour. Since experts with GDPval's level of experience typically earn closer to the 90th percentile of their occupation: it could around 117 dollars per hour on average across six senior occupations (U.S. BLS, May 2025). Applying GDPval's grading time to LineageBench:

Expert validationExpert-hoursCost at $47/h (median wage)Cost at $117/h (90th-percentile wage)
Grade each of the 2,812 agent deliverables once5,108$242k$597k
Review each of the 1,296 derived artifacts and its inputs once2,354$111k$275k
Two graders + artifact review12,571$0.60M$1.47M
Single human gold artifact to compare against1,589$85k$185k

These figures cover wages only, while recruiting and interviews would add to them. Studying a real company would further require data-sharing and privacy agreements, and the resulting data could not be released, so others could not reproduce or extend the benchmark.

To further strengthen the judge evaluation in the absence of experts (UMSe), we have now added a pairwise comparison against the Reference artifact on every rubric item as the primary metric instead of absolute scoring. Agreement between the two judges' rankings on this rises the Kendall scores to 0.82 with pairwise scores, and the leaderboard ranking stabilizes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.