- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DataIntern, a benchmark for evaluating enterprise data agents that can learn from experience across analytical workloads rather than solving queries independently. It identifies three complementary learning capabilities: procedural reuse, data space understanding, and adaptation to data space changes. Construction of Dataintern combines publicly available enterprise-relevant data, feedback from real interactions, agent-generated tasks, execution-based verification and expert annotation. Controllled experiments with the dataset compares language models, agent harness and different memory systems. Results demonstrate substantial reductions in tool calls and computational costs through procedural reuse, and accuracy improvements of 15–28 percentage points from relevant experience for twelve of thirteen models on DataIntern-Lite. However, agents frequently fail to apply stored knowledge in the right context, particularly when underlying conditions evolve. The paper concludes that effective data agents require mechanisms for selective knowledge retention, context-sensitive retrieval, and adaptation to evolving environments.
Critical strengths
-
One of the relevant use-case for AI model is automating data science workloads. And in that regard, DataIntern benchmark is a useful addition to evaluate agentic capabilities across heterogeneous multiple domains from real-world. Specific questions that the paper targets, that of change in data space and the agents ability to adapt to it, use of procedural knowledge and memory gained is relevant to real-world scenario.
-
Authors have carried out systematic and controlled experiments to isolate confounding factors providing insight into agents capabilities and limitations (Table 1 and 3) by evaluating 13 LLMs, 6 Agentic harness and 4 memory mechanisms.
-
The conclusion provides some insight into current generation agentic systems. For example, experiments confirms extraction and reuse of procedure by most agents, at the same time, reveals their inability to transfer the knowledge across data space. An important insight that agents apply knowledge where it does not hold (is that "procedural hallucination"?).
Critical weaknesses
- The benchmark measures experience utilization more convincingly than "real" learning capabilities of the data agents, i.e., the experiments do not clearly separate knowledge acquisition, memory storage, retrieval, and generalization. For example, the LG=C2-C1 is used to conclude that prior knowledge helps in the task but it does not reveal the details as where and how the agents used the prior knowledge. Currently, the "prior knowledge" is buried in the metric.
- Limited evaluation for long-horizon, open-ended tasks. This is very important for enterprise and real-world workloads, especially when the authors have used memory as one of the features to evaluate. Also, there is no measure of the complexity of the tasks, and hence, very difficult to get a sense of agentic capabilties for real-world workloads (mere classifying "hard", "medium", and "easy" gives no clear indication).
- The authors conclusion that current generation agents "reuse procedures well" may well be a reflection of the tasks and domain choosen for the dataset. The authors explicitly observe that agents learn new knowledge but do not know "where" it applies.
Questions for authors
Please see the weakness section. In addition,
- Can the authors reveal the "procedural rule" or "knowledge" reused by the Agents? Specifically, if the procedure is across (1) tasks in the same domain, (2) tasks across domains. Some specific examples would be good evidence of what is geting learnt by the agent and reused.
5 comments
Authors' Response (2/2)
Q1 (examples of reused knowledge). Thank you for the suggestion. We have added concrete examples from the data agent's memory to Appendix E.
| Example | Data space | What the memory holds | Effect |
|---|---|---|---|
| (1) Reused procedure | power sector | "group by plant identity and take the maximum capacity" | tool calls fall from 20 to 4 to 3 on the next two queries |
| (2) Reused rule | aviation | "for diverted flights, DivArrDelay is the final arrival delay" | 61 late arrivals, correct; without the rule, 39 and wrong |
| (3) Procedural hallucination | banking | reporting threshold rises from USD 47M to USD 58M | the new threshold is applied to a 2020 query, giving 315 institutions instead of 411 |
In (1) and (2) the memory saves work or fixes the answer. In (3) the knowledge is right but is applied to a year where it does not hold. In DataIntern, reuse happens within one data space, because a procedure like (1) depends on its tables, and the benchmark models one data agent per enterprise. Transfer across data spaces is a promising direction for future work.
We hope these results clarify the points you raised, and we would appreciate it if you would take them into consideration.
Reply to Authors' rebuttal
I appreciate authors' detailed explanation, additional analysis and revision of the paper. All my concerns have been addressed, and I am happy to increase the score. Thank you.
Authors' Response (1/2)
We sincerely appreciate the reviewer's thoughtful review and helpful suggestions. Changes in the revised PDF are marked in blue.
W1 (acquisition, storage, retrieval, and generalization). We appreciate this observation and agree that separating these stages is important. Section 2 now defines learning as an improvement in accuracy or cost from earlier experience, without updating model weights. To show where knowledge is used and where it is lost, we trace each piece through four observable stages on DataIntern-Full (Section 4 and Appendix A.11):
Memories that record K more often gain more. Full logs are the exception, because K is buried among many commands and outputs.
Two further results show where the gain comes from and where it is lost. First, removing K from structured memory while keeping its length drops LG from 0.13 to 0.07 (Table 4): about half of the gain comes from the knowledge itself. Second, on DataIntern-Lite, when data experts write the notes from the same trajectories, accuracy rises from 0.65 to 0.78 (Appendix A.11), showing that a major bottleneck is writing down the right knowledge. Q1 gives concrete examples.
W2 (long-horizon tasks and complexity). Thank you for raising this point. For long horizons, we insert up to 30 unrelated queries between related ones and report six sequences of up to ten requests (Section 4). With Mem0 and full session logs, accuracy stays at 0.74–0.77 from k = 5 to k = 30, well above 0.52 with unrelated experience (Table 5). With structured memory, accuracy drops to 0.55–0.59, because the data agent can drop earlier knowledge when it rewrites its notes.
DataIntern evaluates whether data agents learn from experience over an enterprise workload. In real-world enterprises, this workload often includes open-ended analysis, recurring reports, and changes in the data. Each DataIntern query has a fixed answer, so every score comes from a program. The path to the answer is open: the data agent explores the tables, files, and documents of a data space to find it.
For complexity, we define easy, medium, and hard by how many models solve a query without experience. We now also report objective measures from each reference program (Appendix A.3, Table 11):
Unlike data agent benchmarks that evaluate multi-step SQL or coding ability, DataIntern focuses on learning capability, so a query is hard when a piece of domain knowledge is missing. Experience helps most on such queries: relevant experience raises accuracy from 0.17 to 0.53 in the hard group and from 0.85 to 0.94 in the easy group (Appendix A.3).
W3 (reuse and where knowledge applies). Thank you for this observation. We agree that reusing a procedure and knowing where it applies are different abilities. In DataIntern, data agents reuse procedures effectively, but they often apply stored knowledge without checking whether it holds for the new query.
This scope error causes 35 of 39 change-test failures on the reference model and is the most common cause across all 13 models (Table 6). We have added the term procedural hallucination to the conclusion.