acceptodds
Under review as a conference paper at ICLR 2027

From Solving Data Tasks to Learning a Data Space: Benchmarking Learning Capabilities of Data Agents

Abstract

Data agents are increasingly deployed in enterprise analytical platforms, where they serve sustained workloads of queries over the data space. These workloads often contain queries that share procedures, depend on domain knowledge specific to the data space, and are affected by changes in it. Without learning capabilities, data agents may repeat costly exploration for each query, miss knowledge they have already found, and produce wrong answers from outdated experience. Existing data agent benchmarks evaluate each query independently and do not measure the learning capabilities of data agents. In this work, we introduce DataIntern, a benchmark that evaluates the learning capabilities of data agents through: (1) 15 real-world enterprise-level data spaces and 742 queries organized into 84 workloads that cover three complementary learning scenarios: procedural reuse, data space understanding, and data space changes; (2) a feedback-aware agentic construction framework, in which real enterprise feedback guides agents to build data spaces from public data and to verify by execution that each query needs this domain knowledge; (3) a controlled evaluation protocol that evaluates the learning capabilities of data agents under different experience conditions and clearly reveals how much they gain from experience and at what cost; (4) extensive experiments across thirteen frontier LLMs, six agent harnesses, and four memory mechanisms that reveal key insights for building future data agents with stronger learning capabilities. Our results show that reusing procedures cuts tool calls by 81 % and cost by 60 % on DataIntern-Full. Compared with unrelated experience from the same data space, experience that contains this domain knowledge raises accuracy by 15–28 points for twelve of thirteen models on DataIntern-Lite. Data space changes remain the hardest scenario. Few data agents answer correctly before and after a change, and most failures come from applying knowledge where it does not hold. Code and data are available at https://anonymous.4open.science/r/DataIntern-E53C.

Primary area:
datasets and benchmarks
Keywords:
Data Agents, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2vCvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DataIntern, a benchmark for evaluating enterprise data agents that can learn from experience across analytical workloads rather than solving queries independently. It identifies three complementary learning capabilities: procedural reuse, data space understanding, and adaptation to data space changes. Construction of Dataintern combines publicly available enterprise-relevant data, feedback from real interactions, agent-generated tasks, execution-based verification and expert annotation. Controllled experiments with the dataset compares language models, agent harness and different memory systems. Results demonstrate substantial reductions in tool calls and computational costs through procedural reuse, and accuracy improvements of 15–28 percentage points from relevant experience for twelve of thirteen models on DataIntern-Lite. However, agents frequently fail to apply stored knowledge in the right context, particularly when underlying conditions evolve. The paper concludes that effective data agents require mechanisms for selective knowledge retention, context-sensitive retrieval, and adaptation to evolving environments.

Critical strengths
  1. One of the relevant use-case for AI model is automating data science workloads. And in that regard, DataIntern benchmark is a useful addition to evaluate agentic capabilities across heterogeneous multiple domains from real-world. Specific questions that the paper targets, that of change in data space and the agents ability to adapt to it, use of procedural knowledge and memory gained is relevant to real-world scenario.

  2. Authors have carried out systematic and controlled experiments to isolate confounding factors providing insight into agents capabilities and limitations (Table 1 and 3) by evaluating 13 LLMs, 6 Agentic harness and 4 memory mechanisms.

  3. The conclusion provides some insight into current generation agentic systems. For example, experiments confirms extraction and reuse of procedure by most agents, at the same time, reveals their inability to transfer the knowledge across data space. An important insight that agents apply knowledge where it does not hold (is that "procedural hallucination"?).

Critical weaknesses
  1. The benchmark measures experience utilization more convincingly than "real" learning capabilities of the data agents, i.e., the experiments do not clearly separate knowledge acquisition, memory storage, retrieval, and generalization. For example, the LG=C2-C1 is used to conclude that prior knowledge helps in the task but it does not reveal the details as where and how the agents used the prior knowledge. Currently, the "prior knowledge" is buried in the metric.
  2. Limited evaluation for long-horizon, open-ended tasks. This is very important for enterprise and real-world workloads, especially when the authors have used memory as one of the features to evaluate. Also, there is no measure of the complexity of the tasks, and hence, very difficult to get a sense of agentic capabilties for real-world workloads (mere classifying "hard", "medium", and "easy" gives no clear indication).
  3. The authors conclusion that current generation agents "reuse procedures well" may well be a reflection of the tasks and domain choosen for the dataset. The authors explicitly observe that agents learn new knowledge but do not know "where" it applies.
Questions for authors

Please see the weakness section. In addition,

  1. Can the authors reveal the "procedural rule" or "knowledge" reused by the Agents? Specifically, if the procedure is across (1) tasks in the same domain, (2) tasks across domains. Some specific examples would be good evidence of what is geting learnt by the agent and reused.
5 comments
AuthorsOpenReview ↗

Authors' Response (1/2)

We sincerely appreciate the reviewer's thoughtful review and helpful suggestions. Changes in the revised PDF are marked in blue.

W1 (acquisition, storage, retrieval, and generalization). We appreciate this observation and agree that separating these stages is important. Section 2 now defines learning as an improvement in accuracy or cost from earlier experience, without updating model weights. To show where knowledge is used and where it is lost, we trace each piece through four observable stages on DataIntern-Full (Section 4 and Appendix A.11):

  1. Acquisition from the source query. When the source query is solved, the target query reaches 0.89 with structured memory against 0.67 with unrelated experience. When the source query fails, there is no gain (0.33 vs. 0.39).
  2. Recording. The knowledge K that a test needs often never reaches the memory (table below).
  3. Retrieval. Once recorded, the knowledge reaches the data agent in nearly every run (Mem0 returned all stored items in its 104 runs).
  4. Generalization. The target query asks a different question from the source query but needs the same knowledge K. In another 5–20% of all runs, K is in memory but the target answer is still wrong.
Memory (reference model)Share where K is recordedLG
Short session logs21%0.05
Zep38%0.11
Mem055%0.13
Full session logs73%0.10
Structured memory78%0.13

Memories that record K more often gain more. Full logs are the exception, because K is buried among many commands and outputs.

Two further results show where the gain comes from and where it is lost. First, removing K from structured memory while keeping its length drops LG from 0.13 to 0.07 (Table 4): about half of the gain comes from the knowledge itself. Second, on DataIntern-Lite, when data experts write the notes from the same trajectories, accuracy rises from 0.65 to 0.78 (Appendix A.11), showing that a major bottleneck is writing down the right knowledge. Q1 gives concrete examples.

W2 (long-horizon tasks and complexity). Thank you for raising this point. For long horizons, we insert up to 30 unrelated queries between related ones and report six sequences of up to ten requests (Section 4). With Mem0 and full session logs, accuracy stays at 0.74–0.77 from k = 5 to k = 30, well above 0.52 with unrelated experience (Table 5). With structured memory, accuracy drops to 0.55–0.59, because the data agent can drop earlier knowledge when it rewrites its notes.

DataIntern evaluates whether data agents learn from experience over an enterprise workload. In real-world enterprises, this workload often includes open-ended analysis, recurring reports, and changes in the data. Each DataIntern query has a fixed answer, so every score comes from a program. The path to the answer is open: the data agent explores the tables, files, and documents of a data space to find it.

For complexity, we define easy, medium, and hard by how many models solve a query without experience. We now also report objective measures from each reference program (Appendix A.3, Table 11):

GroupTarget queriesTablesDocumentsJoinsLines of code
Hard1731268
Medium1231255
Easy1521152

Unlike data agent benchmarks that evaluate multi-step SQL or coding ability, DataIntern focuses on learning capability, so a query is hard when a piece of domain knowledge is missing. Experience helps most on such queries: relevant experience raises accuracy from 0.17 to 0.53 in the hard group and from 0.85 to 0.94 in the easy group (Appendix A.3).

W3 (reuse and where knowledge applies). Thank you for this observation. We agree that reusing a procedure and knowing where it applies are different abilities. In DataIntern, data agents reuse procedures effectively, but they often apply stored knowledge without checking whether it holds for the new query.

  • Procedural reuse. Reuse makes data agents much cheaper. On the 37 reuse chains of DataIntern-Full, across 15 data spaces in 6 domains, the third query needs 12–29% of the tool calls of the first query for every model (Table 25).
  • Where knowledge applies. We provide two examples from DataIntern:
ScenarioWhat the memory holdsWhere the data agent applies itResult
Reusethe procedure of the first queryevery later query of the chain, uncheckedan early mistake repeats (Section 4.3)
Changesthe new reporting threshold, USD 58Ma 2020 query, where USD 47M still holds315 institutions instead of 411 (Appendix A.1)

This scope error causes 35 of 39 change-test failures on the reference model and is the most common cause across all 13 models (Table 6). We have added the term procedural hallucination to the conclusion.

AuthorsOpenReview ↗

Authors' Response (2/2)

Q1 (examples of reused knowledge). Thank you for the suggestion. We have added concrete examples from the data agent's memory to Appendix E.

ExampleData spaceWhat the memory holdsEffect
(1) Reused procedurepower sector"group by plant identity and take the maximum capacity"tool calls fall from 20 to 4 to 3 on the next two queries
(2) Reused ruleaviation"for diverted flights, DivArrDelay is the final arrival delay"61 late arrivals, correct; without the rule, 39 and wrong
(3) Procedural hallucinationbankingreporting threshold rises from USD 47M to USD 58Mthe new threshold is applied to a 2020 query, giving 315 institutions instead of 411

In (1) and (2) the memory saves work or fixes the answer. In (3) the knowledge is right but is applied to a year where it does not hold. In DataIntern, reuse happens within one data space, because a procedure like (1) depends on its tables, and the benchmark models one data agent per enterprise. Transfer across data spaces is a promising direction for future work.

We hope these results clarify the points you raised, and we would appreciate it if you would take them into consideration.

Reviewer_2vCvOpenReview ↗

Reply to Authors' rebuttal

I appreciate authors' detailed explanation, additional analysis and revision of the paper. All my concerns have been addressed, and I am happy to increase the score. Thank you.

Reviewer_K4JbOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces DataIntern, a benchmark for evaluating whether data agents can learn from prior experience across related analytical tasks. The authors construct enterprise-level data spaces with verified knowledge dependencies and evaluate agents under controlled experience conditions. The results show that relevant experience improves accuracy and efficiency, while revealing persistent challenges in applying learned knowledge correctly.

Critical strengths
  1. Well-designed benchmark construction with execution-verified knowledge dependencies, ensuring that the tasks genuinely require the intended domain knowledge.
  2. A convincing evaluation protocol with matched controls, complemented by intuitive case studies and systematic error analysis that reveal meaningful limitations of current agents.
Critical weaknesses

My main concern is the limited evaluation of learning over sustained workloads. The introduction motivates agents operating over the same data space for weeks or months, but most reuse experiments involve only three-query chains, with just six longer sequences. This leaves open how well the findings generalize to more realistic long-term workloads with interleaved tasks and evolving knowledge.

Questions for authors
  1. How well do the observed learning gains persist when related tasks are separated by many unrelated queries?

  2. Do the observed scope errors vary across different types of data space changes?

3 comments
AuthorsOpenReview ↗

Authors' Response

We sincerely appreciate the reviewer's positive assessment and insightful questions. Changes in the revised PDF are marked in blue.

W1 (sustained workloads). We appreciate this important concern. We report three results beyond three-query chains: interleaved workloads, long sequences, and all reuse chains.

  1. Interleaved workloads. We insert up to 30 unrelated queries between related ones on 28 DataIntern-Lite tests. Gains persist with Mem0 and full session logs (Q1). Seven of these tests are change tests, so evolving knowledge is also covered.
  2. Long sequences. On the six sequences of up to ten requests, notes that store reusable programs cut tool calls per request from 16 to 7 and cost from 0.48 to 0.29 at the same accuracy. On held-out requests the data agent has not seen, removing the programs raises the cost from 0.21 to 0.42, matching the no-memory baseline. The saving comes from reusing procedures.
  3. All reuse chains. On the 37 chains of DataIntern-Full, the third query needs 12–29% of the tool calls of the first query for every model (Table 25).
SettingAccuracy (seen / held-out requests)Tool calls per requestCost per request
No memory0.97 / 1.00160.48
Notes0.97 / 1.0090.38
Notes with programs0.97 / 1.0070.29

These results show that learning lasts beyond three-query chains when the memory keeps the knowledge. The long sequences are reported in Appendix A.7 of the original submission, and we have added the other two results to Section 4 and Appendices A.8 and A.12.

Q1 (persistence of gains). Thank you for this question. We insert k = 0, 5, 15, 30 unrelated queries between source and target on 28 DataIntern-Lite tests (21 understanding, 7 change). No inserted query involves the tested knowledge. Accuracy of the reference model:

Memoryk = 0k = 5k = 15k = 30
Structured memory0.740.550.590.59
Mem00.740.770.740.76
Full session logs0.690.770.770.77

With Mem0 and full session logs, accuracy stays at 0.74–0.77 from k = 5 to k = 30, well above 0.52 with unrelated experience. With structured memory, accuracy drops to 0.55–0.59. After every query, the data agent rewrites its notes and keeps them under 80 lines, so earlier knowledge can be dropped. Whether learning lasts depends on how the memory is managed: Mem0 and full session logs keep the gain, while notes with a fixed length need to decide what to keep. We have added this experiment to Section 4 (Table 5) and Appendix A.8.

Q2 (scope errors by change type). We appreciate this question. We grouped the 19 change tests into four types: threshold, business rule, reference list, and new data release. To check for in-family bias, we use Claude Opus 5.5 to label the failures of all 13 models.

Type of changeTestsScope errors, reference modelScope errors, 13 models
New threshold616 of 1775%
Business rule36 of 671%
Reference list79 of 1169%
New data release34 of 554%

Scope error is the most common cause of failure in every type (35 of 39 failures on the reference model). For example, after a reporting threshold rises from 47M to 58M, the data agent records the new threshold but applies it to a 2020 query and reports 315 institutions instead of 411 (Appendix A.1). Data agents record the new knowledge but do not check which version applies to the query. We have added this analysis to Section 4 (Table 6).

We hope these results answer your questions and are glad to discuss any further points.

Reviewer_K4JbOpenReview ↗

Response to Authors' Rebuttal

Thank you for the thorough and thoughtful rebuttal, as well as the additional experiments incorporated into the revised manuscript.

The new interleaved workload experiments directly address my concern regarding sustained workloads. The contrast between full logs/Mem0 and fixed-length structured memory provides a very compelling insight into memory retention and forgetting over long horizons. Additionally, the breakdown of scope errors across different change types clearly demonstrates the generality of the boundary-tracking bottleneck.

All of my concerns have been well addressed. I am happy to maintain my positive rating and support the acceptance of this paper.

AuthorsOpenReview ↗

Thank you for your positive assessment

We are deeply grateful for your time and for your continued support of DataIntern. Your positive assessment truly encourages us to keep improving our work and exploring this exciting direction!

Reviewer_eNAHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a benchmark for cross-session learning of data agents, consisting of 15 real-world data spaces, 742 queries, and 84 workflows that cover three learning scenarios: procedural reuse, data space understanding, and data space changes. A controlled evaluation protocol (C0–C3 for understanding and B0–B3 for changes) provides a matched unrelated-experience control and an oracle upper bound. Thirteen models are evaluated under six harnesses and four memory mechanisms, with execution-based deterministic grading as the primary signal, supplemented by expert review.

Critical strengths
  1. A controlled protocol that isolates "learning" itself. The matched unrelated-experience condition (C1) and the oracle condition (C3) make it possible to attribute gains to relevant experience rather than to additional practice, and the LG and Gap metrics avoid the misleading comparisons of raw scores that pervade agent benchmarks.
  2. A systematic quality-control and leakage-prevention pipeline. The paper reports independent blind solving of all 742 queries, independent adjudication of every disagreement, automated release gates for answer leakage and pre-embedding (checking that the target knowledge is not already present in the source data), and inter-reviewer agreement (Gwet's AC1 = 0.73). The 29 reference programs caught and fixed through blind adjudication are evidence that this pipeline works.
  3. Rigorous, transparent large-scale evaluation. The experiments span 13 models × 6 harnesses × 4 memory mechanisms, use paired bootstrap (10,000 replicates) with Holm correction, report three seeds, and fully disclose cost accounting, list prices, and version pinning for models, harnesses, and seeds.
Critical weaknesses
  1. Small sample for the change scenario. The data-space-change scenario contains only 19 tests on Full (110 queries) and 8 tests on Lite (48 queries). The headline conclusion that data space changes are the hardest scenario is therefore estimated from a small number of units, and the associated confidence intervals are wide; per-model estimates on this scenario are even noisier.
  2. Selection bias in the main comparison. The 13-model comparison is run on DataIntern-Lite, whose tests are selected by a learning-head gate — the oracle must improve accuracy over no experience (together with naturalness ≥ 4 and at most two tests per data space). Conditioning the comparison on tests with demonstrated headroom for learning can inflate the apparent learning gains and compress the apparent gap between models, while Full results are reported for only three models (the reference-model analyses). The benchmark's central model-level claims therefore do not directly characterize performance on a realistic, unfiltered workload.
  3. Conflicts of roles and remaining documentation gaps. GPT-5.6-Sol serves as the naturalness reviewer during construction while also being a tested model (and the reference configuration), and GPT-5.6-Luna performs grading-item extraction for every model under test; free-text/naturalness items judged by the same family that is being evaluated are at risk of in-family bias. In addition, the items are LLM-synthesized, the licensing of the 68 GiB of data is not fully resolved (snapshots are released only "where licensing permits"), and no human-expert baseline is provided to situate the agents' accuracy.
Questions for authors
  1. Can the data-space-change tests be expanded (and per-model confidence intervals reported), and can the authors describe how the scope of the "hardest scenario" conclusion was limited given the current 19/8 tests?
  2. Can the authors provide Full-tier results for all 13 models, and quantify the Lite selection bias — e.g., by comparing LG/Gap and model rankings on Lite versus Full, or by reporting inverse-probability/weighted estimates that account for the learning-head selection rule?
  3. Can the free-text key items (naturalness and rubric-based judgments) be re-checked by judges from a different model family than GPT-5.6, with agreement reported? Can the authors additionally (a) add a human-expert baseline on a representative subset and (b) include a per-source data-license inventory clarifying redistribution and release terms for the 68 GiB of data?
2 comments
AuthorsOpenReview ↗

Authors' Response (1/2)

We thank the reviewer for the careful review and positive comments. Changes in the revised PDF are marked in blue.

W1 (sample size of change tests). We appreciate this observation and agree that per-model estimates on the change tests carry wide confidence intervals. Each of the change tests has about six queries around a documented change in public data, such as the September 2024 Federal Reserve rate cut. To check sensitivity to individual tests, we report two results.

  1. Removing single tests. On the DataIntern-Full change tests, no model answers correctly both before and after the change in more than 0.63 of the tests. Removing any single test changes this share by at most 0.04, and every model stays at or below 0.67.
  2. Stating the new knowledge. Even when the oracle states the new knowledge (B3), accuracy on the query before the change is only 0.74, and scope errors remain the main cause of failure (Table 4, Section 4.7).

We have revised the abstract and introduction to state that data space changes remain challenging, as data agents often miss the query before or after a change. We have added these results to Appendix A.12.

W2 (DataIntern-Lite selection bias). Thank you for this observation. DataIntern-Lite keeps, by design, the tests where knowledge matters. DataIntern-Full shows whether the gains hold on the full benchmark. Relevant experience helps 12 of 13 models on DataIntern-Full, with 95% CI above zero for 9.

ModelLG DataIntern-LiteLG DataIntern-Full [95% CI]
GPT-6-Astra0.280.17 [0.09, 0.27]
GPT-6-Luna0.260.16 [0.05, 0.28]
Kimi-K30.260.18 [0.05, 0.30]
DeepSeek-V4.1-Flash0.260.15 [0.05, 0.25]
GLM-5.3-Flash0.240.09 [−0.02, 0.21]
GPT-5.6-Sol0.220.13 [0.01, 0.25]
GPT-6-Sol0.220.05 [−0.06, 0.16]
HY4-Preview0.200.12 [0.01, 0.22]
Grok-4.60.200.12 [0.02, 0.22]
DeepSeek-V4-Pro0.190.13 [0.04, 0.21]
GLM-5.30.170.12 [0.02, 0.23]
Gemini-3.1-Pro0.150.09 [−0.01, 0.19]
Qwen-3.7-Max−0.040.00 [−0.11, 0.11]

The mean LG falls from 0.20 on DataIntern-Lite to 0.12 on DataIntern-Full, because many DataIntern-Full tests are solved without the knowledge. On DataIntern-Full, experience mainly lowers cost: the cost per solved query falls from 1.14 to 0.49 on the reference model (Table 4). Models rank similarly on both tiers (Spearman 0.91 for accuracy with relevant experience, 0.72 for LG). We have added this comparison to Appendix A.12 (Table 25).

W3 (LLM judge). Thank you for raising this concern. All accuracy scores are computed by programs that compare each answer field with the reference answer. The 13-model comparison uses structured memory, which each model writes itself. The table lists where LLMs enter the evaluation.

UseLLMEffect on scoresCheck with Claude Opus 5.5
Gradingnone (programs)computes every scoreno LLM involved
Mem0 and Zep memoriesGPT-5.6-Lunaused only to compare memories on the reference modelnot used in the 13-model comparison
Naturalness reviewGPT-5.6-Soldecides which queries are released83% agreement (AC1 = 0.80)
Failure labelslabeling modelnone; used only for error analysis77% agreement

To check for in-family bias, we use Claude Opus 5.5 to rerun the naturalness review. Removing the queries that Claude rates below 4 changes the LG of each model by at most 0.03. The queries are written with LLMs, but every query is executed against the data, solved blindly, and reviewed by experts. Two researchers solve 0.80 of a sample without an LLM (Q3). We have added this check to Section 3 and Appendix I.9.

AuthorsOpenReview ↗

Authors' Response (2/2)

Q1 (Change tests). Thank you for the suggestion. The main table compares the 13 models on the DataIntern-Lite change tests. We now evaluate all 13 models on all 19 DataIntern-Full change tests (110 queries) and report per-model 95% CIs (Table 25). Joint is the share of tests answered correctly both before and after the change.

ModelJoint [95% CI]ModelJoint [95% CI]
Grok-4.60.63 [0.42, 0.84]Kimi-K30.37 [0.16, 0.58]
DeepSeek-V4.1-Flash0.58 [0.37, 0.79]GLM-5.30.37 [0.16, 0.58]
GPT-6-Astra0.53 [0.32, 0.74]GLM-5.3-Flash0.32 [0.11, 0.53]
HY4-Preview0.53 [0.32, 0.74]Gemini-3.1-Pro0.32 [0.11, 0.53]
GPT-5.6-Sol0.47 [0.26, 0.68]GPT-6-Luna0.26 [0.11, 0.47]
DeepSeek-V4-Pro0.47 [0.26, 0.68]Qwen-3.7-Max0.26 [0.11, 0.47]
GPT-6-Sol0.42 [0.21, 0.63]

Joint ranges from 0.26 to 0.63 (mean 0.43). Removing any single test changes Joint by at most 0.04 for every model (W1).

Q2 (DataIntern-Full results and weighting). We appreciate this question. Tests where the oracle shows no headroom never enter DataIntern-Lite, so reweighting cannot recover them. Instead, we evaluate all 13 models directly on DataIntern-Full (W2, Tables 24 and 25). On the reference model:

TierNo experience (C0)Unrelated (C1)Relevant (C2)LG = C2 − C1Gap
DataIntern-Lite0.430.430.650.220.55
DataIntern-Full0.600.580.700.130.46

LG and Gap are computed before rounding. As the selection rule intends, DataIntern-Lite tests start lower and leave more room to learn. The gain stays positive on DataIntern-Full.

Q3 (judge bias, human baseline, and licenses). Thank you for these suggestions.

  • Judges from another model family. We use Claude Opus 5.5 to rerun the naturalness review and to label the same failures. Agreement with the original models is 83% on release decisions (AC1 = 0.80) and 77% on change-test failure labels (Table 6). Rubric judgments do not enter any score.
  • Human baseline. Two independent researchers each solve 12 DataIntern-Lite tests (28 queries in 8 data spaces) in workload order. They use SQL and Python without an LLM assistant, under the same time limit and grader. After the source queries, they solve half of the understanding queries. Once the knowledge is stated, they solve all of them, so these queries are well posed and the missing knowledge makes them hard.
SolverReuseUnderstandingChangesAll
Two researchers (mean)0.920.500.790.80
Reference model (three seeds)1.000.330.440.67

We have added the human results to Appendix A.11.

  • Notes written by humans. Two data experts read the source trajectories of 27 DataIntern-Lite understanding tests without the target queries and wrote notes under the same 80-line limit. Expert notes close most of the gap between the data agent's own notes and the oracle, so writing down the right knowledge is a major bottleneck (Appendix A.11).
Notes given to the reference modelAccuracy
Its own structured memory0.65
Notes written by data experts0.78
Oracle (knowledge stated)0.83
  • Licenses. We have added a table of the sources of each data space and their licenses (Appendix F, Table 30). The repository lists the sources with their URLs in data_spaces/*/sources.yaml.

We hope these results address your concerns and are happy to answer further questions.

Comments on the paper

2 comments

Dear reviewers,

We are less than a day away from the end of the discussion period. Please read the authors' response and any revisions, and post a brief follow-up indicating which concerns remain and whether your assessment has changed. Please raise any outstanding questions promptly so the authors have time to respond.

Thank you for your contribution!

Best,

Your AC

AuthorsOpenReview ↗

Summary of the Discussion

Dear AC and reviewers,

We sincerely thank all reviewers for their thoughtful and constructive feedback, and the AC for guiding the discussion. We appreciate the recognition of DataIntern's controlled evaluation protocol, benchmark construction, and large-scale evaluation from Reviewers eNAH, K4Jb, and 2vCv.

Based on the valuable feedback, we have addressed the concerns in our manuscript and added the following results (changes are marked in blue):

  • DataIntern-Full results (eNAH): we evaluate all 13 models on DataIntern-Full with 95% CIs and compare the two tiers (Appendix A.12).
  • Sustained workloads (K4Jb): we test the gains with up to 30 unrelated queries in between and break down scope errors by type of change (Section 4).
  • Learning stages (2vCv): we trace where knowledge is lost on its way to the target query and add complexity measures with concrete examples (Appendices A.3, A.11, and E).
  • Evaluation checks (eNAH): we recheck the LLM judgments with Claude Opus 5.5 and add a human baseline and a license table (Appendices A.11, F, and I.9).

Reviewers K4Jb and 2vCv have indicated that their concerns are addressed, and Reviewer 2vCv is willing to increase the score. We remain happy to discuss any further points.

We believe these updates further strengthen DataIntern as a rigorous testbed for evaluating the learning capabilities of data agents. We hope the revised submission meets your expectations.

Thank you for your constructive feedback and support!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.