acceptodds
Withdrawn from ICLR 2027

Learning the Data Environment: Continual Specialization for Data Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Data agents often answer a succession of questions about the same data sources. Although they have general analytical capabilities, they often repeat exploratory work across requests, recovering dataset-specific knowledge acquired in earlier analyses. We introduce Continual Data Specialist (CDS), a framework that enables a data agent to acquire reusable, environment-specific state before downstream requests are known. Using multi-table data bundles as a concrete instantiation, CDS generates analysis questions, collects tool-executed trajectories, filters them with lightweight verification, and directs later exploration toward observed failures. The resulting experience is retained through two complementary channels: a bundle-specific LoRA adapter learns patterns of analysis and tool use, while persistent memory statements make dataset-specific findings explicitly available in context. Subsequent requests reuse the same adapter and memory bank. For each request, a hypothesis module draws on both channels to propose analysis directions, which are checked against the original data through tool execution. Experiments on three data-agent benchmarks show that CDS improves insight quality over Vanilla Agent and either memory channel alone. CDS offers a route from general-purpose analysis to environment-specific expertise, turning autonomous exploration into knowledge that can support repeated use of the same data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Data Agents, Continual Learning, Continual Specialization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WxdqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Continual Data Specialist (CDS), a framework for turning autonomous data exploration into reusable environment-specific knowledge. As shown in Table 1 of the draft, CDS consistently outperforms Vanilla Agent across InsightBench, InsightEval, and UniDataBench.

Critical strengths

None

Critical weaknesses
  1. Lack of benchmarks. CDS is only verified on InsightBench, InsightEval, and UniDataBench. Meanwhile, there are more benchmarks on data analysis tasks, e.g., DABStep [1], KramaBench [2], DA-Code [3], etc.

  2. Lack of baselines. CDS is only evaluated against Vanilla Agent, and some in-context learning techniques build upon the Vanilla Agent. On the other hand, more baselines exist, e.g., DA-Agent [3], DS-STAR [4]. Moreover, the experiments mainly rely on Qwen3.6-27B. However, these days most of the data analysis agent uses more bigger models or proprietary models like GPTs. At least, I think authors should show that CDS is scalable or generalizable technique to any kinds of LLMs.

  3. Novelty. Acquiring reusable, environment-specific state before downstream requests for data analysis tasks is actually a well-known concept. For example, DS-STAR [4] also generates data description which is reusable before doing downstream tasks. However, in this paper, such concept is not well-mentioned through related works and also not compared.

Overall, even though the problem tackled by CDS is very important, I think it cannot meet an acceptance bar of ICLR due to lack of scientific rigor. This paper needs more experiments and more analysis to be a contribution to the ICLR community.

[1] Egg et al., DABStep: Data Agent Benchmark for Multi-step Reasoning, 2025
[2] Lai et al., KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes, 2025
[3] Huang et al., DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models, 2024
[4] Nam et al., DS-STAR: Data Science Agent for Solving Diverse Tasks across Heterogeneous Formats and Open-Ended Queries, 2025

Questions for authors
  1. Can you provide additional experiment using more bigger models than Qwen3.6-27B?

  2. Can you add additional baselines like DA-Agent [1] or DS-STAR [2]?

  3. What happens when there are too many files to analyze before the downstream tasks? E.g., in KramaBench [3], some tasks require analyzing ~1000 data files before data analysis. I highly recommend to try such benchmarks.

  4. Can CDS be added to proprietary closed models like GPT or Gemini? If not, what can be some issue, and what can be some way to do it? Since CDS have to make LoRA, I think it might be impossible to be compatible with these closed models.

  5. Can you provide some qualitative results? What is the output for the CDS, except for LoRA?

[1] Huang et al., DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models, 2024
[2] Nam et al., DS-STAR: Data Science Agent for Solving Diverse Tasks across Heterogeneous Formats and Open-Ended Queries, 2025
[3] Lai et al., KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes, 2025

Reviewer_QQavOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a tabular data agent that synthesizes questions to come up with a state/pipeline for the table bundle, so when the user's real questions come in, the questions can be accurately answered by the bundle.

Critical strengths

S1: The authors tested their method on 3 different benchmarks, which I appreciate, to make the evaluation more complete.

S2: The related work section is also pretty complete.

S3: Data agent is a meaningful topic that many care about.

Critical weaknesses

W1: In the space of data agents, CLI systems (e.g., Claude Code, Codex, OpenHands, OpenCode) are often used as baselines. Why were they not compared? I'm concerned because (even multi-table) tabular-data-only manipulation could now be relatively straightforward for those systems (but of course, depending on the specific setting of the benchmarks). Have the authors tried to run those systems with the latest models on the 3 benchmarks?

W2: 27B is not a very big model. I get that the authors might be constrained by computational resources, and they wanted to do LoRA, but now I'm not convinced that a small model + LoRA is better than a big model without LoRA. This part of the experiment should also be strengthened.

W3: Looks like the generated questions are the key to the success of this method. How representative are those generated questions of the actual questions at inference time? Are the choices of generated question groups backed by any data management literature? Why are these kinds particularly helpful? The eval doesn't answer this either.

W4: Overall, this paper is trying to combine a bunch of well-established techniques into one data agent, without evaluating why and how each component works thoroughly. I understand it's empirical work, but from the current eval, I cannot judge if this work is necessary or even useful. My main concern (although I have many concerns) is that, with all the razzle-dazzle, this data agent still couldn't outperform other CLI systems. Right now, this is unclear. I encourage the authors to take a look at Matei's new paper: https://arxiv.org/pdf/2609.03141

Questions for authors

Please address my comments/questions in W1-4.

Reviewer_c8ZnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

CDS pre-explores each data bundle before any user request arrives. From a bundle profile (source schemas, column types, basic statistics, representative values, permitted metadata), it constructs an initial question pool from template-based, statistics-driven, and language-model-generated questions, deduplicates the pool, and executes the questions through a Python tool. Trajectories that pass structural checks (a complete interaction, at least one successful execution, and at most two failed tool calls) are further screened by a GPT-4o judge for relevance and coverage. Rejected questions drive a failure-driven curriculum that generates additional questions targeted at missing analyses, up to the synthesis budget (target: 300 successful trajectories per bundle; up to 3,000 attempts). Accepted trajectories are converted into three supervision views—original trajectory, detailed analysis, and final findings—which train a per-bundle LoRA adapter, while compact findings are consolidated into a persistent textual memory bank. At inference, a hypothesis module uses the adapted model together with the memory bank to propose candidate analysis plans and claims, and a source-checking stage re-executes and verifies them against the original sources before answering. CDS is evaluated on InsightBench, InsightEval, and UniDataBench (100 tasks each) with Qwen3.6-27B as the common backbone, using GPT-4o G-Eval-style reference agreement and DeepSeek-V4-Pro-based Grounded Claims as metrics.

Critical strengths
  1. The LoRA adapter retains patterns of analysis and tool use (which tables to consult, how to structure comparisons, how to proceed after an observation), while the textual bank retains concrete, bundle-specific findings. The channels are genuinely complementary, the shared backbone remains frozen, and per-bundle state can be swapped in independently for each checkpoint.
  2. Table 2 isolates the contributions of execution checks alone, analysis-quality filtering (+0.0150), and failure-guided questions (+0.0121); Table 3 cumulatively adds the three supervision views; Table 4 separates hypothesis generation from source refinement and shows that hypotheses without refinement are largely unsupported (Grounded Claims 0.24). The ablations line up with the paper's narrative.
  3. Table 5 reports concrete hyperparameters, synthesis budgets, and temperatures; Appendix C walks through a full archived trajectory; and the Limitations section explicitly acknowledges per-bundle training overhead and the limited benchmark coverage.
Critical weaknesses
  1. With up to 3,000 LM-generated question attempts per bundle targeting the same tables as the benchmark's insight questions—and with multiple benchmark tasks sharing a single checkpoint over the same source files—the exploration phase has ample opportunity to cover the exact analytical directions later tested. Because validation is performed on the training data itself (Appendix A) and no n-gram, embedding, or semantic near-duplicate analysis is reported, the main result (Table 1) cannot distinguish transferable environment learning from question-coverage memorization.
  2. GPT-4o both filters retained trajectories and performs the final G-Eval scoring, so any systematic preference of GPT-4o (e.g., for analyses phrased like its own judgments) can inflate the very metric used to validate the method. Additionally, although TOFFEE, DataCOPE, and SEAL are discussed at length in §2 and are the closest systems in scope, none is included in the experiments; all comparisons are against the authors' own reconfigurations (Vanilla Agent, summary handoff, memory-only, LoRA-only), and no RAG or long-context baseline is reported.
  3. All evidence comes from one backbone (Qwen3.6-27B), 100 questions per benchmark, a single run, with no paired per-question tests and no multiple seeds, so the magnitude of several small deltas (e.g., LoRA over Memory-statement ICL on UniDataBench, 0.7152 vs. 0.7372) is hard to interpret. The Grounded Claims decline (0.91 → 0.86) is acknowledged only in passing, without analysis of which claim types lose support. Finally, per-bundle training cost is named as a limitation but never quantified: no GPU-hours per bundle and no break-even point (number of reuse requests required to amortize synthesis) are given, even though the practical claim of the paper depends on this quantity.
Questions for authors
  1. Please release the full list of generated questions (and retained trajectories) and conduct a decontamination analysis against every benchmark question—n-gram overlap, embedding similarity, and a manual or model-assisted semantic near-duplicate review. Please re-run Table 1 under strict isolation (synthesis completed before benchmark questions are inspected, or on held-out bundles/tables) so that the gains can be attributed to transfer rather than coverage.
  2. Please re-score all outputs with a judge distinct from GPT-4o (e.g., the DeepSeek model already used for Grounded Claims, or a second independent model), report inter-judge agreement and the correlation between each judge's scores and the Grounded Claims rates, and add paired per-question significance tests (e.g., paired bootstrap or permutation tests) over multiple seeds.
  3. Please add experimental comparisons to TOFFEE, DataCOPE, and SEAL, together with RAG and long-context controls under matched token/compute budgets, and report per-bundle GPU-hours for synthesis plus the number of reuse requests at which the upfront cost breaks even.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.