- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Continual Data Specialist (CDS), a framework for turning autonomous data exploration into reusable environment-specific knowledge. As shown in Table 1 of the draft, CDS consistently outperforms Vanilla Agent across InsightBench, InsightEval, and UniDataBench.
Critical strengths
None
Critical weaknesses
-
Lack of benchmarks. CDS is only verified on InsightBench, InsightEval, and UniDataBench. Meanwhile, there are more benchmarks on data analysis tasks, e.g., DABStep [1], KramaBench [2], DA-Code [3], etc.
-
Lack of baselines. CDS is only evaluated against Vanilla Agent, and some in-context learning techniques build upon the Vanilla Agent. On the other hand, more baselines exist, e.g., DA-Agent [3], DS-STAR [4]. Moreover, the experiments mainly rely on Qwen3.6-27B. However, these days most of the data analysis agent uses more bigger models or proprietary models like GPTs. At least, I think authors should show that CDS is scalable or generalizable technique to any kinds of LLMs.
-
Novelty. Acquiring reusable, environment-specific state before downstream requests for data analysis tasks is actually a well-known concept. For example, DS-STAR [4] also generates data description which is reusable before doing downstream tasks. However, in this paper, such concept is not well-mentioned through related works and also not compared.
Overall, even though the problem tackled by CDS is very important, I think it cannot meet an acceptance bar of ICLR due to lack of scientific rigor. This paper needs more experiments and more analysis to be a contribution to the ICLR community.
[1] Egg et al., DABStep: Data Agent Benchmark for Multi-step Reasoning, 2025
[2] Lai et al., KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes, 2025
[3] Huang et al., DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models, 2024
[4] Nam et al., DS-STAR: Data Science Agent for Solving Diverse Tasks across Heterogeneous Formats and Open-Ended Queries, 2025
Questions for authors
-
Can you provide additional experiment using more bigger models than Qwen3.6-27B?
-
Can you add additional baselines like DA-Agent [1] or DS-STAR [2]?
-
What happens when there are too many files to analyze before the downstream tasks? E.g., in KramaBench [3], some tasks require analyzing ~1000 data files before data analysis. I highly recommend to try such benchmarks.
-
Can CDS be added to proprietary closed models like GPT or Gemini? If not, what can be some issue, and what can be some way to do it? Since CDS have to make LoRA, I think it might be impossible to be compatible with these closed models.
-
Can you provide some qualitative results? What is the output for the CDS, except for LoRA?
[1] Huang et al., DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models, 2024
[2] Nam et al., DS-STAR: Data Science Agent for Solving Diverse Tasks across Heterogeneous Formats and Open-Ended Queries, 2025
[3] Lai et al., KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes, 2025