- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose DataPraxis, a data analysis task synthesis framework based on workspace evolution; they also construct the DataPraxis-5K dataset that includes long-horizon professional analytical training examples; they use the generated dataset to train DataPraxis-27B and DataPraxis-35B-A3B, and DataPraxis-27B achieves comparable performance among the evaluated models on DABStep, JobBench, and RideBench.
Critical strengths
The dataset generated by the work proposed in the paper effectively improves the capabilities of Qwen3.6-27B when used for training, enabling it to outperform larger open-source models such as DeepSeek-V4-Pro on three evaluation sets.
Critical weaknesses
The paper only compares against a single method baseline, DataMind, and does not systematically compare with existing related data generation methods; it also only provides evaluations on three datasets, so it is unclear whether the datasets were specifically selected because the proposed method performs better on them.
Questions for authors
The paper discusses the datasets on which the proposed work performs relatively well. It is recommended that it also fairly analyze the dataset scenarios in which the proposed work performs poorly or is unsuitable, and explain why this is the case. This would provide readers with more comprehensive information about the paper’s method and fair suggestions for its application and optimization.
5 comments
Q3: Could you clarify a boundary of the dataset’s applicability and provide practical guidance?
R3: We thank the reviewer for this valuable suggestion. As mentioned in Section 3.2.3 of our submission draft, DataPraxis constructs analytical tasks with domain-appropriate deliverables. We appreciate the reviewer’s helpful suggestion on applicability and clarify the current scope as follows.
Precise spreadsheet editing is outside the current dataset’s focus. Although spreadsheets can serve as analytical inputs or deliverables, the dataset is not specifically designed for changing designated cells while preserving all unrelated formulas, formatting, and workbook structure. For these applications, we recommend adding dedicated spreadsheet-editing trajectories with checks for both the requested changes and unintended modifications.
We thank the reviewer again. We will follow the reviewer’s suggestion to add this applicability discussion and practical guidance to improve our paper.
Kind Reminder
Dear Reviewer pf2h,
Thank you again for your constructive feedback and for recognizing the analytical improvements achieved with DataPraxis-5K and its competitive performance against larger models. As the discussion period is nearing its end, we would like to kindly check whether our responses have addressed your concerns.
We have broadened the synthesis-method comparison by adding DIVE and extended evaluation beyond the original three benchmarks to DAComp, alongside comparisons with BigBang-v1 and Occamy-1.0. We have also clarified the dataset’s applicability boundary: it focuses on analytical tasks, while precise spreadsheet editing would benefit from dedicated editing trajectories and preservation checks. The additional evaluations and applicability discussion are included in the revised manuscript and highlighted in blue.
If you have any remaining questions, we would be happy to address them before the discussion ends. If our responses have resolved your concerns, we would sincerely appreciate it if you could consider updating your score.
Best regards, The Authors
We sincerely thank the reviewer for recognizing the analytical improvements from DataPraxis-5K and its competitive results against larger open-weight models. We address the suggestions on baseline comparisons, evaluation coverage, and practical applicability below.
Q1: Could you broaden the comparison with existing task-synthesis methods?
R1: We thank the reviewer for this helpful suggestion. As mentioned in Section 4.1 of our submission draft, DataMind is our original synthesis baseline. Section 5 describes it as a task-type-guided method focused on data analysis, making it the closest baseline among the related methods. Several other methods target GUI or coding agents, whose task settings differ from ours. Our DataMind comparison matches the training-example count, teacher model, agent harness, and SFT recipe.
We appreciate the suggestion to broaden the comparison. Given the substantial financial cost of data synthesis, we add DIVE$^{[1]}$ as a representative exploration-based baseline. We adapt its implementation to generate 2,000 trajectories from our analytical workspaces.
DIVE explores the available tools through iterative interactions and accumulates execution evidence. It derives tasks from this evidence and uses the resulting queries to guide further exploration. DataPraxis inherits and merges verified executable workspaces, retaining scripts, intermediate data, and findings for continued investigation across complementary branches.
Both methods use the same Qwen3.6-27B backbone, seed workspaces, teacher, and harness. We also match the number of training examples and the SFT recipe:
DataPraxis improves over DIVE by 4.33 percentage points on DABStep and 5.29 points on JobBench. With the backbone, training recipe, and example count held fixed, these gains support the value of verified workspace evolution for analytical training.
We thank the reviewer again. We follow the reviewer’s suggestion to add this comparison and clarify the supporting evidence to improve our paper.
Q2: Could you clarify the benchmark-selection criteria and extend evaluation to another analytical setting?
R2: We thank the reviewer for raising this question. As mentioned in Section 4.1 under “Benchmarks” of our submission draft, the benchmarks cover complementary analytical tasks. DABStep tests multi-step reasoning, JobBench tests professional deliverable construction, and RideBench tests operational business analysis. We appreciate the reviewer’s suggestion to broaden coverage and include DAComp$^{[2]}$ as an additional public benchmark for this response. It covers enterprise data engineering and analytical workflows.
We also include recent agents BigBang-v1$^{[3]}$ and Occamy-1.0$^{[4]}$. All models below share the Qwen3.6-35B-A3B backbone:
DataPraxis improves over the base model by 22.62 points and exceeds the strongest evaluated alternative by 11.48 points, extending the evidence beyond the original three benchmarks. Frontier models and the two additional models we compare against use large-scale training data and complex post-training pipelines, while DataPraxis achieves these results with only 5K SFT trajectories.
We thank the reviewer again. We follow the reviewer’s suggestion to add DAComp and clarify the benchmark-selection criteria and evaluation scope to improve our paper.
References
[1] A. Chen et al. DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use. 2026.
[2] F. Lei et al. DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle. 2025.
[3] The BigBang Team. BigBang: Pursuing Open-Ended Intelligence through Self-Evolving Synthesis of Verifiable Frontier Tasks. 2026.
[4] W. Chen et al. Occamy-1.0: Open Pareto-frontier 35B Intelligence for Co-work. 2026.