acceptodds
Under review as a conference paper at ICLR 2027

DataPraxis: Scaling Analytics Tasks via Verified Workspace Evolution

Abstract

General-purpose agents are increasingly expected to perform professional analytics work, which requires sustained reasoning, iterative investigation, and reliable evidence. Training these capabilities requires diverse and challenging tasks. However, generating tasks directly from raw data often overlooks deeper questions that emerge through analysis and provides limited evidence for verification. To address these limitations, we introduce DataPraxis, an analytics task synthesis framework based on verified workspace evolution. The framework progressively enriches workspaces with validated findings and artifacts, then inherits and combines these workspaces to uncover deeper questions. It constructs tasks from accumulated findings and their dependencies, verifying analytical results both during evolution and after task assembly. Using this framework, we construct DataPraxis-5K, a dataset of 5,000 verified trajectories spanning diverse domains and output formats, to train DataPraxis-27B and DataPraxis-35B-A3B. DataPraxis-27B achieves 55.11 on DABStep and 51.9 on JobBench, matching or surpassing frontier models such as GLM-5.2. We will release DataPraxis-5K to the community for further study.

Primary area:
datasets and benchmarks
Keywords:
Data Synthesis, LLM Agents, Agentic Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pf2hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose DataPraxis, a data analysis task synthesis framework based on workspace evolution; they also construct the DataPraxis-5K dataset that includes long-horizon professional analytical training examples; they use the generated dataset to train DataPraxis-27B and DataPraxis-35B-A3B, and DataPraxis-27B achieves comparable performance among the evaluated models on DABStep, JobBench, and RideBench.

Critical strengths

The dataset generated by the work proposed in the paper effectively improves the capabilities of Qwen3.6-27B when used for training, enabling it to outperform larger open-source models such as DeepSeek-V4-Pro on three evaluation sets.

Critical weaknesses

The paper only compares against a single method baseline, DataMind, and does not systematically compare with existing related data generation methods; it also only provides evaluations on three datasets, so it is unclear whether the datasets were specifically selected because the proposed method performs better on them.

Questions for authors

The paper discusses the datasets on which the proposed work performs relatively well. It is recommended that it also fairly analyze the dataset scenarios in which the proposed work performs poorly or is unsuitable, and explain why this is the case. This would provide readers with more comprehensive information about the paper’s method and fair suggestions for its application and optimization.

5 comments

We sincerely thank the reviewer for recognizing the analytical improvements from DataPraxis-5K and its competitive results against larger open-weight models. We address the suggestions on baseline comparisons, evaluation coverage, and practical applicability below.


Q1: Could you broaden the comparison with existing task-synthesis methods?

R1: We thank the reviewer for this helpful suggestion. As mentioned in Section 4.1 of our submission draft, DataMind is our original synthesis baseline. Section 5 describes it as a task-type-guided method focused on data analysis, making it the closest baseline among the related methods. Several other methods target GUI or coding agents, whose task settings differ from ours. Our DataMind comparison matches the training-example count, teacher model, agent harness, and SFT recipe.

We appreciate the suggestion to broaden the comparison. Given the substantial financial cost of data synthesis, we add DIVE$^{[1]}$ as a representative exploration-based baseline. We adapt its implementation to generate 2,000 trajectories from our analytical workspaces.

DIVE explores the available tools through iterative interactions and accumulates execution evidence. It derives tasks from this evidence and uses the resulting queries to guide further exploration. DataPraxis inherits and merges verified executable workspaces, retaining scripts, intermediate data, and findings for continued investigation across complementary branches.

Both methods use the same Qwen3.6-27B backbone, seed workspaces, teacher, and harness. We also match the number of training examples and the SFT recipe:

Training dataDABStepJobBench
DIVE-2K44.5640.21
DataPraxis-2K48.8945.50

DataPraxis improves over DIVE by 4.33 percentage points on DABStep and 5.29 points on JobBench. With the backbone, training recipe, and example count held fixed, these gains support the value of verified workspace evolution for analytical training.

We thank the reviewer again. We follow the reviewer’s suggestion to add this comparison and clarify the supporting evidence to improve our paper.


Q2: Could you clarify the benchmark-selection criteria and extend evaluation to another analytical setting?

R2: We thank the reviewer for raising this question. As mentioned in Section 4.1 under “Benchmarks” of our submission draft, the benchmarks cover complementary analytical tasks. DABStep tests multi-step reasoning, JobBench tests professional deliverable construction, and RideBench tests operational business analysis. We appreciate the reviewer’s suggestion to broaden coverage and include DAComp$^{[2]}$ as an additional public benchmark for this response. It covers enterprise data engineering and analytical workflows.

We also include recent agents BigBang-v1$^{[3]}$ and Occamy-1.0$^{[4]}$. All models below share the Qwen3.6-35B-A3B backbone:

ModelDAComp
Base model53.72
DataMind-5K54.79
BigBang-v154.31
Occamy-1.064.86
DataPraxis-5K76.34

DataPraxis improves over the base model by 22.62 points and exceeds the strongest evaluated alternative by 11.48 points, extending the evidence beyond the original three benchmarks. Frontier models and the two additional models we compare against use large-scale training data and complex post-training pipelines, while DataPraxis achieves these results with only 5K SFT trajectories.

We thank the reviewer again. We follow the reviewer’s suggestion to add DAComp and clarify the benchmark-selection criteria and evaluation scope to improve our paper.

References

[1] A. Chen et al. DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use. 2026.

[2] F. Lei et al. DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle. 2025.

[3] The BigBang Team. BigBang: Pursuing Open-Ended Intelligence through Self-Evolving Synthesis of Verifiable Frontier Tasks. 2026.

[4] W. Chen et al. Occamy-1.0: Open Pareto-frontier 35B Intelligence for Co-work. 2026.

Q3: Could you clarify a boundary of the dataset’s applicability and provide practical guidance?

R3: We thank the reviewer for this valuable suggestion. As mentioned in Section 3.2.3 of our submission draft, DataPraxis constructs analytical tasks with domain-appropriate deliverables. We appreciate the reviewer’s helpful suggestion on applicability and clarify the current scope as follows.

Precise spreadsheet editing is outside the current dataset’s focus. Although spreadsheets can serve as analytical inputs or deliverables, the dataset is not specifically designed for changing designated cells while preserving all unrelated formulas, formatting, and workbook structure. For these applications, we recommend adding dedicated spreadsheet-editing trajectories with checks for both the requested changes and unintended modifications.

We thank the reviewer again. We will follow the reviewer’s suggestion to add this applicability discussion and practical guidance to improve our paper.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer pf2h,

Thank you again for your constructive feedback and for recognizing the analytical improvements achieved with DataPraxis-5K and its competitive performance against larger models. As the discussion period is nearing its end, we would like to kindly check whether our responses have addressed your concerns.

We have broadened the synthesis-method comparison by adding DIVE and extended evaluation beyond the original three benchmarks to DAComp, alongside comparisons with BigBang-v1 and Occamy-1.0. We have also clarified the dataset’s applicability boundary: it focuses on analytical tasks, while precise spreadsheet editing would benefit from dedicated editing trajectories and preservation checks. The additional evaluations and applicability discussion are included in the revised manuscript and highlighted in blue.

If you have any remaining questions, we would be happy to address them before the discussion ends. If our responses have resolved your concerns, we would sincerely appreciate it if you could consider updating your score.

Best regards, The Authors

Reviewer_aNKGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose DataPraxis, an analytics task synthesis framework to generate tasks instead of generating tasks directly from raw data. Using the DataPraxis framework, the authors conduct a 5000 samples dataset and train DataPraxis-27B and DataPraxis-35B-A3B. The extensive experimental results show that DataPraxis-27B has better performance than GLM-5.2

Critical strengths
  1. This paper studies an very important problem: how to generate tasks for model training
  2. The framework is simple but powerful
  3. The writing of this paper is easy to follow
  4. The authors use both dense model and moe model for experiments and compared the trained open-weight models with frontier models in Table 2
  5. I like the Figure 6 (b), it looks very cool
Critical weaknesses
  1. This paper tries to fix the number of samples, however, I believe that the number of tokens are more important than the number of samples
  2. I feel a bit confused that DataPraxis-5K has sample results as Qwen3.6-35B-A3B on DABStep Easy tasks, could you check it?
  3. is it possible to release RideBench to the public?
Questions for authors

Please check critical weakness

4 comments

We sincerely thank the reviewer for recognizing the importance of task synthesis, the effectiveness of our framework, and the clarity of the presentation. We also appreciate the positive feedback on our evaluations across dense and MoE models and on Figure 6(b). As mentioned in Section 4.4, we use hierarchical clustering inspired by Clio$^{[1]}$ to identify agent behavior types without predefined categories. Figure 6(b) visualizes the action representations using UMAP. We recommend this combination for analyzing agent behavior and diversity across training datasets.

Q1: Could you account for training-token volume when comparing the datasets?

R1: We thank the reviewer for this helpful suggestion. DataPraxis-5K contains 309.83M tokens, while DataMind-5K contains 136.65M tokens. As mentioned in Section 4.1 of our submission draft, the original comparison controls the number of training examples, teacher model, agent harness, and SFT recipe. We appreciate the reviewer’s suggestion to report token volume alongside example count.

We additionally compare DataPraxis-2K, containing 119.857M tokens, with DataMind-5K, containing 136.65M tokens. The corresponding Qwen3.6-27B benchmark scores are reported in Table 2 of Section 4.2 and Table 3 of Section 4.3 of our submission draft:

Training dataTokensDABStepJobBench
DataMind-5K136.65M46.1937.90
DataPraxis-2K119.857M48.8945.50

Under comparable token budgets, DataPraxis-2K outperforms DataMind-5K by 2.70 percentage points on DABStep and 7.60 points on JobBench. Together with the original comparison at a fixed sample count, these gains support the quality of our synthesized data beyond differences in training-data volume.

We thank the reviewer again. We follow the reviewer’s suggestion to add token statistics and highlight this comparison to improve our paper.


Q2: Could you clarify the identical DABStep Easy scores before and after training?

R2: We thank the reviewer for carefully checking these results. As reported in Section 4.2 and Table 2 of our submission draft, both Qwen3.6-35B-A3B and DataPraxis-35B-A3B achieve 81.94% on DABStep Easy. We confirm that these identical scores are correct.

DABStep contains 72 Easy tasks and 378 Hard tasks. Both Easy scores correspond to 59 correct answers out of 72. Each additional correct answer changes the Easy score by approximately 1.39 percentage points, giving this split coarser accuracy resolution. Table 2 also shows similar Easy scores across many evaluated models.

We recommend using DABStep Hard as a more informative reference for challenging analytical capabilities. On this split, DataPraxis improves Qwen3.6-35B-A3B from 27.78% to 47.09%.

We thank the reviewer again. We follow the reviewer’s suggestion to clarify the interpretation of the Easy and Hard results to improve our paper.


Q3: Could RideBench be made publicly available?

R3: We thank the reviewer for their interest in RideBench. As mentioned in Appendix A and the Ethics Statement of our submission draft, RideBench uses simulator-generated order, driver, and passenger records. The simulation is informed by real business data and operational characteristics from our company’s ride-hailing business. We appreciate the reviewer’s helpful suggestion on making the benchmark accessible to the community.

We will consider a public release after reviewing the materials for sensitive business information and completing appropriate sanitization and validation. Because the simulation reflects real business conditions, RideBench supports evaluation of analyses with practical and economic value.

We thank the reviewer again for their interest. We follow the reviewer’s suggestion to clarify the considerations for public release to improve our paper.


References

[1] A. Tamkin et al. Clio: Privacy-Preserving Insights into Real-World AI Use. 2024.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer aNKG,

Thank you again for your thoughtful comments and positive assessment of our framework, presentation, and evaluations across dense and MoE models. We also appreciate your interest in Figure 6(b). As the discussion period is nearing its end, we would like to kindly check whether our responses have addressed your concerns.

We have reported the training-token counts and provided a comparison between DataPraxis-2K and DataMind-5K under comparable token budgets. We have also verified the identical DABStep Easy scores and explained their interpretation. Regarding RideBench, we clarified that its simulated data are informed by real business data and that public release will be considered after sanitization and validation.

If you have any remaining questions, we would be happy to clarify them before the discussion ends. If our responses have resolved your concerns, we would sincerely appreciate it if you could consider updating your score.

Best regards, The Authors

Thanks for your response! I will keep the score.

Dear Reviewer aNKG,

Thank you for taking the time to consider our response. We appreciate your thoughtful feedback and your time throughout the review process.

Best regards, The Authors

Reviewer_DyH5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DataPraxis generates analytical training tasks by extending and combining verified workspace findings, then uses evidence-based filtering to collect trajectories.

Critical strengths

S1: The framework has a clear motivation. Actual analysis can reveal useful follow-up questions that are difficult to generate from raw data descriptions alone.

S2: The downstream improvements are substantial across both model backbones.

S3: The ablations meaningfully examine data volume, evolution depth, verification, and teacher choice.

Critical weaknesses

W1: Some claims are stronger than the evidence supports. The introduction states that DataPraxis-27B achieves "leading performance among the evaluated models," but Table 2 shows that it doesn't lead any of the three benchmarks or the overall average. Similarly, claims that workspace evolution discovers deeper questions beyond existing synthesis methods are not established by the current comparisons. The reuslts support substantial improvements over the base model and DataMind, but broader claims of superiority should be narrowed or supported by additional experiments.

W2: Related to W1, DataMind is the only alternative synthesis method evaluated. The paper claims advantages over exploration-based and compositional approaches in related work, but does not compare against representatives of these methods.

W3: Maybe I'm missing something, but the reliability of the generated tasks and rubrics needs independent validation. Executable results and agreement between models can still share analytical mistakes. A human audit/verification step should be added to assess task correctness, evidential support, and whether rubrics accept valid alternative solutions.

Questions for authors

Please address my comments W1-3.

6 comments

We sincerely thank the reviewer for recognizing the clear motivation of DataPraxis, the substantial improvements across both backbones, and the informative ablation studies. We address the comments on performance claims, synthesis baselines, and task reliability below.

Q1: Could you clarify the scope of the performance claims and provide comparisons with recent agents?

R1: We thank the reviewer for this constructive suggestion. As mentioned in Section 4.2 and Table 2 of our submission draft, DataPraxis-27B achieves an average score of 60.39. This exceeds DeepSeek-V4-Pro at 58.53 and GPT-5.6-Terra at 51.30. On DABStep Hard, DataPraxis-27B scores 49.74, compared with 49.47 for GLM-5.2. On JobBench, it scores 51.90, compared with 51.20 for GLM-5.2. These results demonstrate that our 27B model can match or surpass several frontier models on the evaluated analytical benchmarks.

To make the claim more precise, we will revise the statement in Section 1 from “leading performance among the evaluated models” to: “DataPraxis substantially improves both evaluated backbones and achieves competitive performance with several frontier models on the evaluated analytical benchmarks.”

We also appreciate the reviewer’s suggestion to broaden the comparisons. We add evaluations of BigBang-v1 [1]^{[1]} and Occamy-1.0 [2]^{[2]}. We further include DAComp [3]^{[3]} as an additional public benchmark for this response, covering enterprise data engineering and analysis:

Model (Qwen3.6-35B-A3B backbone)JobBenchDACompDABStep-Hard
Base model31.3053.7227.78
DataMind-5K26.6054.7929.63
BigBang-v133.8454.3134.66
Occamy-1.038.1864.8644.44
DataPraxis-5K41.0076.3447.09

DataPraxis achieves the highest scores across all three benchmarks in this comparison. Frontier models and the two additional models we compare against use large-scale training data and complex post-training pipelines, while DataPraxis achieves these results with only 5K SFT trajectories.

We thank the reviewer again. We follow the reviewer’s suggestion to add these comparisons and clarify the scope of our claims to improve our paper.

References

[1] The BigBang Team. BigBang: Pursuing Open-Ended Intelligence through Self-Evolving Synthesis of Verifiable Frontier Tasks. 2026.

[2] W. Chen et al. Occamy-1.0: Open Pareto-frontier 35B Intelligence for Co-work. 2026.

[3] F. Lei et al. DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle. 2025.

Q2: Could you supplement the DataMind comparison with a representative exploration-based synthesis baseline?

R2: We thank the reviewer for this helpful suggestion. As mentioned in Section 4.1 of our submission draft, DataMind is our original synthesis baseline. Section 5 describes it as a task-type-guided method focused on data analysis, making it the closest baseline among the related methods. Several other methods target GUI or coding agents, whose task settings differ from ours.

We appreciate the suggestion to broaden the comparison. Given the substantial financial cost of data synthesis, we add DIVE$^{[4]}$ as a representative exploration-based baseline. We adapt its open-source implementation to our analytical workspaces and synthesize 2,000 training trajectories.

DIVE explores the available tools through iterative interactions and accumulates execution evidence. It derives tasks from this evidence and uses the resulting queries to guide further exploration. DataPraxis inherits and merges verified executable workspaces, preserving scripts, intermediate data, and findings for continued analysis. It combines complementary branches and constructs standalone tasks from the resulting analytical dependencies.

Both methods use the same seed workspaces, Qwen3.6-27B backbone, GLM-5.2 teacher, and Claude Code harness. We also match the number of training examples and the SFT recipe:

Training dataDABStepJobBench
Base model35.7838.50
DIVE-2K44.5640.21
DataPraxis-2K48.8945.50

DataPraxis improves over DIVE by 4.33 percentage points on DABStep and 5.29 points on JobBench. With the backbone, training recipe, and example count held fixed, these gains support the value of verified workspace evolution for analytical training.

We thank the reviewer again. We follow the reviewer’s suggestion to add this comparison and clarify which advantages are supported by experiments to improve our paper.

References

[4] A. Chen et al. DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use. 2026.

Q3: Could you explain how verification addresses shared errors and valid alternative solutions, and provide human validation of task reliability?

R3: We thank the reviewer for this valuable suggestion. As mentioned in Sections 3.2.2 to 3.3 and Appendices E.5 to E.7 of our submission draft, our verification includes the following safeguards against shared analytical errors.

The evolution validator checks assumptions, metric definitions, and supporting evidence, explicitly distinguishing reproducibility from correctness. Reference solvers independently reconstruct solutions from the initial workspace. As specified in Appendix E.7, “Verifier Validation,” the rubric validator recomputes claims from input data independently of agreement among trajectories.

Appendix B.2, Table 4 illustrates how verification handles valid alternatives. The lift for Bakery and Tea changes from 0.920 to 1.197 under two different basket populations. The validator removes the fixed conclusion about this pair while retaining the requirement to distinguish frequency from positive association. Appendix E.6 also explicitly permits open answers justified by the task and data.

We also appreciate the suggestion on human validation and add a human audit. Eight in-house business analytics experts cross-evaluated 30 business-related tasks. Experts scored each dimension separately on a scale from 1 to 3. The levels were 1 for unreasonable or unreliable, 2 for moderate, and 3 for reasonable or reliable. The average scores were 2.38/3.00 for task reasonableness and 2.78/3.00 for final-result reliability. On overlapping ratings, experts assigned the same rating tier in over 70% of cases. As a complementary model-based check, 99.5% of 200 sampled trajectories passed a correctness audit by Claude Opus 5. Further verification confirmed that the failed case resulted from an incorrect assessment by Claude Opus 5.

We thank the reviewer again. We follow the reviewer’s suggestion to add the audit results and protocols to improve our paper. We will also highlight how verification handles valid alternative solutions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper Revision

Dear Reviewers and Area Chairs,

We sincerely thank the reviewers for their thoughtful and constructive feedback. We have revised the manuscript to clarify our claims and incorporate additional evaluations. All additions and modifications are highlighted in blue. If the paper is accepted, the revised text will appear in black in the final version. The main changes are summarized below:

  1. [Section 1] Clarified the performance claim, stating that DataPraxis substantially improves both evaluated backbones, with DataPraxis-27B achieving competitive performance with frontier models such as GLM-5.2.

  2. [Section 4.1] Reported the token count of DataPraxis-5K, totaling 309.83M tokens.

  3. [Ethics Statement] Clarified that synthesis is automated and that sampled tasks and results undergo an additional expert audit.

  4. [Appendix D] Clarified the dataset’s analytical focus and its applicability boundary for precise spreadsheet editing.

  5. [Appendix F.1 and Table 5] Added a controlled comparison with DIVE, an exploration-based synthesis method.

  6. [Appendix F.2 and Table 6] Added DAComp evaluation and comparisons with BigBang-v1 and Occamy-1.0.

  7. [Appendix G] Added a report on the human audit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.