- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces DataOrchestra, a framework that selects a curation pipeline for each pretraining-data chunk. An orchestrator decides whether to drop, retain, or clean a chunk, selects among noise pruning, surface rectification, and pedagogical augmentation, and generates chunk-specific rewriting instructions. Its training plans are refined through tool execution and verification. Experiments cover pretraining from scratch at 0.5B–7B, several web corpora, and mathematical continued pretraining, with comparisons against individual curation methods, raw/rewritten mixtures, and fixed multi-stage pipelines.
Critical strengths
-
The framework addresses a practical limitation of uniform data processing. Selecting both the required operations and their instructions is a coherent contribution, and the execution-based plan refinement makes the supervision more grounded than a teacher's initial routing decisions alone.
-
The empirical evaluation is broad and includes relevant stronger controls. In particular, Sections 4.4–4.5 compare against mixtures and fixed pipelines rather than relying only on isolated rewriting baselines. Table 2 shows consistent average gains across model sizes, while Table 3 supports a useful quality–processing-compute trade-off.
Critical weaknesses
-
The incremental quality advantage over the strongest fixed pipeline is relatively small at the scale of the controlled comparison: Table 3 reports 40.13 versus 39.83 with mixing. Averaging three neighboring checkpoints reduces reporting noise but does not establish variability across independent training runs. This limits confidence in the size of the incremental gain, although the broader results and processing savings remain encouraging.
-
The fidelity analysis relies substantially on a model also used for generating and verifying training supervision. The human checks reported in Appendices A.3 and F are helpful, but the sample sizes, annotation procedure, and types of remaining disagreements are not specified sufficiently to assess the strength of the content-preservation claim.
Questions for authors
-
Using the existing runs, could you report the checkpoint-wise differences and a paired uncertainty estimate for the strongest Table 3 comparison? Please distinguish evaluation uncertainty from training-seed uncertainty; additional large-model pretraining runs are not necessary for this clarification.
-
Could you specify the sample sizes and annotation protocol underlying the reported >83% and >86% human agreement, and give a short breakdown of disagreements involving content loss or factual errors? A small, clearly described manual audit would be useful if these details are not already recorded.
-
Please clarify whether Table 3's 782 EFLOPs is the deployment-time curation cost only, and separately summarize the one-time supervision-generation and orchestrator-training costs, or explain over what corpus size those costs would be amortized. This would make the efficiency claim easier to interpret.
3 comments
Q3: Please clarify whether Table 3's 782 EFLOPs is the deployment-time curation cost only, and separately summarize the one-time supervision-generation and orchestrator-training costs, or explain over what corpus size those costs would be amortized. This would make the efficiency claim easier to interpret.
Yes. Table 3’s 782 EFLOPs covers deployment-time curation only, including orchestrator inference and NP/SR/PA tool execution. The one-time preparation costs are estimated separately:
| One-time component | Estimated cost (EFLOPs) |
|---|---|
| Supervision generation: coarse plans and plan evolution | 140.25 |
| 1.7B orchestrator SFT, three epochs | 9.95 |
| Total | 150.20 |
Supervision-generation costs are computed from recorded input/output token usage using Appendix G’s approximation, with 22B active parameters for the Qwen3-235B-A22B teacher. SFT uses the Trainer-reported FLOPs estimate. These figures cover the recorded generation and SFT runs, excluding exploratory reruns and seed-pool preparation. Including these one-time costs brings the total for preparing the orchestrator and curating the 40B-token corpus to approximately 932.20 EFLOPs.
The fine-tuning cost itself is small, and the supervision-generation and training costs are incurred only once for a given orchestrator. Reusing the same orchestrator across subsequent curation runs spreads this fixed preparation cost over a larger token volume, progressively reducing its per-token overhead.
Thank you for clarifying the human audit and accounting for the one-time curation costs. These details strengthen the evidence for the efficiency claim, and I will raise my soundness score from 2 to 3. The quality gain over the strongest fixed pipeline is modest, and its stability across independent training runs remains uncertain, so I am keeping my overall rating
We thank the reviewer for the helpful suggestions on evaluation and cost accounting.
Checkpoint-wise comparison. We compare DataOrchestra with the strongest fixed-pipeline baseline in Table 3, using fastText-selected raw-data mixing for both methods. The table reports the unsmoothed average over the same 11 benchmarks at the last three checkpoints:
DataOrchestra leads at all three checkpoints. The mean paired difference is 0.310 percentage points, with a sample standard deviation of 0.109 points across the three checkpoint pairs. This describes checkpoint-to-checkpoint variation.
Evaluation repeatability. As described in Appendix D, the general benchmarks in Table 3 use likelihood-based (PPL) evaluation: we score each candidate answer and select the highest-scoring option. Where generative evaluation is used, we apply greedy decoding without sampling. Conditional on a fixed checkpoint, evaluation example, and prompt configuration, repeated evaluation therefore introduces no decoding-sampling variance, apart from potential numerical nondeterminism.
Training-seed variability. Separately, we conducted two additional runs with different training seeds for the 0.5B × RedPajama-V2 setting in Table 2. Including the original run, the three-seed results are:
The AVG is 40.02 ± 0.09, where SD is the sample standard deviation across three training runs. This quantifies training-seed variability and highlights the significant advantages of our approach.
This question closely relates to Reviewer mG2T’s Q4. We reproduce our response here for completeness:
We acknowledge that the paper omitted important details of the human audits. Both audits assess the reliability of Qwen3-235B judgments: one concerns verification during plan evolution, and the other concerns the rewriting-quality evaluation in Figure 6. Using the same model means that the automated evaluation is not model-independent; the blinded human audits provide an external reliability check.
Sampling. We sampled 200 examples for each audit. For plan evolution, sampling was stratified across datasets and NP/SR/PA operations, with approximately balanced coverage. For rewriting-quality evaluation, we sampled 200 before/after text pairs with approximately balanced coverage across datasets. This sampling aimed to cover diverse sources rather than let a single dataset dominate the audit.
Annotation. We built a Streamlit annotation interface to facilitate human review. Three PhD annotators with LLM research backgrounds independently labeled each example using this interface, using the corresponding teacher-model evaluation criteria. Annotators could not see the model’s judgments. Before formal annotation, we completed a 20-example calibration exercise to align interpretation of the criteria and annotation granularity.
Agreement. We computed each annotator’s exact-match agreement with the model and averaged the three rates, yielding over 83% for plan evolution and over 86% for rewriting-quality evaluation. For NP, exact match requires agreement on the binary
keep/revertverdict. For SR/PA plan-evolution judgments, it requires simultaneous agreement on all three binary labels:no_content_loss,no_factual_error, andno_surface_noise. For rewriting-quality evaluation, it requires simultaneous agreement on the two labels,no_content_lossandno_factual_error.