acceptodds
Under review as a conference paper at ICLR 2027

DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

Abstract

Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples, without adapting to the needs of each example. We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example. Given a chunk of pretraining data, an orchestrator decides whether to drop, untouch, or clean it. For a chunk to be cleaned, it selects one or more downstream operations, ranging from programmatic editing to different forms of LLM-based rewriting. For each rewriting step, it further generates a concrete instruction, which is executed by the corresponding downstream tool model. We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks. DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Pretraining, Data-Centric AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hiQwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces DataOrchestra, a framework that selects a curation pipeline for each pretraining-data chunk. An orchestrator decides whether to drop, retain, or clean a chunk, selects among noise pruning, surface rectification, and pedagogical augmentation, and generates chunk-specific rewriting instructions. Its training plans are refined through tool execution and verification. Experiments cover pretraining from scratch at 0.5B–7B, several web corpora, and mathematical continued pretraining, with comparisons against individual curation methods, raw/rewritten mixtures, and fixed multi-stage pipelines.

Critical strengths
  1. The framework addresses a practical limitation of uniform data processing. Selecting both the required operations and their instructions is a coherent contribution, and the execution-based plan refinement makes the supervision more grounded than a teacher's initial routing decisions alone.

  2. The empirical evaluation is broad and includes relevant stronger controls. In particular, Sections 4.4–4.5 compare against mixtures and fixed pipelines rather than relying only on isolated rewriting baselines. Table 2 shows consistent average gains across model sizes, while Table 3 supports a useful quality–processing-compute trade-off.

Critical weaknesses
  1. The incremental quality advantage over the strongest fixed pipeline is relatively small at the scale of the controlled comparison: Table 3 reports 40.13 versus 39.83 with mixing. Averaging three neighboring checkpoints reduces reporting noise but does not establish variability across independent training runs. This limits confidence in the size of the incremental gain, although the broader results and processing savings remain encouraging.

  2. The fidelity analysis relies substantially on a model also used for generating and verifying training supervision. The human checks reported in Appendices A.3 and F are helpful, but the sample sizes, annotation procedure, and types of remaining disagreements are not specified sufficiently to assess the strength of the content-preservation claim.

Questions for authors
  1. Using the existing runs, could you report the checkpoint-wise differences and a paired uncertainty estimate for the strongest Table 3 comparison? Please distinguish evaluation uncertainty from training-seed uncertainty; additional large-model pretraining runs are not necessary for this clarification.

  2. Could you specify the sample sizes and annotation protocol underlying the reported >83% and >86% human agreement, and give a short breakdown of disagreements involving content loss or factual errors? A small, clearly described manual audit would be useful if these details are not already recorded.

  3. Please clarify whether Table 3's 782 EFLOPs is the deployment-time curation cost only, and separately summarize the one-time supervision-generation and orchestrator-training costs, or explain over what corpus size those costs would be amortized. This would make the efficiency claim easier to interpret.

3 comments

We thank the reviewer for the helpful suggestions on evaluation and cost accounting.

Q1: Using the existing runs, could you report the checkpoint-wise differences and a paired uncertainty estimate for the strongest Table 3 comparison? Please distinguish evaluation uncertainty from training-seed uncertainty; additional large-model pretraining runs are not necessary for this clarification.

Checkpoint-wise comparison. We compare DataOrchestra with the strongest fixed-pipeline baseline in Table 3, using fastText-selected raw-data mixing for both methods. The table reports the unsmoothed average over the same 11 benchmarks at the last three checkpoints:

Training stepDataOrchestra + MixFull NP → SR → PA pipeline + MixPaired difference (pp)
last-339.87239.639+0.233
last-240.28240.019+0.263
last-140.25239.817+0.434
Mean40.13539.825+0.310

DataOrchestra leads at all three checkpoints. The mean paired difference is 0.310 percentage points, with a sample standard deviation of 0.109 points across the three checkpoint pairs. This describes checkpoint-to-checkpoint variation.

Evaluation repeatability. As described in Appendix D, the general benchmarks in Table 3 use likelihood-based (PPL) evaluation: we score each candidate answer and select the highest-scoring option. Where generative evaluation is used, we apply greedy decoding without sampling. Conditional on a fixed checkpoint, evaluation example, and prompt configuration, repeated evaluation therefore introduces no decoding-sampling variance, apart from potential numerical nondeterminism.

Training-seed variability. Separately, we conducted two additional runs with different training seeds for the 0.5B × RedPajama-V2 setting in Table 2. Including the original run, the three-seed results are:

RunARC-EARC-CMMLUHellaS.RACEWinoG.OBQAPIQACSQASIQASciQAVG
Original (Table 2)44.7527.1627.6036.8030.1851.8829.2066.5020.5638.4066.9039.99
Additional seed 144.5527.3727.4837.0730.0351.8129.3666.3420.7538.1366.6739.96
Additional seed 245.0427.0427.8436.6330.4952.0629.1166.7720.4138.7367.2040.12
Mean ± SD44.78 ± 0.2527.19 ± 0.1727.64 ± 0.1836.83 ± 0.2230.23 ± 0.2351.92 ± 0.1329.22 ± 0.1366.54 ± 0.2220.57 ± 0.1738.42 ± 0.3066.92 ± 0.2740.02 ± 0.09

The AVG is 40.02 ± 0.09, where SD is the sample standard deviation across three training runs. This quantifies training-seed variability and highlights the significant advantages of our approach.

Q2: Could you specify the sample sizes and annotation protocol underlying the reported >83% and >86% human agreement, and give a short breakdown of disagreements involving content loss or factual errors? A small, clearly described manual audit would be useful if these details are not already recorded.

This question closely relates to Reviewer mG2T’s Q4. We reproduce our response here for completeness:

We acknowledge that the paper omitted important details of the human audits. Both audits assess the reliability of Qwen3-235B judgments: one concerns verification during plan evolution, and the other concerns the rewriting-quality evaluation in Figure 6. Using the same model means that the automated evaluation is not model-independent; the blinded human audits provide an external reliability check.

Sampling. We sampled 200 examples for each audit. For plan evolution, sampling was stratified across datasets and NP/SR/PA operations, with approximately balanced coverage. For rewriting-quality evaluation, we sampled 200 before/after text pairs with approximately balanced coverage across datasets. This sampling aimed to cover diverse sources rather than let a single dataset dominate the audit.

Annotation. We built a Streamlit annotation interface to facilitate human review. Three PhD annotators with LLM research backgrounds independently labeled each example using this interface, using the corresponding teacher-model evaluation criteria. Annotators could not see the model’s judgments. Before formal annotation, we completed a 20-example calibration exercise to align interpretation of the criteria and annotation granularity.

Agreement. We computed each annotator’s exact-match agreement with the model and averaged the three rates, yielding over 83% for plan evolution and over 86% for rewriting-quality evaluation. For NP, exact match requires agreement on the binary keep/revert verdict. For SR/PA plan-evolution judgments, it requires simultaneous agreement on all three binary labels: no_content_loss, no_factual_error, and no_surface_noise. For rewriting-quality evaluation, it requires simultaneous agreement on the two labels, no_content_loss and no_factual_error.

Q3: Please clarify whether Table 3's 782 EFLOPs is the deployment-time curation cost only, and separately summarize the one-time supervision-generation and orchestrator-training costs, or explain over what corpus size those costs would be amortized. This would make the efficiency claim easier to interpret.

Yes. Table 3’s 782 EFLOPs covers deployment-time curation only, including orchestrator inference and NP/SR/PA tool execution. The one-time preparation costs are estimated separately:

One-time componentEstimated cost (EFLOPs)
Supervision generation: coarse plans and plan evolution140.25
1.7B orchestrator SFT, three epochs9.95
Total150.20

Supervision-generation costs are computed from recorded input/output token usage using Appendix G’s approximation, with 22B active parameters for the Qwen3-235B-A22B teacher. SFT uses the Trainer-reported FLOPs estimate. These figures cover the recorded generation and SFT runs, excluding exploratory reruns and seed-pool preparation. Including these one-time costs brings the total for preparing the orchestrator and curating the 40B-token corpus to approximately 932.20 EFLOPs.

The fine-tuning cost itself is small, and the supervision-generation and training costs are incurred only once for a given orchestrator. Reusing the same orchestrator across subsequent curation runs spreads this fixed preparation cost over a larger token volume, progressively reducing its per-token overhead.

Thank you for clarifying the human audit and accounting for the one-time curation costs. These details strengthen the evidence for the efficiency claim, and I will raise my soundness score from 2 to 3. The quality gain over the strongest fixed pipeline is modest, and its stability across independent training runs remains uncertain, so I am keeping my overall rating

Reviewer_8SXWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DataOrchestra, a framework for example-specific pretraining data curation. For each text chunk, an orchestrator decides whether to drop, keep, or clean it. For cleaning, it selects appropriate operations among Noise Pruning (NP), Surface Rectification (SR), and Pedagogical Augmentation (PA), and generates chunk-specific rewriting instructions. The orchestrator is trained on approximately 300K examples constructed through teacher-generated plans and execution-based plan evolution. Experiments across different model sizes, web corpora, and math continued pretraining demonstrate consistent improvements over existing data-processing methods. DataOrchestra also achieves competitive performance with lower inference compute than fixed multi-stage pipelines.

Critical strengths
  1. The paper proposes an adaptive framework that selects appropriate processing operations and generates chunk-specific rewriting instructions instead of applying a fixed pipeline to all examples. An important contribution is the execution-based plan evolution, which refines the teacher-generated plans based on the actual behavior of downstream tool models.

  2. The paper provides extensive experimental validation across different model sizes, datasets, and pretraining settings.

Critical weaknesses
  1. The paper makes claims based on small performance differences from a single training run. The paper does not report results from multiple training seeds. However, several claims are based on relatively small performance differences. These differences may be affected by training randomness. Without results from multiple seeds, it is difficult to determine whether these improvements are consistent.

  2. It is unclear whether the performance improvements come from better data curation or additional knowledge introduced by the rewriting model. In the PA stage, Qwen3-4B adds relevant knowledge to the original text, increasing the average chunk length by approximately 2.4 times. However, the paper does not report whether the rewritten data overlaps with the evaluation benchmarks or what proportion of the final training tokens comes from PA-generated content.

Questions for authors
  1. Could the authors provide the mean and standard deviation from multiple random seeds for the key experiments using the 0.5B model?
  2. Does DataOrchestra still have a computational efficiency advantage when the costs of generating the orchestrator's training data and fine-tuning are included?
  3. Figure 5 shows that PA significantly increases the number of tokens by adding explanations and knowledge. What proportion of the final pretraining tokens comes from PA-generated content?
  4. How do the authors ensure that the additional knowledge introduced by PA does not cause benchmark contamination?
2 comments

We thank the reviewer for the constructive feedback and suggestions.

Q1: Could the authors provide the mean and standard deviation from multiple random seeds for the key experiments using the 0.5B model?

We conducted two additional runs with different training seeds for the 0.5B × RedPajama-V2 setting in Table 2. Together with the original run, the results across three seeds are:

RunARC-EARC-CMMLUHellaS.RACEWinoG.OBQAPIQACSQASIQASciQAVG
Original (Table 2)44.7527.1627.6036.8030.1851.8829.2066.5020.5638.4066.9039.99
Additional seed 144.5527.3727.4837.0730.0351.8129.3666.3420.7538.1366.6739.96
Additional seed 245.0427.0427.8436.6330.4952.0629.1166.7720.4138.7367.2040.12
Mean ± SD44.78 ± 0.2527.19 ± 0.1727.64 ± 0.1836.83 ± 0.2230.23 ± 0.2351.92 ± 0.1329.22 ± 0.1366.54 ± 0.2220.57 ± 0.1738.42 ± 0.3066.92 ± 0.2740.02 ± 0.09

SD is the sample standard deviation across the three training runs. All three runs outperform that baseline, supporting a consistent advantage across training seeds. It highlights the significant advantages of our approach.

Q2: Does DataOrchestra still have a computational efficiency advantage when the costs of generating the orchestrator's training data and fine-tuning are included?

Table 3’s 782 EFLOPs covers deployment-time curation only, including orchestrator inference and tool execution. We additionally estimate the one-time costs from the saved supervision-generation records and the 1.7B orchestrator’s training log:

One-time componentEstimated cost (EFLOPs)
Coarse-grained plan generation43.45
Plan evolution, including tool calls and iterative verification96.79
1.7B orchestrator SFT, three epochs9.95
Total150.20

Generation uses the approximation in Appendix G, with 22B active parameters for the Qwen3-235B-A22B teacher; SFT uses the Trainer-reported FLOPs estimate. These figures cover the recorded generation and SFT runs, excluding exploratory reruns and seed-pool preparation.

Even when the entire one-time preparation cost is charged to the first 40B-token corpus, DataOrchestra requires approximately 932.20 EFLOPs, retaining a 24.6% compute saving over the full fixed pipeline’s 1,236 EFLOPs while achieving higher downstream accuracy (39.99 vs. 39.59 AVG in Table 3).

The fine-tuning cost itself is small, and supervision generation and SFT are one-time costs for a given orchestrator. The same orchestrator can be reused across subsequent curation runs, spreading this fixed preparation cost over a larger token volume and progressively reducing its per-token overhead.

Q3: Figure 5 shows that PA significantly increases the number of tokens by adding explanations and knowledge. What proportion of the final pretraining tokens comes from PA-generated content?

We report the volume of PA-processed text below, using the same corpus accounting as our response to Reviewer mG2T, Q7.

For DataOrchestra on RedPajama-V2, we summarize the corpus-size trajectory below.

Processing stagecorpus size (B tokens)
Raw40.00
After DROP34.50
After NP31.77
After SR29.49
After PA40.34

Training-data sampling. Using the same text-volume estimate as the table, 19.91B of the final 40.34B tokens come from PA-processed chunks before training-data sampling. Under our 20B-token training budget, approximately 9.87B tokens are expected to come from PA-processed chunks, assuming the sampled documents retain the curated pool’s text-volume composition. Both figures include preserved source content and PA additions.

We agree that PA-expanded documents consume more tokens and can reduce source-document coverage under a fixed training budget, under source-document-level sampling. However, matching source documents while retaining each transformed document in full would instead produce different training-token totals and computational budgets across methods. We consider this a reasonably fair comparison because it evaluates data-curation methods under the same training-token budget. Similar fixed-budget comparisons are widely used in pretraining-data research, including SciPedia[1], RePro[2], and ReWire[3].

The additional experiments in Reviewer mG2T’s Q1 and Table 3 suggest that PA usage alone does not explain DataOrchestra’s gains. Frequency-matched random routing performs worse despite matching the operation-combination frequencies, including PA, while the fixed NP → SR → PA pipeline also underperforms DataOrchestra despite applying PA uniformly to all retained data. These results support selectively matching processing to content: simply increasing PA usage is insufficient to reproduce DataOrchestra’s downstream gains.

[1] SciPedia: Unlocking the Value of Scientific Data for Pre-training

[2] RePro: Training Language Models to Faithfully Recycle the Web for Pretraining

[3] Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models

Q4: How do the authors ensure that the additional knowledge introduced by PA does not cause benchmark contamination?

We audited the six curated corpora using an n-gram-based lexical-overlap detector. Training text and benchmark questions are normalized with Unicode NFKC, case folding, and alphanumeric tokenization. We check for shared 13-token sequences and separately match complete normalized question strings. The reported metric, normalized full-question overlap (exact_high), requires a complete question match satisfying minimum-length, lexical-diversity, and same-document context constraints where applicable. Each evaluation row is counted at most once per corpus, and benchmark-level rates are averaged with equal weights.

CorpusCurated mean overlap
RedPajama-V20.1088%
DCLM-RefinedWeb0.0802%
C40.1645%
FineWeb0.0995%

For the nine science-reasoning benchmarks actually reported in the paper, the corresponding math-corpus averages are:

CorpusRaw mean overlapCurated mean overlap
OpenWebMath5.0749%3.1039%
MegaMath7.5478%5.4692%

The audit finds very low mean lexical overlap in the Web corpora (0.08%–0.16%). For both math corpora, mean overlap decreases after curation, and over 92% of the post-curation matched benchmark rows also match the raw data. Thus, most detected mathematical overlap concerns questions already present before curation, rather than evidence of newly introduced overlap by our DataOrchestra curation method. We will include this additional data contamination report in the revision of the paper. Thanks for pointing out!

Reviewer_mG2TOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DataOrchestra, in which a 1.7B orchestrator decides for each text chunk whether to drop, keep, or clean it, selects a subset of noise pruning (NP), surface rectification (SR) and pedagogical augmentation (PA), and writes chunk-specific rewriting instructions executed by small tool models. The orchestrator is fine-tuned on ~300K plans proposed by a teacher LLM and refined through actual tool execution and verification. Experiments pretrain 0.5B–7B models from scratch on four web corpora and run math continued pretraining, reporting gains over single-operation methods, raw/rewritten mixtures and fixed multi-stage pipelines with lower curation compute.

Critical strengths

Most prior curation methods apply a single operation (filtering, line-level editing or rewriting) or one fixed pipeline with a shared prompt to all data. DataOrchestra places these operations under one per-chunk decision interface and grounds the teacher's plans in actual tool behavior: stages that over-prune, hallucinate or barely change the text are removed, and failed rewrites are retried with corrective instructions. The downstream evidence is broad. The method achieves the best average at 0.5B, 1.5B and 7B, with the margin over ProX growing from 0.90 to 1.92 points, and the gains transfer to three other web corpora and two math corpora. It also uses about 37% less curation compute than the full NP/SR/PA pipeline, which makes it practically relevant for large-scale data curation.

Critical weaknesses

The paper does not isolate the value of learned per-chunk routing, which is its central claim. All baselines are either single operations or fixed pipelines applied uniformly to every chunk. Yet the operations substantially change corpus composition: about 25% of chunks are dropped, NP shortens chunks, and PA expands them from about 670 to about 1,594 tokens (Fig. 5). The gains may therefore come from the resulting mixture of preserved, pruned and expanded text rather than from matching each chunk to the right operations. Two controls with the same tool models and token budget are necessary: a simple conditional policy (e.g., selecting NP/SR/PA from DataMan scores or detectable noise patterns), and a frequency-matched random assignment that keeps the learned proportions of operation combinations but assigns them independently of chunk content.

Questions for authors

The fixed NP → SR → PA order and the independent True/False judgment per stage are not justified. Neither choice is ablated, and the paper does not show whether the order reflects real dependencies between operations or is simply a convention. What motivates this order and factorization over letting the orchestrator produce a joint plan?

For rewrites that pass on the first attempt, the verifier writes the instruction after seeing both the input and the final output. These hindsight instructions can describe edits that are not predictable from the input alone, which creates a mismatch with inference, where the orchestrator only sees the input. This may also explain why the 4B orchestrator writes overly conservative instructions and underperforms the 1.7B one. What fraction of training instructions are hindsight-generated versus corrective?

The same Qwen3-235B model serves as teacher, verifier and rewriting-quality judge, so the faithfulness evaluation in Fig. 6 is not independent of the supervision. The reported 83%/86% human agreement is given without sample size, sampling procedure or error breakdown. The paper should report these details.

The seed pool is rebalanced so that low- and high-quality documents are equally represented, and the drop threshold is deliberately lenient (score < 2). The deployment distribution differs from this rebalanced one, yet the paper does not examine whether the orchestrator's decision rates (25.1% drop, 10.8% untouched) are calibrated on the actual corpora. How sensitive are the downstream results to the drop threshold?

The end-to-end rewriting baseline uses only Qwen3-4B. Without comparing against a stronger model that either cleans the data directly or acts as a zero-shot router for the same tools, it is unclear whether training a dedicated 1.7B orchestrator is the most cost-effective choice. What are the accuracy and curation cost of these alternatives on a subset?

The token accounting is not reported. Starting from 40B raw tokens, how many tokens remain for each method after dropping, pruning and PA expansion, and is any data repeated or subsampled to reach the 20B/30B budgets? Because PA more than doubles chunk length, a PA-heavy corpus covers fewer distinct source documents under a fixed token budget, which affects the comparison.

Chunks of up to 1,024 tokens are processed independently and concatenated. Independent decisions, such as dropping a middle chunk or expanding one chunk but not its neighbors, can break document-level coherence, especially for code and long structured documents. Does the orchestrator receive any document-level context, and how often do such inconsistencies occur?

4 comments

Q1(central claim): The paper does not isolate the value of learned per-chunk routing, which is its central claim. All baselines are either single operations or fixed pipelines applied uniformly to every chunk. Yet the operations substantially change corpus composition: about 25% of chunks are dropped, NP shortens chunks, and PA expands them from about 670 to about 1,594 tokens (Fig. 5). The gains may therefore come from the resulting mixture of preserved, pruned and expanded text rather than from matching each chunk to the right operations. Two controls with the same tool models and token budget are necessary: a simple conditional policy (e.g., selecting NP/SR/PA from DataMan scores or detectable noise patterns), and a frequency-matched random assignment that keeps the learned proportions of operation combinations but assigns them independently of chunk content.

We agree that fixed-pipeline comparisons do not fully isolate content-dependent routing. We added two controls on 0.5B × RedPajama-V2, using the same 40B-token source corpus, 1,024-token chunking, 0.6B NP and 4B SR/PA tools, and 20B training-token budget. Training and evaluation followed Section 4.1, without additional raw-data mixing.

DataMan-score routing. We fixed the policy before evaluation: score ≤1 → DROP; 2 → NP; 3–4 → SR; score-5 chunks are randomly split equally between PA and UNTOUCH. Its cost includes both scoring and subsequent cleaning.

Frequency-matched random routing. We used the same decision-proportion distribution as DataOrchestra in RedPajama-V2, including DROP, UNTOUCH, and all seven operation combinations. We randomly permuted these route labels across chunks with a fixed seed, preserving their exact frequencies while removing content-dependent assignment. Selected stages retained the NP → SR → PA order.

Both controls used fixed general tool prompts.

MethodAVG ↑Curation cost (EFLOPs) ↓
DataOrchestra (Table 3)39.99782
DataMan-score routing39.16952
Frequency-matched random routing36.49732

DataOrchestra’s effectiveness. DataOrchestra outperforms DataMan-score routing. It also outperforms frequency-matched random routing, under the same training-token budget. Thus, matching aggregate operation frequencies alone is insufficient to reproduce DataOrchestra’s gains. These results demonstrate the effectiveness of the combined content-adaptive routing.

Q2: The fixed NP → SR → PA order and the independent True/False judgment per stage are not justified. Neither choice is ablated, and the paper does not show whether the order reflects real dependencies between operations or is simply a convention. What motivates this order and factorization over letting the orchestrator produce a joint plan?

The order is a practical design prior informed by existing pretraining-data curation practice. Nemotron-CC[1] combines extraction and quality filtering with synthetic rephrasing, while SciPedia[2] uses a multi-stage pipeline of content cleaning and pedagogical augmentation. Following this general progression, NP removes obvious noise, SR repairs the remaining text, and PA adds explanatory depth to the cleaned content. This ordering is intended to avoid spending rewriting effort on removable noise or augmenting malformed text. We adopt it as an engineering prior.

We agree that the stages interact: earlier operations can change what later stages need to do. As described in Coarse-grained Plan Generation, the True/False judgments provide only an initial plan. Plan evolution then refines this plan through sequential execution and verification. NP is reverted and removed from the final plan if it over-prunes meaningful content. SR/PA are iteratively refined using corrective instructions. Stages that still fail verification after the maximum five attempts are removed from the final SFT plan and contribute no instruction target. The final training targets therefore incorporate execution and verifier feedback rather than unchanged initial judgments.

[1] Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset

[2] SciPedia: Unlocking the Value of Scientific Data for Pre-training

Q3: For rewrites that pass on the first attempt, the verifier writes the instruction after seeing both the input and the final output. These hindsight instructions can describe edits that are not predictable from the input alone, which creates a mismatch with inference, where the orchestrator only sees the input. This may also explain why the 4B orchestrator writes overly conservative instructions and underperforms the 1.7B one. What fraction of training instructions are hindsight-generated versus corrective?

We agree that output-informed instruction generation requires care. We explicitly considered this issue when designing the verifier prompt (Figure 13). It requires instructions to be "grounded ONLY in what is visibly present in the BEFORE chunk" and, for accepted rewrites, to be “forward-looking from only BEFORE.” Instructions may identify which part needs which treatment, but must not supply the corrected text, reconstructed content, or added explanations.

The audited SR/PA generation records have the following rewrite outcomes before final SFT filtering:

Rewrite-generation outcomeProportion
First-attempt success58.88%
Success following corrective refinement31.91%
Failed all five attempts; excluded from final SFT9.21%

Under the target-construction rule in Paper's Algorithm 1, first-attempt successes receive an instruction synthesized from the input and accepted rewrite. For successes following refinement, the corrective instruction that guided the accepted attempt is retained. Stages that fail all attempts or produce negligible changes are excluded from the final plan. The table reports generation outcomes before this filtering. Both instruction types are subject to the same source-grounding constraints, which are intended to mitigate the training–inference mismatch.

Q4: The same Qwen3-235B model serves as teacher, verifier and rewriting-quality judge, so the faithfulness evaluation in Fig. 6 is not independent of the supervision. The reported 83%/86% human agreement is given without sample size, sampling procedure or error breakdown. The paper should report these details.

We acknowledge that the paper omitted important details of the human audits. Both audits assess the reliability of Qwen3-235B judgments: one concerns verification during plan evolution, and the other concerns the rewriting-quality evaluation in Figure 6. Using the same model means that the automated evaluation is not model-independent; the blinded human audits provide an external reliability check.

Sampling. We sampled 200 examples for each audit. For plan evolution, sampling was stratified across datasets and NP/SR/PA operations, with approximately balanced coverage. For rewriting-quality evaluation, we sampled 200 before/after text pairs with approximately balanced coverage across datasets. This sampling aimed to cover diverse sources rather than let a single dataset dominate the audit.

Annotation. We built a Streamlit annotation interface to facilitate human review. Three PhD annotators with LLM research backgrounds independently labeled each example using this interface, using the corresponding teacher-model evaluation criteria. Annotators could not see the model’s judgments. Before formal annotation, we completed a 20-example calibration exercise to align interpretation of the criteria and annotation granularity.

Agreement. We computed each annotator’s exact-match agreement with the model and averaged the three rates, yielding over 83% for plan evolution and over 86% for rewriting-quality evaluation. For NP, exact match requires agreement on the binary keep/revert verdict. For SR/PA plan-evolution judgments, it requires simultaneous agreement on all three binary labels: no_content_loss, no_factual_error, and no_surface_noise. For rewriting-quality evaluation, it requires simultaneous agreement on the two labels, no_content_loss and no_factual_error.

Q5: The seed pool is rebalanced so that low- and high-quality documents are equally represented, and the drop threshold is deliberately lenient (score < 2). The deployment distribution differs from this rebalanced one, yet the paper does not examine whether the orchestrator's decision rates (25.1% drop, 10.8% untouched) are calibrated on the actual corpora. How sensitive are the downstream results to the drop threshold?

We distinguish two design choices: seed-pool balancing and the teacher’s drop threshold.

Our seed-document balancing between scores 1–3 and 4–5 aims to expose the orchestrator to more examples of data-quality problems while retaining substantial high-quality supervision. It is intended to improve coverage of cleaning needs, rather than reproduce the deployment distribution. An overall quality score also does not directly specify which cleaning operation is needed.

Separately, score < 2 refers to the teacher’s 0–5 recoverable-value rubric, not DataMan’s 1–5 quality score. This lenient gate is intended to preserve noisy but repairable content for subsequent cleaning. The reported drop/untouch rates are observed deployment outcomes, not proportions imposed by the seed balancing.

We acknowledge that these motivations do not establish deployment calibration or downstream robustness to the threshold. Given the computational cost of additional curation and pretraining runs and the experiments prioritized during rebuttal, we do not yet have a downstream threshold-sensitivity study. We leave systematic evaluation of alternative thresholds and sampling ratios to future work.

Q6: The end-to-end rewriting baseline uses only Qwen3-4B. Without comparing against a stronger model that either cleans the data directly or acts as a zero-shot router for the same tools, it is unclear whether training a dedicated 1.7B orchestrator is the most cost-effective choice. What are the accuracy and curation cost of these alternatives on a subset?

Table 3 shows that, at the 4B rewriting-model scale, end-to-end rewriting yields smaller gains than DataOrchestra. Its curation cost is also higher. Our choice of 4B tools targets affordable large-scale curation and maintains comparable model capacity with closely related work: RePro[1] likewise trains its rewriter from Qwen3-4B.

Given the limited compute available during this stage, we have not yet completed a stronger-model accuracy–cost comparison. We therefore provide a rough cost projection using our existing experimental settings, routing workload, and the input/output token distributions observed with the 4B rewriters. We extrapolate to 8B using Appendix G’s FLOPs approximation and the costs reported in Table 3, assuming unchanged routing proportions and token usage. For end-to-end rewriting, we retain the 52-EFLOP drop-gate cost and scale the remaining 772-EFLOP rewriting cost by model size; the combined configuration additionally adjusts for the number of CLEAN chunks:

AlternativeProjected curation cost (EFLOPs)
8B router with the existing NP/SR/PA tools≈1,378
8B end-to-end rewriter with the existing drop gate≈1,596
8B router followed by an 8B end-to-end rewriter on CLEAN chunks≈2,351

The router-only and combined configurations are estimated at 1.76× and 3.01× the current DataOrchestra cost, respectively. We acknowledge that transferring the existing token distributions to an 8B model is approximate: its prompts, routing decisions, and rewrite lengths may differ. These figures indicate only the broad cost trend and motivate our budget-conscious design.

[1] RePro: Training Language Models to Faithfully Recycle the Web for Pretraining

Q7: The token accounting is not reported. Starting from 40B raw tokens, how many tokens remain for each method after dropping, pruning and PA expansion, and is any data repeated or subsampled to reach the 20B/30B budgets? Because PA more than doubles chunk length, a PA-heavy corpus covers fewer distinct source documents under a fixed token budget, which affects the comparison.

For DataOrchestra on RedPajama-V2, we summarize the corpus-size trajectory below.

Processing stagecorpus size (B tokens)
Raw40.00
After DROP34.50
After NP31.77
After SR29.49
After PA40.34

Training-data sampling. Using the same text-volume estimate as the table, 19.91B of the final 40.34B tokens come from PA-processed chunks before training-data sampling. Under our 20B-token training budget, approximately 9.87B tokens are expected to come from PA-processed chunks, assuming the sampled documents retain the curated pool’s text-volume composition. Both figures include preserved source content and PA additions.

We agree that PA-expanded documents consume more tokens and can reduce source-document coverage under a fixed training budget, under source-document-level sampling. However, matching source documents while retaining each transformed document in full would instead produce different training-token totals and computational budgets across methods. We consider this a reasonably fair comparison because it evaluates data-curation methods under the same training-token budget. Similar fixed-budget comparisons are widely used in pretraining-data research, including SciPedia[1], RePro[2], and ReWire[3].

The additional experiments in Q1 and Table 3 suggest that PA usage alone does not explain DataOrchestra’s gains. Frequency-matched random routing performs worse despite matching the operation-combination frequencies, including PA, while the fixed NP → SR → PA pipeline also underperforms DataOrchestra despite applying PA uniformly to all retained data. These results support selectively matching processing to content: simply increasing PA usage is insufficient to reproduce DataOrchestra’s downstream gains.

[1] SciPedia: Unlocking the Value of Scientific Data for Pre-training

[2] RePro: Training Language Models to Faithfully Recycle the Web for Pretraining

[3] Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models

Q8: Chunks of up to 1,024 tokens are processed independently and concatenated. Independent decisions, such as dropping a middle chunk or expanding one chunk but not its neighbors, can break document-level coherence, especially for code and long structured documents. Does the orchestrator receive any document-level context, and how often do such inconsistencies occur?

Chunking long documents poses a general challenge in pretraining-data curation: larger chunks preserve more context, while smaller chunks allow more localized rewriting. We use a 1,024-token target and address boundary risks through both the splitting algorithm and the prompts.

Splitting algorithm. We use a nominal chunk size of 1,024 tokens while prioritizing natural text boundaries: paragraphs first, then line breaks within long paragraphs, with token-level slicing as a fallback for overlong individual lines. This preserves paragraph structure where possible and reduces unnecessary fragmentation. Here, 1,024 denotes a target size rather than a strict upper bound. Retained chunks are concatenated in their original order with paragraph separators.

Prompt safeguards. The drop and verification prompts identify boundary truncation as a normal chunking artifact, rather than a defect. The SR/PA prompts explicitly state: "do not attempt to complete or extend truncated text at chunk boundaries." These instructions are intended to discourage unnecessary edits and invented continuations.

Document-length statistics. A full scan of original RedPajama-V2 document records shows that 91.40% remain a single chunk, while 8.60% are split into multiple chunks. Most documents therefore do not require cross-chunk concatenation. For documents that do, the boundary-aware splitting and prompt safeguards described above are intended to mitigate coherence risks as possible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.