- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Data Manifold Oracle (DMO), which predicts the neural data-scaling exponent from the change in DEFLATE compression rates across different text window lengths. The goal is to reduce the cost of training full scaling-law grids for candidate corpora. The authors calibrate the predictor on 38 corpora and report an MAE of 0.029 on five held-out corpus families. The paper also studies text-order perturbations, document selection, and theoretical limits on recovering source geometry from symbolic compression statistics.
Critical strengths
- Practical motivation. Predicting data-scaling behavior before training could reduce the computational cost of evaluating candidate corpora.
- Simple and efficient method. DMO-Corpus requires only text compression and slope estimation, without training a proxy model for each new corpus.
- Prospective evaluation. The authors freeze the calibration rule before evaluating five new corpus families, reporting an MAE of 0.029 and comparing against simple baselines.
- Broad experimental analysis. The paper investigates text-block shuffling, compressor choices, and document selection, while also reporting cases where the proposed approach does not work well.
Critical weaknesses
- Limitations of the baseline comparison. On the five held-out families, the single-size proxy model has a higher MAE but achieves a Spearman correlation of 0.90, compared with 0.80 for DMO. The current comparison does not establish that DMO is generally superior to training-based approaches for corpus selection.
- Unclear scope of applicability. The fitted exponent depends on the model, training setup, and budget range. It remains unclear whether the frozen linear mapping generalizes reliably to larger models, different training configurations, or substantially different corpus distributions.
- Limited connection between theory and the main predictor. The theoretical results mainly establish limits on identifiability and do not derive the relationship between compression slope and . The central contribution therefore remains primarily empirical, with its underlying mechanism and transferability not yet fully established.
Questions for authors
- Can the frozen predictor be evaluated on more independent corpus families, with uncertainty estimates for its prediction error?
- Can the authors provide a more controlled comparison between DMO, domain-mean baselines, and calibrated lightweight training-based proxies under the same evaluation protocol?
- How sensitive is the predictor to model scale, training budget, tokenizer choice, and data preprocessing? What is the intended scope of applicability?
5 comments
Reply to Reviewer 5mE8 [2]
How sensitive is the predictor to model scale, training budget, tokenizer choice, and data preprocessing? What is the intended scope of applicability?
On 37 panel corpora the slope orders the exponents fitted at each model size alone at ρ = 0.78 to 0.82 from 16M to 1.2B parameters. On identical cells of the same corpora it reaches ρ = 0.81 under each of our two training protocols. Section 4.1 gives the tested range. It covers GPT-2-style models of 16M to 1.2B parameters with the GPT-NeoX-20B tokenizer and sequence length 512. Each run trains on 11M to 900M tokens (nominal budgets 12M to 700M). On 29 language corpora, per-size fits with a free floor give ρ = 0.88 to 0.96 from 16M to 1.2B, with 0.89 at 1.2B (Appendix B.4).
For the training setup, the median per-corpus change in the exponent is +0.002 when one learning rate replaces the better of two (model sizes and budgets held equal). It is +0.005 once the batch schedule also changes. For the training budget, the slope orders the exponents refitted without the 12M-token budget at ρ = 0.78 and 0.67 under the two protocols (Section 4.1).
The predictor uses no tokenizer: the slope is computed on raw UTF-8 text (Appendix B.8). Section 2.1 counts the tokenizer among the settings that define . Every grid uses the GPT-NeoX-20B tokenizer (Section 4.1), so the map's coefficients are calibrated for that tokenizer. On eight non-English corpora the slope keeps their order (ρ = 0.91). For preprocessing and selection inside one pool, DMO-Doc scores individual documents (Section 5).
Section 4.1 describes DMO as suited to screening many candidates from small samples of their text. A ranking of candidates uses the slope's order directly. A predicted value uses the map's coefficients, which apply under the protocol of their calibration grids. A lab that trains under another protocol refits the map once on its own grids.
The theoretical results mainly establish limits on identifiability and do not derive the relationship between compression slope and alpha_D.
Section 3 motivates the slope-to- map from the conditional-entropy tail, and Section 4 tests the frozen map prospectively and by two interventions. Block shuffling inside one corpus changes the slope and together across fifteen conditions on code, web text and mathematics (fixed-effects t = 6.5 to 7.4 for the zlib, PPM and LZ3 slopes; Appendix B.9). Two StarCoder mixtures matched in slope to block-shuffled versions fall within 0.0026 and 0.0035 of those versions' , where the weighted average of their ingredients' exponents misses by 0.0105 and 0.0164 (general response 2/2). On the theory side, Cagnetta et al. (2026) derive neural scaling laws from how conditional entropy decays with context length. Proposition G.2 makes that conditional-entropy tail one term of an exact decomposition of a context- predictor's log-loss. The reply to Ppz4 relates the slope to that tail and tests it on sources with a known tail (Appendices K.1 and P).
Thank you for the detailed response and additional experiments. The rebuttal has addressed most of my concerns, particularly regarding the fairer baseline comparisons and the uncertainty analysis of prediction errors. However, I still have two remaining concerns:
- The second round of independent experiments failed to meet most of the predefined success criteria. Moreover, across both rounds, DMO still achieves lower rank correlation than the simple domain-mean baseline and lightweight training-based proxies. Although DMO offers advantages in computational efficiency and limited-data settings, its reliability for pretraining data selection remains to be established.
- Although the experiments cover models with up to 1.2B parameters, the primary calibration and independent evaluation use training budgets of only 12M–900M tokens, substantially below the Chinchilla compute-optimal reference budgets for larger models. While Appendix B.4 provides correlation analysis at higher token budgets, the predictive accuracy of the frozen mapping in this regime remains unverified. Given the sensitivity of the fitted exponent to the training budget, I remain concerned about the applicability of DMO to practical large-scale pretraining.
I appreciate the authors' efforts and improvements during the rebuttal, but these two concerns remain insufficiently addressed. I would be willing to reconsider my rating if the authors can adequately address them.
Reply to Reviewer 5mE8 [1]
Over 11 held-out families from two prospective rounds the frozen map's MAE is 0.087, with a 95% parent-family bootstrap interval of 0.04 to 0.14 (Appendix B.7). That MAE lies 0.032 below the calibration-set mean, and the 95% interval of the gap runs from 0.014 to 0.050. It is within 0.004 of the domain mean's 0.091.
The ±0.103 half-width is the empirical 95th percentile of the largest absolute residual per family under leave-one-family-out calibration. We set it before training the families of each round. Table 4 gives it together with each round's decision rule and outcome. Table 5 lists every family's prediction and observed αD. Part of the second round's error comes from the training protocol (Appendix B.2). The held-out grids use a critical-batch schedule and one learning rate. On the cells both protocols share, that protocol raises 8 of the panel's 20 code and mathematics exponents by 0.12 to 0.67. The slope orders the panel corpora at ρ = 0.81 under both protocols. On the five web and prose families of both rounds the map's MAE is 0.033.
Both the predicted and the observed αD of the first round have a measured uncertainty (Appendix B.7). On the prediction side, recomputing the slope at 20 window seeds changes the predicted αD of the six first-round corpora by an SD of 0.001 to 0.007. Their ordering keeps a mean rank correlation of 0.997 when the windows are resampled and 0.986 (range 0.88 to 1.00) when 100-document blocks are resampled in 20 replicates. On the target side, the training-seed SD of the fitted αD is 0.003 to 0.024 per first-round family (RMS 0.017). Target noise alone would give an MAE of about 0.014, half of the first round's 0.029.
Tables 2 and 6 of the revision give that comparison. We map DMO and each 16M proxy tier to αD through a line fitted on the panel, refit in each fold for the panel scores. We then score them and both mean baselines on the same 27 leave-one-family-out folds (37 panel corpora for the 1.13B tier) and on the same 11 held-out families. General response 1/2 gives the scores of each tier.
Table 6 makes the comparison at equal text (reply to 6Rhc). Both predictors receive the first 300, 1,000 or 10,000 documents of each corpus. The proxy trains on 90% of each sample. At 300 documents (a median of 0.6M tokens) the slope's panel MAE is 0.050 against the proxy's 0.079, a paired difference whose 95% interval runs from 0.016 to 0.051. On the held-out families the slope's MAE is 0.090 against 0.11.
On the panel DMO's MAE is within 0.005 of the best tier's (exact sign-flip p = 0.21 over 27 parent families). The proxy tiers train for 3 to 58 H100-minutes per candidate. DMO needs about a minute on one CPU node with the download included. It matches the proxy's panel accuracy from a few hundred documents of raw text.