acceptodds
Under review as a conference paper at ICLR 2027

Predicting Neural Scaling Laws without Training: A Data Manifold Oracle

Abstract

Neural scaling laws set large-scale pretraining budgets, yet estimating a data scaling exponent requires fitting dozens of compute-intensive proxy models, one sweep per candidate corpus. We propose the Data Manifold Oracle (DMO), a training-free framework that predicts neural data scaling trajectories directly from the static, lossless compression properties of raw text. By linking the sequence-length scaling behavior of DEFLATE compression to neural loss decay, DMO forecasts the data scaling exponent before any model is trained on the candidate corpus. We pair that predictor with a theoretical boundary: symbol-level compressibility cannot uniquely resolve the dimension of a continuous source, at any sample size. A calibration frozen on pretraining corpora predicts the exponents of five held-out corpus families to an MAE of without GPU overhead, against for a proxy model trained on the candidate corpus itself. By turning scaling law estimation from a multi-GPU training challenge into a data-profiling task, DMO makes each additional candidate corpus cost one compression pass instead of one training sweep.

Primary area:
datasets and benchmarks
Keywords:
Scaling Laws, Compressions

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5mE8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Data Manifold Oracle (DMO), which predicts the neural data-scaling exponent αD\alpha_D from the change in DEFLATE compression rates across different text window lengths. The goal is to reduce the cost of training full scaling-law grids for candidate corpora. The authors calibrate the predictor on 38 corpora and report an MAE of 0.029 on five held-out corpus families. The paper also studies text-order perturbations, document selection, and theoretical limits on recovering source geometry from symbolic compression statistics.

Critical strengths
  • Practical motivation. Predicting data-scaling behavior before training could reduce the computational cost of evaluating candidate corpora.
  • Simple and efficient method. DMO-Corpus requires only text compression and slope estimation, without training a proxy model for each new corpus.
  • Prospective evaluation. The authors freeze the calibration rule before evaluating five new corpus families, reporting an MAE of 0.029 and comparing against simple baselines.
  • Broad experimental analysis. The paper investigates text-block shuffling, compressor choices, and document selection, while also reporting cases where the proposed approach does not work well.
Critical weaknesses
  • Limitations of the baseline comparison. On the five held-out families, the single-size proxy model has a higher MAE but achieves a Spearman correlation of 0.90, compared with 0.80 for DMO. The current comparison does not establish that DMO is generally superior to training-based approaches for corpus selection.
  • Unclear scope of applicability. The fitted exponent depends on the model, training setup, and budget range. It remains unclear whether the frozen linear mapping generalizes reliably to larger models, different training configurations, or substantially different corpus distributions.
  • Limited connection between theory and the main predictor. The theoretical results mainly establish limits on identifiability and do not derive the relationship between compression slope and αD\alpha_D. The central contribution therefore remains primarily empirical, with its underlying mechanism and transferability not yet fully established.
Questions for authors
  • Can the frozen predictor be evaluated on more independent corpus families, with uncertainty estimates for its prediction error?
  • Can the authors provide a more controlled comparison between DMO, domain-mean baselines, and calibrated lightweight training-based proxies under the same evaluation protocol?
  • How sensitive is the predictor to model scale, training budget, tokenizer choice, and data preprocessing? What is the intended scope of applicability?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer 5mE8 [1]

Over 11 held-out families from two prospective rounds the frozen map's MAE is 0.087, with a 95% parent-family bootstrap interval of 0.04 to 0.14 (Appendix B.7). That MAE lies 0.032 below the calibration-set mean, and the 95% interval of the gap runs from 0.014 to 0.050. It is within 0.004 of the domain mean's 0.091.

Can the frozen predictor be evaluated on more independent corpus families, with uncertainty estimates for its prediction error?

familiesnDMO MAE (95% interval)calibration-set meandomain meanρ, DMO / domain meaninside the ±0.103 prediction interval
panel, leave-one-family-out380.0510.0790.0510.80 / 0.54–
first round50.029 (0.013 to 0.046)0.0480.0450.80 / –5 of 5
second round, toward code and math60.135 (0.062 to 0.216)0.1770.1290.66 / –3 of 6
both rounds110.087 (0.04 to 0.14)0.1190.0910.79 / 0.878 of 11

The ±0.103 half-width is the empirical 95th percentile of the largest absolute residual per family under leave-one-family-out calibration. We set it before training the families of each round. Table 4 gives it together with each round's decision rule and outcome. Table 5 lists every family's prediction and observed αD\alpha_D. Part of the second round's error comes from the training protocol (Appendix B.2). The held-out grids use a critical-batch schedule and one learning rate. On the cells both protocols share, that protocol raises 8 of the panel's 20 code and mathematics exponents by 0.12 to 0.67. The slope orders the panel corpora at ρ = 0.81 under both protocols. On the five web and prose families of both rounds the map's MAE is 0.033.

Both the predicted and the observed αD\alpha_D of the first round have a measured uncertainty (Appendix B.7). On the prediction side, recomputing the slope at 20 window seeds changes the predicted αD\alpha_D of the six first-round corpora by an SD of 0.001 to 0.007. Their ordering keeps a mean rank correlation of 0.997 when the windows are resampled and 0.986 (range 0.88 to 1.00) when 100-document blocks are resampled in 20 replicates. On the target side, the training-seed SD of the fitted αD\alpha_D is 0.003 to 0.024 per first-round family (RMS 0.017). Target noise alone would give an MAE of about 0.014, half of the first round's 0.029.

Can the authors provide a more controlled comparison between DMO, domain-mean baselines, and calibrated lightweight training-based proxies under the same evaluation protocol?

Tables 2 and 6 of the revision give that comparison. We map DMO and each 16M proxy tier to αD\alpha_D through a line fitted on the panel, refit in each fold for the panel scores. We then score them and both mean baselines on the same 27 leave-one-family-out folds (37 panel corpora for the 1.13B tier) and on the same 11 held-out families. General response 1/2 gives the scores of each tier.

Table 6 makes the comparison at equal text (reply to 6Rhc). Both predictors receive the first 300, 1,000 or 10,000 documents of each corpus. The proxy trains on 90% of each sample. At 300 documents (a median of 0.6M tokens) the slope's panel MAE is 0.050 against the proxy's 0.079, a paired difference whose 95% interval runs from 0.016 to 0.051. On the held-out families the slope's MAE is 0.090 against 0.11.

On the panel DMO's MAE is within 0.005 of the best tier's (exact sign-flip p = 0.21 over 27 parent families). The proxy tiers train for 3 to 58 H100-minutes per candidate. DMO needs about a minute on one CPU node with the download included. It matches the proxy's panel accuracy from a few hundred documents of raw text.

AuthorsOpenReview ↗

Reply to Reviewer 5mE8 [2]

How sensitive is the predictor to model scale, training budget, tokenizer choice, and data preprocessing? What is the intended scope of applicability?

On 37 panel corpora the slope orders the exponents fitted at each model size alone at ρ = 0.78 to 0.82 from 16M to 1.2B parameters. On identical cells of the same corpora it reaches ρ = 0.81 under each of our two training protocols. Section 4.1 gives the tested range. It covers GPT-2-style models of 16M to 1.2B parameters with the GPT-NeoX-20B tokenizer and sequence length 512. Each run trains on 11M to 900M tokens (nominal budgets 12M to 700M). On 29 language corpora, per-size fits with a free floor give ρ = 0.88 to 0.96 from 16M to 1.2B, with 0.89 at 1.2B (Appendix B.4).

For the training setup, the median per-corpus change in the exponent is +0.002 when one learning rate replaces the better of two (model sizes and budgets held equal). It is +0.005 once the batch schedule also changes. For the training budget, the slope orders the exponents refitted without the 12M-token budget at ρ = 0.78 and 0.67 under the two protocols (Section 4.1).

The predictor uses no tokenizer: the slope is computed on raw UTF-8 text (Appendix B.8). Section 2.1 counts the tokenizer among the settings that define αD\alpha_D. Every grid uses the GPT-NeoX-20B tokenizer (Section 4.1), so the map's coefficients are calibrated for that tokenizer. On eight non-English corpora the slope keeps their order (ρ = 0.91). For preprocessing and selection inside one pool, DMO-Doc scores individual documents (Section 5).

Section 4.1 describes DMO as suited to screening many candidates from small samples of their text. A ranking of candidates uses the slope's order directly. A predicted value uses the map's coefficients, which apply under the protocol of their calibration grids. A lab that trains under another protocol refits the map once on its own grids.

The theoretical results mainly establish limits on identifiability and do not derive the relationship between compression slope and alpha_D.

Section 3 motivates the slope-to-αD\alpha_D map from the conditional-entropy tail, and Section 4 tests the frozen map prospectively and by two interventions. Block shuffling inside one corpus changes the slope and αD\alpha_D together across fifteen conditions on code, web text and mathematics (fixed-effects t = 6.5 to 7.4 for the zlib, PPM and LZ3 slopes; Appendix B.9). Two StarCoder mixtures matched in slope to block-shuffled versions fall within 0.0026 and 0.0035 of those versions' αD\alpha_D, where the weighted average of their ingredients' exponents misses by 0.0105 and 0.0164 (general response 2/2). On the theory side, Cagnetta et al. (2026) derive neural scaling laws from how conditional entropy decays with context length. Proposition G.2 makes that conditional-entropy tail one term of an exact decomposition of a context-mm predictor's log-loss. The reply to Ppz4 relates the slope to that tail and tests it on sources with a known tail (Appendices K.1 and P).

Thank you for the detailed response and additional experiments. The rebuttal has addressed most of my concerns, particularly regarding the fairer baseline comparisons and the uncertainty analysis of prediction errors. However, I still have two remaining concerns:

  • The second round of independent experiments failed to meet most of the predefined success criteria. Moreover, across both rounds, DMO still achieves lower rank correlation than the simple domain-mean baseline and lightweight training-based proxies. Although DMO offers advantages in computational efficiency and limited-data settings, its reliability for pretraining data selection remains to be established.
  • Although the experiments cover models with up to 1.2B parameters, the primary calibration and independent evaluation use training budgets of only 12M–900M tokens, substantially below the Chinchilla compute-optimal reference budgets for larger models. While Appendix B.4 provides correlation analysis at higher token budgets, the predictive accuracy of the frozen mapping in this regime remains unverified. Given the sensitivity of the fitted exponent to the training budget, I remain concerned about the applicability of DMO to practical large-scale pretraining.

I appreciate the authors' efforts and improvements during the rebuttal, but these two concerns remain insufficiently addressed. I would be willing to reconsider my rating if the authors can adequately address them.

Reviewer_Ppz4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper proposes to predict the Chincilla data scaling exponent αD\alpha_D via the compressibility of the data distribution. Many experiments are conducted analyzing the proposed impact of data compressibility as compressed by DEFLATE, revealing

Critical strengths

The sheer volume of experiments is very impressive, convincingly demonstrating

  • The predicability of αD\alpha_D from zlib compression levels
  • The negative effect of block length on αLZ\alpha_{LZ} and αD\alpha_D
  • The non-linear effect of data mixing of αD\alpha_D
Critical weaknesses

The theory in the appendix focuses primarily on approximation theory determining the size of the transformer head (and resulting model scaling exponent αM\alpha_M), and largely disconnected from the empirical results studying αD\alpha_D.

Questions for authors

There is no comparison between DMO-Doc and a simpler compression only baseline. Why? Also rHigr_{Hig} is pulled out of the air and never properly defined, making it hard to understand.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer Ppz4

The requested compression-only baseline is now included in Section 5 and Appendix N.2. It improves upon a random draw by 0.008 bits per byte (bpb) at both evaluated checkpoints (t=−2.9t = -2.9 and −4.5-4.5). RICH utilizes both scores to achieve an additional 0.009 bpb reduction over the baseline at step 12,145, the checkpoint detailed in Section 5 (t=−2.6t = -2.6). Notably, at this step, persistence reduces the loss by 0.020 bpb within the high-compression-rate subset from which the baseline is drawn (RICH vs. COLLOQUIAL, t=−2.9t = -2.9).

There is no comparison between DMO-Doc and a simpler compression only baseline.

The baseline uniformly samples documents from the high-compression-rate half of the second DCLM pool, upon which the 18-run 150M grid is trained (Appendix N.2). Consequently, RICH and COLLOQUIAL documents appear in their natural proportions (54:46). This baseline is trained at the 150M scale and paired by seed with both the RICH configuration and the random draw.

We evaluate all 18 runs from this grid at step 12,145 (out of 22,889), corresponding to the validation step in the training script. Additionally, the baseline, RICH, and the random draw are evaluated at the final saved checkpoint (step 22,500). The paired tt-statistics for the baseline are calculated using seeds 42 through 46 (4 degrees of freedom).When analyzed as a 2 × 2 design, the 12 quadrant runs of this grid (seeds 42–44) yield a persistence effect of −0.024 bits per byte (bpb) (t=−4.9t = -4.9) and a compression effect of −0.019 bpb (t=−3.8t = -3.8).

In the four-size grid presented in Figure 4A, the high-persistence partition achieves lower held-out loss across all model sizes, improving by 0.032 bpb at 150M and 0.023 bpb at 2B (2.6 to 3.7 seed standard deviations). Notably, across 4,000 DCLM documents, persistence and compression rate are nearly uncorrelated (Spearman ρ=+0.09\rho = +0.09).

Also r_Hig is pulled out of the air and never properly defined, making it hard to understand.

Section 5 now defines rHigr_{\rm Hig} in full. A single document is too short for the corpus-level slope of Section 2. DMO-Doc therefore pairs the gzip rate with the Higuchi persistence rHig=2−DHigr_{\rm Hig} = 2 - D_{\rm Hig}, a statistic over lags of 1 to 32 bytes that one document is long enough to estimate. Here DHigD_{\rm Hig} is the fractal dimension (Higuchi, 1988) of the walk traced by the running sum of the document's mean-centred UTF-8 byte values. We estimate DHigD_{\rm Hig} from how the walk's length shrinks when it is sampled every k=1k = 1 to 3232 bytes. Independent bytes give a random walk with rHig≈0.5r_{\rm Hig} \approx 0.5. Runs of above- or below-average byte values that persist over these lags raise rHigr_{\rm Hig}, and alternation lowers it. Algorithm N.1 gives every step. The quadrants split at the pool medians (rHigr_{\rm Hig} 0.4443 and gzip rate 3.962 bits per byte), so high and low are relative to the pool. On 20,000-sample test signals with lags up to 64, the routine returns D = 2.0 for white noise, 1.5 for Brownian motion and 1.0 for a sinusoid.

The theory in the appendix focuses primarily on approximation theory determining the size of the transformer head

Section 3 relates the slope to αD\alpha_D via the conditional-entropy tail, hm−hXh_m - h_X, and links this tail to the training loss. Correspondingly, the head bound of Proposition G.4 and Theorem H.1 governs the parameter exponent. By Proposition G.2, a predictor qmq_m using the last mm symbols exhibits an excess log-loss of L(qm)−hX=(hm−hX)+E KL(pm∥qm)L(q_m) - h_X = (h_m - h_X) + \mathbb{E}\ \mathrm{KL}(p_m \Vert q_m), representing the out-of-context information plus the in-context error. Cagnetta et al. (2026) derive αD=γCE/(2βcorr)\alpha_D = \gamma_{\rm CE}/(2\beta_{\rm corr}), where γCE\gamma_{\rm CE} and βcorr\beta_{\rm corr} are the decay exponents of the first term and the token correlations, respectively.

For an ideal code, the compression rate at window nn exceeds hXh_X by the first term averaged over shorter contexts (Appendix A).Across seven Santa Fe sources, the slope perfectly ranks the known γCE\gamma_{\rm CE} values (Spearman ρ=+1.00\rho = +1.00, Appendix P).

Because hXh_X and γCE\gamma_{\rm CE} share the same Zipf exponent in this setting, the paired comparison in Appendix K.1 isolates the two by fixing the drawn facts and character positions, modifying only whether a repeated fact produces a repeated answer. This causes the slope to rise by 6.4 to 32 times its noise floor, with the maximum increase (0.060) aligning with the highest γCE\gamma_{\rm CE} (0.70). For real corpora, the map's coefficients are derived from trained grids. On real corpora two experiments you list as strengths test the relation. Slope-matched StarCoder mixtures fall within 0.0026 and 0.0035 of the αD\alpha_D of the block-shuffled versions they match, the grid's refit-noise scale. The weighted average of their ingredients' exponents misses by 0.0105 and 0.0164.

Reviewer_6RhcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DMO, a method to predict the data scaling exponent of a data source without directly training models on that dataset. The method does so by estimating the data scaling exponent under DEFLATE and shows this exponent correlates with the data scaling exponent of a transformer language model.

Critical strengths

The method leverages the hypothesized universality of the scaling law exponent with respective to training and architecture details to estimate it cheaply from a simple compression algorithm, making the approach very cheap to implement.

Critical weaknesses

I see very little practical value in the method introduced by this paper for actually guiding practitioner's choice in making training decisions at scale, which was the motivation behind estimating the scaling law exponent in the first place. While the method is extremely cheap, its quality is not convincing. Indeed, the paper treats the exponent estimated by a power law fit from a grid of (N,D) runs as ground truth. By my estimate, running this sweep costs total 6ND flops around that of training a 300M parameter model at D/N=20, which itself is completely negligible for any serious effort for training models at scale. If one were to use to estimated exponent to make decisions at scale (say optimal D vs N setting), then it is preferable to spend this compute to obtain a good estimate. Running scaling ladders that itself consists of different model sizes is also standard practice for model development, which means one can get the "ground truth" exponent without actually paying the additional cost.

Questions for authors
  1. In Appendix B.8, you show the Equal-Target Training Proxy, which fits the exponent by running the 16M models at three budgets, ranks the five new families perfectly. This method is also very cheap and can run in minutes on a single GPU. Is there any reason not to prefer this, more reliable method? The one issue you identified, namely, the bias of this estimate, should be fixable by applying the same calibration procedure used in DMO.
  2. If the intended selling point of DMO is that it can guide scaling choices for large-scale training, could you demonstrate a practical application of using DMO to improve training at scale?
1 comment
AuthorsOpenReview ↗

Reply to Reviewer 6Rhc

Unlike the proposed calibrated proxy that requires a separate training run on 52M to 1.13B tokens for every candidate model, DMO matches its panel accuracy without any candidate-specific training. Under identical calibration, the proxy tiers produce a panel MAE between 0.047 and 0.060, while DMO achieves 0.051. Furthermore, DMO operates entirely without a tokenizer or GPU, providing an unbiased alternative methodology. DMO also maintains its accuracy when restricted to just 300 documents per corpus—a setting where the proxy’s MAE degrades to 0.079 (see table below).

Running scaling ladders that itself consists of different model sizes is also standard practice for model development, which means one can get the "ground truth" exponent without actually paying the additional cost.

A development ladder gives αD\alpha_D only for the corpus a lab has already chosen (Section 1). DMO scores the candidate corpora before that choice, after a one-time calibration on the panel. A full grid costs 30 H100-hours for every candidate corpus. One 6 × 5 grid requires 1.6 × 10¹⁹ FLOPs (Appendix B.2). Scoring each first-round candidate took 25 to 63 seconds of wall time on one CPU node, including the download of its sample (Appendix B.8).

Is there any reason not to prefer this, more reliable method? The one issue you identified, namely, the bias of this estimate, should be fixable by applying the same calibration procedure used in DMO.

On the panel DMO matches the calibrated proxy's MAE with no training on the candidate. On 300 or 1,000 documents it is more accurate. Table 2 runs the calibration you propose, and that calibration needs the same trained panel as DMO. Each proxy tier maps its 16M exponent to αD\alpha_D through a line fitted on the panel's own 16M rows (Appendix B.8). Without that line the five-budget proxy predicts 0.036 to 0.11 for round one, whose observed exponents run from 0.11 to 0.24.

With that line the tier with budgets up to 40M tokens has leave-one-family-out MAE 0.053 and the tier up to 130M tokens 0.047. DMO's is 0.051 (best-tier gap 0.005; exact sign-flip p = 0.21 over 27 families).

The proxy's accuracy depends on how much text it trains on (Table 6). We gave the 12M–130M tier the first 300, 1,000 or 10,000 documents of each corpus (medians of 0.6M, 1.9M and 17M tokens). It trains on 90% of each sample and validates on the rest:

documents per corpuspanel MAE, DMO / proxypanel ρ, DMO / proxyproxy − DMO (95% interval)held-out MAE, DMO / proxy
first 3000.050 / 0.0790.82 / −0.56+0.030 (0.016 to 0.051)0.090 / 0.11
first 1,0000.050 / 0.0790.81 / 0.07+0.029 (0.014 to 0.048)0.089 / 0.11
10,0000.051 / 0.0560.80 / 0.72+0.005 (−0.007 to 0.021)0.087 / 0.19
10,000; proxy on full stream0.051 / 0.0470.80 / 0.90−0.005 (−0.012 to 0.002)0.087 / 0.076

The proxy tiers of Table 2 train on 52M to 1.13B tokens per candidate. DMO keeps its panel MAE on a median of 0.6M tokens of raw text.

If the intended selling point of DMO is that it can guide scaling choices for large-scale training, could you demonstrate a practical application of using DMO to improve training at scale?

On the 37 panel corpora the slope orders the exponents fitted at each model size from 16M to 1.2B at ρ = 0.78 to 0.82 (Section 4.1). On 29 language corpora, per-size fits with a free floor give ρ = 0.88 to 0.96 from 16M to 1.2B, with 0.89 at 1.2B (Appendix B.4).

Section 5 tests a second application. DMO-Doc selects pretraining documents by each document's gzip rate and byte-level persistence. On DCLM the two high-persistence quadrants train to lower held-out Paloma loss than the two low-persistence quadrants at all four sizes from 150M to 2B. The gap is 0.032 bits per byte at 150M and 0.023 at 2B, where each quadrant trains on 11.2 to 11.9B tokens. At each size it is 2.6 to 3.7 times the SD of 0.0086 from the 150M grid (Figure 4A).

The comparison against unselected data is a 150M grid of 18 runs on a second pool, scored at step 12,145 of 22,889 (Figure 4B; Appendix N.2). With two seeds added for a paired comparison, RICH (high rate and high persistence) trains 0.017 below a random draw of the pool on 5 seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.