acceptodds
Under review as a conference paper at ICLR 2027

Residual-Augmented Scaling Laws for LLM Pretraining Performance Prediction

Abstract

The computational cost of pretraining Large Language Models (LLMs) at modern scales makes it essential to predict pretraining outcomes (validation loss on hold-out data) before committing to expensive full-scale pretraining. The scaling laws of parameters and token sizes enable extrapolation from small-scale pilots but miss other pretraining feature-dependent variations (i.e., hyperparameters and data states), while purely data-driven methods require substantial training data and extrapolate unreliably beyond observed regimes. To address this dual limitation, we propose ReScaL , a hybrid method that diversely fits an ensemble of scaling laws on minimal pilot runs; learns ML-predicted residual corrections to capture pretraining feature-dependent variations that scaling laws miss; and dynamically combines scaling law extrapolation with residual corrections to leverage the strengths of both. Through experiments on 52 pairs of pretraining datasets and budget sizes, we show that ReScaL is ranked 1st in 41/52 cases (79%) with the best average rank of 1.21, and attains up to 5.7 lower MAPE than the strongest counterpart. Ablations show that removing each design can degrade accuracy by up to 4.3, while sensitivity analysis confirms our hyperparameter choices as robust operating points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM pretraining, scaling laws, residual learning, performance modeling, extrapolation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_xtSfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Traditional scaling laws struggle to capture how training configurations beyond scale variables affect validation loss, while purely data-driven methods can model these effects but often extrapolate poorly beyond the observed range. To address the limitations of existing scaling laws, this paper introduces ReScaL, a method for predicting large-scale LLM pretraining validation loss from small-scale pilot runs. ReScaL combines nine fitted variants of three scaling-law families, uses a random forest to model residuals associated with additional pretraining features, and adaptively weights the residual corrections at the pilot-set level. The evaluation covers eight datasets and 52 dataset–budget combinations, with baseline comparisons, ablation studies, sensitivity analyses, and assessments of configuration ranking and selection. ReScaL achieves the lowest MAPE in 37 combinations and falls within the top Scott–Knott group in 41, with an average group rank of 1.23. This work contributes to scaling-law research by incorporating feature-dependent residual corrections into an ensemble of scaling laws.

Critical strengths
  1. This paper addresses a practical problem with a well-motivated design. Predicting expensive pretraining outcomes from limited pilot runs can reduce the cost of selecting training configurations.
  2. The evaluation is broad and provides evidence of both predictive accuracy and practical utility. The experiments cover eight datasets and multiple pilot budgets, with comparisons against individual scaling laws, random forests, and hybrid predictors. ReScaL achieves the lowest MAPE in 37 of 52 settings, and the evaluation also examines configuration ranking and selection. Together with the ablation results, these findings support benefits that extend beyond a single dataset or evaluation metric.
Critical weaknesses
  1. The experimental protocol leaves several details unclear, limiting reproducibility and interpretation. Dense4 contains only three scale configurations, yet results are reported for rho=0.1, which would yield zero training configurations under the stated floor-based selection rule. The paper also does not clearly explain whether the best run at each scale is selected before or within five-fold cross-validation, or how cross-validation is handled when very few scale configurations are available. These issues make the evaluation difficult to reproduce and weaken confidence in the extrapolation results.
  2. The training targets of the two auxiliary random forests are not explicitly specified, nor is it clear whether the RSS values are computed from in-sample, out-of-bag, or cross-validated predictions.
  3. Some reported results also require reconciliation: averaging the Dense1 ReScaL results in Table 2 equally across budgets gives approximately 0.02006, whereas Table 4 reports 0.0215. The averaging procedure used in Figure 3 should likewise be stated explicitly. Since adaptive weighting is presented as a central contribution, these details matter for assessing both its reliability and the size of its benefit.
  4. The presentation could be improved in terms of clarity and consistency. The abstract uses MAPE without defining the acronym and reports an average rank of 1.21, whereas the introduction and Table 2 report 1.23. In Figure 1, the right-hand panel labels the vertical axis “Accuracy” without defining the metric or explaining its relationship to the validation-loss prediction task. Several passages also contain grammatical errors and ambiguous phrasing. Addressing these issues would make the method and its supporting evidence easier to follow and assess.
Questions for authors
  1. Please clarify how a nonempty training set is obtained for Dense4 at rho=0.1.
  2. How is the residual correction weight estimated? Do the two auxiliary random forests predict validation loss y or residuals r? Are their RSS values computed from training-set, out-of-bag, or cross-validated predictions?
  3. Could you clarify the aggregation procedures and reconcile the differences across tables? Do Tables 2 and 4 use the same full-model runs and aggregation weights? Are the metrics in Table 3 and Figure 3 averaged over individual dataset–budget cases or first aggregated within each dataset? Which methods or parameter settings are compared when computing the reported ranks?
4 comments

Response to Reviewer xtSf

Thank you for your careful review and for recognizing the motivation of our design and the evidence for predictive accuracy and practical utility across datasets and metrics. We appreciate your close examination of the experimental protocol, result aggregation, and presentation; these comments help us improve reproducibility, consistency, and clarity.

Q1 — Nonempty DENSE4 pilot set at ρ = 0.1

Thank you for raising these concerns. We overlooked the minimum-one-scale safeguard in our description of pilot selection. In the implementation, at least one scale setting is selected even when ⌊ρ∣S∣⌋=0\lfloor\rho|\mathcal S|\rfloor=0. Consequently, DENSE4 at ρ=0.1\rho=0.1 uses its cheapest scale setting, containing 1,096 pilot records, as already reported in Supplementary Table 6 (Appendix B.2). We will make the minimum-one-scale rule explicit in the paper.

Q2 — Auxiliary random forest targets and residual correction weight

Thank you for requesting this clarification.

  • Auxiliary random forest targets. Both auxiliary random forests predict the original validation loss y. Their purpose is to estimate the additional predictive value of configuration features under extrapolation for determining α.

  • RSS estimation. Both RSS values are computed from predictions on an internal expensive pilot holdout, rather than training-set, out-of-bag, or cross-validated predictions.

    • The intended extrapolation-validation procedure holds out the most expensive 20% of pilot records, ranked by ND, and fits both auxiliary random forests on the remaining, cheaper pilot records.
    • RSS_ND and RSS_ALL denote the residual sums of squares on this same expensive holdout for the auxiliary models using only (N, D) and using (N, D) together with configuration features x, respectively.
    • These held-out errors assess whether the additional features remain predictive at larger scales and are used to calculate partial R² below.

    For example, on DENSE2 at ρ = 0.7, the auxiliary models are fitted on the 1,020 cheaper records among the 1,274 pilot records, while the 254 most expensive records are held out for computing both RSS values.

  • Residual correction weight. We calculate partial R² as max(1 − RSS_ALL / RSS_ND, 0) and map it to the extrapolation weight through α = σ(s · (partial R² − 0.5)). The target test samples are excluded from this estimation.

We will explicitly specify the auxiliary models' training targets, the internal holdout, and the RSS calculation in the revised paper.

Q3 — Aggregation across tables and figures, and rank calculation

Thank you for checking these results carefully. Our earlier experiments used a finer pilot-budget step of 0.05, yielding 19 budget settings for DENSE1. To fit the available space, the paper reports a subset with a step of 0.1. The ablation summary retained results from the earlier budget grid, which was not consistently reflected in the presentation. We will correct this inconsistency by aligning the summaries and full-model results in main-paper Tables 2 and 4 with the same reported budget settings.

For main-paper Table 3 (Section 4.3) and Figure 3 (Section 4.5), the aggregate metrics are calculated with equal weight across the 52 dataset–budget cases. In Table 3, ranks compare the prediction methods within each case. In Figure 3, ranks compare the evaluated (t, s) combinations within each case, and mRank, mMAPE, and mSD average the corresponding case-level values. We will make these aggregation rules and comparison sets explicit in the captions.

W1 — Reproducibility of the split and cross-validation protocol

Please refer to Q1 for the minimum-one-scale rule and the DENSE4 example. We clarify the selection of records for scaling-law fitting and the five-fold cross-validation procedure below.

We select the reference records before five-fold cross-validation. The procedure is as follows.

  • For each distinct scale in the pilot set, retain only the k lowest-loss records (k = 5) to fit the scaling law ensemble.
  • Use the ensemble fitted on these selected records to predict the entire pilot set.
  • Calculate each record's residual by subtracting the predicted loss from the observed loss, capturing how its configuration deviates from the high-performing scale reference.
  • Train the RF on these residuals using all pilot records and their scale and configuration features.

Cross-validation on the selected records determines the ensemble weights, after which the scaling laws are refitted on those records. For example, on DENSE2 at ρ=0.7\rho=0.7, the pilot set contains 1,274 records across 11 distinct (N, D) settings. Selecting five records per scale yields 55 records for scaling-law fitting and cross-validation; the residual RF is then trained using residuals from all 1,274 records. We will clarify this sequence and its rationale in the revised paper.

Five-fold cross-validation partitions records, with each fold used in turn for validation. The selected subsets contain at least five records in every evaluated dataset–budget case, including cases with only one pilot scale. For example, DENSE4 at ρ=0.1\rho=0.1 retains five records from its single pilot scale, allowing each fold to use four records for fitting and one for validation. Thus, five-fold cross-validation remains feasible across all 52 cases. It supports ensemble weighting within the pilot data, while the held-out large-scale targets evaluate extrapolation performance.

The paper provides a link to our anonymous repository containing the main code, experimental scripts, and raw data. We hope that these clarifications, together with the released code, help reproduce and interpret our evaluation.

W2 — Auxiliary random forest targets and RSS estimation

Please refer to Q2 for the auxiliary models' training targets, the internal holdout, and the estimation of RSS and the residual correction weight.

W3 — Inconsistent results and aggregation procedures

Please refer to Q3 for the budget-grid inconsistency and the aggregation and rank-calculation procedures used in the tables and figures.

W4 — Presentation and terminology

Thank you for identifying these presentation issues. We will define MAPE as mean absolute percentage error at its first occurrence in the abstract and correct the average rank from 1.21 to 1.23, consistent with the introduction and main-paper Table 2. We will clarify the metric represented by “Accuracy” in the right-hand panel of main-paper Figure 1 and explain its relationship to validation-loss prediction in the caption. We will also carefully revise the manuscript to correct grammatical errors and ambiguous wording and ensure consistent terminology throughout.

Dear Reviewer xtSf,

As the discussion period is nearing its end, we hope our responses have addressed your concerns. Please let us know if any further clarification would be helpful. We would be happy to answer any additional questions. Thank you again for your time and thoughtful review.

Thank you for the detailed responses. These address my main concerns. After considering the response, I believe my original score still reflects my overall assessment of the paper, so I will keep it unchanged.

Reviewer_ysGGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReScaL, a hybrid method that predicts the validation loss of large LLM pretraining runs from a few cheap pilot runs. It has three core components. First, an ensemble of 9 scaling-law variants based on Kaplan, Chinchilla and Farseer, wo first studied scaling lays in LMs and their corrections. Each run is warm-started from published parameters and fully or partially re-fitted under bounds (±35% for adapted parameters, ±1% for fixed ones). The variants are combined with a temperature softmax over their cross-validated MAPE. Second, a random forest that learns the residual between the true loss and the ensemble prediction, using N, D and extra features (hyperparameters, data-mix ratios, MoE settings, etc.). Third, a gate α that weights the residual when the target falls inside the pilot compute range; otherwise α = $\sigma(s·(R²_partial − 0.5)), where R²_partial measures how much the extra features reduce residual error compared with (N, D) alone.

Validation covers 8 public datasets (6 dense, 2 MoE) and 52 dataset × pilot-budget cases. Pilots are the cheapest scale settings by 6ND; tests are the most expensive Pareto-front settings. ReScaL ranks first in 41/52 cases (Scott-Knott on MAPE), has average rank 1.23, and gets up to 5.7× lower MAPE than the best baseline. It also has the best mean rank on ranking and selection metrics (Spearman, pairwise accuracy, top-1, regret). Different ablations, a sensitivity analysis on t and s, an overhead analysis and a pilot-compute break-even analysis support the design.

Critical strengths

The strengths in my opinion are:

  1. The problem is practical and the method is simple, cheap and modular. It anchors on scaling laws for extrapolation and adds a gated, learned correction. Fitting takes seconds, and the method works with any regressor. This matters because a predictor like this could be adopted easily in real scaling decisions. Scaling law papers often disregard hyperparameters for training and dataset distribution and this paper studies precisely that.

  2. Empirical coverage is broad and the gate behaves sensibly. The method is tested on 8 datasets (dense and MoE), with several pilot budgets, ablations of every component, decision metrics and released code. The gate turns the correction off where the extra features don't help (α ≈ 0 on DENSE1 and DENSE6), and dynamic α beats every fixed α on average (Table 8). This matters because it suggests the hybrid design avoids the usual failure of ML predictors when extrapolating.

Critical weaknesses

Weaknesses in my opinion are:

  1. The evaluation is too thin to support the strength of the claims. Most datasets have a single test scale setting. DENSE1 and DENSE6 have only 5 test samples each, and DENSE3 has 8 (Table 7). With so few targets, Scott-Knott ranks and claims like "rank 1 in 41/52 cases" may not be robust.
  2. The main claim, that the residual captures hyperparameter and data-state effects that scaling laws miss, is not substantiated. The large gains appear on DENSE4/5 and MoE2. There, the features include validation-domain indicators, training-step or progress information, and MoE structure (expert count, router type). These encode which eval set is measured or correct a poorly specified scale variable (total vs. active parameters, tokens seen); they are not hyperparameter effects. Where the features are mainly LR and BS (DENSE1, MoE1), the residual adds little: on DENSE1 the error is unchanged without it and α ≈ 0; on MoE1, top-1 is 0 at every budget. Also, the scaling-law baselines predict one value per (N, D). No feature-importance or feature-group analysis is given (eg: shapely values). This matters because it changes what the paper actually shows. Much of the gain may come from information the baselines were never given.
Questions for authors
  1. Can you report feature importance (e.g., permutation importance or SHAP) or a feature-group ablation for the residual model on DENSE4/5, MoE1 and MoE2? The groups for example could bevalidation-domain indicators, step/progress features, MoE structural features, LR/BS, and data-mix ratios. How much of the gain remains without domain indicators and step features?

  2. Would the scaling-law baselines close the gap if given the same information? For example: fitting per validation domain on DENSE4/5, or using active parameters for MoE2.

  3. When training the residual model, is $ ŷ_{SL}$ computed in-sample or out-of-fold? Is R²_partial computed from in-sample RF errors or from out-of-bag / cross-validated errors? In-sample estimates would make both the residuals and the gate optimistic.

  4. The warm-start parameters come from the original scaling-law papers. DENSE2 comes from the Farseer paper and DENSE6 is a Chinchilla replication. Do these priors overlap with the evaluation data? How does ReScaL perform with priors from unrelated sources?

  5. Given the small test sets, can you report bootstrap confidence intervals, or results under alternative test selections (e.g., the top-k most expensive settings)?

  6. Minor: the abstract reports average rank 1.21, but the introduction and Table 2 report 1.23.

Suggestions for future work to extend and improve the paper (no new experiments required at this stage):

  1. Add uncertainty estimates to the predictions.

  2. Test transfer across architectures and datasets, and compare other residual learners (e.g., gradient boosting).

5 comments

Response to Reviewer ysGG

Thank you for your careful review and for recognizing the practical relevance of the problem, the simplicity and modularity of ReScaL, and the breadth of our evaluation. We appreciate your valuable suggestions on feature contributions, evaluation robustness, and estimation details, which help us strengthen the evidence and clarify the method.

Q1 — Feature-group ablation of the residual model

We conducted additional feature-group ablation experiments on the five datasets where residual correction substantially improves mean prediction accuracy: DENSE2, DENSE4, DENSE5, MoE1, and MoE2. For each ablation, we removed one feature group, retrained the residual model, and re-estimated the gate while keeping the original pilot/test split and fitted scaling-law ensemble unchanged. Please refer to Q2 for the analysis of validation-domain indicators through domain-specific fitting.

Each entry reports the relative change in budget-averaged MAPE compared with full ReScaL, calculated as (MAPE_ablated − MAPE_full) / MAPE_full × 100%. Positive values indicate worse prediction accuracy after removing the group, and negative values indicate an improvement. An em dash denotes a feature group that is not applicable to the dataset.

Feature group removedDENSE2DENSE4DENSE5MoE1MoE2
Training steps and progress—+44.61%+33.71%—+138.20%
Data mixture ratios—+30.58%+16.34%——
Learning rate / batch size+27.91%——−15.04%—
Base architecture−2.34%——+0.22%+0.60%
Scale and resource features———−0.34%−4.93%
MoE structure———+6.29%+85.38%
Training-step budget−4.22%——+14.70%—

Training-step and progress information does improve prediction accuracy, as shown by the increased MAPE when this group is removed. The pretraining hyperparameters central to our study also remain important: removing data mixture ratios increases MAPE by 30.58% on DENSE4 and 16.34% on DENSE5, while removing MoE structural features increases MAPE by 85.38% on MoE2.

Q2 — Comparisons with domain-specific fitting and an alternative MoE scale variable

In response to this comment, we conducted two additional experiments, reported below, to examine whether explicitly accounting for validation domains or changing the parameter-count definition narrows the performance gap.

  • Domain-specific fitting on DENSE4 and DENSE5. For each pilot budget, we fitted and evaluated all methods separately within each validation domain, retaining the original pilot/test scale assignments and all other features, including training progress. Thus, the scaling-law baselines can fit a different loss function for each domain. MAPE is averaged equally across domains within each budget and then equally across budgets.

    DatasetReScaL MAPEEnsemble-only MAPEBest baseline: method / MAPE
    DENSE412.121%11.765%RF / 15.642%
    DENSE516.665%17.024%Chinchilla / 19.375%

    Domain-specific fitting substantially narrows the gap between the scaling-law ensemble and full ReScaL. Nevertheless, both full ReScaL and ensemble-only retain lower mean MAPE than the strongest baseline in this setting. The results indicate that much of the remaining predictive advantage is provided by the scaling-law component with diverse fitting and ensembling.

  • Replacing N on MoE2. We replaced total parameter count with the source authors' dense_parameter_count field and refitted all six methods below. Importantly, this field is the authors' reported dense-model parameter count, rather than a strictly defined per-token active parameter count. or each case, MAPE is averaged over test records and then equally over the five pilot budgets. Ranks are calculated among the six methods by case-level MAPE, with ties assigned average ranks, and then averaged over budgets.

    MethodTotal-parameter N: mean MAPE / mean rankAuthor-reported dense N: mean MAPE / mean rank
    Kaplan7.882% / 2.0016.410% / 3.40
    Chinchilla29.826% / 5.2029.439% / 5.80
    Farseer54.013% / 5.8036.835% / 4.20
    Ensemble-only13.567% / 3.6013.151% / 2.80
    ReScaL5.674% / 1.809.923% / 1.40
    RF15.091% / 2.6015.007% / 3.40

    ReScaL retains the lowest mean MAPE and the best mean rank among the compared methods under both parameter-count definitions, showing that its predictive advantage persists after replacing total parameter count with the authors' dense parameter count.

Q3 — Estimation of scaling-law residuals and the residual gate

Thank you for requesting this clarification. The scaling-law predictions used to construct residual targets and the auxiliary RF predictions used to estimate the gate are obtained through different procedures.

  • Scaling-law predictions and residual targets.

    • For each distinct scale in the pilot set, retain only the k lowest-loss records (k = 5) to fit the scaling law ensemble.
    • Use the ensemble fitted on these selected records to predict the entire pilot set.
    • Calculate each record's residual by subtracting the predicted loss from the observed loss, capturing how its configuration deviates from the high-performing scale reference.
    • Train the RF on these residuals using all pilot records and their scale and configuration features.

    For example, on DENSE2 at ρ=0.7\rho=0.7, the pilot set contains 1,274 records spanning 11 distinct (N, D) settings. Five records per setting are selected, yielding 55 records for scaling-law fitting. The fitted ensemble then predicts all 1,274 records, and their residuals are used to train the RF with inputs (N,D,x)(N,D,\mathbf{x}). Within a scale setting, the scaling-law prediction is identical, but the observed losses and residuals can differ across configurations.

  • Partial R² and the gate. Under the intended procedure, partial R² is computed from prediction errors on an internal holdout comprising the most expensive 20% of pilot records, ranked by ND, rather than from in-sample, out-of-bag, or cross-validated RF predictions. The gate is intended to assess the additional predictive value of features beyond (N, D) when extrapolating to larger-scale runs. We estimate this value using an internal holdout drawn exclusively from the pilot set.

    For example, on DENSE2 at ρ=0.7\rho=0.7, this split would use the 254 most expensive pilot records for validation and the remaining 1,020 records for training. Under this procedure, both RSS values are computed from held-out predictions.

Overall, when training the residual model, residuals are computed neither entirely in-sample nor through out-of-fold predictions. Instead, the scaling-law ensemble is fitted on the lowest-loss records at each scale and then used to construct residuals for the entire pilot set. The partial R² is calculated from validation errors on a held-out subset of larger-scale observations within the pilot set.

Q4 — Overlap between warm-start priors and the evaluation data

Following the evaluation setting of the cited scaling-law studies, we fit models using smaller-scale pilot runs and evaluate them on larger-scale targets. In every dataset–budget case, the test samples are excluded from our pilot-based fitting, including scaling-law parameter adaptation, ensemble weighting, residual learning, and gate estimation. We have also verified that these test samples were not used to fit the published parameters adopted as historical initializations. Thus, the test samples were used neither in the original estimation of these priors nor in our subsequent model fitting.

For each scaling-law family, ReScaL's ensemble variants and the corresponding scaling-law baseline start from the same publicly available parameter values. Thus, both have access to the same historical initialization; ReScaL does not receive an initialization unavailable to the corresponding baseline. This comparison controls access to historical priors.

Q5 — Robustness under alternative test selections

Thank you for raising this concern. Our evaluation follows the practical setting of using inexpensive, small-scale pilot runs to predict expensive, large-scale pretraining outcomes. The cost of large-scale pretraining limits the number of available target runs in the public datasets.

Compared with selecting Pareto-frontier settings as test targets, top-k or top-p% splits do not necessarily reflect the practical task of forecasting large-scale pretraining outcomes, because they may include many small-scale models in the test set. Nevertheless, to further assess the robustness of our results to alternative test selections, we conducted additional experiments specifically using the top 10%, 30%, 50%, and 70% most expensive distinct (N, D) settings as test targets. For each protocol, held-out settings are removed from the original pilot set, and all methods are refitted using the remaining pilots. Conditions with insufficient scale settings or no remaining pilots are marked as unavailable. MAPE is first calculated over test records within each case and then averaged equally over valid dataset–budget cases. Mean rank is calculated by ranking methods by MAPE within each case, assigning average ranks to ties, and averaging these ranks over valid cases; it is distinct from the Scott–Knott group rank in the original paper. The lowest-MAPE count is the number of cases in which ReScaL achieves the minimum MAPE, including ties.

Test selectionValid casesReScaL mean MAPEReScaL mean rankCases with lowest MAPEBest fully covered baseline: mean MAPE / mean rank
Top 10%525.89%1.5239/52Kaplan: 12.70% / 4.46
Top 30%525.89%1.5635/52Kaplan: 12.13% / 4.37
Top 50%525.76%1.4637/52Kaplan: 11.53% / 4.15
Top 70%505.57%1.1845/50Kaplan: 11.62% / 4.16

ReScaL achieves a lower mean MAPE than the strongest fully covered baseline under all four reported protocols, achieving the lowest MAPE in 67.31%–90.00% of their valid cases. These results support robustness of the aggregate prediction advantage.

W1 — Small test sets and robustness of the reported ranks

Please refer to our response to Q5, where we discuss the limited number of independent test scales and report additional results using the top 10%, 30%, 50%, and 70% most expensive scale settings as test sets.

W2 — What the residual model learns from additional features

Please refer to Q1 for the additional feature-group ablations and Q2 for the additional comparisons using domain-specific fitting and an alternative MoE parameter-count definition. These experiments address the sources of residual gains and the information available to the baselines.

Comments on the paper

6 comments

Response to Reviewer XMAV

Thank you for your thoughtful review and for recognizing the practical importance of the problem, the rationale for combining scaling laws with configuration-dependent corrections, and the complementary evaluation metrics. Your valuable comments on extrapolation, residual weighting, simpler alternatives, and pilot costs help us clarify the scope and supporting evidence of our claims.

W1 — Extrapolation distances and independent test scales

Thank you for this valuable suggestion. We will report the extrapolation distance for every pilot budget alongside its prediction error, making the extent of extrapolation explicit.

  • Pilot budgets and extrapolation distance.

    Our evaluation follows the setting of the cited scaling-law studies, using inexpensive, small-scale pilot runs to predict expensive, large-scale pretraining outcomes. The main paper already evaluates these pilot budgets in Table 2 (Section 4.3), keeping the test targets fixed within each dataset while varying the pilot budget. This evaluates whether ReScaL remains effective with fewer observed scales and longer extrapolation distances. For example, on DENSE1, reducing the pilot budget from ρ=0.9\rho=0.9 to ρ=0.1\rho=0.1 increases the maximum compute extrapolation factor from 15.33 to 3,925. The budget experiments therefore already cover different extrapolation distances, while also varying the amount of pilot data.

  • Explicit extrapolation factors.

    For each budget, we define the maximum extrapolation factor as the largest test scale divided by the largest pilot scale, calculated separately for parameter count N, token count D, and the compute proxy C = 6ND. As an additional analysis for this rebuttal, we calculated these factors from the original experimental splits; the table below summarizes their ranges across budgets. In the revised paper, we will additionally report the pilot/test boundaries and factors for each budget next to its prediction errors, as requested.

    DatasetBudgetsMaximum N extrapolation factorMaximum D extrapolation factorMaximum C extrapolation factor
    DENSE193.944–37.161.189–6415.33–3925
    DENSE251–5.0021.25–252.845–71.15
    DENSE391–104.81.5–6008.967–4.665e+04
    DENSE422.562–5.85712.562–5.857
    DENSE542.439–14.291.3673.333–19.52
    DENSE691–2.2641.25–13.028.227–259.6
    MoE191–1.0025931–101.002593–10.03
    MoE2510.44–3610110.44–3610
  • Additional test-scale evaluations.

    The cost of large-scale pretraining limits the number of available target runs. To assess sensitivity to test-target selection, we conducted additional experiments for this rebuttal by expanding the test sets to include the top 10%, 30%, 50%, and 70% most expensive distinct (N, D) settings, ranked by 6ND. In these additional experiments, selected test settings are removed from the original pilot set, and all methods are refitted on the remaining pilots. These splits can include many smaller-scale targets, so their practical relevance to forecasting large-scale pretraining outcomes varies.

    MAPE and numerical MAPE ranks are averaged equally across valid dataset–budget cases, with average ranks for ties. Lowest-MAPE counts include ties. The baseline shown has the lowest mean MAPE among those covering every valid case.

    Test selectionValid casesReScaL mean MAPEReScaL mean rankCases with lowest MAPEBest fully covered baseline: mean MAPE / mean rank
    Top 10%525.89%1.5239/52Kaplan: 12.70% / 4.46
    Top 30%525.89%1.5635/52Kaplan: 12.13% / 4.37
    Top 50%525.76%1.4637/52Kaplan: 11.53% / 4.15
    Top 70%505.57%1.1845/50Kaplan: 11.62% / 4.16

    ReScaL achieves lower mean MAPE than the strongest fully covered baseline under all four test selections, with the lowest MAPE in 67.31%–90.00% of valid cases.

  • Interpretation of the 52 cases.

    We will also clarify that the original 52 cases share pilot data and test targets across budgets, so their aggregate ranks summarize performance across budget conditions within the eight datasets.

W2 — Reliability of the residual gate at unseen scales

  • Transferability of residual corrections.

    Predictability on small-scale observations does not establish that residual corrections will transfer to larger models or longer training runs. This is precisely why we introduce dynamic α to regulate the contribution of residual correction and mitigate this risk. In practice, the target model's loss is unknown, so we cannot directly determine whether hyperparameter effects observed at small scales will persist at larger scales. We therefore hold out expensive observations within the pilot set for an internal extrapolation test, assessing whether the predictive value of additional features at smaller scales transfers to larger-scale observations. This evidence guides the residual weight.

  • RSS estimation.

    Both RSS values used to compute partial R² are evaluated on this internal expensive holdout, using predictions from auxiliary models fitted on the remaining pilot observations. Thus, they are held-out prediction errors, rather than in-sample, out-of-fold, or out-of-bag estimates.

  • Pilot-level weighting.

    Each pilot set yields one extrapolation weight. This provides a practical way to regulate the overall residual contribution using the available pilot observations, while the residual model itself still produces a configuration-dependent correction for each target. The observed ND range supplies the operational distinction between the two weighting regimes. Incorporating target-specific extrapolation distances and configuration support into the gate is a valuable direction for further refinement, especially when configuration effects change with scale. Our current design addresses this risk through pilot-based weighting without requiring the target model's true loss.

  • Existing empirical evidence.

    Although dynamic α does not explicitly incorporate each target's extrapolation distance, the retrospective evaluation already reported in Supplementary Table 8 (Appendix C.1) tests its prediction accuracy across the eight systems and pilot budgets, which cover different extrapolation distances. This existing experiment compares the dynamic gate with fixed residual weights from 0 to 1, keeping the other components unchanged; its published table is reproduced below. These results show that pilot-based dynamic weighting already provides strong prediction accuracy across the evaluated systems and extrapolation settings, supporting its practical effectiveness while leaving room for refinement through target-specific distance information. The table reports system-level averages over budgets.

    SystemDynamicα = 0.0α = 0.1α = 0.2α = 0.3α = 0.4α = 0.5α = 0.6α = 0.7α = 0.8α = 0.9α = 1.0
    Dense10.0220.0220.0220.0220.0230.0230.0240.0250.0260.0270.0280.029
    Dense20.0470.0660.0610.0580.0550.0540.0520.0500.0490.0480.0470.046
    Dense30.0760.0780.0840.0930.1020.1110.1200.1290.1390.1490.1590.169
    Dense40.1730.4660.4190.3720.3250.2800.2340.1890.1470.1360.1400.173
    Dense50.1670.4540.3960.3410.2900.2430.2010.1720.1610.1610.1690.197
    Dense60.0370.0370.0330.0340.0360.0400.0430.0470.0510.0550.0590.064
    MoE10.0180.0260.0240.0230.0220.0210.0210.0210.0220.0230.0250.026
    MoE20.0570.1360.1280.1200.1120.1040.0960.0880.0800.0720.0640.057
    Mean0.07460.16040.14590.13270.12060.10930.09890.09020.08420.08380.08640.0951

Overall, the gate addresses the transfer concern by using an internal extrapolation test to determine how strongly residual corrections should contribute at unseen scales. Its RSS estimates come from held-out observations, and the retrospective evaluation across systems and budgets supports the practical effectiveness of this pilot-based strategy, with lower overall mean MAPE than every globally fixed weight tested. Although a target-specific gate could further account for distance and configuration shifts, the current gate already provides an empirically effective way to moderate residual correction using only the available pilot data.

W3 — Incremental contribution over hybrid methods and historical priors

ReScaL builds on the established idea of combining analytical prediction with learned corrections. Its contribution lies in adapting this combination to extrapolation from limited LLM pilot runs through constrained diverse fitting, a scaling-law ensemble, and a pilot-derived residual weight. For example, NCPL learns configuration-dependent residuals relative to a Chinchilla baseline, whereas ReScaL combines fitted variants of three scaling-law families and adjusts the residual contribution using pilot evidence. These designs address uncertainty in the choice of scaling law and in the usefulness of residual corrections beyond the observed scales.

Both simpler hybrid settings discussed in this comment have already been evaluated in the paper, through the component ablation in main-paper Table 4 (Section 4.4) and the fixed-weight comparison in Supplementary Table 8 (Appendix C.1).

  • Single scaling law with RF residual learning. The w/o ensemble ablation replaces ensemble weighting with selection of the single scaling-law variant having the lowest pilot cross-validation MAPE, while retaining diverse fitting, the RF residual learner, and dynamic α. It therefore examines whether selecting one fitted law is sufficient. The relevant columns of main-paper Table 4 are reproduced below; the per-budget ablation results are provided in Supplementary Table 16 (Appendix C.4). Values are MAPE fractions, with mean and SD reported using the original table's aggregation. Ranks are reproduced from the original component-ablation table, where all five variants are compared; lower ranks indicate better performance.

    SystemReScaL rank / MAPE mean (SD)w/o ensemble rank / MAPE mean (SD)
    Dense11 / 0.0215 (0.01)2 / 0.0291 (0.02)
    Dense21 / 0.0467 (0.08)1 / 0.0452 (0.07)
    Dense31 / 0.0767 (0.06)2 / 0.1473 (0.17)
    Dense41 / 0.1728 (0.14)1 / 0.1731 (0.15)
    Dense51 / 0.1674 (0.14)2 / 0.1965 (0.18)
    Dense61 / 0.0365 (0.04)2 / 0.0607 (0.09)
    MoE11 / 0.0191 (0.01)2 / 0.0265 (0.02)
    MoE21 / 0.059 (0.03)1 / 0.059 (0.03)
    Average rank1.001.63

    The full model has lower reported mean MAPE on six datasets, ties on MoE2 at the reported precision, and has slightly higher MAPE on DENSE2. These results support the benefit of ensembling across the evaluated datasets, while showing that the single-law hybrid can also be competitive.

  • Scaling-law ensemble plus RF with constant α. Supplementary Table 8 (Appendix C.1) already evaluates this setting by replacing the dynamic gate with fixed weights from 0 to 1 while retaining the ensemble and RF. We reproduce it below. The dynamic gate achieves a lower overall mean MAPE than every globally fixed weight evaluated, although particular systems favor different fixed values. This experiment provides a retrospective comparison across fixed weights.

    SystemDynamicα = 0.0α = 0.1α = 0.2α = 0.3α = 0.4α = 0.5α = 0.6α = 0.7α = 0.8α = 0.9α = 1.0
    Dense10.0220.0220.0220.0220.0230.0230.0240.0250.0260.0270.0280.029
    Dense20.0470.0660.0610.0580.0550.0540.0520.0500.0490.0480.0470.046
    Dense30.0760.0780.0840.0930.1020.1110.1200.1290.1390.1490.1590.169
    Dense40.1730.4660.4190.3720.3250.2800.2340.1890.1470.1360.1400.173
    Dense50.1670.4540.3960.3410.2900.2430.2010.1720.1610.1610.1690.197
    Dense60.0370.0370.0330.0340.0360.0400.0430.0470.0510.0550.0590.064
    MoE10.0180.0260.0240.0230.0220.0210.0210.0210.0220.0230.0250.026
    MoE20.0570.1360.1280.1200.1120.1040.0960.0880.0800.0720.0640.057
    Mean0.07460.16040.14590.13270.12060.10930.09890.09020.08420.08380.08640.0951
  • Historical parameters and evaluation overlap. All historical parameters used in our experiments are publicly available from the cited scaling-law studies. We have verified that the test samples were excluded from the data used to estimate these historical parameters, as well as from our subsequent pilot-based fitting. Thus, there is no overlap between the test samples and the fitting data used to obtain the adopted priors. For a fair comparison, every corresponding scaling-law baseline uses the same historical parameter values as ReScaL's variants of that family for initialization. The comparisons therefore give ReScaL and the scaling-law baselines equal access to historical priors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.