- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Traditional scaling laws struggle to capture how training configurations beyond scale variables affect validation loss, while purely data-driven methods can model these effects but often extrapolate poorly beyond the observed range. To address the limitations of existing scaling laws, this paper introduces ReScaL, a method for predicting large-scale LLM pretraining validation loss from small-scale pilot runs. ReScaL combines nine fitted variants of three scaling-law families, uses a random forest to model residuals associated with additional pretraining features, and adaptively weights the residual corrections at the pilot-set level. The evaluation covers eight datasets and 52 dataset–budget combinations, with baseline comparisons, ablation studies, sensitivity analyses, and assessments of configuration ranking and selection. ReScaL achieves the lowest MAPE in 37 combinations and falls within the top Scott–Knott group in 41, with an average group rank of 1.23. This work contributes to scaling-law research by incorporating feature-dependent residual corrections into an ensemble of scaling laws.
Critical strengths
- This paper addresses a practical problem with a well-motivated design. Predicting expensive pretraining outcomes from limited pilot runs can reduce the cost of selecting training configurations.
- The evaluation is broad and provides evidence of both predictive accuracy and practical utility. The experiments cover eight datasets and multiple pilot budgets, with comparisons against individual scaling laws, random forests, and hybrid predictors. ReScaL achieves the lowest MAPE in 37 of 52 settings, and the evaluation also examines configuration ranking and selection. Together with the ablation results, these findings support benefits that extend beyond a single dataset or evaluation metric.
Critical weaknesses
- The experimental protocol leaves several details unclear, limiting reproducibility and interpretation. Dense4 contains only three scale configurations, yet results are reported for rho=0.1, which would yield zero training configurations under the stated floor-based selection rule. The paper also does not clearly explain whether the best run at each scale is selected before or within five-fold cross-validation, or how cross-validation is handled when very few scale configurations are available. These issues make the evaluation difficult to reproduce and weaken confidence in the extrapolation results.
- The training targets of the two auxiliary random forests are not explicitly specified, nor is it clear whether the RSS values are computed from in-sample, out-of-bag, or cross-validated predictions.
- Some reported results also require reconciliation: averaging the Dense1 ReScaL results in Table 2 equally across budgets gives approximately 0.02006, whereas Table 4 reports 0.0215. The averaging procedure used in Figure 3 should likewise be stated explicitly. Since adaptive weighting is presented as a central contribution, these details matter for assessing both its reliability and the size of its benefit.
- The presentation could be improved in terms of clarity and consistency. The abstract uses MAPE without defining the acronym and reports an average rank of 1.21, whereas the introduction and Table 2 report 1.23. In Figure 1, the right-hand panel labels the vertical axis “Accuracy” without defining the metric or explaining its relationship to the validation-loss prediction task. Several passages also contain grammatical errors and ambiguous phrasing. Addressing these issues would make the method and its supporting evidence easier to follow and assess.
Questions for authors
- Please clarify how a nonempty training set is obtained for Dense4 at rho=0.1.
- How is the residual correction weight estimated? Do the two auxiliary random forests predict validation loss y or residuals r? Are their RSS values computed from training-set, out-of-bag, or cross-validated predictions?
- Could you clarify the aggregation procedures and reconcile the differences across tables? Do Tables 2 and 4 use the same full-model runs and aggregation weights? Are the metrics in Table 3 and Figure 3 averaged over individual dataset–budget cases or first aggregated within each dataset? Which methods or parameter settings are compared when computing the reported ranks?
4 comments
W1 — Reproducibility of the split and cross-validation protocol
Please refer to Q1 for the minimum-one-scale rule and the DENSE4 example. We clarify the selection of records for scaling-law fitting and the five-fold cross-validation procedure below.
We select the reference records before five-fold cross-validation. The procedure is as follows.
- For each distinct scale in the pilot set, retain only the k lowest-loss records (k = 5) to fit the scaling law ensemble.
- Use the ensemble fitted on these selected records to predict the entire pilot set.
- Calculate each record's residual by subtracting the predicted loss from the observed loss, capturing how its configuration deviates from the high-performing scale reference.
- Train the RF on these residuals using all pilot records and their scale and configuration features.
Cross-validation on the selected records determines the ensemble weights, after which the scaling laws are refitted on those records. For example, on DENSE2 at , the pilot set contains 1,274 records across 11 distinct (N, D) settings. Selecting five records per scale yields 55 records for scaling-law fitting and cross-validation; the residual RF is then trained using residuals from all 1,274 records. We will clarify this sequence and its rationale in the revised paper.
Five-fold cross-validation partitions records, with each fold used in turn for validation. The selected subsets contain at least five records in every evaluated dataset–budget case, including cases with only one pilot scale. For example, DENSE4 at retains five records from its single pilot scale, allowing each fold to use four records for fitting and one for validation. Thus, five-fold cross-validation remains feasible across all 52 cases. It supports ensemble weighting within the pilot data, while the held-out large-scale targets evaluate extrapolation performance.
The paper provides a link to our anonymous repository containing the main code, experimental scripts, and raw data. We hope that these clarifications, together with the released code, help reproduce and interpret our evaluation.
W2 — Auxiliary random forest targets and RSS estimation
Please refer to Q2 for the auxiliary models' training targets, the internal holdout, and the estimation of RSS and the residual correction weight.
W3 — Inconsistent results and aggregation procedures
Please refer to Q3 for the budget-grid inconsistency and the aggregation and rank-calculation procedures used in the tables and figures.
W4 — Presentation and terminology
Thank you for identifying these presentation issues. We will define MAPE as mean absolute percentage error at its first occurrence in the abstract and correct the average rank from 1.21 to 1.23, consistent with the introduction and main-paper Table 2. We will clarify the metric represented by “Accuracy” in the right-hand panel of main-paper Figure 1 and explain its relationship to validation-loss prediction in the caption. We will also carefully revise the manuscript to correct grammatical errors and ambiguous wording and ensure consistent terminology throughout.
Dear Reviewer xtSf,
As the discussion period is nearing its end, we hope our responses have addressed your concerns. Please let us know if any further clarification would be helpful. We would be happy to answer any additional questions. Thank you again for your time and thoughtful review.
Thank you for the detailed responses. These address my main concerns. After considering the response, I believe my original score still reflects my overall assessment of the paper, so I will keep it unchanged.
Response to Reviewer xtSf
Thank you for your careful review and for recognizing the motivation of our design and the evidence for predictive accuracy and practical utility across datasets and metrics. We appreciate your close examination of the experimental protocol, result aggregation, and presentation; these comments help us improve reproducibility, consistency, and clarity.
Q1 — Nonempty DENSE4 pilot set at ρ = 0.1
Thank you for raising these concerns. We overlooked the minimum-one-scale safeguard in our description of pilot selection. In the implementation, at least one scale setting is selected even when ⌊ρ∣S∣⌋=0. Consequently, DENSE4 at ρ=0.1 uses its cheapest scale setting, containing 1,096 pilot records, as already reported in Supplementary Table 6 (Appendix B.2). We will make the minimum-one-scale rule explicit in the paper.
Q2 — Auxiliary random forest targets and residual correction weight
Thank you for requesting this clarification.
Auxiliary random forest targets. Both auxiliary random forests predict the original validation loss y. Their purpose is to estimate the additional predictive value of configuration features under extrapolation for determining α.
RSS estimation. Both RSS values are computed from predictions on an internal expensive pilot holdout, rather than training-set, out-of-bag, or cross-validated predictions.
For example, on DENSE2 at ρ = 0.7, the auxiliary models are fitted on the 1,020 cheaper records among the 1,274 pilot records, while the 254 most expensive records are held out for computing both RSS values.
Residual correction weight. We calculate partial R² as max(1 − RSS_ALL / RSS_ND, 0) and map it to the extrapolation weight through α = σ(s · (partial R² − 0.5)). The target test samples are excluded from this estimation.
We will explicitly specify the auxiliary models' training targets, the internal holdout, and the RSS calculation in the revised paper.
Q3 — Aggregation across tables and figures, and rank calculation
Thank you for checking these results carefully. Our earlier experiments used a finer pilot-budget step of 0.05, yielding 19 budget settings for DENSE1. To fit the available space, the paper reports a subset with a step of 0.1. The ablation summary retained results from the earlier budget grid, which was not consistently reflected in the presentation. We will correct this inconsistency by aligning the summaries and full-model results in main-paper Tables 2 and 4 with the same reported budget settings.
For main-paper Table 3 (Section 4.3) and Figure 3 (Section 4.5), the aggregate metrics are calculated with equal weight across the 52 dataset–budget cases. In Table 3, ranks compare the prediction methods within each case. In Figure 3, ranks compare the evaluated (t, s) combinations within each case, and mRank, mMAPE, and mSD average the corresponding case-level values. We will make these aggregation rules and comparison sets explicit in the captions.