acceptodds
Under review as a conference paper at ICLR 2027

Hinted Scaling Laws

Abstract

Downstream scaling laws let developers predict benchmark performance at large scale from small, cheap proxies. This prediction fails when the small proxies score zero because their accuracies give no signal to extrapolate to larger scale. We introduce hinted scaling laws, which use privileged information to turn a benchmark into a continuous sequence of easier versions so that even the small proxies score above zero. We find that accuracy rises smoothly and predictably with the hint level, enabling new types of scaling laws that operate below the noise threshold. Within pretrained and post-trained model families, hinted scaling laws fit with the smallest models reduce the median forecasting error for larger models from 21 percentage points for a standard scaling law to 6. Hinting also improves the sample efficiency of model ranking. It widens the gaps in accuracy between models while approximately preserving their order, so a single hinted rollout per problem ranks near-indistinguishable models as accurately as 5 unhinted rollouts. Our results suggest that hinting is a general primitive for improving the accuracy and cost of downstream scaling laws and other prediction tasks in model development.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Scaling Laws, Benchmarks, Downstream performance prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_i2kvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a variant of scaling law measurement, Hinted Scaling Law, which fits to the benchmark accuracies. To avoid the scarce accuracy scores on small-scale and uncapable models, the authors propose to include the "hints" by exposing part of answer tokens or teacher's knowledge during accuracy evaluation stage. Applying various levels of hints, the authors manage to fit a downstream accuracy scaling curves, and also apply this method to rank models on benchmark performance.

Critical strengths

(1) The paper addresses the scarcity of benchmark accuracy and directly fit a scaling law to the downstream accuracy scores using various types of induced Hints. They show that the hinted accuracy is predicable from hinted level and model scale.

(2) The Hinted accuracy works well and improves efficiency at ranking models on benchmark performance.

Critical weaknesses

(1) The test loss or perplexity on the targeted benchmarks can also be the continuous proxy of downstream performance, can we alternatively apply it to fit the scaling law or rank models?

(2) What implications we can draw from the scaling curves across different hinted levels?

Questions for authors

As stated in Weaknesses

3 comments

We thank the reviewer for the thoughtful feedback. We have run additional experiments and uploaded a revised PDF with improved results (see also the general response). Among other changes, a hinted scaling law fit on Delphi models of at most 102010^{20} FLOPs now forecasts the GSM8K accuracy of the 102310^{23} FLOPs model to within 3 percentage points, and all forecast errors come with bootstrap intervals. In response to the reviewer's first comment, the main text now also points to our loss-based forecasting baseline. Following are our responses to the reviewer’s comments.


(1) The test loss or perplexity on the targeted benchmarks can also be the continuous proxy of downstream performance, can we alternatively apply it to fit the scaling law or rank models?

We thank the reviewer for raising this. The paper tests both uses of a loss-based proxy and neither works in our setting, but the forecasting experiment was only in the appendix and easy to miss.

  • For forecasting, a two-stage law that maps compute to bits-per-byte on the benchmark's solutions and then bits-per-byte to accuracy predicts an essentially flat line at zero on the Delphi ladder with an error of 16–24 accuracy points (Appendix E.1, Figure 16).
  • For ranking, ordering models by their loss on the reference solutions reaches a Kendall's τ\tau of only 0.33 on AIME and −0.05-0.05 on HLE, below chance (Section 5), whereas a single hinted rollout per problem already reaches τ≈0.5\tau \approx 0.5 on both benchmarks (Figure 7).

Action. We now point to the loss-based forecasting baseline in the main text (Section 4.1), next to our other comparisons.

(2) What implications we can draw from the scaling curves across different hinted levels?

We interpret the question as asking what the scaling curves at different hint levels (Figure 2, right) imply, and see the two implications below. Please let us know if we misunderstood and we can provide more details.

What the curves show. Accuracy varies smoothly with both hint level and model scale (Figure 2). Raising the hint level shifts the scaling curve towards smaller models, with curves rarely crossing, so it moves the point where accuracy starts to rise towards smaller scales (Section 3).

What the curves can be used for. Because hinting reveals this rise at scales where the unhinted benchmark still scores near zero, the hinted curves can be used to forecast benchmark accuracy of larger models (Section 4) and to rank models with fewer rollouts (Section 5).


Thank you again for the review. We hope the revised paper and these answers address both questions, and we are happy to clarify anything else.

Thank you again for your careful review and thoughtful feedback. Have we addressed your concerns in our response or is there anything else we could clarify?

Thanks for the clear response, it address most of my concern! I will keep my positive score.

Reviewer_SDjkOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes continiously hinting parts of the correct response as a scaling axis. Specifically, when small models fail on difficult benchmarks, they find that hinting parts of the answer continiously (through 1) truncating the response 2) masking out parts of the response and 3) leveraging probabilities of a teacher model conditioned on parts of the response) can help reliably predict model performance as a function of C (compute used to train the model) and H (hint level). After they show scaling, they a) propose a functional form and extrapolate to larger models, and b) show that hinting helps with sample-efficient ranking.

Critical strengths
  • One open problem in scaling LMs is that smaller models perform near random on more complicated benchmarks that are not saturated by larger models, which means that it’s impossible to predict model performance at scale on these tasks, and this paper proposes a strategy for studying the scaling behaviour of models on benchmarks / capabilities that smaller models fail on.
  • The paper shows that hinting strategies continiously improves model performance. This is interesting empirical observation, because now we can have less capable models (lower C), given stronger hints (higher H), predict how larger scale models would perform on the benchmark of interest (at H=0).
  • The paper proposes a scaling law with H and C, that extrapolates to predicting the performance of larger models on more difficult math benchmarks that perform near random on smaller models.
  • Results are clear and presented well.
Critical weaknesses
  • In section 4, it’s mentioned that some of the model, hint pairs are removed (e.g. due to all models in the ladder scoring zero on the H=0 benchmark). However, this is one of the most interesting setups for the hinted scaling laws proposed. Specifically, we might have a set of small models that completely fail on the benchmark of interest, but we want to approximate how well a scaled up model might perform, and whether the capability might emerge at some point.
    • I noticed that some of these excluded experiments are in Appendix E. It seems the Qwen2.5 and Gemma 3 results on HLE suggest that hinted scaling laws can overestimate the performance of larger models in settings where their actual performance remains very low. In these cases, the unhinted baseline can actually be more accurate. I think these experiments should be included in the main quantitative evaluation. Otherwise, the reported improvement in forecasting accuracy might not fully reflect how well the method performs across different settings, especially those where a capability does not emerge.
  • The functional form for accuracy scaling with C and H in section 4 seems reasonable, but I wonder if any alternatives were tested. I think the asymptotes of this law make sense, e.g. at H=0, accuracy is a sigmoidal power law with compute, and if compute is fixed, then it’s a sigmoidal power law with the hint level. However, this also assumes that the contributions of C and H are additive (for the logits), which implicitly assumes that increasing the hint level has the same effect on models of different scales. It would have been reasonable to include alternative functional forms, thatg allow for interactions between C and H. e.g. a simple extension is adding a C×HC \times H term.
  • Qwen3’s smaller models were trained with strong-to-weak distillation (according to the technical report), which includes fine-tuning them to match the outputs of a teacher model (either 32B or 235B param models). In this paper, Qwen3-32B is the held-out model on HLE, so the small models used to make the prediction were trained to copy the model they are predicting. This might be a mismatch from a realistic setup where smaller models are not distilled down.
  • None of the forecasting results are reported with uncertainty quantification, while a few models are used to fit the proposed laws.
Questions for authors
  1. What are some limitations of this work / the hinting strategy? Which setups would you expect this to fail?
  2. Are there any guidelines on how to fit the hinted laws? e.g. what hint levels should be evaluated?
  3. Why is parameter count used as the compute scale for the post-trained models? (and not the compute)
  4. Doesn’t filtering for HLE questions that are answerable by frontier models cause some form of a selection bias?
5 comments

We thank the reviewer for the insightful and constructive feedback. We are encouraged that they find the observation that hinting continuously improves model performance interesting and that they find our results clear and well presented. We have run additional experiments and uploaded a revised PDF that addresses all of the reviewer's comments (see also the general response). In summary:

  • All 30 panels. We now report results over all 30 panels in the main text (the hinting ensemble beats the baseline in 22) and acknowledge where hinting overestimates.
  • Alternative functional form. We tested the joint law with the interaction term between model scale and hint level that the reviewer suggested.
  • Uncertainty. All forecast errors now come with bootstrap intervals.
  • Limitations. The discussion now covers distilled model families and overestimation when a capability does not appear within a family, and Appendix A states the selection effect of our HLE subset.

Following are our responses to the reviewer’s comments.


In section 4, it’s mentioned that some of the model, hint pairs are removed (e.g. due to all models in the ladder scoring zero on the H=0 benchmark). However, this is one of the most interesting setups for the hinted scaling laws proposed. Specifically, we might have a set of small models that completely fail on the benchmark of interest, but we want to approximate how well a scaled up model might perform, and whether the capability might emerge at some point.

I noticed that some of these excluded experiments are in Appendix E. It seems the Qwen2.5 and Gemma 3 results on HLE suggest that hinted scaling laws can overestimate the performance of larger models in settings where their actual performance remains very low. In these cases, the unhinted baseline can actually be more accurate. I think these experiments should be included in the main quantitative evaluation. Otherwise, the reported improvement in forecasting accuracy might not fully reflect how well the method performs across different settings, especially those where a capability does not emerge.

We thank the reviewer for raising this point, and we agree that the setting where every small model scores near zero is one of the most interesting ones. Our main-text results do include this setting. On the Delphi ladder, the ten smallest compute budgets score between 0.5% and 2.1% on GSM8K, while the largest model scores 25.9%. Fit only on models that score at most 2.1% (training sets of seven to ten compute budgets), the hinting ensemble forecasts the largest model with an error of 1.7 to 7.4 percentage points, against 19 to 22 for the baseline (Figure 4). So when the capability does appear within the ladder, hinting predicts it from models that all score near zero.

The reviewer is right that on HLE + Qwen2.5 and HLE + Gemma-3 (Figure 13, Appendix E), the baseline forecasts the largest model more accurately: the hinting ensemble overestimates it in all seven panels, by up to 9.6 percentage points. We agree this is a real weakness of our results, since our hinted forecasts misjudge where accuracy starts to rise in these families. As a side note, truncation alone is more robust: it stays within 4.2 percentage points of the target in all seven of these panels (median error 1.7 against 2.7 for the baseline), beats the baseline in 12 of the 14 post-trained main-text panels, and is the method that ranks models best (Section 5).

We left these panels out of the main-text figures because every member of both families scores near guessing performance (the largest members, Qwen2.5 32B and Gemma-3 27B, both score 2.7%). The target model therefore sits below the noise threshold, so a forecast that stays flat at zero is correct by default. If these families were extended to larger members, accuracy should eventually rise above zero, and a flat baseline could then have a larger error than the hinted prediction. How large depends on whether accuracy starts to rise at a model scale 2x or 2000x larger than the largest member here, which these families cannot tell us. These panels therefore show that hinting can overestimate, but they cannot show which method better predicts the model scale at which accuracy rises above zero. Following the reviewer's suggestion, we now report the results over all 30 panels in the main text: the hinting ensemble beats the baseline in 22 of them, with a median error of 5.9 percentage points against 12.6.

Action. We added the forecasting results over all 30 panels to Section 4, where we also acknowledge that the hinting ensemble overestimates accuracy on these two families, and we added this failure mode to the limitations in the discussion.

The functional form for accuracy scaling with C and H in section 4 seems reasonable, but I wonder if any alternatives were tested. I think the asymptotes of this law make sense, e.g. at H=0, accuracy is a sigmoidal power law with compute, and if compute is fixed, then it’s a sigmoidal power law with the hint level. However, this also assumes that the contributions of C and H are additive (for the logits), which implicitly assumes that increasing the hint level has the same effect on models of different scales. It would have been reasonable to include alternative functional forms, thatg allow for interactions between C and H. e.g. a simple extension is adding a term.

We thank the reviewer for this suggestion! The reviewer is right that the joint law (Eq. 2) is additive in CC and HH, so it assumes that raising the hint level has the same effect on models of every scale. We chose this form for two reasons. First, each forecast is fit on as few as two to five models, so we prefer a law with few parameters that is less likely to overfit. Second, the iso-accuracy contours of Figure 3 suggest the assumption is roughly true: for masking and truncation on both Delphi and Qwen3.5, they run as parallel diagonals. It is not exact, which is why we note in Section 4 that it is more reasonable for some family and method pairs than for others, and compare the contours the joint law implies with the empirical ones in Appendix E (Figure 14).

Following the reviewer's suggestion, we tested an interaction directly by adding a cross term,

L(C,H)=max⁡ ⁣(F,  L+(A−L) σ(β1C+β2H+β4 CH+β3)),L(C, H) = \max\!\big(F,\; L + (A - L)\,\sigma(\beta_1 C + \beta_2 H + \beta_4\, C H + \beta_3)\big),

so that the scale coefficient β1+β4H\beta_1 + \beta_4 H can change with the hint level. We fit two versions: one where β4\beta_4 is unconstrained, and one constrained so that predicted accuracy still increases with both model scale and hint level over the training models and the forecast target. Everything else is as in the revised paper: the same data, loss, panels and ensemble of masking and truncation. Over the 19 main-text panels (errors in percentage points):

Joint lawMedian errorLargest errorBeats baselineBeats additive law
Additive (Eq. 2, as in the paper)4.323.017 / 19–
Cross term, constrained11.258.312 / 193 / 19
Cross term, unconstrained12.374.311 / 192 / 19
Baseline20.888.2––

The cross term fits the training data better (median mean absolute error 1.3 against 1.9 points), but it forecasts worse: it beats the additive law in only 3 of 19 panels, and its largest error more than doubles. The fitted β4\beta_4 also has no consistent sign: it is positive in 17 of the 38 fits (19 panels × 2 methods) and negative in 21. So the data do not show a consistent change of the scale coefficient with hint level, and with few training models the extra parameter mostly fits noise. We see the same when we treat pass@k as a hinting method: the pass@k law of prior work (Krajewski et al., 2026) has such an interaction term, and removing it improves its forecasts (Appendix E.2).

Action. We added this comparison to Appendix E, next to Figure 14.

Qwen3’s smaller models were trained with strong-to-weak distillation (according to the technical report), which includes fine-tuning them to match the outputs of a teacher model (either 32B or 235B param models). In this paper, Qwen3-32B is the held-out model on HLE, so the small models used to make the prediction were trained to copy the model they are predicting. This might be a mismatch from a realistic setup where smaller models are not distilled down.

We thank the reviewer for pointing this out and agree this is a weakness. The reviewer is correct that Qwen3's smaller models are distilled from Qwen3-32B and Qwen3-235B-A22B. This is not specific to Qwen3: the Gemma 3 models are also trained with knowledge distillation, and distilling small models from larger ones is common practice in post-trained model families. Our discussion (Section 7) notes that "among the five model families we consider, only Delphi is explicitly a scaling ladder trained on one data recipe", but it did not mention distillation, and we agree it should. Few open post-trained families with enough sizes avoid distillation.

We believe the effect of this on our conclusions is limited, for two reasons. First, the baseline and the hinted law are fit to the same small models, and we have no reason to expect distillation to help one more than the other. Second, the Delphi ladder involves no distillation and hinting forecasts it accurately as well: the hinting ensemble beats the baseline in all five Delphi panels with a median error of 3.5 percentage points against 20.8.

Action. We now state in the discussion that the smaller members of some post-trained families are distilled from larger ones, so the models we fit on are not independent of the model we forecast.

None of the forecasting results are reported with uncertainty quantification, while a few models are used to fit the proposed laws.

We thank the reviewer for this suggestion, and we agree that a single error per panel hides how uncertain these forecasts are. Following the suggestion, we bootstrapped the forecast errors. We resample the benchmark problems with replacement (the same draw for every model, hint level and method), rebuild all accuracies, and refit the baseline and both hinting methods, 1,000 times. Over the 19 main-text panels:

Reported95% interval
Hinting ensemble, median error4.3[2.8, 6.6]
Baseline, median error20.8[11.3, 23.1]
Baseline minus ensemble16.5[6.9, 18.6]
Panels where the ensemble beats the baseline17 / 19[15, 18]

The ensemble's median error is below the baseline's in all 1,000 resamples. Per panel, the ensemble beats the baseline in at least 90% of resamples in 15 of the 19 panels. The bootstrap also shows that hinted forecasts are far more stable than the baseline when few models are available: for Qwen2.5 on AIME with two training models, the baseline's error ranges from 9 to 93 points across resamples (95% interval), against 0 to 10 for the ensemble.

These intervals only capture the sampling noise of the benchmark problems, so they likely underestimate the full uncertainty of a forecast. They do not capture the choice of training models: as the reviewer notes, each forecast is fit on only a few models (two to five, and up to eleven on Delphi), which is too few to resample meaningfully.

Action. We added the bootstrap intervals to the forecasting results in Appendix E (for the headline numbers and per panel), and state there that they capture only the sampling noise of the benchmark problems.

Questions

What are some limitations of this work / the hinting strategy? Which setups would you expect this to fail?

We see three main limitations. (1) Hinting needs a correct reference solution for every problem, so it requires either verified solutions or a more capable model to produce them. (2) Open-source scaling ladders are scarce: among the five model families we consider, only Delphi is explicitly a scaling ladder. (3) We have not tested hinting on agentic benchmarks. Their high cost makes cheaper forecasting and ranking especially valuable there, and we believe hinting can be adapted to them, but designing hinting methods for these benchmarks is future work (Section 7).

Are there any guidelines on how to fit the hinted laws? e.g. what hint levels should be evaluated?

In our experiments we evaluate 11 evenly spaced hint levels from 0 to 1 for masking and truncation (and 29 hint levels between 0 and 16 for logit mixing), and fit the joint law to all of them at once.

From our results, the hint levels should cover the range over which the small models rise from near zero to clearly above it, so the law sees the rise. Rather than selecting hint levels, we fit the law to all of them. For ranking, the best single hint level differs between benchmarks, which is why the hint-prior ranking pools all levels instead of tuning one.

Why is parameter count used as the compute scale for the post-trained models? (and not the compute)

Training compute is not reported for the post-trained models we consider (in particular, post-training compute is not disclosed), while parameter count is public, so we use that. For families whose members are pretrained on the same number of tokens, pretraining compute (≈6ND\approx 6ND) is proportional to parameter count, so the two scales differ only by a constant that the intercept of the joint law absorbs.

Doesn’t filtering for HLE questions that are answerable by frontier models cause some form of a selection bias?

Yes, the filtering introduces a selection bias. Hinting needs a correct reference solution, so we keep only questions for which we could generate one (Appendices A and B), and the accuracies we report are for this subset, not for all of HLE. We have no reason to believe that this bias favors hinting over the baseline or the other way around: the hinted forecasts, the baseline and the ground truth are all measured on the same subset.

Action. We now state this selection effect explicitly in Appendix A.


Thank you again for the careful review. Your comments on the excluded panels, the functional form and the Qwen3 distillation in particular led to changes that we believe make the paper considerably stronger. We are happy to clarify anything else.

Reviewer_o2JLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studied if larger model performance can be predicted using small models when their accuracy is almost 0 because in such case there is little signal for fitting scaling laws. The proposed method is Benchmark Hinting where model get different amount of help from a referece solution using truncation and masking and logit averaging with a teacher that see the solution. Since accuracy was observed to increase smoothly with hints and adding hints has roughly similar effect as increasing model scale a 3 parameter joint law was fitted over both using smallest models. For larger models the prediction was then taken at 0 hint and averaged over 3 hinting methods. The experiments were done on GSM8K and AIME and a HLE subset and models used were pretrained from Delphi and posttrained from Qwen and Gemma. The comparision was done with direct unhinted fitting and 2 stage baseline on Delphi and pass@k based hinting was also tested. Answer mixing was added to check if just making accuracy higher helps forecasting. The results showed median forecasting error dropped from 21 to 6 percentage points with better forecast in 17 of 19 selected settings. Additional ranking experiments were also done on near 0 models which have almost same accuracy and with truncation hinting similar ranking quality as around 5–6 unhinted rollouts was obtained using 1 hinted rollout per question in tested settings.

Critical strengths
  • Overall the motivation of the problem is interesting because if small models get 0 accuracy there is little signal for scaling laws fitted only on unhinted accuracy to predict larger model performance. The proposed hinting method gives extra signal without finetuning and adding hints has roughly similar effect as increasing model scale. The experiments on selected settings showed forecasting error dropped from 21 to 6 percentage points with better forecast in 17 of 19 settings which shows the idea can be useful.

  • The experiments were done with different hint method and model families and answer mixing was also added to check if just making accuracy higher helped forecasting. Since higher accuracy alone may not give useful information for predicting larger models this is a useful check. Pass@k based hinting was also tested which partly checked the benefit coming from more samples.

  • For ranking the reference rankings were obtained using many unhinted rollouts and seperate rollout samples were used for evaluation which helped reduce uncertainty from sampling. This is useful because the models have almost same accuracy. With truncation hinting similar ranking quality was obtained using 1 hinted rollout instead of around 5–6 unhinted rollouts per question in tested settings which shows useful improvement in sample efficiency.

Critical weaknesses
  • The results reported in the paper i.e 17/19 coms from selected setting but the cases where all models were near 0 were not included ad with all the reported settings the results is 21/30 which is slitghly misleading. The selection used a larger model results which generally would not be known when the forecast is made so the comparison reported do not fully represented that use case. Also the experiments on pretained were done on the Delphi model with moly GSM8K so this gives little evidence for the training settings.

  • Since the experiments have used the measurements across the hint levels and the 3 methods were average so there is limited understanding of the separate benefits from the hint and all the extra evaluations been done and the logit averaging hint method needed a teacher model. The Pass @k experiments did check for more sampling but its direct comparison with similar budget unhinted evaluation is not provided in the paper. For some cases the baselines has some prediction which are extreme so the gains would have to be checked and clear constraints for fitting should be specified

  • It is not very clear what loss was used for the fitting and how did paper handled 0 scores because this details can affect the forecast given experiments were done on only few small models. Since the same scale coefficient were used at all hint level for joint law too more experiments needs to be done if such restriction affects the forecasting. Additional detailed for uncertainity of forecast errors need to be reported because experiments of wen3.5 on HLE shows that forecast gets worst when more models gets added.

- Since the ranking experiments were done with only 1 set of 7 models for each benchmark more experiments needs to be done to see if similar result also comes with other model sets. It is also not clear if truncation was selected before evaluation because the paper reported this method as it worked best. For HLE ranking more additional check would be required because an LLM grader was used and some model accuracy differences are only a few point. More rollouts does not check possible mistakes from the grader so these close comparisions need manual or independent grading checks.

Questions for authors
  • Since the forecasts for Qwen2.5 and Gemma3 on HLE were higher than actual results was there any difference in small model hint curves compared with cases where forecast worked well and could this be used to identify such cases before knowing larger model results?

  • Provide more details for the loss and settings used for fitting joint law, baseline and whether similar gain would still come if same budget was used for more unhinted samples with clearly specified constraints for baseline fitting?

  • Also since the other hint methods changed model ordering in the experiments was truncation selected before evaluating these ranking sets and were the close HLE comparisons also checked manually or with an independent grader to see if same ordering and gains remain?

7 comments

We thank the reviewer for the thorough feedback. We are glad they find that we address an interesting problem and that benchmark hinting is a useful idea. We have run additional experiments and uploaded a revised PDF that addresses all of the reviewer's comments (see also the general response). In summary, the revised paper shows that a single hinting method already forecasts well in each setting, reports results over all 30 panels, and adds a budget-matched baseline (also for pass@k), the full fitting procedure with alternative baselines, a joint law with a cross term, and bootstrap intervals. We also reran the ranking comparison on every other valid model set.

Following are our responses to the reviewer's comments.


The results reported in the paper i.e 17/19 coms from selected setting but the cases where all models were near 0 were not included ad with all the reported settings the results is 21/30 which is slitghly misleading. The selection used a larger model results which generally would not be known when the forecast is made so the comparison reported do not fully represented that use case. Since the forecasts for Qwen2.5 and Gemma3 on HLE were higher than actual results was there any difference in small model hint curves compared with cases where forecast worked well and could this be used to identify such cases before knowing larger model results?

The reviewer is right. Over all 30 panels the hinting ensemble beats the baseline in 22 (median error 5.9 percentage points against 12.6), and we now report this in the main text next to the 17 of 19. The 11 panels outside the main text are those of AIME + Qwen3 (4 panels), HLE + Qwen2.5 (4) and HLE + Gemma-3 (3) in Figure 13 (Appendix E). The ensemble loses to the baseline in 8 of the 30 panels: 2 of the 19 main-text panels and 6 of these 11. On AIME + Qwen3 it beats the baseline in all four panels, so all six losses are in HLE + Qwen2.5 and HLE + Gemma-3, where every model in the family scores near guessing performance.

On these two families the baseline indeed forecasts the largest model more accurately: the hinting ensemble overestimates it in all seven panels, by up to 9.6 percentage points. We agree this is a real weakness of our results, since our hinted forecasts misjudge where accuracy starts to rise in these families. As a side note, truncation alone is more robust: it stays within 4.2 percentage points of the target in all seven of these panels (median error 1.7 against 2.7 for the baseline), and it beats the baseline in 12 of the 14 post-trained main-text panels (see also our response on ensembling).

We left these panels out of the main-text figures because the largest members of both families score only 2.7%. The target model therefore sits below the noise threshold, so a forecast that stays flat at zero is correct by default. If these families were extended to larger members, accuracy should eventually rise above zero, and a flat baseline could then have a larger error than the hinted prediction. How large depends on whether accuracy starts to rise at a model scale 2x or 2000x larger than the largest member here, which these families cannot tell us. These panels therefore show that hinting can overestimate, but they cannot show which method better predicts the model scale at which accuracy rises above zero.

On the reviewer's related question, we have not yet investigated whether the small models' hint curves differ from those of the families where the forecast works well. Detecting in advance when a hinted forecast will be unreliable is an interesting question that we leave to future work.

Action. The main text stated that these families score around zero but not why that makes their panels uninformative for evaluating forecasts. We added this reasoning where we introduce the selection criterion (Section 4), together with the results over all 30 panels, and we now acknowledge in the main text that the hinting ensemble overestimates accuracy on these families.

the experiments on pretained were done on the Delphi model with moly GSM8K so this gives little evidence for the training settings.

Indeed, for the pretraining setting we only show one model family (Delphi) and one benchmark (GSM8K). In the revised paper we strengthened the Delphi results: we trained additional models so that every compute budget below 102310^{23} FLOPs has three random seeds, added two new budgets, and show that a hinted scaling law fit on models of at most 102010^{20} FLOPs forecasts the accuracy of the 102310^{23} FLOPs model to within 3 percentage points (Section 4.2).

The limited scope is mostly an availability issue: few open-source pretraining scaling ladders contain enough models to make predictions. Delphi is also the only one we know of with several random seeds per compute budget, and we found that averaging over seeds makes the Delphi forecasts more stable. The choice of benchmark is constrained as well. To measure the effectiveness of hinting, we need a benchmark on which the smaller members of the ladder score near zero while the larger members score clearly above zero. GSM8K meets this for Delphi, whose largest member (102310^{23} FLOPs) scores 25.9%. To our knowledge, GSM8K and HumanEval are the only widely used benchmarks without multiple-choice answers on which the largest Delphi model scores clearly above zero.

Action. We now state more explicitly in the discussion that our pretraining results cover only one ladder and one benchmark, and explain why.

Since the experiments have used the measurements across the hint levels and the 3 methods were average so there is limited understanding of the separate benefits from the hint and all the extra evaluations been done and the logit averaging hint method needed a teacher model.

We thank the reviewer for this point. We agree it is important to separate the effect of the hint from that of ensembling and of the extra evaluations, and we address each below.

Ensembling. In the revised paper, a single hinting method already forecasts well in each setting, so our results do not depend on ensembling. On the Delphi pretraining ladder, logit mixing with a teacher beats the baseline in all nine training sets, with a mean error of 3.3 percentage points (Section 4.2). On the post-trained families, truncation alone beats the baseline in 12 of the 14 main-text panels, stays within 4.2 percentage points of the target on the near-guessing families discussed above, and is the method that ranks models best (Section 5). We still report the ensemble of masking and truncation, with the individual forecasts shown as grey curves in Figures 4 and 5, because different hints give forecasts whose errors are not necessarily correlated, which we see as a feature of hinting. On Delphi, for example, masking over-forecasts the largest model and truncation under-forecasts it, and their average beats the baseline in all five panels. Over the 19 main-text panels, masking alone beats the baseline in 15, truncation alone in 13, and the ensemble in 17.

Teacher model. We dropped logit averaging, so the ensemble no longer needs a teacher model: masking and truncation only change the prompt of the evaluated model. Logit mixing, a new hinting method that does need a teacher, is reported separately for the Delphi ladder (Section 4.2).

Extra rollouts. On AIME and HLE, masking and truncation each use 100 hinted rollouts per problem (10 hint levels × 10) and share the 10 unhinted ones, so the ensemble uses 210 rollouts per problem, whereas the baseline uses 10. To separate the effect of the extra rollouts, we refit the baseline on 256 unhinted rollouts per problem, which is more than the ensemble uses. These deeper unhinted pools exist for the post-trained models because we collected them for the ranking experiments; we do not have them for Delphi. On the 14 AIME and HLE panels of the main text, the extra rollouts lower the baseline's median error from 17.9 to 11.4 percentage points, but the hinting ensemble (5.0) still beats it in 11 of 14 panels.

We agree that a full conceptual account of what makes a hint informative remains open, and we see it as an interesting direction for future work.

Action. We added the forecast errors of each hinting method on its own and the budget-matched baseline to Appendix E, and we now state in Section 4 which single method forecasts well in each setting: truncation on the post-trained families and logit mixing on the Delphi ladder.

The Pass @k experiments did check for more sampling but its direct comparison with similar budget unhinted evaluation is not provided in the paper.

The reviewer is right that on AIME and HLE, Figure 17 compares pass@k, estimated from 256 rollouts per problem, against a baseline fit on only 10 rollouts per problem. On Delphi the comparison is already budget-matched since pass@k and the baseline use the same 64 rollouts per problem and seed. We would also like to clarify that pass@k is not a baseline in our paper but another hinting method: we treat log⁡2k\log_2 k as the hint level and fit our joint law to it (Appendix E.2).

Using the experiment described in our previous answer, we refit the baseline on 256 unhinted rollouts per problem. On the 14 AIME and HLE panels of the main text, the matched baseline has a median error of 11.4 percentage points. pass@k lowers this to 7.7 and beats the matched baseline in 12 of 14 panels, so our joint law does extract more from the same samples than simply averaging them. The hinting ensemble lowers the error further, to 5.0 with fewer rollouts (210 per problem).

The functional form of the joint law is important for this. Prior work (Krajewski et al., 2026) fits pass@k with a power law in the negative log accuracy that includes an interaction term between training compute and kk. We reproduced this law on our data and used its k=1k = 1 slice to forecast the largest model. It beats the baseline in only 7 of the 19 main-text panels (median error 11.0 percentage points), whereas our joint law fit to the same pass@k data beats it in 17 of 19 (median error 8.5).

Action. We now state more clearly in Section 4.1 and Appendix E.2 that pass@k is a hinting method fit with our joint law, not a baseline. We added the budget-matched comparison and the comparison with the law of Krajewski et al. to Appendix E.2, and note the matched budget in Section 4.1.

Comments on the paper

3 comments

We thank all reviewers for their thoughtful feedback. Reviewers found the problem of predicting benchmark performance from small models that score near zero well motivated (o2jl, SDjk), the predictable scaling of hinted accuracy with hint level and model scale a useful basis for forecasting (o2jl, SDjk, i2kv), and the gains in ranking efficiency valuable (o2jl, i2kv). We have responded to each reviewer and uploaded a revised PDF. Here we summarize the improved results and what changed in the paper since submission.

Improved results

Reviewer o2jl asked us to separate the benefits of the individual hinting methods from those of the ensemble. In the revised paper:

  • One hint works in each setting. To separate the individual hints from the ensemble, we now also report each hinting method on its own. A single method works well in each setting even without ensembling: truncation across the post-trained families, and logit mixing for the pretraining setting. Truncation beats the baseline in 12 of the 14 post-trained settings, stays close to the true accuracy on the near-zero post-trained families, and is the best hint method for ranking. On the Delphi pretraining ladder, logit mixing fit on models of at most 102010^{20} FLOPs forecasts the GSM8K accuracy of the 102310^{23} FLOPs model (25.9%) to within 3 percentage points, where a standard scaling law under-forecasts it by 13 points.
  • Combining hints helps further. Ensembling is not required to predict benchmark performance, but it is beneficial: the errors of different hints may not be correlated, so averaging their forecasts reduces the error. Across the 19 main-text settings the ensemble of masking and truncation beats the baseline in 17, and its median error decreases from 6.0 to 4.3 percentage points (against 20.8 for the baseline). This follows from the changes below, in particular the more flexible joint law and the Delphi accuracies averaged over three seeds.

Changes to the method

These improvements come from the following changes to the paper.

We dropped logit averaging. We removed logit averaging from the paper, so the ensemble now combines the forecasts of masking and truncation only. This also addresses a point raised by reviewer o2jl, that logit averaging needed a teacher model: because truncation and masking change only the prompt of the evaluated model, the ensemble requires only input access to the model (and no longer its logits or a teacher model).

Logit mixing is a new hinting method for the pretraining setting (Sections 2.1 and 4.2). It uses a stronger external teacher (Llama-3.1-8B-Instruct, which is given the reference solution) and adds its logits to those of the evaluated model. Because it needs a teacher that shares the evaluated models' tokenizer, we use it on the Delphi ladder and report it separately from the ensemble. This gives the Delphi result above.

Joint law (Section 4). The reviewers had questions about the functional form of the joint law and how it is fit (o2jl, SDjk). We revisited the functional form and replaced the three-parameter sigmoid with a floored sigmoid, which lets accuracy stay flat until it starts to rise and saturate below 100%. As the reviewers suggested, we also tested a form with an interaction term between model scale and hint level: it fits the training models better but forecasts the largest model worse, so we keep the simpler law (Appendix E).

Delphi ladder (Section 3). We trained additional models so that every compute budget below 102310^{23} FLOPs has three random seeds, and added two new budgets. A re-evaluation with strict answer matching changes the GSM8K accuracy of the largest model from 24.1% to 25.9%.

Additional experiments

We ran a new experiment for each concern the reviewers raised about how our results are evaluated (Section 4.1 and Appendix E).

Do the results hold on all model families? (o2jl, SDjk) The main text reported 19 forecasting settings and left out the model families in which every model scores near zero. We now report all 30 settings: the hinting ensemble still beats the baseline in 22 of them. Most of its losses are on two HLE families in which even the largest model scores near guessing performance. There the ensemble predicts an accuracy that is too high, which we now state as a limitation, while truncation alone stays close to the true accuracy.

Is hinting more accurate only because it uses more rollouts? (o2jl) The hinted forecasts use more model rollouts than the baseline, which is fit on unhinted accuracies alone (210 against 10 rollouts per problem). The extra rollouts do not explain why hinting forecasts more accurately: when we refit the baseline on 256 unhinted rollouts per problem, more than the ensemble uses, the ensemble's error is still less than half as large (5.0 against 11.4 percentage points).

Do the results depend on the baseline's extreme predictions? (o2jl) Reviewer o2jl rightly noted that the baseline, a standard scaling law fit to the unhinted accuracies of the small models, sometimes makes extreme predictions, and asked us to check our results against this. These predictions arise when the small models score near zero: many curves then fit them equally well but forecast very different accuracies for the largest model. They therefore follow from how little the near-zero accuracies say about larger models, and not from a particular choice in our fitting procedure, which the paper now describes in full. To check that our results do not depend on the sometimes extreme baseline predictions, we ran two comparisons. First, we left out the three settings where the baseline is off by more than 50 percentage points: hinting is still more accurate there (median error 3.9 against 15.6). Second, we compared against a piecewise-linear baseline, which extrapolates more conservatively: hinting is more accurate in 14 of the 19 settings, and that baseline is better where the small models already show a rising trend.

How much do the forecast errors vary? (o2jl, SDjk) The reviewers asked for uncertainty on the forecast errors. We now report bootstrap intervals obtained by resampling the benchmark problems (Appendix E). The ensemble's median error is lower than the baseline's in every resample. However, these intervals only capture the sampling noise of the benchmark problems. They are not confidence intervals for the forecasts, since each forecast is fit on only a few models, which are too few to resample.

Does the ranking result depend on the model set? (o2jl) Our ranking experiment used one set of seven models per benchmark, and reviewer o2jl asked whether the result also holds for other model sets. Only four valid sets of seven models exist per benchmark, so we repeated the comparison on every valid set of five, six or seven models in our pools: with one rollout per problem, the hinted ranking is closer to the true ranking than the unhinted one in nearly all of them (152 of 153 sets on AIME and 279 of 292 on HLE).

Limitations (o2jl, SDjk). The discussion now states that our pretraining results cover one ladder and one benchmark, that some post-trained families are distilled, and that hinting can overestimate accuracy when a capability does not appear within a model family.

The individual responses give more detail on each of these.

AuthorsOpenReview ↗

Summary of the discussion

Dear Area Chair and Reviewers,

We thank the reviewers for the constructive discussion and summarize it here.

Our contribution is hinted scaling laws, which use hinted benchmarks to fit more accurate scaling laws for benchmark accuracy. A hinted benchmark turns a benchmark into a sequence of easier versions which we create using three methods. The hinted forecast beats a standard scaling law in 17 of the 19 main-text settings and in 22 of all 30 settings we evaluate. Over all 30, its median error is 5.9 percentage points against 12.6. In the pretraining setting, models trained with at most 102010^{20} FLOPs forecast the accuracy of a 102310^{23} FLOPs model to within 3 points whereas the standard law is off by 13.

The reviewers found the problem well motivated and the idea useful but were concerned with the robustness of our results (whether they hold beyond the settings we reported, whether the comparison with the baseline is fair, whether the functional form of the scaling law is justified, how uncertain our forecasts are). We addressed each with a new experiment in the revised PDF and our main conclusions hold in each case. Our general response describes all the additional experiments and the changes made to the paper.

Reviewer SDjk wrote that their concern about the excluded settings "is addressed" and that the result of the interaction-term evaluation "is interesting". They wrote that their "final evaluation is between a 3 and a 4".

We believe the revision addresses the concerns raised and that the paper is substantially stronger as a result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.