- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a variant of scaling law measurement, Hinted Scaling Law, which fits to the benchmark accuracies. To avoid the scarce accuracy scores on small-scale and uncapable models, the authors propose to include the "hints" by exposing part of answer tokens or teacher's knowledge during accuracy evaluation stage. Applying various levels of hints, the authors manage to fit a downstream accuracy scaling curves, and also apply this method to rank models on benchmark performance.
Critical strengths
(1) The paper addresses the scarcity of benchmark accuracy and directly fit a scaling law to the downstream accuracy scores using various types of induced Hints. They show that the hinted accuracy is predicable from hinted level and model scale.
(2) The Hinted accuracy works well and improves efficiency at ranking models on benchmark performance.
Critical weaknesses
(1) The test loss or perplexity on the targeted benchmarks can also be the continuous proxy of downstream performance, can we alternatively apply it to fit the scaling law or rank models?
(2) What implications we can draw from the scaling curves across different hinted levels?
Questions for authors
As stated in Weaknesses
3 comments
Thank you again for your careful review and thoughtful feedback. Have we addressed your concerns in our response or is there anything else we could clarify?
Thanks for the clear response, it address most of my concern! I will keep my positive score.
We thank the reviewer for the thoughtful feedback. We have run additional experiments and uploaded a revised PDF with improved results (see also the general response). Among other changes, a hinted scaling law fit on Delphi models of at most 1020 FLOPs now forecasts the GSM8K accuracy of the 1023 FLOPs model to within 3 percentage points, and all forecast errors come with bootstrap intervals. In response to the reviewer's first comment, the main text now also points to our loss-based forecasting baseline. Following are our responses to the reviewer’s comments.
We thank the reviewer for raising this. The paper tests both uses of a loss-based proxy and neither works in our setting, but the forecasting experiment was only in the appendix and easy to miss.
Action. We now point to the loss-based forecasting baseline in the main text (Section 4.1), next to our other comparisons.
We interpret the question as asking what the scaling curves at different hint levels (Figure 2, right) imply, and see the two implications below. Please let us know if we misunderstood and we can provide more details.
What the curves show. Accuracy varies smoothly with both hint level and model scale (Figure 2). Raising the hint level shifts the scaling curve towards smaller models, with curves rarely crossing, so it moves the point where accuracy starts to rise towards smaller scales (Section 3).
What the curves can be used for. Because hinting reveals this rise at scales where the unhinted benchmark still scores near zero, the hinted curves can be used to forecast benchmark accuracy of larger models (Section 4) and to rank models with fewer rollouts (Section 5).
Thank you again for the review. We hope the revised paper and these answers address both questions, and we are happy to clarify anything else.