- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces RADS, a data-selection method for weak-to-strong generalization. Presented with weak labels and hidden representations from a weak teacher, RADS 1) first fits a MLP from weak representation to weak labels and computes residuals 2) projects those residuals through a ridge-regression smoother based on strong representations 3) prioritizes examples with large values from 2).
At a 5-% selection budget, RADS has the best average PGR among the tested methods across nine classification datasets. Reward modeling experiments are also reported in the paper.
Critical strengths
- Important problem. Data selection for weak selection is underexplored, and even in a broader context.
- Statistically motivated simple methods. RADS does not require ground true labels and are built upon classical statistical results.
- Comprehensive experiments across datasets and baselines.
Critical weaknesses
- Results are mainly empirically only. The theoretical motivation under a linear ridge-regression setting needs ground-truth labels. In RADS however, weak labels are used. The theorem motivated examine the norms of v but not the chosen selection direction.
- Experimental uncertainty is high. For example from Table 1, RADS's wins are concentrated on certain datasets such as FEVER-NLI and WANLI, while not for others. Table 2 is missing stand errors.
- Limited scalability and practical applicability. RADS requires running both models over the entire candidate pool, the cost of constructing kernels, solving ridge regressions, etc are not properly documented.
Questions for authors
See weaknesses.
2 comments
Dear Reviewer vMF6,
Thank you for your time and effort in reviewing our paper. As the author–reviewer discussion period is coming to a close, we would be happy to answer any remaining questions you may have. We appreciate your engagement.
We thank the reviewer for their interest in the setting and for recognizing it as an important problem, especially since data selection in weak-to-strong generalization is underexplored. We also appreciate the reviewer noting that our approach is motivated by simple methods, and we thank them for highlighting our hard work in covering diverse empirical settings. We address each of the mentioned weaknesses below.
W1) Empirical-centric results. We first want to describe the use of v again. This vector has components corresponding to each data point, where vi captures the predicted separation between training on ground-truth labels and weak labels for that data point. Thus, v acts more as a collection of values with ∥v∥2 being their sum of squares, rather than a proper vector with magnitude and direction.
Second, we note that most prior work in theoretical settings relies on assumptions that do not hold in practical, real-world settings. Therefore, the scope of our paper is primarily empirical. Many of the i.i.d. assumptions are broken when performing data selection, which makes it challenging to produce a clean theoretical result that holds beyond very simple, Gaussian-distributed settings. Regarding the selection direction, we report results for both the high and low ∣v∣ points in Table 6 and Table 8, and the high ∣v∣ points perform better on average.
W2) High experimental variance. This high variance relates to W2 in the common response. In summary, we used two types of datasets: some common ones and some that showed the greatest likelihood of W2S generalization based on the weak and strong performances. The NLI datasets were all of this second type, with high weak model performance and low base-strong model performance. For the OLMo and Gemma models in Table 12, RADS also shows an improvement over random selection on these datasets. Additionally, we have added the error bars to Table 2.
W3) Limited scaling and cost summary. In the revised paper in Appendix B.5, we document the compute required for RADS along with the expected scaling to larger models and a measured running time. We note that for a fine-tuning setting where the data pool is relatively small, RADS uses less compute than a training run over all of the data once.