acceptodds
Under review as a conference paper at ICLR 2027

Aligning Prediction Residuals: Representation-Aware Data Selection for Weak-to-Strong Generalization

Abstract

Weak-to-strong generalization (W2S), where a weaker teacher supervises the training of a more capable student, offers a path to advancing frontier models beyond what costly human expert supervision can offer. Existing W2S work focuses on model-centric aspects such as adapting the training objective. Far less is known about which data to use in order to maximize the student's improvement over its teacher. We study this problem, beginning with a characterization of how bias and structured noise in weak labels limit the performance of the student. Building on this, we develop a data selection framework, **RADS**, measuring differences in predictive capability between the two models' representations. RADS prioritizes points with features the stronger model encodes but the weaker teacher misses, and training on them yields the largest performance gains. Across nine datasets, RADS improves over the strongest baseline by more than 9% on average; this holds across data budgets. We show this advantage does not come from simply selecting samples the teacher already labels correctly, but from data with features the student can additionally learn from. The effect carries over to reward modeling, where RADS outperforms naive sampling by 10% on average. Finally, we present a probing study that estimates potential W2S gains prior to training

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Weak-to-Strong Generalization, Representation Alignment, Data Selection

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_vMF6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RADS, a data-selection method for weak-to-strong generalization. Presented with weak labels and hidden representations from a weak teacher, RADS 1) first fits a MLP from weak representation to weak labels and computes residuals 2) projects those residuals through a ridge-regression smoother based on strong representations 3) prioritizes examples with large values from 2).

At a 5-% selection budget, RADS has the best average PGR among the tested methods across nine classification datasets. Reward modeling experiments are also reported in the paper.

Critical strengths
  • Important problem. Data selection for weak selection is underexplored, and even in a broader context.
  • Statistically motivated simple methods. RADS does not require ground true labels and are built upon classical statistical results.
  • Comprehensive experiments across datasets and baselines.
Critical weaknesses
  • Results are mainly empirically only. The theoretical motivation under a linear ridge-regression setting needs ground-truth labels. In RADS however, weak labels are used. The theorem motivated examine the norms of v but not the chosen selection direction.
  • Experimental uncertainty is high. For example from Table 1, RADS's wins are concentrated on certain datasets such as FEVER-NLI and WANLI, while not for others. Table 2 is missing stand errors.
  • Limited scalability and practical applicability. RADS requires running both models over the entire candidate pool, the cost of constructing kernels, solving ridge regressions, etc are not properly documented.
Questions for authors

See weaknesses.

2 comments

We thank the reviewer for their interest in the setting and for recognizing it as an important problem, especially since data selection in weak-to-strong generalization is underexplored. We also appreciate the reviewer noting that our approach is motivated by simple methods, and we thank them for highlighting our hard work in covering diverse empirical settings. We address each of the mentioned weaknesses below.

W1) Empirical-centric results. We first want to describe the use of v\mathbf{v} again. This vector has components corresponding to each data point, where vi\mathbf{v}_i captures the predicted separation between training on ground-truth labels and weak labels for that data point. Thus, v\mathbf{v} acts more as a collection of values with ∥v∥2\|\mathbf{v}\|^2 being their sum of squares, rather than a proper vector with magnitude and direction.

Second, we note that most prior work in theoretical settings relies on assumptions that do not hold in practical, real-world settings. Therefore, the scope of our paper is primarily empirical. Many of the i.i.d. assumptions are broken when performing data selection, which makes it challenging to produce a clean theoretical result that holds beyond very simple, Gaussian-distributed settings. Regarding the selection direction, we report results for both the high and low ∣v∣|\mathbf{v}| points in Table 6 and Table 8, and the high ∣v∣|\mathbf{v}| points perform better on average.

W2) High experimental variance. This high variance relates to W2 in the common response. In summary, we used two types of datasets: some common ones and some that showed the greatest likelihood of W2S generalization based on the weak and strong performances. The NLI datasets were all of this second type, with high weak model performance and low base-strong model performance. For the OLMo and Gemma models in Table 12, RADS also shows an improvement over random selection on these datasets. Additionally, we have added the error bars to Table 2.

W3) Limited scaling and cost summary. In the revised paper in Appendix B.5, we document the compute required for RADS along with the expected scaling to larger models and a measured running time. We note that for a fine-tuning setting where the data pool is relatively small, RADS uses less compute than a training run over all of the data once.

Dear Reviewer vMF6,

Thank you for your time and effort in reviewing our paper. As the author–reviewer discussion period is coming to a close, we would be happy to answer any remaining questions you may have. We appreciate your engagement.

Reviewer_CTQNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes RADS, a data selection method for weak-to-strong generalization. It scores samples using weak-label residuals that are predictable from strong representations. Experiments evaluate the method on classification and reward modeling tasks.

Critical strengths

Studying data selection without access to any ground-truth labels is meaningful.

Critical weaknesses
  1. The motivation and justification for the method are insufficient. The paper develops a data selection heuristic based on Xue et al. (2025), but the transition from theory to algorithm is not clearly explained. The presentation also lacks rigor. For example, the main text does not provide formal definitions of PredGap or (δw,γ^w,γ~w)(\delta_w,\hat{\gamma}_w,\tilde{\gamma}_w)-decomposability.

  2. The experiments provide insufficient support for the method’s generality. The main-text experiments use teachers and students from the Qwen2.5 family. Validation across model families is needed.

Questions for authors
  1. Could you explain how Algorithm 1 follows from Theorem 3.8 and why this transition is justified? Does Theorem 3.8 require any additional assumptions on the data or models?

  2. Beyond Qwen2.5-0.5B and Qwen2.5-7B as the weak and strong models, could you provide more model combinations to validate the method’s generality?

  3. Are there other data selection methods that operate without access to any ground-truth labels?

3 comments

We appreciate the reviewers' attention to the theoretical basis of the method and to the notation not fully explained. We have modified the draft to add some details and hope to explain the choice to omit others.

W1 and Q1) Clarity and rigor. We acknowledge that PredGap was not cleanly defined and (δ,γ^,γ~)(\delta, \hat{\gamma}, \tilde{\gamma})-decomposability was not rigorously defined. For the former, we have added a clearer description in the revised work. For the latter, we omit the precise definition due to its length. On page 4 of Xue et al., the definition covers 1/3 of a page, and we find the precise definition provides little intuition, nor does it assist in establishing RADS. We also note that this theorem was used to guide our algorithm design for testing representation alignment for performing data selection; a precise theoretical characterization of the method was not our primary focus.

W2 and Q2) Additional model families. We agree that our selection of a single model family did not sufficiently defend this as a general phenomenon. We have added Table 12 to the revised draft, including results for OLMo-2 (1B to 7B) and Gemma-2 (2B to 9B) pairs. These show mostly similar behaviors to the Qwen family: the 5 datasets favorable to any W2S generalization show an improvement of RADS over random selection, and on average RADS outperforms random selection; this includes situations where the base strong model already outperforms its weak teacher.

W3) Other ground-truth-free methods. We thank the reviewer for raising this. To our knowledge, the only other ground-truth-free method designed for W2S generalization is that of Shin et al., which uses overlap density to predict W2S performance. We report this method in Table 1 and Table 6. In our work, we show that RADS outperforms the overlap density method by 0.15 points on average.

Dear Reviewer CTQN,

Thank you for your time and effort in reviewing our paper. As the author–reviewer discussion period is coming to a close, we would be happy to answer any remaining questions you may have. We appreciate your engagement.

Thank you for the response and the additional experiments. These address some of my concerns, but the substantive contribution beyond Xue et al. (2025) remains limited. I will maintain my original rating.

Reviewer_2KLnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors of the paper propose a new algorithm to tackle the important problem of sample selection for weak to strong generalization. Termed RADS, the proposed method scores individual training points for weak-to-strong (W2S) training without using ground-truth labels. It is motivated by Xue et al.'s result that established a bound on weak-to-strong performance based on the representation spaces of each of the strong and weak mode. RADS works in three steps:

  1. Fit a cross-fitted MLP from the weak model's features to the weak labels, and keep the residual a.
  2. Ridge-smooth a on the strong model's features to get v = P_s a.
  3. Train on the half of the pool with the largest |v|.

The authors of the paper conducted experiments with With Qwen2.5-0.5B → 7B on nine classification tasks and demonstrate superior performance.

Critical strengths
  • The authors of the paper tackles an important problem.
  • The band comparison shows that the score does order the importance of data effectively.
  • The gain over baseline methods are large.
Critical weaknesses
  • The main results shown in table 1 only compares the proposed method to baseline methods of 9 datasets, while the appendix included much more datasets (23 in Table 7). How does the performance of the performance look on the other datasets? how did you pick the 9?
  • Moreover, based on my understanding with Table 6, the primary variant was picked by test PGR on Table 1's own datasets. How does RADS perform when applied to a new dataset that you haven't done any hyper-parameter selection with? Because of these, I am not fully convinced about the advantage of the proposed method.
Questions for authors
  • What is the MLP used?
  • How are the 9 datasets chosen?
2 comments

We thank the reviewer for their careful read of our results, noting that we show that RADS sufficiently orders data by how well they improve W2S generalization. We have revised the current draft of the paper to address the weaknesses and questions raised. We summarize the changes here.

W1 and Q1) Dataset choices. We refer the reviewer to W2 in the common response. In summary, we chose datasets based on the difference in performance between the weak teacher and the base-strong student. We chose a few datasets with a large difference, a few with a small difference, and a handful of common datasets even though this selection criterion would indicate they might not exhibit any positive W2S generalization.

W2) Further dataset questions. Only a small handful of datasets were selected using the scoring metric, most notably IMPPRES. The rest of the datasets were chosen as described in W1 in the common response, so the results seen are on randomly selected datasets.

Q1) MLP settings. We agree we did not supply the MLP settings in the draft. In Section B.3 of the revised draft, we have added additional MLP settings used in the experiments.

Dear Reviewer 2KLn,

Thank you for your time and effort in reviewing our paper. As the author–reviewer discussion period is coming to a close, we would be happy to answer any remaining questions you may have. We appreciate your engagement.

Reviewer_QzXxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes RADS, a data selection method inspired by Xue et al. (2025) for improving weak-to-strong generalization. The authors report average performance gains in classification and reward modeling and investigate representation similarity as a predictor of the performance difference between the selected and discarded subsets.

Critical strengths

None

Critical weaknesses

Presentation

The paper is not sufficiently self-contained. Notation essential to following the main argument is deferred to Appendix A, which provides only a table of brief descriptions rather than adequate explanations of the quantities and their roles. Similarly, the theoretical result from Xue et al. (2025) is stated without sufficient background, so its applicability is difficult to understand without consulting the original paper. Essential aspects of the experimental setup (how representations are extracted and used, how the classification tasks are formulated, and how the auxiliary predictors are trained) are either relegated to the appendix or insufficiently specified. These details are necessary for understanding the method and interpreting the experiments and should be clearly explained in the main text.

Novelty

Algorithm 1 relies heavily on Xue et al. (2025), and I find its novelty limited. The main changes are converting the existing dataset-level expression into a per-sample selection score using weak labels and replacing the weak-side linear predictor with an MLP. Adapting the prior analysis to data selection could be a contribution, but the adaptation appears relatively straightforward, and the paper does not provide substantial new technical insight.

Empirical Evidence

The empirical signal is not strong enough to compensate for the paper's weak presentation and limited novelty. The main gains are concentrated on ANLI-R1, FEVER-NLI, and WANLI, whereas averaging the reported results for the 14 additional datasets in Table 7 yields approximately 0.499 PGR for RADS versus 0.511 for random selection.

The reward-modeling results are similarly mixed: RADS underperforms random selection on two of the four datasets at the 10% budget and exceeds token-length selection by only 0.014 mean PGR at the 30% budget. Overall, these results do not appear to convincingly establish a robust advantage.

Questions for authors

Could the authors clarify how the nine datasets in Table 1 were selected from the broader set of 23 datasets reported in Table 7?

4 comments

We thank the reviewer for carefully checking the prior work and how it compares to our paper.

W1) Presentation. We acknowledge a typo in Appendix A that uses ψ\psi rather than θ\theta to represent the models from an earlier version of the draft; this has been corrected. However, we note that all notation from the appendix is described in the main text. Specifically, in Algorithm 1, we provide comments that indicate what each line is doing. When PsP_s and PwP_w are introduced in the main text, we describe their purpose in Section 3.2 after providing our complete algorithm. For the full definition of (δ,γ^,γ~)(\delta, \hat{\gamma}, \tilde{\gamma})-decomposability and other assumptions for Theorem 3.8, we omitted these due to brevity. The definition requires 1/3 of a page in the prior work and provides little intuition to defend the structure of RADS. As we did not extend this prior theory, we focused on the portion of the result that most directly applies to this work.

W2) Novelty. We refer the reviewer to W1 in the common response.

W3) Empirical evidence. Table 7 adds supplementary results from the other datasets tested in Figure 4. Our method showed promise on some datasets and less so on others, which is the motivation for the metric in Figure 4. We do not claim our method works universally; rather, we provide a mechanism to predict performance before we train anything. Also, the average performance of RADS in Table 7 is brought down significantly by WikiToxic, a dataset that was predicted to have low performance; using a median to omit both positive and negative outliers maintains a positive benefit of RADS over random selection. Note: In the revision, the mentioned table is now indexed as Table 8.

Q1) Dataset choices. We agree we were not clear on our dataset selection criteria in the original draft. We clarify this in W2 in the common response. Summarized, we chose some common datasets regardless of how well suited they would be to test W2S generalization, and the rest were selected for their ability to exhibit W2S generalization. Note that this does not mean we selected them to perform well with RADS; rather, we selected data so that any W2S method should have a favorable situation to work.

Dear Reviewer QzXx,

Thank you for your time and effort in reviewing our paper. As the author–reviewer discussion period is coming to a close, we would be happy to answer any remaining questions you may have. We appreciate your engagement.

I thank the authors for the response. However, my main concerns remain.

Generality

On the 14 additional datasets in Table 8, RADS does not outperform Random. The median argument holds only for column medians, not for per-dataset differences. On IMPPRES, which Table 5 marks as favorable and which was added to test the probing metric, RADS clearly underperforms Random, as it does on WikiToxic. The probing metric predicts the high-vs-low band gap but not the gain over Random. Table 12 extends the gains to other model families, but only on the Table 1 datasets.

Theoretical Motivation

The weak labels come from a thresholded linear probe on the same representation used as XwX_w (Appendix B.1), so no part of y^\hat{y} lies beyond the weak representation. The nonzero residual aa thus reflects the fitting procedure rather than missing weak-model capacity; Appendix C.6 notes that without weight decay the MLP nearly perfectly fits the labels. Moreover, as the authors acknowledge, Theorem 3.8 does not justify selecting high-∣v∣|v| points.


The rebuttal frames the contribution as predicting performance before training, while the paper claims RADS is the best method on average (C1–C4). Under the first framing, the metric does not predict the gain over Random; under the second, the gains are limited to a few datasets.

For these reasons, I still consider this a clear rejection.

Thank you for the discussion. On the first concern, we agree that IMPPRES and WikiToxic both underperform random selection. However, 3 other datasets (CosmosQA, ANLI-R3, RiddleSense) perform significantly better than random within the additional 14 datasets. We agree this is not a uniform win, but we would not expect a guarantee across all datasets. Two of 14 datasets underperforming is not evidence that RADS underperforms random on average.

There also is not a direct conflict between C1 and the probing metric. On average, across the 23 datasets, RADS performs better than random. The metric asks a different question: is the ordering of ∣v∣|v| informative on each dataset? Does the partition of the datasets select a half of the data that typically performs better? We observe this is indeed the case, as on 21 datasets, the high band partition outperforms training on the remaining low partition. We agree our rebuttal phrased this too broadly, and we have refined this distinction in the draft.

The results in Table 12 are meant to answer a different concern. Rather than these being a defense for the generality of RADS, this is used to compare the performance on Qwen to other architectures. Both OLMo and Gemma, our new results, show similar behaviors for these datasets, even with different representations guiding data selection.

As for Theorem 3.8, our position stays the same as in the rebuttal. We use it to motivate the score, and the case for picking high-|v| points comes from the experiments. We agree that the fitting procedure determines the deviation of the weak labels and the probe, and we will be clearer about this in a later revision. We suspect the flimsy linear boundary for the weak labels had to compromise certain near-boundary points, and these points can be detected and removed through RADS.

We hope we have addressed some of your remaining concerns.

Comments on the paper

2 comments

We thank the reviewers for the careful reading of our paper and attention to prior work. Among the comments provided by each reviewer, there were two main threads we address here.

W1) Novelty. Mentioned by Reviewer QzXx, our method is based on prior theoretical work by Xue et al. This previous work provides a theoretical result, requiring a lengthy assumption, of predicting the loss gap between training on ground-truth labels and training on weak-generated labels. This gap is found to be asymptotically close to a term using the ground-truth labels. In this work, we i) probe the ability for an extension of this theorem to be applied to data selection, and ii) establish a predictor of when aligning representations provides a good candidate for data selection, neither of which uses the ground-truth labels.

Additionally, we want to reinforce that the data selection from the theorem is nuanced. The theorem establishes how the PredGap, the difference in performance when fine-tuning on ground-truth labels rather than the weak labels, behaves asymptotically through the norm of v. The points with large components in v increase the RHS of the theorem the most. It is not immediately obvious that selecting these points would improve W2S generalization. However, we empirically observe this in RADS.

W2) Dataset Choices. In the work, we selected datasets with two favorable properties for studying W2S generalization: the base strong model cannot already have very high performance, and the weak model must not have too low of performance. Otherwise, we test settings where the strong model can only barely improve, or the weak model provides no useful signal. With these two criteria, the datasets WANLI, ANLI-R1, and FEVER-NLI were very promising. Two other datasets, HellaSwag, and ETHICS, were moderately promising, giving us a wider range of tested situations. We additionally included a few other common classification datasets to test, even though the base 7B model already outperformed our 0.5B weak model on most of them. We suspect these datasets may have been within the training pools of these models, making the strong model already perform well. We provide a table here of the weak model and base strong model accuracy on each of the datasets to show these differences between the different datasets. Additionally, we have added a table with all 23 datasets as Table 5 in the revised draft, along with some elaboration on dataset choice in Section B.2 and B.4.

We also note that the datasets selected for the main results came from an earlier, smaller pool where the NLI datasets showed the most promise. Later, we expanded the pool of datasets to test the proposed metric, which produced some additional datasets that would have also shown promise for testing W2S generalization for Table 1.

DatasetWeak AccuracyBase Strong Accuracy
ANLI-R10.7170.493
FEVER-NLI0.6930.518
WANLI0.6790.512
HellaSwag0.5580.522
ETHICS justice0.5700.501
AG News0.8880.860
SST-20.8310.907
SciQ0.6780.883
IMDb0.8890.954

Dear Reviewers,

As the discussion deadline (Nov 18, AoE) approaches, please review the authors' rebuttal if you have not done so, and post a follow-up indicating whether your main concerns have been addressed or whether any issues remain.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.