acceptodds
Withdrawn from ICLR 2027

Beyond Fixed Rules: Best-of-N Alignment with Learned Set-wise Selection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Best-of-, which generates candidate responses from an LLM and selects the one with the highest reward-model score, is widely used for inference-time alignment but is susceptible to reward hacking as grows. Some recent work seeks to mitigate this problem by designing proxy-reward correction rules based on signals over the candidate responses. However, these methods encode predefined assumptions about reward-model errors and the rigidity of such fixed-form rules may limit their ability to capture reward-model error patterns that vary across inference-time conditions, making it difficult to reliably distinguish reward-hacked responses from genuinely high-quality ones. To move beyond fixed rules under varying inference-time conditions, we propose LESS, a LEarned Set-wise Selection framework for Best-of- alignment, which learns a tailored selection strategy from the corresponding data to select the best response. It represents each candidate using reward and representation features, then employs a set transformer encoder to capture interactions among the candidates and assign each response a selection logit, returning the highest-scoring candidate as the final output. The set-based design naturally accommodates varying candidate-set sizes. Extensive experiments across diverse benchmarks and varying values of demonstrate the effectiveness of the proposed approach.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Best-of-N, alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BGb9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LESS, a learned set-wise selection framework for improving inference time BoN alignment. Instead of selecting the response with the highest proxy reward using a fixed reward model or other rule, LESS combines reward statistics and semantic embeddings, and employs a set transformer to model interactions among candidate responses. Experiments across several benchmarks and reward models demonstrate improvements over conventional BoN methods.

Critical strengths

Standard Best-of-N selection can suffer from reward miscalibration and reward overoptimization as the candidate pool grows. Learning a data-driven selection rule is a reasonable direction for improving the reliability of inference-time selection. The authors evaluate several benchmarks, reward models, and candidate-set sizes, and include ablation studies and architectural sensitivity analyses.

Critical weaknesses

My main concern is whether the additional set-wise transformer is necessary, especially given the provided experimental results. Although LESS generally outperforms conventional BoN methods, the ablations in Figure 2 suggest that removing the set-wise encoder leads to relatively modest performance drops on several benchmarks, particularly MMLU-Pro and MBPP-Plus. It seems like that the improvements mainly come from using reward statistics and semantic embeddings, rather than explicitly modeling interactions among candidates. An experiment that provides a stronger comparison with a simpler supervised pointwise reranker trained using the same ground truch labels would help clarify the contribution of the proposed set-wise architecture.

I think the generalizability of LESS could be explored further. Most experiments focus on relatively short responses from standard benchmarks with available correctness labels. I am not sure how well the method would work for more complex tasks, such as long-horizon agents, where trajectories can easily exceed the 4,096-token input limit of the reward and embedding models. Besides, the improvements over competitive baselines are sometimes quite small or unstable, especially on GPQA.

Questions for authors

Please refer to the Weaknesses. I would like to increase my score if the authors could address my concersn appropriately.

Reviewer_Em5oOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes LESS, a learned set-wise selection framework for Best-of-NN alignment. The authors argue that existing methods rely on hand-crafted reward correction rules that may not adapt well to different inference-time conditions. To address this, LESS combines reward statistics and prompt-response embeddings, and uses a Set Transformer to assign selection scores to candidate responses. The selector is trained using ground-truth correctness labels with a set-level objective. Experiments on five benchmarks with three reward models show improvements over standard Best-of-NN and several existing selection methods.

Critical strengths
  • The paper addresses a reasonable problem in Best-of-NN alignment, where selecting the response with the highest proxy reward may lead to reward hacking. Learning a selection strategy from data is a straightforward way to explore this problem.
Critical weaknesses
  • I find the novelty and motivation of this work quite limited. The proposed method essentially trains a small supervised reranker using reward scores and embeddings, with a standard Set Transformer architecture. More importantly, LESS requires ground-truth correctness labels from 200 prompts for each benchmark, which is a much stronger assumption than simply using an existing reward model for inference-time alignment. If these labels are available, it is unclear why training another supervised selector is an interesting solution to reward hacking, rather than directly improving the reward model or training a verifier. The cross-dataset transfer experiments in Figure 14 also show that LESS can perform significantly worse than standard BoN under distribution shift. This makes me question whether the method actually addresses reward hacking in general, or simply learns benchmark-specific pattern.
  • I am not convinced that the empirical results justify the additional supervision and computation. While LESS achieves some large improvements in specific settings, the gains on MMLU-Pro, MATH, and IFEval are generally small, often only around 1–2 percentage points. The authors report standard deviations from five random candidate subsamplings, but do not report multiple training seeds or training splits. In particular, when N=32N=32, the subsampling uses the entire fixed candidate pool, so these repetitions do not provide independent estimates. Moreover, LESS introduces an additional 0.6B embedding model and a separately trained selector, but there is no end-to-end compute-matched comparison with existing methods. Overall, I think the practical benefit of this additional complexity is not sufficiently demonstrated.
Questions for authors

Here are my main questions and concerns.

  • What is the main contribution of LESS beyond training a supervised reranker on benchmark-specific correctness labels? The paper includes a reward-model fine-tuning baseline, but the learned selector still benefits from additional supervised training and a separate embedding model. I would like the authors to better justify why this is a general solution to reward hacking rather than simply learning a better verifier for a particular benchmark. I also think the authors should clarify what makes the proposed set-wise selection fundamentally different from standard supervised reranking.
  • How is LESS expected to generalize to new tasks where ground-truth correctness labels are not available? Figure 14 shows that directly transferring the selector across datasets can significantly hurt performance, sometimes even below standard BoN. Although fine-tuning on 50 target samples helps, this still requires additional labels for each new setting. I think this is a major limitation for a method motivated by inference-time alignment, and the authors should explain how LESS can be practically applied beyond the benchmarks used for training.
  • Can the authors provide multi-seed results and a fair comparison of the additional computation? Many of the improvements are quite small, but the current standard deviations only reflect candidate subsampling rather than independent training runs. I would like to see results across different training seeds and data splits. Also, how much additional inference cost does the embedding model introduce compared with standard BoN, Soft BoN, and MBR-BoN? An end-to-end compute-matched comparison would help determine whether the proposed method is actually useful in practice.
1 comment

As the authors have not provided a rebuttal so far, my main concerns regarding the novelty, generalizability, and empirical evaluation of the proposed method remain unaddressed. I therefore maintain my original rating.

Reviewer_jEPTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper works on the reward hacking problem in Best-of-N alignment. The paper proposes LESS (LEarned Set-wise Selection) framework for Best-of-N alignment, which learns a tailored selection strategy from a set of candidates instead of relying on fixed reward-correction heuristics. Specifically, it takes the proxy-reward features and prompt-response embedding for each candidate response, uses a set transformer encoder to capture interactions among N candidates, and trains a prediction head to map the contextualized representation to a scalar calibrated score. This method requires a set of training data with groundtruth reward to train the framework for each particular task. They compare their method with several representative Best-of-N alignment baselines and show better performance.

Critical strengths
  1. This method is a simple and intuitive design. It replaces manually designed reward-correction rules with a learned set-wise selection framework.
  2. It combines proxy-reward statistics with prompt–response semantic embeddings, and ablation studies demonstrate the contributions of both feature types and set-wise contextualization.
Critical weaknesses
  1. The advantage of this method comes with a need for labelled in-domain data to train the whole framework. The heavy dependence on ground-truth reward labels is non-negligible. This method becomes less practical when ground-truth quality judgments are expensive or difficult to define.
  2. The experiments are conducted on in-domain data. It's unclear whether this can generalize beyond the training distribution, either prompt change or response change (e.g., response style change from different models, even with in-domain task). Their experiment shows clear performance degrade transferring across datasets.
  3. The paper's main experiments are conducted on binary tasks, which is a simple and narrow setting of the paper's methodology. And answer correctness may not reflect actual response quality. For open-ended tasks, however, they use another reward model as the oracle and train the framework to select the candidate receiving the highest oracle score, while this oracle can still be imperfect. So in this case, does this framework primarily learn to reproduce another reward model's judgments?
  4. Although the paper's motivation is from reward hacking in best-of-N selection, its method and experiment show little connection to reward hacking.
Questions for authors
  1. Can the authors provide further analysis on generalizability of this method? Specifically, show whether the performance degrades when (i) prompt distribution shift, even within the same task; (ii) response distribution shift, such as responses generated by a different model with distinct style from the training data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.