- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LESS, a learned set-wise selection framework for improving inference time BoN alignment. Instead of selecting the response with the highest proxy reward using a fixed reward model or other rule, LESS combines reward statistics and semantic embeddings, and employs a set transformer to model interactions among candidate responses. Experiments across several benchmarks and reward models demonstrate improvements over conventional BoN methods.
Critical strengths
Standard Best-of-N selection can suffer from reward miscalibration and reward overoptimization as the candidate pool grows. Learning a data-driven selection rule is a reasonable direction for improving the reliability of inference-time selection. The authors evaluate several benchmarks, reward models, and candidate-set sizes, and include ablation studies and architectural sensitivity analyses.
Critical weaknesses
My main concern is whether the additional set-wise transformer is necessary, especially given the provided experimental results. Although LESS generally outperforms conventional BoN methods, the ablations in Figure 2 suggest that removing the set-wise encoder leads to relatively modest performance drops on several benchmarks, particularly MMLU-Pro and MBPP-Plus. It seems like that the improvements mainly come from using reward statistics and semantic embeddings, rather than explicitly modeling interactions among candidates. An experiment that provides a stronger comparison with a simpler supervised pointwise reranker trained using the same ground truch labels would help clarify the contribution of the proposed set-wise architecture.
I think the generalizability of LESS could be explored further. Most experiments focus on relatively short responses from standard benchmarks with available correctness labels. I am not sure how well the method would work for more complex tasks, such as long-horizon agents, where trajectories can easily exceed the 4,096-token input limit of the reward and embedding models. Besides, the improvements over competitive baselines are sometimes quite small or unstable, especially on GPQA.
Questions for authors
Please refer to the Weaknesses. I would like to increase my score if the authors could address my concersn appropriately.
As the authors have not provided a rebuttal so far, my main concerns regarding the novelty, generalizability, and empirical evaluation of the proposed method remain unaddressed. I therefore maintain my original rating.