acceptodds
Under review as a conference paper at ICLR 2027

Efficient Best-Of- with Radial Consensus Score

Abstract

Solving complex problems with large language models (LLMs) often requires selecting the best answer from multiple candidate responses (best-of-), yet identifying the most reliable one remains challenging, particularly when correctness does not align with majority agreement. Existing approaches based on voting or probability scoring rely on surface-level signals (e.g., frequency or token-level likelihood), often missing high-quality rare answers and ignoring the geometric structure of candidate representations, where correctness may emerge as semantic consistency rather than surface agreement. To address these limitations, we introduce **Radial Consensus Score (RCS)**, a simple, efficient, and training-free method for best-of- selection. RCS models semantic consensus by computing a weighted Frechet mean (semantic center) of answer embeddings and ranking candidates by their radial distance to this center. This enables it to recover semantically coherent correct answers despite lexical fragmentation, even when they are not supported by the majority. Importantly, RCS offers a flexible framework that supports multiple weighting schemes, enabling the integration of arbitrary signals, such as semantic agreement or model confidence, with the position of the center adapting accordingly. Extensive experiments across five open-weight models and various benchmarks demonstrate that our metrics consistently outperform strong baselines by 2–7%, with advantages growing as increases. Moreover, RCS serves as an effective drop-in replacement for majority voting in multi-agent debate and remains effective in black-box scenarios. These results suggest that geometric consensus can serve as a scalable and reliable alternative to voting-based methods for answer selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Best-of-$N$ Selection, Test-time Scaling, Self-Consistency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CT55OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Radial Consensus Score (RCS), a training-free approach for best-of-N answer selection in LLMs. The method embeds candidate answers into a semantic space, computes a weighted Fréchet mean, and selects the candidate closest to the resulting centroid.

Critical strengths
  • The method provides a straightforward, training-free alternative to conventional majority voting, with an efficient centroid-based implementation.
  • The evaluation covers multiple language models, reasoning tasks, and sampling budgets.
  • The framework also allows different confidence and semantic weighting mechanisms to be incorporated without requiring task-specific training
Critical weaknesses
  • The novelty seems a bit limited to me. The proposed selection rule is equivalent to weighted cosine-similarity aggregation or centroid-based. The paper should better establish which components are genuinely novel beyond applying this formulation to final-answer embeddings.
  • Equations (3)–(8) imply equivalent candidate rankings for the proposed weighted medoid and nearest-centroid methods. Nevertheless, the experimental results show different accuracies. This raises concerns about implementation consistency.
  • Proposition 1 establishes convergence of the estimated semantic center but does not establish that the center represents a correct answer.
  • Proposition 2 also appears to conflate centroid-level cluster decisions with candidate-level selection.
  • Some McNemar p-values in Appendix Table 9 appear incompatible with the reported accuracy differences. These results require verification.(eg - Qwen2.5-3B on Arithmetics)
  • RCS primarily uses final-answer embeddings, while some competing methods use full reasoning trajectories. Therefore, the reported improvements do not fully isolate the contribution of the scoring method itself. Could the authors provide details on evaluation of SCW and ModeX using the same final-answer embeddings as RCS?
Questions for authors

1)Why do RCSmedoid and RCSuni produce different results despite the equivalence implied by the equations? 2) Could the authors point out which assumptions ensure that convergence to the semantic centroid implies convergence to a correct answer? 3) the computational complexity discussion omits embedding-encoder inference costs, and the cosine-weighted implementation. is this the case ? if so its not a fair comparison interms of efficiency. 4) How are negative cosine-derived weights handled?

3 comments

Dear Reviewer CT55,

Thanks again for the careful review, which caught two real errors (RCS_medoid row and McNemar table), both now fixed. We also revised Prop. 1-2 and ran SCW and ModeX on the same final-answer embeddings. Do these resolve your concerns?

We are happy to clarify anything further.

Thank you for the response.

I appreciate the authors' acknowledgment and correction of the inconsistencies in the medoid formulation and McNemar statistical analysis, as well as the revisions to Propositions 1 and 2.

However, my primary concern regarding methodological novelty remains. In particular, the acknowledged equivalence between RCS and existing inner-product MBR formulations limits the novelty of the proposed selection mechanism.

Based on the new experiments, it seems to me that SCW and ModeX perform superior when evaluated using the same final-answer embeddings, hinting to me that the originally reported improvements are primarily attributable to the choice of representation rather than the proposed scoring formulation.

Hence it weakens the argument for a distinct methodological advancement- and I remain concerned about whether the resulting contribution offers sufficient novelty for increasing the review score.

Dear Reviewer CT55, thank you for the follow up and for acknowledging the corrections.

We agree that the selection rule is not new, as now mentioned throughout the paper (Sec. 1, 2, Sec. 3.3 and Table 1). Our contributions are instead:

  • Representation is a main finding. The SCW/ModeX results show that embedding final answers rather than trajectories explains most of the gap in Table 3. We have revised Sec. 5.3 to state this explicitly, since it changes how existing methods should be used.
  • A simple, unified baseline. With final-answer embeddings, RCS is the minimal form of consensus-based selection with a single center, exact O(Nd) selection, and no graph selection. As the table shows, it is competitive with both SCW and ModeX without claiming to dominate them.
  • Analysis of when RCS works. Prop. 1-3 and the analyses of the gain source, minority regime and case studies (App. Tables 6, 10, 13-15) characterize when RCS helps and where it fails.

We thank you again for the feedback, which improved the paper's claims and corrected several errors.

Reviewer_xmLHOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RCS, which selects the best answer from multiple LLM responses based on their distance to a semantic center.

Critical strengths
  1. The problem studied in this paper, best-of-N, is meaningful. The proposed method (despite the concerns below about limited novelty and the lack of refinement in the method design) is overall effective.

  2. The authors provide code in the appendix. In the experiments, they validate across models and tasks; the experimental volume is relatively sufficient.

Critical weaknesses
  1. Insufficient novelty. This is the most critical concern. Specifically:
    (i) When handling best-of-N, the authors essentially use the idea of centroid-based MBR decoding. The idea is also very similar to Smoothie: ``Label Free Language Model Routing'' (although that method is for LLM ensemble, it can be directly applied to best-of-N).
    (ii) The several variants proposed by the authors, which mainly handle the weighting part, are not complicated. They exist only as simple processing components.

  2. On the effectiveness of the method. The authors first use an embedding model to obtain vectors of the responses. This approach is not unusable in the current LLM era, but it is no longer advanced. It is relatively difficult for this approach to capture the subtle and fine-grained differences between responses. The theoretical conclusions rely on strong assumptions.

Some other concerns. The authors’ asymptotic theory depends on the candidate vectors having a specific semantic center structure. Even with more samples, the algorithm can only more stably select the answer closest to this center; this itself does not guarantee that the answer is correct. Semantically similar wrong answers can also form a center.

Questions for authors

Please refer to the above comments.

1 comment

Dear Reviewer xmLH, thanks again for your review.

We added a Smoothie comparison (close to a random sample for i.i.d. generations), results with two larger encoders (no improvement over MiniLM), and made the correctness assumption explicit (Assumption 1, App. Table 6). Do these address your concerns on novelty and assumptions?

We are happy to clarify anything further.

Reviewer_AnHcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Radial Consensus Score (RCS), a training-free geometric aggregation method for Best-of-N answer selection in large language models (LLMs). By exploiting embedding geometry, it offers a perspective that differs from conventional majority voting. The method has clear practical value, and the experimental evaluation is reasonably comprehensive. However, the distinction between the paper's main contribution and existing MBR- and centroid-based selection methods needs to be clarified further. In addition, the paper lacks direct empirical validation of semantic fragmentation, which is a central motivation for the proposed method. A direct comparison with MBR, together with a more detailed analysis of the answer spaces in which the method is genuinely effective, would substantially strengthen the paper.

Critical strengths
  1. The problem is important and practically relevant. The paper addresses an important problem in LLM inference scaling: how to select a more reliable answer from multiple sampled responses. As Best-of-N, self-consistency, and multi-agent debate become increasingly common, efficient answer aggregation methods have significant practical value.

  2. The paper offers a simple and intuitive methodological perspective. RCS reformulates answer selection as a consensus-estimation problem in embedding space rather than relying on conventional surface-level voting. By using a semantic center instead of string frequency, it provides a more natural way to handle answers that have different surface forms but convey the same meaning.

  3. The method is efficient and requires no additional training. Unlike reward models, verifiers, or internal probability signals, RCS does not require additional training. Some of its variants can also operate in black-box LLM settings, giving the method promising practical deployment potential.

  4. The experimental evaluation is relatively comprehensive. The authors evaluate the method across multiple open-source models, benchmarks, and sampling budgets. They also compare voting-based, probability-based, and semantic-based methods, resulting in a reasonably broad empirical study.

  5. Some analysis experiments help justify the design choices. For example, the authors compare final-answer embeddings with full-trajectory embeddings. The results suggest that directly modeling reasoning trajectories may be affected by shared prefixes and redundant information, whereas final answers are more suitable representations for consensus estimation.

Critical weaknesses
  1. The distinction between RCS and MBR decoding / centroid-based selection needs further clarification. The core idea of the paper is to estimate a semantic center of the candidate answers using a weighted centroid and then select the candidate closest to that center. This formulation has a strong connection to existing Minimum Bayes Risk (MBR) decoding and centroid-based decoding methods. Although the paper discusses this relationship, it does not provide a direct empirical comparison. It therefore remains unclear whether RCS offers a substantive advantage beyond the existing MBR objective or primarily provides a more efficient implementation.

  2. The paper lacks a sufficiently detailed analysis of the settings in which RCS is applicable. The main motivation for RCS is to address lexical or semantic fragmentation, in which multiple correct answers cannot be effectively aggregated by majority voting because they use different expressions. However, the experiments include mathematical reasoning, multiple-choice questions, and short-answer QA, where the final answer spaces are relatively standardized in some cases. The paper does not yet analyze where the gains from embedding-based consensus come from under different answer structures.

  3. There is no direct validation of the relationship between semantic fragmentation and performance gains. The paper emphasizes that RCS can address surface-form fragmentation, but the current experiments mainly report overall accuracy. They do not provide analyses targeted at this factor. For example, the authors could stratify examples according to the lexical diversity of candidate answers, the number of semantic clusters, or the degree of fragmentation among correct answers, and then test whether RCS obtains its main advantage in high-fragmentation settings.

  4. The analysis of failure cases is insufficient. RCS implicitly assumes that the correct answer is usually located near the center of the candidate-answer distribution. In practice, however, many models may jointly produce the same incorrect answer, while the correct answer may correspond to a semantically isolated, low-probability mode. Although the paper provides theoretical analysis, it lacks a systematic investigation of these failure cases in realistic settings.

Questions for authors
  1. What is the precise relationship between RCS and MBR decoding? The paper states that, under certain conditions, selecting the candidate closest to the weighted centroid is equivalent to MBR decoding. Could the authors clarify the key differences between RCS and existing MBR or centroid-based methods, and provide a direct empirical comparison?

  2. Are the gains from RCS primarily attributable to semantic fragmentation? Could the authors analyze the relationship between RCS's performance gains and factors such as the surface-form diversity of candidate answers and the number of semantic clusters?

  3. Where does RCS's advantage come from on tasks such as mathematics and multiple-choice question answering, where the answer space is relatively small? Because these tasks typically have well-defined standard answers, the theoretical advantage of embedding-based consensus may be limited. Could the authors further analyze the source of the observed improvements on these tasks?

  4. How does RCS avoid an incorrect consensus when the correct answer is not part of the dominant semantic mode? For example, if multiple models produce the same incorrect answer while a minority of models produce the correct answer using different expressions, can RCS still make a reliable selection?

  5. Is the assumption of a single semantic center appropriate for all tasks? For questions with multiple valid answer modes, could the method be extended to support multi-center consensus or mixture-based selection?

  6. What motivates the choice of weighting strategy in RCS? The paper introduces several weighting schemes, including uniform, frequency-based, cosine-similarity-based, and probability-based weighting. However, it remains unclear how to choose the most appropriate consensus definition for different tasks.

3 comments

Dear Reviewer AnHc, thanks again for your review.

Following your Q3, we traced the math gains to unparseable answers and revised our claims (Sec. 5.2, App. Tables 10, 13, 14). We also added the stratified analysis (W2/W3), failure analysis (W4) and kernel multi-center variant (Q5). Do these address your remaining concerns?

We are happy to clarify anything further.

Reviewer_AnHcOpenReview ↗

For authors

Thank you to the authors for addressing my concerns. I maintain my original positive score.

Dear Reviewer AnHc,

Thank you for the response. We appreciate your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We sincerely thank all reviewers for their thoughtful and constructive feedback.

We appreciate the reviewers' recognition of several strengths of our work:

  • Important and practical problem for inference-time scaling (AnHc, xmLH).
  • Simple, training-free, efficient method that handles semantically equivalent answers with some variants work in black-box settings (AnHc, CT55).
  • Broad and comprehensive evaluation across models, benchmarks and sampling budgets, with released code (AnHc, xmLH, CT55).
  • Useful analysis of final-answer vs. full-trajectory embeddings (AnHc).

Below, we address the main concerns raised by the reviewers, with details provided in the individual responses.

  • Relation to MBR and centroid methods: stated the equivalence of selection and added Table 1 positioning RCS (AnHc, xmLH, CT55). Source of the gains: the math gain comes from unparseable answers, and the remaining advantage is on free-text answers. We revised our claims accordingly (AnHc).
  • Theory: added Assumption 1, corrected Prop. 1-2, added Prop. 3 and a failure analysis by vote split (CT55, xmLH, AnHc).
  • Errors fixed: removed the redundant RCS_medoid row and corrected the McNemar table (CT55).
  • New experiments: SCW and ModeX on final-answer embeddings (CT55), Smoothie (xmLH), larger encoders (xmLH), kernel multi-center (AnHc), encoding costs (CT55).

We welcome any further questions before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.