- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Radial Consensus Score (RCS), a training-free approach for best-of-N answer selection in LLMs. The method embeds candidate answers into a semantic space, computes a weighted Fréchet mean, and selects the candidate closest to the resulting centroid.
Critical strengths
- The method provides a straightforward, training-free alternative to conventional majority voting, with an efficient centroid-based implementation.
- The evaluation covers multiple language models, reasoning tasks, and sampling budgets.
- The framework also allows different confidence and semantic weighting mechanisms to be incorporated without requiring task-specific training
Critical weaknesses
- The novelty seems a bit limited to me. The proposed selection rule is equivalent to weighted cosine-similarity aggregation or centroid-based. The paper should better establish which components are genuinely novel beyond applying this formulation to final-answer embeddings.
- Equations (3)–(8) imply equivalent candidate rankings for the proposed weighted medoid and nearest-centroid methods. Nevertheless, the experimental results show different accuracies. This raises concerns about implementation consistency.
- Proposition 1 establishes convergence of the estimated semantic center but does not establish that the center represents a correct answer.
- Proposition 2 also appears to conflate centroid-level cluster decisions with candidate-level selection.
- Some McNemar p-values in Appendix Table 9 appear incompatible with the reported accuracy differences. These results require verification.(eg - Qwen2.5-3B on Arithmetics)
- RCS primarily uses final-answer embeddings, while some competing methods use full reasoning trajectories. Therefore, the reported improvements do not fully isolate the contribution of the scoring method itself. Could the authors provide details on evaluation of SCW and ModeX using the same final-answer embeddings as RCS?
Questions for authors
1)Why do RCSmedoid and RCSuni produce different results despite the equivalence implied by the equations? 2) Could the authors point out which assumptions ensure that convergence to the semantic centroid implies convergence to a correct answer? 3) the computational complexity discussion omits embedding-encoder inference costs, and the cosine-weighted implementation. is this the case ? if so its not a fair comparison interms of efficiency. 4) How are negative cosine-derived weights handled?
3 comments
Thank you for the response.
I appreciate the authors' acknowledgment and correction of the inconsistencies in the medoid formulation and McNemar statistical analysis, as well as the revisions to Propositions 1 and 2.
However, my primary concern regarding methodological novelty remains. In particular, the acknowledged equivalence between RCS and existing inner-product MBR formulations limits the novelty of the proposed selection mechanism.
Based on the new experiments, it seems to me that SCW and ModeX perform superior when evaluated using the same final-answer embeddings, hinting to me that the originally reported improvements are primarily attributable to the choice of representation rather than the proposed scoring formulation.
Hence it weakens the argument for a distinct methodological advancement- and I remain concerned about whether the resulting contribution offers sufficient novelty for increasing the review score.
Dear Reviewer CT55, thank you for the follow up and for acknowledging the corrections.
We agree that the selection rule is not new, as now mentioned throughout the paper (Sec. 1, 2, Sec. 3.3 and Table 1). Our contributions are instead:
- Representation is a main finding. The SCW/ModeX results show that embedding final answers rather than trajectories explains most of the gap in Table 3. We have revised Sec. 5.3 to state this explicitly, since it changes how existing methods should be used.
- A simple, unified baseline. With final-answer embeddings, RCS is the minimal form of consensus-based selection with a single center, exact O(Nd) selection, and no graph selection. As the table shows, it is competitive with both SCW and ModeX without claiming to dominate them.
- Analysis of when RCS works. Prop. 1-3 and the analyses of the gain source, minority regime and case studies (App. Tables 6, 10, 13-15) characterize when RCS helps and where it fails.
We thank you again for the feedback, which improved the paper's claims and corrected several errors.
Dear Reviewer CT55,
Thanks again for the careful review, which caught two real errors (RCS_medoid row and McNemar table), both now fixed. We also revised Prop. 1-2 and ran SCW and ModeX on the same final-answer embeddings. Do these resolve your concerns?
We are happy to clarify anything further.