- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for reducing the cost of evaluating large language models by selecting a smaller subset of questions for annotation. The uncertainty is calculated based on the self-consistency of a smaller LLM based on the consistency of these responses. The questions are then grouped according to their self-consistency, and approximate Neyman allocation determines the number of samples that will be label from each group. The method aims to provide an unbiased estimate of a larger model's performance without requiring additional training. The authors evaluate this approach on many language and multimodal benchmarks and report a 13% average reduction in estimation error compared to uniform sampling, along while saving 15–28% of the labels .
Critical strengths
The problem is important, since reducing the cost of LLM evaluation when obtaining ground-truth labels requires expensive expert annotation. . The proposed approach is simple and practical, combining uncertainty estimates from a smaller LLM with self-consistency. The method is also training-free and can be applied to different language and multimodal models.
Critical weaknesses
The novelty appears limited, as it mainly combines existing ideas from active testing, semantic entropy, and stratified sampling. The proposed allocation strategy lacks a theoretical guarantees of improved efficiency. The method also assumes that uncertainty in a smaller LLM reflects the errors of a larger model, but this assumption is not sufficiently justified. The additional gains over proportional allocation are modest on MMLU-Pro (88.7% to 87.1% relative MSE) and almost negligible on MMBench (95.4% to 95.0%), raising questions about the practical benefit of the method. Finally, generating ten responses per question introduces additional computational costs, and the experiments on existing labeled benchmarks do not fully demonstrate the claimed benefits in realistic expert-annotation settings. Overall, the problem is important and the approach is practical, but I am not fully convinced that the methodological novelty and additional empirical gains are sufficient to justify the contribution.
Questions for authors
Could the authors provide a qualitative analysis of how uncertainty in the smaller surrogate LLM relates to the errors of the larger target model? The authors provide comparisons with adaptations of Berrada et al. (2025), which perform substantially worse than uniform sampling. Could the authors clarify whether this poor performance is an inherent limitation of the previous approach or a consequence of adapting it to multi-token reasoning? Could the authors report the absolute estimation error in accuracy percentage points, rather than only relative MSE? This would make it easier to understand how meaningful the reported improvements are in practice. The authors report computational overhead in Table 3, but the method uses a fixed number of ten surrogate generations per question. How sensitive are the estimation accuracy and computational cost to this choice?
2 comments
Dear Authors,
Thank you for trying to address my comments. I now appreciate the paper more. However: This type of pipelines are usually accompanied by some kind of theoretical guaranties. This would boost the contribution and soundness significantly. I am still sceptical about the paper soundness since the proposed method, as its effectiveness relies heavily on the assumption that uncertainty signals produced by a smaller surrogate model are sufficiently informative about the error distribution of a larger target model. thus it might provide unreliable signals and lead to suboptimal budget allocation that will hurt the performance. Finally while the problem is important and practically relevant. I still believe the contribution of the paper is incremental to the previous work that the authors very well identify.
Overall, I appreciate the authors' clarifications and efforts, but these concerns prevent me from substantially changing my assessment of the paper.
Dear Reviewer,
Thank you for the careful review. We are grateful for this opportunity to address and explain your thoughtful questions.
Response to Weaknesses (Contribution): We agree that several things in our method are classical. However, we view our contributions as follows in short: 1) We study the failure modes of prior art extensions. 2) We re-design the whole active testing pipeline, which is largely different from prior art (e.g. Berrada et al. 2025 [1]), as we list the differences in the reply to Reviewer KHMq. 3) We show practical results in the multi-token generation setting, which is essentially the dominant use case in the LLM era.
Response to Question 1 (Concern about smaller surrogate):
As we reply to Reviewer SGgD, we conduct additional experiments on two smaller surrogates (SmolVLM2 500M & Qwen3.5-0.8B). Combined with previous results, the table reports five different surrogates estimating GPT-5.4 as a target model on GPQA-Diamond:
*Relative MSE (% of Uniform Sampling, lower is better).
We can see that the performance struggles with Qwen3.5-2B or weaker as a surrogate, but tends to be helpful when Qwen3.5-4B or GPT-5.4 Nano is the surrogate. Although comprehensive experiments are deferred to the camera-ready version, we believe we have provided a preliminary conclusion that once the surrogate surpasses a certain capability threshold, our method becomes highly effective. It is also worth noting that, in our experiments even with incapable surrogates, our framework gracefully degrades to the uniform sampling baseline (100% relative MSE is equal to uniform sampling), instead of being harmful to the evaluation. We believe that this is quite useful in practical deployment.
Response to Question 2 (poor performance of prior art): The poor performance is a consequence of adapting it to multi-token reasoning. In the original paper of Berrada et al. 2025, their setting is to use an LLM as a classifier (which restricts the LLM to outputting only one token). Our setting is the multi-token generation setting without that restriction because that's the dominant use case in the LLM era. We first try to naturally extend Berrada et al. 2025 in this setting and then study its failure modes.
Response to Question 3 (absolute value of estimation error):
We report mean absolute error (MAE) in percentage points at two label budgets. For example, if full-pool accuracy is 80%, 6.97 in the table means the average absolute gap between estimation and oracle-80% is 6.97 percentage points. And please note that, as the label budget increases, the absolute value of estimation error decreases naturally.
Budget M=20:
Budget M=100:
Response to Question 4 (choice of k):
Regarding the choice of K, we conduct an additional ablation study on GPQA-Diamond as follows:
*Relative MSE (% of Uniform Sampling, lower is better). Target is GPT-5.4, surrogate is Qwen3.5-4B. Budget M is 100.
We view K=10 as a sweet spot between surrogate signal diversity and computational overhead. We plan to conduct comprehensive experiments over benchmarks in the camera-ready version to give a bigger picture of this question.
[1] Gabrielle Berrada et al. “Scaling Up Active Testing to Large Language Models.” NeurIPS 2025.