- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 1(−1)
- Presentation
- 2(−1)
- Soundness
- 1(−1)
Read review
Paper summary
This paper asks how we can tell whether an LLM's answer is reliable without looking inside the model. The idea is intuitive: the target model answers first, then a few smaller models from the same family answer the same question, and we check how much they agree with the target model. The more the smaller models arrive at the same answer, the more reliable it is considered. The authors plot this agreement along model size and use the area under the curve as the score, which they call AnchorAUC. Experiments on five QA datasets, four target models and two sets of small models show that it outperforms several baselines at a much lower cost. But this work still miss some components and need to clarify something.
Critical strengths
The presentation of this paper is visually appealing, and the overall structure is complete. The main text is concise and easy to follow. The proposed method is simple and low-cost, the evaluation covers a broad range of settings, and the ablation studies are well designed and directly address the key claims.
Critical weaknesses
-
While checking the tables, I noticed an unusual pattern. In Table 1, the SESDLG–SE difference is identical across all four anchor models (for example, 9.0 on TriviaQA), and in Table 2, TU's AUROC on the two closed-source anchors differs by exactly 0.5 on every dataset. I may be misunderstanding how the baselines were computed, and I would appreciate it if the authors could clarify how these numbers were obtained.
-
As far as I can tell, the method can be viewed as a weighted average of three probes' agreement, and the gain over the plain mean is 2.3 AUROC points, so I have some reservations about the degree of novelty.
-
I am a bit concerned that the score mostly reflects how hard a question is for small models. Cases where the anchor is correct but a probe is wrong account for 14%–21% of examples, and these correct answers would receive high uncertainty.
-
A few case studies of failure cases would help readers understand when the method breaks down. Besides, the current tasks are all short-answer QA. I would be curious to see how the method performs on long-form generation or mathematical reasoning. Additionally, the correctness labels come mainly from an LLM judge. A small human-checked sample with agreement statistics would make the results more convincing.
Questions for authors
Hi, if you can add some experiments as mentioned in weakness and clarify some concerns, I'm willing to improve my score. Your presentation is very good.
5 comments
Responses to w3
3. On the Concern That the Score Primarily Reflects Question Difficulty for Smaller Models
Thank you for raising this important concern. We would like to clarify that 14%–21% represents the average proportion of cases where the Anchor is correct but a Probe is incorrect, rather than the false-positive rate of the final score. An incorrect answer from an individual Probe does not necessarily lead to high uncertainty, as the final score depends on the consistency between multiple Probes and the Anchor,.
As you pointed out, our method does not completely separate question difficulty from answer reliability, since we believe question difficulty itself serves as a useful risk signal:
greater task difficulty is generally associated with a higher risk of incorrect answers.
Therefore, the key question is:
whether scale-related behavioral signals can still help distinguish correct and incorrect Anchor answers, even when smaller Probes have lower standalone accuracy.
To investigate this, we conducted additional experiments using Qwen3-14B as the Anchor and Qwen3-8B, Qwen3-4B, and Qwen3-1.7B as Probes on more challenging mathematical and reasoning tasks. The results show that our method can retain meaningful discriminative ability even when the smaller model has substantially lower accuracy than the Anchor:
| Task Type | Task | 8B Accuracy (%) | 14B Accuracy (%) | AUROC (%) |
|---|---|---|---|---|
| Mathematical Reasoning | MATH-500 | 89.18 | 92.27 | 90.97 |
| Mathematical Reasoning | BBH-Object Counting | 46.40 | 56.40 | 79.06 |
| Logical Reasoning | BBH-Five-Object Logical Reasoning | 58.80 | 62.00 | 73.71 |
For example, on BBH Object Counting, the 8B Probe achieves only 46.40% accuracy, compared with 56.40% for the 14B Anchor, yet our method achieves an AUROC of 79.06%. Similarly, on BBH Five-Object Logical Reasoning, the 8B Probe achieves 58.80% accuracy, while our method still achieves an AUROC of 73.71%. These results suggest that the usefulness of Probes does not simply depend on their standalone accuracy or their ability to answer every question correctly.
We nevertheless acknowledge that when a task substantially exceeds the capabilities of multiple Probes (e.g., when the accuracy of all probes approaches zero), their behavioral signals may become less informative. We have clarified this point in the revised manuscript.
Responses to w4
4.1 Failure Case Analyses
We identify two major types of failure cases.
Case 1: Shared model errors lead to low uncertainty for an incorrect answer.
Question: Alice and Bob each have a bag containing five balls, one in each of the following colors: blue, green, orange, red, and purple. Alice randomly transfers one ball to Bob's bag. Bob then randomly selects one ball from his bag and returns it to Alice. What is the probability that the two bags have identical compositions at the end?
Correct answer: .
Model responses:14B, 8B, and 4B: ; 1.7B:
The anchor and the two stronger probes share the same incorrect answer. Consequently, the mathematical-equivalence matching variant assigns this incorrect anchor answer a low uncertainty score of .
Case 2: Errors introduced by the NLI evaluator itself.
Question: When was Episode 122 of Dragon Ball Super aired?
Correct answer: Japanese version: January 7, 2018. English version: August 3, 2019.
Model responses: 14B: "The main series has only 121 episodes, so there is no official release date for episode 122." 8B: "The original series ended with episode 122, which aired on July 1, 2015."
Neither response matches the reference answers, and the two responses contradict each other. However, the original full-text NLI evaluator assigns them a bidirectional average retention score as high as 0.9785, causing the incorrect anchor answer to fall into the low-uncertainty region. This case illustrates that the NLI evaluator itself can also be a source of errors in our framework.
4.2 Performance on Mathematical Reasoning and Long-Form Generation Tasks
Mathematical reasoning.
We added experiments on MATH-500 using Qwen3-14B as the anchor and Qwen3-8B, 4B, and 1.7B as probes.
| Task | 8B Accuracy (%) | 14B Accuracy (%) | AUROC (%) | AUARC (%) |
|---|---|---|---|---|
| MATH-500 | 89.18 | 92.27 | 90.97 | 98.65 |
Long-form generation.
For long-form generation, we conducted closed-book generation experiments on 200 questions from ASQA. The primary bottleneck in this setting is the context-length limitation of the NLI evaluator. The nli-deberta-v3-base model used in our original experiments has a maximum input length of 512 tokens. We investigate two potential solutions:
- Sentence-level matching: Split the original answers into individual sentences, compute NLI scores at the sentence level, select the maximum matching score, and then average the resulting scores.
- Using a longer-context evaluator: Replace the original NLI evaluator with a longer-context discriminator, such as Kev-Qwen3-4B, a JEV-style open-source reproduction.
| Method | AUROC (%) | AUARC (%) |
|---|---|---|
| nli-deberta-v3-base, full-answer matching | 55.77 | 9.15 |
| nli-deberta-v3-base, sentence-level matching | 61.81 | 10.56 |
| Kev-Qwen3-4B, full-answer matching | 78.24 | 18.01 |
4.3 Human Validation of Correctness Labels
We randomly sampled 50 examples from each of the four datasets that use LLM-assisted correctness evaluation and manually verified their correctness labels. We then compared the human annotations with the automatically assigned labels on an example-by-example basis.
Among the 200 manually checked examples, 193 labels were consistent with the automatic evaluations, yielding an overall agreement rate of 96.5%. The results for individual datasets are reported below.
StrategyQA was excluded from this validation because its correctness labels are determined using deterministic rule-based evaluation rather than an LLM judge.
| Dataset | Number of Verified Examples | Agreement between Automatic and Human Labels | Agreement Rate |
|---|---|---|---|
| TriviaQA | 50 | 50 | 100% |
| Natural Questions | 50 | 48 | 96% |
| HotpotQA | 50 | 49 | 98% |
| CoQA | 50 | 46 | 92% |
| Total | 200 | 193 | 96.5% |
Responses to w1 and w2
We thank the reviewer for the positive feedback on the presentation of our work. We have further polished Figure 1 to enhance its visual appeal and make the overall presentation more intuitive and engaging. We address the reviewer's comments and questions point by point below.
1. Unusual Numerical Patterns in Tables 1 and 2
Thank you for pointing out these unusual numerical patterns. In an earlier version of our experiments, some results were still pending. We temporarily used placeholder values based on fixed differences for these entries. After the corresponding experiments were completed, we obtained the actual results, but inadvertently failed to update all the placeholder entries in the final LaTeX tables.
We have now replaced these placeholders with the actual experimental results, including the SE_SDLG results for Phi-4-14B, OLMo2-13B, and Mistral-Nemo-12B in Table 1, and the TU results for Claude Opus 5 and GPT-5.6 Sol in Table 2. All corresponding averages have also been recalculated. These discrepancies were confined to the reported tables and did not affect the metric calculations, or method implementations. We have cross-checked the corrected values against the original experimental records, and the main conclusions remain unchanged.
We apologize for this oversight and sincerely thank the reviewer for identifying it. The corrected tables are included in the revised manuscript.
2. On the Concern That AnchorAUC Is Essentially a Weighted Average
We do not deny the basic principle of weighted averaging itself, but it is worth emphasizing that, prior to our work, no study had organized models of different scales within the same model family into an ordered probe sequence, even using simple averaging. This organization is itself a new attempt, allowing us to systematically examine how model behavior changes with scale while controlling for differences across model families.
On this basis, we further recognize that simple averaging may overlook the differences in information carried by models of different scales, and find that incorporating model scale into the aggregation weights can further improve method performance. Furthermore, we systematically examine semantic retention, correctness changes, and behavioral transitions between neighboring model scales, revealing systematic associations between changes in model scale, answer semantics, and correctness. These findings not only provide empirical support for scale-aware aggregation strategies but also further demonstrate that models at different scales are not simply interchangeable independent voters; their relative scales and behavioral changes themselves contain valuable information about answer reliability.
Therefore, we believe the novelty of our work lies not merely in the weighted averaging itself, but more importantly in the scale-ordered probe organization and the systematic use of scale-related behavioral information to assess answer reliability.