acceptodds
Under review as a conference paper at ICLR 2027

What English-Only, Non-Frontier Evaluation Misses About Consistency-Based Confidence Estimation

Abstract

Confidence estimation helps users decide when to trust an AI model's answer. Most common black-box confidence estimation methods, such as self-consistency, measure agreement among answers sampled from the model given the question. Analyzing recent work, we find that research results on consistency-based methods largely come from English-only evaluations of small models, leaving other languages and recent models under-explored. We therefore evaluate consistency-based confidence estimation across nine languages and a range of small open-source to frontier models. Our results show that self-consistency performs worse outside English for all 17 models we tested. Moreover, verbalized confidence using recent Claude models outperforms self-consistency, despite using only one LLM call. For a more effective consistency-based approach across models and languages, we introduce Rosetta, which uses translation as a principled perturbation method and measures agreement between the resulting multilingual samples and the original answer to estimate confidence. Rosetta improves confidence estimation over same-language rewording in English and outside English at a matched call budget and without labeled calibration data, while narrowing the average language gap.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty Estimation, Confidence Estimation, Large Language Models, Multilingual Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Wp5KOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors tackle the problem of black-box LLM confidence calibration. They focus on frontier models (which they show are understudied in recent calibration literature) and evaluate on multiple languages. They introduce a method called Rosetta, which computes self-consistency under translated prompts. They show that with frontier models on English and non-English datasets, Rosetta has better calibration than self-consistency under same-language prompt perturbations.

Critical strengths
  • Presents a simple and well-motivated method for black-box LLM confidence calibration, outperforming baselines. They have an insightful analysis showing that Rosetta leverages the multilingual capabilities of modern LLMs, with Rosetta’s calibration improving with a given model’s non-English accuracy.
Critical weaknesses
  • Does not consider settings with long-form generation or reasoning (CoT). This is in spite of the fact that the two datasets shared across all languages (PolyMath and MMLU-ProX) are meant to be solved with CoT.
  • The method relies on the multilingual proficiency of the LLM being used. This is probably one reason why the paper focuses less on non-frontier models, since their multilingual capabilities are weaker.
Questions for authors
  • How does Rosetta compare to DINCO (Wang and Stengel-Eskin, 2026)? DINCO uses verbalization consistency as a complementary signal to generation consistency, beating generation consistency alone, whereas generation consistency is the main family considered in this paper; vanilla verbalized confidence is the only verbalization-based method considered and shows catastrophic performance in almost all cases. It would be interesting to compare your current single-language and multi-language methods (Paraphrase and Rosetta) with versions that incorporate a DINCO-like verbalization component. A simple way to do this would be to have each of the k prompts ask for a verbalized confidence after the predicted answer, as in the vanilla prompt in Xiong et al. (2024). Then you can average the verbalized confidence over duplicates of a given unique answer, then normalize (with denominator >= 1) verbalized confidence over unique answers. The normalized verbalized confidence can then be combined with the usual generation consistency confidence. Maybe you want to have the LLM do this processing in the last call as in your current setup. I would like to see these comparisons for at least a subset of the models and datasets.
  • In the abstract, the framing of translation as a “principled” perturbation method is not justified. It is not clear why translation is more principled than using no perturbation or same-language question perturbations.

References:

Wang and Stengel-Eskin, 2026. Calibrating Verbalized Confidence with Self-Generated Distractors.

Xiong et al., 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Wp5K (1/2)

We appreciate the reviewer’s comment that Rosetta is a "simple and well-motivated method", and for highlighting our "insightful analysis" showing that Rosetta leverages the multilingual capabilities of modern LLMs.


Q1: How does Rosetta compare to DINCO (Wang and Stengel-Eskin, 2026)? [..] I would like to see these comparisons for at least a subset of the models and datasets.

Summary: We implemented your suggested hybrid step and also ran DINCO. Combining normalized verbalized confidence with agreement improves self-consistency, Paraphrase, and Rosetta. Rosetta +NVC or Rosetta +DINCO are the best methods in 10 of 12 cases across both models, both datasets, and three metrics; the two exceptions are ECE, where DINCO is better.

We see hybrid methods such as DINCO as orthogonal to our method and analysis.

As requested, Appendix E.7 of our revised manuscript includes results for both DINCO and your proposed hybrid step.

  1. DINCO with 8 samples and 8 self-generated distractors, matching Rosetta's eight panel answers, and using an LLM in place of the NLI model for grouping equivalent answers. We also test integration of Rosetta with DINCO in the hybrid setup. Rosetta +DINCO replaces DINCO's self-consistency component with Rosetta's agreement score and keeps the rest unchanged.

  2. Hybrid method (your proposal) applies normalized verbalized confidence (NVC) to the three consistency-based methods in our paper. We elicit the verbalized confidence of panel answers by asking the LLM to assign a confidence score.

We evaluate DINCO and Hybrid on MMLU-ProX in English, Chinese, Japanese, and Arabic and on open-ended questions in PopQA with GPT-5.6 Luna and Sonnet 4.6.

Takeaway: The hybrid method improves the Brier score for every sampling method. Rosetta with NVC outperforms all other methods on Brier, ECE, and AUROC, except for DINCO's ECE on Sonnet 4.6.

MMLUPROX:

The table below reports mean Brier ↓\downarrow / ECE ↓\downarrow / AUROC ↑\uparrow, across the four languages.

MethodGPT-5.6 LunaSonnet 4.6
Verbalized confidence29.1 / 28.5 / 63.621.5 / 12.6 / 71.3
Paraphrase20.5 / 17.9 / 78.325.9 / 23.4 / 66.2
Paraphrase +NVC17.1 / 11.1 / 82.020.7 / 14.4 / 76.1
DINCO18.7 / 9.4 / 77.020.4 / 7.7 / 72.3
Rosetta17.7 / 14.2 / 81.621.3 / 16.5 / 74.6
Rosetta +DINCO17.9 / 12.3 / 79.918.6 / 8.3 / 76.3
Rosetta +NVC15.5 / 8.2 / 84.218.3 / 9.4 / 78.5

POPQA:

Takeaway: Rosetta with NVC achieves the best Brier score across both models. For AUROC, Rosetta + DINCO was the top performer for Sonnet, while Rosetta + NVC performed best on Luna.

The table below reports Brier ↓\downarrow / ECE ↓\downarrow / AUROC ↑\uparrow on PopQA.

MethodGPT-5.6 LunaSonnet 4.6
Verbalized confidence35.1 / 36.6 / 77.023.1 / 21.7 / 81.2
Paraphrase20.8 / 18.4 / 81.122.3 / 20.8 / 80.7
Paraphrase +NVC17.8 / 8.3 / 82.918.1 / 11.6 / 83.5
DINCO18.2 / 5.4 / 80.120.3 / 7.9 / 77.6
Rosetta18.2 / 10.1 / 81.817.6 / 8.5 / 82.6
Rosetta +DINCO18.2 / 10.6 / 82.417.5 / 10.1 / 83.7
Rosetta +NVC17.3 / 7.8 / 83.217.3 / 6.9 / 82.3

Thank you for these suggested experiments. Your proposed hybrid method with NVC helps further demonstrate Rosetta’s generalizability and effectiveness.

Q2: In the abstract, the framing of translation as a “principled” perturbation method is not justified. It is not clear why translation is more principled than using no perturbation or same-language question perturbations.

We agree that “principled” is not sufficiently justified here. Our revised manuscript now uses “multilingual perturbation” in place of “principled perturbation”.

AuthorsOpenReview ↗

Response to Reviewer Wp5K (2/2)

W1: Does not consider settings with long-form generation or reasoning (CoT). This is in spite of the fact that the two datasets shared across all languages (PolyMath and MMLU-ProX) are meant to be solved with CoT.

Summary: We initially excluded CoT generation due to its prohibitively high cost. Our revised manuscript now adds an evaluation with chain-of-thought prompting on MMLU-ProX and PolyMath. Averaged across the six languages and two models, Rosetta achieves the best mean Brier and ECE.

We did not initially evaluate CoT because of its high cost. Figure 3 alone covers 17 models, 9 languages, 2 datasets, and 5 methods (>1,500 setup combinations). Duplicating these experiments with CoT would roughly cost $20K at API list prices. Nonetheless, Appendix E.8 of our revised manuscript now includes CoT evaluation for GPT-5.6 Luna and Gemma 4 31B on both MMLU-ProX and PolyMath. Each cell below reports Brier ↓\downarrow / ECE ↓\downarrow / AUROC ↑\uparrow, averaged across six languages: English, Chinese, Arabic, Spanish, Russian, and Japanese.

Takeaway: Rosetta retains its advantage with chain of thought, achieving the lowest mean Brier and ECE in all four comparisons, although the winning margins are smaller.

Brier ↓\downarrow / ECE ↓\downarrow / AUROC ↑\uparrow

DatasetModelSelf-consistencyRosetta-AOParaphraseRosetta
MMLU-ProXGPT-5.6 Luna15.1 / 12.4 / 78.913.5 / 9.0 / 82.313.9 / 10.5 / 81.612.6 / 7.3 / 84.7
Gemma 4 31B16.8 / 15.8 / 68.616.1 / 13.8 / 69.315.8 / 13.8 / 70.314.9 / 12.2 / 73.8
PolyMathGPT-5.6 Luna16.3 / 19.2 / 90.714.6 / 14.7 / 90.115.4 / 17.1 / 90.213.8 / 13.9 / 90.5
Gemma 4 31B15.0 / 12.3 / 81.314.9 / 11.8 / 81.814.5 / 10.6 / 81.914.4 / 9.4 / 82.6

W2: The method relies on the multilingual proficiency of the LLM being used. This is probably one reason why the paper focuses less on non-frontier models, since their multilingual capabilities are weaker.

Summary: Rosetta relies on multilingual proficiency, but its benefits are not limited to frontier models. Rosetta achieves a lower Brier score than both baselines outside English for nine of ten open models from 2B to 31B. We showed AUROC gains depend on multilingual proficiency, with improvements over both baselines for every tested model with at least 30% non-English accuracy (Section 5.2).

We agree that Rosetta relies on the model's multilingual capability. Still, our results show that Rosetta improves calibration for non-frontier models. Our evaluation includes ten open models from 2B to 31B. Outside English, Rosetta achieves a lower Brier score than both baselines for nine of ten open models, with a mean Brier improvement of 8.0 points over self-consistency and 3.5 points over Paraphrase.

As mentioned in Section 5.2, AUROC gains depend more on multilingual proficiency. Rosetta improves AUROC over both baselines for all eleven models with at least 30% non-English accuracy and even two small Gemma models achieve AUROC gains comparable to frontier models like GPT-5.4 (Table 3):

ModelAUROC gain over self-consistencyAUROC gain over Paraphrase
Gemma 4 12B+5.8+4.7
Gemma 4 31B+9.1+2.2
GPT-5.4+6.0+2.2

We hope these clarifications and additional experiments address your concerns. Thank you again for your time and feedback.

Thank you for taking the time to run additional experiments to address my comments. I am happy with the authors' response and will keep my positive score.

Reviewer_eZLWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Rosetta which uses agreement between question/answers in different languages as a black-box confidence estimation methods. On multiple datasets and models, the paper demonstrates improved AUROC, Brier, and ECE over existing methods.

Critical strengths
  1. The paper investigates an interesting research question and the topic is valuable to the community.
  2. Extensive results on multiple models and datasets demonstrates the improvement over self-consistency and paraphrase.
Critical weaknesses
  1. The description from ln41 to ln46 together with the observation in Figure 1 relates to the use of frontier/non-frontier models and English/non-English data for LLM research in general, while the paper uses it to motivate the research gap in consistency-based confidence estimation methods. For a more concrete motivation, the statistics and Figure 1 should also be collected from papers on consistency-based confidence estimation methods.
  2. The first contribution relates the gap in confidence estimation between English and non-English tasks. This, however, is a trivial observation with English being the primary languages on which those models are trained.
  3. Under Rosetta, confidence estimation also becomes a function of languages and models' capability in them. That is, confidence estimation now approximates the probability of correctness of the answer to a question in a particular language. There seems to be a lack of discussion around this. Please see questions.
Questions for authors
  1. Does Rosetta require models to have similar capability between the original language and the additional tested languages? When models have uneven capabilities between the original language and the additional languages, could it lead to poorly calibrated confidence estimates?
  2. The answer agreement evaluation can also be problematic, as it requires model to correctly recognize answers in one languages matches the other. In practice, there are often answers that is unique to a particular language and do not have direct translations to other languages.
  3. Is k=8 a common choice used in previous consistency-based methods? How does method perform over different values of k?
5 comments
AuthorsOpenReview ↗

Response to Reviewer eZLW (1/4)

We thank the reviewer for finding our research question "interesting" and "valuable to the community". We provide additional experiments and corrections based on your feedback and our revised manuscript.

W1: The description from ln41 to ln46 together with the observation in Figure 1 relates to the use of frontier/non-frontier models and English/non-English data for LLM research in general, while the paper uses it to motivate the research gap in consistency-based confidence estimation methods. For a more concrete motivation, the statistics and Figure 1 should also be collected from papers on consistency-based confidence estimation methods.

Our survey only included consistency-based methods papers. Appendix J of our original submission explained the survey’s focus on consistency-based confidence estimation in detail. Our updated manuscript makes this scope explicit and clear (ln44).

W2: The first contribution relates the gap in confidence estimation between English and non-English tasks. This, however, is a trivial observation with English being the primary languages on which those models are trained.

We agree that an English–non-English confidence-estimation gap may be unsurprising. Our contribution goes beyond documenting its existence. We systematically evaluate 17 models across nine languages to investigate the gap and compare how different confidence estimators handle it.

Beyond quantifying the gap, our main results and additional analyses reveal two findings about its nature:

  • We investigate whether lower answer accuracy explains the gap. To test this explanation, we compare English and non-English on the same questions at equal accuracy. We retain only questions answered correctly in both languages or incorrectly in both. Self-consistency’s mean AUROC gap remains at 2.9 points, compared with 2.7 before matching. The confidence-estimation gap therefore persists even when answer accuracy is equal across languages (Appendix E.5).
  • The evaluation provides a common testbed for comparing confidence estimators across languages. We compare estimators on the same models and answers, allowing us to compare their performance in English and outside English. Self-consistency and Paraphrase have mean AUROC gaps of 2.7 and 2.2 points, respectively. Verbalized confidence has almost no gap (0.2 points), while Rosetta reverses the mean gap. These comparisons show how the choice of estimator affects performance across languages and provide a common basis for evaluating improvements.

W3: Under Rosetta, confidence estimation also becomes a function of languages and models' capability in them. That is, confidence estimation now approximates the probability of correctness of the answer to a question in a particular language. There seems to be a lack of discussion around this.

We agree that Rosetta's confidence estimates are affected by the model's multilingual performance and by the choice of panel languages. Our revised manuscript now states this dependency clearly in new Section 5.4.

What Rosetta estimates: Estimating "the probability of correctness of the answer to a question in a particular language" is not unique to Rosetta. Verbalized confidence, self-consistency, and Paraphrase also estimate the correct probability, for the language of the original question and answer. Rosetta estimates the same quantity.

How the languages enter: The only difference for Rosetta’s confidence estimate is that it also depends on the model’s performance in the kk panel languages. Appendix E.2 compares three panel language sets, and our answer to Q1 below measures the effect of uneven capability between the original language and the panel languages.

AuthorsOpenReview ↗

Response to Reviewer eZLW (2/4)

Q1: Does Rosetta require models to have similar capability between the original language and the additional tested languages? When models have uneven capabilities between the original language and the additional languages, could it lead to poorly calibrated confidence estimates?

Summary: Rosetta does not require or assume similar capability between the original language and the panel languages. However, the model’s ability in the panel languages affects its confidence estimates. Our original results already include cases with uneven capability. Our new analysis shows better mean Brier and ECE than Paraphrase with both more accurate and less accurate panel languages (new Section 5.4).

To test whether Rosetta’s gains remain when language capability is uneven, we analyze the effect of language capability differences already present in our MMLU-ProX results. We quantify capability by measuring how accurately the model answers the dataset’s questions in each language. More capable languages have higher accuracy. Section 5.1 experiments test MMLU-ProX over 9 languages in a leave-one-out setting. Put simply, one language is selected as the anchor language and the remaining 8 languages serve as the panel. We then measure the language capability gap by taking the accuracy difference between the anchor language and the average of the 8 panel languages. A positive gap of xx points means that, on average, the panel languages’ average accuracy is xx points higher than the accuracy of the anchor language. If x<0x < 0, that means the model is more accurate in the anchor language than the average of the panel languages. For each model and original language, we find the largest accuracy difference between the original language and any single panel language. The “Largest gap” column reports the average size of these differences within each group.

The table below stratifies results by this language capability gap xx.

Takeaway: Rosetta improves mean Brier and ECE, across all values of xx. When panel languages are more accurate (x>0x > 0), Rosetta improves all three metrics on average. When x<0x < 0, Rosetta’s mean Brier and ECE still improve, with slightly worse AUROC compared to Paraphrase.

Avg. Capability Gap (xx)Largest gap (mean)Brier gainECE gainAUROC gainBetter Brier than Paraphrase
xx > 7 points15.2+5.7+6.1+7.2100%
5 points < xx <= 7 points10.5+3.8+6.2+1.6100%
-5 points <= xx <= 5 points7.1+2.7+4.7+2.289%
-7 points <= xx < -5 points11.1+2.1+6.1−1.589%
xx < -7 points12.3+3.7+8.0−0.8100%

We see the same pattern even in cases with the largest positive and negative capability gaps. Each cell reports Brier ↓\downarrow / ECE ↓\downarrow / AUROC ↑\uparrow.

Model, question languageAccuracy: question lang → panel langsAccuracy gapParaphraseRosetta
Gemma 4 12B, Japanese33.0 → 52.1+19.142.8 / 43.5 / 60.532.1 / 31.6 / 72.2
Granite 4.1 8B, English48.8 → 40.5−8.335.6 / 36.3 / 69.231.4 / 29.2 / 67.5

These results show that similar capability is not necessary for Rosetta’s gains. They also highlight a practical benefit for Rosetta. Rosetta takes advantage of languages where the model is more accurate to improve confidence estimation in languages where it is less accurate.

Thank you for prompting this analysis. It shows that Rosetta’s gains do not require equal capability across languages. Our updated manuscript now discusses these findings in the new Section 5.4

AuthorsOpenReview ↗

Response to Reviewer eZLW (3/4)

Q2 Part 2: In practice, there are often answers that is unique to a particular language and do not have direct translations to other languages.

Summary: For answers or concepts unique to a language, Rosetta often translates the surrounding question while keeping the unique concept and the options in the original language. We evaluate Rosetta on INCLUDE’s language-specific questions, including grammar, idioms, and national literature. Rosetta achieves lower Brier and ECE than both baselines on these questions.

We agree that some answers are specific to a language and may not have direct translations. However, in our main experiments, we focus on questions whose correct answers are not unique to a particular language.

Additionally, as a stress test of Rosetta, we evaluate it on INCLUDE, a benchmark of questions taken from local exams in each language with region-specific/agnostic labels. To examine language-specific content more directly, we focus on its language and literature subset, including grammar, spelling, idioms, and national literature. This allows us to compare language and literature questions with region-agnostic questions. For example, a Spanish question asks which of four words has the most syllables: “murciélago”, “aceitunitas”, “destruiditos”, or “autobiografía”.

We sample 500 language-and-literature questions and 500 language-agnostic questions from INCLUDE, with an equal number of questions per language up to the number available (19 languages: count per language differs). We report Brier ↓\downarrow / ECE ↓\downarrow / AUROC ↑\uparrow for GPT-5.6 Luna and Gemma 4 31B.

Takeaway: Rosetta retains its Brier and ECE advantage on language and literature questions containing answers unique to a particular language.

MethodLuna: language and literatureLuna: agnosticGemma: language and literatureGemma: agnostic
Self-consistency21.1 / 20.0 / 68.720.2 / 18.4 / 67.926.9 / 26.5 / 59.724.2 / 23.6 / 60.1
Paraphrase19.5 / 17.2 / 75.117.2 / 14.5 / 76.423.0 / 20.4 / 67.920.3 / 17.9 / 70.4
Rosetta19.3 / 13.3 / 72.816.7 / 11.7 / 76.419.4 / 12.0 / 74.317.6 / 11.5 / 72.8

Our qualitative analysis shows that Rosetta quotes language-specific text in its original language while writing the surrounding question or answer in the target language. We quantify this behavior by checking whether panels contain text copied verbatim from the original question or options. On language and literature questions, this occurs in 48% of GPT-5.6 Luna’s panels and 45% of Gemma 4 31B’s, compared with 7% and 12% on region-agnostic questions.

For example, for a Chinese question about the idiom "请君入瓮" (成语“请君入瓮”,“请”的是谁), Rosetta translates the surrounding question while keeping the idiom and options in Chinese:

Panel languageTranslated QAnswer
EnglishIn the idiom "请君入瓮", who is the one being "请"ed?The one being "invited" is 周兴.
GermanAuf wen bezieht sich im chinesischen Idiom „请君入瓮“ das „请“?Das „请“ bezieht sich auf 周兴.
SpanishEn el modismo chino "请君入瓮", ¿a quién se refiere "请"?"请" se refiere a 周兴.

Q3: Is k = 8 a common choice, and how do results change with k?

k=8k=8 is not a standard choice. We chose it following Gao et al. (2024), a prior work on perturbation-based confidence estimation, who evaluated up to eight samples.

Figure 5 in Section 6 and Figure 7 in Appendix E.1 already report results for k∈{1,2,4,6,8}k \in \{1,2,4,6,8\}. Larger panels generally result in better Brier scores and Rosetta’s advantage persists across panel sizes: it outperforms the consistency-based baselines at nearly all tested values of kk. Rosetta (k=4) outperforms self-consistency and Paraphrase at k=8 in Appendix K. We have updated the manuscript to make this clear.


Gao, X., Zhang, J., Mouatadid, L., and Das, K. SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models. EACL 2024.

Reviewer_XopcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the effectiveness of consistency-based confidence estimation across different languages and frontier LLMs. Through experiments on 17 models and nine languages, the authors find that self-consistency performs consistently worse outside English. To address this issue, they propose Rosetta, which estimates confidence by measuring agreement among answers generated from questions translated into different languages. Experiments show that Rosetta generally outperforms self-consistency and paraphrase-based methods, particularly in non-English settings. The paper also finds that verbalized confidence from certain frontier models, such as Claude Opus 4.6, can outperform consistency-based methods with fewer LLM calls.

Critical strengths
  • The evaluation covers 17 models and nine languages, including both open-weight and frontier models. The paper also considers multiple metrics, including AUROC, Brier Score, and ECE.
  • The observation that verbalized confidence from certain frontier models can outperform self-consistency is interesting and practically relevant.
  • Rosetta is simple yet effective and can outperform other baselines, especially for non-English texts
Critical weaknesses
  • Rosetta is not very novel. It simply translates questions into different languages. This paper does not explain why multilingual perturbation can lead to improvements. More in-depth studies should be done here.
  • The practical advantage of Rosetta is not clear. Rosetta requires multiple LLM calls for translation and answer generation, which introduces additional cost. If the target model is powerful, verbalised confidence can outperform self-consistency. When should Rosetta be preferred is not well-explained.
Questions for authors
  • Why does Rosetta sometimes improve the Brier Score while reducing AUROC?
  • Can the authors provide a comparison of the inference cost, such as latency?
  • Why does translating a question into multiple languages provide better confidence estimates than paraphrasing it within the same language?
4 comments
AuthorsOpenReview ↗

Response to Review Xopc (1/2)

We thank our reviewer for describing Rosetta as "simple yet effective" and for finding our verbalized-confidence result "interesting and practically relevant". We are happy to address your concerns:

Q2: Can the authors provide a comparison of the inference cost, such as latency?

Summary: Rosetta (k=8) takes 7.2 seconds per question for GPT-5.6 Luna on average, the same as Paraphrase at k=8. At k=4, Rosetta is faster, cheaper, and has a better Brier than Paraphrase (k=8). Similarly, Rosetta-AO (k=4) is as fast, cheaper, and has a better Brier than self-consistency at k=8. A similar pattern holds for Sonnet 4.6. See new Appendix K plots.

Thank you for the suggestion. Appendix K in our revised draft now includes the cost and latency of all five estimators on MMLU-ProX and PolyMath questions in six languages (English, Chinese, Arabic, Spanish, Russian, and Japanese) with GPT-5.6 Luna and Sonnet 4.6. Each value is the mean over the six languages and two datasets MMLU-ProX and PolyMath. We measured end-to-end latency per question and calculated cost from the recorded token usage at API list prices.

GPT-5.6 Luna

Method (#calls)Seconds$/1k questionsBrier ↓\downarrow (sorted worst to best)
Verbalized confidence (1)1.7$0.1539.3
Self-consistency, k=4 (5)1.9$0.5130.9
Self-consistency, k=8 (9)2.0$0.8530.2
Paraphrase, k=4 (6)5.1$1.0225.1
Rosetta-AO, k=4 (5)1.9$0.4924.1
Paraphrase, k=8 (10)7.2$1.8223.9
Rosetta-AO, k=8 (9)2.0$0.8323.4
Rosetta, k=4 (6)5.0$1.0421.5
Rosetta, k=8 (10)7.2$1.8520.7

Sonnet 4.6

Method (#calls)Seconds$/1k questionsBrier ↓\downarrow (sorted worst to best)
Self-consistency, k=4 (5)2.5$10.0533.0
Self-consistency, k=8 (9)2.6$17.0232.9
Rosetta-AO, k=4 (5)2.8$10.0925.6
Rosetta-AO, k=8 (9)3.1$17.0825.6
Paraphrase, k=4 (6)10.0$20.1124.2
Paraphrase, k=8 (10)15.1$35.2523.7
Verbalized confidence (1)2.6$3.2523.2
Rosetta, k=4 (6)10.5$20.5620.1
Rosetta, k=8 (10)16.7$36.1219.7

Our results show that Rosetta (k=8) is the best confidence estimator, at basically the same cost as baseline Paraphrase (k=8). Additionally, Rosetta (k=4) and Rosetta-AO (k=4) are cheaper alternatives that provide better performance than Paraphrase and self-consistency at k=8 respectively.


W2: The practical advantage of Rosetta is not clear. Rosetta requires multiple LLM calls for translation and answer generation, which introduces additional cost. If the target model is powerful, verbalised confidence can outperform self-consistency. When should Rosetta be preferred is not well-explained.

Summary: Rosetta should be preferred for multilingual models when confidence quality matters out of the box and verbalized confidence has not been validated for both the target model and domain. Verbalized confidence (VC) is cheaper but potentially brittle, as multiple powerful models, e.g., GPT-5.6 Sol, GPT-5.4, and DeepSeek V4.1 are poorly calibrated. In Table 1, only Opus 4.6's verbalized confidence was competitive with Rosetta. Compared to consistency-based methods, we have shown simlar cost/latency and better Brier with more efficient options (See Q2 above).

Cost or latency mattersQuality matters
Well-calibrated VC (e.g., Sonnet 4.6)VCRosetta or VC if validated as better (e.g., Opus 4.6)
Unknown or poorly calibrated VC (e.g., GPT-5.6 Luna)Rosetta-AORosetta

Compared to VC: We agree that verbalized confidence can be a cheap option. However, its quality varies widely across models. In Table 1, only Opus 4.6’s verbalized confidence slightly outperforms Rosetta, with a Brier score of 17.1 versus 17.3 for Rosetta. Even frontier models such as GPT-5.4 and GPT-5.6 Sol perform poorly with verbalized confidence. Therefore, model strength alone does not tell a user which estimator to choose. Additionally, establishing whether verbalized confidence works well for each new model and domain requires validating with labeled data and is often impractical. In domains where uncertainty estimation is important, and a model’s calibration is unknown, Rosetta is the safer choice compared to verbalized confidence. Even for Sonnet 4.6, whose verbalized confidence is well calibrated, Rosetta achieves a better Brier across six languages (19.7 versus 23.2, see Q2).

Compared to consistency-based methods: As shown in our answer to Q2 above, Rosetta-AO (k=4) and Rosetta (k=4) are cheaper and achieve lower Brier than self-consistency and Paraphrase at k=8, respectively. Please see Q2 above and Appendix K in our updated draft for more details.

AuthorsOpenReview ↗

Response to Review Xopc (2/2)

W1/Q3: [...] This paper does not explain why multilingual perturbation can lead to improvements. More in-depth studies should be done here. Why does translating a question into multiple languages provide better confidence estimates than paraphrasing it within the same language?

Summary: Translation perturbs the same question more strongly than paraphrasing (New Appendix L.2). In Rosetta, both answer diversity (New Appendix L.1) and answer novelty (Section 5.3) increase more when the anchor answer is wrong than when it is correct. Rosetta has the largest diversity gap between wrong and correct answers, making agreement more informative about correctness and reducing overconfidence. Our new studies examine both the representations and the behavior of Rosetta (New Appendix L).

Translation changes the representation more while remaining close to the original question. We examine three open models: Qwen3.5-9B, Gemma-4-12B, and Gemma-4-31B. For each question, we compare the representations of the original question, its paraphrases, and its translations across layers. Our results show that at every layer, translated questions remain much closer to their originals than to unrelated questions (0.04–0.20 versus approximately 1.0). They come closest in the middle layers (0.04–0.07), but remain farther from the original than paraphrases do (0.02–0.04). This difference holds at every layer in all three models. This shows that translation produces a stronger perturbation throughout the model while keeping the representations close to the original question. Please see Appendix L for more details.

The additional diversity is greater on wrong answers (Section 5.3, New Appendix L.1). Section 5.3 already shows that Rosetta produces more novel answers when the anchor answer is wrong and has a bigger novelty gap than Paraphrase and self-consistency (Figure 4b). We now measure diversity within each panel directly, using the entropy of its selected options on MMLU-ProX (in bits; higher values indicate greater diversity). Averaged over eight models and nine languages, Rosetta produces the most diverse panels when the anchor answer is wrong. For confidence estimation, the gap matters more than diversity alone: panels should stay consistent on correct answers and become more diverse on wrong ones. Rosetta has the largest such gap in all eight models.

MethodAnchor answer wrongAnchor answer correctGap
Self-consistency0.590.190.41
Paraphrase0.830.280.55
Rosetta0.980.370.61

As a result, Rosetta is less often confidently wrong (Section 5.3, Figure 4a). For GPT-5.4 on English MMLU-ProX and PolyMath, it assigns high confidence (≥\geq 90%) to only 14% of wrong answers, compared with 21% for Paraphrase and 36% for self-consistency.

Together, these results explain Rosetta's advantage. Translation perturbs the same question more strongly, and wrong answers repeat less often under this perturbation.

W1: Rosetta is not very novel. It simply translates questions into different languages.

Rosetta's simplicity is its strength. To the best of our knowledge, we are the first to show that simple cross-language agreement is a more useful signal of correctness than same-language agreement at a matched call budget. This holds with or without translating the question (Rosetta and Rosetta-AO) and the Brier gains hold in English and outside English in an evaluation covering nine languages and various models.

Q1: Why does Rosetta sometimes improve the Brier Score while reducing AUROC?

AUROC only measures ranking quality, while Brier measures both calibration and ranking quality. When Rosetta improves Brier but reduces AUROC, ranking has become (slightly) worse, while calibration has improved by more than enough to compensate for the small drop in AUROC.


We hope these clarifications and additional experiments address your concerns. Thank you again for your time and feedback.

Thank you again for your review. As the discussion period is coming to an end, we wanted to kindly check whether our responses and new experiments have addressed your concerns. If so, we would appreciate it if you would consider updating your score. Please let us know if there is anything else we can clarify or do.

I truly appreciate the authors' effort and response. I agree that the new findings and explanations strengthen the manuscript. However, the novelty is still an issue and the verbalised confidence is working very well, which limits the contribution of this work. Therefore, I would like to maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.