- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors tackle the problem of black-box LLM confidence calibration. They focus on frontier models (which they show are understudied in recent calibration literature) and evaluate on multiple languages. They introduce a method called Rosetta, which computes self-consistency under translated prompts. They show that with frontier models on English and non-English datasets, Rosetta has better calibration than self-consistency under same-language prompt perturbations.
Critical strengths
- Presents a simple and well-motivated method for black-box LLM confidence calibration, outperforming baselines. They have an insightful analysis showing that Rosetta leverages the multilingual capabilities of modern LLMs, with Rosetta’s calibration improving with a given model’s non-English accuracy.
Critical weaknesses
- Does not consider settings with long-form generation or reasoning (CoT). This is in spite of the fact that the two datasets shared across all languages (PolyMath and MMLU-ProX) are meant to be solved with CoT.
- The method relies on the multilingual proficiency of the LLM being used. This is probably one reason why the paper focuses less on non-frontier models, since their multilingual capabilities are weaker.
Questions for authors
- How does Rosetta compare to DINCO (Wang and Stengel-Eskin, 2026)? DINCO uses verbalization consistency as a complementary signal to generation consistency, beating generation consistency alone, whereas generation consistency is the main family considered in this paper; vanilla verbalized confidence is the only verbalization-based method considered and shows catastrophic performance in almost all cases. It would be interesting to compare your current single-language and multi-language methods (Paraphrase and Rosetta) with versions that incorporate a DINCO-like verbalization component. A simple way to do this would be to have each of the k prompts ask for a verbalized confidence after the predicted answer, as in the vanilla prompt in Xiong et al. (2024). Then you can average the verbalized confidence over duplicates of a given unique answer, then normalize (with denominator >= 1) verbalized confidence over unique answers. The normalized verbalized confidence can then be combined with the usual generation consistency confidence. Maybe you want to have the LLM do this processing in the last call as in your current setup. I would like to see these comparisons for at least a subset of the models and datasets.
- In the abstract, the framing of translation as a “principled” perturbation method is not justified. It is not clear why translation is more principled than using no perturbation or same-language question perturbations.
References:
Wang and Stengel-Eskin, 2026. Calibrating Verbalized Confidence with Self-Generated Distractors.
Xiong et al., 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs.
3 comments
Response to Reviewer Wp5K (2/2)
W1: Does not consider settings with long-form generation or reasoning (CoT). This is in spite of the fact that the two datasets shared across all languages (PolyMath and MMLU-ProX) are meant to be solved with CoT.
Summary: We initially excluded CoT generation due to its prohibitively high cost. Our revised manuscript now adds an evaluation with chain-of-thought prompting on MMLU-ProX and PolyMath. Averaged across the six languages and two models, Rosetta achieves the best mean Brier and ECE.
We did not initially evaluate CoT because of its high cost. Figure 3 alone covers 17 models, 9 languages, 2 datasets, and 5 methods (>1,500 setup combinations). Duplicating these experiments with CoT would roughly cost $20K at API list prices. Nonetheless, Appendix E.8 of our revised manuscript now includes CoT evaluation for GPT-5.6 Luna and Gemma 4 31B on both MMLU-ProX and PolyMath. Each cell below reports Brier / ECE / AUROC , averaged across six languages: English, Chinese, Arabic, Spanish, Russian, and Japanese.
Takeaway: Rosetta retains its advantage with chain of thought, achieving the lowest mean Brier and ECE in all four comparisons, although the winning margins are smaller.
Brier / ECE / AUROC
| Dataset | Model | Self-consistency | Rosetta-AO | Paraphrase | Rosetta |
|---|---|---|---|---|---|
| MMLU-ProX | GPT-5.6 Luna | 15.1 / 12.4 / 78.9 | 13.5 / 9.0 / 82.3 | 13.9 / 10.5 / 81.6 | 12.6 / 7.3 / 84.7 |
| Gemma 4 31B | 16.8 / 15.8 / 68.6 | 16.1 / 13.8 / 69.3 | 15.8 / 13.8 / 70.3 | 14.9 / 12.2 / 73.8 | |
| PolyMath | GPT-5.6 Luna | 16.3 / 19.2 / 90.7 | 14.6 / 14.7 / 90.1 | 15.4 / 17.1 / 90.2 | 13.8 / 13.9 / 90.5 |
| Gemma 4 31B | 15.0 / 12.3 / 81.3 | 14.9 / 11.8 / 81.8 | 14.5 / 10.6 / 81.9 | 14.4 / 9.4 / 82.6 |
W2: The method relies on the multilingual proficiency of the LLM being used. This is probably one reason why the paper focuses less on non-frontier models, since their multilingual capabilities are weaker.
Summary: Rosetta relies on multilingual proficiency, but its benefits are not limited to frontier models. Rosetta achieves a lower Brier score than both baselines outside English for nine of ten open models from 2B to 31B. We showed AUROC gains depend on multilingual proficiency, with improvements over both baselines for every tested model with at least 30% non-English accuracy (Section 5.2).
We agree that Rosetta relies on the model's multilingual capability. Still, our results show that Rosetta improves calibration for non-frontier models. Our evaluation includes ten open models from 2B to 31B. Outside English, Rosetta achieves a lower Brier score than both baselines for nine of ten open models, with a mean Brier improvement of 8.0 points over self-consistency and 3.5 points over Paraphrase.
As mentioned in Section 5.2, AUROC gains depend more on multilingual proficiency. Rosetta improves AUROC over both baselines for all eleven models with at least 30% non-English accuracy and even two small Gemma models achieve AUROC gains comparable to frontier models like GPT-5.4 (Table 3):
| Model | AUROC gain over self-consistency | AUROC gain over Paraphrase |
|---|---|---|
| Gemma 4 12B | +5.8 | +4.7 |
| Gemma 4 31B | +9.1 | +2.2 |
| GPT-5.4 | +6.0 | +2.2 |
We hope these clarifications and additional experiments address your concerns. Thank you again for your time and feedback.
Thank you for taking the time to run additional experiments to address my comments. I am happy with the authors' response and will keep my positive score.
Response to Reviewer Wp5K (1/2)
We appreciate the reviewer’s comment that Rosetta is a "simple and well-motivated method", and for highlighting our "insightful analysis" showing that Rosetta leverages the multilingual capabilities of modern LLMs.
Summary: We implemented your suggested hybrid step and also ran DINCO. Combining normalized verbalized confidence with agreement improves self-consistency, Paraphrase, and Rosetta. Rosetta +NVC or Rosetta +DINCO are the best methods in 10 of 12 cases across both models, both datasets, and three metrics; the two exceptions are ECE, where DINCO is better.
We see hybrid methods such as DINCO as orthogonal to our method and analysis.
As requested, Appendix E.7 of our revised manuscript includes results for both DINCO and your proposed hybrid step.
DINCO with 8 samples and 8 self-generated distractors, matching Rosetta's eight panel answers, and using an LLM in place of the NLI model for grouping equivalent answers. We also test integration of Rosetta with DINCO in the hybrid setup. Rosetta +DINCO replaces DINCO's self-consistency component with Rosetta's agreement score and keeps the rest unchanged.
Hybrid method (your proposal) applies normalized verbalized confidence (NVC) to the three consistency-based methods in our paper. We elicit the verbalized confidence of panel answers by asking the LLM to assign a confidence score.
We evaluate DINCO and Hybrid on MMLU-ProX in English, Chinese, Japanese, and Arabic and on open-ended questions in PopQA with GPT-5.6 Luna and Sonnet 4.6.
Takeaway: The hybrid method improves the Brier score for every sampling method. Rosetta with NVC outperforms all other methods on Brier, ECE, and AUROC, except for DINCO's ECE on Sonnet 4.6.
MMLUPROX:
The table below reports mean Brier ↓ / ECE ↓ / AUROC ↑, across the four languages.
POPQA:
Takeaway: Rosetta with NVC achieves the best Brier score across both models. For AUROC, Rosetta + DINCO was the top performer for Sonnet, while Rosetta + NVC performed best on Luna.
The table below reports Brier ↓ / ECE ↓ / AUROC ↑ on PopQA.
Thank you for these suggested experiments. Your proposed hybrid method with NVC helps further demonstrate Rosetta’s generalizability and effectiveness.
We agree that “principled” is not sufficiently justified here. Our revised manuscript now uses “multilingual perturbation” in place of “principled perturbation”.