acceptodds
Under review as a conference paper at ICLR 2027

When Confidence Doesn’t Travel: Calibration of Open-Weight LLMs on African-Language MCQs Under Prompt Perturbations

Abstract

Research question: Are open-weight LLMs well-calibrated on African-language multiple-choice questions, does their calibration survive prompt perturbations, and at what compute cost? Answer: Mostly no. The failure is model-specific in ways aggregate benchmarks hide, and the full audit costs under 40 USD. We audit four open-weight instruct models (Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B, Mistral-7B) on 422,000 MCQ items spanning 24 Global-MMLU languages (9 African, 15 European) and 18 afrimmlu configurations, under 5 prompt perturbations, for a total cost of 39.48 USD on serverless A100s. Accuracy drops 16-31 points from European to African language cells for every model. The calibration gap, measured as Expected Calibration Error (ECE), is real but model-dependent at baseline: large for Qwen (+0.140 ECE [0.099, 0.177]) and Gemma (+0.133 [0.109, 0.159]), absent for Llama, and reversed for Mistral. Perturbation effects flip sign across models; a format that calibrates one model can break another, and Gemma loses 20 accuracy points in African language cells simply when asked to "answer with a single letter", with a 26.1% fallback rate across evaluated African cells that reaches 100% in some format_A afrimmlu cells. Miscalibration is largely repairable in-distribution: regularized logistic regression fit on approximately 250 items per cell cuts held-out ECE to 0.04-0.10 depending on region. We provide code, parsed outputs, and an executable reproduction notebook.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM calibration, uncertainty quantification, multilingual language models, African languages, large language models, prompt robustness, confidence estimation, evaluation, responsible AI

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3H9NOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates multilingual calibration for multiple-choice QA with several open-weight LLMs, looking at both language variation and prompt sensitivity. It reports that calibration and accuracy degrade in lower-resource settings and under prompt perturbations, and it explores a simple post-hoc recalibration step.

Critical strengths
  1. Broad empirical coverage at low reported cost. The paper evaluates four open-weight models across many language configurations and prompt variants. This produces a potentially useful descriptive record of model- and prompt-specific failures, including cases where a model fails to emit an answer option.

  2. Some transparency about limitations and reproducibility. The authors disclose the restricted-logit procedure, fallback cases, compute cost, and the in-distribution nature of the calibration repair. They also state that code and parsed outputs will be released. These choices make the reported audit easier to inspect and reproduce.

Critical weaknesses
  1. The central calibration measurement is not reliable across the evaluated conditions. Confidence is obtained by renormalizing only the A/B/C/D tokens that happen to occur among the top-5 next-token logits. The effective candidate set can therefore vary across examples, languages, prompts, and models. If no answer letter appears, the paper assigns confidence 0.25; in some Gemma conditions the fallback rate approaches 100%. Consequently, changes in ECE can reflect truncation and answer-extraction behavior rather than genuine changes in model calibration. The paper acknowledges this limitation but nevertheless bases its main conclusions on these measurements.

  2. The contribution is largely descriptive and does not establish a new mechanism or general remedy. Lower performance and reduced robustness in underrepresented languages are already plausible consequences of uneven multilingual training. The paper neither measures language representation in the models' training corpora nor tests another causal explanation. Its proposed repair is separate in-distribution calibration for every model–language–prompt cell using approximately 250 labeled examples. This does not address the prompt instability highlighted by the paper and provides no evidence of transfer across prompts, languages, datasets, or models. The short main text and limited analysis leave the submission closer to an experimental audit report than a complete research contribution.

Questions for authors

1.Can the authors recompute the results using the complete logits for all four answer options, without top-5 truncation or an artificial 0.25 fallback value? Do the Africa–Europe and prompt-perturbation conclusions remain after this correction?

  1. What evidence supports attributing the observed differences specifically to language or region rather than dataset source, subject composition, tokenization, or instruction-following failures? A subject-matched, within-dataset analysis would materially affect my assessment.

3.Is there evidence that any proposed calibration repair transfers to a new prompt, language, or dataset without collecting a new labeled calibration set? If not, the paper should substantially narrow its claim that the problem is “repairable.”

Reviewer_uaP4OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the calibration of open-weight LLMs on African language MCQ. It evaluates 4 models on 422k MCQ items spanning 24 Global MMLU languages and 18 afrimmlu configurations, under 5 prompt perturbations. The result show that all models lose substantial accuracy on African languages compared to European languages, the calibration gap is model-dependent (large for Qwen and Gemma, absent for Llama, and reversed for Mistral), and the effect of each perturbation is also model-specific. The authors further show that the miscalibration is largely repairable in-distribution via post-hoc calibration.

Critical strengths
  1. This work audit 422k MCQ items across 4 models and 42 language cells and 5 prompt perturbations for only 39.48$, and release code, parsed outputs, and an executable reproduction notebook
  2. This work reveal current multilingual evaluations under-cover failure modes: Gemma collapse under prompt "answer with a single letter", which provides insights for future evaluation and task design
Critical weaknesses
  1. Miscalibrations of LLMs in low-resource languages, sensitivity to prompt formatting, and post-hoc calibration repair are well established before (On the Calibration of Multilingual Question Answering LLMs, N2C2: Nearest Neighbor Enhanced Confidence Calibration for Cross-Lingual In-Context Learning). The paper combines these points but provides no new mechanism, metric, and analysis
  2. The experiment covers only four 7-9B models, and 5 perturbations. The evidence is too narrow.
Questions for authors
  1. Confidence is computed from a restricted softmax over top-5 first-token logprobs, which inflates confidence when fewer than 4 answer letters appear. Could you provide control results with the full 4-letter softmax?
  2. All four audited models are 7–9B models from 2024. Could you run a spot check on one newer or larger open-weight model? Does the African-language miscalibration phenomenon persist in newer generations, or has it been mitigated?
Reviewer_z5qROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors evaluate the log-probs-based confidence estimation of several open-weights language models on multilingual multiple-choice question benchmarks, providing a direct comparison of the models’ calibrations in European languages versus African languages. They additionally test the effect of changing the prompting format, employ several calibration repair strategies, and report the total cost of the evaluation.

Critical strengths
  • The authors evaluate the calibration of open-weights LLMs on the AfriMMLU dataset, which, as far as the reviewer is aware, has not been done in previous work.

  • The authors evaluate the effect of perturbing the prompting format, a step that is missing from related work.

  • The authors identify that instances where the model fails the question format corrupt the calibration measurements, a relevant insight for related work on confidence estimation.

  • The evaluation code is promised, allowing for reproducibility.

Critical weaknesses

The central aim of this work, to evaluate calibration, is not performed effectively. Although reliability diagrams are provided, only absolute calibration is evaluated, primarily with Expected Calibration Error (their ‘class-conditional Expected Calibration Error’ is calculated only over correct predictions, meaning that it reduces to 1 minus the mean confidence on correct answers). ECE alone is not sufficient for identifying meaningful confidence estimates; e.g. a model that always gives 60% confidence on a dataset for which it’s 60% accurate will have very low ECE. It is essential to report a failure prediction metric, such as AUROC, to determine whether the confidence estimates effectively distinguish between correct and incorrect answers. Additional analysis of the distribution of confidence estimates across language families would also have given needed insight into the models’ behaviours.

The method of extracting confidence estimates is flawed as the authors only extract the top-5 log-probs meaning that, if any of the options are missing from the top 5 tokens predicted by the model, the confidence assigned to the remaining options will be inflated. While the authors report how often none of the options appear in the top-5, they do not characterize the frequency at which 1, 2, or 3 of the options are missing. The resulting confidence estimates are therefore unreliable. While it is claimed that the within-model regional comparisons are robust to this, the differing fallback rates (e.g. Gemma 26.1% vs. 18.1%) contradict this.

The comparison is not fair. The authors claim that the evaluation is ‘dataset-matched’ which is inaccurate: the Africa region has 9 cells from Global-MMLU as well as 18 from AfriMMLU, whereas the Europe region only has 15 Global-MMLU cells. In addition, the models show a large difference in accuracy between the African and European languages. The relationship between accuracy and calibration is not characterized. Because a failure prediction metric, such as AUROC, is not reported, it is impossible to know whether the confidence estimates extracted for European languages are any more meaningful than those extracted for African languages, or whether the calibration gap derives largely from the difference in accuracy between language groups.

The paper overclaims its novelty. Huang et al. (EACL 2026: https://aclanthology.org/2026.eacl-short.1/) evaluate the calibration of several open-weights models on Global-MMLU, including the 9 Global-MMLU African languages evaluated in this paper. Zhou et al. (arXiv 2510.03136) evaluate log-probs-based confidence for MCQ answering on MMMLU and Belebele, which covers several African languages, and report AUROC alongside ECE. This paper’s novelty is limited to the evaluation on AfriMMLU, as well as some smaller details such as the prompt perturbation and format failure analysis.

Questions for authors
  • How do the models’ confidence estimates perform at failure prediction, as measured by AUROC? Is there a significant difference in this metric between the language groups?

  • What is the relationship between accuracy and calibration? If you plot accuracy against calibration for all language cells across the regions, does this explain the calibration gap between African and European languages?

  • How frequently are letter options excluded from the top-5 log-probs of the models? If it is frequent, is it possible to increase the number of top log-probs considered to mitigate this issue?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.