- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper evaluates multilingual calibration for multiple-choice QA with several open-weight LLMs, looking at both language variation and prompt sensitivity. It reports that calibration and accuracy degrade in lower-resource settings and under prompt perturbations, and it explores a simple post-hoc recalibration step.
Critical strengths
-
Broad empirical coverage at low reported cost. The paper evaluates four open-weight models across many language configurations and prompt variants. This produces a potentially useful descriptive record of model- and prompt-specific failures, including cases where a model fails to emit an answer option.
-
Some transparency about limitations and reproducibility. The authors disclose the restricted-logit procedure, fallback cases, compute cost, and the in-distribution nature of the calibration repair. They also state that code and parsed outputs will be released. These choices make the reported audit easier to inspect and reproduce.
Critical weaknesses
-
The central calibration measurement is not reliable across the evaluated conditions. Confidence is obtained by renormalizing only the A/B/C/D tokens that happen to occur among the top-5 next-token logits. The effective candidate set can therefore vary across examples, languages, prompts, and models. If no answer letter appears, the paper assigns confidence 0.25; in some Gemma conditions the fallback rate approaches 100%. Consequently, changes in ECE can reflect truncation and answer-extraction behavior rather than genuine changes in model calibration. The paper acknowledges this limitation but nevertheless bases its main conclusions on these measurements.
-
The contribution is largely descriptive and does not establish a new mechanism or general remedy. Lower performance and reduced robustness in underrepresented languages are already plausible consequences of uneven multilingual training. The paper neither measures language representation in the models' training corpora nor tests another causal explanation. Its proposed repair is separate in-distribution calibration for every model–language–prompt cell using approximately 250 labeled examples. This does not address the prompt instability highlighted by the paper and provides no evidence of transfer across prompts, languages, datasets, or models. The short main text and limited analysis leave the submission closer to an experimental audit report than a complete research contribution.
Questions for authors
1.Can the authors recompute the results using the complete logits for all four answer options, without top-5 truncation or an artificial 0.25 fallback value? Do the Africa–Europe and prompt-perturbation conclusions remain after this correction?
- What evidence supports attributing the observed differences specifically to language or region rather than dataset source, subject composition, tokenization, or instruction-following failures? A subject-matched, within-dataset analysis would materially affect my assessment.
3.Is there evidence that any proposed calibration repair transfers to a new prompt, language, or dataset without collecting a new labeled calibration set? If not, the paper should substantially narrow its claim that the problem is “repairable.”