- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes REAL, a method to improve multilingual MCQA performance using inference time scaling. The method works in 2 steps. First step is to generate multiple candidates. Here the authors propose adding perturbation to the hidden states of the question tokens. The perturbation vectors are generated offline using high and low resource non-test questions. This setup is used to generate three candidate answers to each question.
For selection, the authors train classifiers that essentially select the best answer from the pool of candidates. The selection is framed as predicting repair vs damage: REAL switches to the best perturbed answer only if predicted repair minus λ times predicted damage passes a threshold tuned on validation data, and otherwise keeps the original answer.
The authors' main results are on MMMLU test sets in Yoruba and Swahili with Qwen3-8B and Llama-3.1-8B-Instruct. The results show improvements of up to 11.12 pp. The paper compares against fixed perturbations (which reduce accuracy), a base-feature-only selector (+7.02), a confidence gate (+7.52) and an NLLB translation baseline. Finally the paper reports smaller gains on Tibetan and no reliable gain on free-form AfriQA.
Critical strengths
- The paper shows rigorous and well-documented evaluation. The paper evaluates on the full official MMMLU test sets and reports paired bootstrap confidence intervals. The authors also carefully keep the bank, classifier and test data disjoint and documented (Table 7). The paper also reports negative results faithfully, for example fixed perturbations reduce accuracy, the AfriQA results, and more. This matters to my assessment because I trust the reported numbers, even where I disagree with how they are interpreted.
Critical weaknesses
- The gain is not shown to come from the cross-lingual perturbations as several results suggest the perturbations mostly act as noise that changes the answer, while the classifier decides when to switch:
- Shuffled and covariance-sampled residuals do as well as the matched ones (Table 4).
- The base-only selector already gets +7.02 of REAL’s +9.04 (Table 3).
- About 98% of the answer changes happen on questions the base model got wrong, and of those changed answers 36% become correct, close to the 33% you would get by picking one of the other three options at random (computed from Table 3, L089 and Appendix E, L802–803).
- On free-form AfriQA, where you cannot just switch to another option, the selectors give no reliable gain.
This matters because the title and main claims credit cross-lingual alignment. Without these controls, the gain could come entirely from switching away from low-confidence answers.
- The paper is hard to read and the method is overly complex.
- The perturbation alone has many tuned settings (PCA size, removed language directions, cluster and neighbor counts, temperature, layer, scale and cap). On top of that come the three-perturbation library, two classifier stages, λ and the threshold.
- Key baselines are not defined. The confidence gate is described only as ‘a validation-tuned confidence/margin gate’ (L300).
- The text relies on terms without giving context about them first, such as preserving oracle (used in the abstract and introduction before Section 3 defines it) and SemSub (never spelled out), and run labels such as P25 and P29 (L727) that appear before or without explanation.
- Tables 1, 3 and 4 are repeated as Tables 8, 9 and 10.
This matters because it makes the contribution hard to assess.
Questions for authors
-
[Suggestion] Can you run the same classifier on candidates from temperature sampling, and on random perturbations of the same norm? If these match REAL, then it tells us the gains comes from the selection step alone.
-
[Clarification] How are the confidence/margin gate and the base-feature-only selector defined?
-
[Suggestion] I wonder how an off-the-shelf reward model compares with your trained classifier when picking among temperature samples, or among your perturbations.