- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Here the authors examine the discrepancy in factual claim verification performance between English and other languages using LLMs. They explore whether this reflects a lack of factual knowledge or difficulties in retrieving information already encoded in the models' internal weights.
The authors compare the performance of the models' original read-out mechanisms with linear probes trained on intermediate representations. They introduce RoSh, an inference-time technique which applies orthogonal rotations to the residual stream at selected transformer layers. These transformations are estimated using Orthogonal Procrustes, without altering the original model weights.
The authors evaluate across eight languages and eight models from five families. The authors report improvements in factual verification performance, particularly for smaller models and languages with larger initial performance gaps. The evaluation includes comparisons with existing methods, component ablations and controls to examine the importance of geometric transformations, layer selection and cross-lingual correspondences.
Critical strengths
1. Interesting research question The paper addresses an interesting question regarding multilingual factual knowledge in LLMs: whether differences in factual verification performance across languages reflect missing knowledge or difficulties in recovering information from internal representations. The distinction between linearly decodable information and the model's actual predictions provides a useful perspective for investigating multilingual model behaviour. The proposed RoSh method is simple and well motivated.
2. Extensive experimental validation and well-designed controls The paper provides a several experimental evaluation across multiple languages, model families, and model sizes. The reported improvements are particularly notable for smaller models. They also include several ablation studies and control experiments. The comparisons involving shift-only and rotation-only interventions, random transformations, scrambled correspondences, and alternative layer choices help establish the importance of different components of the proposed method. These experiments strengthen the empirical evidence supporting RoSh beyond simply reporting improvements over the baseline.
Critical weaknesses
1. Interpretation of the read-out gap My main concern relates to the interpretation of the difference between the performance of linear probes and the models' original read-out mechanisms. The authors argue that factual knowledge is already encoded in the models' internal representations but is not adequately recovered in languages other than English. While the probe results support the presence of linearly decodable information, this does not necessarily establish that the original model can functionally access or use that knowledge. Similarly, the improvements obtained through RoSh provide evidence that cross-lingual representation alignment is beneficial, but they do not uniquely establish the mechanism responsible for the original performance gap.
The authors acknowledge some of these limitations but certain statements about recovering existing knowledge and eliminating read-out failures appear stronger than the evidence directly supports. A more cautious interpretation would help distinguish the demonstrated empirical improvements from the proposed explanation of their underlying mechanism.
2. Dependence on the read-out formulation and limited comparison with simpler alternatives The evaluation relies on a fixed binary verdict formulation, using the output logits associated with the English tokens TRUE and FALSE across all languages. It remains unclear to what extent the observed performance gaps depend on this particular choice of verbalizers. Authors may evaluate translated verdict labels or alternative prompt formulations in order to determine whether part of the multilingual gap can be reduced without modifying internal activations.
Questions for authors
- Have the authors investigated whether the multilingual performance gap and the improvements obtained with RoSh remain consistent when using alternative verbalizers, particularly verdict labels translated into each target language? This would help clarify whether the observed read-out gap is partly dependent on the fixed TRUE/FALSE output formulation.
- How does RoSh compare with directly adapting the read-out mechanism, for example through a lightweight linear classification head trained on the same fitting data? Given that linear probes already recover substantial factual information from the representations, such a comparison could help clarify the advantages of correcting intermediate activations rather than replacing or adapting the final decision mechanism.
2 comments
Assessment after rebuttal
I would like to thank the authors for their response and efforts. The new TRUE/FALSE experiments are useful for analyzing the multilingual performance gap. I also appreciate the clarification regarding the comparison with linear probes.
Furthermore, the revised interpretation of the read-out gap addresses my main conceptual concern. Presenting the proposed explanation as a hypothesis brings the conclusions closer to the experimental evidence. Overall, I believe the rebuttal substantially addressed my concerns and strengthened the paper. Therefore, I maintain my positive recommendation.
Thank you. Both of your questions led to new experiments in the revision, and one of them changed what we believe.
Q1 and W2: does the gap depend on the English TRUE and FALSE words? No.
We changed the prompt in each language to ask for that language's own words (VERDADEIRO/FALSO, PRAWDA/FAŁSZ, WAHR/FALSCH, صحيح/خطأ, 真/假, ВЕРНО/НЕВЕРНО, DOĞRU/YANLIŞ), scored as in Eq. 7. Separately we kept the English instruction and left only the claim in its own language (Section 5.2, Table 4; every model in Appendix Q, Table 22).
The mean gap is 0.047 with native words and 0.049 with the English instruction, against 0.050 with our prompt. Single models move a lot: native words halve Llama-3B's gap, from 0.111 to 0.051, and widen Mistral-7B's, from 0.130 to 0.147. On average they barely move it.
We then refitted RoSh from scratch with the native words. It narrows the gap on all eight models, from 0.047 to 0.021, against 0.015 with English words. So on average neither the gap nor the correction depends on the English answer tokens, although the correction does recover somewhat less without them.
W1: we were claiming more than the evidence supports.
You are right, and the revision now says only what the results show.
Section 3.1 presents the account as our working hypothesis, which the results support but do not prove: the activations carry the truth value in every language, but in a different frame from the English one, meaning a different orientation and position of the activation cloud. The abstract and Section 1 now speak of what a probe recovers against what the verdict expresses, by the probe's measure. They no longer speak of knowledge the model already holds, and "no longer a read-out failure" is gone. Section 5.1 no longer calls read-out inefficiency a general property of the model, and Appendix A says the probe estimates rather than bounds what a linear reader can extract. Section 6 has a new part, "What the probe shows, and what it does not". A probe shows that a linear reader can extract the information, not that the model uses it. The split is an estimate, because the probe is trained on the same templates and the head is not. And the change-of-frame account is offered as the reading most consistent with the results, not as an established mechanism. The Limitations say we do not identify which part of the head fails, or what puts non-English activations in a different frame.
What the evidence does support, we kept. The gain needs the claim-level pairing: scrambling it drops the score to 0.621 and a random rotation gives 0.658, against 0.853 for the fitted map. The map is invertible, so the gain cannot come from deleting information (Proposition E.1). In a re-run the fitted map beats all 20 scrambled pairings in each of 20 model and split runs. Corrected activations land near their own English counterparts without becoming copies (Figure 4).
Q2: how does RoSh compare with a lightweight linear head trained on the same data?
Our probe is exactly that head, fitted at an intermediate layer rather than on the final state: an L2-regularised logistic regression on the same 750 fitting claims, at one layer (Table 11). On held-out claims it scores higher than RoSh, 0.910 outside English against 0.855, and we say so rather than leaving it out. We use it as the measuring instrument that splits the gap, not as a competitor.
Correcting the activations differs from replacing the read-out in two ways. The model's own head still produces the verdict, so the gain is a statement about the model rather than about a new classifier. And the two use the labels differently: a head fits all of its weights to the labels, while RoSh fits its map from the claim pairing alone and uses labels only to rank candidate layers. Section 6 now makes this comparison in the paper.