- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies cross-lingual multimodal fact verification in small vision-language models and constructs MOCHEG-XL, a machine-translated extension of MOCHEG to eight target languages that yields a 9×9 grid of claim/evidence language configurations with visual content held fixed. The authors show that high-resource languages degrade modestly while low-resource evidence languages (Javanese, Welsh, Māori, Tigrinya) collapse to near-random performance, with evidence language rather than claim language driving the failure. They propose ORBIT, a training-free, test-time intervention that estimates span-specific language offset vectors from parallel calibration instances and adds them to the claim and evidence hidden states at an intermediate layer. Across three SVLMs and 81 configurations, ORBIT improves low-resource accuracy (e.g., +2.12 points on Qwen3-VL, with bootstrap CIs excluding zero) at lower overhead than translation pipelines, though the absolute performance remains near chance.
Critical strengths
-
The controlled diagnostic design is the paper's strongest point. By holding visual content constant while sweeping claim and evidence languages over an 81-configuration grid, MOCHEG-XL isolates the linguistic effect from the visual encoder, and the pattern is replicated across three different architectures. The evidence-language-dominant collapse — and the disappearance of the same-language diagonal for jw/cy/mi/ti — is a non-obvious finding that turns a vague intuition about cross-lingual failure into a quantified, locatable phenomenon.
-
The intervention is genuinely lightweight and deployment-friendly. ORBIT requires no training, no labels at calibration, and only a single-layer injection with modest overhead, which is well aligned with the resource-constrained, local deployment setting that motivates the paper. The offset norm also offers a promising, cheap proxy for screening language risk without running a full evaluation.
Critical weaknesses
-
The comparison is unfair because the methods are not given translation resources of comparable quality. Both translation baselines rely on deliberately weak translators — Self-translate CoT asks the 4B model itself to translate evidence passages up to 1,774 words, and Pivot NLLB uses the 600M distilled NLLB — whereas ORBIT's own calibration pairs are produced by Google Cloud Translation, a much stronger system. The baselines' weaker performance may therefore stem from their translators rather than from ORBIT itself; I would suggest re-running the comparison with the same translation source on both sides.
-
The paper only compares against two relatively old baselines and does not cover closely related work from 2025–2026. In particular, CLAIM (Ye et al., ACL 2025), a training-free, test-time intervention that likewise estimates English-to-target shift vectors for multilingual VLMs, is neither cited nor run. I would suggest including CLAIM as a baseline and explicitly positioning ORBIT against it so that the novelty of the proposed method is clear.
-
A question about the method: in Eqs. (4)–(5) the offset is defined as μ(target) − μ(anchor), and Eqs. (9)–(10) add this vector to the target-language hidden states, which pushes them one step further away from the anchor — the opposite of what "realigning toward the anchor" suggests. Should the vector not be subtracted, or defined in the reverse direction? As written, the intervention seems to amplify language specificity rather than realign it, which would also explain why the NEI class is oversteered in Tables 6 and 10; a +v vs. −v ablation would clarify this.
Questions for authors
Please See Weakness.
5 comments
Weakness 2
W2: Missing comparison with novel, related methods
A: We thank the reviewer for their great suggestion. We ran the experiment on CLAIM (Ye et al., ACL 2025) and we have added the results with discussion to our manuscript. Here are the results on test set:
Table 1a: Qwen3-VL 4B
| Method | High-res | Low-res | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| en | es | ru | vi | ja | mean | jw | cy | mi | ti | mean | |
| ΔAccuracy | |||||||||||
| Pivot | -1.48 | -1.99 | -0.50 | -1.52 | +0.61 | -0.98 | +0.99 | -1.28 | +0.42 | +2.64 | +0.69 |
| CoT | -1.77 | -0.34 | +0.05 | -0.08 | +0.84 | -0.26 | +1.91 | +0.64 | +0.62 | +0.20 | +0.84 |
| CLAIM | +0.01 | -0.03 | +0.05 | +0.55 | +0.92 | +0.30 | +0.76 | +0.27 | +1.84 | +0.89 | +0.94 |
| ORBIT | +0.49 | +0.80 | +1.44 | +1.37 | +2.09 | +1.24 | +2.70 | +2.22 | +1.80 | +1.76 | +2.12 |
| ΔF1-score | |||||||||||
| Pivot | -3.08 | -3.98 | -2.51 | -3.28 | -1.50 | -2.87 | -0.40 | -2.13 | -1.80 | +3.05 | -0.32 |
| CoT | -3.20 | -2.02 | -1.67 | -1.77 | -1.25 | -1.98 | +0.40 | -0.37 | +0.17 | -2.96 | -0.69 |
| CLAIM | -1.71 | -1.39 | -1.40 | -0.75 | -0.46 | -1.14 | +0.02 | -0.12 | +2.61 | +1.77 | +1.07 |
| ORBIT | -1.63 | -1.09 | -0.53 | -0.44 | +0.16 | -0.71 | +1.70 | +1.84 | +2.56 | +2.68 | +2.20 |
Table 1b: Gemma 4 E2B
| Method | High-res | Low-res | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| en | es | ru | vi | ja | mean | jw | cy | mi | ti | mean | |
| ΔAccuracy | |||||||||||
| Pivot | -1.32 | +0.10 | +1.05 | +0.41 | +1.44 | +0.34 | +1.09 | +0.80 | -0.27 | +3.23 | +1.21 |
| CoT | -1.87 | +0.77 | +0.94 | +0.72 | +0.74 | +0.26 | +0.41 | +0.12 | +0.06 | +0.73 | +0.33 |
| CLAIM | +0.08 | +0.72 | +0.28 | +0.50 | -1.24 | +0.07 | +0.52 | +1.15 | +1.02 | +0.49 | +0.79 |
| ORBIT | +1.48 | +2.59 | +2.54 | +2.61 | +2.32 | +2.31 | +2.45 | +1.45 | +1.39 | +1.34 | +1.66 |
| ΔF1-score | |||||||||||
| Pivot | -1.16 | -0.20 | +1.07 | +0.34 | +1.42 | +0.37 | +1.77 | +0.61 | -2.32 | +4.61 | +1.17 |
| CoT | -1.81 | +0.43 | +0.39 | +0.26 | +0.31 | -0.08 | +1.00 | +1.36 | +1.27 | +2.14 | +1.44 |
| CLAIM | +0.09 | +0.70 | -0.05 | +0.19 | -1.62 | -0.14 | +0.49 | +0.36 | +0.44 | -0.86 | +0.11 |
| ORBIT | +0.58 | +1.30 | +0.57 | +0.65 | +0.58 | +0.74 | +2.86 | +2.40 | +2.26 | +4.05 | +2.89 |
Table 1c: LFM2-VL 3B
| Method | High-res | Low-res | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| en | es | ru | vi | ja | mean | jw | cy | mi | ti | mean | |
| ΔAccuracy | |||||||||||
| Pivot | +0.65 | +0.09 | +2.78 | +3.77 | +1.40 | +1.74 | +2.57 | +1.13 | +1.39 | +5.56 | +2.66 |
| CoT | -2.31 | -1.41 | +1.35 | +3.53 | +1.47 | +0.53 | +2.98 | +2.18 | +2.52 | +2.13 | +2.45 |
| CLAIM | -2.03 | -2.10 | +0.23 | +4.70 | +0.19 | +0.20 | +1.20 | +2.50 | +3.15 | +4.31 | +2.79 |
| ORBIT | +2.11 | +2.28 | +2.65 | +5.15 | +2.78 | +3.00 | +3.06 | +4.42 | +3.64 | +4.30 | +3.85 |
| ΔF1-score | |||||||||||
| Pivot | +0.85 | +0.34 | +3.12 | +4.92 | +1.24 | +2.09 | +3.75 | +0.67 | -0.33 | +3.96 | +2.01 |
| CoT | -4.16 | -2.37 | +0.21 | +5.23 | +3.23 | +0.43 | +3.39 | +1.27 | +1.40 | -2.32 | +0.93 |
| CLAIM | -4.69 | -4.26 | -1.29 | +3.17 | -1.04 | -1.62 | +1.13 | +3.78 | +2.81 | +4.31 | +3.01 |
| ORBIT | +2.98 | +3.40 | +3.64 | +5.69 | +3.66 | +3.87 | +4.33 | +5.30 | +4.58 | +4.01 | +4.55 |
ORBIT surpassed CLAIM in almost all configurations because of the design. Firstly, CLAIM estimates attention shifts and intervenes only at the final input token's attention, limiting its ability to model attention distribution across long contextual evidence in our domain (fact checking). Secondly, CLAIM masks attention toward textual tokens during probe training to select language-specific cross-modal attention heads through an SVM classifier, whereas ORBIT steers full hidden states to preserve tri-modal reasoning across claim, textual evidence, and image evidence. Lastly, original CLAIM processes one target language at a time and requires training an SVM for each attention head with a uniform scaling factor alpha, while ORBIT requires no probe training and automatically balances misaligned claim-evidence language pairs via inverse-norm weighting.
Weakness 3
W3: Reviewer concerns the equation (4)-(5), with the definition μ(target) − μ(anchor), that the addition in equation (9)-(10) pushes the hidden states further from the anchor, causing oversteer in the NEI class.
A: We would like to clarify that the offset in these equations is μ(anchor) − μ(target), leading to vector v pointing from the target toward the anchor. Hence, adding λ * v in equations 9 and 10 pushes the representation to the anchor like our intention. We have added a short explanation about this choice of sign in our manuscript. However, we agree with the reviewer that the ablation of the sign of equations (9)-(10) is helpful, so we ran a small ablation on it with Qwen3-VL and we have added it to the manuscript:
| Method | cy-ru | mi-cy | vi-jw | |||
|---|---|---|---|---|---|---|
| Accuracy | F1-score | Accuracy | F1-score | Accuracy | F1-score | |
| Zero-shot | 44.63 | 43.87 | 39.66 | 39.20 | 46.30 | 45.64 |
| a-v | 43.99 | 42.29 | 36.49 | 36.42 | 46.57 | 45.64 |
| a+v (ORBIT) | 46.81 | 45.13 | 39.85 | 39.74 | 49.48 | 47.45 |
The results confirm our formulation. Steering with a+v consistently improves over zero-shot, whereas reversing the sign a-v pushes representations further away from the anchor and severely degrades both accuracy and F1.
Weakness 1
W1: Reviewer concerns that when ORBIT uses Google Cloud Translation (GCT) for vector calibration, while other translation baselines use weak translators, leading to the question of whether the improvement might come from translation quality, not the algorithm itself.
A: We thank the reviewer for their thorough comments. We would like to clarify the role of Google Cloud Translation in our experiments. We use GCT to replace humans in constructing the dataset. If we apply it again to the translation baselines at inference time, it would result in a round-trip translation and largely reconstruct English text in MOCHEG. Moreover, NLLB-600M is trained on high-quality parallel corpora (combining human-translated NLLB-seed data and large-scale mined bitext), and both baselines run on the same GCT-translated inputs as ORBIT.
They also play different roles. ORBIT's calibration phase is a one-time, offline cost that only uses development split with no leakage from the test set. In practice, these pairs can come from high-quality human translators. Since our focus is resource-constrained deployment with SVLMs, using a stronger commercial translator at inference time would answer a different question (an upper-bound pipeline). We have clarified the deployment framing and GCT's role in the revision.