acceptodds
Under review as a conference paper at ICLR 2027

ORBIT: Offset-based Activation Steering for Cross-lingual Multimodal Fact Verification

Abstract

Multimodal fact verification faces a fundamental accessibility bottleneck: while large proprietary models exhibit cross-lingual robustness, small vision-language models suffer a severe, disproportionate accuracy collapse on low-resource languages. Closing this gap traditionally requires prohibitive scale or extensive language-specific fine-tuning. In this work, we propose Offset Recovery via Bilingual Inference-time Technique (ORBIT), an algorithmic intervention that mitigates this collapse at inference time. ORBIT estimates span-specific offset vectors from parallel claim–evidence instances and applies them to claim and evidence representations, targeting the cross-modal grounding failure induced by low-resource evidence. Experiments across 81 configurations reveal that ORBIT substantially recovers both accuracy and F1-score in low-resource settings, exposes calibration offset norms as a diagnostic proxy, and incurs lower overhead than translation-based pipelines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Fact Verification, Cross-Lingual Transfer, Small Vision-Language Models, Low-Resource Languages, Activation Steering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rNbKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies cross-lingual multimodal fact verification in small vision-language models and constructs MOCHEG-XL, a machine-translated extension of MOCHEG to eight target languages that yields a 9×9 grid of claim/evidence language configurations with visual content held fixed. The authors show that high-resource languages degrade modestly while low-resource evidence languages (Javanese, Welsh, Māori, Tigrinya) collapse to near-random performance, with evidence language rather than claim language driving the failure. They propose ORBIT, a training-free, test-time intervention that estimates span-specific language offset vectors from parallel calibration instances and adds them to the claim and evidence hidden states at an intermediate layer. Across three SVLMs and 81 configurations, ORBIT improves low-resource accuracy (e.g., +2.12 points on Qwen3-VL, with bootstrap CIs excluding zero) at lower overhead than translation pipelines, though the absolute performance remains near chance.

Critical strengths
  1. The controlled diagnostic design is the paper's strongest point. By holding visual content constant while sweeping claim and evidence languages over an 81-configuration grid, MOCHEG-XL isolates the linguistic effect from the visual encoder, and the pattern is replicated across three different architectures. The evidence-language-dominant collapse — and the disappearance of the same-language diagonal for jw/cy/mi/ti — is a non-obvious finding that turns a vague intuition about cross-lingual failure into a quantified, locatable phenomenon.

  2. The intervention is genuinely lightweight and deployment-friendly. ORBIT requires no training, no labels at calibration, and only a single-layer injection with modest overhead, which is well aligned with the resource-constrained, local deployment setting that motivates the paper. The offset norm also offers a promising, cheap proxy for screening language risk without running a full evaluation.

Critical weaknesses
  1. The comparison is unfair because the methods are not given translation resources of comparable quality. Both translation baselines rely on deliberately weak translators — Self-translate CoT asks the 4B model itself to translate evidence passages up to 1,774 words, and Pivot NLLB uses the 600M distilled NLLB — whereas ORBIT's own calibration pairs are produced by Google Cloud Translation, a much stronger system. The baselines' weaker performance may therefore stem from their translators rather than from ORBIT itself; I would suggest re-running the comparison with the same translation source on both sides.

  2. The paper only compares against two relatively old baselines and does not cover closely related work from 2025–2026. In particular, CLAIM (Ye et al., ACL 2025), a training-free, test-time intervention that likewise estimates English-to-target shift vectors for multilingual VLMs, is neither cited nor run. I would suggest including CLAIM as a baseline and explicitly positioning ORBIT against it so that the novelty of the proposed method is clear.

  3. A question about the method: in Eqs. (4)–(5) the offset is defined as μ(target) − μ(anchor), and Eqs. (9)–(10) add this vector to the target-language hidden states, which pushes them one step further away from the anchor — the opposite of what "realigning toward the anchor" suggests. Should the vector not be subtracted, or defined in the reverse direction? As written, the intervention seems to amplify language specificity rather than realign it, which would also explain why the NEI class is oversteered in Tables 6 and 10; a +v vs. −v ablation would clarify this.

Questions for authors

Please See Weakness.

5 comments
AuthorsOpenReview ↗

Weakness 1

W1: Reviewer concerns that when ORBIT uses Google Cloud Translation (GCT) for vector calibration, while other translation baselines use weak translators, leading to the question of whether the improvement might come from translation quality, not the algorithm itself.

A: We thank the reviewer for their thorough comments. We would like to clarify the role of Google Cloud Translation in our experiments. We use GCT to replace humans in constructing the dataset. If we apply it again to the translation baselines at inference time, it would result in a round-trip translation and largely reconstruct English text in MOCHEG. Moreover, NLLB-600M is trained on high-quality parallel corpora (combining human-translated NLLB-seed data and large-scale mined bitext), and both baselines run on the same GCT-translated inputs as ORBIT.

They also play different roles. ORBIT's calibration phase is a one-time, offline cost that only uses development split with no leakage from the test set. In practice, these pairs can come from high-quality human translators. Since our focus is resource-constrained deployment with SVLMs, using a stronger commercial translator at inference time would answer a different question (an upper-bound pipeline). We have clarified the deployment framing and GCT's role in the revision.

AuthorsOpenReview ↗

Weakness 2

W2: Missing comparison with novel, related methods

A: We thank the reviewer for their great suggestion. We ran the experiment on CLAIM (Ye et al., ACL 2025) and we have added the results with discussion to our manuscript. Here are the results on test set:

Table 1a: Qwen3-VL 4B

MethodHigh-resLow-res
enesruvijameanjwcymitimean
ΔAccuracy
Pivot-1.48-1.99-0.50-1.52+0.61-0.98+0.99-1.28+0.42+2.64+0.69
CoT-1.77-0.34+0.05-0.08+0.84-0.26+1.91+0.64+0.62+0.20+0.84
CLAIM+0.01-0.03+0.05+0.55+0.92+0.30+0.76+0.27+1.84+0.89+0.94
ORBIT+0.49+0.80+1.44+1.37+2.09+1.24+2.70+2.22+1.80+1.76+2.12
ΔF1-score
Pivot-3.08-3.98-2.51-3.28-1.50-2.87-0.40-2.13-1.80+3.05-0.32
CoT-3.20-2.02-1.67-1.77-1.25-1.98+0.40-0.37+0.17-2.96-0.69
CLAIM-1.71-1.39-1.40-0.75-0.46-1.14+0.02-0.12+2.61+1.77+1.07
ORBIT-1.63-1.09-0.53-0.44+0.16-0.71+1.70+1.84+2.56+2.68+2.20

Table 1b: Gemma 4 E2B

MethodHigh-resLow-res
enesruvijameanjwcymitimean
ΔAccuracy
Pivot-1.32+0.10+1.05+0.41+1.44+0.34+1.09+0.80-0.27+3.23+1.21
CoT-1.87+0.77+0.94+0.72+0.74+0.26+0.41+0.12+0.06+0.73+0.33
CLAIM+0.08+0.72+0.28+0.50-1.24+0.07+0.52+1.15+1.02+0.49+0.79
ORBIT+1.48+2.59+2.54+2.61+2.32+2.31+2.45+1.45+1.39+1.34+1.66
ΔF1-score
Pivot-1.16-0.20+1.07+0.34+1.42+0.37+1.77+0.61-2.32+4.61+1.17
CoT-1.81+0.43+0.39+0.26+0.31-0.08+1.00+1.36+1.27+2.14+1.44
CLAIM+0.09+0.70-0.05+0.19-1.62-0.14+0.49+0.36+0.44-0.86+0.11
ORBIT+0.58+1.30+0.57+0.65+0.58+0.74+2.86+2.40+2.26+4.05+2.89

Table 1c: LFM2-VL 3B

MethodHigh-resLow-res
enesruvijameanjwcymitimean
ΔAccuracy
Pivot+0.65+0.09+2.78+3.77+1.40+1.74+2.57+1.13+1.39+5.56+2.66
CoT-2.31-1.41+1.35+3.53+1.47+0.53+2.98+2.18+2.52+2.13+2.45
CLAIM-2.03-2.10+0.23+4.70+0.19+0.20+1.20+2.50+3.15+4.31+2.79
ORBIT+2.11+2.28+2.65+5.15+2.78+3.00+3.06+4.42+3.64+4.30+3.85
ΔF1-score
Pivot+0.85+0.34+3.12+4.92+1.24+2.09+3.75+0.67-0.33+3.96+2.01
CoT-4.16-2.37+0.21+5.23+3.23+0.43+3.39+1.27+1.40-2.32+0.93
CLAIM-4.69-4.26-1.29+3.17-1.04-1.62+1.13+3.78+2.81+4.31+3.01
ORBIT+2.98+3.40+3.64+5.69+3.66+3.87+4.33+5.30+4.58+4.01+4.55

ORBIT surpassed CLAIM in almost all configurations because of the design. Firstly, CLAIM estimates attention shifts and intervenes only at the final input token's attention, limiting its ability to model attention distribution across long contextual evidence in our domain (fact checking). Secondly, CLAIM masks attention toward textual tokens during probe training to select language-specific cross-modal attention heads through an SVM classifier, whereas ORBIT steers full hidden states to preserve tri-modal reasoning across claim, textual evidence, and image evidence. Lastly, original CLAIM processes one target language at a time and requires training an SVM for each attention head with a uniform scaling factor alpha, while ORBIT requires no probe training and automatically balances misaligned claim-evidence language pairs via inverse-norm weighting.

AuthorsOpenReview ↗

Weakness 3

W3: Reviewer concerns the equation (4)-(5), with the definition μ(target) − μ(anchor), that the addition in equation (9)-(10) pushes the hidden states further from the anchor, causing oversteer in the NEI class.

A: We would like to clarify that the offset in these equations is μ(anchor) − μ(target), leading to vector v pointing from the target toward the anchor. Hence, adding λ * v in equations 9 and 10 pushes the representation to the anchor like our intention. We have added a short explanation about this choice of sign in our manuscript. However, we agree with the reviewer that the ablation of the sign of equations (9)-(10) is helpful, so we ran a small ablation on it with Qwen3-VL and we have added it to the manuscript:

Methodcy-rumi-cyvi-jw
AccuracyF1-scoreAccuracyF1-scoreAccuracyF1-score
Zero-shot44.6343.8739.6639.2046.3045.64
a-v43.9942.2936.4936.4246.5745.64
a+v (ORBIT)46.8145.1339.8539.7449.4847.45

The results confirm our formulation. Steering with a+v consistently improves over zero-shot, whereas reversing the sign a-v pushes representations further away from the anchor and severely degrades both accuracy and F1.

Reviewer_PFZ8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MOCHEG-XL, a machine-translated, 9-language version of MOCHEG. They find that small VLMs drop to near-chance accuracy when the evidence is in a low-resource language. They propose ORBIT, a training-free method that adds offset vectors, estimated from parallel data, to the hidden states of the claim and evidence spans. With low-resource evidence, ORBIT improves results by about 2–4 points across three models, and it is faster than translation-based baselines.

Critical strengths
  1. Clean diagnostic. The images stay fixed while all 81 language pairs are tested. This shows that low-resource evidence drives a near-chance collapse in small VLMs, and the result holds across all three models. It is the paper’s most solid and reusable contribution.

  2. Practical, training-free fix. ORBIT is cheap (about 19% overhead) and improves low-resource results on three architectures, which suits local, low-resource fact-checking. This only holds if the gains are shown to come from the estimated directions themselves.

Critical weaknesses
  1. No evidence that the gains come from realignment. The effect barely changes with calibration size (N=32–372) or layer, and λ=0.05 sits at the bottom of the search grid. The gains also come mostly from fewer NEI predictions (NEI F1 −8 to −12 on high-resource evidence). Without random-vector, wrong-language, or label-prior calibration controls, ORBIT may simply be a label-bias shift rather than cross-lingual steering.

  2. The core claim is not supported. The paper attributes the collapse to cross-modal grounding failure, but there is no text-only ablation to separate it from simply failing to understand low-resource text, or from translation noise (mi/ti evidence adequacy ≈ 76).

Questions for authors
  1. Under Eq. 11, English claims or evidence give a zero shift. Why do Figs. 8/11/14 show non-zero gains in the en row and column?

  2. How does ORBIT compare with a random vector of the same norm, or with another language’s vector (e.g., using jw vectors on ti evidence)?

  3. How does the distribution of predicted labels change after steering, and does a simple label-bias calibration give similar gains?

I am willing to raise my score if the authors address these questions.

6 comments

Q1: Equation (11) appears to give a zero shift whenever either the claim or evidence is in English (n_c = 0 or n_e = 0), so why do Figures 8, 11, and 14 show non-zero gains in the en row and column?

A: We thank the reviewer for pointing out this omission in equation (11). The results in figures 8, 11, and 14 are unchanged because we unintentionally left out the two single-English boundary cases when writing the equation (11). The inverse-norm weighting only applies when both spans are non-English (n_c, n_e > 0). And when one span is non-English, we apply lambda_e = lambda if n_c = 0 and lambda_c = lambda if n_e = 0, which explains the non-zero changes in the English row and column (it was also our initial single-span steering version). We have updated equation (11) in the revised manuscript to include these two cases explicitly.


Q2: How does ORBIT compare with a wrong-language steering vector or a same-norm random vector to confirm that the steering direction actually matters?

A: We evaluated ORBIT on Qwen3-VL on the validation split with a mismatched language vector (es-ti) and a label-bias calibration baseline across three representative pairs (cy-ru, mi-cy, vi-jw):

Methodcy-rumi-cyvi-jw
AccuracyF1-scoreAccuracyF1-scoreAccuracyF1-score
Zero-shot44.6343.8739.6639.2046.3045.64
Different vector (es-ti)44.6342.9936.4936.4148.8446.95
Label-bias calibration44.9943.1939.2539.3145.9145.79
ORBIT46.8145.1339.8539.7449.4847.45

Using a mismatched vector is unstable: it causes a drop on mi-cy and lowers F1-score on cy-ru. By contrast, ORBIT consistently improves both accuracy and F1-score across all three pairs, showing that language-matched steering directions matter. We have included these results in the revised version.


Q3: How does steering change the predicted label distribution, and can a simple label-bias calibration achieve similar gains?

A: Yes, steering shifts predictions away from NEI toward Supported and Refuted, as shown per class in appendix H. On high-resource evidence, this lowers NEI F1. While on low-resource evidence, Supported and Refuted F1 improve strongly with much smaller NEI drops, producing net positive Macro-F1. Simple label-bias calibration does not achieve similar gains. As shown in the two tables in Q2, label-bias calibration barely changes performance and even degrades it relative to zero-shot pairs, whereas ORBIT outperforms both zero-shot and label-bias calibration across all pairs.

Thank you for the response. The clarification of Eq. 11 resolves Q1. The new experiments are helpful, but two main points remain:

  1. The control experiments are not yet convincing. It cover only one model and three pairs, and the same-norm random vector is still missing.

  2. The text-only ablation is not addressed.

In addition, could you clarify how the label-bias calibration baseline is implemented?

AuthorsOpenReview ↗

Reply to point 1

Thank you for reading our rebuttal and pointing out 2 missing main points.

Point 1 (Q2 & Q3): The control experiments are not yet convincing (only one model and three pairs), and the same-norm random vector is still missing.

A: We ran more experiments on 3 models (following the setup of the other experiments in the paper) and 6 representative pairs (cy-ru, mi-cy, vi-jw, mi-ti, ti-ja, ru-jw), which cover low-resource languages on the claim side, the evidence side, and both sides. We also ran the same-norm random vector ablation. All the ablation results below are from the validation split.

Table 1: Compact results

MethodQwen3-VLGemma4LFM2-VL
AccuracyF1-scoreAccuracyF1-scoreAccuracyF1-score
Different vector (es-ti)-0.12-0.96-0.10-0.34+1.53+1.21
Same-norm random vector+0.50-0.19+0.70-0.14+2.41+1.51
Label-bias calibration-0.29+0.02+0.13+0.65+1.48+1.56
ORBIT+1.98+1.46+3.19+2.18+5.76+5.18

Table 2 shows that ORBIT's improvements are not explained by any of the controls. On all models we tested, ORBIT rises in both metrics. Using a mismatched vector is unstable in different pairs/models. Label-bias calibration barely improves the performance in the two better models (Qwen3-VL and Gemma4), and a random vector with the same norm improves accuracy a bit, but F1 remains unchanged. On LFM2-VL (the weakest model), all three controls also improve over zero-shot – each recovers about a quarter or more of ORBIT's gain – but ORBIT still exceeds the best control by more than three points. We have added these results in our revised manuscript.

Regarding label-bias calibration implementation, we learn three per-class offsets for the label logits with 372 calibration samples, so that the mean predicted probability of each class matches the label prior and add them at inference time. Since the label distribution is balanced (Table 3), this is a uniform prior and no ground-truth labels are used. This baseline corrects for over- or under-predicting a class such as NEI, so it tests whether ORBIT's gain is just a shift in predicted label proportions.

Table 2: Full ablated results

Methodcy-rumi-cyvi-jwmi-titi-jaru-jw
AccF1AccF1AccF1AccF1AccF1AccF1
Qwen3-VL
Zero-shot44.6343.8739.6639.2046.3045.6435.1933.5537.6136.4446.5145.85
Different vector (es-ti)44.6342.9936.4936.4148.8446.9534.4429.6636.9437.0047.8245.77
Same-norm random vector44.5442.8238.2838.2547.5545.2635.8732.1337.3937.7149.2747.22
Label-bias calibration44.9943.1939.2539.3145.9145.7935.2133.1836.6236.5946.1546.63
ORBIT46.8145.1339.8539.7449.4847.4537.3535.5038.9438.0649.3747.43
Gemma4
Zero-shot34.5334.8733.4531.7736.7636.5830.7728.2635.7835.8235.3335.52
Different vector (es-ti)34.3733.6533.4532.3236.8835.9930.2330.3436.5534.5834.5233.92
Same-norm random vector37.2235.2232.4632.7237.0235.4430.6529.1337.4334.5936.0634.91
Label-bias calibration35.8734.9233.9034.1836.4036.2929.8729.8536.1236.1335.2435.36
ORBIT38.9136.8234.3534.9741.2338.0131.5731.7038.8236.4140.8837.96
LFM2-VL
Zero-shot30.1629.4529.6527.4730.7930.0129.5327.7934.8131.5830.5228.27
Different vector (es-ti)32.7531.5231.5329.2233.8431.5031.2029.0034.1631.5731.1429.00
Same-norm random vector35.0132.8632.8828.5031.9230.8630.2028.2135.7034.0034.2329.22
Label-bias calibration33.9430.9630.1330.2932.1530.3330.9329.0734.5332.5132.6330.77
ORBIT38.5536.1935.1532.6637.6634.2932.2930.9240.4338.6535.9632.91
Reviewer_n3bSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-lingual multimodal fact verification with small vision-language models. It constructs MOCHEG-XL by translating MOCHEG into eight additional languages and evaluates 81 claim-evidence language configurations. The paper observes substantial degradation when textual evidence is in low-resource languages. To mitigate this issue, it proposes ORBIT, which estimates language-specific activation offsets from parallel calibration instances and applies them separately to claim and evidence spans. Experiments on Qwen3-VL 4B, Gemma 4 E2B, and LFM2-VL 3B report improvements over zero-shot and translation-based baselines, with a trade-off in Qwen3-VL between low-resource recovery and high-resource macro F1.

Critical strengths
  • The paper addresses a useful problem for locally deployable vision-language models. Evaluating 81 claim-evidence language configurations exposes cross-lingual weaknesses that aggregate multilingual scores can hide.
  • The method requires no gradient updates and is evaluated on three model architectures with separate calibration, validation, and test sets. Full language-pair results, inference-time measurements, and paired bootstrap intervals for Qwen3-VL help assess the magnitude and cost of the reported gains.
Critical weaknesses
  • There is an apparent inconsistency between the equations and results. For English claims, Eq. (4) gives v_c = 0 and n_c = 0. If the evidence offset is nonzero, Eq. (11) gives lambda_e = 0, while lambda_c * v_c is also zero. Eqs. (9)-(11) therefore apply no intervention, yet Figure 8 reports nonzero changes in the English row and column, including +3.04 accuracy points for English claims with Javanese evidence. The near-zero-total-norm fallback does not resolve this single-zero case. This directly affects reproducibility of the central method.
  • The proposed explanation and mechanism are not sufficiently isolated. Keeping images fixed does not distinguish cross-modal grounding failure from text comprehension or translation errors, especially given the lower translation scores for some low-resource languages in Appendix B. Comparisons also lack suitable steering baselines and component controls for sign correction and span weighting. Thus, the gains are interesting, but the current evidence does not establish why this intervention works.
Questions for authors

Please refer to the weaknesses.

1 comment

W1: The equation (11) contradicts figure 8, when one span is English (norm = 0), inverse-norm weighting zeroes out the other span's weight, implying no steering

A: We thank the reviewer for pointing out this omission in equation (11). The results in figure 8 are unchanged because we unintentionally left out the two single-English boundary cases when writing the equation. To be specific, the inverse-norm weighting only applies when n_c and n_e > 0 (non-English). By contrast, when only one span is non-English, we apply lambda_e = lambda if n_c = 0 and lambda_c = lambda if n_e = 0, which explains the nonzero changes in the English row and column – it was also our initial single-span steering version. We have updated equation (11) in the revised manuscript to include these two cases and ensure full reproducibility.


W2: Keeping fixed images cannot isolate visual grounding failure from text comprehension or translation errors. The paper lacks steering baselines and ablation for sign correction and span weighting.

A: We thank the reviewer for this constructive feedback. We agree that keeping images fixed cannot completely separate text reading errors from visual grounding errors. However, appendix B shows that the translated texts still keep the main facts, and ORBIT improves results on the same translated inputs without changing any words, so we believe that the drop is not just caused by translation errors. For steering baselines, we recently ran CLAIM (Ye et al., ACL 2025) (thanks to the reviewer rNbK's suggestion). Our method beats CLAIM on almost all configurations, especially on low-resource evidence (like +2.12 vs. +0.94 accuracy on Qwen3-VL). We believe that it is because ORBIT steers the full claim and evidence spans, while CLAIM only steers the last token.

Finally, our direction test (a+v vs. a-v in reply 3 to reviewer rNbK) shows that the gain comes from moving toward English. We also ran component ablations on Qwen3-VL for both sign-aligned aggregation (equation 8) and inverse-norm span weighting (equation 11).

Table 1: Aggregation (equation 8)

Methodcy-rumi-cyvi-jw
AccuracyF1-scoreAccuracyF1-scoreAccuracyF1-score
Zero-shot44.6343.8739.6639.2046.3045.64
Plain mean43.8342.3137.5737.3548.1946.11
Unit-normalized mean43.8342.3337.3037.1147.9345.96
Sign-aligned mean (ORBIT)46.8145.1339.8539.7449.4847.45

Table 2: Span weighting (equation 11)

Methodcy-rumi-cyvi-jw
AccuracyF1-scoreAccuracyF1-scoreAccuracyF1-score
Zero-shot44.6343.8739.6639.2046.3045.64
Direct-norm weighting44.6543.2238.3938.1548.9346.88
Uniform weighting44.4543.1738.5738.3348.5746.42
Inverse-norm weighting (ORBIT)46.8145.1339.8539.7449.4847.45

Both tables confirm that each component is necessary. Replacing sign alignment with plain mean (directly averaging per-sample shifts) or unit-normalized mean (normalizing each sample shift to unit length before averaging) results in weaker performance. Similarly, replacing inverse-norm weighting with direct-norm weighting (lambda_c = lambda * n_c/sum_n and lambda_e = lambda * n_e/sum_n) or uniform weighting (lambda_c = lambda_e = lambda/2) drops both accuracy and F1 below zero-shot on cy-ru and mi-cy, whereas ORBIT consistently improves across all pairs. We have included these full ablation tables in the revised appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Author Response

Dear PC, SAC, AC, and reviewers,

We sincerely thank you for your constructive feedback and valuable suggestions, which have helped us improve our manuscript. Our work investigates cross-lingual multimodal fact verification for small VLMs, with the key contributions:

  • Controlled diagnostic analysis: We expose an evidence-language-dominant collapse in low-resource languages that aggregate multilingual scores often hide by holding visual evidence fixed across claim-evidence language configurations.
  • Lightweight, training-free intervention (ORBIT): We propose a label-free, single-layer activation-steering method with separate claim and evidence offset vectors. On average, it recovers more low-resource accuracy than translation-based and steering baselines, and it is cheaper at inference than translation pipelines.
  • Cheap language-risk proxy: On Qwen3-VL, calibration offset norms strongly correlate with zero-shot collapse, making it cheap to screen without full evaluation.

Moreover, we have updated our manuscript based on the received feedback (highlighted in blue):

  • We have added the baseline results of CLAIM (Ye et al., ACL 2025) across 3 small VLMs on the test set to compare with ORBIT. The results demonstrate consistent gains of ORBIT, and we also explicitly position ORBIT against it.
  • We have added control experiments on 3 models and 6 language pairs in the appendix, including mismatched-language vector, same-norm random vector, and label-bias calibration. These results help confirm that ORBIT improvements come from language-matched steering rather than from random noise or label-prior shifts.
  • We have added component and direction experiments to validate sign-aligned aggregation (equation 8), inverse-norm span weighting (equation 11), and steering direction (a+v vs. a-v in equations 9-10). Additionally, we clarified the offset sign definition (equations 4-5 and equations 9-10) as well as the role of Google Cloud Translation in our experimental setup.
  • We have added text-only zero-shot experiments on 3 models across 6 language pairs to distinguish understanding low-resource languages and cross-modal grounding, made our discussion of the collapse's causes and limitations more precise.
  • We have updated equation 11 to include the single-English boundary cases for full reproducibility.

We have provided further details in our responses and we are glad that our rebuttal successfully addressed the main concerns of reviewers rNbK and PFZ8. We greatly appreciate all reviewers' support in raising or maintaining their scores to 3 (confidence 4, reviewer rNbK), 3 (confidence 3, reviewer PFZ8), and 3 (confidence 3, reviewer n3bS), leading to a final average score of 3.

Thank you again for your time and dedication in reviewing our submission.

Best regards,

The Authors of Submission #23051

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.