- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates why vision-language models (VLMs) often perform worse on visual inputs than on semantically equivalent textual inputs. Through causal interventions, the authors identify a cross-modal relay state where visual and textual representations become executable by the shared decoder, but show that identical relay states can still produce different predictions under different contexts, revealing that VLM decisions are jointly determined by state formation and context-conditioned execution. They further decompose the modality gap into Formation and Execution effects, finding that state formation is generally the dominant factor while post-relay Attention-driven context reading also contributes to visual errors. Based on these insights, they propose Factorized Counterfactual Relay Distillation (FCRD), a lightweight correction module that separately updates relay states and surrounding contexts without modifying the original VLM, achieving significant improvements in visual accuracy and closing over 60% of the text–vision performance gap across multiple VLMs and tasks.
Critical strengths
- Rigorous causal analysis of VLM decision mechanisms. The paper goes beyond correlational interpretability by applying systematic causal interventions to identify how visual and textual information contribute to final predictions. The introduction of the cross-modal relay state and the state interchange experiments provide strong evidence that modality-specific representations converge into a shared decision interface before the final prediction stage. This enables the authors to distinguish whether errors originate from incorrect state formation or downstream execution, offering a more precise explanation of the vision-language performance gap than previous studies.
- Strong connection between mechanistic understanding and practical improvement. A major strength of the work is that it does not stop at analyzing model behavior but translates the discovered mechanism into an effective correction method. The proposed Factorized Counterfactual Relay Distillation (FCRD) directly leverages the identified Formation and Execution factors to design lightweight state and context corrections while keeping the original VLM frozen. The method achieves consistent improvements across multiple VLMs and tasks with only 0.0018%–0.0032% trainable parameters, demonstrating that mechanistic insights can lead to practical and parameter-efficient model enhancement.
Critical weaknesses
- Limited task diversity and uncertainty of generalization. Although the paper evaluates four VLMs across five paired tasks, these tasks are relatively controlled and simplified (e.g., counting, arithmetic, color ordering, factual recall, and sentiment analysis). Such benchmarks are effective for mechanistic analysis, but they may not fully represent the complexity of real-world multimodal reasoning tasks. It remains unclear whether the identified Formation–Execution decomposition and the effectiveness of FCRD generalize to more challenging settings such as open-ended visual reasoning, long-context understanding, or embodied multimodal tasks.
- The identification of the relay state relies on heuristic intervention criteria. The paper defines the relay layer as the earliest layer where bidirectional cross-modal interchange accuracy exceeds a fixed 75% threshold. While this provides an operational definition, the choice of threshold is somewhat heuristic, and the sensitivity of the discovered relay states to this criterion is not fully explored. A stronger validation would include demonstrating that the relay phenomenon remains consistent under different thresholds or alternative definitions of decision transfer, ensuring that the proposed Formation–Execution decomposition is not dependent on a particular experimental setting.
Questions for authors
- How sensitive are the identified relay states and the Formation–Execution decomposition to the choice of the 75% IIA threshold? The relay layer is defined as the earliest layer where bidirectional cross-modal interchange accuracy exceeds a fixed 75% threshold. Could the authors provide an analysis showing whether the discovered relay locations and the resulting Formation/Execution effects remain consistent under different threshold choices or alternative relay definitions?
- How well does FCRD generalize beyond the controlled paired tasks used in this paper? The proposed method is evaluated mainly on five paired tasks designed for mechanistic analysis, along with IsoBench and TabMWP. Could the authors clarify whether FCRD maintains its effectiveness on more challenging real-world multimodal reasoning benchmarks or open-ended generation tasks, where the decision process may involve longer contexts and more complex visual reasoning?
- Does the proposed Formation–Execution decomposition remain consistent across different VLM architectures beyond adapter-based models? The experiments focus on four transformer-based VLMs with similar adapter architectures. Could the authors discuss whether the observed relay state and context-conditioned execution mechanism are expected to hold for other architectures, such as models with different vision encoders, multimodal fusion strategies, or end-to-end trained vision-language systems?
17 comments
Response to Reviewer Ared of Critical Weakness 1 [part 1/2]: Generalization to Public Reasoning and Visual Question Answering
To further evaluate generalization beyond the five controlled tasks, we assessed FCRD on IsoBench [1], TabMWP [2] and TextVQA [3]. Specifically, the added TextVQA experiment applies correctors trained on paired source inputs to 5000 validation examples with the correctors frozen. Consequently, deployment uses target images and questions without paired target descriptions or target training. Additionally, CE only and LoRA [4] use target answer labels for adaptation and are evaluated on the same target examples.
Complete public benchmark results: accuracy on IsoBench and TabMWP; consensus scores on TextVQA (%)
| Dataset | Setting | Qwen2-VL-7B | Pixtral-12B | Gemma-3-12B | Qwen3-VL-4B |
|---|---|---|---|---|---|
| IsoBench | Baseline | 64.54% | 58.67% | 62.08% | 71.25% |
| IsoBench | FCRD | 65.67% (+1.13) | 70.84% (+12.17) | 63.49% (+1.41) | 77.50% (+6.25) |
| TabMWP | Baseline | 68.37% | 75.65% | 71.23% | 72.64% |
| TabMWP | FCRD | 71.79% (+3.42) | 77.66% (+2.01) | 76.26% (+5.03) | 75.40% (+2.76) |
| TextVQA | Vision | 78.764% | 68.702% | 67.662% | 82.010% |
| TextVQA | CE only | 77.680% | 68.024% | 67.796% | 80.378% |
| TextVQA | LoRA | 76.730% | 67.934% | 67.670% | 80.272% |
| TextVQA | Frozen FCRD | 78.650% | 68.756% | 67.082% | 82.026% |
| FCRD parameters | Trainable ratio (count) | 0.0024% (0.20M) | 0.0023% (0.28M) | 0.0018% (0.22M) | 0.0032% (0.14M) |
Parentheses in the IsoBench and TabMWP rows give gains in percentage points; results cover the evaluated benchmark subsets.
Analysis
-
Additional visual reasoning demands: Specifically, TextVQA requires reasoning about text in images together with the visual scene and question [3]. Meanwhile, TabMWP requires mathematical reasoning over questions and tabular information [2]. Additionally, IsoBench tests reasoning through matched representations of the same underlying problem [1]. Thus, these evaluations extend the application of FCRD to visual reading and reasoning over structured information beyond the five controlled tasks.
-
Consistent gains on public reasoning benchmarks: Moreover, FCRD improves accuracy in all eight model and benchmark combinations on IsoBench and TabMWP. Specifically, the mean gains computed from the table are 5.24 and 3.31 percentage points, respectively. For example, Pixtral improves from 58.67% to 70.84% on IsoBench and Qwen3 improves from 71.25% to 77.50%. Similarly, Gemma improves from 71.23% to 76.26% on TabMWP and Qwen2 improves from 68.37% to 71.79%. Accordingly, the benefit appears for every evaluated VLM on both public reasoning benchmarks.
-
Frozen deployment on a new visual task: Furthermore, the TextVQA correctors retain their source parameters throughout evaluation on 5,000 examples. Specifically, their mean score is 74.13% compared with 73.47% for CE only and 73.15% for LoRA. Therefore, the frozen source correction exceeds adaptation using target labels by 0.66 and 0.98 percentage points on average. Moreover, FCRD exceeds both comparisons for Qwen2, Pixtral and Qwen3.
-
Concrete advantages on TextVQA: For example, Qwen3 scores 82.026% with frozen FCRD compared with 80.378% for CE only and 80.272% for LoRA, giving advantages of 1.648 and 1.754 percentage points. Similarly, Qwen2 scores 78.650% compared with 77.680% and 76.730%, giving advantages of 0.970 and 1.920 points. Additionally, Pixtral scores 68.756% compared with 68.024% and 67.934%, giving advantages of 0.732 and 0.822 points.
-
The mechanism motivates a reusable correction: First, paired source inputs supply separate targets for state Formation and context conditioned Execution during training. Subsequently, the learned branches compute both updates from native Vision activations on TextVQA. Thus, the effect supervision learned from source tasks supports deployment on a visual question answering benchmark without constructing textual counterparts for its images. Moreover, FCRD uses 0.0018%–0.0032% trainable parameters while keeping the pretrained VLM frozen. Specifically, Qwen3 uses 0.14M corrector parameters and gains 6.25 points on IsoBench and 2.76 points on TabMWP.
Together, the public benchmark gains and frozen TextVQA deployment provide complementary evidence of generalization beyond the original controlled tasks.
Accordingly, Section 5.3, Table 4 and Appendix M.3 report these results.
Reference
[1] Deqing Fu et al. IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations. arXiv:2404.01266, 2024.
[2] Pan Lu et al. Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning. arXiv:2209.14610, 2022.
[3] Amanpreet Singh et al. Towards VQA Models That Can Read. CVPR, 2019.
[4] Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021.
Response to Reviewer Ared of Critical Weakness 1 [part 2/2]: Transfer of Frozen Correctors between Tasks
To test whether a corrector learned on one task can benefit other tasks, we evaluated every Qwen3 source corrector on all four remaining tasks. Specifically, both its learned parameters and source relay layer remain fixed. Subsequently, each target is evaluated with native Vision inputs and no target retraining. Accordingly, the complete matrix measures reuse of both the correction and its insertion location across 20 distinct source and target pairs. Moreover, each row applies one source corrector to every target, so its four transfer results test the breadth of reuse from that source.
Complete Qwen3 transfer matrix: visual accuracy (%)
| Source | Counting | Arithmetic | Color Ordering | Factual Recall | Sentiment | |
|---|---|---|---|---|---|---|
| Counting | 28 | 88.24% | 88.60% | 79.90% | 62.66% | 91.80% |
| Arithmetic | 24 | 81.41% | 95.40% | 79.64% | 63.37% | 90.98% |
| Color Ordering | 30 | 79.32% | 89.10% | 79.95% | 62.26% | 92.21% |
| Factual Recall | 31 | 80.63% | 91.60% | 79.69% | 65.63% | 90.98% |
| Sentiment | 24 | 81.68% | 90.80% | 79.64% | 62.66% | 95.90% |
| Target Vision | — | 80.37% | 88.80% | 79.69% | 62.26% | 90.98% |
Rows denote source tasks and columns denote target tasks; transfer summaries use all 20 off diagonal entries.
Analysis
-
Positive transfer across source and target tasks: Specifically, the frozen correctors preserve or improve accuracy in 80% of the 20 transfers and achieve an overall mean gain of 0.53 percentage points. Moreover, every source corrector has a positive mean gain over its four target tasks. Furthermore, every target task has a positive mean gain over its four source correctors. Thus, the positive average transfer is supported from both the source and target perspectives with all 20 transfers included.
-
One source corrector supports all four other tasks: For example, the Factual Recall corrector preserves or improves accuracy on every target while retaining . Specifically, Arithmetic rises from 88.80% to 91.60% and Counting rises from 80.37% to 80.63%. Meanwhile, Color Ordering and Sentiment retain their original accuracies of 79.69% and 90.98%. Consequently, a correction learned for factual retrieval is reusable for numerical, spatial and sentiment tasks at the same insertion layer.
-
Arithmetic benefits from several different source tasks: Additionally, the Sentiment corrector raises Arithmetic accuracy from 88.80% to 90.80%, giving a gain of 2.00 percentage points. Similarly, the Factual Recall corrector gives a gain of 2.80 points and the Color Ordering corrector raises accuracy to 89.10%. Accordingly, improvements on Arithmetic arise from three distinct source tasks with relay layers 24, 31 and 30. Thus, transfer benefits include corrections learned for different prediction objectives and inserted at different source locations.
-
Transfer also reaches counting, factual recall and sentiment: Furthermore, the Sentiment corrector raises Counting from 80.37% to 81.68%, a gain of 1.31 percentage points. Similarly, the Arithmetic corrector raises Factual Recall from 62.26% to 63.37%, a gain of 1.11 points. Additionally, the Color Ordering corrector raises Sentiment from 90.98% to 92.21%, a gain of 1.23 points. Moreover, the Counting corrector raises Sentiment to 91.80% and Factual Recall to 62.66%. Therefore, useful transfer spans several source and target combinations beyond the largest Arithmetic improvement.
-
The learned correction operates on target representations: First, Formation computes an update from the target decision state and its change from the preceding layer. Subsequently, Execution conditions its context updates on that corrected target state. Consequently, the same learned maps act on the representations produced by each new target input. Furthermore, both updates are applied before the frozen decoder resumes from the next layer, retaining the coupling between state correction and context correction. Moreover, the source layer is retained throughout transfer, so these evaluations use no target Text inputs to select a new relay. Thus, the results test whether the state and context updates learned on a source task remain useful when the task changes.
Together, the complete transfer matrix complements the public benchmark and TextVQA results in the preceding response. Specifically, public benchmarks test FCRD on additional reasoning problems, while frozen deployment tests reuse without fitting the corrector to the target task. These experiments connect the Formation and Execution design to useful correction beyond its original training setting, with the pretrained VLM unchanged.
The complete results appear in Section 5.3, Table 5 and Appendix M.3, with the correction mechanism described in Section 5.1 and Figure 9.
Overall Response to Reviewer Ared
We sincerely thank you for your careful reading and insightful suggestions!
To further provide a better presentation, we have modified and submitted the modified paper PDF to improve the presentation and make the terminology and notation easier to follow.
Furthermore, we have drawn and added a new overview figure of FCRD (Figure 9) to illustrate the two correction branches, which may give readers a better understanding. The changes made in the paper PDF have been marked in blue.
We sincerely recommend that you take a look at the modified paper PDF please.
Below, we will address each of your suggestions and concerns in turn.