acceptodds
Under review as a conference paper at ICLR 2027

Same State, Different Decisions: Disentangling Formation and Context-Conditioned Execution Mechanisms in VLMs

Abstract

Vision-language models (VLMs) often achieve higher accuracy on textual inputs than on semantically matched visual inputs describing the same task instance, with both modalities ultimately processed by the same decoder. We investigate this performance gap by causally delving into the decision process from modality-specific computation, through a cross-modal relay, to the downstream computation that continues to shape the final prediction. Our analysis identifies a relay state at which decision states transfer bidirectionally between visual and textual continuations. Yet identical relay states can still produce different predictions under different retained contexts, revealing that cross-modal executability can emerge before final decision commitment. This exposes a factorized decision process with distinct causal contributions from state formation and context-conditioned execution, while further localization attributes the residual execution effect primarily to post-relay attention reading distributed prefix information. Guided by these findings, we introduce Factorized Counterfactual Relay Distillation (FCRD), which distills Formation and Execution effects into a lightweight corrector that jointly updates the relay state and its surrounding context. Across three VLMs and five paired tasks, our method closes over 60% of the text-vision performance gap on average, demonstrating the consistency of the identified decision structure across model architectures and task types. Our analysis dissects the internal mechanisms underlying this gap and provides principled targets for correcting visual errors.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language Models, Mechanistic Interpretability, Cross-Modal Performance Gap

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AredOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why vision-language models (VLMs) often perform worse on visual inputs than on semantically equivalent textual inputs. Through causal interventions, the authors identify a cross-modal relay state where visual and textual representations become executable by the shared decoder, but show that identical relay states can still produce different predictions under different contexts, revealing that VLM decisions are jointly determined by state formation and context-conditioned execution. They further decompose the modality gap into Formation and Execution effects, finding that state formation is generally the dominant factor while post-relay Attention-driven context reading also contributes to visual errors. Based on these insights, they propose Factorized Counterfactual Relay Distillation (FCRD), a lightweight correction module that separately updates relay states and surrounding contexts without modifying the original VLM, achieving significant improvements in visual accuracy and closing over 60% of the text–vision performance gap across multiple VLMs and tasks.

Critical strengths
  1. Rigorous causal analysis of VLM decision mechanisms. The paper goes beyond correlational interpretability by applying systematic causal interventions to identify how visual and textual information contribute to final predictions. The introduction of the cross-modal relay state and the state interchange experiments provide strong evidence that modality-specific representations converge into a shared decision interface before the final prediction stage. This enables the authors to distinguish whether errors originate from incorrect state formation or downstream execution, offering a more precise explanation of the vision-language performance gap than previous studies.
  2. Strong connection between mechanistic understanding and practical improvement. A major strength of the work is that it does not stop at analyzing model behavior but translates the discovered mechanism into an effective correction method. The proposed Factorized Counterfactual Relay Distillation (FCRD) directly leverages the identified Formation and Execution factors to design lightweight state and context corrections while keeping the original VLM frozen. The method achieves consistent improvements across multiple VLMs and tasks with only 0.0018%–0.0032% trainable parameters, demonstrating that mechanistic insights can lead to practical and parameter-efficient model enhancement.
Critical weaknesses
  1. Limited task diversity and uncertainty of generalization. Although the paper evaluates four VLMs across five paired tasks, these tasks are relatively controlled and simplified (e.g., counting, arithmetic, color ordering, factual recall, and sentiment analysis). Such benchmarks are effective for mechanistic analysis, but they may not fully represent the complexity of real-world multimodal reasoning tasks. It remains unclear whether the identified Formation–Execution decomposition and the effectiveness of FCRD generalize to more challenging settings such as open-ended visual reasoning, long-context understanding, or embodied multimodal tasks.
  2. The identification of the relay state relies on heuristic intervention criteria. The paper defines the relay layer as the earliest layer where bidirectional cross-modal interchange accuracy exceeds a fixed 75% threshold. While this provides an operational definition, the choice of threshold is somewhat heuristic, and the sensitivity of the discovered relay states to this criterion is not fully explored. A stronger validation would include demonstrating that the relay phenomenon remains consistent under different thresholds or alternative definitions of decision transfer, ensuring that the proposed Formation–Execution decomposition is not dependent on a particular experimental setting.
Questions for authors
  1. How sensitive are the identified relay states and the Formation–Execution decomposition to the choice of the 75% IIA threshold? The relay layer is defined as the earliest layer where bidirectional cross-modal interchange accuracy exceeds a fixed 75% threshold. Could the authors provide an analysis showing whether the discovered relay locations and the resulting Formation/Execution effects remain consistent under different threshold choices or alternative relay definitions?
  2. How well does FCRD generalize beyond the controlled paired tasks used in this paper? The proposed method is evaluated mainly on five paired tasks designed for mechanistic analysis, along with IsoBench and TabMWP. Could the authors clarify whether FCRD maintains its effectiveness on more challenging real-world multimodal reasoning benchmarks or open-ended generation tasks, where the decision process may involve longer contexts and more complex visual reasoning?
  3. Does the proposed Formation–Execution decomposition remain consistent across different VLM architectures beyond adapter-based models? The experiments focus on four transformer-based VLMs with similar adapter architectures. Could the authors discuss whether the observed relay state and context-conditioned execution mechanism are expected to hold for other architectures, such as models with different vision encoders, multimodal fusion strategies, or end-to-end trained vision-language systems?
17 comments
AuthorsOpenReview ↗

Overall Response to Reviewer Ared

We sincerely thank you for your careful reading and insightful suggestions!

To further provide a better presentation, we have modified and submitted the modified paper PDF to improve the presentation and make the terminology and notation easier to follow.

Furthermore, we have drawn and added a new overview figure of FCRD (Figure 9) to illustrate the two correction branches, which may give readers a better understanding. The changes made in the paper PDF have been marked in blue.

We sincerely recommend that you take a look at the modified paper PDF please.

Below, we will address each of your suggestions and concerns in turn.

AuthorsOpenReview ↗

Response to Reviewer Ared of Critical Weakness 1 [part 1/2]: Generalization to Public Reasoning and Visual Question Answering

To further evaluate generalization beyond the five controlled tasks, we assessed FCRD on IsoBench [1], TabMWP [2] and TextVQA [3]. Specifically, the added TextVQA experiment applies correctors trained on paired source inputs to 5000 validation examples with the correctors frozen. Consequently, deployment uses target images and questions without paired target descriptions or target training. Additionally, CE only and LoRA [4] use target answer labels for adaptation and are evaluated on the same target examples.

Complete public benchmark results: accuracy on IsoBench and TabMWP; consensus scores on TextVQA (%)

DatasetSettingQwen2-VL-7BPixtral-12BGemma-3-12BQwen3-VL-4B
IsoBenchBaseline64.54%58.67%62.08%71.25%
IsoBenchFCRD65.67% (+1.13)70.84% (+12.17)63.49% (+1.41)77.50% (+6.25)
TabMWPBaseline68.37%75.65%71.23%72.64%
TabMWPFCRD71.79% (+3.42)77.66% (+2.01)76.26% (+5.03)75.40% (+2.76)
TextVQAVision78.764%68.702%67.662%82.010%
TextVQACE only77.680%68.024%67.796%80.378%
TextVQALoRA76.730%67.934%67.670%80.272%
TextVQAFrozen FCRD78.650%68.756%67.082%82.026%
FCRD parametersTrainable ratio (count)0.0024% (0.20M)0.0023% (0.28M)0.0018% (0.22M)0.0032% (0.14M)

Parentheses in the IsoBench and TabMWP rows give gains in percentage points; results cover the evaluated benchmark subsets.

Analysis

  1. Additional visual reasoning demands: Specifically, TextVQA requires reasoning about text in images together with the visual scene and question [3]. Meanwhile, TabMWP requires mathematical reasoning over questions and tabular information [2]. Additionally, IsoBench tests reasoning through matched representations of the same underlying problem [1]. Thus, these evaluations extend the application of FCRD to visual reading and reasoning over structured information beyond the five controlled tasks.

  2. Consistent gains on public reasoning benchmarks: Moreover, FCRD improves accuracy in all eight model and benchmark combinations on IsoBench and TabMWP. Specifically, the mean gains computed from the table are 5.24 and 3.31 percentage points, respectively. For example, Pixtral improves from 58.67% to 70.84% on IsoBench and Qwen3 improves from 71.25% to 77.50%. Similarly, Gemma improves from 71.23% to 76.26% on TabMWP and Qwen2 improves from 68.37% to 71.79%. Accordingly, the benefit appears for every evaluated VLM on both public reasoning benchmarks.

  3. Frozen deployment on a new visual task: Furthermore, the TextVQA correctors retain their source parameters throughout evaluation on 5,000 examples. Specifically, their mean score is 74.13% compared with 73.47% for CE only and 73.15% for LoRA. Therefore, the frozen source correction exceeds adaptation using target labels by 0.66 and 0.98 percentage points on average. Moreover, FCRD exceeds both comparisons for Qwen2, Pixtral and Qwen3.

  4. Concrete advantages on TextVQA: For example, Qwen3 scores 82.026% with frozen FCRD compared with 80.378% for CE only and 80.272% for LoRA, giving advantages of 1.648 and 1.754 percentage points. Similarly, Qwen2 scores 78.650% compared with 77.680% and 76.730%, giving advantages of 0.970 and 1.920 points. Additionally, Pixtral scores 68.756% compared with 68.024% and 67.934%, giving advantages of 0.732 and 0.822 points.

  5. The mechanism motivates a reusable correction: First, paired source inputs supply separate targets for state Formation and context conditioned Execution during training. Subsequently, the learned branches compute both updates from native Vision activations on TextVQA. Thus, the effect supervision learned from source tasks supports deployment on a visual question answering benchmark without constructing textual counterparts for its images. Moreover, FCRD uses 0.0018%–0.0032% trainable parameters while keeping the pretrained VLM frozen. Specifically, Qwen3 uses 0.14M corrector parameters and gains 6.25 points on IsoBench and 2.76 points on TabMWP.

Together, the public benchmark gains and frozen TextVQA deployment provide complementary evidence of generalization beyond the original controlled tasks.

Accordingly, Section 5.3, Table 4 and Appendix M.3 report these results.

Reference

[1] Deqing Fu et al. IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations. arXiv:2404.01266, 2024.

[2] Pan Lu et al. Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning. arXiv:2209.14610, 2022.

[3] Amanpreet Singh et al. Towards VQA Models That Can Read. CVPR, 2019.

[4] Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021.

AuthorsOpenReview ↗

Response to Reviewer Ared of Critical Weakness 1 [part 2/2]: Transfer of Frozen Correctors between Tasks

To test whether a corrector learned on one task can benefit other tasks, we evaluated every Qwen3 source corrector on all four remaining tasks. Specifically, both its learned parameters and source relay layer remain fixed. Subsequently, each target is evaluated with native Vision inputs and no target retraining. Accordingly, the complete matrix measures reuse of both the correction and its insertion location across 20 distinct source and target pairs. Moreover, each row applies one source corrector to every target, so its four transfer results test the breadth of reuse from that source.

Complete Qwen3 transfer matrix: visual accuracy (%)

SourceHHCountingArithmeticColor OrderingFactual RecallSentiment
Counting2888.24%88.60%79.90%62.66%91.80%
Arithmetic2481.41%95.40%79.64%63.37%90.98%
Color Ordering3079.32%89.10%79.95%62.26%92.21%
Factual Recall3180.63%91.60%79.69%65.63%90.98%
Sentiment2481.68%90.80%79.64%62.66%95.90%
Target Vision—80.37%88.80%79.69%62.26%90.98%

Rows denote source tasks and columns denote target tasks; transfer summaries use all 20 off diagonal entries.

Analysis

  1. Positive transfer across source and target tasks: Specifically, the frozen correctors preserve or improve accuracy in 80% of the 20 transfers and achieve an overall mean gain of 0.53 percentage points. Moreover, every source corrector has a positive mean gain over its four target tasks. Furthermore, every target task has a positive mean gain over its four source correctors. Thus, the positive average transfer is supported from both the source and target perspectives with all 20 transfers included.

  2. One source corrector supports all four other tasks: For example, the Factual Recall corrector preserves or improves accuracy on every target while retaining H=31H=31. Specifically, Arithmetic rises from 88.80% to 91.60% and Counting rises from 80.37% to 80.63%. Meanwhile, Color Ordering and Sentiment retain their original accuracies of 79.69% and 90.98%. Consequently, a correction learned for factual retrieval is reusable for numerical, spatial and sentiment tasks at the same insertion layer.

  3. Arithmetic benefits from several different source tasks: Additionally, the Sentiment corrector raises Arithmetic accuracy from 88.80% to 90.80%, giving a gain of 2.00 percentage points. Similarly, the Factual Recall corrector gives a gain of 2.80 points and the Color Ordering corrector raises accuracy to 89.10%. Accordingly, improvements on Arithmetic arise from three distinct source tasks with relay layers 24, 31 and 30. Thus, transfer benefits include corrections learned for different prediction objectives and inserted at different source locations.

  4. Transfer also reaches counting, factual recall and sentiment: Furthermore, the Sentiment corrector raises Counting from 80.37% to 81.68%, a gain of 1.31 percentage points. Similarly, the Arithmetic corrector raises Factual Recall from 62.26% to 63.37%, a gain of 1.11 points. Additionally, the Color Ordering corrector raises Sentiment from 90.98% to 92.21%, a gain of 1.23 points. Moreover, the Counting corrector raises Sentiment to 91.80% and Factual Recall to 62.66%. Therefore, useful transfer spans several source and target combinations beyond the largest Arithmetic improvement.

  5. The learned correction operates on target representations: First, Formation computes an update from the target decision state and its change from the preceding layer. Subsequently, Execution conditions its context updates on that corrected target state. Consequently, the same learned maps act on the representations produced by each new target input. Furthermore, both updates are applied before the frozen decoder resumes from the next layer, retaining the coupling between state correction and context correction. Moreover, the source layer is retained throughout transfer, so these evaluations use no target Text inputs to select a new relay. Thus, the results test whether the state and context updates learned on a source task remain useful when the task changes.

Together, the complete transfer matrix complements the public benchmark and TextVQA results in the preceding response. Specifically, public benchmarks test FCRD on additional reasoning problems, while frozen deployment tests reuse without fitting the corrector to the target task. These experiments connect the Formation and Execution design to useful correction beyond its original training setting, with the pretrained VLM unchanged.

The complete results appear in Section 5.3, Table 5 and Appendix M.3, with the correction mechanism described in Section 5.1 and Figure 9.

Reviewer_PEL6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the text–vision performance gap in VLMs through causal interventions. The authors identify cross-modal relay states and decompose their effects into state Formation and context-conditioned Execution. Based on these findings, they propose FCRD, a lightweight correction method that improves visual accuracy by 6.33 percentage points on average across four VLMs and five tasks.

Critical strengths
  1. The paper presents a systematic causal analysis of cross-modal decision-making, providing interesting insights into how intermediate states and contextual information jointly influence predictions.

  2. The analysis combines multiple intervention strategies, including state interchange, factorial decomposition, and component-level localization, providing complementary evidence for the observed phenomena.

  3. FCRD translates the mechanistic findings into a lightweight correction method, achieving encouraging improvements across multiple VLMs and tasks with minimal trainable parameters.

Critical weaknesses
  1. The Formation–Execution decomposition lacks sufficient causal justification, as replacing contextual representations also changes the task-relevant information available to the model rather than isolating the execution mechanism.

  2. The paper lacks comparisons with simpler supervised representation correction methods, such as CE-only training and parameter-matched representation editing, making it difficult to attribute the improvements specifically to the proposed factorization and counterfactual distillation.

  3. You select the relay layer using a fixed 75% interchange accuracy threshold without sufficient justification. The sensitivity of the mechanistic findings and correction performance to this threshold and alternative relay layers remains unclear.

  4. How do you distinguish the proposed Execution effect from changes in the task-relevant information encoded in contextual representations? Can you provide additional comparisons with simpler supervised correction methods to demonstrate the necessity of counterfactual effect distillation?

Questions for authors

See weeknesses.

10 comments
AuthorsOpenReview ↗

Overall Response to Reviewer PEL6

We sincerely thank you for your careful reading and insightful suggestions!

To further provide a better presentation, we have modified and submitted the modified paper PDF to improve the presentation and make the terminology and notation easier to follow.

Furthermore, we have drawn and added a new overview figure of FCRD (Figure 9) to illustrate the two correction branches, which may give readers a better understanding. The changes made in the paper PDF have been marked in blue.

We sincerely recommend that you take a look at the modified paper PDF please.

Below, we will address each of your suggestions and concerns in turn.

AuthorsOpenReview ↗

Response to Reviewer PEL6 of Critical Weakness 1: Causal Evidence for Formation and Context-Conditioned Execution

To establish the causal basis of the Formation and Execution decomposition, we combine factorial interventions with matched component and source interventions. Specifically, Execution measures the contribution of retained context to the continuation of a fixed relay state, including the task information read from that context. Accordingly, all four conditions combine states and contexts from the same paired instance at HH.

Subsequently, reader tests use the same examples and Vision state to exchange Attention or MLP outputs over the same layers. Moreover, each recipient retains its context at HH and unselected components follow its own computation. Next, source tests exchange one Attention contribution and preserve the others, output projection and attention normalization. Thus, both directions test which computations and source positions carry the context effect.

Complete component and source results (mean bidirectional logit margin scores)

ModelTasknAttentionMLPA+MImagePrefixSelf
Qwen2Counting120.760.410.760.360.43-0.17
Qwen2Arithmetic921.120.181.120.200.940.01
Qwen2Color Ordering250.74-0.630.740.100.620.01
Qwen2Factual Recall250.440.180.440.110.34-0.01
Qwen2Sentiment110.930.180.930.050.93-0.03
PixtralCounting213.27-0.053.27-0.073.240.10
PixtralArithmetic0——————
PixtralColor Ordering60.57-0.850.570.570.16-0.09
PixtralFactual Recall1833.301.713.300.563.010.02
PixtralSentiment655.535.475.530.855.270.21
GemmaCounting364.051.084.050.423.26-0.13
GemmaArithmetic416.02-0.426.021.254.880.10
GemmaColor Ordering2248.98-4.358.980.618.54-0.07
GemmaFactual Recall20615.19-6.1415.190.9013.810.29
GemmaSentiment125.800.485.800.285.61-0.02
Qwen3Counting301.800.611.800.421.46-0.04
Qwen3Arithmetic740.820.390.82-0.151.200.03
Qwen3Color Ordering162.52-0.152.521.011.80-0.11
Qwen3Factual Recall472.530.222.530.142.45-0.20
Qwen3Sentiment84.471.954.47-0.044.580.00

n counts examples where the same Vision state yields the original error under Vision context and the correct answer under Text context; A+M exchanges both components; Prefix denotes earlier nonimage positions and Self the decision position; — denotes n=0.

Analysis

  1. Separating the intervention coordinates: Specifically, Formation averages state effects over both contexts and Execution averages context effects over both states. Thus, both contributions are measured at the same HH and exactly recover the native gap as G=F+EG=F+E. Moreover, the interaction term records how the effect of one coordinate depends on the other. For example, Text context recovers the answer in 90.3% of eligible Pixtral Sentiment cases with the native Vision state fixed. Moreover, the full population Execution effect is 12.91 percentage points for this setting.

  2. Locating the computation carrying contextual information: Furthermore, Attention exceeds MLP in all 19 eligible settings and the paired 95% interval for their difference is above zero in 18. For example, Gemma Factual Recall yields 15.19 for Attention and -6.14 for MLP, with a paired difference of 21.33. Consequently, this matched comparison identifies Attention as a principal carrier of the context effect after the fixed initial state.

  3. Establishing direction through reciprocal interventions: Additionally, Attention exchange increases support for the correct answer in Vision and for the original Vision error in Text in all 19 settings. Specifically, Qwen3 Factual Recall shifts the correct answer versus original error margin by +2.53 in the forward direction and -2.53 in reverse. Thus, the intervention transfers the direction of the contextual decision through a specific component.

  4. Identifying the source of the effect: Moreover, Prefix exceeds Image in 18 of 19 settings. Gemma Factual Recall yields 13.81 versus 0.90 with a paired difference of 12.91. During prefix exchange, image and self contributions retain their current recipient values. Therefore, the experiment localizes the effect to Attention reading preceding context representations. Accordingly, task information carried by these representations contributes through this measured pathway, motivating FCRD's context update alongside its state update.

The definitions and complete results appear in Sections 3.2 and 4, Figures 5–8 and Appendices D, E, G and H in the modified PDF.

AuthorsOpenReview ↗

Response to Reviewer PEL6 of Critical Weakness 2 [part 1/2]: Comparisons with Supervised Representation Correction

To determine how counterfactual distillation contributes beyond simpler supervised representation correction, we compared FCRD with CE only and LoRA [1]. Specifically, CE only retains both FCRD branches, their trainable parameters, the insertion layer and the update regularizer while disabling the two effect losses. Accordingly, it provides supervised representation editing with exactly matched architecture and parameter count. Additionally, LoRA uses the same ground truth labels, seeds, and training budget.

Complete supervised comparison: visual accuracy (%)

ModelTaskVisionCE onlyLoRAFCRD
Qwen2-VL-7BCounting70.94%73.82%75.13%76.18%
Qwen2-VL-7BArithmetic67.60%79.00%76.50%80.10%
Qwen2-VL-7BColor Ordering74.18%74.03%72.47%74.18%
Qwen2-VL-7BFactual Recall68.12%70.17%70.49%71.04%
Qwen2-VL-7BSentiment93.03%95.49%97.13%96.72%
Pixtral-12BCounting66.67%74.47%72.97%74.77%
Pixtral-12BArithmetic21.70%28.00%29.00%29.80%
Pixtral-12BColor Ordering81.05%80.85%78.83%81.45%
Pixtral-12BFactual Recall45.02%47.39%50.95%53.48%
Pixtral-12BSentiment70.08%91.80%95.90%95.08%
Gemma-3-12BCounting73.04%75.13%77.75%82.46%
Gemma-3-12BArithmetic87.60%94.00%95.00%95.70%
Gemma-3-12BColor Ordering46.39%50.65%50.91%52.31%
Gemma-3-12BFactual Recall66.53%68.75%69.62%70.57%
Gemma-3-12BSentiment93.03%96.72%95.90%97.54%
Qwen3-VL-4BCounting80.37%82.98%84.82%85.60%
Qwen3-VL-4BArithmetic88.80%93.90%91.40%95.40%
Qwen3-VL-4BColor Ordering79.69%79.84%79.38%79.95%
Qwen3-VL-4BFactual Recall62.26%64.24%65.27%65.43%
Qwen3-VL-4BSentiment90.98%91.80%93.85%95.90%
MeanAll 20 settings71.35%75.65%76.16%77.68%

Means over five tasks (%) and matched corrector parameters

ModelCE onlyLoRAFCRDParameters in CE only / FCRD (each)
Qwen2-VL-7B78.50%78.34%79.65%200,704
Pixtral-12B64.50%65.53%66.92%286,720
Gemma-3-12B77.05%77.84%79.72%215,040
Qwen3-VL-4B82.55%82.94%84.46%143,360

Analysis

  1. Effect supervision improves the identical corrector: Specifically, FCRD exceeds CE only in all 20 settings and raises mean accuracy from 75.65% to 77.68%. Moreover, this 2.03 point advantage uses exactly the same update architecture and trainable parameter counts shown above. Therefore, the controlled comparison directly attributes the additional improvement to including the counterfactual effect losses.

  2. The advantage holds for every model: Furthermore, the mean gains over CE only are 1.15, 2.42, 2.67 and 1.91 percentage points for Qwen2, Pixtral, Gemma and Qwen3. Additionally, FCRD exceeds LoRA in 18 of 20 settings with a mean advantage of 1.52 percentage points. Thus, each VLM achieves its highest task average with FCRD among these three training methods.

  3. Concrete benefits on individual tasks: For example, Gemma Counting reaches 82.46% compared with 75.13% for CE only and 77.75% for LoRA, yielding advantages of 7.33 and 4.71 percentage points. Similarly, Pixtral Factual Recall reaches 53.48% compared with 47.39% and 50.95%, yielding advantages of 6.09 and 2.53 points. Moreover, Qwen3 Sentiment reaches 95.90% compared with 91.80% and 93.85%, yielding advantages of 4.10 and 2.05 points.

  4. Supervision follows the identified intervention effects: Specifically, Formation matches the logit change from inserting the Text state into Vision, while Execution matches the change from placing the Vision state under Text context. Meanwhile, cross entropy supervises the final prediction produced by their composition. Consequently, the same corrector receives separate targets for its state and context updates in addition to the answer label. Thus, the accuracy improvement measures the practical value of these targets with model capacity fixed. Furthermore, the effect targets are used during training, so inference retains the same two branches and uses only native Vision activations.

Together, these comparisons establish a consistent contribution from counterfactual effect distillation beyond label supervision.

Accordingly, the complete results appear in Section 5.2, Table 2, Appendix M.1 and Table I.1

Reference

[1] Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021.

Reviewer_Mn6cOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to explain and reduce the performance gap between textual and visual inputs in vision-language models. The authors first identify task- and model-specific relay layers, where either modality’s decision-position state can guide the other’s continuation. Each relay layer is selected as the earliest layer at which interchanging this state between modalities reaches 75% interchange intervention accuracy in both directions.

At this layer, the authors split the hidden representation into the state (the vector at the final prompt position) and the context (the vectors at all preceding positions). They call the effect of replacing the Vision state with the Text state “Formation,” and the effect of replacing the Vision context with the Text context “Execution.” Through a 2×2 state–context factorization, they find that Formation accounts for most of the gap, while Execution also contributes, so the same state can still lead to different decisions under different contexts. They further attribute this context effect mainly to post-relay attention reading the non-image prefix positions.

Based on these findings, they propose FCRD, a lightweight corrector at the relay layer that updates both the state and the context. The corrector consists of two small low-rank MLP branches and is trained with ground-truth labels and with the logit changes produced by inserting the Text state or Text context. At inference, it uses only the visual input. The method is evaluated on four VLMs across five paired text–vision tasks, as well as on IsoBench and TabMWP, two public reasoning benchmarks whose problems are available in both visual and textual forms, where it improves visual accuracy in all four models.

Critical strengths
  • Method design follows from the analysis. The design of FCRD is directly motivated by the analysis.
  • Systematic analysis of where the modality gap arises. The paper traces the gap through the model across four VLMs and five tasks, and identifies a consistent structure: a relay layer where the decision state transfers across modalities, with the gap arising mostly from the state and partly from the context.
Critical weaknesses
  • Insufficient validation of the method. The FCRD corrector is trained on ground-truth labels, but it is not compared with other post-training methods (e.g., parameter-matched LoRA, or the same architecture trained with cross-entropy only). It is therefore unclear whether the gains in Table 2 come from label supervision alone or from the design derived from the causal analysis. Similarly, the paper does not test whether inserting the corrector at H is more effective than at other layers (e.g., H ± k).
  • Overstated and limited evaluation. The GC metric can overstate the results. For example, Pixtral-Arithmetic improves from 21.70% to 29.80% against a text accuracy of 25.50%, yielding a GC of 213.16%, which inflates the “over 60%” average in the abstract. In addition, FCRD requires paired Text inputs during training, and the additional benchmarks in Table 3 also provide paired representations, so the paper does not show that the method applies beyond tasks with textual counterparts.
  • Presentation. Sections 3.2, 3.3, and 4 largely report different views of the same 2×2 state–context interchange at H. These could be consolidated, and the space used for the validation experiments above. The terminology and notation are also harder to follow than necessary. “Formation” and “Execution” suggest a split across layers, but the factorization actually splits the residual stream at H by position. Section 6.1 introduces many symbols for a relatively simple two-branch module.
Questions for authors
  • Can FCRD be applied to visual tasks without textual counterparts, and does a corrector trained on one task transfer to others?
  • How sensitive are H and the FCRD results to the 75% IIA threshold?
19 comments
AuthorsOpenReview ↗

Overall Response to Reviewer Mn6c

We sincerely thank you for your careful reading and insightful suggestions!

Follow your valuable feedback, we have modified and submitted the modified paper PDF to improve the presentation and make the terminology and notation easier to follow.

Furthermore, we have drawn and added a new overview figure of FCRD (Figure 9) to illustrate the two correction branches, which may give readers a better understanding. Also, we have accepted your advice and made more room for the validation of experiments. The changes made in the paper PDF have been marked in blue.

Below, we will address each of your suggestions and concerns in turn.

AuthorsOpenReview ↗

Response to Reviewer Mn6c of Critical Weakness 1 [part1/2]: Comparison with CE only and LoRA

To evaluate the contribution of our causal analysis beyond label supervision, we compared FCRD with CE only and LoRA. Specifically, CE only trains the same architecture with cross entropy and the original regularizer while disabling both effect losses. Additionally, LoRA provides a comparison with another post training method using the same ground truth labels. Accordingly, all comparisons use identical seeds, training budgets and five evaluation folds across four VLMs and five tasks.

Complete supervision comparison: visual accuracy (%), with equal weight for all 20 settings

ModelTaskVisionCE onlyLoRAFCRD
Qwen2-VL-7BCounting70.94%73.82%75.13%76.18%
Qwen2-VL-7BArithmetic67.60%79.00%76.50%80.10%
Qwen2-VL-7BColor Ordering74.18%74.03%72.47%74.18%
Qwen2-VL-7BFactual Recall68.12%70.17%70.49%71.04%
Qwen2-VL-7BSentiment93.03%95.49%97.13%96.72%
Pixtral-12BCounting66.67%74.47%72.97%74.77%
Pixtral-12BArithmetic21.70%28.00%29.00%29.80%
Pixtral-12BColor Ordering81.05%80.85%78.83%81.45%
Pixtral-12BFactual Recall45.02%47.39%50.95%53.48%
Pixtral-12BSentiment70.08%91.80%95.90%95.08%
Gemma-3-12BCounting73.04%75.13%77.75%82.46%
Gemma-3-12BArithmetic87.60%94.00%95.00%95.70%
Gemma-3-12BColor Ordering46.39%50.65%50.91%52.31%
Gemma-3-12BFactual Recall66.53%68.75%69.62%70.57%
Gemma-3-12BSentiment93.03%96.72%95.90%97.54%
Qwen3-VL-4BCounting80.37%82.98%84.82%85.60%
Qwen3-VL-4BArithmetic88.80%93.90%91.40%95.40%
Qwen3-VL-4BColor Ordering79.69%79.84%79.38%79.95%
Qwen3-VL-4BFactual Recall62.26%64.24%65.27%65.43%
Qwen3-VL-4BSentiment90.98%91.80%93.85%95.90%
AverageAll 20 settings71.35%75.65%76.16%77.68%
  1. Contribution of effect supervision: Specifically, FCRD reaches 77.68% mean accuracy compared with 75.65% for CE only. Moreover, FCRD improves over CE only in all 20 settings with a mean advantage of 2.03% accuracy. Because the architecture, trainable parameters and regularizer are unchanged, this comparison isolates the contribution of the two effect losses.

  2. Comparison with LoRA: Additionally, FCRD exceeds LoRA in 18 settings with a mean advantage of 1.52%. Furthermore, FCRD achieves higher mean accuracy than both comparisons for every VLM. For example, Qwen3 achieves 84.46% mean accuracy with FCRD compared with 82.55% for CE only and 82.94% for LoRA.

  3. Concrete gains from the causal supervision: Specifically, Gemma Counting reaches 82.46% with FCRD compared with 75.13% for CE only and 77.75% for LoRA. Similarly, Pixtral Factual Recall reaches 53.48% compared with 47.39% and 50.95%, respectively. Therefore, adding the effect targets improves the same architecture by 7.33% and 6.09% on these two settings.

Accordingly, these comparisons and analyses are included in Section 5.2, Table 2 and Appendix M.1 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Mn6c of Critical Weakness 1 [part2/2]: Insertion Layer Sensitivity

To test the choice of insertion layer, we trained FCRD at H−2H-2, HH and H+2H+2 for all 20 settings. Specifically, HH is selected as the earliest layer reaching 75% IIA in both directions before corrector training. Subsequently, each insertion uses the same corrector architecture, objective, seeds, training budget and five evaluation folds. Accordingly, the table reports every setting with the absolute layer and visual accuracy in each cell.

Complete insertion layer comparison: layer index and accuracy (%)

ModelTaskH−2H-2HHH+2H+2
Qwen2-VL-7BCounting20: 76.70%22: 76.18%24: 75.13%
Qwen2-VL-7BArithmetic21: 80.50%23: 80.10%25: 79.90%
Qwen2-VL-7BColor Ordering24: 75.01%26: 74.18%28: N/A
Qwen2-VL-7BFactual Recall21: 71.20%23: 71.04%25: 70.57%
Qwen2-VL-7BSentiment18: 98.77%20: 96.72%22: 96.31%
Pixtral-12BCounting35: 74.17%37: 74.77%39: N/A
Pixtral-12BArithmetic36: 30.10%38: 29.80%40: N/A
Pixtral-12BColor Ordering31: 80.65%33: 81.45%35: 81.25%
Pixtral-12BFactual Recall29: 53.88%31: 53.48%33: 53.16%
Pixtral-12BSentiment17: 93.85%19: 95.08%21: 95.08%
Gemma-3-12BCounting37: 83.25%39: 82.46%41: 82.98%
Gemma-3-12BArithmetic43: 96.40%45: 95.70%47: N/A
Gemma-3-12BColor Ordering42: 51.58%44: 52.31%46: 53.25%
Gemma-3-12BFactual Recall40: 71.28%42: 70.57%44: 68.83%
Gemma-3-12BSentiment26: 97.13%28: 97.54%30: 95.90%
Qwen3-VL-4BCounting26: 84.82%28: 85.60%30: 85.60%
Qwen3-VL-4BArithmetic22: 95.60%24: 95.40%26: 95.10%
Qwen3-VL-4BColor Ordering28: 79.64%30: 79.95%32: 79.84%
Qwen3-VL-4BFactual Recall29: 66.93%31: 65.43%33: 65.59%
Qwen3-VL-4BSentiment22: 96.72%24: 95.90%26: 95.90%

N/A marks a location outside the valid suffix range.

Analysis

  1. Stable performance around the relay: Specifically, the mean absolute accuracy change from HH is 0.62% and the median is 0.50 across the 36 valid neighboring comparisons, which is a very small fluctuation and verifies the robustness of our method as well as the mechanism analysis. Moreover, 83.33% of these comparisons remain within one percentage point. Furthermore, the absolute changes on average are 0.68, 0.48, 0.91 and 0.42 percentage points for Qwen2, Pixtral, Gemma and Qwen3, respectively.

  2. A competitive location selected by causal analysis: Additionally, HH achieves the highest tested accuracy or lies within one percentage point of it in 18 of 20 settings. Thus, the bidirectional IIA criterion identifies a competitive correction location before FCRD training.

  3. Correction gains persist at neighboring layers: For example, Qwen2 Arithmetic achieves 80.50%, 80.10% and 79.90% at H−2H-2, HH and H+2H+2, respectively. Similarly, Qwen3 Arithmetic achieves 95.60%, 95.40% and 95.10%. Consequently, the gains over native Vision remain 12.30 to 12.90 percentage points for Qwen2 and 6.30 to 6.80 for Qwen3 throughout the tested neighborhood.

These results are included in Section 5.4, Table 6, Figure 11 and Appendix M.4 (Table 17) of the revised manuscript.

Figure 11(top) in the modified PDF gives a clearer line chart showing the sensitivity. I sincerely recommend that you take a look at it please.

Comments on the paper

2 comments

Dear Reviewers,

The rebuttal deadline is approaching. I am leaving this friendly reminder to encourage active discussion. Thank you for your valuable time and engagement!

AC

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewers:

We have submitted the revised manuscript as a PDF. Please download the updated PDF from OpenReview to see the revisions.

All changes are marked in blue or red.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.