acceptodds
Desk rejected at ICLR 2027

Mitigating Modality Interference for Unified Reasoning and Perception in Multimodal Large Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent text-only reasoning models show that System-2 reasoning can greatly improve math and code, but multimodal large language models (MLLMs) still rely on fast visual intuition with limited visuospatial gains. A dominant textual-proxy transfer recipe naively mixes supervision by converting images into caption-like intermediates and jointly training distilled long-form reasoning traces with multimodal perception supervision under a single objective. To address this gap, we conduct a systematic empirical analysis and attribute the trade-off to modality interference, where reasoning and perception objectives produce conflicting gradients under naive joint optimization. Building on this diagnosis, we propose UPRSD (Unified Perceptual and Reasoning via Staged Distillation), a training recipe that mitigates modality interference through decoupling, unification, and verifiable-reward reinforcement optimization, thereby improving reasoning without sacrificing perceptual grounding. Across multiple textual and multimodal benchmarks and different MLLMs scales, UPRSD strengthens multimodal and text reasoning while maintaining strong general multimodal understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Multimodal Reasoning, Visual Perception, Modality Interference, Staged Distillation, System-2 Reasoning, Chain-of-Thought, Reinforcement Learning with Verifiable Rewards

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mV9DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates modality interference between textual reasoning and visual perception objectives in multimodal large language models (MLLMs). Through preliminary experiments, the authors observe that naively combining textual reasoning distillation with multimodal supervision produces non-additive performance gains and negatively aligned gradients.

To address this issue, the paper proposes UPRSD (Unified Perceptual and Reasoning via Staged Distillation), a three-stage training framework:

  1. Structural Priming: Separates reasoning and perception behaviors through prefix-conditioned supervision and bootstraps multimodal reasoning trajectories.
  2. Logic Integration:Unifies reasoning supervision through a shared reasoning interface while using task-dependent routing to avoid unnecessary deliberation.
  3. Reinforced Evolution:Applies GRPO with verifiable rewards and potency-aware data selection to improve reasoning performance.

The framework is evaluated using Uni-VL models initialized from Qwen3-VL at 4B and 8B scales. The reported results demonstrate substantial improvements over several open-source baselines on multimodal and textual reasoning benchmarks while maintaining competitive general multimodal understanding.

The paper further provides gradient-alignment analysis, stage-wise ablations, and training dynamics to support its proposed approach.

Critical strengths

Strength 1: Strong and broad empirical improvements across reasoning modalities.

The paper demonstrates substantial performance improvements across a diverse collection of textual and multimodal reasoning benchmarks.

In particular, Uni-VL-8B achieves a multimodal reasoning average of 57.0, compared with 46.2 for the strongest open-source baseline in the same model-size group. Its textual reasoning average also reaches 60.9, compared with 31.0 for the strongest corresponding baseline.

The improvements at both 4B and 8B scales strengthen the empirical contribution. These results suggest that the proposed training framework can effectively improve multiple capabilities without an obvious general degradation in the reported multimodal benchmarks.

Strength 2: A practically relevant training framework supported by informative diagnostics.    

The paper addresses an important challenge in multimodal model training: simultaneously improving deliberative reasoning and visual understanding.

The combination of prefix-conditioned training, multimodal trajectory bootstrapping, unified reasoning supervision, and reinforcement learning provides a coherent practical training recipe.

The gradient-alignment analysis is also valuable. The reported change in mean gradient cosine similarity from approximately -0.12 under naive mixing to +0.18 after the proposed staged training is consistent with the authors' interference hypothesis.

Together with the stage-wise experiments, this provides a useful empirical starting point for understanding reasoning–perception interactions.

Critical weaknesses

Weakness 1: Insufficient causal isolation of the proposed modality-interference mitigation mechanism.

The paper attributes its improvements primarily to mitigating interference between reasoning and perception objectives. However, the experimental evidence does not sufficiently isolate this mechanism from other factors.

First, Stage 2 simultaneously introduces a unified reasoning interface and additional bootstrapped multimodal reasoning trajectories. Consequently, the improvements after Stage 2 cannot be attributed specifically to the proposed integration strategy.

Second, the gradient-alignment analysis compares different training checkpoints after different optimization histories. The observed shift from negative to positive gradient similarity is informative but does not establish that the reduced interference causes the downstream improvements.

Third, the paper does not clearly establish a training-token- and compute-matched comparison between the complete proposed pipeline and a conventional mixed-supervision pipeline using the same data and reinforcement-learning budget.

These limitations are consequential because the paper's central claim is not simply that staged training performs well, but that mitigating modality interference is the key mechanism responsible for its effectiveness.

Weakness 2: Limited independent validation of the routing and potency-aware reinforcement-learning components.    

Two important components of the proposed framework remain insufficiently evaluated.

The inference-time router is a text-only classifier trained using domain-based reasoning/perception labels. Although the reported classification accuracy is high, this evaluation may not adequately capture visually dependent or ambiguous queries for which the appropriate reasoning behavior depends on the image content. A controlled comparison against always-on reasoning and direct-response inference would help establish its actual end-to-end contribution.

Similarly, the effectiveness of potency-aware data selection is not isolated from standard GRPO using random or difficulty-matched training examples. The description of the held-out mining pool and its relationship to the eventual RL training prompts also requires clarification.

Without these analyses, it is difficult to determine whether these components contribute the claimed efficiency and robustness improvements or whether most gains arise from additional supervised data and reinforcement-learning optimization.

This weakens the evidence supporting the distinct contributions of the complete framework.

Questions for authors

Q1. [Clarification and additional evidence — training-budget fairness]

Could the authors clarify the number of training examples, total training tokens, and optimizer updates used by the naive-mixed baseline and each stage of UPRSD?

In particular, is there a baseline that performs conventional mixed-supervision SFT followed by GRPO using the same backbone, training data, total optimization budget, and reward configuration as the complete proposed pipeline?

If such a comparison is available, reporting it would materially strengthen the evidence for the benefits of staged training.

Q2. [Additional evidence — mechanism attribution]    

Could the authors disentangle the effects of the unified reasoning interface and the additional bootstrapped multimodal trajectories introduced in Stage 2?

For example, how does a model trained with the additional trajectories but without the proposed interface integration compare with the complete Stage-2 model?

It would also be useful to report gradient alignment for controls with more closely matched training histories. These results would materially affect my assessment of the central interference-mitigation claim.

Q3. [Clarification and additional evidence — routing and RL data selection]    

Could the authors clarify how potency scores computed on a held-out mining pool are transferred to the selection of RL training prompts, given that the mining pool is described as disjoint from those prompts?

Additionally, could the authors report the performance of standard GRPO without potency mining under the same training budget, and compare the learned router against always-on reasoning on visually ambiguous queries?

Reviewer_cDxgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates modality interference between reasoning and perception objectives in multimodal large language models (MLLMs). The authors propose UPRSD, a staged distillation and reinforcement learning recipe designed to improve reasoning while preserving perceptual grounding. The approach combines decoupled training, unified reasoning formats, and verifiable-reward optimization. Experiments across multiple model scales and textual and multimodal benchmarks show improvements in reasoning while maintaining general multimodal understanding.

Critical strengths
  • The paper addresses the trade-off between reasoning improvements and perceptual capabilities in MLLMs, an important challenge for developing models that can both reason effectively and remain visually grounded.

  • The method is evaluated across multiple model scales and textual and multimodal benchmarks. The staged training results provide evidence that the overall recipe can improve reasoning while maintaining strong multimodal performance.

Critical weaknesses
  • Previous studies have investigated perception degradation during reasoning training, conflicts in mixed-data training, and staged approaches to coordinating reasoning with visual grounding [1–4]. Although the paper provides a gradient-based analysis of modality interference, it remains unclear what new insights this diagnosis offers beyond these existing findings. The current analysis also does not sufficiently distinguish modality interference from conflicts caused by task types, output formats, or differences in supervision data.

  • Uni-VL is built on Qwen3-VL, whereas most open-source baselines in Table 1 use Qwen2.5-VL. Comparisons against these methods therefore cannot reliably isolate the contribution of UPRSD from differences in the backbone and training setup. The comparison with RAPID [5] is also not controlled, as it uses a different visual backbone and evaluation protocol. Consequently, the results demonstrate the effectiveness of the overall training recipe but do not establish its superiority over existing approaches under comparable conditions.

  • The training stages change multiple factors simultaneously, making it difficult to attribute improvements to specific components. For example, Stage 2 combines a unified reasoning interface with newly generated and filtered multimodal reasoning traces, so the effects of interface unification cannot be separated from those of improved supervision data. Similarly, the gains from Stage 3 do not independently establish the value of potency mining. Comparisons against simpler alternatives under matched data and training budgets are needed to validate these design choices.

[1] Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning.

[2] More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models.

[3] MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning.

[4] Learning When to Look: A Disentangled Curriculum for Strategic Perception in Multimodal Reasoning.

[5] Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning.

Questions for authors
  1. How does the proposed modality-interference analysis differ from prior studies on perception degradation, mixed-data training conflicts, and staged reasoning-perception training [1–4]? What evidence establishes that the observed gradient conflicts arise specifically from modality interference rather than differences in task types or supervision formats?

  2. Can UPRSD be compared with representative existing training approaches using the same backbone, comparable training data, and matched compute budgets? Does its advantage persist when evaluated on Qwen2.5-VL?

  3. Can the authors isolate the effects of reasoning-interface unification and training-stage order using matched data and training budgets? For Stage 3, how does potency mining compare with random sampling or conventional difficulty-based selection under the same GRPO setup?

Reviewer_njZyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes UPRSD, a three-stage training pipeline for improving multimodal reasoning while preserving perceptual capabilities. It combines prefix-conditioned training, verified self-generated reasoning trajectories, unified supervised fine-tuning, and GRPO with a sampling-based data-selection heuristic. The resulting models show improvements.

Critical strengths

The complete pipeline achieves improvements in both textual and multimodal reasoning across two model scales.

Critical weaknesses

Limited technical novelty. The approach is built largely from established techniques, including prefix conditioning, self-generated reasoning trajectories with verification and filtering, supervised fine-tuning, task routing, and GRPO. The distinguishing choices mainly concern prefix definitions, training-stage scheduling, output formatting, and a sampling-based data-selection heuristic. These differences appear to be incremental implementation details. The paper does not clearly establish a substantively new algorithmic mechanism, learning principle, or nontrivial combination of existing methods. Organizing familiar components into a multi-stage pipeline does not, by itself, provide a sufficiently compelling methodological contribution.

Insufficient ablations. Stage 2 changes the reasoning interface and adds training data simultaneously, preventing clear attribution of gains. Potency-based selection also lacks comparisons with simpler strategies. Matched-data and matched-budget experiments are needed to establish the independent benefits of these design choices.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.