acceptodds
Withdrawn from ICLR 2027

Unified Multimodal Models Do Not Learn in Unison: Active Probing of Cross-Capability Transfer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Shared parameters in unified multimodal models do not imply shared learning. In our controlled setting, optimizing one capability can reduce the other by up to 3.70 percentage points on average, and similar passive gradient statistics can precede opposite receiver outcomes. We introduce Active Mechanism-Separating Probe (AMSP). AMSP first selects a bounded native low-rank probe that maximizes robust separation among action-distinct finite-response hypotheses. It measures source and receiver responses on a disposable branch, then commits a source-feasible intervention from the untouched optimizer state; insufficient evidence leads to abstention. On Show-o2-7B, AMSP reaches a joint understanding–generation harmonic score of 82.36%, exceeding Pareto LoRA by 1.68 points and equal-budget lookahead by 1.85 points; both margins remain positive under paired seed-and-combination resampling. On InternVL-U-4B, it again achieves the highest mean score at 84.42%. Lookahead repairs more destructive events, but AMSP preserves 98.15% of constructive updates versus 79.63% for lookahead.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
unified multimodal models, cross-capability transfer, negative transfer, shared computation, multimodal understanding and generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_u8pPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines cross-capability interference between understanding and generation in unified multimodal models, focusing on the limitations of passive gradient statistics in predicting update outcomes. The proposed Active Mechanism-Separating Probe (AMSP) uses low-rank probes on a temporary optimizer branch to distinguish action-relevant hypotheses and select updates under a source-feasibility constraint. Experiments on Show-o2-7B and InternVL-U-4B using the Chop & Learn benchmark report higher mean joint understanding–generation scores than the compared baselines, alongside an analysis of destructive-update repair and constructive-transfer retention.

Critical strengths
  1. The paper identifies a potential limitation of passive gradient statistics, illustrating that updates with similar gradient characteristics may lead to different cross-capability outcomes. The distinction between transfer sign and intervention value provides a reasonable motivation for investigating alternative update decisions.

  2. AMSP presents a structured high-level procedure involving probe selection, finite-response measurement, hypothesis filtering, and intervention selection. Using a temporary optimizer branch allows probing without modifying the original training state, which is a reasonable design choice for stateful optimizers.

Critical weaknesses

1. Insufficient specification and justification of the core methodology.

The core decision mechanism relies on a hypothesis set V\mathcal{V}, predictive response distributions PhP_h, and robust separation bounds, yet their construction and calibration are insufficiently specified. The constrained/reduced update actions and fallback policy also lack implementation details. Given the limited set of predefined actions, the methodological distinction from existing active experimental design and robust action-selection approaches remains insufficiently established. While Appendix B provides a conditional first-order analysis, it does not establish the reliability of the proposed decisions under response-model misspecification, a concern further supported by the limited misspecification detection reported in Appendix F. These gaps hinder reproducibility and make the technical contribution difficult to assess.

2. Limited empirical support for the motivation and generality.

The reported 3.70-percentage-point degradation is introduced without sufficiently clarifying its controlled experimental setting. While Figures 1 and 3 illustrate limitations of passive statistics, the conditions and prevalence of this phenomenon beyond the controlled setting remain unclear. Both backbones are evaluated solely on the small-scale Chop & Learn benchmark, without larger-scale or more diverse understanding–generation evaluations. The joint-score improvements on InternVL-U are statistically inconclusive, and the wall-clock and memory overhead of active probing is not empirically evaluated. These limitations weaken the evidence for the method's generalizability and practical benefits.

Questions for authors
  1. How are the hypothesis set V\mathcal{V}, predictive response distributions PhP_h, and robust separation bounds constructed and calibrated in practice? Could the authors clarify the implementation of the constrained/reduced update actions and fallback policy?

  2. What distinguishes AMSP from existing active experimental design and robust action-selection methods, given its limited set of predefined actions? What supports the reliability of its decisions under response-model misspecification, particularly considering the limited detection rates in Appendix F?

  3. How were the R0/R2 examples in Figure 1 selected, and what evidence supports the observability gap beyond the controlled setting? Are there additional results on larger or more diverse understanding–generation benchmarks, as well as measured wall-clock and memory overhead relative to the baselines?

Reviewer_RdY5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method to adapt multimodal models, focusing on improving cross capability transfer, such as generation and understanding. Concretely, given a training update, they propose to test different modification to this update and verify its impact on specific capabilities. Hypotheses are designed to predict how different updates would impact performance of a capability, and probes are used to make small changes to the update and test multiple hypotheses at once. Experiments adapt the show-o2 and intern-VL models using LoRA layers, showing promising results.

Critical strengths
  • Understanding the impact of training on multiple capabilities, and ensuring positive transfer across them is very useful and relevant.
  • The paper comprises a large set of experiments, including different architectures and several baselines. Results show gains compared to other methods.
  • The idea of designing experiment such that multiple hypotheses can be tested at once is nice, and allows for reduced compute.
Critical weaknesses
  • The paper is very difficult to read and poorly presented. Authors use an overly complicated language, a lot of jargon, concepts and variables are not explained, or introduced properly. Many variable names are overloaded throughout the paper (e.g. L,r,D,q). Figures are difficult to understand and receive very little explanation and context in the text.
  • Crucially, key methodological choices are not described. Authors never explain concretely what hypotheses are, how they are constructed and what they represent. Similarly, the actual probes are never described. This not only prevents reproducibility, but makes it impossible to properly understand how the method is designed and its validity. How are hypotheses constructed? Are they handcrafted? How do they connect to probes?
  • Beyond potentially handcrafted hypotheses, the method risks being manually engineered for the specific dataset considered with three parameters having arbitrary values chosen (\eta, \kappa, \epsilon). How these parameters are chosen is not discussed, nor is their impact on method performance.
  • While the method generally outperform state of the art methods, the gains seem incremental compare to a simpler approach like lookahead. Are there computational gains to the proposed approach?
  • State of the art methods compared to in Table 1 are not discussed or cited at all in the main paper. It is important to discuss related work properly. These discussions should not be moved to the appendix as supplementary detail.
Questions for authors
  • How are hypotheses designed concretely? Are they handcrafted? How do they generalise across datasets and capabilities? How do they relate to probes? How are probes implemented concretely?
  • The paper focuses on generation and understanding. What about multimodal capabilities? E.g. text generation vs image generation.
  • How are the three parameters chosen? What is their influence on performance?
  • Can authors discuss how their work relates to the state of the art methods compared to in the experiment section?
Reviewer_rob7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies interference between understanding and generation during low-rank adaptation of unified multimodal models. Its main argument is that knowing whether an ordinary update helps or harms the receiving capability is not enough to decide whether another update would be better. The paper therefore formulates the problem as finite action identification. AMSP evaluates native-LoRA probe responses on a disposable copy of the optimizer state, uses source and receiver-proxy responses to narrow a set of action-distinct hypotheses, and then commits one source-feasible action from the untouched state or falls back when the evidence is insufficient.

On Show-o2-7B, AMSP reports a higher joint understanding–generation score than Pareto LoRA and equal-budget lookahead. On InternVL-U-4B, it again obtains the highest mean score, although the differences from the two strongest baselines are statistically unresolved.

Critical strengths

I like the paper’s decision-oriented framing. Separating transfer sign from intervention value is conceptually important, since identifying a harmful ordinary update does not by itself identify the best alternative. The clone–measure–discard–recompute protocol also correctly treats optimizer moments, randomness, buffers, and data position as part of the learning state rather than restoring model parameters alone.

The controlled Show-o2 results are also encouraging. AMSP improves generation and the joint score over Pareto LoRA and equal-budget lookahead, and the event analysis reveals a useful trade-off between repairing harmful updates and preserving beneficial transfer. The ablations are directionally consistent with the proposed design.

Critical weaknesses

My main concern is reproducibility. The paper does not instantiate the default hypotheses or response model, define how response uncertainty is estimated, or provide the actual probe library. The constrained and reduced actions, fallback behavior, and central decision thresholds are also missing. These are core parts of AMSP rather than secondary implementation details, so the method cannot be reconstructed from the submission.

The evaluation is also narrow. Both backbones use the same small object–state benchmark, generation is measured by a frozen classifier, and the InternVL-U joint-score differences are unresolved. The misspecification check misses many injected failures, which is important for a method whose decisions depend on the response model being trustworthy.

Questions for authors

What hypotheses and response models were used in the reported experiments, and how were they fitted and calibrated?

What are the exact probe set, action transitions, fallback behavior, and decision thresholds?

Do the results persist with more seeds and broader understanding and generation benchmarks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.