acceptodds
Withdrawn from ICLR 2027

MARKER: Missing-Aware Low-Rank Expert Routing for Incomplete Multimodal Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Missing modalities commonly arise in real-world applications due to device failures, network instability, or privacy concerns. Recent studies address this issue with prompt-based fine-tuning. However, most of these prompt-based methods struggle with two aspects: (1) they rely on scenario-conditioned prompt sets and require explicit prompt selection for different modality-availability conditions, leading to a less unified adaptation pipeline; (2) most methods are modality-missing-aware but instance-agnostic, while a few recent approaches achieve instance-level adaptivity through retrieval-based mechanisms at the cost of additional components and increased system complexity. To address these issues, we propose MARKER, a Missing-Aware Low-Rank Expert Routing framework for incomplete multimodal learning. Specifically, we insert a LoRA-based mixture-of-experts module into a pretrained multimodal backbone. Trained on data with missing modalities, MAIL dynamically routes each input to a sparse combination of LoRA experts, conditioned on both the missingness pattern and instance-specific features, enabling adaptive prediction without maintaining per-scenario prompt banks. Importantly, this design decouples adaptation from per-pattern prompt specification, providing a unified mechanism across varying missingness conditions. We further conduct ablation studies on routing strategies and layer-wise expert-allocation shapes to assess their impact under different modality-missing scenarios. Extensive experiments on three real-world datasets show that MAIL consistently outperforms 11 strong baselines, demonstrating its effectiveness and robustness for incomplete multimodal learning.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multimodal Learning, Modality Incomplete

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4Rg5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes MARKER, a mixture of LoRA experts for incomplete image–text learning. Results on three datasets are competitive, and the routing analyses provide useful evidence that expert selection varies with modality availability and individual inputs. However, several major concerns remain.

Critical strengths
  • Addresses a practical problem through parameter-efficient adaptation.

  • Routing diagnostics provide evidence of sample-dependent expert selection.

  • Reports computational cost and measured inference speed transparently.

Critical weaknesses
  • The architectural contribution is incremental, combining established LoRA experts, softmax routing, and top-k selection without a clearly new mechanism.

  • Related-work positioning is inadequate: previously requested Flex-MoE and DCP are absent, as are closely related Omni-SMoLA, LLaVA-MoLE, and MLE-SAM.

  • Comparisons with ordinary LoRA, per-missingness adapters, and existing LoRA-MoE methods under matched parameter budgets are missing.

  • Evaluation is restricted to ViLT and two modalities, leaving broader scalability and generalisation unverified.

  • Efficiency benefits are mixed, as lower FLOPs accompany substantially higher inference latency than prompt-based baselines.

  • Presentation requires substantial revision: Figure 1 contains accidental editing-interface artifacts, its caption mischaracterises prompt-based methods, and grammatical and notation issues persist.

Questions for authors

See weakness.

Reviewer_MZZyOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses incomplete multimodal learning, in which pretrained VLMs must stay accurate when modalities are missing. The authors claim that existing prompt-based methods rely on scenario-specific prompt sets and apply static, instance-agnostic corrections. Instance-aware alternatives introduce auxiliary query or retrieval modules. To address these limitations, the paper proposes a new method which inserts sparse mixtures of LoRA experts into a frozen ViLT backbone. A lightweight router selects the top two experts per module, conditioned on both the missingness pattern and instance features. This yields a unified and parameter-level adaptation mechanism without per-scenario prompt banks. The method includes linear-wise, attention-wise, and combined variants.

Critical strengths
  1. This paper is addressing an important problem. The incomplete multimodal inputs are what a deployed system actually see, and the combinatorics make it a exponential problem, eg M modalities produces 2^M - 1 patterns.

  2. The paper has contributed good conceptual insights, it claims that one should not inject prompts/imputing inputs if they are missing, instead one should make the weight update itself condition on the input, which could be an interesting direction to think of.

  3. The findings in A.1.4 is interesting - a router can have nearly uniform distribution while consistently changing the top-k expert it picks. This gives people insights that entropy alone is not sufficient so need more fine-grained analysis metrics.

Critical weaknesses
  1. I think one obvious drawback is the paper only considered random missingness patterns, and they are split uniformly. In reality many missingness and modality missingness are dependent on the data or label, or a particular situation. The authors haven't studies this basic scenarios, which makes the approach impractical.

  2. In addition, it looks like the method has to see every missing pattern during training so that the router can learn it. The assumption is that training and testing pattern always match, which is unrealistic. In addition, only test-time missing rate is varied.

  3. The paper's base models and tasks are outdated: it only studies 2-modality case on simple classification tasks, and the base model it used is ViLT-B/32 which dates back to ~5 years ago. The authors could consider using more cutting-edge base models.

  4. Also, I suspect there's no missingness signal reach the router. The router only sees mean-pooled token representations without anything else. So the "missing-aware" claim is questionable.

Questions for authors
  1. It looks like fig 1 is not ready for presentation - "Our MARKER" module is selected with an editing mode. Other parts like "trained prompt sets" look confusing, how can a prompt set be trained? Please take back and revise the figure better.

  2. Missing relevant citations that worked on the same problem and are well-known baselines: (1) Fusemoe: Mixture-of-experts transformers for fleximodal fusion (2) Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts

  3. How does this method perform under complete inputs?

Reviewer_aNyQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MARKER, a framework for incomplete multimodal learning that introduces LoRA experts into a frozen multimodal Transformer. A sample-level router selects a sparse combination of experts based on input representations, aiming to adapt to both modality availability and instance characteristics. The authors investigate expert placement in feed-forward layers, attention projections, and their combination, and introduce a retrieval-augmented variant. Experiments on MM-IMDb, Hateful Memes, and UPMC Food-101 evaluate predictive performance, routing behavior, and computational efficiency.

Critical strengths
  1. The proposed low-rank expert adaptation provides a reasonable and parameter-efficient way to leverage a frozen multimodal backbone under missing-modality settings. The reported results show consistent improvements over strong non-retrieval baselines across different missing conditions, indicating the potential effectiveness of the approach.

  2. The evaluation preserves sample identity and labels while varying modality availability, enabling a controlled analysis of how missing modalities affect routing behavior. The additional cross-dataset analyses and the explicit discussion of the gap between FLOPs reduction and actual inference latency further strengthen the empirical transparency of the paper.

Critical weaknesses
  1. The claimed missing-aware mechanism is insufficiently justified.
As specified, the router operates on pooled hidden representations, with missingness reflected through dummy-filled inputs. This allows routing to respond to modality availability, but the paper does not establish what missingness-specific failure this mechanism resolves beyond ordinary input-conditioned MoE routing. Changes in expert selection do not, by themselves, demonstrate reduced cross-condition interference or more effective use of the remaining modality.


  2. The motivation also treats static prompt parameters as essentially static corrections. However, fixed prompt parameters can produce input-dependent effects through attention. The paper therefore needs a more precise account of why explicit expert routing provides a capability that existing prompt-based adaptation lacks. Without this distinction, the central contribution remains an application of generic conditional adaptation rather than a clearly established missing-modality mechanism.

  3. The theoretical argument does not characterize the implemented model.
Equation (11) equates the loss of the mixed model with a weighted average of individual expert losses. MARKER instead combines low-rank updates inside a nonlinear Transformer, for which this equality generally does not hold. The resulting routing-regret decomposition therefore cannot directly justify the claimed robustness advantage.
More broadly, a richer family of conditional updates does not by itself guarantee useful specialization or improved robustness. The paper does not establish that the learned router selects experts with lower conditional loss, or that its observed routing patterns correspond to the corrective behavior assumed by the analysis. This leaves a substantive gap between the proposed architecture, its theoretical explanation, and the claimed mechanism.

Questions for authors
  1. What specifically makes the proposed mechanism missing-aware beyond ordinary input-conditioned routing? Please identify the missingness-induced failure that the router is designed to correct and explain how the formulation addresses it.

  2. In particular, how does the argument account for the input-dependent effects of fixed prompts through attention, and what connects the observed routing changes to useful corrective behavior?

  3. How does Equation (11) apply to the actual MARKER architecture? The model mixes low-rank updates before subsequent nonlinear computation, rather than averaging independent expert losses. Under what assumptions is the stated equality valid, and what robustness conclusion remains justified for the implemented model if those assumptions do not hold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.