acceptodds
Withdrawn from ICLR 2027

Joint, Decoupled, or Agent: When Each Multimodal Alignment Paradigm Wins

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Jointly pretrained multimodal encoders are a common way to combine modalities, while alternatives integrate their information during downstream training or inference. When does each approach win? We compare three paradigms, instantiated with public checkpoints, across 16 paired datasets in five domains and multiple training-label budgets. Joint trains a fusion head on frozen, jointly pretrained encoders. Decoupled trains a fusion head on frozen, independently pretrained encoders. Agent gives a frozen LLM task instructions, labelled examples, available sample text, and supervised unimodal readouts. No paradigm dominates the complete dataset–budget combinations: Decoupled wins , Joint , and Agent by mean utility. ScienceQA, MELD, and MetMeme shift from Agent to fusion as supervision increases. Between the fusion paradigms, Joint leads on four of five QA datasets across all evaluated budgets, while higher unimodal -NN accuracy is associated with a relative shift toward Decoupled across eight classification datasets (Spearman ). We test whether representation and task diagnostics predict these preferences on held-out datasets before fitting their candidate systems. A two-step cascade combines these diagnostics with task and budget metadata to select Agent or fusion, then Joint or Decoupled. Under leave-one-group-out evaluation, it selects the winner in of cells versus for metadata-only routing, with mean relative regret versus . Using geometry, supervised representation diagnostics, and text–label matching together yields higher accuracy and lower regret than using any one of these sources with metadata. These results connect paradigm preferences to supervision budget, task format, and unimodal -NN accuracy. Target-task diagnostics thus support selecting among the three paradigms before candidate fitting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-modal, Alignment, Pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Bo8cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper compares joint pretraining, supervised fusion of independently pretrained encoders, and LLM-based inference-time composition across 16 datasets and multiple label budgets. It examines how the preferred approach changes with label availability and introduces a diagnostic-based router to predict which approach will work best on a new task. The evaluation uses shared training and selection protocols, component controls, and grouped validation. The results show budget-dependent changes in the preferred system and suggest that task and representation diagnostics can improve selection within the evaluated configurations.

Critical strengths
  1. The paper addresses two practically useful questions: how the preferred approach depends on the amount of supervision, and whether that preference can be predicted before fitting all candidate systems. I find this combination particularly interesting because it connects empirical comparisons to decisions practitioners actually need to make.
  2. The experimental design is careful and systematic. Shared evaluation protocols and component controls help distinguish overall system performance from the benefit of combining modalities, while grouped validation provides meaningful evidence of selection performance on unseen datasets. These choices strengthen the credibility of the empirical findings.
Critical weaknesses
  1. It is unclear how much the observed preferences reflect alignment paradigms rather than the particular checkpoints and their pretraining histories. The paper does not sufficiently establish that its main findings persist across competitive implementations of each approach. The absence of multimodal LLMs that directly process images also leaves an important practical alternative untested on vision–language tasks. Together, these limitations make broader recommendations about which paradigm to choose less convincing.
  2. The evidence for the router’s generalizability remains limited. Although the benchmark covers several domains, the routing evaluation contains only 15 dataset groups and largely fixed candidate configurations. Held-out-dataset performance is useful evidence, but it does not establish that the selection rules transfer to new domains or different candidate models. Since providing guidance for new tasks is a central contribution, this uncertainty materially affects my assessment.
Questions for authors

Clarification requests:

  1. What considerations drove the choice of default checkpoints, and how consistently do the existing encoder and controller ablations preserve the main budget-dependent preferences? A clear account of which conclusions remain stable across these changes would help assess their generality.
  2. What is the rationale for excluding multimodal LLMs that directly receive images from the vision–language comparisons? How should this exclusion limit the practical interpretation of the Agent results?
  3. What specific predictive capability or scientific insight does the router add beyond prior diagnostic-based model-selection work? Clarifying this distinction would help assess the contribution beyond the benchmark itself.

Suggestions for additional validation: Evaluation on held-out domains and alternative candidate models would strengthen the generalization evidence. Testing under a fixed total annotation budget, including validation labels, would also clarify the practical meaning of the budget-dependent findings. These are requests for additional evidence, rather than ambiguities in the current protocol that can be resolved through explanation alone.

Reviewer_7a7YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares three ways to combine multiple modalities for a downstream task. Joint trains a fusion head on frozen encoders that were pretrained jointly. Decoupled trains the same kind of head on frozen unimodal encoders that were never jointly aligned. Agent gives a frozen text LLM (Qwen3-32B by default) the task, a few in-context examples, the sample text (when available) and readouts. The study compares these methods on 16 datasets, five domains, several label budgets. This leads to 144 dataset-budget cells. The headline analysis leads to a conclusion that no paradigm wins everywhere. Joint trends to win on tasks spanning QA. Decoupled does better when unimodal 5-NN accuracy is high. The authors also train a two-step "router" on cheap dataset features to predict the winner on new datasets. It gets 71.4% accuracy agains 54.2% for a router that uses only task and budget information.

Critical strengths
  1. The analysis is fairly comprehensive and something that practitioners could find useful. The coverage is pretty broad. Also, each fusion paradigm gets a search over eight head types, making the fusion baselines pretty strong.
  2. The control experiments are useful and honestly reported.
  3. The router is evaluated carefully (leave-one-group-out, related datasets held out together) and the authors plan to release predictions and features, which I can also see being of value.
Critical weaknesses
  1. The big weakness is that there is no new method. The main finding is not surprising. The takeaways for practitioners, despite the exhaustive study, is that "no paradigm wins everywhere, and it depends on the task and the amount of labeled data". There is no explanation or analysis into why any of these trends are the way they are. So as is, it adds little value for someone starting a new experiment in this field, even with these results published.
  2. The "Agent" is not really an agent. It is one prompt with the probe outputs filled in. The version that really calls tools is usually worse. On several tasks, Agent is at or below chance (Fracture, Memotion, DrugBank). It wins no regression task. Where it wins is mostly where text does most of the work. The LLM also never sees the image. And the paper has no multimodal LLM baseline, which is also what most practitioners would try first. 3, The router, probably the only novel contribution, is weak. It rests on 15 dataset groups and the gain over the router with only task and budget features has a 95% interval of [0.001, 0.346]. Against a simple task-and-budget-lookup, the paper's own appendix shows the interval crossing zero at 4/5 tolerance levels.

Overall, it reads more as a workshop or datasets-and-benchmarks paper than a main-track at ICLR. The experimental analysis would help people who already work on this, but a broad audience might not learn anything new from the paper as is.

Questions for authors
  1. Can you add a multimodal LLM that sees the image? Why call the current system an agent? Is it also using some harness?
  2. How does picking the paradigm by dev-set score compare with the router, on accuracy, regret, and time?
Reviewer_mTSjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares three types of multimodal systems on downstream multimodal tasks. 1. CLIP model which does the alignment in pre-training. 2. DINOv2 and GPT2, which train encoders independently on different modalities with decoupled representation 3. Agentic method that uses one LLM to summarize the readout and raw text from each modality.

The paper conducts experiments on 16 datasets with different training budgets and finds that candidate-matching QA usually prefers  JOINT. While in classification, the higher 5-NN accuracy is, the better model prefers DECOUPLED.  The paper also uses the results from three types of model training as labels, and uses the task information as input to train a classification model. The model could predict which system to choose for different tasks and is verified with leave-one-dataset-out experiments.

Critical strengths
  1. As an empirical study, the paper conducts experiments on different datasets and compares three types of models with different training budgets. Also the paper summarizes some empirical rules from a lot of ablations experiments. E.g., candidate-matching QA usually prefers JOINT.
  2. The proposed classifier model is tested in a cross-dataset setting. It can be used in practice to select models and could have the potential to reduce the training compute for a specific task.
Critical weaknesses
  1. The comparison is system-level; without apple-to-apple control over pretraining data and compute, it is challenging to claim which paradigm wins. Also, each paradigm has different variants of models; the result is also not consistent across them. The pre-training compute, data, and objective are not controlled, so the performance difference could not come only from the alignment method. The conclusion on three types of models may not be generalizable to three paradigms.

  2. The results is mainly relevance on empirical level and lacks explanation. For example, why is JOINT better for candidate-matching QA. Does it come from some properties of the task or from some capabilities of the encoder? Also, it is not clear whether the classifier prediction on model selection aligns with the observations and can be generalized across different models.

Questions for authors

Please refer to the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.