Joint, Decoupled, or Agent: When Each Multimodal Alignment Paradigm Wins

Jointly pretrained multimodal encoders are a common way to combine modalities, while alternatives integrate their information during downstream training or inference. When does each approach win? We compare three paradigms, instantiated with public checkpoints, across 16 paired datasets in five domains and multiple training-label budgets. Joint trains a fusion head on frozen, jointly pretrained encoders. Decoupled trains a fusion head on frozen, independently pretrained encoders. Agent gives a frozen LLM task instructions, labelled examples, available sample text, and supervised unimodal readouts. No paradigm dominates the complete $144$ dataset–budget combinations: Decoupled wins $54$, Joint $44$, and Agent $46$ by mean utility. ScienceQA, MELD, and MetMeme shift from Agent to fusion as supervision increases. Between the fusion paradigms, Joint leads on four of five QA datasets across all evaluated budgets, while higher unimodal $5$-NN accuracy is associated with a relative shift toward Decoupled across eight classification datasets (Spearman $\rho =-0.67$). We test whether representation and task diagnostics predict these preferences on held-out datasets before fitting their candidate systems. A two-step cascade combines these diagnostics with task and budget metadata to select Agent or fusion, then Joint or Decoupled. Under leave-one-group-out evaluation, it selects the winner in $71.4\%$ of cells versus $54.2\%$ for metadata-only routing, with mean relative regret $0.028$ versus $0.057$. Using geometry, supervised representation diagnostics, and text–label matching together yields higher accuracy and lower regret than using any one of these sources with metadata. These results connect paradigm preferences to supervision budget, task format, and unimodal $5$-NN accuracy. Target-task diagnostics thus support selecting among the three paradigms before candidate fitting.