Preference-Routed Interest Synthesis: Modality Responsibility as a Latent Variable in Multimodal Recommendation

Multimodal recommenders enrich item representations with visual and textual content, and they consistently outperform recommenders that see interactions alone. Almost all of them, however, decide once—globally, for the whole population—how much each modality should count. We argue that this is the wrong place to make the decision. Whether a purchase is driven by how an item looks, by what its description promises, or by who else bought it is a property of the user, and it varies widely across a population; a single fusion rule can at best fit the average of that variation. We recast the fusion weight as a latent per-user variable, which we call modality responsibility, and we build a recommender around inferring it. Our model, , amortises the posterior over responsibilities with a sparse simplex-valued head, so each user is explained by a subset of the channels rather than a soft blend of all of them; modulates the routed weights by an item-side reliability prior, so items whose content is uninformative are not trusted; keeps propagation user-independent, so personalised routing costs a constant number of inner products at scoring time; and corrects pathologies routing alone does not fix—a channel that monopolises the gradient signal, and the geometric gap between channel subspaces—with a margin-equalising update and an entropic optimal-transport alignment. We prove that the excess risk of any statically fused scorer is bounded below by a quantity proportional to the dispersion of the true responsibility distribution, which makes explicit what static fusion gives up, and that the sparse routing head recovers the correct support under a margin condition. Across public benchmarks, improves on strong graph-based and self-supervised multimodal baselines, converges in fewer epochs than the methods it outperforms, and gains most where the routing story predicts: on sparse users, cold items, and the most visually driven catalogue.