Learning Evidence-Adaptive Composition for Compositional Generalization

Modern vision systems increasingly combine pretrained models that provide complementary yet uncertain evidence about the same instance. However, existing approaches combine such evidence using fixed-parameter fusion functions or predefined composition rules, even when the appropriate interaction depends on the surrounding evidence. This limitation becomes more pronounced in multi-source settings, where an intermediate composition can change how subsequent evidence should be interpreted. We therefore ask whether a model should learn not only what evidence to combine, but also how that evidence should be composed. We introduce AdaCompose, a general framework for evidence-adaptive composition in which the complete evidence states being composed determine the parameters of the composition operator itself. The generated operator is applied to local evidence representations to produce pairwise relations and a new evidence state, which can then determine the operator for subsequent compositions. AdaCompose incorporates within-source permutation consistency, prediction-score gating, controlled low-rank operator adaptation, bounded relations, and a common state interface that enables recursive composition. We evaluate AdaCompose on video anomaly detection, compositional zero-shot recognition, and visual question answering, covering context-dependent recognition, unseen semantic combinations, and multimodal reasoning. Across these tasks, these evaluations assess generalisation to unseen attribute-object combinations in CZSL and the integration of heterogeneous evidence in VAD and multimodal question answering. These results demonstrate the potential of making the composition operator itself dependent on the evidence being composed, providing a general mechanism for composing heterogeneous evidence beyond fixed composition rules.