DepthRoute: Adaptive Visual-Depth Selection for Multimodal Sentiment Analysis

Multimodal sentiment analysis has mainly focused on how visual and textual features are fused, while paying less attention to how representations from different depths of a pretrained visual encoder should be utilized before fusion. Representations from multiple visual depths may provide complementary information, yet existing approaches typically rely on the final visual layer or aggregate multiple layers using a fixed strategy. Such choices impose a largely input-independent depth-utilization pattern, although the visual cues relevant to sentiment can vary across image–text pairs and not all information encoded at different depths is equally useful for sentiment prediction. We therefore formulate visual-depth utilization as an input-dependent, sentiment-oriented representation-selection problem and introduce DepthRoute. DepthRoute explores complementary routing configurations that vary in scope and granularity: layer-wise routing over the full visual hierarchy, routing restricted to deeper layers, and structured routing over contiguous depth groups. All variants use a shared expert pool, with routing conditioned on the paired textual representation and optimized jointly with the sentiment objective. Experiments on MVSA-Single and TumEmo with frozen SigLIP2 and CLIP visual encoders demonstrate the effectiveness of adaptive visual-depth routing over fixed final-layer representations and fixed multi-layer aggregation. An expert-enhanced final-layer control shows that additional expert capacity explains only part of the improvement. Late-restricted routing remains highly competitive, while structured full-depth routing achieves the strongest overall mean performance in our controlled comparisons. Class-wise routing analysis further reveals sentiment-dependent preferences over depth-associated routes, showing that the learned routing behavior varies with sentiment categories.