acceptodds
Under review as a conference paper at ICLR 2027

DepthRoute: Adaptive Visual-Depth Selection for Multimodal Sentiment Analysis

Abstract

Multimodal sentiment analysis has mainly focused on how visual and textual features are fused, while paying less attention to how representations from different depths of a pretrained visual encoder should be utilized before fusion. Representations from multiple visual depths may provide complementary information, yet existing approaches typically rely on the final visual layer or aggregate multiple layers using a fixed strategy. Such choices impose a largely input-independent depth-utilization pattern, although the visual cues relevant to sentiment can vary across image–text pairs and not all information encoded at different depths is equally useful for sentiment prediction. We therefore formulate visual-depth utilization as an input-dependent, sentiment-oriented representation-selection problem and introduce DepthRoute. DepthRoute explores complementary routing configurations that vary in scope and granularity: layer-wise routing over the full visual hierarchy, routing restricted to deeper layers, and structured routing over contiguous depth groups. All variants use a shared expert pool, with routing conditioned on the paired textual representation and optimized jointly with the sentiment objective. Experiments on MVSA-Single and TumEmo with frozen SigLIP2 and CLIP visual encoders demonstrate the effectiveness of adaptive visual-depth routing over fixed final-layer representations and fixed multi-layer aggregation. An expert-enhanced final-layer control shows that additional expert capacity explains only part of the improvement. Late-restricted routing remains highly competitive, while structured full-depth routing achieves the strongest overall mean performance in our controlled comparisons. Class-wise routing analysis further reveals sentiment-dependent preferences over depth-associated routes, showing that the learned routing behavior varies with sentiment categories.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal representation learning, visual-depth routing, hierarchical representations, mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_P9giOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DepthRoute, a text-conditioned framework for adaptive visual-depth selection in multimodal sentiment analysis. It studies three routing strategies over individual layers, late layers, and contiguous layer groups, and evaluates them on MVSA-Single and TumEmo with frozen SigLIP2 and CLIP backbones. The experiments on two datasets validate the effectiveness of the proposed method.

Critical strengths
  1. The motivation is clear and well grounded: rather than focusing on cross-modal fusion, the paper studies how visual representations from different encoder depths should be adaptively utilized. The unified comparison of full, late, and group-wise routing makes this question easy to follow.
Critical weaknesses
  1. The experimental scope is somewhat limited, as the evaluation is conducted only on MVSA-Single and TumEmo. I suggest adding experiments on MVSA-Multiple to provide stronger evidence of the method's generalizability. This is particularly important given that the reported performance on TumEmo is lower than that of CoDe [1] and MVSA-Single is lower than that of SPP=SCL [2].

  2. The citation of references is problematic, for example:

    "including multiview calibration Wei et al. (2023) " should be "including multiview calibration (Wei et al. ,2023) "

Questions for authors
  1. The current experiments do not clearly isolate the effect of adaptive routing from the group-wise representation design. Adding a fixed Group baseline with the same group construction and projections but without input-dependent routing will be helpful.
  2. The current routing analysis does not clearly establish whether the learned depth selection is genuinely image-dependent, since the router is also conditioned on text. Controlled analyses with fixed text and varied images, or text permutation, would provide stronger evidence for adaptive visual-depth selection.
  3. The experimental scope is somewhat limited, as the evaluation is conducted only on MVSA-Single and TumEmo. I suggest adding experiments on MVSA-Multiple to provide stronger evidence of the method's generalizability. This is particularly important given that the reported performance on TumEmo is lower than that of CoDe [1] and MVSA-Single is lower than that of SPP=SCL [2].

[1] WU D, YANG D, SHEN H, et al. Resolving sentiment discrepancy for multimodal sentiment detection via semantics completion and decomposition[J]. Pattern Recognition, 2026, 172: 112719.URL https://doi.org/10.1016/j.patcog.2025.112719.

[2] Wu J, Li S. SPP-SCL: Semi-Push-Pull Supervised Contrastive Learning for Image-Text Sentiment Analysis and Beyond[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2026, 40(3): 2173-2181.URL https://doi.org/10.1609/aaai.v40i3.37200.

3 comments

Thank you for recognizing the motivation for adaptive visual-depth selection and for suggesting controls to isolate the role of input-dependent routing.

1. Fixed Group baselines. We compared DepthRoute-Group with two input-independent controls that use the same depth groups, group projections, shared expert pool, and classifier. Uniform Group assigns equal weights to all group-expert pairs. Learned Static Group learns a single set of routing weights shared across examples. The table reports Weighted-F1 with SigLIP2 as mean ± standard deviation over three seeds.

MethodMVSA-SingleTumEmo
Uniform Group0.7516 ± 0.00550.6984 ± 0.0016
Learned Static Group0.7534 ± 0.00840.6968 ± 0.0017
DepthRoute-Group0.7794 ± 0.00730.7193 ± 0.0035

DepthRoute-Group improves mean Weighted-F1 over Learned Static Group by 2.60 percentage points on MVSA-Single and 2.25 points on TumEmo. The gains over Uniform Group are 2.78 and 2.09 points, respectively.

2. Sensitivity to image and text inputs. We varied one modality at a time at inference and measured how the routing weights changed across visual depth groups. We summed the expert weights within each group and used Jensen–Shannon divergence (JSD) to compare the resulting depth-group distributions. For the three-group checkpoint trained with seed 42, the mean JSD was 0.00150 when we held the text fixed and changed the image, 0.06353 when we held the image fixed and varied the text, and 0.05546 under text permutation. Under this perturbation protocol, depth allocation showed modest sensitivity to image changes and substantially greater sensitivity to text changes.

3. MVSA-Multiple and comparison with recent methods. We agree that evaluation on MVSA-Multiple would strengthen the evidence for generalization. Our current findings are limited to MVSA-Single and TumEmo.

We acknowledge that SPP-SCL and CoDe report higher scores on MVSA-Single and TumEmo, respectively. DepthRoute examines visual encoder depth as a design choice for image–text sentiment classification. The fixed-routing comparisons above show that adapting depth weights to each image–text pair improves mean Weighted-F1 on both datasets with the frozen visual backbone and grouped architecture held fixed. We will clarify this contribution in the revision and discuss the published scores of SPP-SCL and CoDe as reference results.

4. Citation formatting. We will revise the example to “including multiview calibration (Wei et al., 2023)” and use parenthetical citations in similar contexts throughout the manuscript.

AuthorsOpenReview ↗

Q3: Siglip results for MVSA-Multiple datasets

Thank you for suggesting MVSA-Multiple. We added this dataset to evaluate adaptive visual layer selection on a third image–text sentiment benchmark. All methods use the same SigLIP visual backbone. We report Accuracy and Weighted-F1 as mean ± standard deviation over three seeds.

MethodAccuracyWeighted-F1
Final-Layer0.6943 ± 0.00110.6829 ± 0.0022
All-Layer Concat0.6972 ± 0.00410.6886 ± 0.0039
DepthRoute-Layer0.7122 ± 0.00270.6903 ± 0.0075
DepthRoute-Group0.7031 ± 0.00190.6916 ± 0.0010
DepthRoute-Late0.7075 ± 0.00400.6941 ± 0.0036

All three DepthRoute variants achieve higher mean Accuracy and Weighted-F1 than both fixed baselines. Relative to All-Layer Concat, DepthRoute-Layer improves mean Accuracy by 1.50 percentage points, and DepthRoute-Late improves mean Weighted-F1 by 0.55 percentage points. DepthRoute-Group achieves the highest mean Weighted-F1 on MVSA-Single and TumEmo, while DepthRoute-Late achieves the highest mean on MVSA-Multiple. These results suggest that the most effective visual-depth selection strategy can vary across datasets.

I appreciate the authors' efforts in addressing my concerns. However, the results on the additional datasets presented in the rebuttal are still quite poor. Therefore, I remain highly skeptical about the effectiveness of the proposed method and do not believe the paper meets the acceptance bar for ICLR.

Reviewer_6mtwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper observes that intermediate visual encoder layers contain sentiment-relevant information, and that different layers appear useful for different inputs. It formulates this as an input-dependent representation-selection problem: rather than always using the final visual layer or a fixed multi-layer aggregation, DepthRoute constructs a set of "visual-depth candidates" (individual layers, late layers, or contiguous depth groups) and routes them through a shared expert pool conditioned on the paired text representation. Routing logits are adjusted by a learned per-route log-scale, and the routed visual representation is concatenated with the text representation for classification. Experiments on MVSA-Single and TumEmo with frozen SigLIP2 and CLIP backbones show that group-wise routing outperforms final-layer and fixed all-layer fusion.

Critical strengths

Whereas the predominant focus of MSA research has been on how textual and visual information should be combined, this paper identifies visual-depth utilization as an orthogonal modeling axis that has received comparatively little explicit treatment. The observation that access to multiple visual layers does not by itself guarantee their effective use is well taken, and the distinction drawn between having multi-layer representations and selecting among them in a task- and input-appropriate manner is conceptually sound. Framing this as a representation-selection problem rather than a fusion problem is a reasonable and useful reframing.

Critical weaknesses
  1. The motivation "sentiment-relevant information is distributed across visual depth, and different inputs need different depths" is interesting, but the method is a fairly direct application of top-k/mixture-of-experts routing over a new candidate set (visual layers). Sparse MoE with a text-conditioned router, per-route scale prediction, and log-scale regularization are all standard components. The paper does not introduce a new routing principle, a new theoretical argument for why visual depth should be routed rather than fused, or a new mechanism specific to the sentiment task. Thus, the overall novelty is relatively marginal.
  2. The empirical evaluation is confined to two image–text datasets, both in English. Video-based benchmarks such as CMU-MOSI, CMU-MOSEI, and CH-SIMS, which constitute the dominant evaluation setting in multimodal sentiment analysis, are not included.
  3. The comparison against recent methods (D2R, CF-MSA, DIB, and others) is drawn from results reported in the respective original papers, whereas the controlled baselines and the proposed variants are evaluated under a different protocol using frozen SigLIP2 and CLIP visual encoders. This constitutes a non-equivalent comparison: differences in backbone, training regime, preprocessing, and evaluation splits are not controlled, and the reported gaps therefore cannot be attributed to the proposed method with confidence.
Questions for authors
  1. How much per-sample variation is there in the routing distribution? What is the entropy of the routing distribution across samples within a class, and does it differ from across classes? This directly tests the "input-dependent" claim.
  2. What is the comparison against a LoRA/adapter on the final visual layer?
1 comment

Thank you for recognizing visual-depth utilization as a distinct question in multimodal sentiment analysis and for suggesting additional analyses and baselines.

1. Novelty and contribution. DepthRoute builds on established mixture-of-experts components to investigate whether the useful visual depths of a frozen encoder vary across image–text pairs. To assess the contribution of input-dependent routing, we compared DepthRoute-Group with Uniform Group, which retains the same depth groups, group projections, shared expert pool, and classifier. With SigLIP2, DepthRoute-Group improves mean Weighted-F1 by 2.78 percentage points on MVSA-Single (0.7794 vs. 0.7516) and 2.09 points on TumEmo (0.7193 vs. 0.6984). It also exceeds Learned Static Stage Weighting by 1.64 points on MVSA-Single (0.7794 vs. 0.7630).

MethodMVSA-Single Acc.MVSA-Single W-F1TumEmo Acc.TumEmo W-F1
Uniform Group0.7600 ± 0.00800.7516 ± 0.00670.6989 ± 0.00150.6984 ± 0.0020
Learned Static Group0.7615 ± 0.00720.7534 ± 0.01020.6973 ± 0.00230.6968 ± 0.0021
DepthRoute-Group0.7830 ± 0.00710.7794 ± 0.00730.7190 ± 0.00380.7193 ± 0.0035

2. Evaluation scope. Our evaluation focuses on static image-text sentiment classification. CMU-MOSI, CMU-MOSEI, and CH-SIMS are video-based multimodal sentiment benchmarks that provide visual, audio, and text inputs. Extending DepthRoute to these settings would require defining how to aggregate visual-depth features across video frames. We will clarify this scope in the revised manuscript and discuss extending DepthRoute to video-based sentiment analysis as a direction for future work.

3. Comparability with published methods. We agree that differences in backbones, training regimes, preprocessing, and data splits limit the interpretation of cross-paper comparisons. We will label the published scores as reference results and base our claims about visual-depth routing on the controlled comparisons within each backbone and evaluation protocol.

4. Sample-level routing variation. We analyzed the three-group DepthRoute-Group checkpoint trained with seed 42. For each test example, we summed the expert weights within each depth group to obtain a distribution over the Early, Middle, and Late groups. We then measured how much these distributions differed between examples with the same ground-truth sentiment label. The mean pairwise Jensen–Shannon divergences were 0.0294 for Negative, 0.0393 for Neutral, and 0.0356 for Positive. Among Positive examples, 52.2% assigned their highest routing weight to the Middle group and 47.8% to the Late group. Thus, the highest-weighted visual depth varies across examples even within a single sentiment class.

5. Final-layer adapter. We added a trainable adapter that uses only the final-layer visual representation and kept the visual backbone frozen. The table reports Weighted-F1 as mean ± standard deviation over three seeds.

DatasetBackboneFinal AdapterDepthRoute-GroupDifference (percentage points)
MVSA-SingleSigLIP20.7673 ± 0.00620.7794 ± 0.0073+1.21
MVSA-SingleCLIP ViT-B/160.7404 ± 0.01280.7654 ± 0.0108+2.50
TumEmoSigLIP20.7092 ± 0.00110.7193 ± 0.0035+1.01

With SigLIP2, the adapter improves over the original Final baseline on both datasets. DepthRoute-Group achieves higher mean Weighted-F1 than Final Adapter in all three settings shown.

Reviewer_qm9SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DepthRoute studies text-conditioned selection of representations from different depths of a frozen visual encoder for multimodal sentiment analysis. It compares layer-wise, late-restricted, and grouped routing through shared experts on two datasets with SigLIP2 and CLIP backbones.

Critical strengths

Visual-depth utilization is a clearly motivated modeling question. The grouped variant consistently improves over the fixed baselines in Table 1.

Critical weaknesses
  1. Section 3.3, Equation 5, states that Late retains layer-wise routing over deeper layers. However, models/depthroute_late.py constructs grouped features and enables only the final group; the supplied configurations combine layers 8–11 into that single group. This implements expert routing over a pooled late-group representation, rather than selection among individual late layers. It is therefore unclear which definition produced Table 1 and how the scope/granularity comparisons should be interpreted.
  2. Final+Expert controls expert capacity at the final layer, but not the grouped projections introduced in Equation 6. A control retaining the same group projections, experts, and classifier while replacing input-dependent routing with fixed or globally learned weights is needed to distinguish dynamic depth selection from the broader architectural changes.
Questions for authors
  1. Which Late implementation generated Table 1? Please provide the corresponding configuration and reconcile it with Equation 5.
  2. With group construction, expert capacity, and optimization held fixed, does input-dependent routing outperform uniform or globally learned routing weights? Please also specify the exact All-Layer baseline implementation.
1 comment

Thank you for identifying the discrepancy in the Late implementation and for suggesting fixed-routing controls within the grouped architecture.

1. Which Late implementation generated Table 1?

The original DepthRoute-Late results used the submitted Late-Group implementation. In both submitted configurations, late_stage_index: 2 selects the group comprising layers 8-11. The model combines these layers into a single visual candidate and routes it through four experts. Eq. (5) treats the four layers as separate routing candidates.

We evaluated Late-Layer-Wise as described in Eq. (5) using both SigLIP2 and CLIP-B/16. The complete results, including Accuracy and Weighted-F1 on MVSA-Single and TumEmo, are reported below (mean ± SD over three seeds).

BackboneMVSA-Single Acc.MVSA-Single W-F1TumEmo Acc.TumEmo W-F1
SigLIP20.7674 ± 0.00380.7548 ± 0.00470.7134 ± 0.00150.7131 ± 0.0020
CLIP-B/160.7570 ± 0.01210.7479 ± 0.02000.7001 ± 0.00110.7001 ± 0.0016

Across both backbones and datasets, DepthRoute-Group consistently outperforms DepthRoute-Late, suggesting that the group-wise routing configuration is more effective than the late-layer-wise alternative. These results further highlight the importance of routing scope and granularity in adaptive visual-depth selection. We have updated the corresponding DepthRoute-Late entries in Table 1 of the revised submission with these results and revised the discussion of routing scope accordingly.

2. Does input-dependent routing outperform fixed routing?

We compared DepthRoute-Group with two input-independent controls using the same depth groups, group projections, shared expert pool, and classifier. Uniform Group assigns equal weights to all group–expert pairs. Learned Static Group learns a single set of routing weights shared across examples. The table reports ACC and Weighted-F1 with SigLIP2 (mean ± SD over three seeds).

MethodMVSA-Single Acc.MVSA-Single W-F1TumEmo Acc.TumEmo W-F1
Uniform Group0.7600 ± 0.00800.7516 ± 0.00670.6989 ± 0.00150.6984 ± 0.0020
Learned Static Group0.7615 ± 0.00720.7534 ± 0.01020.6973 ± 0.00230.6968 ± 0.0021
DepthRoute-Group0.7830 ± 0.00710.7794 ± 0.00730.7190 ± 0.00380.7193 ± 0.0035

DepthRoute-Group improves mean Weighted-F1 over Learned Static Group by 2.60 percentage points on MVSA-Single and 2.25 points on TumEmo. The corresponding gains over Uniform Group are 2.78 and 2.09 points. These comparisons support the benefit of input-dependent routing within the grouped architecture on both datasets.

3. How is All-Layer implemented?

At each visual token position, All-Layer concatenates the corresponding features from all 12 layers of the frozen visual encoder and applies a learned projection. It then mean-pools the projected tokens and concatenates the resulting visual vector with the text representation for classification. The same projection is applied to every example.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Comments on the updated version

Thanks for the suggestions for our submission. We have revised the manuscript in response to the reviewers’ feedback. The main changes are as follows.

  1. Table 1 correction. We corrected the results reported for DepthRoute-Late in Table 1. The revised table now reports the results of the Late-layer-wise variant.
  2. Citation corrections. We corrected several in-text citations. For example, we changed “including multiview calibration Wei et al. (2023)” to “including multiview calibration (Wei et al., 2023).”
  3. Evaluation on a third dataset. We added experiments on MVSA-Multiple and expanded the discussion of how the effect of adaptive visual-depth selection varies across datasets.
  4. Additional comparisons in the main results (Table 1). We added results for two published work, CoDe and SPP-SCL, to broaden the comparison with prior work.
  5. Additional analysis in the appendix. We expanded the ablation discussion to clarify what each control tests. In particular, we compare final-layer and globally weighted visual features before expert processing, and examine input-dependent routing within DepthRoute-Group using uniform and learned static routing controls. Thank you again for your time. We hope the revision clarifies the goal of our work, and we would be grateful if you could take it into account in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.