acceptodds
Withdrawn from ICLR 2027

Vertical Meets Horizontal: Disentangling Affective Geometry for Plug-and-Play Structured Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal sentiment analysis has made substantial progress in learning expressive cross-modal representations, yet final prediction is still commonly performed by flat classifiers that treat sentiment categories as independent targets. Such representations implicitly form a task-specific affective space whose inter-class organization defines a learned affective geometry that may remain underexploited during prediction. To address this limitation, Vertical-Meets-Horizontal Prediction (VMH-P) is introduced as a plug-and-play structured prediction framework that exploits learned affective geometry from fully trained multimodal models without modifying their representation space. VMH-P aligns available multimodal representations into a shared evidence space and disentangles the resulting affective geometry along two complementary directions. Vertically, latent affective regions are induced from class-center geometry and embedded into a Lorentz hierarchy, where path-wise routing and leaf-level discrimination jointly produce hierarchical class evidence. Horizontally, centered class directions establish supportive and contrasting relations whose strengths are adaptively refined to enable inter-class evidence propagation. The resulting structured evidence is aggregated across sources and introduced into the host logits through a confidence-gated residual correction. Experiments on three multimodal affective datasets with three heterogeneous host models show consistent improvements across all nine host–dataset combinations, with average gains of 3.84% and 4.12% in ACC and F1, respectively. The code is publicly available at https://anonymous.4open.science/r/VMH-P-5692.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Sentiment Analysis, Structured Prediction, Plug-and-Play, Affective Geometry Disentanglement, Emotion Recognition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NWxkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Vertical-Meets-Horizontal Prediction (VMH-P), a post-hoc structured prediction framework for multimodal sentiment analysis. Using representations from a frozen host model, it constructs a hyperbolic class hierarchy for coarse-to-fine prediction and a signed relation field for inter-class evidence propagation. The resulting scores refine the host logits through a confidence-gated residual correction.

Critical strengths

1.VMH-P achieves substantial performance gains across multiple host models and datasets.

  1. The ablation studies provide useful evidence that the proposed components contribute to the overall performance.
Critical weaknesses

1.The novelty over prior work is unclear. Exploiting inter-class emotion relations overlooked by flat classifiers has been explored extensively, and the paper does not clearly establish what VMH-P adds beyond these approaches.

2.The paper does not convincingly establish why the Vertical Hyperbolic Sentiment Hierarchy (VHSH) and Horizontal Affective Relation Field (HARF) are appropriate for the stated problem. The ablations show that these components help, but do not validate their claimed mechanisms. Without stronger evidence, the method appears to be an ad hoc architectural modification rather than a well-justified approach to exploiting learned affective geometry.

Questions for authors

1.Why this particular architecture? How would other designs that follow the same idea perform?

2.The problem of exploiting inter-class emotion relations beyond flat classification has already received considerable attention. Rather than proposing another architecture with better benchmark results, could you investigate which structural choices help address this problem, and why?

Reviewer_Rnz3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VMH-P, a plug-and-play post-hoc prediction framework for multimodal sentiment analysis (MSA). Given a frozen, fully trained host model, VMH-P caches its textual, visual, and fused representations, projects them into a shared evidence space, and derives class centers. From this geometry, it constructs two complementary structures: (1) a vertical hyperbolic hierarchy (VHSH), where K-means over class centers induces a coarse-to-fine tree embedded in Lorentz space, with path-wise routing and leaf-level discrimination producing hierarchical evidence; and (2) a horizontal affective relation field (HARF), where signed supportive/contrasting relations are initialized from angular affinities between class directions and adaptively refined. The resulting structured evidence is aggregated and injected as a confidence-gated residual correction to the host logits. Experiments on MVSA-Single, MVSA-Multiple, and TumEmo with three heterogeneous host models show consistent gains, with average improvements of 3.84% ACC and 4.12% F1.

Critical strengths
  1. The vertical (cross-granularity) vs. horizontal (same-level inter-class) distinction is intuitive and well motivated.
  2. VMH-P operates only on cached representations, introduces at most 0.47M trainable parameters, and adds negligible latency (≤0.02 ms/sample).
Critical weaknesses
  1. My main reservation echoes the concern that MSA is a well-explored area, and the novelty here is not sufficient to clearly clear the bar. Specifically, each ingredient is well established: hyperbolic embeddings for hierarchies, post-hoc residual logit correction, and class-relation modeling. The contribution is the specific composition. The paper does not introduce a new learning principle, geometric construction, or theoretical insight. The "vertical meets horizontal" framing is a useful expository device but not a conceptual advance. The authors should sharpen what is conceptually new beyond the assembly.

  2. The empirical scope is limited in several ways. All three datasets (MVSA-Single, MVSA-Multiple, TumEmo) are English image–text benchmarks; no video-based benchmarks such as CMU-MOSI, CMU-MOSEI, or CH-SIMS are included, even though video is a major setting for MSA. The plug-and-play claim would be much stronger with at least one video-based benchmark. Moreover, the three host models (CLMLF, D2R, SPP-SCL) come from the same research lineage, so the backbone-agnostic claim is tested only within a fairly narrow architectural family. It would be informative to test on a substantially different host, e.g., a large pretrained MLLM backbone to test whether the geometry premise holds for models trained at scale.

  3. Table 1 shows that the largest gains are on SPP-SCL (up to +8.89 F1 on MVSA-Single), while gains on TumEmo are modest (+1.44 to +5.32). The paper attributes this to host-specific representation properties, but this is post-hoc. A more systematic analysis would turn a narrative into evidence. Currently, the explanation reads as plausible but unverified.

Questions for authors
  1. How stable is the induced hierarchy under bootstrapping or different random seeds for K-means? Does tree stability correlate with downstream gains?
  2. What happens if the host is a large pretrained MLLM model Does the geometry premise still hold?
  3. On MVSA, where K=2 forces a merge of two of three polarities, how do the authors interpret the host-specific groupings? Are these semantically meaningful or artifacts?
  4. Could the horizontal relation field be replaced by a learned adjacency matrix without the geometry prior? If so, how much does the geometry initialization contribute?
Reviewer_n6s5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VMH-P (Vertical-Meets-Horizontal Prediction), a plug-and-play structured prediction framework for multimodal sentiment analysis. The key motivation is that pretrained multimodal models may already encode meaningful inter-class geometry in their representation spaces, while conventional flat classifiers largely ignore such structure. VMH-P decomposes the recovered class geometry into two complementary components: a vertical structure, which models coarse-to-fine class organization through a hyperbolic hierarchy, and a horizontal structure, which captures supportive and contrasting relations among classes at the same level. The resulting structured evidence is aggregated across modalities and used to perform confidence-gated residual correction of the original predictions. Experiments on three multimodal affective datasets with three different host models show consistent improvements, with average gains of 3.84% in accuracy and 4.12% in weighted F1.

Critical strengths
  1. The paper identifies a meaningful limitation of flat classifiers: they do not explicitly exploit the inter-class geometry already encoded in learned multimodal representations.

  2. The vertical-horizontal decomposition provides an intuitive way to model both hierarchical class organization and same-level class interactions.

  3. VMH-P improves all evaluated host-dataset combinations, demonstrating good compatibility with different pretrained multimodal models.

Critical weaknesses
  1. Proximity in the representation space does not necessarily imply a hierarchical relationship. Could the authors provide further justification or empirical evidence showing that the induced structure captures meaningful hierarchical organization rather than simply imposing a tree structure on flat clusters?

  2. Geometric alignment or opposition does not necessarily correspond to semantic or predictive support/contrast. Could the authors provide additional evidence validating this interpretation or clarify the conditions under which such geometric relations can be regarded as supportive or contrasting?

  3. Although the current ablations show that generic residual heads and randomized structures cannot fully explain the gains, it remains unclear whether better latent class geometry systematically leads to larger improvements from VMH-P. An analysis relating the quality of the recovered geometry to the corresponding performance gains would provide stronger support for the paper’s central hypothesis.

Questions for authors

How sensitive is VMH-P to the choice of the number of latent groups KK?

How much of the improvement comes from the evidence-alignment adapters rather than the structured prediction module itself?

Does the entropy-based confidence gate reliably correlate with prediction correctness?

How different are the learned horizontal relations from their geometry-based initialization after training?

Why is a Lorentz hyperbolic space necessary for the relatively shallow three-level hierarchy used in VMH-P?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.