acceptodds
Withdrawn from ICLR 2027

From Language Reasoning to Cross-Modal Sentiment Evidence: Recursive Harness Search for Multimodal Sentiment Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The performance of multimodal sentiment analysis depends not only on the model itself, but also on how information is organized and on how the model is coordinated with external reasoning. Recursive self-improvement (RSI) provides an automated route to this by iteratively searching, executing, and modifying executable programs around a base model. However, existing RSI work has focused almost exclusively on text-based reasoning tasks. Unlike purely textual settings, sentiment cues in multimodal sentiment analysis are distributed across text, audio, and visual modalities, where they may complement or conflict with one another and vary from sample to sample. Consequently, the acquisition and utilization of sentiment cues must be searched jointly with the reasoning strategy and carried by the same executable program. To operationalize this view, we present SentiHarness, a recursive Harness search framework that jointly edits sentiment-cue acquisition, the organization of modality-specific evidence across model calls, cross-modal evidence use, and reasoning within one executable program. With the inner-loop model fixed, a matched CH-SIMS run improves the validation frontier from 81.14% under Fixed-Exposure Reasoning Search to 89.0% when multimodal evidence handling is made searchable. We further find that Harness effectiveness depends strongly on the state of the inner-loop model: after Plain SFT, the marginal benefit of the same Harness can shrink substantially or even reverse, revealing a concrete model–Harness interaction. Motivated by this coupling, we further explore Harness-aware SFT, which improves retained-Harness Acc-2 from 88.18% with Plain SFT to 90.37% under a matched-budget CH-SIMS training setting. These results identify executable multimodal evidence handling and model–Harness interaction as two central dimensions of recursive Harness optimization for multimodal sentiment analysis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Sentiment Recognition, Multimodal Large Language Models, Recursive Improvement, Harness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_viRFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SentiHarness, a recursive framework that jointly optimizes multimodal evidence acquisition, cross-modal integration, and reasoning strategies for sentiment recognition. It further investigates model–Harness interactions and introduces Harness-aware SFT. Experiments on CH-SIMS, CMU-MOSI, and MELD demonstrate performance improvements over the base model and several existing methods.

Critical strengths
  1. The paper extends recursive Harness optimization to multimodal sentiment recognition, jointly searching evidence acquisition and reasoning strategies rather than optimizing prompts alone.

  2. The experiments show consistent improvements over fixed-exposure reasoning search across three CH-SIMS search trajectories. The model–Harness interaction analysis also provides useful insights into model adaptation.

Critical weaknesses
  1. The search jointly changes modality exposure, call structure, evidence organization, and decision rules. Without isolating these components, the source of the performance improvements remains unclear.

  2. Although recent methods such as GEPA and Meta-Harness are discussed, direct experimental comparisons are missing, making it difficult to assess the benefits of the proposed search framework over existing optimization approaches.

  3. The main controlled adaptation result relies on a single training run. The marginal Harness-utility recovery is not statistically significant, while CMU-MOSI and MELD use unmatched training budgets, weakening the generality of the adaptation claim.

  4. The experiments primarily use one backbone model. Moreover, Appendix L shows that directly transferring the CH-SIMS Harness to CMU-MOSI substantially reduces accuracy, raising concerns about dataset-specific optimization.

Questions for authors
  1. Can the authors provide component-level ablations and compute-matched comparisons to identify which search operations contribute most to the improvements?

  2. How does SentiHarness compare with recent automatic program or Harness optimization methods under the same backbone and search budget?

  3. Does the proposed framework remain effective with different backbone models, and how can the substantial performance degradation in cross-dataset Harness transfer be explained?

Reviewer_pGWiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a recursive program-search framework for multimodal sentiment recognition. A proposer LLM revises evidence acquisition, model-call organization, and decision logic around a frozen multimodal model using execution feedback. The paper further examines how Harness utility changes after model fine-tuning and explores Harness-aware SFT through a retained request–response interface.

Critical strengths
  1. The study of model–Harness interaction provides a useful empirical observation.
  2. well performance
Critical weaknesses
  1. The manuscript gives me the impression of an insufficiently curated auto-research output. It lacks an overall pipeline diagram, and have some formatting issues (e.g., Table 9 and Appendix G.1).

  2. The main contribution appears to extend existing Harness-search methods to MSA. Beyond demonstrating that a broader search space improves performance, I think that the paper provides limited insight into which discovered strategies are reliably useful.

Questions for authors
  1. Could targeted ablations isolate the contributions of the visual mosaic, modality-specific queries, programmatic decision rules, and conditional verification?

  2. In Harness-aware SFT, all voice_verdict and face_verdict targets are fixed to none. Why?

Reviewer_sDJEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SentiHarness, a recursive search framework for multimodal sentiment analysis (MSA) that jointly optimizes sentiment-cue acquisition,

Critical strengths

Novel problem formulation that unifies evidence acquisition and reasoning as jointly searchable in MSA task.

Critical weaknesses
  1. While CH-SIMS provides matched-budget comparisons, the CMU-MOSI and MELD results use unmatched training budgets (Table 13), making it unclear whether observed differences arise from the Harness-aware training interface or from different optimization resources. More critically, Appendix L shows that direct cross-dataset transfer of the CH-SIMS-searched H to CMU-MOSI degrades performance, with the Harness applying overly conservative decision rules. This suggests that searched Harnesses may absorb dataset-specific annotation priors and cue distributions rather than discovering generalizable multimodal reasoning strategies. The lack of cross-dataset search or controlled transfer experiments limits confidence in the method's broader applicability.

  2. The paper establishes that the full editable space outperforms fixed-exposure search (RQ1), but does not ablate the separate contributions of modality dropping, conditional branching, structured evidence labels, or cross-call organization. Without this decomposition, it's unclear which mechanisms are essential versus incidental. Additionally, the inference cost is substantial: H requires up to 2278 model calls per dataset, and search itself involves 50 rounds of proposal-evaluation cycles with a separate proposer model. The paper does not report wall-clock search time, total inference cost comparison with baselines, or a guide on when the search investment is justified versus simpler prompt engineering.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.