acceptodds
Under review as a conference paper at ICLR 2027

From Evidence Content to Decision Roles: Role-Conditioned Reasoning for Multimodal Intent Recognition

Abstract

Multimodal intent recognition (MIR) integrates linguistic, acoustic, and visual evidence for inferring a speaker’s intent. Existing methods primarily model what this evidence conveys or how strongly it should contribute to fusion, leaving a more fundamental question underexplored: what role should the evidence play in the current decision? Reliable evidence may be redundant, while conflicting evidence can either correct predictions or introduce noise. Therefore, we propose RCER, a Role-Conditioned Evidence Reasoning framework that organizes multimodal intent inference into an evidence–role–decision pipeline. RCER employs a latent evidence structuring mechanism, which uses shared latent queries to organize heterogeneous modality sequences into slot-aligned, modality-specific evidence and explicitly encodes pairwise relations within each slot. It then performs role-conditioned evidence reasoning to characterize sample-dependent evidence roles through predictive conflict and signed counterfactual utility learned from leave-one-modality-out supervision. RCER encodes these signals as explicit role tokens that condition joint reasoning over unimodal and relational evidence, thereby guiding evidence interactions beyond scalar fusion weighting. Extensive experiments on two challenging MIR benchmarks, MIntRec and MIntRec2.0, demonstrate that RCER achieves state-of-the-art performance, highlighting the value of modeling evidence roles alongside evidence content for MIR.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Intent Recognition, Evidence Reasoning, Role-Conditioned Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8rbJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work presents a novel framework for multimodal intent recognition that decomposes the task into two main components: (1) cross-modal alignment via a shared query bank to produce structured evidence embeddings that capture modality-specific information and pairwise relations, and (2) role conditioning via tokens that capture inter-modal predictive conflict and relevance to the prediction task (measured by signed counterfactual a utility metric). This aims to generate predictions that directly account for cross-modal predictive agreement and predictive utility of each modality. Results show improved performance on two MIR benchmarks over multiple baselines, and ablation studies demonstrate that evidence structuring, role-conditioning, and more direct role supervision drive performance improvements in the model. Further analyses show relationships between utility, predictive reliability, and cross-modal predictive conflict across text, audio, and video modalities, and that the proposed method retains superior performance over the baselines when textual evidence is corrupted.

Critical strengths

• The method is clearly described, and examples show how the utility measure allows for better understanding of each modality’s role in the multimodal prediction, regardless of prediction correctness or disagreement. This is helpful for auditing the model and improving explainability/transparency, and it is an important practical contribution beyond the novelty and reported performance gains.

Critical weaknesses

• Lack of variance measures on the performance – from the currently reported results, the reader is unable to get a sense of how significant performance gains are relative to baselines.

• The representative role cases are a nice showcase of how the utility measure provides useful insights into the role of each modality, but it would be helpful to have more results analyzing how this utility measure varies, for example, in the ‘Robustness under textual evidence corruption’ experiments, showing how the utility measure for the text modality varies across corruption rates would strengthen the argument that this measure (which is fairly central to the method via role-conditioning) is what drives the improved robustness over baselines. This is important to the assessment because it helps to more thoroughly show how the utility measure contributes to the model’s performance: whether it’s working as expected, or if there are other aspects of the model that are driving the improvements.

Questions for authors

Clarification Requests

  1. Measure of variation on the performance metrics – what are the standard deviations across seeds for all methods? This would be helpful for a more informed comparison against baselines.

  2. For the shared-query evidence extraction, how does the framework ensure the modality evidence queries are not redundant with one another / don’t collapse?

  3. In pairwise relation construction, it seems that absolute difference and elementwise multiplication is used to measure element-wise agreement – however, doesn’t this depend on the scale of the vectors? Are the vectors normalized in any way before computing these measures?

  4. Why do self-attention across all evidence content and role context tokens as opposed to separating evidence content and role context and doing cross-attention between them? In the current formulation, which does self-attention across all evidence and role tokens together, wouldn’t the resulting embeddings include some elements of the evidence/role tokens (rather than simply using these to help guide the attention patterns between the modality-specific/paired evidence tokens)?

  5. When is the pairwise modality evidence MLP (Φrel\Phi_{rel}) trained? It seems that the stepwise training process starts by training the unimodal classifiers (gunig^{uni}), but then the pairwise modality evidence (rr) vectors do not get trained until the complete model optimization. So, the role-conditioned reasoning, which depends on the pairwise modality evidence (rr), also depends on un-optimized vectors – have the authors considered supervising the training of these vectors as well via additional classifier heads (similar to gunig^{uni})? Then could the qmn\mathbf{q}_{mn} predictions be generated directly from this predictor rather than doing the averaging operation? I know this would increase the number of parameters that need to be trained, but could this benefit the overall training of the complete model without it having to learn the rr vectors from scratch? Or is there a clever way to initialize these rr vectors?

  6. Please provide more detailed description on how the ablations were done – it can be difficult to understand exactly what was meant by the different setting names (for evidence structuring: ‘pairwise relations’, ‘pooled raw features’, ‘matched-capacity control’).

  7. For the ‘Robustness under textual evidence corruption’ experiment, did the authors also observe how the utility value the model assigned to text changed as corruption rate increased?

  8. The framework uses a complicated training procedure with different auxiliary parts to avoid circularity, that other parts of the model depend on – if initial parts aren’t sufficiently optimized, presumably this could have an additive detrimental effect on the full aggregate model. For example, the signed counterfactual utility component relies on a role-free branch that is trained for a different prediction than what it’s used to train at the end – does this have negative impact on the ‘ground truth’ counterfactual utility target value used to train the final modality-specific utility estimator, and could this be improved in future work?

Suggestions for Future Work

  1. Could consider expanding the methodology to handle missing modalities.

  2. Could consider the case of more than 3 modalities, and accordingly, expanding the framework to handle higher order modality relations above pairwise.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8rbJ

We sincerely thank the reviewer for the careful and constructive evaluation. We particularly appreciate the recognition that distinguishing a modality’s standalone confidence from its conditional usefulness provides a meaningful motivation for our work, as well as the positive assessment of the clarity of our method and the practical value of interpreting modality roles beyond predictive correctness or disagreement.

We organize our responses into four themes: (A1) Statistical reliability (Q1/W1), (A2) Utility validity and effectiveness (Q7/W2, Q8), (A3) Evidence construction and optimization (Q2, Q3, Q5), and (A4) Reasoning design and ablation settings (Q4, Q6).


(A1) Statistical Reliability (Q1/W1)

As described in Section 4.1, the main results are averaged over three random seeds. We summarize the mean and sample standard deviation across seeds 0–2 in R3–Table 1. On MIntRec, RCER improves mean accuracy over MVCL-DAF by 2.03 percentage points and mean weighted F1 by 1.77 percentage points. On MIntRec2.0, RCER achieves modest mean gains over DUMIR in accuracy (+0.66 percentage points), weighted F1 (+0.36), and weighted precision (+0.99). Together with the controlled ablations in Section 4.3, these comparisons support the empirical evaluation of structured evidence and explicit role conditioning.

Following a related suggestion from Reviewer azwc, we further evaluated RCER on the CMU-MOSI benchmark, where it outperforms all evaluated baselines across five metrics, supporting transfer beyond multimodal intent recognition (see R2–Table 1).

R3–Table 1: Main comparison results reported as mean ± sample standard deviation over three random seeds.

DatasetMIntRecMIntRec2.0
MethodACC (%)R (%)WF1 (%)WP (%)ACC (%)R (%)WF1 (%)WP (%)
MAG-BERT70.86 ± 0.3567.34 ± 0.4070.55 ± 0.3671.61 ± 0.7959.91 ± 0.7052.57 ± 2.1855.67 ± 2.6052.45 ± 2.13
MISA71.46 ± 0.5968.72 ± 1.0171.93 ± 0.8073.57 ± 1.3057.27 ± 0.0551.31 ± 0.1057.21 ± 0.0257.63 ± 0.05
MulT71.09 ± 1.0467.39 ± 0.3171.05 ± 0.7771.83 ± 0.2759.89 ± 0.2451.58 ± 1.3653.53 ± 1.6951.79 ± 1.21
TCL-MAP73.48 ± 1.8070.26 ± 2.4973.04 ± 2.0173.33 ± 2.3957.73 ± 0.4553.36 ± 0.5357.45 ± 0.3957.97 ± 0.57
DUMIR74.31 ± 2.4872.38 ± 3.2874.03 ± 2.2275.10 ± 2.0260.30 ± 0.9853.93 ± 3.2759.25 ± 1.6459.43 ± 1.65
MVCL-DAF75.13 ± 0.3573.39 ± 0.5375.11 ± 0.2775.53 ± 0.1759.79 ± 0.5452.94 ± 1.1858.63 ± 0.4158.98 ± 0.81
RCER77.16 ± 1.6973.58 ± 2.6376.88 ± 1.8977.43 ± 2.1160.96 ± 0.5153.53 ± 1.3959.61 ± 0.6160.42 ± 0.31

(A2) Utility Behavior and Target Validity (Q7/W2, Q8)

(a) Utility under Textual Evidence Corruption

Thank you for encouraging a more detailed analysis of the utility signal. We evaluated the utility of textual evidence under progressively corrupted textual features on both MIntRec and MIntRec2.0. From 10% to 90% corruption, the mean text utility decreases from 0.709 to −0.673 on MIntRec and from 0.768 to −0.419 on MIntRec2.0. Over the same range, the proportion of samples with negative text utility increases from 13.03% to 85.39% and from 10.53% to 73.34%, respectively.

R3–Table 2: Counterfactual text utility under progressive textual feature corruption.

Corruption rate10%20%30%40%50%60%70%80%90%
MIntRec mean utility0.7090.6960.6950.6040.4800.189−0.142−0.436−0.673
Negative utility (%)13.0313.7113.7118.6524.7240.4556.8574.3885.39
MIntRec2.0 mean utility0.7680.7670.7400.6770.5620.3430.081−0.225−0.419
Negative utility (%)10.5310.2811.5614.5120.1731.9745.9962.5773.34

The mean utility becomes negative at 70% corruption on MIntRec and 80% corruption on MIntRec2.0. Thus, the counterfactual analysis captures the transition from beneficial textual evidence to interfering textual evidence as corruption becomes severe. This extends the representative cases to a controlled degradation setting and shows that the utility behaves consistently with the increasing unreliability of the textual modality.

AuthorsOpenReview ↗

Response to Reviewer 8rbJ

(b) Role-Free Target and Staged Optimization (Q8)

The reviewer raises an important concern about the dependency between the role-free reference branch and the final role-conditioned predictor. We clarify that the role-free branch is trained for the same intent prediction task and uses the same structured multimodal evidence, but without sample-dependent role signals. In Eq. (6), um⋆u_m^\star measures the change in label-conditioned loss when modality mm and its incident relational evidence are removed. It is therefore a model-relative supervision target, rather than an independently annotated utility or a causal-effect estimate.

This construction reduces direct circularity: the evidence backbone and unimodal heads are trained first, followed by utility supervision and then role conditioning. Stop-gradient operations further prevent the final classification loss from directly modifying the role signals. We nevertheless acknowledge that the role-free predictor may be imperfect, so its targets may not exactly match the modality effects of the final predictor. The conflict-only ablation supports the usefulness of the learned utility signal, but does not establish that the reference targets are error-free.


(A3) Evidence Construction and Optimization (Q2, Q3, Q5)

(a) Query Diversity and Parameter-Level Collapse

The shared query bank provides common latent slot indices across modalities, while modality-specific attention and feed-forward parameters allow each extractor to retrieve source-dependent evidence. RCER does not impose an explicit orthogonality or diversity loss, and therefore does not provide a formal guarantee that every query retrieves nonredundant semantics.

To examine whether the learned queries collapse geometrically, we analyzed the query parameters on both datasets. As shown in R3–Table 3, the low absolute cosine similarities and near-full effective ranks indicate that the inspected query banks retain distinct parameter directions and do not exhibit substantial parameter-level collapse.

R3–Table 3: Geometric diagnostics of the learned shared query banks.

DatasetNumber of queriesMean pairwise cosine similarityMean absolute cosine similarityEffective rank
MIntRec180.00500.042617.88
MIntRec2.06−0.00710.05135.98

(b) Scale of the Relation Descriptors

The evidence vectors entering Eq. (2) are outputs of the residual attention–FFN evidence extractors described below Eq. (1), including LayerNorm before relation construction. The relation MLP also applies LayerNorm before the pair-identity embedding is added.

There is no explicit L2 normalization of the evidence endpoints. LayerNorm regulates their activation scale, but its learned affine parameters do not guarantee identical vector norms or strict scale invariance. Accordingly, the absolute-difference and elementwise-product terms should be understood as learned interaction features exposed to the relation encoder, rather than as standalone scale-invariant similarity measures.

(c) Relation-Encoder Optimization and the Definition of qmnq_{mn}

The initial stage jointly trains the role-free multimodal content predictor and the unimodal heads. Since the content predictor already includes the relation MLP, Φrel\Phi_{\mathrm{rel}} receives gradients from the multimodal classification loss from the first epoch. Utility supervision and dynamic role conditioning are activated subsequently. The relation encoder is therefore already trained when role-conditioned reasoning begins.

It is also useful to distinguish the relation vector rmnkr_{mn}^{k} from qmnq_{mn} in Eq. (5). The former represents cross-modal evidence content, whereas the latter is the midpoint distribution qmn=12(pm+pn)q_{mn}=\frac{1}{2}(p_m+p_n) required by the Jensen–Shannon divergence between the unimodal beliefs. Thus, qmnq_{mn} is not a prediction from the relation vector. An auxiliary pair classifier could provide additional supervision for relational evidence, but its output would serve a different purpose: substituting it for qmnq_{mn} would no longer compute the stated JSD.

We appreciate the suggestion of auxiliary pairwise supervision as a possible extension. In the current model, relational evidence is learned through the multimodal prediction objective, without requiring a separate pair classifier or a special initialization scheme. We will revise the training description to make this optimization order explicit.


AuthorsOpenReview ↗

Response to Reviewer 8rbJ

(A4) Reasoning Design and Ablation Definitions (Q4, Q6)

(a) Joint versus Separated Attention

Joint self-attention allows role information to enter the resulting representations. This is an intended part of the design: role tokens provide contextual representations that participate in evidence reasoning, rather than serving solely as attention biases. Each slot contains one fusion token, six evidence-content tokens, and six role tokens, with the fusion-token output passed to decision aggregation. The evidence–role distinction therefore specifies how the inputs are constructed, without requiring them to remain separate throughout reasoning.

Motivated by the reviewer’s suggestion, we conducted an additional ablation comparing joint self-attention with a separated design, in which the fusion and content tokens first undergo self-attention and then attend to role tokens through cross-attention. Both variants use two attention–FFN blocks with identical parameter counts and the same training protocol.

As shown in R3–Table 4, joint self-attention outperforms the separated alternative by 0.61 ACC and 0.66 WF1 points. The separated alternative remains competitive, indicating that structured evidence and explicit role context can also be used effectively under a different interaction design. This comparison supports joint self-attention as a suitable design choice for RCER, while also clarifying that the broader evidence–role formulation is not restricted to a single attention mechanism.

R3–Table 4: Comparison of joint self-attention and separated self-/cross-attention on MIntRec.

Interaction designACC (%)R (%)WF1 (%)WP (%)
Separated self-/cross-attention76.5572.3776.2276.58
Joint self-attention (RCER)77.1673.5876.8877.43

(b) Evidence-Structuring Ablations

Thank you for pointing out that the original setting names were insufficiently explicit. Settings (a)–(d) of the ablation study vary the evidence construction while retaining the same downstream role-conditioned reasoning framework:

  • Raw features: Encoded modality features are pooled without shared-query extraction or explicit pairwise relations. Here, “raw” refers to features before latent evidence structuring, not unprocessed inputs.
  • Raw + pairwise relations: The pooled modality features in (a) are augmented with pairwise representations constructed using Eq. (2), which combines the two endpoints, their absolute difference, and their elementwise product.
  • Latent evidence: Shared-query extraction replaces raw pooling, while explicit pairwise relation representations are omitted.
  • Raw + matched capacity: The pooled modality features are augmented with modality-specific attention–FFN refinement and a local adapter matching the architecture of the relation encoder. These additions operate within each modality, without shared-query extraction from the original sequences or explicit cross-modal relation content. This control examines whether additional parameterization alone can account for the observed improvement.

Future-Work Suggestions

We thank the reviewer for these thoughtful suggestions, which invite a broader investigation of how multimodal systems assess evidence as its availability and interactions vary. As an initial step, we evaluated RCER with missing non-textual modalities, providing initial evidence of RCER’s robustness to modality absence (see R2–Table 3). We plan to further explore role estimation under changing modality availability and selective modeling of higher-order relations, extending the evidence–role formulation toward more flexible multimodal reasoning beyond intent recognition.


We sincerely appreciate the reviewer’s detailed and constructive comments. We hope that these additional analyses and clarifications make the scope and empirical support of our claims clearer, and we will incorporate the corresponding discussions into the camera-ready version.

Reviewer_azwcOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RCER for multimodal intent recognition. It organizes text, audio, and visual features into shared latent evidence slots and models their pairwise relationships. The framework further estimates sample-dependent evidence roles using predictive conflict and signed counterfactual utility, which are incorporated as role tokens for joint reasoning. Experiments on MIntRec and MIntRec2.0 show improvements over existing multimodal fusion methods.

Critical strengths
  1. The paper provides a clear motivation for distinguishing evidence reliability from its actual contribution to a prediction. The evidence–role–decision formulation is reasonable.

  2. The experiments include recent MIR baselines, component ablations, and robustness analysis. The comparisons between direct readout, scalar gating, and role tokens are useful.

Critical weaknesses
  1. The independent contribution of signed counterfactual utility is relatively small. In Table 2, adding utility supervision improves accuracy from 76.48% to 77.16%, while a substantial portion of the improvement comes from unimodal supervision and evidence structuring.

  2. The quality of the learned utility estimator is not directly evaluated. The paper analyzes counterfactual utility distributions but provides limited evidence that the estimated utility accurately identifies beneficial, redundant, or harmful modalities at inference time.

  3. The evaluation is limited to two closely related benchmarks. The improvement on MIntRec2.0 is modest, and RCER does not achieve the best macro recall. Statistical significance analysis is also missing.

  4. Existing work has already explored modality contribution, uncertainty, predictive conflict, and counterfactual valuation. The distinction between RCER and these approaches is not sufficiently established.

  5. The robustness experiment uses synthetic corruption of textual features while leaving other modalities unchanged. This does not sufficiently establish robustness to realistic modality conflicts, missing information, or unreliable non-textual evidence.

Questions for authors
  1. Provide direct evaluation of the utility estimator.

  2. Further isolate the contribution of role tokens and utility supervision.

  3. Provide statistical significance and broader robustness analysis.

6 comments
AuthorsOpenReview ↗

Response to Reviewer azwc

We sincerely thank the reviewer for recognizing the motivation behind our evidence–role–decision formulation and for providing detailed and constructive feedback.

We organize our response into four themes: (A1) Evaluation scope and statistical reliability (W3, Q3), (A2) Robustness under modality degradation (W5, Q3), (A3) Utility validation and role ablations (W1–W2, Q1–Q2), and (A4) Methodological distinctions from prior MIR methods (W4).


(A1) Evaluation Scope and Statistical Reliability (W3, Q3)

(a) Cross-Task Evaluation on CMU-MOSI benchmark

To assess whether RCER’s gains generalize beyond multimodal intent recognition, we further evaluated RCER on CMU-MOSI, a widely used multimodal sentiment analysis benchmark. This extends the evaluation from discrete communicative-intent prediction to sentiment prediction. As shown in R2–Table 1, RCER outperforms all six evaluated baselines on all five metrics. RCER achieves an MAE of 0.663, compared with 0.738 for the strongest baseline, and improves ACC-7 from 44.31% to 48.83%. These results provide additional evidence that structured evidence and role-conditioned reasoning can generalize beyond the original intent-recognition setting. We will include these additional results in the revised manuscript.

R2–Table 1: Additional comparison results on the CMU-MOSI benchmark.

MethodMAE ↓Corr ↑ACC-2 (%) ↑F1-score (%) ↑ACC-7 (%) ↑
MISA0.7380.78485.2185.2344.31
MulT0.7760.75885.5185.4142.12
MAG-BERT0.7810.77484.9084.9543.29
TCL-MAP0.7820.78984.6084.6341.10
DUMIR0.7470.78784.9084.8943.00
MVCL-DAF0.7600.76283.6883.7744.02
RCER (Ours)0.6630.81586.1386.0948.83

(b) Seed-Level Results and Run-to-Run Variability

As described in Section 4.1, the main results are averaged over three random seeds. We summarize the mean and sample standard deviation across seeds 0–2 in R2–Table 2. On MIntRec, RCER improves mean accuracy over MVCL-DAF by 2.03 percentage points and mean weighted F1 by 1.77 percentage points. On MIntRec2.0, RCER achieves positive gains over DUMIR in accuracy (+0.66), weighted F1 (+0.36), and weighted precision (+0.99). DUMIR achieves a slightly higher mean macro recall (53.93% vs. 53.53%), whereas RCER shows lower variation in macro recall across the three runs (1.39 vs. 3.27 standard deviation points).

R2–Table 2: Main comparison results reported as mean ± sample standard deviation over three random seeds.

DatasetMIntRecMIntRec2.0
MethodACC (%)R (%)WF1 (%)WP (%)ACC (%)R (%)WF1 (%)WP (%)
MAG-BERT70.86 ± 0.3567.34 ± 0.4070.55 ± 0.3671.61 ± 0.7959.91 ± 0.7052.57 ± 2.1855.67 ± 2.6052.45 ± 2.13
MISA71.46 ± 0.5968.72 ± 1.0171.93 ± 0.8073.57 ± 1.3057.27 ± 0.0551.31 ± 0.1057.21 ± 0.0257.63 ± 0.05
MulT71.09 ± 1.0467.39 ± 0.3171.05 ± 0.7771.83 ± 0.2759.89 ± 0.2451.58 ± 1.3653.53 ± 1.6951.79 ± 1.21
TCL-MAP73.48 ± 1.8070.26 ± 2.4973.04 ± 2.0173.33 ± 2.3957.73 ± 0.4553.36 ± 0.5357.45 ± 0.3957.97 ± 0.57
DUMIR74.31 ± 2.4872.38 ± 3.2874.03 ± 2.2275.10 ± 2.0260.30 ± 0.9853.93 ± 3.2759.25 ± 1.6459.43 ± 1.65
MVCL-DAF75.13 ± 0.3573.39 ± 0.5375.11 ± 0.2775.53 ± 0.1759.79 ± 0.5452.94 ± 1.1858.63 ± 0.4158.98 ± 0.81
RCER77.16 ± 1.6973.58 ± 2.6376.88 ± 1.8977.43 ± 2.1160.96 ± 0.5153.53 ± 1.3959.61 ± 0.6160.42 ± 0.31

AuthorsOpenReview ↗

Response to Reviewer azwc

(A2) Robustness under Modality Degradation (W5, Q3)

We thank the reviewer for the helpful suggestion to broaden the robustness evaluation. We additionally evaluated a setting in which a non-textual modality is unavailable from the beginning. The original robustness experiment corrupts textual features at inference time while keeping audio and video unchanged. In the new experiments, we remove video in the T+A setting and audio in the T+V setting throughout training, validation, and testing. All methods are retrained using only the modalities available in each setting.

As shown in R2–Table 3, RCER achieves the highest accuracy, weighted F1, and weighted precision in both settings. In T+A, RCER obtains 74.83% ACC, 74.58% WF1, and 75.09% WP. In T+V, it obtains 76.40% ACC, 76.16% WF1, and 76.35% WP. Its accuracy exceeds that of the strongest baseline in accuracy, MVCL-DAF, by 0.45 and 1.79 percentage points, respectively. Together, these experiments show that RCER maintains a performance advantage both when textual evidence is corrupted at inference time and when a non-textual modality is absent during training and inference.

R2–Table 3: Robustness results on MIntRec when a non-textual modality is unavailable.

SettingT+AT+V
MethodACC (%)R (%)WF1 (%)WP (%)ACC (%)R (%)WF1 (%)WP (%)
MAG-BERT72.1368.1271.6172.1572.8170.9572.5773.95
MISA72.1368.7172.1473.9773.0370.2472.7873.73
MulT73.0369.2872.5672.8674.3873.2474.6575.78
TCL-MAP72.5870.0972.3473.0074.1672.3473.8974.50
DUMIR73.9370.3373.6974.6374.3872.8174.1575.14
MVCL-DAF74.3871.6174.4174.7574.6171.9674.5075.09
RCER74.8371.8474.5875.0976.4072.5276.1676.35

(A3) Utility Validation and Role Ablations (W1–W2, Q1–Q2)

(a) Utility-Estimator Validation

We first examined whether the role-free utility targets remain relevant to the final role-conditioned predictor. Specifically, we recomputed leave-one-modality-out effects in the final predictor, re-estimating the role representations after removing each modality. The resulting effects show Spearman correlations of 0.809, 0.695, and 0.452 with the role-free utility targets for text, audio, and video, respectively. This indicates that the role-free targets capture modality effects that remain relevant after role-conditioned reasoning, particularly for text and audio.

We also evaluated utility-estimation accuracy for samples with near-zero utility targets, where small prediction errors are particularly important. For near-zero utility targets (∣u∗∣≤0.05|u^*|\leq0.05), the held-out prediction MAEs are 0.0203 and 0.0306 for audio and video, respectively, with 93.98% and 84.67% of predictions remaining within the same interval. In our held-out set, no text utility target falls within this interval.

As reported in our response to Reviewer 8rbJ, progressive textual corruption reduces the mean utility, while increasing the proportion of negative-utility samples. This further indicates that the utility signal tracks changes in evidence quality (see R3–Table 2).

(b) Role Tokenization and Utility Supervision

R2–Table 4 compares Full RCER with four controls on MIntRec. We agree that the gain over Conflict only is modest, at 0.68 percentage points in accuracy and 0.64 points in weighted F1. This comparison measures the incremental benefit of adding learned utility to a model that already uses structured evidence and predictive conflict.

In RCER, conflict characterizes disagreement among unimodal predictions and also provides context for utility estimation. Signed utility complements it by estimating whether retaining a modality helps or hurts prediction given the other evidence. Both signals inform joint reasoning over structured evidence and its sample-dependent roles.

Role tokenization. Scalar role gating and Full RCER use the same types of conflict and utility signals. Encoding them as explicit role tokens improves accuracy and weighted F1 by 1.95 and 1.83 percentage points, respectively. The Static role token control retains the token layout, role identities, and reasoning architecture while disabling both sample-dependent inputs. Full RCER improves the two metrics by 1.43 and 1.52 points over this control, showing the value of dynamic role information with the token structure held fixed.

AuthorsOpenReview ↗

Response to Reviewer azwc

Utility supervision. The w/o utility supervision control retains the conflict input and feeds the untrained utility estimator’s output into the role tokens. It achieves 76.30% accuracy and 75.85% weighted F1, slightly below Conflict only, where the utility input is disabled. With the utility channel held fixed, supervision improves accuracy and weighted F1 by 0.86 and 1.03 points, respectively. Together, these comparisons show that supervising the estimator improves performance when its output is used and that the learned utility signal adds value beyond conflict alone.

R2–Table 4: Controlled ablations of role tokenization and utility supervision on MIntRec.

SettingRole representationConflict inputUtility inputACC (%)R (%)WF1 (%)WP (%)
Scalar role gatingScalar gates✓Supervised estimate75.2171.7575.0575.41
Static role tokensRole tokens——75.7371.6775.3675.88
Conflict onlyRole tokens✓—76.4873.1476.2476.88
w/o utility supervisionRole tokens✓Untrained estimate76.3072.1675.8576.22
Full RCERRole tokens✓Supervised estimate77.1673.5876.8877.43

(A4) Methodological Distinctions from Prior MIR Methods (W4)

We thank the reviewer for this constructive comment. We agree that modality contribution, uncertainty, predictive conflict, and counterfactual valuation have each been explored in recent MIR approaches. Our distinction is therefore not that RCER is the first method to consider these signals, but that it organizes them at a different level and for a different purpose.

Prior MIR methods mainly regulate modality-level contributions or use auxiliary signals to guide fusion. DUMIR [1] adjusts modality contributions according to estimated reliability, MVCL-DAF [2] dynamically allocates attention across modality views, and ARL [3] calibrates modality imbalance. LGSRR [4] reasons over semantic importance, complementarity, and inconsistency, while CDPR [5] uses consensus and inconsistency to guide two adaptive pathways. Existing uncertainty- and counterfactual-based methods likewise primarily use these signals to calibrate, reweight, or evaluate modality contributions.

In contrast, RCER explicitly separates evidence content—what a piece of evidence conveys—from evidence role—how it should inform the prediction for a given sample—and reasons over both jointly. This separation is implemented through two coupled components. Latent evidence structuring uses shared queries to organize heterogeneous modality sequences into aligned latent slots and constructs pairwise relations within each slot. Role-conditioned reasoning encodes predictive conflict and estimated signed utility into modality-role and relation-role tokens, which interact with structured evidence.

The two components are complementary: evidence structuring defines the units and relations over which the model reasons, while role conditioning determines how these structured units inform the final prediction. This design separates evidence reliability from its contribution to the joint prediction, which is the key methodological distinction from the above approaches.

References

[1] DUMIR: Dynamic Uncertainty-aware Multimodal Intent Recognition. Gong et al. Pattern Recognition, 2026.
[2] Adaptive Multimodal Fusion: Dynamic Attention Allocation for Intent Recognition. Hu et al. AAAI, 2025.
[3] Adaptive Re-calibration Learning for Balanced Multimodal Intention Recognition. Yang et al. NeurIPS, 2025.
[4] LLM-Guided Semantic Relational Reasoning for Multimodal Intent Recognition. Zhou et al. EMNLP, 2025.
[5] Mitigating Multimodal Inconsistency via Cognitive Dual-pathway Reasoning for Intent Recognition. Wang et al. ACM ICMR, 2026.


We hope that these additional results and clarifications address the reviewer’s concerns and make the evaluation scope, robustness, utility analysis, and methodological distinction of RCER clearer. We would appreciate your consideration of these additional results and clarifications when reassessing the submission and will incorporate any remaining clarifications in the camera-ready version.

Reviewer_nPzuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RCER for multimodal intent recognition. Shared latent queries extract modality-specific evidence slots, and pairwise descriptors encode relations within each slot. Predictive disagreement and a learned estimate of signed leave-one-modality-out utility are encoded as role tokens, which condition a Transformer reasoner before decision aggregation. The model is evaluated on MIntRec and MIntRec2.0, with ablations and synthetic text-feature corruption experiments on MIntRec.

Critical strengths
  1. Distinguishing a modality's standalone confidence from its conditional usefulness is a meaningful motivation. The method specifies how utility targets are constructed during training and how a learned estimator replaces label-dependent target computation at inference. The paper also appropriately distinguishes this supervised utility from a causal-effect estimate.

  2. The empirical study includes useful controls: reproduced baselines with common inputs and splits, a matched-capacity evidence-structuring control, and a comparison of scalar gates and explicit role tokens using the same signals. RCER improves over the best reported MIntRec baseline by 2.03 accuracy points and 1.77 weighted-F1 points.

Critical weaknesses
  1. The central claim about learning useful decision roles is not sufficiently validated at inference. Section 4.4 and Figures 3-4 analyze the target utility u*, whereas the deployed model consumes predicted utility u-hat. The manuscript does not report held-out utility prediction error, sign discrimination, or correlation by modality. Moreover, the targets are computed with a role-free branch, so their relevance to the final role-conditioned predictor needs to be established. The qualitative examples demonstrate that the defined target can describe different cases, but not that the learned estimator reliably identifies those cases. Table 2 shows a relatively modest additional gain from the utility stage (76.48 to 77.16 accuracy; 76.24 to 76.88 weighted F1). This makes direct verification of the proposed mechanism particularly important.

  2. The comparisons do not yet establish the advantage over the closest reasoning-based alternatives. Section 2 discusses LGSRR and CDPR as approaches to semantic relations and multimodal conflict, but neither appears in Table 1, and their exclusion is not explained. The larger MIntRec2.0 benchmark shows only +0.66 accuracy and +0.36 weighted F1 over DUMIR, with lower macro recall. Although results are averaged over three seeds, dispersion or paired uncertainty is not reported, and the mechanism ablations are restricted to MIntRec. Thus, the evidence for a robust advance over nearby methods remains limited.

Questions for authors
  1. On held-out samples, how accurately does u-hat predict u*, including sign discrimination for beneficial versus interfering evidence and performance near zero utility? Does utility estimated from the role-free branch agree with modality effects in the final predictor?

  2. With the backbone, auxiliary supervision, schedule, and reasoner held fixed, how do predicted utilities compare with shuffled or constant utilities and an explicitly labeled oracle-utility diagnostic? Can the main utility ablation be confirmed on MIntRec2.0?

  3. Can you provide comparable results for LGSRR and CDPR, or explain the obstacles to a fair comparison, together with seed-level variability for the principal comparisons?

3 comments
AuthorsOpenReview ↗

Response to Reviewer nPzu

We sincerely thank the reviewer for the careful and constructive evaluation. We particularly appreciate the recognition that distinguishing a modality’s standalone confidence from its conditional usefulness provides a meaningful motivation for our work.

We organize our response into three themes: (A1) Utility estimation and target validity (W1, Q1), (A2) Controlled utility and role ablations (W1–W2, Q2), and (A3) Comparative evaluation and statistical reliability (W2, Q3).


(A1) Utility Estimation and Target Validity (W1, Q1)

We first examined whether the role-free utility targets remain relevant to the final role-conditioned predictor. Specifically, we recomputed leave-one-modality-out effects in the final predictor, re-estimating the role representations after removing each modality. The resulting effects show Spearman correlations of 0.809, 0.695, and 0.452 with the role-free utility targets for text, audio, and video, respectively. This indicates that the role-free targets capture modality effects that remain relevant after role-conditioned reasoning, particularly for text and audio.

We also evaluated utility-estimation accuracy for samples with near-zero utility targets, where small prediction errors are particularly important. For near-zero utility targets (∣u∗∣≤0.05|u^*|\leq0.05), the held-out prediction MAEs are 0.0203 and 0.0306 for audio and video, respectively, with 93.98% and 84.67% of predictions remaining within the same interval. In our held-out set, no text utility target falls within this interval.

As reported in our response to Reviewer 8rbJ, progressive textual corruption reduces the mean utility, while increasing the proportion of negative-utility samples. This further indicates that the utility signal tracks changes in evidence quality (see R3–Table 2).


(A2) Controlled Utility and Role Ablations (W1–W2, Q2)

(a) Matched Utility-Conditioning Ablations

Following the reviewer’s suggestion, we extended the utility-conditioning study to MIntRec2.0. We compared three settings: sample-specific predicted utilities, utilities shuffled across samples, and a constant zero utility vector. The backbone, evidence encoder, reasoner, auxiliary objectives, and training schedule were kept fixed across all settings.

R1–Table 1: Utility-conditioning controls on MIntRec2.0

Utility conditioningACC (%)R (%)WF1 (%)WP (%)
Shuffled59.8653.5158.9959.86
Zero constant59.8151.5858.5060.06
Predicted (RCER)60.9653.5359.6160.42

As shown in R1–Table 1, the predicted-utility model improves over shuffled utility by 1.10 ACC and 0.62 WF1 points, and over zero utility by 1.15 ACC and 1.11 WF1 points. Because all three settings retain the same model capacity, auxiliary objectives, and optimization schedule, these results indicate that the gain is not explained solely by the presence of a utility pathway or by the auxiliary supervision associated with it. The degradation under shuffled utilities further suggests that the correspondence between a utility vector and its own sample is important.

(b) Role-Tokenization and Utility-Supervision Ablations

We further added two controls on MIntRec to separate dynamic role information from token structure and to isolate the effect of utility supervision.

R1–Table 2: Controlled ablations of role representation and utility supervision on MIntRec

SettingRole representationConflict inputUtility inputACC (%)R (%)WF1 (%)WP (%)
Scalar role gatingScalar gates✓Supervised estimate75.2171.7575.0575.41
Static role tokensRole tokens——75.7371.6775.3675.88
Conflict onlyRole tokens✓—76.4873.1476.2476.88
w/o utility supervisionRole tokens✓Untrained estimate76.3072.1675.8576.22
Full RCERRole tokens✓Supervised estimate77.1673.5876.8877.43

As shown in R1–Table 2, the static-token control preserves the token layout, role identities, and reasoning architecture while removing sample-dependent conflict and utility inputs. Full RCER improves over this control by 1.43 ACC and 1.52 WF1 points, indicating that the benefit comes from dynamic role information rather than from adding role tokens alone. The comparison between the model without utility supervision and the full model further shows that supervising the utility estimator improves ACC and WF1 by 0.86 and 1.03 points, respectively, when the utility channel is retained. Relative to conflict-only conditioning, the learned utility provides an additional 0.68 ACC and 0.64 WF1 points.

AuthorsOpenReview ↗

Response to Reviewer nPzu

We agree that the isolated gain from adding signed utility is smaller than the gain from the full evidence–role framework. This comparison evaluates utility after latent evidence structuring and predictive-conflict supervision are already present. It therefore measures the incremental contribution of one role signal rather than the value of RCER as a whole. The results nevertheless show that signed utility provides complementary directional information beyond predictive conflict, while the overall improvement arises from its integration with structured evidence and role-conditioned reasoning.


(A3) Comparative Evaluation and Statistical Reliability (W2, Q3)

(a) Comparisons with CDPR and LGSRR

We additionally reproduced CDPR on both benchmarks and LGSRR on MIntRec2.0 using their publicly released implementations. For LGSRR, we retained the released LLM-generated semantic descriptions, ranking supervision, and training configuration. These resources are provided for MIntRec2.0 but not for MIntRec. Therefore, we report LGSRR only on MIntRec2.0 to avoid introducing additional choices in semantic-resource construction or training configuration.

R1–Table 3: Comparisons with CDPR and LGSRR across the available benchmarks

DatasetMIntRecMIntRec2.0
MethodACC(%)R(%)WF1(%)WP(%)ACC(%)R(%)WF1(%)WP(%)
CDPR74.31 ± 0.2671.19 ± 0.5574.42 ± 0.2575.36 ± 0.8660.57 ± 0.3652.69 ± 0.9959.17 ± 0.5560.05 ± 0.60
LGSRR————59.58 ± 0.5053.37 ± 0.3258.90 ± 0.3059.45 ± 0.73
RCER77.16 ± 1.6973.58 ± 2.6376.88 ± 1.8977.43 ± 2.1160.96 ± 0.5153.53 ± 1.3959.61 ± 0.6160.42 ± 0.31

As shown in R1–Table 3, RCER improves over CDPR by 2.85 ACC and 2.46 WF1 points on MIntRec, and by 0.39 ACC and 0.44 WF1 points on MIntRec2.0. On MIntRec2.0, RCER also improves over LGSRR by 1.38 ACC and 0.71 WF1 points. Thus, RCER achieves the highest mean ACC and WF1 among the compared methods in every available benchmark setting.

(b) Seed-Level Variability and Benchmark-Dependent Gains

We additionally report the mean and sample standard deviation across three runs with seeds 0, 1, and 2 for all principal methods and metrics.

R1–Table 4: Main comparison results reported as mean ± sample standard deviation over three random seeds

DatasetMIntRecMIntRec2.0
MethodACC(%)R(%)WF1(%)WP(%)ACC(%)R(%)WF1(%)WP(%)
MAG-BERT70.86 ± 0.3567.34 ± 0.4070.55 ± 0.3671.61 ± 0.7959.91 ± 0.7052.57 ± 2.1855.67 ± 2.6052.45 ± 2.13
MISA71.46 ± 0.5968.72 ± 1.0171.93 ± 0.8073.57 ± 1.3057.27 ± 0.0551.31 ± 0.1057.21 ± 0.0257.63 ± 0.05
MulT71.09 ± 1.0467.39 ± 0.3171.05 ± 0.7771.83 ± 0.2759.89 ± 0.2451.58 ± 1.3653.53 ± 1.6951.79 ± 1.21
TCL-MAP73.48 ± 1.8070.26 ± 2.4973.04 ± 2.0173.33 ± 2.3957.73 ± 0.4553.36 ± 0.5357.45 ± 0.3957.97 ± 0.57
DUMIR74.31 ± 2.4872.38 ± 3.2874.03 ± 2.2275.10 ± 2.0260.30 ± 0.9853.93 ± 3.2759.25 ± 1.6459.43 ± 1.65
MVCL-DAF75.13 ± 0.3573.39 ± 0.5375.11 ± 0.2775.53 ± 0.1759.79 ± 0.5452.94 ± 1.1858.63 ± 0.4158.98 ± 0.81
RCER77.16 ± 1.6973.58 ± 2.6376.88 ± 1.8977.43 ± 2.1160.96 ± 0.5153.53 ± 1.3959.61 ± 0.6160.42 ± 0.31

On MIntRec, RCER improves over the strongest baseline, MVCL-DAF, by 2.03 ACC and 1.77 WF1 points. On MIntRec2.0, the gains are more modest but positive on the primary overall metrics: +0.66 ACC, +0.36 WF1, and +0.99 WP over DUMIR. DUMIR has a slightly higher mean macro recall, whereas RCER has lower run-to-run variation on this metric (1.39 vs. 3.27 standard deviation points). We report the per-seed results in the supplementary material to make the variability and benchmark-dependent performance differences fully transparent.

Following a related suggestion from Reviewer azwc, we also evaluated RCER on the CMU-MOSI benchmark, where it outperforms all evaluated baselines across five metrics, supporting transfer beyond MIR (see R2–Table 1).


We hope that these responses and additional results clarify the empirical support and scope of our claims. We would appreciate your consideration of them when reassessing the submission and will incorporate any remaining clarifications in the camera-ready version.

Dear Reviewer nPzu,

We hope this message finds you well. As the discussion period draws to a close, we just wanted to respectfully check in and see whether our response has helped address the initial concerns you raised. If there are any remaining points you would like us to consider, please let us know. Your insights are invaluable to us, and we would be happy to address any remaining issues to improve our work.

Thank you again for your time and constructive feedback.

Best regards!

Comments on the paper

2 comments
Area_Chair_Pt7sOpenReview ↗

Discussion reminder

Dear Reviewers,thank you for your time and thoughtful evaluation of the Paper. As the discussion period is closing soon, please carefully review the authors' responses alongside the other reviews, and participate in the discussion to share your updated thoughts and comments.

AC

AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged by their recognition of the paper’s central perspective, the clarity of the proposed framework, and its potential to support more effective and interpretable multimodal reasoning.

In response to the reviewers’ comments, we have conducted additional experiments and analyses, clarified several methodological aspects, and incorporated the corresponding revisions into the manuscript. Below, we summarize the core contributions of RCER and the main revisions made during the rebuttal.


Core Contributions of RCER

  1. Evidence–role–decision formulation: We formulate multimodal intent recognition (MIR) from an evidence–role–decision perspective, distinguishing between the information conveyed by task-conditioned evidence and its sample-dependent role in the current joint decision.
  2. Latent Evidence Structuring: We develop a shared-query mechanism that organizes heterogeneous modality streams into slot-aligned modality-specific evidence and explicit pairwise relational evidence, establishing a structured representation for subsequent reasoning.
  3. Role-Conditioned Evidence Reasoning: We characterize evidence roles through predictive conflict and estimated signed counterfactual utility, and encode these signals as explicit role tokens to condition slot-wise reasoning. This allows evidence content and role context to interact during reasoning, rather than reducing modality contributions to scalar fusion weights.

Manuscript Updates during Rebuttal

  • Main Text: Added cross-task evaluation on CMU-MOSI to assess applicability beyond intent recognition (Section 4.6, Table 4; azwc); clarified the evidence-structuring ablation settings (Section 4.3; 8rbJ); and clarified initial-stage optimization and the scope of the role-free utility targets (Section 3.4; nPzu, 8rbJ).
  • Appendix: Added missing-modality experiments under T+A and T+V settings (Appendix E, Table 6; azwc, 8rbJ); and added controls for role representation and utility supervision on MIntRec, together with sample-specific utility conditioning on MIntRec2.0 (Appendix F, Tables 7–8; nPzu, azwc).

All revisions to the manuscript are highlighted in blue for ease of reference.


We hope that these revisions and the additional evidence address the concerns raised and assist the reviewers in their further assessment. We sincerely appreciate their time and effort in evaluating our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.