- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work presents a novel framework for multimodal intent recognition that decomposes the task into two main components: (1) cross-modal alignment via a shared query bank to produce structured evidence embeddings that capture modality-specific information and pairwise relations, and (2) role conditioning via tokens that capture inter-modal predictive conflict and relevance to the prediction task (measured by signed counterfactual a utility metric). This aims to generate predictions that directly account for cross-modal predictive agreement and predictive utility of each modality. Results show improved performance on two MIR benchmarks over multiple baselines, and ablation studies demonstrate that evidence structuring, role-conditioning, and more direct role supervision drive performance improvements in the model. Further analyses show relationships between utility, predictive reliability, and cross-modal predictive conflict across text, audio, and video modalities, and that the proposed method retains superior performance over the baselines when textual evidence is corrupted.
Critical strengths
• The method is clearly described, and examples show how the utility measure allows for better understanding of each modality’s role in the multimodal prediction, regardless of prediction correctness or disagreement. This is helpful for auditing the model and improving explainability/transparency, and it is an important practical contribution beyond the novelty and reported performance gains.
Critical weaknesses
• Lack of variance measures on the performance – from the currently reported results, the reader is unable to get a sense of how significant performance gains are relative to baselines.
• The representative role cases are a nice showcase of how the utility measure provides useful insights into the role of each modality, but it would be helpful to have more results analyzing how this utility measure varies, for example, in the ‘Robustness under textual evidence corruption’ experiments, showing how the utility measure for the text modality varies across corruption rates would strengthen the argument that this measure (which is fairly central to the method via role-conditioning) is what drives the improved robustness over baselines. This is important to the assessment because it helps to more thoroughly show how the utility measure contributes to the model’s performance: whether it’s working as expected, or if there are other aspects of the model that are driving the improvements.
Questions for authors
Clarification Requests
-
Measure of variation on the performance metrics – what are the standard deviations across seeds for all methods? This would be helpful for a more informed comparison against baselines.
-
For the shared-query evidence extraction, how does the framework ensure the modality evidence queries are not redundant with one another / don’t collapse?
-
In pairwise relation construction, it seems that absolute difference and elementwise multiplication is used to measure element-wise agreement – however, doesn’t this depend on the scale of the vectors? Are the vectors normalized in any way before computing these measures?
-
Why do self-attention across all evidence content and role context tokens as opposed to separating evidence content and role context and doing cross-attention between them? In the current formulation, which does self-attention across all evidence and role tokens together, wouldn’t the resulting embeddings include some elements of the evidence/role tokens (rather than simply using these to help guide the attention patterns between the modality-specific/paired evidence tokens)?
-
When is the pairwise modality evidence MLP () trained? It seems that the stepwise training process starts by training the unimodal classifiers (), but then the pairwise modality evidence () vectors do not get trained until the complete model optimization. So, the role-conditioned reasoning, which depends on the pairwise modality evidence (), also depends on un-optimized vectors – have the authors considered supervising the training of these vectors as well via additional classifier heads (similar to )? Then could the predictions be generated directly from this predictor rather than doing the averaging operation? I know this would increase the number of parameters that need to be trained, but could this benefit the overall training of the complete model without it having to learn the vectors from scratch? Or is there a clever way to initialize these vectors?
-
Please provide more detailed description on how the ablations were done – it can be difficult to understand exactly what was meant by the different setting names (for evidence structuring: ‘pairwise relations’, ‘pooled raw features’, ‘matched-capacity control’).
-
For the ‘Robustness under textual evidence corruption’ experiment, did the authors also observe how the utility value the model assigned to text changed as corruption rate increased?
-
The framework uses a complicated training procedure with different auxiliary parts to avoid circularity, that other parts of the model depend on – if initial parts aren’t sufficiently optimized, presumably this could have an additive detrimental effect on the full aggregate model. For example, the signed counterfactual utility component relies on a role-free branch that is trained for a different prediction than what it’s used to train at the end – does this have negative impact on the ‘ground truth’ counterfactual utility target value used to train the final modality-specific utility estimator, and could this be improved in future work?
Suggestions for Future Work
-
Could consider expanding the methodology to handle missing modalities.
-
Could consider the case of more than 3 modalities, and accordingly, expanding the framework to handle higher order modality relations above pairwise.
5 comments
Response to Reviewer 8rbJ
(b) Role-Free Target and Staged Optimization (Q8)
The reviewer raises an important concern about the dependency between the role-free reference branch and the final role-conditioned predictor. We clarify that the role-free branch is trained for the same intent prediction task and uses the same structured multimodal evidence, but without sample-dependent role signals. In Eq. (6), measures the change in label-conditioned loss when modality and its incident relational evidence are removed. It is therefore a model-relative supervision target, rather than an independently annotated utility or a causal-effect estimate.
This construction reduces direct circularity: the evidence backbone and unimodal heads are trained first, followed by utility supervision and then role conditioning. Stop-gradient operations further prevent the final classification loss from directly modifying the role signals. We nevertheless acknowledge that the role-free predictor may be imperfect, so its targets may not exactly match the modality effects of the final predictor. The conflict-only ablation supports the usefulness of the learned utility signal, but does not establish that the reference targets are error-free.
(A3) Evidence Construction and Optimization (Q2, Q3, Q5)
(a) Query Diversity and Parameter-Level Collapse
The shared query bank provides common latent slot indices across modalities, while modality-specific attention and feed-forward parameters allow each extractor to retrieve source-dependent evidence. RCER does not impose an explicit orthogonality or diversity loss, and therefore does not provide a formal guarantee that every query retrieves nonredundant semantics.
To examine whether the learned queries collapse geometrically, we analyzed the query parameters on both datasets. As shown in R3–Table 3, the low absolute cosine similarities and near-full effective ranks indicate that the inspected query banks retain distinct parameter directions and do not exhibit substantial parameter-level collapse.
R3–Table 3: Geometric diagnostics of the learned shared query banks.
| Dataset | Number of queries | Mean pairwise cosine similarity | Mean absolute cosine similarity | Effective rank |
|---|---|---|---|---|
| MIntRec | 18 | 0.0050 | 0.0426 | 17.88 |
| MIntRec2.0 | 6 | −0.0071 | 0.0513 | 5.98 |
(b) Scale of the Relation Descriptors
The evidence vectors entering Eq. (2) are outputs of the residual attention–FFN evidence extractors described below Eq. (1), including LayerNorm before relation construction. The relation MLP also applies LayerNorm before the pair-identity embedding is added.
There is no explicit L2 normalization of the evidence endpoints. LayerNorm regulates their activation scale, but its learned affine parameters do not guarantee identical vector norms or strict scale invariance. Accordingly, the absolute-difference and elementwise-product terms should be understood as learned interaction features exposed to the relation encoder, rather than as standalone scale-invariant similarity measures.
(c) Relation-Encoder Optimization and the Definition of
The initial stage jointly trains the role-free multimodal content predictor and the unimodal heads. Since the content predictor already includes the relation MLP, receives gradients from the multimodal classification loss from the first epoch. Utility supervision and dynamic role conditioning are activated subsequently. The relation encoder is therefore already trained when role-conditioned reasoning begins.
It is also useful to distinguish the relation vector from in Eq. (5). The former represents cross-modal evidence content, whereas the latter is the midpoint distribution required by the Jensen–Shannon divergence between the unimodal beliefs. Thus, is not a prediction from the relation vector. An auxiliary pair classifier could provide additional supervision for relational evidence, but its output would serve a different purpose: substituting it for would no longer compute the stated JSD.
We appreciate the suggestion of auxiliary pairwise supervision as a possible extension. In the current model, relational evidence is learned through the multimodal prediction objective, without requiring a separate pair classifier or a special initialization scheme. We will revise the training description to make this optimization order explicit.
Response to Reviewer 8rbJ
(A4) Reasoning Design and Ablation Definitions (Q4, Q6)
(a) Joint versus Separated Attention
Joint self-attention allows role information to enter the resulting representations. This is an intended part of the design: role tokens provide contextual representations that participate in evidence reasoning, rather than serving solely as attention biases. Each slot contains one fusion token, six evidence-content tokens, and six role tokens, with the fusion-token output passed to decision aggregation. The evidence–role distinction therefore specifies how the inputs are constructed, without requiring them to remain separate throughout reasoning.
Motivated by the reviewer’s suggestion, we conducted an additional ablation comparing joint self-attention with a separated design, in which the fusion and content tokens first undergo self-attention and then attend to role tokens through cross-attention. Both variants use two attention–FFN blocks with identical parameter counts and the same training protocol.
As shown in R3–Table 4, joint self-attention outperforms the separated alternative by 0.61 ACC and 0.66 WF1 points. The separated alternative remains competitive, indicating that structured evidence and explicit role context can also be used effectively under a different interaction design. This comparison supports joint self-attention as a suitable design choice for RCER, while also clarifying that the broader evidence–role formulation is not restricted to a single attention mechanism.
R3–Table 4: Comparison of joint self-attention and separated self-/cross-attention on MIntRec.
| Interaction design | ACC (%) | R (%) | WF1 (%) | WP (%) |
|---|---|---|---|---|
| Separated self-/cross-attention | 76.55 | 72.37 | 76.22 | 76.58 |
| Joint self-attention (RCER) | 77.16 | 73.58 | 76.88 | 77.43 |
(b) Evidence-Structuring Ablations
Thank you for pointing out that the original setting names were insufficiently explicit. Settings (a)–(d) of the ablation study vary the evidence construction while retaining the same downstream role-conditioned reasoning framework:
- Raw features: Encoded modality features are pooled without shared-query extraction or explicit pairwise relations. Here, “raw” refers to features before latent evidence structuring, not unprocessed inputs.
- Raw + pairwise relations: The pooled modality features in (a) are augmented with pairwise representations constructed using Eq. (2), which combines the two endpoints, their absolute difference, and their elementwise product.
- Latent evidence: Shared-query extraction replaces raw pooling, while explicit pairwise relation representations are omitted.
- Raw + matched capacity: The pooled modality features are augmented with modality-specific attention–FFN refinement and a local adapter matching the architecture of the relation encoder. These additions operate within each modality, without shared-query extraction from the original sequences or explicit cross-modal relation content. This control examines whether additional parameterization alone can account for the observed improvement.
Future-Work Suggestions
We thank the reviewer for these thoughtful suggestions, which invite a broader investigation of how multimodal systems assess evidence as its availability and interactions vary. As an initial step, we evaluated RCER with missing non-textual modalities, providing initial evidence of RCER’s robustness to modality absence (see R2–Table 3). We plan to further explore role estimation under changing modality availability and selective modeling of higher-order relations, extending the evidence–role formulation toward more flexible multimodal reasoning beyond intent recognition.
We sincerely appreciate the reviewer’s detailed and constructive comments. We hope that these additional analyses and clarifications make the scope and empirical support of our claims clearer, and we will incorporate the corresponding discussions into the camera-ready version.
Response to Reviewer 8rbJ
We sincerely thank the reviewer for the careful and constructive evaluation. We particularly appreciate the recognition that distinguishing a modality’s standalone confidence from its conditional usefulness provides a meaningful motivation for our work, as well as the positive assessment of the clarity of our method and the practical value of interpreting modality roles beyond predictive correctness or disagreement.
We organize our responses into four themes: (A1) Statistical reliability (Q1/W1), (A2) Utility validity and effectiveness (Q7/W2, Q8), (A3) Evidence construction and optimization (Q2, Q3, Q5), and (A4) Reasoning design and ablation settings (Q4, Q6).
(A1) Statistical Reliability (Q1/W1)
As described in Section 4.1, the main results are averaged over three random seeds. We summarize the mean and sample standard deviation across seeds 0–2 in R3–Table 1. On MIntRec, RCER improves mean accuracy over MVCL-DAF by 2.03 percentage points and mean weighted F1 by 1.77 percentage points. On MIntRec2.0, RCER achieves modest mean gains over DUMIR in accuracy (+0.66 percentage points), weighted F1 (+0.36), and weighted precision (+0.99). Together with the controlled ablations in Section 4.3, these comparisons support the empirical evaluation of structured evidence and explicit role conditioning.
Following a related suggestion from Reviewer azwc, we further evaluated RCER on the CMU-MOSI benchmark, where it outperforms all evaluated baselines across five metrics, supporting transfer beyond multimodal intent recognition (see R2–Table 1).
R3–Table 1: Main comparison results reported as mean ± sample standard deviation over three random seeds.
(A2) Utility Behavior and Target Validity (Q7/W2, Q8)
(a) Utility under Textual Evidence Corruption
Thank you for encouraging a more detailed analysis of the utility signal. We evaluated the utility of textual evidence under progressively corrupted textual features on both MIntRec and MIntRec2.0. From 10% to 90% corruption, the mean text utility decreases from 0.709 to −0.673 on MIntRec and from 0.768 to −0.419 on MIntRec2.0. Over the same range, the proportion of samples with negative text utility increases from 13.03% to 85.39% and from 10.53% to 73.34%, respectively.
R3–Table 2: Counterfactual text utility under progressive textual feature corruption.
The mean utility becomes negative at 70% corruption on MIntRec and 80% corruption on MIntRec2.0. Thus, the counterfactual analysis captures the transition from beneficial textual evidence to interfering textual evidence as corruption becomes severe. This extends the representative cases to a controlled degradation setting and shows that the utility behaves consistently with the increasing unreliability of the textual modality.