acceptodds
Under review as a conference paper at ICLR 2027

Forge and Bridge: Strictly Inductive Multimodal Intent Discovery

Abstract

Unsupervised multimodal intent discovery aims to uncover potential intent categories from unlabeled multimodal interactions and generalize the discovered structure to unseen samples. A key challenge lies in the fact that although multimodal large language models can provide rich sample-level semantic descriptions, if we directly rely on MLLMs for classification, it often leads to category collapse. In this paper, we propose a unified framework named IntentForge–IntentBridge, which separates semantic descriptions, intent category discovery, and prediction for unseen samples: IntentForge organizes samples based on their consensus and organizes them into a structured semantic view, from which fixed training partitions are discovered, while IntentBridge migrates this partition to unseen samples through reliability-guided pointwise prediction. The framework uses MLLMs to describe what each sample means, thereby mitigating dominant-category attraction, redundant categories, and fragmented intent structures caused by direct MLLM categorization, and discovers the intent taxonomy from semantic consensus across the entire training set. Our algorithm achieves 62.95%, 47.70%, and 39.49% accuracy on MIntRec, MELD-DA, and IEMOCAP-DA. Especially on the MIntRec dataset, the accuracy rate is far superior to that of existing works, and it also achieves excellent performance on the MELD-DA dataset.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-View Clustering, Unsupervised Multimodal Intent Discovery, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zgvBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the problem of strictly inductive unsupervised multimodal intent discovery and proposes the IntentForge–IntentBridge (IntentFB) framework.

First, the authors employ a frozen Multimodal Large Language Model (MLLM) to generate a "semantic card" for each utterance, devoid of any predefined category labels. Subsequently, IntentForge derives multiple semantic views from these cards to construct a consensus neighborhood graph. By integrating spectral clustering, core-prototype refinement, and anchored propagation, the framework establishes a fixed set of anonymous training categories. Next, IntentBridge transfers this categorization to unseen samples through reliability-weighted student–EMA training, supported by a frozen exemplar retrieval mechanism. During inference, each query is processed independently using only the frozen artifacts derived from the training phase; no relationships between test samples are established, and no test-time updates are performed. The proposed method achieves an accuracy (ACC) of 62.92%, 47.70%, and 39.49% on the MIntRec, MELD-DA, and IEMOCAP-DA datasets, respectively. The appendix provides further analysis regarding reliability, the impact of pseudo-label noise, the stability of representations and decision-making, and the sufficient conditions required for anchored propagation.

Critical strengths
  1. The author’s motivation is clear and well-defined. The paper highlights a crucial distinction: while Multimodal Large Language Models (MLLMs) can provide reasonable semantic interpretations for individual utterances, this does not imply they can construct a consistent category system across an entire dataset. Decoupling "per-sample semantic description" from "category system construction" is a well-motivated research approach. This differs from the logic of previous works, which typically perform clustering first and then use representative samples to generate semantic guidance.

  2. The paper presents rigorous controlled experiments for the "category partitioning module," accompanied by a thorough analysis of the underlying mechanisms. By keeping factors such as prompts/instructions, semantic features, context, input budget, data partitioning, and random seeds consistent, the authors demonstrate that IntentForge achieves strong performance across multiple benchmarks compared to baseline methods.

  3. The appendix provides a direct analysis of the relationship between "reliability" and "pseudo-label error," while also offering a detailed breakdown of the dynamics of "effective supervision noise." This analysis is highly valuable for understanding why the proposed method is effective.

Critical weaknesses
  1. The benchmark advantages have not been validated under conditions where models and representation schemes are fully aligned, leading to unfair comparisons. The current evaluation mixes results obtained from local re-runs with those cited directly from the original GLAD/MCSP papers. While methods from various studies retain their respective original models and input configurations, the proposed approach—IntentFB—utilizes MiniCPM-o-4.5 to generate semantic "cards" and employs BGE-M3 for encoding. The appendix acknowledges that discrepancies in models and input configurations persist in these comparisons. Consequently, the substantial performance gains observed on MIntRec and MELD-DA cannot be definitively attributed to the Forge/Bridge algorithms themselves, as opposed to the more robust semantic representation pipeline. The authors must provide a reasonable explanation or additional experimental evidence to address this weakness.

  2. The authors should address the contradiction between the "category-collapse" argument and the decline in certain experimental metrics—specifically, the lower macro-recall on MELD-DA/IEMOCAP-DA, as well as the lower ARI, NMI, and FMI scores on IEMOCAP-DA.

Questions for authors
  1. The benchmark advantages have not been validated under conditions where models and representation schemes are fully aligned, leading to unfair comparisons. The current evaluation mixes results obtained from local re-runs with those cited directly from the original GLAD/MCSP papers. While methods from various studies retain their respective original models and input configurations, the proposed approach—IntentFB—utilizes MiniCPM-o-4.5 to generate semantic "cards" and employs BGE-M3 for encoding. The appendix acknowledges that discrepancies in models and input configurations persist in these comparisons. Consequently, the substantial performance gains observed on MIntRec and MELD-DA cannot be definitively attributed to the Forge/Bridge algorithms themselves, as opposed to the more robust semantic representation pipeline. The authors must provide a reasonable explanation or additional experimental evidence to address this weakness.

  2. The authors should address the contradiction between the "category-collapse" argument and the decline in certain experimental metrics—specifically, the lower macro-recall on MELD-DA/IEMOCAP-DA, as well as the lower ARI, NMI, and FMI scores on IEMOCAP-DA.

  3. How were the shared hyperparameters in Table 6 determined? Please explicitly state the development criteria and the specific data used at each step of the model selection process.

  4. There are grammatical errors in the paper, the author should revise it.

7 comments
AuthorsOpenReview ↗

Answer to Question 1

We sincerely thank the reviewer for this insightful comment. We deeply apologize for the confusion caused by our insufficiently clear presentation and commit to clarifying these issues in the revised manuscript.

Our contribution goes beyond representation construction. IntentFB integrates three complementary stages: a frozen MLLM generates sample-level semantic Cards without target category names; IntentForge discovers a consistent anonymous training partition through semantic consensus; and IntentBridge transfers this fixed partition to unseen samples through reliability-guided, strictly inductive prediction. Unlike MCSP, which generates semantic concepts from representatives selected after initial clustering, our Cards are constructed before category discovery, enabling semantic evidence to guide the initial partition.

Regarding comparison fairness, IntentFB and UMC, MCSP, and GLAD address the same task on shared benchmark datasets. All locally rerun baselines, including UMC, use identical training/test splits and a unified evaluation protocol. Literature-reported MCSP and GLAD results retain their original settings. We acknowledge that shared benchmarks do not imply identical representation interfaces: UMC and MCSP use pre-extracted modality features, and directly replacing them with Cards would change their original mechanisms. A common-Card evaluation would therefore require adapted implementations rather than native-system comparisons. We also acknowledge that MCSP's code is publicly available and will correct the outdated statement in our manuscript.

Crucially, Table 3 directly isolates the contributions of Forge and Bridge beyond the semantic frontend. We fix cached Cards, BGE-M3 features, context, input budgets, preprocessing, data splits, category counts, and five matched seeds. Compared with K-means plus nearest-centroid assignment (49.84%/32.09% ACC), the complete framework achieves 62.92%/47.70% on MIntRec/MELD-DA, improving ACC by 13.08/15.61 points. Under nearest-centroid assignment, Forge contributes 11.28/14.10 points; with Forge fixed, Bridge further contributes 1.80/1.51 points. Table 4 additionally validates Bridge under identical Forge partitions.

Thus, Table 2 demonstrates complete-framework performance, while Tables 3–4 provide controlled evidence that Forge and Bridge contribute additional improvements beyond the shared semantic frontend. We do not interpret native-system comparisons as representation-controlled evaluations.

We sincerely apologize again for the insufficient clarification. We commit to explicitly explaining the shared data splits, comparison protocols, baseline-result provenance, and module-level contributions in the revised manuscript and future versions. We are truly grateful for the reviewer's constructive feedback, which will help strengthen the clarity and rigor of our work.

AuthorsOpenReview ↗

Answer to Question 2

We sincerely thank the reviewer for this thoughtful and constructive observation. We fully agree that higher ACC alone does not establish more complete recovery of semantic categories, and we appreciate the reviewer's careful attention to the limitations revealed by our results. This feedback helps us clarify the scope of our claims and improve the presentation of our findings. (Our Table 2, p.8; Figure 4 and Section 5, p.9.)

The motivating observation in Table 1 and Appendix F concerns the tested direct-MLLM category-generation procedure on MIntRec, which produces duplicated categories and highly concentrated assignments. (Our Figure 1 and Table 1, p.2; Appendix F, p.20.) This motivates separating sample-level semantic description from dataset-level category formation. We agree that IntentFB does not necessarily eliminate semantic merging or recover every ground-truth category across all datasets. In particular, producing KK nonempty clusters ensures a valid partition but does not guarantee recovery of all KK semantic categories. (Our Section 3.2, p.5; Appendix B.1, p.14; Rosenberg and Hirschberg, 2007, Section 2.)

We particularly appreciate the reviewer for highlighting the mixed experimental results. On MELD-DA, IntentFB achieves higher ACC than GLAD (47.70% versus 37.79%), while Figure 4 reports lower macro-recall. On IEMOCAP-DA, the ACC improvement is smaller (39.49% versus 38.26%), accompanied by lower macro-recall, ARI (20.11 versus 24.84), NMI (24.55 versus 28.22), and FMI (37.05 versus 37.37). These findings indicate that improvements in ACC do not necessarily translate into uniformly better category recovery or overall partition quality. We agree that this important distinction deserves greater emphasis in the main text. (Our Section 4.2, p.7; Table 2 and Section 4.4, p.8; Figure 4, p.9.)

ACC weights each category's recall by its sample frequency, whereas macro-recall gives categories equal weight. (Confusion-matrix formulation: Narasimhan et al., 2015, Section 2.) This explains how higher ACC can coexist with lower macro-recall, while the latter remains an important indication of uneven category-level performance. Furthermore, the lower ARI, NMI, and FMI on IEMOCAP-DA suggest additional limitations in partition quality. A more detailed attribution of these differences would require matched class-level predictions.

Inspection of available IEMOCAP utterances and Card records suggests one possible challenge for our description-based pipeline. Short responses such as “yeah” can express different conversational functions depending on context and vocal delivery. (Shang et al., 2020, Section 1; Saha et al., 2020, Sections 1 and 3.2–3.4) In the inspected records, responses associated with different dialogue-act labels sometimes receive the same broad description, “to confirm understanding.” [E1] Although other Card fields may preserve additional evidence, the intent description itself may not sufficiently distinguish these functions.

Such semantic ambiguity may affect neighborhood construction and subsequently propagate into the discovered partition and unseen-sample assignments. (Our Theorem 1, p.5; Section 3.3, pp.5–6; Appendix B.1, p.14; Appendix I.5 and Figure 10, pp.26–27.) We regard this as a plausible failure mechanism suggested by qualitative observations, and we appreciate the reviewer's comment for motivating a clearer discussion of this limitation.

Overall, IntentFB connects semantic description, category discovery, and strictly inductive prediction for unseen samples. Our results demonstrate dataset-dependent benefits, with the strongest overall clustering performance on MIntRec, while also revealing limitations in fine-grained semantic discrimination and balanced category recovery. (Our Table 2, p.8; Appendix I.2 and Figure 7, p.23.)

We sincerely thank the reviewer again for identifying this important issue. We apologize that our original wording may have suggested a broader claim than the evidence supports. We commit to explicitly clarifying these limitations in the revised main text, particularly in Sections 4.2, 4.4, and 5. Specifically, we will qualify “effectively addressing category collapse” to refer to the failures observed in our motivating direct-MLLM diagnostic, and clearly acknowledge that semantic merging, uneven category recovery, and dataset-dependent performance remain limitations of the current framework.

We are truly grateful for the reviewer's valuable feedback, which will help us present our contributions and limitations more accurately and transparently.

AuthorsOpenReview ↗

Answer to Questions 3 and 4

We sincerely thank the reviewer for raising this important question and for the opportunity to clarify our hyperparameter development procedure. The shared hyperparameters in Table 6 were developed through a three-stage process aimed at identifying a unified configuration across MIntRec, MELD-DA, and IEMOCAP-DA, rather than independently optimizing each dataset. Hungarian-aligned ACC was the primary development criterion, while ARI, NMI, and FMI served as complementary diagnostics of clustering quality.

Stage 1: Initial configuration development. We first explored candidate configurations using MIntRec training data, drawn from the official training partition containing 1,779 samples. This stage focused on semantic view construction, anonymous partition stability, and partition-transfer strategies. In particular, we examined fixed versus dynamically updated pseudo-partitions, reliability-guided supervision, and student–EMA learning. Candidate configurations were compared using training-label-informed ACC, with ARI, NMI, and FMI as additional diagnostics. These experiments informed the adoption of fixed anonymous partitions and reliability-guided partition transfer.

Stage 2: Cross-dataset assessment. We subsequently extended the assessment to the MELD-DA and IEMOCAP-DA training data, whose official training partitions contain 7,990 and 7,532 samples, respectively. The same evaluation criteria were applied to examine the behavior of candidate settings across different dataset scales and intent distributions. We also investigated alternative multimodal representation strategies, including temporal feature aggregation and learnable audio–visual sequence alignment, to better integrate acoustic and visual information.

Stage 3: Joint refinement and shared configuration selection. Finally, candidate adjustments were reviewed jointly using training data from all three benchmarks, with ACC prioritized and ARI, NMI, and FMI providing supporting evidence. Rather than selecting separate parameter combinations for individual datasets, we focused on their overall behavior under a common configuration. This process established the shared hyperparameters reported in Table 6.

Key hyperparameter settings. The discovery, sequence, and transfer representation dimensions were all set to 256. The neighborhood sizes for view weighting and graph construction were 20 and 30, respectively, with a reserved text-view weight of 0.42. Reliability-guided curriculum supervision expanded from 20% to 100% over 12 epochs, with EMA momentum set to 0.995. The sequence alignment branch used 20 epochs, a learning rate of 3 × 10⁻⁴, and batch size 64, while the student–EMA branch used 40 epochs, a learning rate of 3 × 10⁻³, and batch size 256. Frozen exemplar retrieval used 16 neighbors with temperature 0.10. These settings were shared across all three benchmarks, with the benchmark-defined number of categories K being 20 for MIntRec and 12 for both MELD-DA and IEMOCAP-DA.

We distinguish unsupervised model fitting from label-informed hyperparameter development. Training annotations were used to evaluate and compare candidate configurations through clustering metrics, but were not provided to semantic Card generation, anonymous partition discovery, or the model-fitting objectives. The development assessments used training data rather than an independently held-out validation partition. Following configuration selection, the final models were fitted on the full official training partitions, with data-dependent artifacts frozen before inference. The reported benchmark results follow the five-run evaluation protocol with seeds {0, 1, 2, 3, 4}, as specified in Section 4.1 and Appendix D.3.

We sincerely appreciate the reviewer's constructive feedback and will clarify the development criteria, the training data used at each stage, and the distinction between unsupervised model fitting and label-informed hyperparameter selection in the revised manuscript accompanying Table 6.

Regarding Comment 4, we sincerely apologize for the grammatical errors in our manuscript and commit to carefully correcting them, while also ensuring that our entire rebuttal is expressed with greater clarity, sincerity, and respect for the reviewers' valuable feedback.

Reviewer_2w6EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IntentForge–IntentBridge for unsupervised multimodal intent discovery. It first uses a frozen MLLM to generate semantic descriptions for each utterance. IntentForge then discovers categories through multi-view consensus clustering, prototype refinement and label propagation. IntentBridge learns to assign unseen samples to these fixed categories using reliability-guided training and retrieval. The method achieves large improvement on MIntRec, while the results on other datasets are more mixed.

Critical strengths
  1. Separating semantic description from category discovery is a reasonable idea. MLLMs are good at understanding individual samples, but directly asking them to organize a consistent taxonomy can be problematic. This motivation makes sense to me.

  2. The controlled ablation is useful. It shows that most improvement comes from IntentForge, while IntentBridge provides relatively small additional gain. Also, the strictly inductive setting makes the method more practical for unseen samples.

Critical weaknesses
  1. I am still not very clear about the actual contribution of IntentForge. Even using MLLM-generated Cards with BGE-M3 and simple K-means already gives quite strong results. IntentForge further improves performance, but it combines several existing techniques, including spectral clustering, prototype refinement and label propagation. How much improvement really comes from the proposed consensus mechanism? Or just from applying stronger clustering techniques on good semantic embeddings? The comparison with previous methods also involves different pretrained models and semantic inputs. So the large overall improvement is encouraging, but its source is not fully clear.

  2. Some results seem inconsistent with the motivation of addressing category collapse. On MELD-DA and IEMOCAP-DA, ACC improves, but macro recall decreases, and ARI is worse or almost unchanged. This makes me wonder whether the method mainly improves frequent categories while sacrificing some minority ones. If so, the discovered taxonomy may not actually be better.

Questions for authors
  1. With the same Card embeddings, how does plain spectral clustering perform? What are the individual contributions of prototype refinement and label propagation?

  2. What happens if replacing MLLM-generated Cards with transcript embeddings while keeping the remaining pipeline unchanged? I want to understand how much the semantic generation actually contributes.

  3. Can you show the predicted cluster size distribution and per-class recall? Especially for MELD-DA and IEMOCAP-DA, are the ACC improvements mainly from frequent categories?

5 comments
AuthorsOpenReview ↗

Answer to Question 1

We sincerely thank the reviewer for carefully reading our paper and providing this valuable suggestion. Following the reviewer's recommendation, we conducted a controlled 2×2 factorial ablation to evaluate plain spectral clustering, prototype refinement, label propagation, and their combination.

For each dataset and seed, all configurations share identical frozen Card embeddings, consensus graphs, spectral initialization, and clustering settings. Propagation anchors are selected from the initial spectral partition or the refined partition, depending on the configuration.

Table R1. Component ablation (%, mean ± SD over five seeds).

DatasetConfigurationACCARINMIFMI
MIntRecSpectral only45.32 ± 0.4723.49 ± 0.6844.73 ± 0.3528.51 ± 0.60
+ Refinement49.97 ± 0.2029.68 ± 0.2048.10 ± 0.3034.18 ± 0.19
+ Propagation48.10 ± 0.6026.63 ± 0.9347.97 ± 0.5331.51 ± 0.83
+ Both49.76 ± 0.1928.73 ± 0.2549.12 ± 0.1133.43 ± 0.23
MELD-DASpectral only35.16 ± 0.005.79 ± 0.0015.32 ± 0.0035.73 ± 0.00
+ Refinement27.76 ± 0.168.72 ± 0.0416.60 ± 0.0120.62 ± 0.05
+ Propagation36.63 ± 0.036.74 ± 0.0217.55 ± 0.0438.05 ± 0.01
+ Both30.21 ± 0.2310.07 ± 0.1118.27 ± 0.0423.18 ± 0.13
IEMOCAP-DASpectral only34.28 ± 0.048.05 ± 0.0216.99 ± 0.0228.76 ± 0.02
+ Refinement27.37 ± 0.038.20 ± 0.0214.57 ± 0.0620.55 ± 0.02
+ Propagation35.25 ± 0.127.38 ± 0.0618.89 ± 0.0131.47 ± 0.06
+ Both30.59 ± 0.129.70 ± 0.1017.82 ± 0.0923.06 ± 0.08

Prototype refinement improves MIntRec ACC, ARI, and FMI by 4.65, 6.19, and 5.67 percentage points, respectively. On MELD-DA and IEMOCAP-DA, the combined configuration increases ARI over plain spectral clustering by 4.28 and 1.65 points, respectively, while exhibiting dataset-dependent ACC trade-offs.

Label propagation consistently improves ACC when applied directly to spectral clustering, yielding gains of 2.78, 1.47, and 0.97 points on MIntRec, MELD-DA, and IEMOCAP-DA, respectively. The corresponding NMI improvements are 3.24, 2.23, and 1.90 points, demonstrating the value of neighborhood information.

To further quantify individual contributions, we calculate factorial marginal effects and interactions:

Table R2. Factorial ACC effects (percentage points, mean ± SD).

DatasetRefinementPropagationInteraction
MIntRec+3.16 ± 0.42+1.28 ± 0.23−2.99 ± 0.49
MELD-DA−6.91 ± 0.18+1.96 ± 0.05+0.98 ± 0.10
IEMOCAP-DA−5.79 ± 0.06+2.09 ± 0.08+2.25 ± 0.18

These results highlight complementary but non-additive component effects. Propagation provides positive marginal ACC contributions across all datasets, while refinement exhibits dataset- and metric-dependent behavior. Together, the components offer distinct mechanisms for prototype-based partition adjustment and neighborhood-based assignment refinement.

These findings complement our main-paper Table 3, where complete IntentForge improves ACC by 11.28 points on MIntRec and 14.10 points on MELD-DA under controlled shared-Card conditions. The full IntentForge–IntentBridge framework achieves 62.92%, 47.70%, and 39.49% ACC on the three official test sets.

The new component analysis evaluates frozen training partitions from cached Card cohorts, whereas the main paper evaluates the complete framework on official test splits. We report them separately to maintain a clear evaluation scope.

We will include the complete 2×2 ablation, factorial analysis, and experimental details in the revised Appendix. We thank the reviewer again for this constructive suggestion, which enables a more detailed evaluation of IntentForge's component contributions.

AuthorsOpenReview ↗

Answer to Question 2

Q2. What happens if replacing MLLM-generated Cards with transcript embeddings while keeping the remaining pipeline unchanged? How much does semantic generation actually contribute?

We sincerely thank the reviewer for this valuable suggestion and apologize for not including a direct transcript-based comparison in the original manuscript. Following the reviewer's recommendation, we conducted additional experiments to evaluate the role of MLLM-generated Cards in our framework. We will include these results and the corresponding experimental details in the revised Appendix.

First, we evaluate a transcript-only baseline (raw_none), which uses only the dataset-provided utterance transcripts, without MLLM-generated Cards or additional audio/video sequence features. We also examine a matched representation comparison in which the downstream sequence branch, sample IDs, random seeds, number of clusters, and training configuration are retained.

Table R3. Transcript-based and Card-based comparisons (test ACC, %, mean ± SD over five seeds).

DatasetTranscript only (raw_none)Transcript with sequence branchFull Card with sequence branch
MIntRec47.82 ± 1.2647.36 ± 1.8862.92 ± 1.00
MELD-DA26.72 ± 2.3926.59 ± 1.3647.70 ± 1.10
IEMOCAP-DA24.72 ± 1.0924.04 ± 0.7939.49 ± 0.18

The transcript-only baseline demonstrates that original text provides a meaningful semantic foundation. Notably, adding the sequence branch to transcript embeddings alone does not improve ACC in this setting. More importantly, when the downstream sequence branch and training configuration are kept fixed, the full Card-derived representation path outperforms the transcript representation by 15.56, 21.11, and 15.45 percentage points on MIntRec, MELD-DA, and IEMOCAP-DA, respectively.

These findings highlight the role of Cards in our framework. Unlike raw transcript embeddings, which primarily encode the linguistic content of an utterance, MLLM-generated Cards organize source-grounded descriptions and multimodal evidence into complementary semantic views. IntentForge exploits agreement among these views to discover a coherent training partition. Thus, the contribution of Card generation extends beyond producing a single semantic embedding: it enables the structured multi-view semantic construction central to our approach.

This design is also motivated by previous findings. UMC (Zhang et al., ACL 2024, Table 2) shows that incorporating audio and visual information improves ACC over its text-only counterpart on MIntRec (42.46% to 43.73%), MELD-DA (33.40% to 35.31%), and IEMOCAP-DA (32.76% to 33.87%). Similarly, TAPE (He et al., ICLR 2024) demonstrates the potential of LLM-generated explanations as additional features for downstream representation learning. These studies provide complementary motivation for enriching original text with additional semantic information.

Building on this principle, our framework uses MLLMs to construct structured semantic evidence before category discovery, rather than relying on direct MLLM category assignment. The complete IntentForge–IntentBridge framework achieves 62.92%, 47.70%, and 39.49% ACC on the official test sets, outperforming UMC in the main-paper benchmark comparison.

We note that the matched supplemental comparison measures the contribution of the full Card-derived representation path, including its additional semantic and evidence views, rather than the isolated effect of generated description text.

We will add the transcript-only baseline, matched representation comparison, and detailed experimental protocol to the Appendix. We thank the reviewer again for helping us clarify the contribution of MLLM-generated Cards to multi-view semantic construction and intent discovery.

AuthorsOpenReview ↗

Answer to Question 3

We sincerely thank the reviewer for this valuable suggestion. Examining predicted cluster sizes and per-class recall provides insights beyond overall accuracy. We will incorporate these analyses into the revised Appendix.

Both datasets contain 12 dialogue-act categories: Acknowledge (a), Agreement (ag), Answer (ans), Apology (ap), Backchannel (b), Command (c), Disagreement (dag), Greeting (g), Statement-Opinion (o), Others (oth), Question (q), and Statement-Non-Opinion (s).

Our main paper presents row-normalized confusion matrices for IntentFB and GLAD in Figure 4. We summarize the corresponding diagonal recall values below.

Table R4. Per-class recall on MELD-DA and IEMOCAP-DA (%, approximately).

Dialogue actMELD GLADMELD IntentFBIEMOCAP GLADIEMOCAP IntentFB
Acknowledge (a)1601557
Agreement (ag)1435170
Answer (ans)59162858
Apology (ap)13364942
Backchannel (b)5902211
Command (c)1937700
Disagreement (dag)5929828
Greeting (g)47556530
Statement-Opinion (o)202286
Others (oth)26795084
Question (q)61674563
Statement-Non-Opinion (s)5440463

Approximate integer percentages from Figure 4; bold indicates higher recall.

First, IntentFB demonstrates clear improvements in several communicative categories. On MELD-DA, our method improves recall for Agreement (14%→35%), Apology (13%→36%), Command (19%→37%), Greeting (47%→55%), Others (26%→79%), and Question (61%→67%). On IEMOCAP-DA, notable gains occur for Acknowledge (15%→57%), Answer (28%→58%), Disagreement (8%→28%), Others (50%→84%), and Question (45%→63%). The consistent improvements for Others and Question across both datasets illustrate the effectiveness of our Card-based semantic consensus framework for certain communicative functions.

Second, IntentFB and GLAD exhibit complementary strengths. Our method improves overall ACC from 37.79% to 47.70% (+9.91 points) on MELD-DA and from 38.26% to 39.49% (+1.23 points) on IEMOCAP-DA. Meanwhile, GLAD achieves higher macro recall (37.25% versus 32.91% on MELD-DA; 36.92% versus 31.69% on IEMOCAP-DA) and performs better on several categories. These findings highlight their complementary properties: IntentFB emphasizes structured multimodal semantic consensus and inductive partition transfer, while GLAD achieves stronger recall balance across certain categories.

Third, regarding class-frequency effects, higher overall ACC alongside lower macro recall indicates heterogeneous class-level improvements, potentially influenced by class frequencies. However, per-class recall alone cannot quantify the contribution of frequent categories. We will therefore add predicted cluster-size distributions, class supports, and per-class recall statistics from the official evaluation predictions to the revised Appendix.

These findings also suggest a promising direction for future research. We plan to integrate IntentFB's structured semantic guidance with UMC's contrastive representation learning and reliable-sample selection strategies. This combination may improve discrimination among closely related dialogue acts while preserving multimodal semantic information, enabling more balanced performance across categories.

Reviewer_aLLcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper proposes IntentForge–IntentBridge (IntentFB), a strictly inductive framework for unsupervised multimodal intent discovery. IntentForge uses MLLM-generated semantic Cards and consensus clustering to discover intent categories, while IntentBridge transfers the discovered partitions to unseen samples through reliability-guided learning and retrieval. Experiments on three benchmarks demonstrate substantial improvements on MIntRec and MELD-DA, with more limited and mixed gains on IEMOCAP-DA. The rebuttal provides additional controlled ablations, alternative Card generator evaluations, and computational cost analyses, further clarifying the contributions and limitations of the framework.

Critical strengths
  1. The paper identifies category collapse in direct MLLM-based intent discovery and proposes a reasonable separation between semantic description, category discovery, and inductive prediction. This provides a meaningful perspective on MLLM-assisted clustering.

  2. The authors provide additional controlled experiments that clarify the contributions of the semantic frontend and the IntentForge–IntentBridge modules. The results demonstrate substantial improvements from the proposed framework on MIntRec and MELD-DA.

  3. The rebuttal includes evaluations with alternative Card generators and computational cost analyses. These experiments provide useful evidence of generator compatibility and reveal that the semantic quality of generated Cards can substantially affect downstream clustering performance.

Critical weaknesses
  1. The contribution of the semantic frontend is better supported, but the current ablations still combine MLLM-generated descriptions and multimodal evidence. The individual contributions of different modalities and generated semantic descriptions remain unclear. In addition, the full-method ACC values reported in the rebuttal's controlled frontend ablation differ from those in the main benchmark results, and the corresponding experimental settings should be clarified.

  2. The alternative Card generator experiments are conducted only on MIntRec. Although the results demonstrate compatibility with different generators, performance varies considerably across models (62.95% with MiniCPM, 60.68% with Qwen3-VL-8B, and 56.14% with Gemma-4-12B). The experiments therefore provide limited evidence of cross-dataset robustness and sensitivity to generator choice. The use of a fixed seed setting also makes the stability of these comparisons unclear.

  3. The concerns about category-level performance remain. IntentFB achieves higher ACC but lower macro recall on both dialogue-act datasets, with particularly weak ARI, NMI, and FMI on IEMOCAP-DA. The additional per-class results reveal that some categories have zero or very low recall. Although the authors acknowledge these limitations, the proposed explanation based on insufficient pragmatic information in Cards remains primarily qualitative.

Questions for authors
  1. How much improvement comes specifically from the semantic Cards and multimodal information? Can the authors provide corresponding controlled ablations?

  2. Can the authors report end-to-end results with alternative Card generators, together with training costs and per-sample inference latency?

3 comments
AuthorsOpenReview ↗

Answer to Question 1

We thank the reviewer for carefully reading our paper and raising this valuable question. We agree that the roles of semantic Cards, multimodal information, and the subsequent clustering and partition-transfer mechanisms should be distinguished. We clarify our input processing and representation, discuss our motivation, and report a controlled ablation study.

1. Input processing and semantic representation

UMC’s official implementation uses swin_feats.pkl and wavlm_feats.pkl to store offline-extracted audio and video features. Video frames are processed by a pretrained Swin Transformer, yielding a sequence of length LVL_V with 1,024 dimensions per timestep. Audio waveforms are processed by pretrained WavLM, yielding a sequence of length LAL_A with 768 dimensions per timestep. The features are grouped by sample ID and serialized as .pkl files. During training, they are truncated or padded, passed through linear layers and Transformer encoders, and fused with BERT text representations for contrastive learning and clustering (UMC paper; official implementation).

2. Motivation and method

UMC shows that text provides a strong base representation, while audio and video offer additional information. Compared with its text-only version, UMC’s three-modal version raises ACC from 42.00% to 44.00% on MIntRec (+2.00 percentage points), from 32.54% to 35.58% on MELD-DA (+3.04 points), and from 32.40% to 36.36% on IEMOCAP-DA (+3.96 points) (UMC paper, Table 2). TAPE (He et al., ICLR 2024) also reports that LLM-generated explanations can improve downstream representations and classification in specific settings (TAPE). These findings motivate our exploration of explicit semantic descriptions for organizing multimodal evidence.

Our frontend organizes each transcript, corresponding audio or video path, and sample ID into a structured JSON record according to a predefined modality configuration. It reads the text, samples video frames from .mp4 files under a fixed budget, decodes audio into waveforms, and sends the inputs to a frozen multimodal model. The model generates sample-level descriptions of wording, observable behavior, or audible expression, forming semantic Cards. BGE-M3 encodes the Cards; IntentForge discovers anonymous training partitions, and IntentBridge predicts for unseen samples.

Unlike direct fusion of pre-extracted audio and video features, our method uses an LLM to convert heterogeneous multimodal evidence into explicit, unified, and comparable semantic descriptions before intent discovery. Since methods differ in their inputs, feature extraction, representations, and downstream learning, comparisons of overall performance cannot isolate the contributions of Cards and multimodal information. We therefore conduct controlled comparisons within our framework.

3. Controlled ablation

We compare the full method with a variant that replaces the generated multimodal Cards with the original transcript, while retaining BGE-M3, IntentForge, and IntentBridge. Both experiments use the same effective samples, paired training configuration, and five predefined random seeds.

ACC (%), mean ± standard deviation

DatasetFull methodWithout semantic frontendImprovement (points)
MIntRec60.77 ± 0.5246.05 ± 1.09+14.72
MELD-DA45.94 ± 0.8126.50 ± 0.88+19.44
IEMOCAP-DA39.19 ± 0.4037.30 ± 6.06+1.89

The full method improves ACC by 14.72 and 19.44 points on MIntRec and MELD-DA, respectively. On IEMOCAP-DA, it also has a higher mean ACC, though the gain is smaller and the variant without the frontend has greater run-to-run variability.

With the downstream pipeline held constant, these results show that the multimodal semantic frontend improves intent discovery on some datasets. They provide further evidence for organizing multimodal evidence through explicit semantic descriptions and clarify the scope of the conclusions supported by our results. We thank the reviewer again for this constructive suggestion.

AuthorsOpenReview ↗

Answer to Question 2

We thank the reviewer for this valuable suggestion. To evaluate the robustness and computational efficiency of our framework, we compared alternative Card generators on MIntRec and analyzed their downstream performance, computational costs, and inference latency.

1. Performance with alternative Card generators

We evaluated MiniCPM, Qwen3-VL-8B, and Gemma-4-12B using a common Card-generation prompt and the same experimental configuration as reported in our paper. The new experiment used the same configuration as in the paper, only replacing the card generator with our model.The reported ACC values are averaged over five runs. The previous MiniCPM result is included as a reference.

Card generatorACC (%)
MiniCPM (reference)62.95
Qwen3-VL-8B60.68
Gemma-4-12B56.14

These results demonstrate that our framework supports different Card generators, although downstream performance varies with the generator. Importantly, the reported ACC measures Hungarian-aligned downstream clustering performance rather than the intrinsic correctness of individual Cards.

2. Analysis of generator differences

To investigate the performance gap, we conducted controlled field-swapping experiments between Qwen and Gemma, keeping the data, training configuration, and evaluation protocol unchanged.

ACT + INTENTVISACC (%)
QwenQwen60.68
QwenGemma62.05
GemmaQwen58.18
GemmaGemma56.14

Interestingly, replacing Qwen's visual descriptions with Gemma's improves ACC from 60.68% to 62.05%. Thus, Gemma's lower overall performance cannot be attributed simply to weaker visual descriptions.

Our qualitative analysis identifies two possible factors. First, semantically appropriate Cards may still form clusters that do not align with the dataset's category boundaries. For example, Gemma correctly describes some Praise samples as compliments, but these samples are distributed across multiple clusters, causing errors under one-to-one Hungarian matching. Second, some Cards omit task-relevant pragmatic information. For instance, Gemma describes "sorry to interrupt" primarily as an interruption without preserving its apologetic function.

These findings suggest that performance depends on both the semantic content of Cards and their compatibility with downstream representation learning and clustering. Although ACC is averaged over five runs, we do not claim a stable generator ranking across different random seeds, since the experiments use a fixed seed setting (42).

3. Computational costs and inference latency

We additionally recorded the wall-clock time for training-set Card generation, 40-epoch downstream training, and test-set Card generation on MIntRec.

GeneratorTrain Cards40 epochsTest CardsTotal
MiniCPM45m21s25.47s8m39s54m25s
Qwen3-VL-8B60m19s26.88s16m24s77m10s
Gemma-4-12B40m39s27.11s8m04s49m10s

The corresponding average test-set Card-generation times, calculated over 445 samples, are 1.17 s/sample for MiniCPM, 2.21 s/sample for Qwen, and 1.09 s/sample for Gemma. These are observed generation-stage averages rather than isolated single-request or full-pipeline inference latencies.

All Card generators are frozen, so no generator fine-tuning is required. The measured downstream training takes approximately 25–27 seconds for 40 epochs, whereas Card generation accounts for most of the recorded computation time.

For context, our recorded UMC run on MIntRec required approximately 3 hours 46 minutes, with around 12 GB GPU memory usage. However, differences in preprocessing, runtime accounting, and computational settings prevent a strict efficiency comparison. Comparable full-run peak GPU memory measurements were not available for the three Card generators, so we avoid drawing conclusions about their relative memory requirements.

Overall, these experiments demonstrate the applicability of our framework to alternative Card generators, quantify the observed computational costs, and reveal how generator-dependent semantic representations affect downstream clustering performance. We will incorporate the additional experimental results and analyses presented in this rebuttal into the supplementary material of the revised manuscript. We thank the reviewer for this constructive suggestion.

AuthorsOpenReview ↗

Thank You for Your Thoughtful Reconsideration

We sincerely thank the reviewer for taking the time to carefully reconsider our work and for increasing the evaluation score. We truly appreciate your recognition of our contributions and the constructive feedback you have provided throughout the review process. Your insightful comments have been invaluable in helping us identify areas for improvement and strengthen the clarity and rigor of our manuscript. We will carefully incorporate your suggestions into the revised version. Thank you again for your time, thoughtful consideration, and encouragement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.