acceptodds
Under review as a conference paper at ICLR 2027

SpeechLens: Multi-Perspective Speech Clustering with Reasoning Audio-Language Models

Abstract

Speech clustering is a fundamental task for organizing rapidly growing speech collections, supporting applications such as conversational analysis and speech-driven discovery. However, existing methods rely on fixed acoustic similarity metrics or ASR-based text pipelines, limiting their ability to reorganize the same audio collection under different user-specified perspectives, especially when clustering depends on both linguistic and paralinguistic cues. We introduce speech multi-perspective clustering, where a model directly partitions speech recordings according to a natural-language perspective while inferring both the number of clusters and their assignments. To study this setting, we construct SpeechLens-Bench, a benchmark spanning multiple application domains and evaluating both in-perspective and cross-perspective generalization. We further propose SpeechLens-R1, an end-to-end large audio-language model trained with reasoning distillation and preference optimization. Experiments show that SpeechLens-R1 consistently outperforms all baselines, improving overall ARI by 12.99 points and V-measure by 11.62 points. These results demonstrate the promise of native audio-language models for flexible, perspective-conditioned structure discovery over speech collections.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio clustering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mc7sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the "speech multi-view clustering" task, which requires a model to directly cluster a collection of speech samples based on a view specified in natural language, while simultaneously inferring both the number of clusters and the cluster assignments. To support this task, the authors developed the SpeechLens-Bench benchmark, which spans multiple application domains and maintains a strict distinction between in-view and cross-view generalization evaluations. Furthermore, the authors introduce SpeechLens-R1, an end-to-end large audio-language model trained via reasoning distillation and preference optimization to learn view-based clustering reasoning. Experiments demonstrate that SpeechLens-R1 significantly outperforms both ASR-based pipelines and existing native audio-language model baselines in terms of ARI and V-measure.

Critical strengths

This paper proposes the "speech multi-view clustering" task, which requires a model to directly cluster a collection of speech samples based on a view specified in natural language, while simultaneously inferring both the number of clusters and the cluster assignments. To support this task, the authors developed the SpeechLens-Bench benchmark, which spans multiple application domains and maintains a strict distinction between in-view and cross-view generalization evaluations. Furthermore, the authors introduce SpeechLens-R1, an end-to-end large audio-language model trained via reasoning distillation and preference optimization to learn view-based clustering reasoning. Experiments demonstrate that SpeechLens-R1 significantly outperforms both ASR-based pipelines and existing native audio-language model baselines in terms of ARI and V-measure.

Critical weaknesses

1.The paper claims to outperform the ASR+LLM baseline, but it does not specify whether an optimal Chain-of-Thought (CoT) prompt was designed for the LLM baseline (e.g., GPT-4o) or if sufficient engineering optimization was applied. Furthermore, when an end-to-end LALM processes multiple long audio segments, the computational cost of its autoregressive generation and cross-attention is extremely high. The paper downplays these substantial training and inference costs, casting doubt on the method's applicability to large-scale, real-world audio repositories. 2.The model is required to simultaneously infer the number of clusters (K) and the cluster assignments. If K is inferred incorrectly (e.g., resulting in over-merging or over-splitting), the entire clustering result becomes completely invalid. The paper lacks an independent evaluation of the inference accuracy for K, and fails to analyze how errors in estimating K cascade to impact the final ARI and V-measure metrics.

Questions for authors

1.Please provide the complete prompt used for the ASR+LLM baseline (e.g., GPT-4o). Have you tried having the LLM output its reasoning process (Chain-of-Thought, CoT) before performing the clustering? Does the current baseline performance represent the true performance upper bound of this paradigm? 2.If the correct number of clusters K is directly provided to the model via an oracle during inference, how much would the performance of SpeechLens-R1 improve? This would help distinguish whether the model's performance bottleneck stems from its "feature representation and reasoning capabilities" or merely from failures in "cluster number inference." 3.What are the average inference time and memory consumption for a single forward pass of SpeechLens-R1 when processing an instance containing 10 audio segments? How significant is the efficiency gap compared to an ASR+Embedding-based pipeline? For larger-scale audio collections, are there any feasible chunking or hierarchical clustering strategies?

3 comments

We thank the reviewer for recognizing the task formulation, the distinction between held-in and held-out perspectives, and the improvements over both ASR-based and native audio-language model baselines.

Q1: We would like to clarify that the complete ASR+LLM prompting template is already provided in Appendix D (page 31). The prompt explicitly instructs the models to:

  1. Examine the indexed ASR transcripts under the given clustering perspective.
  2. Reason about the appropriate number of clusters.
  3. Assign every item to a cluster.
  4. Verify that all indices are covered exactly once.
  5. Produce separate structured reasoning and final-answer fields.

The template also includes the clustering perspective and the ASR-transcribed contents of all indexed audio segments. Therefore, the reported baseline is not a simple direct-labeling prompt; it already incorporates a multi-step, reasoning-oriented clustering protocol with explicit output constraints.

Importantly, there is no established prompting protocol that guarantees the theoretical upper bound of the ASR+LLM paradigm. Our comparison therefore focuses on a clearly documented, reasoning-oriented baseline rather than claiming prompt-level optimality.

In addition, the central results do not rely exclusively on GPT-4o. SpeechLens-R1 is also evaluated against multiple native audio-language models. SpeechLens-R1 outperforms the ASR+GPT-4o baseline by 16.39 ARI points and 13.01 V-measure points. It also surpasses the strongest native audio-language baseline, GPT-audio-1.5, by 12.99 ARI points and 11.62 V-measure points.

These comparisons, together with the answer-only SFT and RD/DPO ablations in Table 4, establish the effectiveness of task-specific reasoning and preference optimization for the proposed clustering objective.

We will make reference to the complete GPT-4o prompt more explicitly in the main experimental section to improve clarity.

Q2: We would like to correct an important premise in this concern: an incorrect number of clusters does not make the entire clustering result invalid under either ARI or V-measure.

Both metrics compare partition structures and assign partial credit to predictions that preserve some correct groupings, even when the predicted cluster count differs from the ground truth.

For example, consider a gold partition [[1,2],[3,4]][[1,2],[3,4]] with K=2K=2 and a predicted partition [[1,2],[3],[4]][[1,2],[3],[4]] with K=3K=3. Despite the cluster-count mismatch, the prediction obtains ARI=57.14ARI = 57.14 and V−measure=80.00V-measure = 80.00.

Therefore, the two reported evaluation metrics already reflect the joint effects of cluster-count errors, over-merging, over-splitting, and incorrect item assignments. A KK mismatch is a clustering error, not necessarily a structurally invalid prediction.

Moreover, jointly inferring KK and the assignments is a defining property of our proposed task (Section 3.1). Providing oracle KK at inference time would introduce privileged information unavailable in the intended setting and would evaluate a different, easier problem.

Our baseline protocol already illustrates this distinction: K-Means and GMM are supplied with the gold number of clusters, whereas SpeechLens-R1 must infer it. Despite this more demanding prediction setting, SpeechLens-R1 achieves the strongest overall ARI and V-measure.

Finally, our preference-optimization pipeline explicitly uses valid but incorrect partitions, including over-merging and over-splitting errors, as hard-negative supervision. The method therefore directly targets structural clustering decisions, rather than merely learning the required output format.

An oracle-KK diagnostic, therefore, is not required to interpret the reported ARI and V-measure results or to establish performance on the original joint-inference task.

Q3: To provide quantitative evidence, we conducted additional inference profiling on a single NVIDIA H200 NVL GPU, comparing our audio-native clustering approach against two complete ASR-based pipelines: Whisper large-v2 + MiniLM + GMM and Whisper large-v2 + Instructor + GMM.

We evaluated two representative test instances, including an approximately mean-duration sample and the longest-duration test sample. Each configuration was measured over three runs, excluding model loading time.

Test instanceSpeechLensASR + MiniLM + GMMASR + Instructor + GMM
mean-duration4.48s22.99s23.02s
longest-duration6.62s33.97s34.01s

The SpeechLens measurements include audio preprocessing and generation, while the ASR baselines include full-audio transcription, embedding extraction, and GMM clustering. SpeechLens achieves approximately 5.1× lower processing latency on both instances. Regarding memory consumption, SpeechLens requires 17.32–18.14 GB of peak allocated GPU memory, compared with 6.34–6.35 GB for the ASR-based pipelines. Thus, our approach provides a substantial latency advantage on the measured instances, with a higher GPU memory footprint.

Importantly, SpeechLens jointly processes multiple audio segments and generates a complete clustering partition in a single inference request, avoiding separate full-transcript generation and downstream embedding/clustering stages. These measurements demonstrate the practical feasibility of multi-minute, perspective-conditioned audio clustering within seconds on a single GPU.

For larger repositories, SpeechLens can naturally serve as a perspective-conditioned grouping component within hierarchical clustering or retrieval-based workflows. Recordings can first be organized into subsets, followed by cross-group comparison and iterative refinement under the same user-specified perspective.

Overall, the additional measurements demonstrate a favorable latency–memory trade-off for joint audio-native clustering, complementing the substantial accuracy improvements reported in our main experiments.

Reviewer_3cHxOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces perspective-conditioned speech clustering, where a model groups recordings according to a natural-language instruction while inferring the number of clusters. It constructs SpeechLens-Bench using synthesized speech derived from four source corpora, with evaluation splits covering familiar and held-out perspectives. SpeechLens-R1 combines reasoning distillation and preference optimization, outperforming the reported baselines on this benchmark.

Critical strengths
  • S1: The formulation provides a unified interface for clustering speech according to linguistic or paralinguistic criteria. The held-out-perspective evaluation is a useful step beyond testing only familiar clustering instructions.
  • S2: Table 4 includes answer-only supervised fine-tuning and separate RD/DPO ablations. These comparisons provide evidence that the combined training procedure improves performance beyond basic task adaptation on this benchmark.
Critical weaknesses
  • W1: Generalization beyond the speech synthesis pipeline is untested. Appendix A.7 specifies a single TTS system with nine preset voices, instruction-controlled emotions, and dialogues assembled using fixed inter-turn silences. Although L1/L2 contain unseen recordings, they retain the same synthesis pipeline, and no speaker-disjoint evaluation is reported. Hence, the results cannot distinguish robust speech understanding from adaptation to generator-specific acoustic patterns, particularly for emotion and speaker-related perspectives.
  • W2: The practical significance of the benchmark remains insufficiently established. The introduction motivates organizing growing speech collections, but evaluation is restricted to constructed episodes containing 4-10 clips, with perspectives and categories designed through the benchmark pipeline. No representative real-world workflow or evaluation on larger collections demonstrates that these gains translate into useful speech organization. Appendix H acknowledges these limitations, but the current evidence still leaves the contribution primarily as a controlled proof of concept.
Questions for authors
  • Q1: Do the improvements persist on naturally recorded speech? Evaluation using a different TTS system and disjoint voices would also help determine how much performance depends on the synthesis pipeline, although it would not establish real-speech transfer by itself.
  • Q2: Can the authors demonstrate the method on a representative speech-organization workflow with independently specified user perspectives, and report how clustering quality and inference cost change as collection size increases? This would help establish the practical value beyond small constructed episodes.
9 comments

We thank the reviewer for recognizing the unified task formulation, held-out-perspective evaluation, and the effectiveness of our RD/DPO training strategy. We would like to clarify the scientific objectives and evaluation scope of our work.

Q1: Controlled speech synthesis is a deliberate methodological choice motivated by the unique requirements of multi-perspective speech clustering. To the best of our knowledge, existing real-speech benchmarks do not provide the comprehensive annotations required for this task: multiple linguistic and paralinguistic clustering perspectives over the same speech collection, corresponding ground-truth partitions, and held-out perspective evaluations.

Controlled synthesis enables systematic evaluation with reliable ground-truth annotations and controlled acoustic factors. Specifically, we can vary emotion, speaker configuration, and background conditions while approximately balancing irrelevant attributes across semantic categories. This design reduces unintended correlations between target clustering criteria and unrelated acoustic factors, allowing us to better isolate perspective-conditioned clustering capabilities. We additionally employ transcript validation, lexical-leakage filtering, and independent human quality checks to ensure benchmark reliability.

For audio generation, we adopt Qwen3-TTS CustomVoice, a state-of-the-art speech synthesis model. Our research focuses on perspective-conditioned speech understanding and structure discovery, rather than speech synthesis fidelity.

Importantly, the purpose of SpeechLens-Bench is to evaluate perspective-conditioned clustering under controlled acoustic conditions, rather than to benchmark cross-generator acoustic robustness. The controlled assignment and balancing of paralinguistic attributes reduce unintended correlations between clustering labels and irrelevant acoustic factors.

We acknowledge that the shared TTS system does not independently establish robustness to generator-specific acoustic patterns. However, cross-perspective generalization and cross-generator acoustic robustness are distinct evaluation objectives. The absence of the latter does not invalidate the controlled comparisons or the cross-perspective generalization results established by L2.

Our current evaluation focuses on controlled synthesized speech. Generalization to naturally recorded speech constitutes an additional acoustic-transfer evaluation, distinct from the cross-perspective generalization examined in this work. We will clarify these evaluation dimensions in the revised manuscript.

Q2: We respectfully disagree that establishing a new capability through controlled evaluation constitutes insufficient scientific contribution or methodological soundness. A central objective of scientific research is to establish and systematically evaluate previously unexplored capabilities under well-defined experimental conditions. A controlled proof of concept is a scientifically meaningful contribution when supported by rigorous evaluation and reproducible empirical evidence.

Our work goes beyond a preliminary demonstration. It introduces a formally defined task, establishes a benchmark spanning four application domains and multiple linguistic and paralinguistic perspectives, evaluates generalization to unseen clustering criteria, and demonstrates substantial improvements over competing paradigms. Specifically, SpeechLens-R1 achieves the highest ARI across all 12 evaluation settings and the highest V-measure in 11 of 12 settings, improving overall ARI and V-measure over the strongest baselines by 12.99 and 11.62 points, respectively.

Our setting should not be interpreted as a trivial clustering setting. As reported in Appendix B.4, training episodes contain an average of 373.5 seconds of audio, with durations reaching approximately nine minutes. Additionally, 94.1% of training episodes contain at least one non-singleton cluster, requiring the model to identify meaningful within-cluster relationships while distinguishing different groups. Regarding practical applications, Appendix G.1 presents a representative ECHR case study in which seven spoken legal cases are organized according to their relationships with responsible actors. The model distinguishes direct State conduct, administrative responsibility, detention conditions, and judicial protection under the specified natural-language perspective. This demonstrates the intended speech-organization functionality in a concrete application-domain scenario.

Regarding inference cost, we conducted additional profiling on a single NVIDIA H200 NVL GPU, comparing SpeechLens against complete ASR + LLM pipelines. For two test instances (mean-duration and max-duration test samples), SpeechLens completed preprocessing and generation in approximately 4.47s and 6.62s, respectively. In comparison, Whisper large-v2 + MiniLM + GMM required 22.99s and 33.97s, while Whisper large-v2 + Instructor + GMM required 23.02s and 34.01s. These results demonstrate approximately 5.1× lower processing latency for the profiled SpeechLens checkpoint. Peak allocated GPU memory was 17.32–18.14 GB for SpeechLens, compared with 6.34–6.35 GB for the ASR-based pipelines, reflecting a latency–memory trade-off. These measurements provide concrete evidence of the computational feasibility of jointly clustering multi-minute audio collections within seconds on a single GPU.

For larger speech repositories, perspective-conditioned clustering can naturally serve as a component within hierarchical or retrieval-based workflows. Recordings can first be organized into subsets, followed by cross-group comparison and iterative refinement under the same natural-language perspective. Such approaches preserve the core structure-discovery operation while enabling repository-level organization.

Crucially, establishing the effectiveness of perspective-conditioned speech clustering and demonstrating production-scale repository deployment are distinct research objectives. Our work investigates the former through controlled evaluation, held-out perspectives, comprehensive baseline comparisons, and systematic ablations. We believe these contributions provide a substantive scientific foundation for flexible speech organization, with controlled benchmark evaluation serving as a concrete means of establishing this new capability.

Thank you for the clarification. I agree that L2 shows generalization to unseen clustering perspectives within the same synthesis pipeline. What I'm still unsure about is whether those gains transfer to human speech. This is especially relevant for emotion, where prior work has found a performance gap between synthesized and human speech (https://aclanthology.org/2026.acl-long.372/).

The case study and profiling are useful and show that the approach is feasible on small synthetic sets. Still, I'm less convinced about its practical value on a representative speech set. Even a small evaluation on human recordings with one or two annotated perspectives would help address my concern. Testing with at least two other TTS families (like Gemini, Fish Audio, or VoxCPM2) would also be a useful starting point. Unless meaningful evidence of real speech or TTS transferability is provided, I'm retaining my original score.

Reviewer_2KUKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the task of "speech multi-perspective clustering": a model directly partitions speech recordings according to a user perspective described in natural language, while simultaneously inferring the number of clusters and their assignments. The authors construct SpeechLens-Bench, a benchmark covering four domains—legal, financial, banking, and task-oriented dialogue—including both linguistic and paralinguistic perspectives. They further propose SpeechLens-R1, an end-to-end large audio-language model trained with reasoning distillation and preference optimization. Experiments show that SpeechLens-R1 outperforms all baselines on both ARI and V-measure, with an overall ARI improvement of 12.99 points and a V-measure improvement of 11.62 points.

Critical strengths

The task definition and benchmark construction have practical value: Clustering is shifted from "fixed similarity metrics" to "natural-language perspective conditioning," requiring the model to simultaneously infer the number of clusters and assignments, which is closer to real needs such as conversational analysis than conventional settings. The benchmark construction pipeline (perspective induction, refinement, transcript validation, TTS synthesis, and paralinguistic attribute injection) includes relatively complete prompts and validation steps in the appendix, giving it good reproducibility.

Critical weaknesses

The synthetic nature of the benchmark audio limits the external validity of the conclusions: All audio is synthesized by TTS, and paralinguistic attributes (emotion, speaker identity, background noise) are "injected" rather than naturally occurring. The abstract does not emphasize this, but the full text shows that all corpora go through the synthesis pipeline. It remains unclear whether the perspective-conditioning ability learned on synthetic speech can transfer to real recordings (with natural paralinguistic variation and differences in recording conditions)—precisely the scenario that "paralinguistic perspective clustering" most needs to validate.

Questions for authors

All audio in SpeechLens-Bench is synthesized by TTS, with paralinguistic attributes injected under control. Have you evaluated SpeechLens-R1's performance on real recordings (natural emotion, real background noise, uncontrolled speaker variation)? If there is a domain gap between synthetic and real speech, does the "comparison-based reasoning over audible cues" learned in the RD stage still hold?

1 comment

We thank the reviewer for recognizing the practical significance of our task formulation and the reproducibility of SpeechLens-Bench.

Controlled speech synthesis is a deliberate methodological choice motivated by the unique requirements of multi-perspective speech clustering. To the best of our knowledge, existing real-speech benchmarks do not provide the comprehensive annotations required for this task: multiple linguistic and paralinguistic clustering perspectives over the same speech collection, corresponding ground-truth partitions, and held-out perspective evaluations.

Controlled synthesis enables systematic evaluation with reliable ground-truth annotations and controlled acoustic factors. Specifically, we can vary emotion, speaker configuration, and background conditions while approximately balancing irrelevant attributes across semantic categories. This design reduces unintended correlations between target clustering criteria and unrelated acoustic factors, allowing us to better isolate perspective-conditioned clustering capabilities. We additionally employ transcript validation, lexical-leakage filtering, and human quality checks to ensure benchmark reliability.

For audio generation, we adopt Qwen3-TTS CustomVoice, a state-of-the-art speech synthesis model. Our research focuses on perspective-conditioned speech understanding and structure discovery, rather than speech synthesis fidelity.

Regarding real-world transfer and RD generalization, our current evaluation focuses on controlled synthesized speech rather than naturally recorded speech. Notably, for paralinguistic perspectives, our RD pipeline employs an audio-capable teacher to generate comparison-based reasoning grounded in audible cues, explicitly encouraging cross-recording comparisons rather than independent attribute classification. This supervision targets the relational grouping process, not simply the identification of predefined acoustic labels. Furthermore, our L2 evaluation holds out clustering perspectives, their underlying taxonomies, and speech recordings. SpeechLens-R1 achieves 44.10 ARI and 72.50 V-measure on L2, demonstrating generalization to unseen clustering criteria within the controlled evaluation setting.

Cross-perspective generalization and synthetic-to-real acoustic transfer represent independent and orthogonal evaluation dimensions. Our results directly establish the former, while transfer to naturally recorded speech is left for future work. We will make the controlled TTS-based evaluation setting explicit in the abstract and clarify the corresponding generalization scope.

We believe this controlled benchmark provides a concrete foundation for evaluating multi-perspective speech clustering, together with strong empirical evidence for the effectiveness of our reasoning-based approach.

Comments on the paper

1 comment
Area_Chair_o2PrOpenReview ↗

Reminder: Discussion Phase Ends November 18 (11:59 PM AoE)

Hi reviewers, a quick reminder that the discussion phase ends on November 18 at 11:59 PM AoE. Please take some time to read the author responses, engage in further discussions if you have any remaining questions or concerns, and update your reviews and scores as appropriate. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.