- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the "speech multi-view clustering" task, which requires a model to directly cluster a collection of speech samples based on a view specified in natural language, while simultaneously inferring both the number of clusters and the cluster assignments. To support this task, the authors developed the SpeechLens-Bench benchmark, which spans multiple application domains and maintains a strict distinction between in-view and cross-view generalization evaluations. Furthermore, the authors introduce SpeechLens-R1, an end-to-end large audio-language model trained via reasoning distillation and preference optimization to learn view-based clustering reasoning. Experiments demonstrate that SpeechLens-R1 significantly outperforms both ASR-based pipelines and existing native audio-language model baselines in terms of ARI and V-measure.
Critical strengths
This paper proposes the "speech multi-view clustering" task, which requires a model to directly cluster a collection of speech samples based on a view specified in natural language, while simultaneously inferring both the number of clusters and the cluster assignments. To support this task, the authors developed the SpeechLens-Bench benchmark, which spans multiple application domains and maintains a strict distinction between in-view and cross-view generalization evaluations. Furthermore, the authors introduce SpeechLens-R1, an end-to-end large audio-language model trained via reasoning distillation and preference optimization to learn view-based clustering reasoning. Experiments demonstrate that SpeechLens-R1 significantly outperforms both ASR-based pipelines and existing native audio-language model baselines in terms of ARI and V-measure.
Critical weaknesses
1.The paper claims to outperform the ASR+LLM baseline, but it does not specify whether an optimal Chain-of-Thought (CoT) prompt was designed for the LLM baseline (e.g., GPT-4o) or if sufficient engineering optimization was applied. Furthermore, when an end-to-end LALM processes multiple long audio segments, the computational cost of its autoregressive generation and cross-attention is extremely high. The paper downplays these substantial training and inference costs, casting doubt on the method's applicability to large-scale, real-world audio repositories. 2.The model is required to simultaneously infer the number of clusters (K) and the cluster assignments. If K is inferred incorrectly (e.g., resulting in over-merging or over-splitting), the entire clustering result becomes completely invalid. The paper lacks an independent evaluation of the inference accuracy for K, and fails to analyze how errors in estimating K cascade to impact the final ARI and V-measure metrics.
Questions for authors
1.Please provide the complete prompt used for the ASR+LLM baseline (e.g., GPT-4o). Have you tried having the LLM output its reasoning process (Chain-of-Thought, CoT) before performing the clustering? Does the current baseline performance represent the true performance upper bound of this paradigm? 2.If the correct number of clusters K is directly provided to the model via an oracle during inference, how much would the performance of SpeechLens-R1 improve? This would help distinguish whether the model's performance bottleneck stems from its "feature representation and reasoning capabilities" or merely from failures in "cluster number inference." 3.What are the average inference time and memory consumption for a single forward pass of SpeechLens-R1 when processing an instance containing 10 audio segments? How significant is the efficiency gap compared to an ASR+Embedding-based pipeline? For larger-scale audio collections, are there any feasible chunking or hierarchical clustering strategies?
3 comments
Q2: We would like to correct an important premise in this concern: an incorrect number of clusters does not make the entire clustering result invalid under either ARI or V-measure.
Both metrics compare partition structures and assign partial credit to predictions that preserve some correct groupings, even when the predicted cluster count differs from the ground truth.
For example, consider a gold partition with and a predicted partition with . Despite the cluster-count mismatch, the prediction obtains and .
Therefore, the two reported evaluation metrics already reflect the joint effects of cluster-count errors, over-merging, over-splitting, and incorrect item assignments. A mismatch is a clustering error, not necessarily a structurally invalid prediction.
Moreover, jointly inferring and the assignments is a defining property of our proposed task (Section 3.1). Providing oracle at inference time would introduce privileged information unavailable in the intended setting and would evaluate a different, easier problem.
Our baseline protocol already illustrates this distinction: K-Means and GMM are supplied with the gold number of clusters, whereas SpeechLens-R1 must infer it. Despite this more demanding prediction setting, SpeechLens-R1 achieves the strongest overall ARI and V-measure.
Finally, our preference-optimization pipeline explicitly uses valid but incorrect partitions, including over-merging and over-splitting errors, as hard-negative supervision. The method therefore directly targets structural clustering decisions, rather than merely learning the required output format.
An oracle- diagnostic, therefore, is not required to interpret the reported ARI and V-measure results or to establish performance on the original joint-inference task.
Q3: To provide quantitative evidence, we conducted additional inference profiling on a single NVIDIA H200 NVL GPU, comparing our audio-native clustering approach against two complete ASR-based pipelines: Whisper large-v2 + MiniLM + GMM and Whisper large-v2 + Instructor + GMM.
We evaluated two representative test instances, including an approximately mean-duration sample and the longest-duration test sample. Each configuration was measured over three runs, excluding model loading time.
| Test instance | SpeechLens | ASR + MiniLM + GMM | ASR + Instructor + GMM |
|---|---|---|---|
| mean-duration | 4.48s | 22.99s | 23.02s |
| longest-duration | 6.62s | 33.97s | 34.01s |
The SpeechLens measurements include audio preprocessing and generation, while the ASR baselines include full-audio transcription, embedding extraction, and GMM clustering. SpeechLens achieves approximately 5.1× lower processing latency on both instances. Regarding memory consumption, SpeechLens requires 17.32–18.14 GB of peak allocated GPU memory, compared with 6.34–6.35 GB for the ASR-based pipelines. Thus, our approach provides a substantial latency advantage on the measured instances, with a higher GPU memory footprint.
Importantly, SpeechLens jointly processes multiple audio segments and generates a complete clustering partition in a single inference request, avoiding separate full-transcript generation and downstream embedding/clustering stages. These measurements demonstrate the practical feasibility of multi-minute, perspective-conditioned audio clustering within seconds on a single GPU.
For larger repositories, SpeechLens can naturally serve as a perspective-conditioned grouping component within hierarchical clustering or retrieval-based workflows. Recordings can first be organized into subsets, followed by cross-group comparison and iterative refinement under the same user-specified perspective.
Overall, the additional measurements demonstrate a favorable latency–memory trade-off for joint audio-native clustering, complementing the substantial accuracy improvements reported in our main experiments.
We thank the reviewer for recognizing the task formulation, the distinction between held-in and held-out perspectives, and the improvements over both ASR-based and native audio-language model baselines.
Q1: We would like to clarify that the complete ASR+LLM prompting template is already provided in Appendix D (page 31). The prompt explicitly instructs the models to:
The template also includes the clustering perspective and the ASR-transcribed contents of all indexed audio segments. Therefore, the reported baseline is not a simple direct-labeling prompt; it already incorporates a multi-step, reasoning-oriented clustering protocol with explicit output constraints.
Importantly, there is no established prompting protocol that guarantees the theoretical upper bound of the ASR+LLM paradigm. Our comparison therefore focuses on a clearly documented, reasoning-oriented baseline rather than claiming prompt-level optimality.
In addition, the central results do not rely exclusively on GPT-4o. SpeechLens-R1 is also evaluated against multiple native audio-language models. SpeechLens-R1 outperforms the ASR+GPT-4o baseline by 16.39 ARI points and 13.01 V-measure points. It also surpasses the strongest native audio-language baseline, GPT-audio-1.5, by 12.99 ARI points and 11.62 V-measure points.
These comparisons, together with the answer-only SFT and RD/DPO ablations in Table 4, establish the effectiveness of task-specific reasoning and preference optimization for the proposed clustering objective.
We will make reference to the complete GPT-4o prompt more explicitly in the main experimental section to improve clarity.