acceptodds
Under review as a conference paper at ICLR 2027

SURE-Tagger: Structured Speech Representations for Compositional Evaluation and Targeted Data Selection

Abstract

Speech conveys linguistic content, speaker characteristics, temporal interactions, paralinguistic cues, and acoustic environments. While recent speech captioning models describe these dimensions in natural language, inconsistent attribute coverage and ambiguous omissions hinder systematic analysis across recordings. We introduce SURE-Tagger, a framework that constructs semi-structured speech representations with explicit attribute values, supplementary descriptions, and abstention under insufficient evidence. By integrating specialist observations through language-model reasoning, SURE-Tagger enables speech conditions to be consistently identified and jointly queried across datasets. We further introduce Tag-Bench, combining real recordings with controlled, agentically synthesized speech scenes and verified attribute annotations. On Tag-Bench, SURE-Tagger achieves the highest scores on 25 of 29 evaluated dimensions for real speech and 24 of 29 for synthetic speech, outperforming caption-based baselines across structural and descriptive attributes. Beyond annotation, we use these representations to construct cross-dataset evaluation subsets and reveal condition-specific ASR errors. Using a separate candidate pool, SURE-Tagger-guided selection reduces the overall ASR error rate of Qwen3-ASR-1.7B by 14.3% relative to the unadapted model under a one-hour fine-tuning budget, while caption-guided and random selection increase error in the same setting. These results demonstrate how structured speech representations can connect rich annotation, compositional evaluation, and targeted model improvement.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech caption, rich speech annotation, compositional evaluation, data selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YueNOpenReview ↗
Rating
3(+2)
Confidence
3(+1)
Informativeness
1
Presentation
1(−1)
Soundness
2
Read review
Paper summary

This paper mainly presents SURE-Tagger, which is a framework that integrates the observation results of existing professional speech models through reasoning based on large language models, and constructs semi-structured speech representations. This framework organizes speech information into five predefined semantic levels, combined with explicit attribute values, uncertainty states, and supplementary descriptions, to support consistent annotation and combined analysis across datasets. The authors also introduce Tag-Bench, which includes real recordings and synthesized generated speech scenarios, for evaluating the performance of annotation quality in both structured and descriptive dimensions.

Critical strengths

This article examines a related limitation of free-form voice subtitles, namely the inconsistency in attribute coverage and the omission of ambiguity, which hinders the systematic comparison and analysis of different recordings.

Critical weaknesses

1.The proposed method mainly involves an integration based on large language models, using existing speech models to perform reasoning on the original data and organizing it into an artificially predefined structure. Its essence is merely an engineering-oriented technical description. Its technical contributions beyond traditional attribute extraction, structured annotation, and information aggregation are not yet clear. The authors should more clearly distinguish their methodological contributions from those of existing methods.

2.This paper fails to provide a sufficient explanation for why these specific attributes and organizational structures were chosen, nor does it clarify whether there are other design schemes that could achieve similar results. More importantly, the lack of systematic reduction experiments at the architectural level makes it difficult to determine the necessity and effectiveness of the proposed representation design.

3.Tag-Bench constructs controlled synthetic speech scenarios based on the attributes extracted from the proposed framework, and simultaneously evaluates the annotation performance through the corresponding predefined dimensions. It is difficult to determine whether this benchmark objectively measures the general speech understanding ability or is mainly biased towards representations that conform to its own attribute definitions. Using independent annotation standards or established benchmarks for additional evaluation will help enhance the validity and generalization of the research results.

4.The citation format of this article is currently incorrect and does not comply with the template requirements. Correction should be made.

Questions for authors
  1. Which theories or empirical evidence support the selection of these five semantic levels and their predefined attributes? Has the author examined whether simpler or alternative configuration schemes could achieve similar labeling and downstream performance?

  2. To what extent did the reported improvements stem from the proposed structured representation and the integration based on large language models, rather than the capabilities of the underlying professional speech models?

Reviewer_Yu5EOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that free-form speech captions are unreliable as sample-level attribute records because an omitted attribute cannot be interpreted as confirmed absence ("not mentioned ≠ not present"), preventing consistent cross-dataset analysis. It proposes (1) a "semi-structured speech representation" where each recording receives explicit attribute values (over five layers: lexical, speaker, temporal, acoustic scenes/events, plus open-ended descriptions) with a dedicated null state for "insufficient evidence", (2) "SURE-Tagger", a framework that constructs these representations by integrating observations from specialized perception models (FireRedLID/AED/ASR, MOSS-Diarize, DiaRiZen) via one LLM call under schema constraints, (3) "Tag-Bench", a new 1,150-clip benchmark (500 agentically synthesized scenes + 650 recordings from AMI/FLEURS/LibriSpeech/DESED/FSD50K) with verified reference annotations scored across 29 axes, and (4) a closed-loop demonstration in which the same representations enable compositional cross-dataset ASR evaluation (6 attribute-defined condition clusters over 1,000 recordings from 10 datasets) and condition-guided training-data selection. Under a one-hour fine-tuning budget, SURE-Tagger-guided selection reduces Qwen3-ASR-1.7B error by 14.3% relative (5.50%→4.71%), while caption-guided and random selection increase error. This is a solid, well-executed paper with a genuinely useful representation and benchmark, and I found neither fatal correctness flaws nor overclaims that I consider indefensible. However, the descriptive-axis scoring shares the judge with the generating pipeline, the human validation is small, and the headline selection result is fragile under transfer, seed variance, and tail-cluster dependence. Each concern is addressable with additional analysis. On balance, the core idea is novel, clear, and consequential, and the paper provides enough validation to warrant acceptance, though a strengthened revision would merit a higher score.

Critical strengths

The "not mentioned != not confirmed absent" framing sharpens a genuine gap in speech captioning, and the semi-structured schema with an explicit abstention state is a clean, generalizable design that survives scrutiny: the known-case/null-as-zero ablations (Appendix D) show SURE-Tagger's 76.28% is not an abstention artifact, the language-tag ablation (88.90->98.62 with LLM evidence integration) isolates the mechanism, and Tag-Bench (verifiable references, from rendering records/measurements rather than specifications) is a reusable resource. This is consequential because the paper's claims rest on the representation itself, and this contribution would likely remain valuable even if every downstream number were discounted.

The paper goes beyond "better annotation numbers" and demonstrates that the same representation measurably improves a downstream product: condition-guided selection reduces Qwen3-ASR-1.7B WER by 14.3% under a one-hour budget while caption-guided and random selection hurt, and the effect transfers to Whisper-small in the Eval.-guided variant. The η² comparison (0.348 vs. 0.022 vs. 0.005) further shows the attributes are predictive of model errors in a way that captions are not. This closes the loop from annotation -> evaluation -> improvement and makes the work internally coherent and externally consequential for practitioners.

Critical weaknesses

The descriptive axes (including the per-paper "Caption" axis) are generated by an LLM pipeline and scored by DeepSeek-V4-Pro — the same LLM family that produces the schema and judges semantic agreement. The Grok 4.7 re-scoring only swaps one judge for another of the same generation and does not control for self-preference toward system-consistent outputs. Since 10/29 of the scores and one of the three claims depend on these axes, and since the human study is small (100 items), the descriptive part of the evaluation is not yet convincing.

Questions for authors

For the 10 descriptive axes scored by DeepSeek-V4-Pro: (a) was the judge shown the outputs blind to which system produced them, and (b) have you checked for self-preference by having a judge excluded entirely from the generating pipeline (e.g., a judge not used anywhere in schema construction) re-score the same outputs and re-rank the systems? A short human-rated subset on D would also help.

Reviewer_7gDJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SURE-Tagger integrates specialist speech-model observations into a semi-structured representation containing explicit attribute values, null values for insufficient evidence, and descriptive fields. The paper introduces Tag-Bench with 650 existing-corpus recordings and 500 controlled synthetic scenes, evaluating 19 structured and 10 descriptive axes. It then uses the representation for cross-dataset ASR condition clustering and targeted training-data selection. On Qwen3-ASR-1.7B, candidate-pool-guided selection under a one-hour audio budget reduces the reported pooled recognition error from 5.50% to 4.71%. The appendices include judge robustness, human layer preferences, null-score diagnostics, synthesis details, and an additional Whisper-small adaptation experiment.

Critical strengths

The representation is practically useful and its annotation gains have complementary support. Explicit values and abstentions make conjunctions of conditions queryable across datasets. Also, the work tests downstream usefulness rather than stopping at annotation quality. It forms cross-dataset error profiles and evaluates one-hour data selection under a shared sampling/fine-tuning recipe.

Critical weaknesses

The role of the proposed evidence integration is not sufficiently isolated from the stronger specialist observations. Both baselines use the same final LLM, but their inputs are general-model observations or free-form captions rather than the specialist evidence available to SURE-Tagger. Thus the main gain could largely reflect richer measurements instead of the integration procedure. A direct specialist-to-schema mapping or rule-based reconciliation using the identical observations would clarify the methodological increment.

The data-selection advantage is narrower than a general model-improvement result. Pool-guided selection estimates target-ASR errors on the candidate pool, making error- or loss-based sampling using the same available labels an important comparison alongside random and caption-guided sampling.

Questions for authors
  1. How does an identical-observation pipeline with direct mapping or deterministic reconciliation compare on all structured axes? Which fields benefit specifically from the LLM integration, beyond language identification?
  2. Under the same one-hour budget and available candidate labels, how do individual-error sampling, error-stratified sampling, and simple diversity/duration-balanced selection compare? Please include per-language/source results so the pooled WER/CER change can be interpreted.
  3. Can the independent Pool result be validated on an untouched evaluation set with speaker/session-disjointness documented, and across the second ASR model? Please distinguish the scope of the Qwen result from the mixed Whisper-small findings and from Eval-guided adaptation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.