acceptodds
Under review as a conference paper at ICLR 2027

AUDIO-COMPASS: REFERENCE-FREE COMPONENT-WISE EVALUATOR FOR AUDIO–TEXT SEMANTIC ALIGNMENT

Abstract

Many audio–text evaluation metrics summarize semantic alignment with a single overall score, motivating explicit assessment along consistent semantic dimensions. We introduce \model, a 30B reference-free audio judge that produces overall and component-wise scores for sound events, sources, acoustic attributes, and temporal relations, along with structured explanations of matched, missing, and incorrect information. To train \model, we construct \dataset with 105k fine-grained caption evaluations over 17.5k audio clips. Our perception–reasoning cascade consolidates descriptions from selected audio-language models into high-quality synthetic references, against which GPT-5-mini evaluates candidate captions to generate scores and explanations as training targets. Fine-tuned on this supervision, Audio-Compass evaluates audio–text alignment directly without references at inference time, supporting both audio-to-text caption evaluation and text-to-audio generation evaluation. Across BRACE, RELATE, PAM, and our new long-form benchmark BRACE-Long, \model shows strong agreement with human judgments, achieving a Spearman correlation of 0.636 on PAM and a temporal-relation correlation of 0.370 on RELATE(OS), compared with up to 0.295 for the evaluated CLAP-based metrics. Further analysis suggests that holistic human judgments are more strongly associated with sound-event information than with acoustic attributes or temporal relations, motivating explicit component-wise evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-text semantic alignment, Reference-free evaluation, audio-language models, fine-grained and explainable evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pgfNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Audio-Compass, a reference-free audio–text evaluator that produces an overall alignment score and component-wise judgments for sound events, sources, acoustic attributes, and temporal relations. Its training pipeline first uses multiple audio-language models to describe each clip, consolidates their outputs into textual anchors, and prompts GPT-5-mini to assess candidate captions against those anchors. The resulting fine-grained scores and explanations are distilled into an audio judge, which at inference time evaluates the audio and candidate caption directly, without reference captions. The authors also construct Compass-105k for training and introduce BRACE-Long to assess preferences between long-form captions. The work addresses a meaningful limitation of holistic metrics and offers a potentially useful evaluation pipeline and resource. However, the results do not establish consistent superiority: LALM + CLAP baselines often outperform Audio-Compass. The work’s main contribution therefore appears to be diagnostic, component-wise evaluation rather than clear gains in overall accuracy. The paper also does not sufficiently isolate the benefit of Compass-105k fine-tuning or its dependence on training-set scale, rule out overlap with external benchmarks. Overall, the work is promising as an evaluation framework and dataset contribution, but its claims require stronger ablations and data contamination analysis.

Critical strengths
  1. The paper targets an important practical problem: holistic alignment scores offer little diagnosis of what a caption gets right or wrong.

  2. The study covers the full process from data collection and reference synthesis to teacher annotation, student-model training, and evaluation on multiple benchmarks.

  3. Reference captions are used only during training-data construction. At deployment time, the evaluator directly takes an audio clip and a candidate caption as input, making it suitable for real-world scenarios.

  4. The planned release of Compass-105k could substantially benefit the community by providing a source of audio–text data for fine-tuning and evaluation.

Critical weaknesses
  1. The LALM + CLAP systems outperform Audio-Compass in several experiments. Audio-Compass is stronger in some settings, notably PAM, and offers richer component-wise outputs, but the paper does not sufficiently explain why its additional fine-grained modeling does not translate into consistent gains in overall accuracy. Its main advantage therefore appears to be diagnostic functionality rather than clearly superior performance.

  2. There is no ablation study examining how Audio-Compass’s capabilities depend on the scale and composition of Compass-105k. In particular, the reported performance of Audio-Compass is often close to that of the original Qwen3-Omni-Instruct baseline, making it unclear where the actual benefit of SFT comes from. The paper should analyze performance as a function of training-set size and provide stronger comparisons across different subsets or target formats.

  3. The paper does not provide a data leak analysis. Although Compass-105k is split by audio ID to prevent overlap among its internal training, validation, and test sets, this does not establish independence from the external evaluation benchmarks. Compass-105k is constructed from sources including AudioCaps and Clotho-v2, while BRACE, BRACE-Long, and PAM may contain audio derived from these collections.

  4. Tables 1-4 evaluate different sets of models. The inclusion and exclusion criteria for reference-free models do not clearly explain why certain baselines are omitted from particular benchmarks or whether their exclusion is technically justified.

Questions for authors
  1. Does Compass-105k contain any audio, captions, or metadata that overlap with the evaluation benchmarks? If there is any data leakage, please report exact overlap statistics between the Compass-105k training split and each benchmark.

  2. How does Audio-Compass performance depend on the scale and composition of Compass-105k? Please provide controlled experiments using different training-set sizes and supervision subsets, together with a comparison to the original Qwen3-Omni-Instruct model, to clarify the actual benefit of SFT and whether performance saturates with the current data scale.

  3. Why do the reference-free baselines differ across Tables 1–4? Beyond the stated inapplicability of reference-based metrics to some benchmarks, please explain the inclusion and exclusion criteria for each CLAP-based and LALM-based evaluator.

5 comments
AuthorsOpenReview ↗

(W1) Performance gains and Compass-Hallucination

W1: The LALM + CLAP systems outperform Audio-Compass in several experiments. Audio-Compass is stronger in some settings, notably PAM, and offers richer component-wise outputs, but the paper does not sufficiently explain why its additional fine-grained modeling does not translate into consistent gains in overall accuracy. Its main advantage therefore appears to be diagnostic functionality rather than clearly superior performance.


We appreciate your thoughtful feedback on the benefits of fine-grained evaluation. We agree that LALM + CLAP hybrid evaluation method (CAF-Score) performs better on most benchmarks. To check whether the benefits of Audio-Compass go beyond rationale outputs, we conducted an additional evaluation measuring its ability to distinguish specific caption errors.

Compass-Hallucination compares a human caption with a negative caption containing a component-specific semantic error. Pairwise accuracy measures whether the evaluator assigns the human caption a higher overall score; Macro Avg. is the unweighted mean across four components. Please see our Official Comment, “Compass-Hallucination,” for construction details, filtering criteria, and full results.

Audio-Compass achieved 81.25% on Acoustic Attribute, 79.93% on Temporal Relation, and 87.27% Macro Avg., exceeding the strongest baseline for each metric by 4.17, 4.93, and 1.52 percentage points, respectively. Baselines remained stronger on Sound Event and Source. These results demonstrate measurable gains in detecting attribute and temporal errors, while also showing that the gains are component-dependent.

One possible explanation for the inconsistent gains on existing benchmarks is that holistic human ratings do not reflect all semantic components equally. In RELATE (REL; Section 6.1), Sound Event scores correlated more strongly with human ratings than Overall scores (SRCC: 0.450 vs. 0.428), while Acoustic Attribute and Temporal Relation scores(SRCC: 0.029 and 0.178) showed lower correlations. This pattern suggests that improved detection of attribute and temporal errors may not translate directly into higher agreement with holistic ratings, although it does not establish the cause of the performance gap.

AuthorsOpenReview ↗

(W2, Q2) Experiments on training data scaling

W2: There is no ablation study examining how Audio-Compass’s capabilities depend on the scale and composition of Compass-105k. In particular, the reported performance of Audio-Compass is often close to that of the original Qwen3-Omni-Instruct baseline, making it unclear where the actual benefit of SFT comes from. The paper should analyze performance as a function of training-set size and provide stronger comparisons across different subsets or target formats.

Q2: How does Audio-Compass performance depend on the scale and composition of Compass-105k? Please provide controlled experiments using different training-set sizes and supervision subsets, together with a comparison to the original Qwen3-Omni-Instruct model, to clarify the actual benefit of SFT and whether performance saturates with the current data scale.


We thank the reviewer for requesting a more detailed analysis of the contribution of Compass-105k. To examine the benefit of SFT as the number of processed training samples increases, we evaluated three checkpoints from the Audio-Compass training run and compared them with the original Qwen3-Omni-Instruct model.

Audio-Compass is trained on 104,401 samples derived from score and rationale annotations. We evaluated the checkpoints using the same evaluation prompt on PAM and BRACE-Main, reporting both text score (Raw) and FLEUR scores. Text scores are directly extracted from the model's generated text, whereas FLEUR scores are obtained through a post-processing method that smooths the scores using text token probabilities.

ModelTrainset SizePAM SRCC (Text Score / FLEUR)BRACE-Main Accuracy, % (Text Score / FLEUR)
Qwen3-Omni-Instruct, before SFT—0.384 / 0.42644.51 / 68.31
Audio-Compass38.35k (36.73%)0.586 / 0.58759.50 / 68.75
Audio-Compass76.69k (73.46%)0.625 / 0.62465.99 / 71.75
Audio-Compass104.40k (100%)0.636 / 0.62864.74 / 71.19

In the table, Trainset Size denotes the estimated number of samples processed at each checkpoint, corresponding to 36.73%, 73.46%, and 100% of the full 104,401-example training set, respectively.

These results demonstrate the benefit of evaluation-specific SFT over the original Qwen3-Omni-Instruct model. At the final checkpoint (104.40k), Audio-Compass improves PAM SRCC by 0.252 / 0.202 and BRACE-Main accuracy by 20.23 / 2.88 percentage points under Raw / FLEUR scoring, respectively. The larger improvements under Raw scoring, particularly on BRACE-Main, suggest that SFT improves the model's task-specific scoring behavior. In contrast, FLEUR already improves the baseline's BRACE-Main accuracy from 44.51% to 68.31% through score-token probability-based post-processing, leaving a smaller margin for additional gains.

Regarding training scale, PAM SRCC improves consistently, but gains from 76.69k to 104.40k are small (+0.011 / +0.004). Meanwhile, BRACE-Main reaches its highest accuracy at 76.69k (65.99% / 71.75%) and slightly declines at 104.40k (64.74% / 71.19%). These results indicate diminishing returns during training and confirm the reviewer's concern that additional training does not consistently improve performance across benchmarks. While checkpoint comparisons cannot establish dataset-size saturation or isolate the effects of data composition, we will include these findings in the revised manuscript to clarify the contribution and limitations of Compass-105k.

AuthorsOpenReview ↗

(W3, Q1) Data Leakage Analysis

W3: The paper does not provide a data leak analysis. Although Compass-105k is split by audio ID to prevent overlap among its internal training, validation, and test sets, this does not establish independence from the external evaluation benchmarks. Compass-105k is constructed from sources including AudioCaps and Clotho-v2, while BRACE, BRACE-Long, and PAM may contain audio derived from these collections.

Q1: Does Compass-105k contain any audio, captions, or metadata that overlap with the evaluation benchmarks? If there is any data leakage, please report exact overlap statistics between the Compass-105k training split and each benchmark.


We appreciate you drawing attention to potential overlap between the training data and evaluation benchmarks. In our additional analysis, we found some overlaps in original recording IDs between the benchmarks and our training data sources.

For datasets with official splits, including AudioCaps, Clotho-v2, FSD50K, CochlScene, and TUT Urban, we used only the training splits (the development split for Clotho-v2). The train/valid/test splits mentioned in our paper refer to internal splits created from the collected data for model training and internal evaluation. We omit to explain the difference between these internal splits and the official dataset splits.

However, even when we follow the official splits, the same original recordings may still appear in the benchmarks through other datasets that do not have official splits. Therefore, we defined audio clip overlap as cases where both the original recording ID and audio duration matched exactly. We also checked the official source interval metadata to verify these matches.

BenchmarkExact Audio Clip OverlapNormalized Caption Overlap
RELATE-REL2 / 437 (0.46%)2.09–2.32%
RELATE-IS2 / 778 (0.26%)1.94–2.07%
RELATE-OS0 / 395 (0.00%)0.00%
BRACE-Main3 / 575 (0.52%)0.82–0.85%
BRACE-Hallucination4 / 2,027 (0.20%)0.25–0.27%
BRACE-Long0 / 129 (0.00%)0.00%
PAM0 / 100 (0.00%)3.03–4.04%

For audio overlap, the denominator is the number of real audio clips in each benchmark. For caption overlap, we compared captions in the training set with benchmark captions after normalizing capitalization, punctuation, and whitespace. The denominator is the number of unique normalized captions in each benchmark. However, matching caption text alone does not necessarily indicate audio leakage. For example, we found that the same caption, "A woman is speaking," was used for audio clips with different original recording IDs.

Through this audit, we confirmed some overlaps with the benchmarks. In the revised manuscript, we will report the overlap counts for each benchmark.

Reviewer_N8TwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors contribute AudioCompass a reference free LLM judge for audio-text alignment. AudioCompass is trained on a synthetic training set created by the authors following a multi-model data curation and filtering pipeline. AudioCompass compares favorably to CLAP-based and other LLM judge models on multiple benchmarks, including BRACE and PAM.

Critical strengths
  1. Strong performance and Reproducibility: AudioCompass outperforms SOTA LLM as judge models, including proprietary ones like Gemini, while being open-source and therefore much reliable as a judge model for future work.
Critical weaknesses
  1. Missing comparisons with frame-level CLAP-like models. It is known that CLAP-like models struggle with long-form audio and audio with more than 1 sound event due to the temporal pooling of their training, which makes the baselines a bit weak. The paper would benefit from a comparison with frame-level CLAP-like models, like PE-A [1] or OpenFLAM [2]. The score could be a simple average score over the sound sources, like done by [3], or a more complex version that considers temporal boundaries.
  2. Limited novelty: A reductionist view of the paper would be that it is essentially distilling various LLMs into a single model, which is arguably much more complex than cited CLAP+Judge pipeline (CAF-Score) while performing worse.

[1] Vyas, Apoorv, et al. "Pushing the frontier of audiovisual perception with large-scale multimodal correspondence learning." CVPR. 2026.

[2] Wu, Yusong, et al. "FLAM: Frame-wise language-audio modeling." ICML 2025.

[3] Chen, William, et al. "Audiochat: Unified audio storytelling, editing, and understanding with transfusion forcing." ICML 2026.

Questions for authors

Can you show how this work is more than just a clever data filtering and cleaning pipeline for distillation? This is my main barrier for a higher score.

3 comments
AuthorsOpenReview ↗

(W1) Additional baselines : frame-level CLAP-like models

W1: Missing comparisons with frame-level CLAP-like models. It is known that CLAP-like models struggle with long-form audio and audio with more than 1 sound event due to the temporal pooling of their training, which makes the baselines a bit weak. The paper would benefit from a comparison with frame-level CLAP-like models, like PE-A [1] or OpenFLAM [2]. The score could be a simple average score over the sound sources, like done by [3], or a more complex version that considers temporal boundaries.


Thank you for recommending these frame-level baselines. Following your recommendation, we additionally evaluated PE-A-Frame and OpenFLAM to provide a more comprehensive comparison with frame-level models. We also applied a multiFLAM-style SourceMax aggregation method to both models, which selects the highest frame-level matching score for each sound-source query and averages these scores to obtain the final audio–caption score.

The additional results are shown below.

BenchmarkPE-A-FrameOpenFLAMAudio-Compass
BRACE-Main58.81%52.40%71.19%
BRACE-Hallu86.91%78.15%96.65%
RELATE(REL)0.47280.13950.428
RELATE(IS)0.43080.17850.378
RELATE(OS)0.30410.06080.404
PAM0.37970.37730.636

BRACE-Main and BRACE-Hallu are evaluated using pairwise accuracy (%), while RELATE and PAM use Spearman's rank correlation coefficient (SRCC).

Audio-Compass outperforms both baselines in four out of six evaluation settings, including BRACE-Main, BRACE-Hallucination, RELATE (OS), and PAM. It also achieves better results than OpenFLAM across all settings. However, PE-A-Frame performs better on RELATE (REL) and RELATE (IS).

These results show that Audio-Compass remains competitive with frame-level models across multiple benchmarks, although it does not achieve the best performance in every setting. We will include these additional results and evaluation details in the revised manuscript to provide a more comprehensive and balanced baseline comparison.

AuthorsOpenReview ↗

(W2,Q1) Novelty

W2: Limited novelty: A reductionist view of the paper would be that it is essentially distilling various LLMs into a single model, which is arguably much more complex than cited CLAP+Judge pipeline (CAF-Score) while performing worse.

Q1: Can you show how this work is more than just a clever data filtering and cleaning pipeline for distillation? This is my main barrier for a higher score.


We thank the reviewer for raising a question regarding the novelty of this work and its distinction from conventional knowledge distillation. However, Audio-Compass's contributions lie not merely in data filtering or distillation, but in the fine-grained evaluation supervision we construct and the evaluation capabilities it enables.

1. High-quality Fine-grained Supervision Dataset

Compass-105k provides 105k samples annotated with Component-wise Scores and Structured Rationales. In Human Validation (100 audio samples, three annotators each; see Rxnj W2), Synthetic References achieved 96% Factual Accuracy, 93% Completeness, and 90% on both criteria. Teacher Scores correlated with Human Mean Scores (Spearman ρ = 0.672). These results support the quality of the reference and score supervision.

2. Explainable Evaluation with Structured Rationales

Audio-Compass evaluates Audio–Caption semantic alignment without Reference Captions, generating Component-wise Scores and Structured Rationales for Sound Event, Source, Acoustic Attribute, and Temporal Relation. The rationales identify Matched, Missing, and Incorrect information, providing textual explanations for the scores and allowing users to inspect specific semantic mismatches. In Human Validation (see Rxnj W2), the Teacher Rationales used to train this capability achieved 95% Overall Validity, supporting the quality of the supervision for explanation generation.

3. Improved Detection of Acoustic Attribute and Temporal Relation Errors

To directly investigate its advantages, we newly construct Compass-Hallucination (please see official comment), which evaluates the ability to detect component-specific semantic errors. The results reveal that CAF-Score performs strongly on Sound Event and Source, but is comparatively less effective at detecting errors in Acoustic Attribute and Temporal Relation. In contrast, Audio-Compass achieves higher accuracy on Acoustic Attribute (81.25% vs. 77.08%) and Temporal Relation (79.93% vs. 75.00%), yielding the highest Macro Average (87.27% vs. 85.75%).

These findings suggest that CLAP + LALM hybrid evaluation methods(CAF-Score) are effective at identifying what sound events and sources are present, but may be less sensitive to how they sound and how they unfold over time, including event ordering and overlap. Although Audio-Compass performs comparably to CAF-Score on existing holistic benchmarks, its ability to detect these fine-grained semantic errors demonstrates a distinct advantage beyond diagnostic functionality.

We appreciate the reviewer's feedback, which has helped us better articulate the novelty of our work. We will revise the manuscript accordingly, while explicitly acknowledging the limitations in holistic evaluation performance.

Thank you for the additional results and explanation. They make the strengths of the paper more clear. I encourage the authors to better highlight the positioning of their work, and what it enables, in future drafts.

I will raise my score to 3.

Reviewer_RxnjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Audio-Compass, an audio–text evaluator based on Qwen3-Omni. It produces overall and component scores for sound events, sources, acoustic attributes, and temporal relations, together with explanations. Training uses Compass-105k, with synthetic references consolidated from audio models and judgments generated by GPT-5-mini. The evaluator uses audio directly without references at inference. The paper also introduces BRACE-Long and evaluates agreement with human judgments across several benchmarks.

Critical strengths

S1: Useful evaluation formulation and resources. Evaluating directly from audio and returning a consistent semantic breakdown could support detailed caption assessment and text-to-audio evaluation. The training dataset and human preference benchmark for long captions are useful resources for this problem.

S2: Promising agreement with human judgments. On PAM, Audio-Compass achieves a Spearman correlation of 0.636, compared with 0.601 for AnyAudio-Judge. On BRACE-Main, its FLEUR-smoothed accuracy is 71.19%, compared with 68.31% for the Qwen backbone. These results support its usefulness for overall scoring.

Critical weaknesses

W1: Limited diagnostic granularity and insufficient validation. The four dimensions aggregate diverse details into single scores; acoustic attributes, for example, include loudness, pitch, timbre, count, and distance. Although the model generates explanations, the paper does not independently validate their accuracy and completeness or how they justify the numerical scores. RELATE provides partial evidence for events and ordering, but source and acoustic-attribute scores lack direct human validation. It remains unclear how reliably the outputs identify individual errors or what diagnostic advantage they offer over verifying individual atomic facts.

W2: Reliability of synthetic supervision is insufficiently established. Training judgments depend on model-generated references. The text-only GPT teacher cannot check the waveform and is instructed to treat the references as perfect (Figure 7). Consensus and CLAP filtering may reduce noise, but do not establish that fine acoustic details are correct and complete. A correct detail omitted from the references could therefore be penalized. I acknowledge the evaluation against human judgments, but an independent audit of the synthetic references and teacher labels would strengthen the claimed reliability of fine-grained evaluation.

Questions for authors

Have synthetic references and teacher judgments been checked against the audio? What proportion contains incorrect or missing details, and how do these affect the resulting evaluation labels?

4 comments
AuthorsOpenReview ↗

(W1) Validation and diagnositc granularity

W1: Limited diagnostic granularity and insufficient validation. The four dimensions aggregate diverse details into single scores; acoustic attributes, for example, include loudness, pitch, timbre, count, and distance. (1) Although the model generates explanations, the paper does not independently validate their accuracy and completeness or how they justify the numerical scores. (2) RELATE provides partial evidence for events and ordering, but source and acoustic-attribute scores lack direct human validation. (3) It remains unclear how reliably the outputs identify individual errors or what diagnostic advantage they offer over verifying individual atomic facts.


We thank the reviewer for pointing out the need for further validation of our fine-grained evaluation. We address the concerns as follows.

1. Validation on Explanations and Scores of Teacher Supervision: To assess the quality of teacher-generated training supervision, we selected 100 audio samples, each evaluated by three human annotators. Annotators assessed the Matched, Missing, and Incorrect statements and rated the rationale for each dimension, as well as the overall explanation, as Valid, Invalid, or Uncertain. They also assigned scores assuming the explanations were correct. Rationale validity was 90% for Sound Event, 88% for Source, 83% for Acoustic Attribute, and 86% for Temporal Relation. The overall rationale validity was 95%. In addition, teacher scores showed a Spearman correlation of 0.672 with human mean scores. Due to time constraints, we were not able to conduct human evaluation on Audio-Compass outputs, but we will complete the evaluation using the same evaluation protocol and reflect the results in our final manuscript.

2. Additional Evaluation of Source and Acoustic Attribute Errors: Our original description did not clearly explain how we evaluated Source errors. We used BRACE-Hallucination (Table 1), which compares a human caption with a negative caption created by replacing its sound source with an incorrect source.

To extend the evaluation beyond Source, we constructed Compass-Hallucination. For Acoustic Attribute in particular, we used GPT-5-mini to generate negative captions designed to change only the target attribute while keeping the other semantic information unchanged. For example, “playing a guitar loudly” becomes “playing a guitar quietly.”

We counted a prediction as correct when the model assigned the original caption a higher overall score than the negative caption. Audio-Compass achieved 96.54% accuracy on Source and 81.25% on Acoustic Attribute. Its Acoustic Attribute accuracy was 4.17 percentage points higher than the strongest baseline. Please see our Official Comment, “Compass-Hallucination,” for the construction process, filtering criteria, and full results.

3. Diagnostic Granularity: We agree that each dimension includes various details. However, not all acoustic attributes appear in every audio sample, so it is difficult to assign separate scores to each attribute. Instead, our evaluation instructions ask the model to consider relevant properties, such as loudness, pitch, timbre, count, and distance. By providing component-level scores and explanations in Figure 16, we aim to evaluate these details without requiring separate scores for every attribute.

We appreciate the reviewer's feedback and will include these additional results and explanations in the revised manuscript.

AuthorsOpenReview ↗

(W2, Q1) Reliability of synthetic supervision (1/2)

W2: Reliability of synthetic supervision is insufficiently established. Training judgments depend on model-generated references. The text-only GPT teacher cannot check the waveform and is instructed to treat the references as perfect (Figure 7). Consensus and CLAP filtering may reduce noise, but do not establish that fine acoustic details are correct and complete. A correct detail omitted from the references could therefore be penalized. I acknowledge the evaluation against human judgments, but an independent audit of the synthetic references and teacher labels would strengthen the claimed reliability of fine-grained evaluation.

Q1: Have synthetic references and teacher judgments been checked against the audio? What proportion contains incorrect or missing details, and how do these affect the resulting evaluation labels?


We thank the reviewer for pointing out the importance of validating synthetic supervision for the more reliable synthesis pipeline. To address concerns, we conducted a human validation study where annotators listen to audio directly and validate synthetic references and teacher judgements.

1. Synthetic Reference Validation: To validate the synthetic references, we selected 100 audio samples, each evaluated by three human annotators. Annotators independently assessed the factual accuracy and completeness of each reference using binary Yes/No judgments. Based on majority judgments, 96% of synthetic references were factually accurate, 93% were complete, and 90% satisfied both conditions. Additionally, in our human pairwise evaluation, synthetic references outperformed captions generated by the five constituent captioning models used in their construction, achieving a 75.4% win rate (excluding ties) and an average score advantage of 0.808 points. In a separate exploratory comparison against 28 human-written captions, they achieved a 92.6% win rate (excluding ties).


Table 1. Human pairwise evaluation of synthetic references against five constituent captioning models.

ModelPairsW / T / LWin Rate (%)Synthetic Ref (Mean ± SD)Model (Mean ± SD)Δ
AF Next10073 / 11 / 1682.03.93 ± 0.692.84 ± 0.86+1.090***
Gemini 3 Flash10061 / 14 / 2570.93.82 ± 0.653.31 ± 0.86+0.507**
MOSS Audio 8B10068 / 15 / 1780.03.93 ± 0.692.70 ± 1.10+1.230***
Qwen3 Omni10053 / 22 / 2567.93.69 ± 0.583.40 ± 0.63+0.297*
Step Audio R1.110067 / 11 / 2275.33.80 ± 0.612.88 ± 0.93+0.917***
Total500322 / 73 / 10575.43.83 ± 0.343.03 ± 0.42+0.808

W = Synthetic Ref win; T = Tie; L = Comparator win. Win rate = W / (W + L), excluding ties. Δ = Synthetic Ref score − comparator score. Each pair was evaluated by three raters.


Table 2. Exploratory comparison against selected human-written captions.

ComparatorPairsW / T / LWin Rate (%)Synthetic Ref (Mean ± SD)Human (Mean ± SD)Δ
Human captions2825 / 1 / 292.64.05 ± 0.492.49 ± 0.75+1.560***

The exploratory comparison used 14 AudioCaps and 14 Clotho-v2 samples.

AuthorsOpenReview ↗

(W2, Q1) Reliability of synthetic supervison (2/2)

2. Teacher (GPT-5-mini) Judgment Validation: As detailed in our response to W1, human evaluators assessed the teacher-generated rationales against the audio, yielding validity rates of 90% for Sound Event, 88% for Source, 83% for Acoustic Attribute, and 86% for Temporal Relation, with 95% validity for the overall rationale. Teacher scores also achieved a Spearman correlation of 0.672 with human mean scores. To contextualize this correlation, we evaluated CLAP-based methods and CAF-Score variants on the same 100 samples against the same human mean scores. CLAP-based baselines achieved correlations of 0.358–0.390, while CAF-Score variants achieved 0.419–0.424. The teacher’s higher correlation indicates that its scores more closely reflect human rankings than these baselines under our evaluation protocol, providing additional support for the quality of the score supervision.

ModelSpearman ρ
LAION-CLAP0.358
M2D-CLAP0.390
Qwen3-Omni-Instruct+LAION-CLAP0.419
Qwen3-Omni-Instruct+M2D-CLAP0.424
Audio-Compass0.649
Teacher(GPT-5-mini) label0.672

3. Teacher Judgment Reliability and Reference Errors or Omissions: In our validation of 100 audio samples, 95% of overall rationales were judged Valid after aggregating three annotators’ judgments per sample. Across the four components and Overall, 88.4% of the 500 aggregated rationale judgments were Valid, 6.2% were Uncertain, and 5.4% were Invalid. These results indicate a low rate of Invalid rationales within the audited sample.

Reference errors or omissions were identified in 10 of the 100 samples, while the remaining 90 satisfied both accuracy and completeness criteria. Even among these 10 samples, each component had only one to three Invalid rationale judgments, and nine of the ten overall rationales were Valid. Thus, reference errors or omissions were not always accompanied by invalid teacher rationales.

Taken together, the high reference accuracy and completeness, 95% overall rationale validity, and Audio-Compass’s performance on PAM and BRACE provide evidence supporting the practical utility and reliability of the current dataset as training supervision.

As the author–reviewer discussion period is coming to a close, we would appreciate hearing any remaining concerns you may have. We would be happy to provide further clarification. Thank you again for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Additional Experiment : Compass-Hallucination (New Benchmark)

1. Motivation

Existing holistic benchmarks provide limited evidence about sensitivity to individual semantic errors, and BRACE-Hallucination mainly tests Source substitutions. To our knowledge, no existing benchmark systematically evaluates hallucination sensitivity across the remaining three components: Sound Event, Acoustic Attribute, and Temporal Relation. We therefore constructed Compass-Hallucination to extend evaluation to Sound Event, Acoustic Attribute, and Temporal Relation errors.

2. Construction and quality control

(1) Audio and Human Caption Selection. We used 2,027 high-quality audio clips previously selected for BRACE-Hallucination: 765 from AudioCaps and 1,262 from Clotho. For each audio clip, we chose the longest of the five human captions as the positive caption (H).

(2) Component-wise Negative Generation. We used GPT-5-mini to independently generate a negative caption (N) for each of three components: Sound Event, Acoustic Attribute, and Temporal Relation. All three negatives were generated from the same H. Each perturbation was designed to change only the target component while keeping all other semantic information unchanged. The model was instructed to return “none” if the target component was absent or a valid perturbation was not possible. For example, an Acoustic Attribute perturbation changes “playing a guitar loudly” to “playing a guitar softly.” Out of 6,081 generation attempts, we obtained 5,221 negative captions and excluded 860 “none” outputs.

(3) Quality Control. We used GPT-5-mini to evaluate H and N independently against the same human reference set (R*, excluding H). Without access to the audio, the model produced a rationale and a score from 0 to 100 for each component and for Overall. We kept only pairs that met both conditions below:

  • Target-component score drop: s_target(H) − s_target(N) ≥ 20
  • Overall score drop: s_overall(H) − s_overall(N) > 0

(4) Final Benchmark. After filtering, we retained 2,813 pairs: 1,053 for Sound Event, 624 for Acoustic Attribute, and 1,136 for Temporal Relation. We then separately added 2,026 Source pairs from the existing BRACE-Hallucination benchmark. The final benchmark contains 4,839 pairs across four components.

3. Evaluation protocol and results

We report pairwise accuracy (%): a prediction is correct when the evaluator assigns H a higher overall score than N. Macro Avg. is the unweighted mean across the four components. The table reports all six evaluated methods; the two Qwen3-Omni-Instruct + CLAP variants are CAF-Score baselines.

Model (Method)Sound EventSourceAcoustic AttributeTemporal RelationMacro Avg.
M2D-CLAP85.3885.4966.3561.4474.66
LAION-CLAP83.6786.4369.3961.0075.12
Qwen3-Omni-Instruct (FLEUR)90.6097.7876.2874.9184.89
Qwen3-Omni-Instruct (FLEUR) + LAION-CLAP90.5095.7175.9671.3983.39
Qwen3-Omni-Instruct (FLEUR) + M2D-CLAP93.9296.9977.0875.0085.75
Audio-Compass (FLEUR)91.3696.5481.2579.9387.27

Compass-Hallucination are evaluated using pairwise accuracy (%)

Audio-Compass achieves the highest Acoustic Attribute accuracy (81.25%), Temporal Relation accuracy (79.93%), and Macro Avg. (87.27%) among the evaluated methods. These exceed the strongest baseline for each metric by 4.17, 4.93, and 1.52 percentage points, respectively. Baselines remain stronger on Sound Event and Source, so the gains are concentrated in attribute and temporal error detection.

In the revised manuscript, we will add the construction process, filtering criteria, and results of Compass-Hallucination. We will also clearly explain the difference between these component-level strengths and the results on existing benchmarks. Furthermore, we will publicly release the Compass-Hallucination benchmark to support reproducibility and future research.

AuthorsOpenReview ↗

General Response Summary for AC

We thank all three reviewers for their constructive feedback.

Reviewers highlighted the practical value of reference-free component-wise evaluation and our datasets (Rxnj, pgfN), promising agreement with human judgments (Rxnj), and competitive performance and reproducibility (N8Tw). Their main concerns focused on the reliability of synthetic supervision and fine-grained evaluation (Rxnj), as well as novelty, practical advantages, and baseline coverage (N8Tw, pgfN).

In response, we conducted a human audit of 100 audio samples, each evaluated by three annotators. Synthetic references achieved 96% factual accuracy and 93% completeness, while teacher rationale validity reached 83–90% across components and 95% for Overall. We also introduced Compass-Hallucination to assess component-specific errors. On this benchmark, Audio-Compass outperformed the strongest baseline by 4.17 and 4.93 percentage points on Acoustic Attribute and Temporal Relation accuracy, respectively. We further analyzed the relationship between component scores and holistic human judgments. Additionally, we expanded our baseline comparisons to include PE-A-Frame, M2D-CLAP, and CAF-Score.

All of these results and analyses have been incorporated into the revised manuscript. We have also carefully addressed the remaining comments and concerns raised by each reviewer in our individual responses. While we have not yet received follow-up feedback from all reviewers, we would be sincerely grateful if the Area Chair could take our detailed rebuttals, additional experiments, and manuscript revisions into consideration when evaluating the extent to which the reviewers' concerns have been addressed. We sincerely appreciate the time and thoughtful consideration of the reviewers and the Area Chair.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.