- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Audio-Compass, a reference-free audio–text evaluator that produces an overall alignment score and component-wise judgments for sound events, sources, acoustic attributes, and temporal relations. Its training pipeline first uses multiple audio-language models to describe each clip, consolidates their outputs into textual anchors, and prompts GPT-5-mini to assess candidate captions against those anchors. The resulting fine-grained scores and explanations are distilled into an audio judge, which at inference time evaluates the audio and candidate caption directly, without reference captions. The authors also construct Compass-105k for training and introduce BRACE-Long to assess preferences between long-form captions. The work addresses a meaningful limitation of holistic metrics and offers a potentially useful evaluation pipeline and resource. However, the results do not establish consistent superiority: LALM + CLAP baselines often outperform Audio-Compass. The work’s main contribution therefore appears to be diagnostic, component-wise evaluation rather than clear gains in overall accuracy. The paper also does not sufficiently isolate the benefit of Compass-105k fine-tuning or its dependence on training-set scale, rule out overlap with external benchmarks. Overall, the work is promising as an evaluation framework and dataset contribution, but its claims require stronger ablations and data contamination analysis.
Critical strengths
-
The paper targets an important practical problem: holistic alignment scores offer little diagnosis of what a caption gets right or wrong.
-
The study covers the full process from data collection and reference synthesis to teacher annotation, student-model training, and evaluation on multiple benchmarks.
-
Reference captions are used only during training-data construction. At deployment time, the evaluator directly takes an audio clip and a candidate caption as input, making it suitable for real-world scenarios.
-
The planned release of Compass-105k could substantially benefit the community by providing a source of audio–text data for fine-tuning and evaluation.
Critical weaknesses
-
The LALM + CLAP systems outperform Audio-Compass in several experiments. Audio-Compass is stronger in some settings, notably PAM, and offers richer component-wise outputs, but the paper does not sufficiently explain why its additional fine-grained modeling does not translate into consistent gains in overall accuracy. Its main advantage therefore appears to be diagnostic functionality rather than clearly superior performance.
-
There is no ablation study examining how Audio-Compass’s capabilities depend on the scale and composition of Compass-105k. In particular, the reported performance of Audio-Compass is often close to that of the original Qwen3-Omni-Instruct baseline, making it unclear where the actual benefit of SFT comes from. The paper should analyze performance as a function of training-set size and provide stronger comparisons across different subsets or target formats.
-
The paper does not provide a data leak analysis. Although Compass-105k is split by audio ID to prevent overlap among its internal training, validation, and test sets, this does not establish independence from the external evaluation benchmarks. Compass-105k is constructed from sources including AudioCaps and Clotho-v2, while BRACE, BRACE-Long, and PAM may contain audio derived from these collections.
-
Tables 1-4 evaluate different sets of models. The inclusion and exclusion criteria for reference-free models do not clearly explain why certain baselines are omitted from particular benchmarks or whether their exclusion is technically justified.
Questions for authors
-
Does Compass-105k contain any audio, captions, or metadata that overlap with the evaluation benchmarks? If there is any data leakage, please report exact overlap statistics between the Compass-105k training split and each benchmark.
-
How does Audio-Compass performance depend on the scale and composition of Compass-105k? Please provide controlled experiments using different training-set sizes and supervision subsets, together with a comparison to the original Qwen3-Omni-Instruct model, to clarify the actual benefit of SFT and whether performance saturates with the current data scale.
-
Why do the reference-free baselines differ across Tables 1–4? Beyond the stated inapplicability of reference-based metrics to some benchmarks, please explain the inclusion and exclusion criteria for each CLAP-based and LALM-based evaluator.
5 comments
(W2, Q2) Experiments on training data scaling
W2: There is no ablation study examining how Audio-Compass’s capabilities depend on the scale and composition of Compass-105k. In particular, the reported performance of Audio-Compass is often close to that of the original Qwen3-Omni-Instruct baseline, making it unclear where the actual benefit of SFT comes from. The paper should analyze performance as a function of training-set size and provide stronger comparisons across different subsets or target formats.
Q2: How does Audio-Compass performance depend on the scale and composition of Compass-105k? Please provide controlled experiments using different training-set sizes and supervision subsets, together with a comparison to the original Qwen3-Omni-Instruct model, to clarify the actual benefit of SFT and whether performance saturates with the current data scale.
We thank the reviewer for requesting a more detailed analysis of the contribution of Compass-105k. To examine the benefit of SFT as the number of processed training samples increases, we evaluated three checkpoints from the Audio-Compass training run and compared them with the original Qwen3-Omni-Instruct model.
Audio-Compass is trained on 104,401 samples derived from score and rationale annotations. We evaluated the checkpoints using the same evaluation prompt on PAM and BRACE-Main, reporting both text score (Raw) and FLEUR scores. Text scores are directly extracted from the model's generated text, whereas FLEUR scores are obtained through a post-processing method that smooths the scores using text token probabilities.
| Model | Trainset Size | PAM SRCC (Text Score / FLEUR) | BRACE-Main Accuracy, % (Text Score / FLEUR) |
|---|---|---|---|
| Qwen3-Omni-Instruct, before SFT | — | 0.384 / 0.426 | 44.51 / 68.31 |
| Audio-Compass | 38.35k (36.73%) | 0.586 / 0.587 | 59.50 / 68.75 |
| Audio-Compass | 76.69k (73.46%) | 0.625 / 0.624 | 65.99 / 71.75 |
| Audio-Compass | 104.40k (100%) | 0.636 / 0.628 | 64.74 / 71.19 |
In the table, Trainset Size denotes the estimated number of samples processed at each checkpoint, corresponding to 36.73%, 73.46%, and 100% of the full 104,401-example training set, respectively.
These results demonstrate the benefit of evaluation-specific SFT over the original Qwen3-Omni-Instruct model. At the final checkpoint (104.40k), Audio-Compass improves PAM SRCC by 0.252 / 0.202 and BRACE-Main accuracy by 20.23 / 2.88 percentage points under Raw / FLEUR scoring, respectively. The larger improvements under Raw scoring, particularly on BRACE-Main, suggest that SFT improves the model's task-specific scoring behavior. In contrast, FLEUR already improves the baseline's BRACE-Main accuracy from 44.51% to 68.31% through score-token probability-based post-processing, leaving a smaller margin for additional gains.
Regarding training scale, PAM SRCC improves consistently, but gains from 76.69k to 104.40k are small (+0.011 / +0.004). Meanwhile, BRACE-Main reaches its highest accuracy at 76.69k (65.99% / 71.75%) and slightly declines at 104.40k (64.74% / 71.19%). These results indicate diminishing returns during training and confirm the reviewer's concern that additional training does not consistently improve performance across benchmarks. While checkpoint comparisons cannot establish dataset-size saturation or isolate the effects of data composition, we will include these findings in the revised manuscript to clarify the contribution and limitations of Compass-105k.
(W3, Q1) Data Leakage Analysis
W3: The paper does not provide a data leak analysis. Although Compass-105k is split by audio ID to prevent overlap among its internal training, validation, and test sets, this does not establish independence from the external evaluation benchmarks. Compass-105k is constructed from sources including AudioCaps and Clotho-v2, while BRACE, BRACE-Long, and PAM may contain audio derived from these collections.
Q1: Does Compass-105k contain any audio, captions, or metadata that overlap with the evaluation benchmarks? If there is any data leakage, please report exact overlap statistics between the Compass-105k training split and each benchmark.
We appreciate you drawing attention to potential overlap between the training data and evaluation benchmarks. In our additional analysis, we found some overlaps in original recording IDs between the benchmarks and our training data sources.
For datasets with official splits, including AudioCaps, Clotho-v2, FSD50K, CochlScene, and TUT Urban, we used only the training splits (the development split for Clotho-v2). The train/valid/test splits mentioned in our paper refer to internal splits created from the collected data for model training and internal evaluation. We omit to explain the difference between these internal splits and the official dataset splits.
However, even when we follow the official splits, the same original recordings may still appear in the benchmarks through other datasets that do not have official splits. Therefore, we defined audio clip overlap as cases where both the original recording ID and audio duration matched exactly. We also checked the official source interval metadata to verify these matches.
| Benchmark | Exact Audio Clip Overlap | Normalized Caption Overlap |
|---|---|---|
| RELATE-REL | 2 / 437 (0.46%) | 2.09–2.32% |
| RELATE-IS | 2 / 778 (0.26%) | 1.94–2.07% |
| RELATE-OS | 0 / 395 (0.00%) | 0.00% |
| BRACE-Main | 3 / 575 (0.52%) | 0.82–0.85% |
| BRACE-Hallucination | 4 / 2,027 (0.20%) | 0.25–0.27% |
| BRACE-Long | 0 / 129 (0.00%) | 0.00% |
| PAM | 0 / 100 (0.00%) | 3.03–4.04% |
For audio overlap, the denominator is the number of real audio clips in each benchmark. For caption overlap, we compared captions in the training set with benchmark captions after normalizing capitalization, punctuation, and whitespace. The denominator is the number of unique normalized captions in each benchmark. However, matching caption text alone does not necessarily indicate audio leakage. For example, we found that the same caption, "A woman is speaking," was used for audio clips with different original recording IDs.
Through this audit, we confirmed some overlaps with the benchmarks. In the revised manuscript, we will report the overlap counts for each benchmark.
(W1) Performance gains and Compass-Hallucination
W1: The LALM + CLAP systems outperform Audio-Compass in several experiments. Audio-Compass is stronger in some settings, notably PAM, and offers richer component-wise outputs, but the paper does not sufficiently explain why its additional fine-grained modeling does not translate into consistent gains in overall accuracy. Its main advantage therefore appears to be diagnostic functionality rather than clearly superior performance.
We appreciate your thoughtful feedback on the benefits of fine-grained evaluation. We agree that LALM + CLAP hybrid evaluation method (CAF-Score) performs better on most benchmarks. To check whether the benefits of Audio-Compass go beyond rationale outputs, we conducted an additional evaluation measuring its ability to distinguish specific caption errors.
Compass-Hallucination compares a human caption with a negative caption containing a component-specific semantic error. Pairwise accuracy measures whether the evaluator assigns the human caption a higher overall score; Macro Avg. is the unweighted mean across four components. Please see our Official Comment, “Compass-Hallucination,” for construction details, filtering criteria, and full results.
Audio-Compass achieved 81.25% on Acoustic Attribute, 79.93% on Temporal Relation, and 87.27% Macro Avg., exceeding the strongest baseline for each metric by 4.17, 4.93, and 1.52 percentage points, respectively. Baselines remained stronger on Sound Event and Source. These results demonstrate measurable gains in detecting attribute and temporal errors, while also showing that the gains are component-dependent.
One possible explanation for the inconsistent gains on existing benchmarks is that holistic human ratings do not reflect all semantic components equally. In RELATE (REL; Section 6.1), Sound Event scores correlated more strongly with human ratings than Overall scores (SRCC: 0.450 vs. 0.428), while Acoustic Attribute and Temporal Relation scores(SRCC: 0.029 and 0.178) showed lower correlations. This pattern suggests that improved detection of attribute and temporal errors may not translate directly into higher agreement with holistic ratings, although it does not establish the cause of the performance gap.