- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SITAR, a reference-based ASR metric intended to measure how transcription errors alter the situation conveyed by an utterance. Its construction combines deterministic gap delineation, LLM-assisted severity annotation, and a bounded aggregation rule that prioritizes severity over error count. A DeBERTa regressor learns the resulting scores from 12,347 transcript pairs and requires no LLM calls at inference. Evaluation covers seven ASR systems on three English datasets, regression ablations, and 330 controlled variants from 30 references. The results demonstrate agreement with constructed targets and expected severity orderings, although independent validation of practical consequences remains limited.
Critical strengths
The strongest aspect is the separation of expensive supervision construction from routine scoring. The resulting metric offers deterministic, single-pass evaluation, while the appendix documents its training objectives, ablations, and variation across five seeds. This makes the proposed approach technically concrete and potentially useful for repeated ASR evaluation. The controlled edit construction is also valuable: holding the reference and individual edits fixed allows severity and error accumulation to be examined without confounding changes in surrounding text.
Critical weaknesses
The central validity claim is not sufficiently established by independent human judgments. Annotators label individual gaps, but the expected transcript ordering is subsequently derived from the paper’s own tier-priority aggregation rule rather than elicited from humans. Initial model labels also resolve annotation disagreements. Consequently, the reported 84.07% agreement primarily supports consistency with the proposed rubric, rather than demonstrating that SITAR better captures human assessments of overall situational distortion. The controlled evaluation contains only 30 reference groups, further limiting this conclusion.
The comparative evaluation does not establish the benefit over closely related learned metrics. LASER is discussed but not evaluated, and existing baselines are not compared on the human-annotated controlled set. System-level scaling trends, selected rank reversals, and greater numerical dispersion do not establish superior validity; dispersion alone depends on score scaling. Without a common independent evaluation, it remains unclear whether the proposed construction improves on existing semantic or learned ASR metrics.
Questions for authors
For clarification of validity, how do SITAR and the baselines compare against independent human judgments of complete transcripts, rather than rankings derived from the proposed aggregation rule? Are the reported conclusions preserved when disputed gaps are excluded instead of resolved using initial model labels?
For clarification of evaluation independence, were training, validation, test, and the controlled benchmark separated by reference utterance and source recording? Appendix A.2 describes a pair-level split, although multiple recognizers can produce hypotheses for the same reference. Please clarify whether references or related clips cross these boundaries. These questions concern the reliability of the submitted evaluation, rather than suggestions for future work.
8 comments
(2) Comparing transcripts with identical WER
We next focus on transcripts with identical WER for the same utterance. This yields 1,492 transcripts from 300 LibriSpeech utterances and 1,187 transcripts from 298 VoxPopuli utterances. Each utterance originally has seven candidate transcripts, one from each ASR system. Two VoxPopuli utterances were excluded because all seven candidates had different WER values. We aggregate the human rankings within these groups into a system-level ranking for each dataset. The table reports Spearman’s ρ.
| Dataset | BERTScore | SemDist | AER | LASER | SITAR |
|---|---|---|---|---|---|
| LibriSpeech | 0.4144 | 0.1786 | 0.5357 | 0.4643 | 0.5714 |
| VoxPopuli | 0.3571 | 0.2500 | 0.3571 | 0.3571 | 0.4643 |
SITAR achieves the highest agreement with human judgments on both datasets. Among the compared metrics, it best matches the human-derived system rankings in this setting, demonstrating its usefulness for evaluating error consequences beyond lexical accuracy.
(3) Case study of different consequences at identical WER
We additionally present four examples showing how deleting the same word can have different situational consequences. Each hypothesis deletes one occurrence of “not”, and all four examples have identical WER.
N0: Reason for the flight choice reversed
Reference: I chose the 8:15 flight because I am not worried about the price. Please book the 8:15 refundable ticket.
Hypothesis: I chose the 8:15 flight because I am worried about the price. Please book the 8:15 refundable ticket.
N1: Core flight choice reversed
Reference: The 8:15 flight is available. I have decided not to take the 9:40 flight. Please book the earlier one.
Hypothesis: The 8:15 flight is available. I have decided to take the 9:40 flight. Please book the earlier one.
N2: Secondary availability reversed
Reference: The meeting is on Thursday at three. Maya is not available for lunch, but she will attend the meeting.
Hypothesis: The meeting is on Thursday at three. Maya is available for lunch, but she will attend the meeting.
N3: Meeting attendance reversed
Reference: The meeting is on Thursday at three. Maya is available for lunch, but she will not attend the meeting.
Hypothesis: The meeting is on Thursday at three. Maya is available for lunch, but she will attend the meeting.
| Example | BERTScore F1 ↑ | WER ↓ | SITAR ↓ |
|---|---|---|---|
| N0: Reason for the flight choice | 0.9866 | 0.0526 | 0.6815 |
| N1: Core flight choice | 0.9668 | 0.0526 | 0.7114 |
| N2: Lunch availability | 0.9514 | 0.0526 | 0.8006 |
| N3: Meeting attendance | 0.9534 | 0.0526 | 0.8293 |
In the flight examples, N0 changes the explanation for the choice while preserving the booking request. N1 changes the explicit choice and creates a conflict with the request to book the earlier flight. Both BERTScore and SITAR assign greater severity to N1.
In the meeting examples, N2 changes lunch availability while preserving meeting attendance. N3 reverses attendance itself. BERTScore assigns slightly higher similarity to N3, while SITAR assigns greater severity to the change affecting the meeting itself.
These examples illustrate SITAR’s ability to distinguish an error’s situational consequences even when its lexical operation and WER remain identical.
(4) Evaluation on the controlled benchmark
Following the reviewer’s helpful suggestion, we also obtained independent human ratings of the original complete transcripts. Annotators assessed each transcript as a whole, without applying the aggregation rule.
| Measure | WER | NE-WER | BERTScore F1 | SemDist | AER | LASER | SITAR |
|---|---|---|---|---|---|---|---|
| Agr. | 40.97% | 27.69% | 41.20% | 38.43% | 54.40% | 48.38% | 69.21% |
| Spearman’s ρ | -0.2388 | -0.1373 | -0.2475 | -0.3245 | 0.2440 | 0.0114 | 0.5488 |
We observe positive rank correlations only for the three LLM-based approaches: AER, LASER, and SITAR. We find this clear contrast with traditional metrics encouraging. It suggests that LLMs make it possible to assess aspects of error severity that traditional metrics struggle to capture. The strength of this association varies across the three approaches, with LASER’s rank correlation close to zero. SITAR achieves considerably higher pairwise agreement and rank correlation than both AER and LASER.
Q1.2 Results after excluding disputed annotations
Following the reviewer’s suggestion, we retained only controlled variants for which the two annotators agreed on every included gap. This leaves 268 of the original 330 pairs, without using initial model labels to resolve disagreements.
| Measure | WER | NE-WER | BERTScore F1 | SemDist | AER | LASER | SITAR |
|---|---|---|---|---|---|---|---|
| Agr. (330 pairs) | 79.69% | 17.27% | 84.96% | 83.49% | 56.26% | 67.47% | 84.07% |
| Spearman’s ρ (330 pairs) | 0.8373 | 0.4503 | 0.7895 | 0.7637 | 0.6805 | 0.6086 | 0.8001 |
| Agr. (268 pairs) | 81.61% | 9.55% | 86.24% | 84.67% | 50.91% | 71.45% | 87.15% |
| Spearman’s ρ (268 pairs) | 0.8484 | 0.3464 | 0.8036 | 0.7752 | 0.6498 | 0.6438 | 0.8385 |
We applied the same criterion to the 30 original complete transcripts, retaining 21 transcripts with agreement on every gap. We evaluated this subset against the independent human ratings of complete transcripts.
| Measure | WER | NE-WER | BERTScore F1 | SemDist | AER | LASER | SITAR |
|---|---|---|---|---|---|---|---|
| Agr. (30 transcripts) | 40.97% | 27.69% | 41.20% | 38.43% | 54.40% | 48.38% | 69.21% |
| Spearman’s ρ (30 transcripts) | -0.2388 | -0.1373 | -0.2475 | -0.3245 | 0.2440 | 0.0114 | 0.5488 |
| Agr. (21 transcripts) | 44.02% | 20.45% | 41.63% | 38.28% | 49.76% | 56.46% | 69.86% |
| Spearman’s ρ (21 transcripts) | -0.1286 | -0.2254 | -0.1832 | -0.3527 | 0.0778 | 0.2531 | 0.5645 |
SITAR improves on both measures in both evaluations after excluding disputed annotations. It also retains its lead over all compared baselines in the complete-transcript evaluation. We believe these results provide reassuring evidence that the reported conclusions are preserved without model-based resolution of annotation disagreements.
Answer for Q2
We sincerely thank the reviewer for this question. Our study uses several datasets for different purposes. We first clarify their roles and introduce consistent names to make the discussion easier to follow.
We use sitar_train_data, sitar_val_data, and sitar_test_data to train, validate, and test the fine-tuned DeBERTa regressor, respectively. The examples in sitar_train_data, sitar_val_data, and sitar_test_data are drawn primarily from AppTek.
AppTek contains long recordings. We first segmented the recordings at silence intervals into clips of no more than 30 seconds. As described in the manuscript, we split individual reference–hypothesis pairs rather than grouping the pairs by reference transcript. Some reference transcripts are therefore shared across sitar_train_data, sitar_val_data, and sitar_test_data. We sincerely apologize for not sufficiently explaining this aspect of the split in the original manuscript.
We would also like to clarify the role of the internal evaluation. We used sitar_val_data for model development and checkpoint selection, while sitar_test_data was used for reporting the regression results. The corresponding regression results and fine-tuning details are reported in the appendix, rather than the main text. The main evidence for SITAR’s effectiveness comes from the external ASR evaluation in Table 1, not from results on sitar_test_data.
Table 1 reports results on fleurs_en_test, LibriSpeech, and VoxPopuli. Neither LibriSpeech nor VoxPopuli was included in sitar_train_data, sitar_val_data, or sitar_test_data. LibriSpeech and VoxPopuli therefore provide an evaluation on datasets unseen during SITAR’s development. We believe the results on LibriSpeech and VoxPopuli provide strong, independent evidence of SITAR’s reliability, as both are out-of-domain datasets entirely unseen during SITAR’s development.
Answer for Q1
Q1.1 Independent Human Evaluation of Complete Transcripts
We sincerely thank the reviewer for this valuable suggestion. SITAR explores ASR errors from the perspective of situational distortion. Our original evaluation combined human annotations of individual gaps with controlled comparisons. These tests examined how SITAR responds to different error types and combinations. We appreciate the reviewer’s guidance on extending this analysis to independent human judgments of complete transcripts.
Following the reviewer’s suggestion, we conducted an additional human evaluation with five PhD students. We randomly sampled 300 utterances from LibriSpeech and 300 from VoxPopuli. The annotators assessed the complete ASR transcripts directly, without using the proposed aggregation rule. We now report the agreement of SITAR and all baseline metrics with these human judgments.
We first report (1) overall agreement with human judgments on real ASR outputs and (2) comparisons of transcripts with identical WER. We then present (3) a case study illustrating different situational consequences at identical WER. Finally, we report (4) direct human evaluation of the 30 original complete transcripts.
(1) Agreement with human judgments on real ASR outputs
We first report system-level agreement with human judgments on real ASR outputs. For each dataset, we compare the ranking of the seven ASR systems with the ranking obtained from human annotations. The table reports Spearman’s ρ.
WER achieves the highest correlation on LibriSpeech and shares the highest correlation on VoxPopuli. We believe two factors may help explain its strong performance. First, ASR training primarily optimizes predictions against reference token sequences, making word-level accuracy a central objective. This alignment may contribute to the strong performance of WER as an evaluation metric. Second, word-level transcription errors, including substitutions, deletions, and insertions, directly reflect deviations from the reference transcript, which may also influence human judgments of transcription quality. These findings provide empirical support for the continued effectiveness of WER, which has long served as the de facto standard for ASR evaluation.
The comparison between AER and SITAR is particularly informative. On VoxPopuli, AER is the only evaluated metric with a negative correlation with human judgments (−0.3214). SITAR, in contrast, achieves 0.9643. This finding helps explain the disagreement reported in our paper, where the correlation between SITAR and AER was −0.11. The additional human evidence supports our diagnostic interpretation by showing that SITAR’s system ordering closely follows human judgments.
The strengths of SITAR may not be fully apparent in this overall comparison. This is in line with our expectations, since SITAR was designed to complement WER rather than replace it. We believe its relatively high agreement with human judgments across both datasets already provides strong support for its effectiveness. We next examine its advantages in cases where WER alone cannot distinguish between transcripts.