acceptodds
Under review as a conference paper at ICLR 2027

SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation

Abstract

In automatic speech recognition (ASR), transcription errors with similar lexical or semantic discrepancies can have markedly different consequences for the situation conveyed by speech. Capturing these differences requires not merely contextual assessment, but an understanding of what the spoken discourse describes, establishes, or seeks to bring about. However, relying on LLM judges introduces computational cost and scoring variability. We introduce SITAR, a situational distortion metric for ASR, realized through a comprehensive end-to-end construction framework. Specifically, this framework aggregates situational severity and cumulative extent into a unified score via a bounded recursive formulation. Using targets derived from this formulation with limited offline LLM support, we train SITAR as a lightweight BERT-family regressor that scores reference–hypothesis pairs in a single forward pass, entirely free of any prohibitive runtime costs. Recognizing that introducing a new metric demands rigorous justification, we formalize a principled validation protocol spanning both extrinsic validity, assessed at macro and micro levels, and intrinsic scorer reliability. The former, evaluated across diverse datasets and models, demonstrates that SITAR reliably tracks broad capability scaling while providing orthogonal diagnostic utility, whereas the latter, examined on a purpose-built controlled benchmark, confirms the operational soundness of the scoring instrument itself. Through systematic validation, we demonstrate that SITAR holds strong potential as a practical, lightweight complement to existing ASR metrics.

Primary area:
datasets and benchmarks
Keywords:
Automatic Speech Recognition, Evaluation Metrics, Situational Distortion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_C1dbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SITAR, a reference-based ASR metric intended to measure how transcription errors alter the situation conveyed by an utterance. Its construction combines deterministic gap delineation, LLM-assisted severity annotation, and a bounded aggregation rule that prioritizes severity over error count. A DeBERTa regressor learns the resulting scores from 12,347 transcript pairs and requires no LLM calls at inference. Evaluation covers seven ASR systems on three English datasets, regression ablations, and 330 controlled variants from 30 references. The results demonstrate agreement with constructed targets and expected severity orderings, although independent validation of practical consequences remains limited.

Critical strengths

The strongest aspect is the separation of expensive supervision construction from routine scoring. The resulting metric offers deterministic, single-pass evaluation, while the appendix documents its training objectives, ablations, and variation across five seeds. This makes the proposed approach technically concrete and potentially useful for repeated ASR evaluation. The controlled edit construction is also valuable: holding the reference and individual edits fixed allows severity and error accumulation to be examined without confounding changes in surrounding text.

Critical weaknesses

The central validity claim is not sufficiently established by independent human judgments. Annotators label individual gaps, but the expected transcript ordering is subsequently derived from the paper’s own tier-priority aggregation rule rather than elicited from humans. Initial model labels also resolve annotation disagreements. Consequently, the reported 84.07% agreement primarily supports consistency with the proposed rubric, rather than demonstrating that SITAR better captures human assessments of overall situational distortion. The controlled evaluation contains only 30 reference groups, further limiting this conclusion.

The comparative evaluation does not establish the benefit over closely related learned metrics. LASER is discussed but not evaluated, and existing baselines are not compared on the human-annotated controlled set. System-level scaling trends, selected rank reversals, and greater numerical dispersion do not establish superior validity; dispersion alone depends on score scaling. Without a common independent evaluation, it remains unclear whether the proposed construction improves on existing semantic or learned ASR metrics.

Questions for authors

For clarification of validity, how do SITAR and the baselines compare against independent human judgments of complete transcripts, rather than rankings derived from the proposed aggregation rule? Are the reported conclusions preserved when disputed gaps are excluded instead of resolved using initial model labels?

For clarification of evaluation independence, were training, validation, test, and the controlled benchmark separated by reference utterance and source recording? Appendix A.2 describes a pair-level split, although multiple recognizers can produce hypotheses for the same reference. Please clarify whether references or related clips cross these boundaries. These questions concern the reliability of the submitted evaluation, rather than suggestions for future work.

8 comments

Answer for Q1

Q1.1 Independent Human Evaluation of Complete Transcripts

We sincerely thank the reviewer for this valuable suggestion. SITAR explores ASR errors from the perspective of situational distortion. Our original evaluation combined human annotations of individual gaps with controlled comparisons. These tests examined how SITAR responds to different error types and combinations. We appreciate the reviewer’s guidance on extending this analysis to independent human judgments of complete transcripts.

Following the reviewer’s suggestion, we conducted an additional human evaluation with five PhD students. We randomly sampled 300 utterances from LibriSpeech and 300 from VoxPopuli. The annotators assessed the complete ASR transcripts directly, without using the proposed aggregation rule. We now report the agreement of SITAR and all baseline metrics with these human judgments.

Before presenting the results, we would like to offer two clarifications.

Inclusion of LASER. We thank the reviewer for highlighting the importance of this comparison. LASER was originally developed for Hindi ASR evaluation, which informed our initial choice of baselines. Following the reviewer’s suggestion, we have now evaluated LASER in English using its publicly available implementation. Its results are included in the comparisons below.

LASER scores (higher is better):

Datasetwhisper-tinywhisper-basewhisper-smallwhisper-mediumwhisper-largewav2vec2-basewav2vec2-large
FLEURS0.92220.95060.97320.98060.98270.92960.9461
LibriSpeech0.95800.97650.98750.99020.99420.98610.9902
VoxPopuli0.92050.93810.94830.95340.93890.87120.8952

Dataset selection for human evaluation. SITAR’s training data include FLEURS, while our FLEURS evaluation uses the official test split. For the additional human evaluation, we focus exclusively on LibriSpeech and VoxPopuli. Neither dataset was used to train SITAR. This allows us to examine generalization to external datasets through independent human judgments. We discuss dataset separation in detail in our response to Q2.

We first report (1) overall agreement with human judgments on real ASR outputs and (2) comparisons of transcripts with identical WER. We then present (3) a case study illustrating different situational consequences at identical WER. Finally, we report (4) direct human evaluation of the 30 original complete transcripts.

(1) Agreement with human judgments on real ASR outputs

We first report system-level agreement with human judgments on real ASR outputs. For each dataset, we compare the ranking of the seven ASR systems with the ranking obtained from human annotations. The table reports Spearman’s ρ.

DatasetWERNE-WERBERTScoreSemDistAERLASERSITAR
LibriSpeech1.00000.92860.57660.82140.75000.96430.8214
VoxPopuli0.96430.88290.96430.8929-0.32140.96430.9643

WER achieves the highest correlation on LibriSpeech and shares the highest correlation on VoxPopuli. We believe two factors may help explain its strong performance. First, ASR training primarily optimizes predictions against reference token sequences, making word-level accuracy a central objective. This alignment may contribute to the strong performance of WER as an evaluation metric. Second, word-level transcription errors, including substitutions, deletions, and insertions, directly reflect deviations from the reference transcript, which may also influence human judgments of transcription quality. These findings provide empirical support for the continued effectiveness of WER, which has long served as the de facto standard for ASR evaluation.

The comparison between AER and SITAR is particularly informative. On VoxPopuli, AER is the only evaluated metric with a negative correlation with human judgments (−0.3214). SITAR, in contrast, achieves 0.9643. This finding helps explain the disagreement reported in our paper, where the correlation between SITAR and AER was −0.11. The additional human evidence supports our diagnostic interpretation by showing that SITAR’s system ordering closely follows human judgments.

The strengths of SITAR may not be fully apparent in this overall comparison. This is in line with our expectations, since SITAR was designed to complement WER rather than replace it. We believe its relatively high agreement with human judgments across both datasets already provides strong support for its effectiveness. We next examine its advantages in cases where WER alone cannot distinguish between transcripts.

(2) Comparing transcripts with identical WER

We next focus on transcripts with identical WER for the same utterance. This yields 1,492 transcripts from 300 LibriSpeech utterances and 1,187 transcripts from 298 VoxPopuli utterances. Each utterance originally has seven candidate transcripts, one from each ASR system. Two VoxPopuli utterances were excluded because all seven candidates had different WER values. We aggregate the human rankings within these groups into a system-level ranking for each dataset. The table reports Spearman’s ρ.

DatasetBERTScoreSemDistAERLASERSITAR
LibriSpeech0.41440.17860.53570.46430.5714
VoxPopuli0.35710.25000.35710.35710.4643

SITAR achieves the highest agreement with human judgments on both datasets. Among the compared metrics, it best matches the human-derived system rankings in this setting, demonstrating its usefulness for evaluating error consequences beyond lexical accuracy.

(3) Case study of different consequences at identical WER

We additionally present four examples showing how deleting the same word can have different situational consequences. Each hypothesis deletes one occurrence of “not”, and all four examples have identical WER.

N0: Reason for the flight choice reversed

Reference: I chose the 8:15 flight because I am not worried about the price. Please book the 8:15 refundable ticket.

Hypothesis: I chose the 8:15 flight because I am worried about the price. Please book the 8:15 refundable ticket.


N1: Core flight choice reversed

Reference: The 8:15 flight is available. I have decided not to take the 9:40 flight. Please book the earlier one.

Hypothesis: The 8:15 flight is available. I have decided to take the 9:40 flight. Please book the earlier one.


N2: Secondary availability reversed

Reference: The meeting is on Thursday at three. Maya is not available for lunch, but she will attend the meeting.

Hypothesis: The meeting is on Thursday at three. Maya is available for lunch, but she will attend the meeting.


N3: Meeting attendance reversed

Reference: The meeting is on Thursday at three. Maya is available for lunch, but she will not attend the meeting.

Hypothesis: The meeting is on Thursday at three. Maya is available for lunch, but she will attend the meeting.

ExampleBERTScore F1 ↑WER ↓SITAR ↓
N0: Reason for the flight choice0.98660.05260.6815
N1: Core flight choice0.96680.05260.7114
N2: Lunch availability0.95140.05260.8006
N3: Meeting attendance0.95340.05260.8293

In the flight examples, N0 changes the explanation for the choice while preserving the booking request. N1 changes the explicit choice and creates a conflict with the request to book the earlier flight. Both BERTScore and SITAR assign greater severity to N1.

In the meeting examples, N2 changes lunch availability while preserving meeting attendance. N3 reverses attendance itself. BERTScore assigns slightly higher similarity to N3, while SITAR assigns greater severity to the change affecting the meeting itself.

These examples illustrate SITAR’s ability to distinguish an error’s situational consequences even when its lexical operation and WER remain identical.

(4) Evaluation on the controlled benchmark

Following the reviewer’s helpful suggestion, we also obtained independent human ratings of the original complete transcripts. Annotators assessed each transcript as a whole, without applying the aggregation rule.

MeasureWERNE-WERBERTScore F1SemDistAERLASERSITAR
Agr.40.97%27.69%41.20%38.43%54.40%48.38%69.21%
Spearman’s ρ-0.2388-0.1373-0.2475-0.32450.24400.01140.5488

We observe positive rank correlations only for the three LLM-based approaches: AER, LASER, and SITAR. We find this clear contrast with traditional metrics encouraging. It suggests that LLMs make it possible to assess aspects of error severity that traditional metrics struggle to capture. The strength of this association varies across the three approaches, with LASER’s rank correlation close to zero. SITAR achieves considerably higher pairwise agreement and rank correlation than both AER and LASER.

Q1.2 Results after excluding disputed annotations

Following the reviewer’s suggestion, we retained only controlled variants for which the two annotators agreed on every included gap. This leaves 268 of the original 330 pairs, without using initial model labels to resolve disagreements.

MeasureWERNE-WERBERTScore F1SemDistAERLASERSITAR
Agr. (330 pairs)79.69%17.27%84.96%83.49%56.26%67.47%84.07%
Spearman’s ρ (330 pairs)0.83730.45030.78950.76370.68050.60860.8001
Agr. (268 pairs)81.61%9.55%86.24%84.67%50.91%71.45%87.15%
Spearman’s ρ (268 pairs)0.84840.34640.80360.77520.64980.64380.8385

We applied the same criterion to the 30 original complete transcripts, retaining 21 transcripts with agreement on every gap. We evaluated this subset against the independent human ratings of complete transcripts.

MeasureWERNE-WERBERTScore F1SemDistAERLASERSITAR
Agr. (30 transcripts)40.97%27.69%41.20%38.43%54.40%48.38%69.21%
Spearman’s ρ (30 transcripts)-0.2388-0.1373-0.2475-0.32450.24400.01140.5488
Agr. (21 transcripts)44.02%20.45%41.63%38.28%49.76%56.46%69.86%
Spearman’s ρ (21 transcripts)-0.1286-0.2254-0.1832-0.35270.07780.25310.5645

SITAR improves on both measures in both evaluations after excluding disputed annotations. It also retains its lead over all compared baselines in the complete-transcript evaluation. We believe these results provide reassuring evidence that the reported conclusions are preserved without model-based resolution of annotation disagreements.

Answer for Q2

We sincerely thank the reviewer for this question. Our study uses several datasets for different purposes. We first clarify their roles and introduce consistent names to make the discussion easier to follow.

We use sitar_train_data, sitar_val_data, and sitar_test_data to train, validate, and test the fine-tuned DeBERTa regressor, respectively. The examples in sitar_train_data, sitar_val_data, and sitar_test_data are drawn primarily from AppTek.

AppTek contains long recordings. We first segmented the recordings at silence intervals into clips of no more than 30 seconds. As described in the manuscript, we split individual reference–hypothesis pairs rather than grouping the pairs by reference transcript. Some reference transcripts are therefore shared across sitar_train_data, sitar_val_data, and sitar_test_data. We sincerely apologize for not sufficiently explaining this aspect of the split in the original manuscript.

We would also like to clarify the role of the internal evaluation. We used sitar_val_data for model development and checkpoint selection, while sitar_test_data was used for reporting the regression results. The corresponding regression results and fine-tuning details are reported in the appendix, rather than the main text. The main evidence for SITAR’s effectiveness comes from the external ASR evaluation in Table 1, not from results on sitar_test_data.

Table 1 reports results on fleurs_en_test, LibriSpeech, and VoxPopuli. Neither LibriSpeech nor VoxPopuli was included in sitar_train_data, sitar_val_data, or sitar_test_data. LibriSpeech and VoxPopuli therefore provide an evaluation on datasets unseen during SITAR’s development. We believe the results on LibriSpeech and VoxPopuli provide strong, independent evidence of SITAR’s reliability, as both are out-of-domain datasets entirely unseen during SITAR’s development.

Reviewer_2kSkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SITAR, a metric for evaluating automatic speech recognition (ASR). SITAR measures error distortion severity based on the situational changes it can introduce. Its construction pipeline uses Constrained Gap Delineation (CGD) to identify paired discrepancies, called gaps, between reference and hypothesis transcripts. For each gap, a single-gap hypothesis is constructed by fixing all other discrepancies to match the reference. After that, an LLM annotation and review pipeline assigns a distortion tier and directional boundary adjustments to each of those single-gap hypotheses, based on the situational severity of the distortion. Finally, those scores are aggregated into one, notably the most severe error determines the base score. To eliminate LLM usage in inference, previously described targets are used for training a DeBERTa-v3-large regressor with BiLSTM pooling.

Critical strengths

Explicit and clear scoring construction. The paper defines a bounded scoring formula combining error severity with gap counts at each tier. Two independent annotators achieved 90.00% gap-label agreement and Cohen's kappa equal to 0.8396, supporting reproducibility of the coarse rubric on the selected examples.

Substantial implementation evidence. Training a regressor on the constructed dataset allows for deterministic inference that does not rely on LLM calls at runtime. The paper includes ablations of pooling, losses, input representations, encoder adaptation, and hyperparameters.

Critical weaknesses

Independent validity of the proposed severity scale is not established. The annotators labeled 120 single-gap hypotheses, but didn’t rank the 330 complete hypotheses. It was established that SITAR ranking and the ranking induced by aggregating the annotator-labeled gaps using the specific rule defined in the paper mostly agree with each other. But the paper does not show whether annotators agree with SITAR ranking itself.

Table 1 shows the evaluation of ASR systems using different metrics and compares them, but does not show which metric better reflects the error severity. No baseline metric is evaluated against the controlled ranking. On top of that, the AER scores, as mentioned in the paper, were imported from a study using different evaluation samples. So the differences between AER and SITAR could partly be a result of that inconsistency.

Questions for authors
  1. Can you compare SITAR and baseline metrics against independent human evaluation of complete hypotheses, without using the proposed aggregation rule to determine the expected ranking?

  2. Were all the hypotheses that share the same reference kept in the same data split, meaning that they were all in either training, validation or test split?

  3. Could you compare SITAR and AER metrics for the same reference-hypothesis pairs to get comparable results?

6 comments

Answer for Q1

We sincerely thank the reviewer for this valuable suggestion. SITAR explores ASR errors from the perspective of situational distortion. Our original evaluation combined human annotations of individual gaps with controlled comparisons. These tests examined how SITAR responds to different error types and combinations. We appreciate the reviewer’s guidance on extending this analysis to independent human judgments of complete transcripts.

Following the reviewer’s suggestion, we conducted an additional human evaluation with five PhD students. We randomly sampled 300 utterances from LibriSpeech and 300 from VoxPopuli. The annotators assessed the complete ASR transcripts directly, without using the proposed aggregation rule. We now report the agreement of SITAR and all baseline metrics with these human judgments.

Before presenting the results, we would like to offer two clarifications.

Inclusion of LASER. To broaden the baseline comparison, we have also evaluated LASER in English using its publicly available implementation. LASER was originally developed for Hindi ASR evaluation, which informed our initial choice of baselines. We now include LASER in the comparisons below.

LASER scores (higher is better):

Datasetwhisper-tinywhisper-basewhisper-smallwhisper-mediumwhisper-largewav2vec2-basewav2vec2-large
FLEURS0.92220.95060.97320.98060.98270.92960.9461
LibriSpeech0.95800.97650.98750.99020.99420.98610.9902
VoxPopuli0.92050.93810.94830.95340.93890.87120.8952

Dataset selection for human evaluation. SITAR’s training data include FLEURS, while our FLEURS evaluation uses the official test split. For the additional human evaluation, we focus exclusively on LibriSpeech and VoxPopuli. Neither dataset was used to train SITAR. This allows us to examine generalization to external datasets through independent human judgments. We discuss the data split in our response to Q2.

We first report (1) overall agreement with human judgments on real ASR outputs and (2) comparisons of transcripts with identical WER. We then present (3) a case study illustrating different situational consequences at identical WER. Finally, we report direct human evaluations of (4) the 30 original complete transcripts and (5) all 330 complete hypotheses.

(1) Agreement with human judgments on real ASR outputs

We first report system-level agreement with human judgments on real ASR outputs. For each dataset, we compare the ranking of the seven ASR systems with the ranking obtained from human annotations. The table reports Spearman’s ρ.

DatasetWERNE-WERBERTScoreSemDistAERLASERSITAR
LibriSpeech1.00000.92860.57660.82140.75000.96430.8214
VoxPopuli0.96430.88290.96430.8929-0.32140.96430.9643

WER achieves the highest correlation on LibriSpeech and shares the highest correlation on VoxPopuli. We believe two factors may help explain its strong performance. First, ASR training primarily optimizes predictions against reference token sequences. This emphasis on lexical accuracy may contribute to WER’s strong performance as an evaluation metric. Second, substitutions, deletions, and insertions directly reflect deviations from the reference transcript. Such deviations may also influence human judgments of transcription quality. These findings provide empirical support for the continued effectiveness of WER, which has long served as the de facto standard for ASR evaluation.

The comparison between AER and SITAR is particularly informative. On VoxPopuli, AER is the only evaluated metric with a negative correlation with human judgments (−0.3214). SITAR, in contrast, achieves 0.9643. This finding helps interpret the disagreement reported in our paper, where the correlation between SITAR and AER was −0.11. The additional human evidence supports our diagnostic interpretation by showing that SITAR’s system ordering closely follows human judgments.

The strengths of SITAR may not be fully apparent in this overall comparison. This is in line with our expectations, since SITAR was designed to complement WER rather than replace it. We believe its relatively high agreement with human judgments across both datasets already provides strong support for its effectiveness. We next examine its advantages in cases where WER alone cannot distinguish between transcripts.

(2) Comparing transcripts with identical WER

We hope this analysis also helps address the reviewer’s concern about which metric better reflects error severity. The comparison uses independent human judgments as the reference, rather than relying on differences between metric scores alone.

We next focus on transcripts with identical WER for the same utterance. This yields 1,492 transcripts from 300 LibriSpeech utterances and 1,187 transcripts from 298 VoxPopuli utterances. Each utterance originally has seven candidate transcripts, one from each ASR system. Two VoxPopuli utterances were excluded because all seven candidates had different WER values. We aggregate the human rankings within these groups into a system-level ranking for each dataset. The table reports Spearman’s ρ.

DatasetBERTScoreSemDistAERLASERSITAR
LibriSpeech0.41440.17860.53570.46430.5714
VoxPopuli0.35710.25000.35710.35710.4643

SITAR achieves the highest agreement with human judgments among the compared metrics on both datasets. We believe this supports its usefulness for evaluating error consequences beyond lexical accuracy.

(3) Case study of different consequences at identical WER

We additionally present four examples showing how deleting the same word can have different situational consequences. Each hypothesis deletes one occurrence of “not”, and all four examples have identical WER.

N0: Reason for the flight choice reversed

Reference: I chose the 8:15 flight because I am not worried about the price. Please book the 8:15 refundable ticket.

Hypothesis: I chose the 8:15 flight because I am worried about the price. Please book the 8:15 refundable ticket.


N1: Core flight choice reversed

Reference: The 8:15 flight is available. I have decided not to take the 9:40 flight. Please book the earlier one.

Hypothesis: The 8:15 flight is available. I have decided to take the 9:40 flight. Please book the earlier one.


N2: Secondary availability reversed

Reference: The meeting is on Thursday at three. Maya is not available for lunch, but she will attend the meeting.

Hypothesis: The meeting is on Thursday at three. Maya is available for lunch, but she will attend the meeting.


N3: Meeting attendance reversed

Reference: The meeting is on Thursday at three. Maya is available for lunch, but she will not attend the meeting.

Hypothesis: The meeting is on Thursday at three. Maya is available for lunch, but she will attend the meeting.

ExampleBERTScore F1 ↑WER ↓SITAR ↓
N0: Reason for the flight choice0.98660.05260.6815
N1: Core flight choice0.96680.05260.7114
N2: Lunch availability0.95140.05260.8006
N3: Meeting attendance0.95340.05260.8293

In the flight examples, N0 changes the explanation for the choice while preserving the booking request. N1 changes the explicit choice and creates a conflict with the request to book the earlier flight. Both BERTScore and SITAR assign greater severity to N1.

In the meeting examples, N2 changes lunch availability while preserving meeting attendance. N3 reverses attendance itself. BERTScore assigns slightly higher similarity to N3, while SITAR assigns greater severity to the change affecting the meeting itself.

These examples illustrate SITAR’s ability to distinguish an error’s situational consequences even when its lexical operation and WER remain identical.

(4) Evaluation on the controlled benchmark

Following the reviewer’s helpful suggestion, we also obtained independent human ratings of the 30 original complete transcripts underlying the controlled benchmark. Annotators assessed each transcript as a whole, without applying the aggregation rule. We report pairwise ordering agreement (Agr.) and Spearman’s ρ.

MeasureWERNE-WERBERTScore F1SemDistAERLASERSITAR
Agr.40.97%27.69%41.20%38.43%54.40%48.38%69.21%
Spearman’s ρ-0.2388-0.1373-0.2475-0.32450.24400.01140.5488

We observe positive rank correlations only for the three LLM-based approaches: AER, LASER, and SITAR. We find this clear contrast with traditional metrics encouraging. It suggests that LLMs make it possible to assess aspects of error severity that traditional metrics struggle to capture. The strength of this association varies across the three approaches, with LASER’s rank correlation close to zero.

Within this group, SITAR achieves considerably higher pairwise agreement and rank correlation than both AER and LASER.

(5) Human evaluation of the 330 complete hypotheses

We hope this evaluation also helps address the concern about the independent validity of the proposed severity scale (Weaknesses, paragraph 1).

We also conducted direct human evaluation of all 330 complete hypotheses, rather than deriving their rankings from the proposed aggregation rule.

SITAR results by domain. N denotes the number of hypothesis pairs with a strict human ordering within the same reference group. Agr. measures agreement with this ordering, with metric ties counted as disagreement. Agr. and Spearman’s ρ are calculated within each reference group and then averaged with equal weight per reference.

MeasureCultureFinanceServicesCivicGeneralOverall
N4952203852752721,647
Agr.88.48%80.45%80.00%83.64%85.16%84.07%
Spearman’s ρ0.87580.76140.70780.81820.80620.8001

Comparison across metrics. We compare all metrics against the same human judgments.

MeasureWERNE-WERBERTScore F1SemDistAERLASERSITAR
Agr.79.69%17.27%84.96%83.49%56.26%67.47%84.07%
Spearman’s ρ0.83730.45030.78950.76370.68050.60860.8001

SITAR shows strong agreement with human judgments across all five domains. We are also encouraged by its higher pairwise agreement and rank correlation than both AER and LASER.

Answer for Q2

We sincerely thank the reviewer for requesting this clarification. As described in the manuscript, we split individual reference–hypothesis pairs rather than grouping all hypotheses by their shared reference. Some reference transcripts are therefore shared across sitar_train_data, sitar_val_data, and sitar_test_data. We sincerely apologize for not sufficiently explaining this aspect of the split in the original manuscript.

We use sitar_train_data, sitar_val_data, and sitar_test_data to train, validate, and test the fine-tuned DeBERTa regressor, respectively. The examples in sitar_train_data, sitar_val_data, and sitar_test_data are drawn primarily from AppTek. AppTek contains long recordings, which we first segmented at silence intervals into clips of no more than 30 seconds.

We would also like to clarify the role of the internal evaluation. We used sitar_val_data for model development and checkpoint selection, while sitar_test_data was used for reporting the regression results. The corresponding regression results and fine-tuning details are reported in the appendix, rather than the main text. The main evidence for SITAR’s effectiveness comes from the external ASR evaluation in Table 1, not from results on sitar_test_data.

Table 1 reports results on fleurs_en_test, LibriSpeech, and VoxPopuli. Neither LibriSpeech nor VoxPopuli was included in sitar_train_data, sitar_val_data, or sitar_test_data. LibriSpeech and VoxPopuli therefore provide an evaluation on datasets unseen during SITAR’s development.

For English FLEURS, fleurs_en_train contributed samples to sitar_train_data, while external evaluation used the official fleurs_en_test partition. English FLEURS contributed only to sitar_train_data, not to sitar_val_data or sitar_test_data.

We also constructed a separate controlled dataset, ctrl_data. As disclosed in the paper, the examples in ctrl_data were created from scratch and were not drawn from existing datasets. We found it challenging to construct examples that met all the required conditions. As a result, ctrl_data is relatively small. No reference transcripts are shared between ctrl_data and sitar_train_data, sitar_val_data, or sitar_test_data. We use ctrl_data for illustration and controlled analysis of SITAR’s behavior, rather than to establish its performance on real ASR outputs.

Overall, our study involves seven datasets: sitar_train_data, sitar_val_data, sitar_test_data, fleurs_en_test, LibriSpeech, VoxPopuli, and ctrl_data. Despite its name, sitar_test_data serves as an internal evaluation set for the regressor rather than the main basis for SITAR’s performance claims. Both LibriSpeech and VoxPopuli are out-of-domain datasets entirely unseen during SITAR’s development. We hope the results on LibriSpeech and VoxPopuli could provide reassuring, independent support for SITAR’s reliability.

Reviewer_p6UDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SITAR, a reference-based metric for automatic speech recognition that measures how much a hypothesis distorts the situation conveyed by the source utterance. The underlying view is that the severity of a transcription error depends on whether it changes what the speaker describes, establishes, or seeks to bring about, rather than on the number of words or the amount of meaning affected.

The method moves all LLM involvement into the construction of explicit training targets. A deterministic alignment procedure segments each reference-hypothesis pair into paired gaps. Each gap is evaluated in isolation and assigned one of three severity tiers by an LLM pipeline with committee review. The tiers correspond to changes in expression only, changes in uptake with the intended outcome preserved, and changes to the outcome itself. A closed-form rule combines the tiers into a scalar in which the most severe gap determines the base level and additional gaps contribute a bounded, saturating refinement. A DeBERTa regressor is trained on approximately twelve thousand scored pairs to predict this scalar from the raw transcript pair.

The evaluation covers two forms of validity. Extrinsic validity is examined by showing that SITAR decreases with ASR model scale on FLEURS, LibriSpeech, and VoxPopuli and that it produces wider separation between systems than WER or SemDist while occasionally ordering them differently. Intrinsic reliability is examined on a controlled set of thirty references with eleven edit-based variants each. Two annotators assign tier labels to the underlying gaps and the regressor is evaluated on whether it reproduces the expected severity ordering within each reference.

Critical strengths

The paper addresses a real and well-documented gap in ASR evaluation. It presents convincing evidence that errors with similar lexical or semantic distance can have very different downstream consequences, and that neither edit-based nor embedding-based metrics capture this. Defining severity in terms of what the utterance is meant to accomplish is a principled response, and the decision to keep LLM involvement offline so that scoring remains deterministic and inexpensive is a sensible engineering choice that the paper follows through on. The resulting scorer requires no LLM, alignment, or prompt at inference time, and the cost advantage over question-answering based metrics such as AER is quantified rather than asserted.

The methodology is unusually well specified. The score construction is accompanied by proofs of boundedness, of the priority of severity over quantity, of lexicographic ordering among count configurations, and of diminishing increments, together with worked examples that cover edge cases such as an empty intermediate tier. Whatever one concludes about the design choices themselves, their consequences are fully laid out. The regressor is likewise tuned with care, with pooling, auxiliary losses, input representation, encoder adaptation, learning rates, and ordinal thresholds each ablated over multiple seeds and the final configuration retrained independently five times.

The controlled dataset is the most valuable part of the evaluation. By holding the reference and the wording of each edit fixed while varying which edits are applied, it isolates how the scorer responds to error composition in a way that natural ASR output cannot. The human annotation of this set is carried out independently of the initial model labels and shows high agreement between the two annotators, which gives the within-reference ordering results a credible footing.

Critical weaknesses

The central weakness is that the metric is never evaluated against human judgment of transcript quality, even though closer alignment with human judgment is its primary stated advantage. The paper contains no human assessment of real ASR output and reports no correlation between SITAR and such assessments. It therefore cannot show whether SITAR agrees with people more closely than WER, SemDist, BERTScore, AER, or an LLM judge. The only human data are tier labels on 120 gaps drawn from thirty references, and these are used to check that the scorer orders hypothesis variants in the way the tiers imply. That is a consistency check between the scorer and the paper’s own construct. It does not establish that the construct, or the scorer built on it, agrees with how people perceive transcript quality. The cross-model experiments do not fill this gap. The observation that SITAR decreases from whisper-tiny to whisper-large holds for every baseline in the same table and does not distinguish SITAR from WER. The cases where SITAR disagrees with the baselines, such as ranking whisper-small above wav2vec2-large on LibriSpeech or correlating at negative 0.11 with AER on VoxPopuli, are presented as diagnostic value. Without human judgments on those systems, a disagreement is equally consistent with error on SITAR’s part.

A second weakness is that the deployed scorer cannot resolve the fine structure of the score it was trained to predict. The score construction, with its one-third directional adjustments, its saturation rate derived from the convention that seven gaps fill ninety percent of the refinement interval, and its lexicographic priority of counts, occupies much of the method and appendix. None of these choices is validated by humans, who assign only nominal tiers. More importantly, the regressor’s test error is approximately 0.36 RMSE on a scale whose tiers are one third apart, with Pearson and Spearman correlations of 0.82 and 0.81 against the targets. A typical prediction error is therefore comparable to a full tier. The controlled-set results confirm this. Ordering agreement is 91 percent when variants differ in outcome-level gaps and 83 percent for uptake-level gaps, but only 65 percent when variants differ solely in the number of expression-level gaps, which is the regime the saturating refinement is designed to capture. The paper’s own worked example shows a variant with two added disfluencies scoring above a variant with an additional uptake-level change. In practice, SITAR behaves as an approximate three-way severity classifier, and the recursive formulation is not reflected in its behavior. The paper reports the regression quality only in the appendix and does not discuss what a one-tier error means for the use cases described in the introduction.

Finally, the contribution of the alignment procedure to the final metric is not measured. The paper’s own ablation shows that exposing the gap structure to the regressor does not improve performance, and no experiment compares targets derived from the proposed alignment with targets derived from standard Levenshtein alignment.

Questions for authors
  • Is there any evidence that SITAR agrees with human judgment of real ASR transcripts more closely than existing metrics do? A study in which annotators rate or compare transcripts from the evaluated systems, with SITAR’s correlation reported alongside WER, SemDist, BERTScore, AER, and an LLM judge, would directly support the paper’s central claim.
  • How would WER, SemDist, or an LLM judge perform on the within-reference ordering task used in Section 4.3? This would show whether the 84 percent agreement reflects an advantage of SITAR or a property of the task.
  • Given that the regressor’s error is comparable to one tier and that agreement on expression-level count differences is 65 percent, does the recursive refinement contribute anything at inference time? A comparison against a scorer trained on the base tier alone would answer this.
  • Were the AER baselines computed on the same utterance subsets as SITAR, and is there any indication of how the scorer behaves on domains or utterance lengths outside the call-center and FLEURS training data?
11 comments

Answer for Q1

We sincerely thank the reviewer for emphasizing the importance of human evaluation on real ASR transcripts. SITAR explores ASR errors from the perspective of situational distortion. Our original evaluation combined human annotations of individual gaps with controlled comparisons of error types and combinations. We appreciate the reviewer’s suggestion to extend this evaluation to human judgments of complete ASR transcripts.

Following this suggestion, we conducted an additional human evaluation with five PhD students, using 300 randomly sampled utterances from LibriSpeech and 300 from VoxPopuli. The annotators assessed complete transcripts directly, without using the proposed aggregation rule. Neither dataset was used to train SITAR.

We also include LASER, evaluated using its publicly available implementation. The AER rankings are derived from the published scores. All correlations below are calculated against our human judgments.

We first report (1) overall agreement with human judgments on real ASR outputs, followed by (2) comparisons of transcripts with identical WER. We then present (3) a case study illustrating different situational consequences at identical WER.

(1) Agreement with human judgments on real ASR outputs

This evaluation also helps address the first part of Weaknesses, paragraph 1. We compare SITAR and the baseline metrics against independent human judgments of complete ASR transcripts.

For each dataset, we compare the ranking of the seven ASR systems produced by each metric with the ranking obtained from human annotations. The table reports Spearman’s ρ.

DatasetWERNE-WERBERTScoreSemDistAERLASERSITAR
LibriSpeech1.00000.92860.57660.82140.75000.96430.8214
VoxPopuli0.96430.88290.96430.8929−0.32140.96430.9643

WER achieves the highest correlation on LibriSpeech and shares the highest correlation on VoxPopuli. We think its strong performance may be partly related to how ASR systems are trained. Minimizing prediction loss against reference token sequences may indirectly favor the lexical accuracy measured by WER. These results also reaffirm WER’s practical effectiveness, which has supported its long-standing use as the standard metric for ASR evaluation.

The human evaluation also helps interpret the correlation of −0.11 between SITAR and AER reported in our paper. On VoxPopuli, SITAR’s system ranking closely follows human judgments, whereas the ranking derived from the published AER scores is negatively correlated with them. This provides additional evidence for the diagnostic interpretation discussed in the manuscript.

SITAR’s particular strengths may not be fully apparent in this overall comparison. This is in line with its intended role: SITAR was designed to complement WER rather than replace it. Its relatively high agreement with human judgments across both datasets provides encouraging evidence of its effectiveness. We next examine its contribution in cases where WER cannot distinguish between transcripts.

(2) Comparing transcripts with identical WER

Regarding the latter part of Weaknesses, paragraph 1, the human rankings above provide a reference for interpreting disagreements between metrics. The comparison below further examines SITAR’s complementary value when WER assigns identical scores.

We next focus on transcripts with identical WER for the same utterance. This yields 1,492 transcripts from 300 LibriSpeech utterances and 1,187 transcripts from 298 VoxPopuli utterances. Each utterance originally had seven candidate transcripts, one from each ASR system. Two VoxPopuli utterances were excluded because all seven candidates had different WER values. We aggregate the human rankings within these groups into a system-level ranking for each dataset. The table reports Spearman’s ρ.

DatasetBERTScoreSemDistAERLASERSITAR
LibriSpeech0.41440.17860.53570.46430.5714
VoxPopuli0.35710.25000.35710.35710.4643

SITAR achieves the highest rank correlation with human judgments among the compared metrics on both datasets. We hope these results could provide direct support for SITAR’s complementary value in identifying differences in situational distortion among transcripts that WER rates equally.

(3) Case study of different consequences at identical WER

We additionally present four examples showing how deleting the same word can have different situational consequences. Each hypothesis deletes one occurrence of “not”, and all four examples have identical WER.

N0: Reason for the flight choice reversed

Reference: I chose the 8:15 flight because I am not worried about the price. Please book the 8:15 refundable ticket.

Hypothesis: I chose the 8:15 flight because I am worried about the price. Please book the 8:15 refundable ticket.


N1: Core flight choice reversed

Reference: The 8:15 flight is available. I have decided not to take the 9:40 flight. Please book the earlier one.

Hypothesis: The 8:15 flight is available. I have decided to take the 9:40 flight. Please book the earlier one.


N2: Secondary availability reversed

Reference: The meeting is on Thursday at three. Maya is not available for lunch, but she will attend the meeting.

Hypothesis: The meeting is on Thursday at three. Maya is available for lunch, but she will attend the meeting.


N3: Meeting attendance reversed

Reference: The meeting is on Thursday at three. Maya is available for lunch, but she will not attend the meeting.

Hypothesis: The meeting is on Thursday at three. Maya is available for lunch, but she will attend the meeting.

ExampleBERTScore F1 ↑WER ↓SITAR ↓
N0: Reason for the flight choice0.98660.05260.6815
N1: Core flight choice0.96680.05260.7114
N2: Lunch availability0.95140.05260.8006
N3: Meeting attendance0.95340.05260.8293

In the flight examples, N0 changes the explanation for the choice while preserving the booking request. N1 changes the explicit choice and creates a conflict with the request to book the earlier flight. Both BERTScore and SITAR assign greater severity to N1.

In the meeting examples, N2 changes lunch availability while preserving meeting attendance. N3 reverses attendance itself. BERTScore assigns slightly higher similarity to N3, while SITAR assigns greater severity to the change affecting the meeting itself.

These examples illustrate SITAR’s ability to distinguish an error’s situational consequences even when its lexical operation and WER remain identical.

Answer for Q2

We sincerely thank the reviewer for suggesting this comparison. We have now evaluated the baseline metrics on the same controlled hypotheses. We report the within-reference ordering results first, followed by a complementary evaluation of the original complete transcripts.

(1) Within-reference ordering of the 330 complete hypotheses

We conducted direct human evaluation of all 330 complete hypotheses, without using the proposed aggregation rule to determine their expected ordering. We compare the metrics against these human judgments within each reference group.

Agr. measures the proportion of strictly ordered human comparisons that a metric orders consistently. Metric ties count as disagreement. Agr. and Spearman’s ρ are calculated within each reference group and then averaged with equal weight per reference.

MeasureWERNE-WERBERTScore F1SemDistAERLASERSITAR
Agr.79.69%17.27%84.96%83.49%56.26%67.47%84.07%
Spearman’s ρ0.83730.45030.78950.76370.68050.60860.8001

SITAR achieves higher pairwise agreement than WER and SemDist, and higher agreement and rank correlation than both AER and LASER. The established metrics also perform strongly on this task, with BERTScore F1 leading in pairwise agreement and WER leading in rank correlation. This comparison places SITAR’s reported agreement in the context of competitive baseline performance on the same hypotheses.

(2) Human evaluation of the 30 original complete transcripts

We additionally obtained independent human ratings of the 30 original complete transcripts underlying the controlled benchmark. Annotators assessed each transcript as a whole, without applying the aggregation rule. This evaluation compares severity across complete transcripts, complementing the comparisons among variants of the same reference above.

MeasureWERNE-WERBERTScore F1SemDistAERLASERSITAR
Agr.40.97%27.69%41.20%38.43%54.40%48.38%69.21%
Spearman’s ρ−0.2388−0.1373−0.2475−0.32450.24400.01140.5488

We observe positive rank correlations only for the three LLM-based approaches: AER, LASER, and SITAR. We find this contrast with conventional metrics encouraging. It suggests that incorporating LLMs into evaluation can help assess aspects of overall error severity that conventional metrics may struggle to capture.

Within this group, SITAR achieves considerably higher pairwise agreement and rank correlation than both AER and LASER. We hope these results could provide direct support for SITAR’s ability to reflect human judgments of situational distortion in complete transcripts.

Answer for Q3

This comparison also helps address the concern in Weaknesses, paragraph 2, about whether the deployed scorer retains useful distinctions beyond the three base tiers.

We sincerely thank the reviewer for suggesting this ablation and for proposing a direct way to examine the contribution of refinement. Following this suggestion, we trained a three-class classifier using only the base-tier labels and compared its predictions with those of the trained SITAR scorer.

A three-class output can assign the same label to many distinct transcripts. When all candidates for an utterance receive the same label, their predicted ranks have zero variance, making Spearman’s ρ undefined. We therefore report pairwise ordering agreement (Agr.). Agreement requires the predicted ordering to match a strict human preference; predicted ties do not count as agreement.

(1) Distinguishing differences within T0

We sincerely thank the reviewer for raising this concern about expression-level refinement. To address it directly, we conduct a focused analysis of transcripts containing only T0 errors and compare SITAR with the three-class classifier.

We analyzed all 330 reference–hypothesis pairs and found that 60 pairs involve only expression-level (T0) errors. These pairs cover 30 references, each with two hypothesis versions containing one and two T0 errors, respectively. We compare both models against the human ordering of these versions.

MeasureThree-class classifierSITAR
Correctly ordered pairs028
Tied pairs300
Reversed pairs02
Agr.0.00%93.33%

The classifier assigns the same tier to both versions in every comparison. SITAR preserves the distinction introduced by the additional T0 error and closely follows the human ordering. This shows that the trained scorer can retain useful information about error accumulation even when the base tier remains unchanged.

(2) Preserving differences among transcripts classified as T0

We further examine whether this distinction matters beyond the controlled examples. On English FLEURS, VoxPopuli, and LibriSpeech, we identify transcripts assigned to T0 by the classifier and examine the scores produced by SITAR.

The first row reports how frequently the classifier predicts T0. The second reports how often SITAR distinguishes textually different transcripts assigned to T0. The final row reports agreement on the human-evaluated pairs with strict preferences.

MeasureEnglish FLEURSVoxPopuliLibriSpeech
Transcripts predicted as T02,471/4,529 (54.56%)7,052/12,810 (55.05%)14,666/18,277 (80.24%)
Textually distinct T0 pairs receiving different SITAR scores2,239/2,344 (95.52%)5,571/5,674 (98.18%)8,628/8,941 (96.50%)
SITAR Agr.16/22 (72.73%)53/57 (92.98%)37/53 (69.81%)

These results show that shared T0 predictions occur frequently in real ASR outputs. Across the three datasets, SITAR recovers the human ordering in 106 of 132 comparisons (80.30%). These preferences cannot be expressed by the shared T0 label, showing the practical value of retaining differences within a severity class.

(3) Comparing transcripts with identical WER

We next examine whether the refinement contributes to SITAR’s intended role as a complement to WER. We compare transcripts with identical WER for the same utterance. Even the same type of edit, such as deleting “not,” can have different consequences depending on the information it changes.

DatasetThree-class classifier Agr.SITAR Agr.
LibriSpeech55.23%83.06%
VoxPopuli56.84%84.20%

SITAR achieves substantially higher agreement with human preferences on both datasets. This comparison supports the usefulness of retaining graded severity scores when word error counts are identical.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author-Reviewer Discussion

Dear Area Chair and Reviewers,

Thank you for your time and care in reviewing our paper. As the discussion draws to a close, we would like to summarize the main points raised and how we addressed them:

  • Core Claim and Value (Reviewer p6UD): While we sincerely appreciate the reviewer's recognition and the higher expectations held for our work, we wish to clarify the exact scope of our contribution accordingly. Our central claim is that SITAR is lightweight and complements WER. It measures transcription errors where WER fails, requiring only a 0.4B model to rival or even surpass LLMs.

  • Datasets and Splits (Reviewers 2kSk, C1db): We thank Reviewers 2kSk and C1db for pointing out the confusion caused by our naming of sitar_test_data. Despite its name, it is a held-out split used for regressor development (reported only in the appendix). None of SITAR’s reported performance results as an ASR metric are based on sitar_test_data. All main-text results come from three external benchmarks (the official English FLEURS test split, VoxPopuli, and LibriSpeech) and one internally constructed controlled dataset (ctrl_data).

  • Additional Experiments with AER (Reviewers p6UD, 2kSk) and LASER (Reviewer C1db): We thank the reviewers for suggesting these additional experiments. We have conducted a comprehensive set of experiments comparing AER and LASER with SITAR and reported how closely each metric agrees with human judgments.

  • Large-Scale Human Evaluation (All Reviewers): We thank all three reviewers for their suggestions on strengthening our evaluation. To rigorously validate SITAR's reliability, five PhD students annotated 4,200 complete ASR transcripts across seven systems on LibriSpeech and VoxPopuli (300 utterances per dataset). We now report four measures: (i) Spearman correlation with human system rankings, both overall and on transcripts with identical WER, (ii) transcript-level balanced accuracy, (iii) inter-annotator pairwise agreement, and (iv) pairwise ordering agreement with human judgments on the controlled dataset.

We once again thank all reviewers for their time and feedback. Since the reviewers' inquiries were largely concentrated on a few shared aspects, we have made every effort to address all raised questions and weaknesses with thorough explanations and empirical evidence.

Finally, we deeply value the constructive comments provided by the reviewers. We hope that our work can provide new insights into ASR evaluation metrics and contribute to their further development.

Sincerely,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.