acceptodds
Under review as a conference paper at ICLR 2027

SpeechIEM: A Label-Free Perceptual Distance for Speech

Abstract

We present SpeechIEM, a label-free full-reference distance on self- supervised speech latents, and—the point of the paper—the criterion that decides whether such a construction is a distance at all. The construction is the Information-Estimation Metric of Ohayon et al. (2026): a denoiser trained on the data becomes a distance by comparing residuals across noise levels. Its failure mode is silent. The distance becomes a fixed rescaling of the Euclidean distance on the same representation, ordering every pair identically to it, while every training curve improves and every monotonic- ity plot looks healthy. We give the exact condition for that collapse—the denoiser’s action on the difference integrating to a multiple of the identity— and show a scalar action to be sufficient but not necessary, which alone rules out gating on the residual’s direction. Three routes reach the regime in or- dinary practice, one of them an untrained preconditioned denoiser, so a gate is a mandatory step rather than a diagnostic nicety. Which diagnostic to gate on we settle by measurement over 27 denoisers from three grids, scoring each candidate against what it stands in for: the measure-weighted residual cosine rejects two metrics carrying a full advantage, the dispersion of the distance-to-Euclidean ratio rejects none and separates the classes with a factor-of-fifteen empty margin. Admissible, SpeechIEM beats the Euclidean baseline the construction itself contains by +0.058 τb across six distortion families against PESQ, with no quality labels—but the sign of that advantage depends on the reference it is scored against (−0.012 against ESTOI), and on twelve of the thirteen hidden states the trained denoiser adds almost nothing. Hence a reporting standard for this family of metrics: beat the zero-denoiser baseline on the same representation, pair the com- parison, and name the reference quality and the corruption distribution it was measured on.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
deep learning, speech enhancement, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nXGUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper applies the Information-Estimation Metric (IEM) of Ohayon et al. (2026) to frozen WavLM latents (SpeechIEM): a denoiser head is trained on clean latents and the distance compares its residuals across noise levels. The focus is on when this construction silently collapses into a rescaled latent L2. The authors give a collapse criterion and identify three common routes to collapse, including an untrained preconditioned head and alignment-style auxiliary losses. They propose the dispersion (CV) of IEM/‖Δz‖ as a gate and show, over 27 checkpoints, that it makes no false rejections while a residual-cosine gate makes two. On a six-family distortion probe, the layer-0 metric beats latent L2 by +0.058 τb against PESQ but loses against ESTOI. The advantage nearly vanishes on the other 12 layers. On NISQA human MOS it holds on recorded conditions but reverses on codec conditions. The paper ends with a reporting standard for this family of metrics.

Critical strengths

Rigorous and honest evaluation. Every comparison is paired against the exact zero-denoiser IEM on the same latents, which cleanly separates the denoiser's contribution from the encoder's. The authors correct for layer-selection leakage, replicate the key cells at a second seed, and openly report unfavorable results (ESTOI, codec MOS, the training loss). The resulting reporting standard is a useful methodological contribution for learned-metric research.

Critical weaknesses

(1) The gate detects collapse, not usefulness. By Proposition 2, CV > 0 is only a necessary condition. In practice, all 12 transformer layers pass the gate yet gain only about +0.008 over L2. The "empty margin" also rests on just two collapsed cells from a single, self-constructed failure mode, so the gate's generality is not established.

(2) Limited theoretical and empirical contribution. The criterion is a direct consequence of the Mahalanobis form in Ohayon et al. The metric's gain is small and confined to layer 0, and it is significantly worse than L2 on the largest human-rated subset (2118 NISQA codec files). There is also no comparison to existing label-free speech metrics or stronger encoders.

(3) Missing comparisons with existing speech perceptual metrics. Although comparisons to latent L2 isolate the denoiser’s contribution, they do not establish SpeechIEM’s practical competitiveness. Evaluation against methods such as SpeechBERTScore, CDPAM/DPAM, PESQ, and ViSQOL is needed.

(4) Limited evidence of perceptual validity. The human-rating improvement is small and limited to recorded conditions, while performance is neutral or worse on simulated conditions, especially codec distortions. A dedicated pairwise listening study is needed to support the claim that SpeechIEM reflects human perception.

Questions for authors

Clarification: (1) Among the 25 non-collapsed checkpoints, does CV correlate with the size of the advantage? If not, the gate is only a collapse detector.

(2) Does the diagonal Mahalanobis baseline pass the CV gate? If so, how does that fit the claim that the gate decides "whether the construction is a distance at all"?

(3) The rotation counterexample against the cosine gate uses a skew-symmetric action, but an MMSE denoiser's Jacobian is symmetric. Is there a realistic symmetric case that fools the cosine?

Suggestion: (4) A comparison with SpeechBERTScore and a WavLM-large variant on the same probe and NISQA splits.

Reviewer_4MfmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper adapts the Information-Estimation Metric (IEM) to frozen WavLM representations, using a trained denoiser to compare residual differences integrated over noise levels. Its principal contribution is presented as a criterion and practical gate for detecting when this construction reduces to a rescaled Euclidean distance on the same representation.

Under a common linear denoiser action, the paper characterizes proportionality to Euclidean distance through isotropy of the induced quadratic form. It then proposes thresholding the coefficient of variation of the IEM-to-Euclidean ratio and compares this diagnostic with a residual-cosine diagnostic across 27 checkpoints. Experiments examine cross-distortion rankings, thirteen WavLM hidden states, denoiser architectures and training budgets, human quality ratings, and speech enhancement training objectives.

The reported benefit is conditional: the largest improvement occurs at the feature-projection layer, agreement improves against PESQ but worsens against ESTOI, and the human-rating results vary substantially across corruption conditions. The paper advocates comparisons against the same-representation Euclidean baseline and explicit reporting of the evaluation reference and corruption distribution.

Critical strengths

1. The same representation comparison is well motivated and produces useful empirical findings.

Comparing SpeechIEM with latent Euclidean distance on exactly the same frozen features is an appropriate way to measure the denoiser's incremental contribution. The cross-family evaluation is also more informative in this setting than simply showing monotonic behavior as a single distortion becomes stronger. The paired bootstrap over utterances appropriately addresses the dependence between the methods being compared (sec. 5 and Appendix A).

The layer 0 PESQ improvement is supported by the additional holdout analysis: Appendix E reports an advantage of approximately +0.058 in Kendall's (\tau_b) on the 136 utterances that did not participate in layer selection. This strengthens the evidence that the improvement is not merely an artifact of the initial selection set.

2. The paper reports important limitations and unfavorable results transparently.

The authors examine several ways in which the initial result may fail to generalize. These include the small incremental gains at most transformer layers, the reversal against ESTOI, mixed results against human MOS, and the limitations of using SpeechIEM as an enhancement loss.

In particular, Appendix F contains an independent human rating evaluation, including both a small positive result on recorded live/anchor conditions and a negative result on simulated conditions containing codecs. This evidence is consequential because it reveals the limits of the proposed metric and could support a useful empirical study of when denoiser-based distances add value.

Critical weaknesses

1. The proposed gate is not validated against the property it is claimed to certify.

The paper's main contribution is repeatedly described as a criterion and gate determining whether the construction is a distance at all, or whether it carries ordering information beyond latent (L_2). These are different properties. A positive rescaling of Euclidean distance is already a valid distance. The relevant question here is whether the additional construction changes the ordering and, separately, whether that change is useful.

Proposition 2 correctly acknowledges an important limitation: a nonconstant ratio (IEM/∣Δz∣)(\mathrm{IEM}/|\Delta z|) is only a necessary condition for a different ordering. A strictly increasing nonlinear transformation of Euclidean distance can have a dispersed ratio while preserving every ordering. Thus, passing the proposed CV gate does not establish additional ranking information.

The empirical validation does not directly resolve this gap. Appendix C, particularly lines 630-632, defines the comparison target through whether the paired confidence interval for

τb(IEM,q)−τb(L2,q)\tau_b(\mathrm{IEM},q)-\tau_b(L_2,q)

excludes zero, where (q) is a quality reference such as PESQ. This measures whether the methods have distinguishable agreement with a third reference. It does not directly measure whether their orderings differ from one another. Two different orderings can have the same Kendall correlation with the reference. Conversely, an interval containing zero may reflect insufficient power. The resulting classification also depends on the selected reference and sample size.

Consequently, the reported 25/27 agreement supports an empirical association between the diagnostic and reference-correlation differences on the studied probe. It does not establish the advertised accuracy of a general test for ordering collapse. This matters because the gate, rather than the underlying IEM construction, is presented as the principal contribution.

How to improve: Separate three claims explicitly: proportionality to latent Euclidean distance, direct ranking disagreement with that baseline, and improvement against a quality reference. Using the existing predictions, report direct IEM versus (L_2) rank disagreement, including ties, on the same within-utterance comparisons for all 27 checkpoints. Evaluate perceptual reference agreement separately. The CV threshold should be presented as an empirical screen for approximate proportionality on the tested distribution unless stronger evidence supports a broader interpretation.

2. The theoretical explanation and experimental controls do not establish the claimed role of nonlinear denoising.

Sections 3 and 4.1 state that a (σ)(\sigma) independent, or weakly (σ)(\sigma) dependent, map is linear on the pair by construction. This implication does not hold. For example,

D(y,σ)=tanh⁡(y)D(y,\sigma)=\tanh(y)

is independent of (σ)(\sigma) but nonlinear in its input. Conversely,

D(y,σ)=a(σ)yD(y,\sigma)=a(\sigma)y

can depend strongly on the noise level while remaining exactly in the scalar-action regime described by Proposition 1. Noise conditioning, input nonlinearity, and isotropy are distinct properties.

This distinction also affects the interpretation of the architecture experiment. Section 4.2 concludes that AdaLN improves admissibility reliability. However, Table 1 shows identical CV gate decisions for additive conditioning and AdaLN in every matched objective/budget comparison: both fail for the alignment-augmented objective at 500 steps, and both pass in the other three settings. Differences in residual cosine do not establish greater admission reliability, especially when the paper itself demonstrates that cosine is an unreliable admission diagnostic.

Furthermore, the evidence attributing the improvement specifically to nonlinearity is incomplete. The covariance-aware baseline is diagonal variance whitening (sec. 5 and Appendix E). Beating this baseline does not exclude improvements obtainable from off-diagonal linear structure. Accordingly, describing it as the best linear reweighting in Figure 4 and attributing the remaining gain to nonlinearity are stronger conclusions than the comparison supports. The original IEM's Gaussian case already produces a full-covariance Mahalanobis distance from linear denoisers, making a stronger linear control particularly relevant here (Ohayon et al., 2026, sec. 2.2).

The external results also limit the practical conclusion that can compensate for this explanatory gap. Table 5 reports a human MOS advantage of +0.009 on 480 live/anchor files, approximately no difference on 382 simulated files without codecs, and a disadvantage of -0.029 on 2118 simulated files containing codecs. These mixed findings are valuable and appropriately disclosed, but they do not establish that passing the gate identifies a generally preferable perceptual distance.

How to improve: Correct the relationship asserted between noise conditioning and linearity, and limit the AdaLN conclusion to what the reported gate outcomes demonstrate. Add a matched linear, noise conditioned denoiser or a regularized full covariance baseline on the same features and training data. Evaluate that control on the existing cross-family and human rating data. This would directly test whether the observed benefit requires nonlinear denoising. A comparison with an alternative feature-based metric could also preserve the encoder: SpeechBERTScore, already cited in the paper, supports WavLM Base+ features, so the representation need not change when comparing alternative feature-comparison rules.

Questions for authors

These questions address the paper's existing central claims and could materially change my assessment.

  • Clarification and reanalysis of existing results: For the 27 checkpoints, what are the direct ordering disagreements between SpeechIEM and latent (L_2), measured on the same within-utterance cross-family comparisons? Can these be reported separately from differences in agreement with PESQ, ESTOI, and SISDR? This would determine whether the gate's empirical validation supports its stated structural purpose.

  • Theoretical clarification: What assumptions justify the implication from weak or absent (σ)(\sigma) dependence to input linearity in sec. 3-4.1? If that implication is withdrawn, how should the third collapse route and the claimed admissibility benefit of AdaLN be reformulated? Is there evidence of different admission reliability beyond Table 1's identical matched gate outcomes?

  • Targeted additional validation: How does the main checkpoint compare with a linear denoiser using the same noise conditioning and integration, or with a regularized full-covariance baseline on the same representations? This control would materially affect whether the reported advantage can be attributed to nonlinear denoising.

Reviewer_ofxwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SpeechIEM, a label-free, full-reference metric for measuring the distance between degraded speech and its clean reference. It applies the Information-Estimation Metric to frozen speech representations, using a denoiser to capture the structure of the latent distribution without human quality labels. The paper also analyzes when the resulting metric collapses to rescaled Euclidean distance, proposes a diagnostic gate, and evaluates its usefulness for quality assessment and as a training objective.

Critical strengths
  1. Convincing failure-mode analysis. The paper clearly illustrates how the denoiser-based metric can collapse to a rescaled L2 distance despite apparently successful training. The analysis provides useful guidance for determining whether the denoiser contributes information beyond the underlying representation.
  2. Transparent reporting of limitations. The authors report both favorable and unfavorable results. In particular, the human-rating evaluation shows a small advantage on live/anchor conditions but worse performance than latent L2 on codec simulations. These results help establish the conditions under which the method is useful.
Critical weaknesses
  1. Limited novelty in the metric construction. The underlying distance formulation is inherited from IEM, making its application to speech an incremental contribution. Although the collapse analysis and diagnostic gate add value, the manuscript could more clearly establish the significance of these contributions beyond the speech-domain adaptation.
  2. Limited evidence of generalization across representations. All latent-based experiments use the same pretrained wavlm-base-plus encoder. Sweeping its hidden states provides useful evidence about layer selection, but does not establish whether the findings extend to other representations, such as HuBERT
  3. Insufficient comparison with existing label-free speech metrics. The same-encoder L2 baseline appropriately isolates the denoiser’s contribution. However, without direct comparisons against established methods such as SpeechBERTScore, the experiments do not establish SpeechIEM’s practical advantage over existing label-free approaches.
  4. The linear baseline is incomplete. Diagonal Mahalanobis distance accounts for feature variances but ignores correlations between features. Outperforming this baseline does not establish that nonlinear denoising is necessary to obtain the observed gains. A regularized full-covariance Mahalanobis distance or a linear-denoiser baseline would strengthen this analysis.
  5. Perceptual benefits as a training objective remain unverified. Although Appendix F evaluates metric scores against existing human ratings, Tables 10–12 do not include human evaluation of the outputs produced by enhancers trained with SpeechIEM. Improvements in spectral distance alone do not establish improvements in perceived speech quality. A listening study comparing the enhanced outputs would provide stronger evidence for this application.
Questions for authors
  1. Comparison with existing methods: Could you compare SpeechIEM with an existing label-free speech metric, such as SpeechBERTScore, on the same human-rated recordings? Where feasible, using the same encoder would help isolate differences in the distance computation.
  2. Representation and layer selection: Hidden state 0 is selected because the denoiser contributes the largest improvement over L2. How does SpeechIEM compare with the strongest latent-L2 baseline when each method’s layer is selected independently on validation data? Is there evidence that the benefit extends beyond wavlm-base-plus?
  3. Contribution of nonlinear denoising: Does the advantage persist against a regularized full-covariance Mahalanobis distance or a linear denoiser? Such controls would help determine whether the gains require nonlinear modeling or can be explained by accounting for correlations between latent features.
  4. Perceptual benefit during optimization: Appendix F validates metric scores against human ratings, while Tables 10–12 evaluate enhanced outputs using automatic measures. Do you have listening-test evidence that adding SpeechIEM to the training objective improves perceived quality? Without it, how narrowly should the training-objective results be interpreted?
  5. Statistical implementation: Could you clarify how the exact sign-flip p-values in Tables 10–12 were calculated? With six paired cells and 64 assignments, values such as 0.046 and 0.077 appear consistent with an added-pseudocount convention rather than ordinary exhaustive enumeration. Please specify the test statistic, tail rule, and correction used.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.