- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tests whether higher accuracy from contextual fusion also improves confidence estimates on remaining errors. Using fixed-candidate MEG speech retrieval, the authors add contextual priors to local neural scores and measure confidence by the top-two score margin.
They uncover a confidence-ordering reversal based on the correct answer's initial rank: on MEG-MASC, overall correctness AUROC is 0.872, but the AUROC separating fixed errors from unfixed ones drops from 0.704 (initial ranks 2–3) to 0.392 (ranks 21–50), with MOUS showing the same pattern. This happens because a fixed answer must first overcome its initial score gap, keeping its final margin small, whereas an unfixed error can build a wide gap between two wrong choices.
The authors confirm this mechanism through fusion-weight and prior tests, then evaluate estimators using local, fused, and prior features. On MEG-MASC with history aggregation, their full estimator raises emission from 56.7% to 74.5% at a matched 92.24% coverage. Overall, the paper explains post-fusion confidence failures and shows which signals help estimate confidence reliably.
Critical strengths
The rank-conditioned analysis uncovers a critical issue masked by overall scores: while pooled AUROC reaches 0.872, confidence ordering drops below chance for lower-ranked candidates. Since errors starting beyond rank 20 make up 46.6% of post-fusion errors on MEG-MASC, this finding has clear practical impact. Results across two datasets, within-sentence checks, and repairability controls further back this up, and the authors rightly note sparse repairs and higher uncertainty in the lowest MOUS rank group.
The fusion-weight sweep keeps score streams, histories, and candidate sets fixed to cleanly isolate how prior weights shape confidence ordering. Score-deficit tests and matched feature comparisons also clarify the gap between accuracy, correction reach, and confidence ordering. Finally, the authors transparently report non-monotonic free-running trends, loose LM-prior bounds, and test-time violations of validation policies.
Critical weaknesses
The feature comparisons show that keeping local and prior signals helps—raising emission from 66.6% (fused summaries) to 72.5% (local-plus-fused) and 74.5% (all features)—but do not prove they are strictly necessary over richer fused-only features. Without stronger fused-score baselines (as seen in Li et al., 2021 and SR-CEM), advising against using fused scores alone is overreaching.
The proposed explanation and remedy also need clearer boundaries. Margin bounds are conditional and do not guarantee monotonic boundary shifts, while weight sweeps mix margin effects with group membership changes. Furthermore, gains on hard cases remain unclear: Table 23 lacks fused-only or prior-only baselines at matched operating points, Table 20 shows prior features beating the full model on deep-group top-five discrimination, and coverage stays at 7.3% beyond rank 50. Since learned tests only cover MEG-MASC with history aggregation, the Discussion should separate proven findings from broader claims and limit ordering-invariance to strictly increasing scalar recalibration.
Questions for authors
-
Can you test a richer fused-only estimator using normalized top-candidate scores or probability profiles, and then measure the added gains from local and prior features? Please match the training data, model size, and validation policy selection, and report paired uncertainty for these gains (including the final 2.0-point gain from prior features).
-
Can you separate margin shifts from group membership changes during weight sweeps? Tracking pairwise ordering for examples that keep the same repair or residual status—alongside shifts between groups—would make the mechanism clear. Does this ordering shift still happen within stable subsets under both history aggregation and a free-running LM prior?
-
Can you compare group-specific reliability–emission tradeoffs for the full, decoder-score, fused-only, and prior-feature estimators at ranks 11–20, 21–50, and beyond 50? Please keep output sizes fixed (including K = 5), apply thresholds globally, and use initial rank only for evaluation. Does the prior-feature estimator's higher AUROC in deep groups lead to better selective output? Testing these features on MOUS or with an LM prior would also show how well the results generalize.
3 comments
Response to Reviewer yunW [2/2]
Q3 (group-wise selective output, K = 5, and generality)
Yes. In the deep groups, the prior-feature estimator gives the highest coverage at every emission level, so its stronger AUROC does translate into better selective output (General Response, Table R4).
The comparison follows the controls you specified: one global threshold rule, output size fixed at K = 5, and initial rank used only to group the results. For example, when 20% of the windows at ranks 11–20 are emitted, the prior-feature estimator covers 86.4%, the full estimator 72.8%, and the fused profile 43.3%, which is the group's base rate. Figure 7 shows the curves for all three groups, including beyond rank 50.
The same holds on MOUS (Figure 7D–F; the feature comparison on MOUS is in Tables 32 and 33) and, under an LM prior, in generative decoding (General Response, Table R1).
W2 (effectiveness in the difficult groups; original Tables 20 and 23)
The remedy is effective in the difficult groups, and following your suggestion, we have completed the comparison: all four estimators, at a fixed output size and at matched operating points, on both datasets (Tables 27, 29, and 30; Figure 7).
The prior features' advantage over the full estimator (original Table 20, now Table 23) carries into selective output (Q3): in the deep groups, the information is in the prior scores.
The prior clearly improves coverage. In the most difficult group (beyond rank 50), the correct candidate is rarely among the emitted candidates, and the estimator's role is not to pick the correct answer but to decide whether to emit. Our results show that retaining the prior still doubles coverage there: at matched overall emission, from 3.6% with decoder scores to 7.4% with the full estimator (+3.9 [+3.1, +4.6]), and from 0.7% to 4.4% on MOUS.
Original Table 23 (now Table 26) is unchanged; the comparison you asked for is in the new Tables 27 and 29.
W2 (limits of the account and the Discussion)
We have revised the text as you suggested.
The bound (Section 4.2). It is now stated as a sufficient condition for reversal that constrains ordering at each weight rather than fixing how the boundary moves between weights; the boundary's movement is tested directly by the stable-subset analysis in Q2 (General Response R3).
Recalibration (Discussion). "No calibration can undo the reversal" is now "Any strictly increasing recalibration of the margin preserves its ordering and so cannot undo the reversal".
Scope (Discussion). The conclusion about priors is restricted to "the priors and decoding settings we test".
Summary of revisions. For W1 and Q1, we added the fused profile and the paired gains (Table 2(b), Section 4.4, Appendices J.2 and J.4, Tables 23 and 31–33). For W2 and Q3, we revised the bound in Section 4.2 and two sentences in the Discussion, and we added selective output by rank group on both datasets (Appendix J.3, Figure 7, Tables 27, 29, 30 and 35). For Q2, we added the stable-subset analysis under both priors (Section 4.3, Appendices G.1 and G.4, Tables 11 and 17).
Thank you again for the time and effort you have given to the review. Your suggestions improved our paper. If our responses resolve your concerns, we would be grateful if you would consider raising your rating.
Thank you for your reply. I will keep my score. The paper still lacks a comparison with other fusion methods.
Response to Reviewer yunW [1/2]
Dear Reviewer yunW,
Thank you for your detailed, specific review; for noting that the rank-conditioned analysis reveals "a meaningful failure hidden by pooled evaluation"; and for appreciating our transparent reporting of nonmonotonic free-running results and loose LM-prior bounds. We have run all three analyses you asked for. Because new tables were added, your original Tables 20 and 23 are Tables 23 and 26 in the revision.
W1 and Q1 (richer fused-only estimator and paired gains)
The richer fused-only estimator you suggested still falls short of the estimators that use the prior's information: the gains from retaining the local and prior scores are larger, and each is significant (General Response R2, Table R2).
Following your suggestion, the fused profile contains the 20 largest probabilities of the softmax over fused scores together with the fused entropy-based confidence, 21 features in total. Like SR-CEM (Jia & Van hamme, 2026), it reads only output scores and their ranks and learns confidence with a small network; here, the scores are the fused ones. Training data, model capacity, and validation-based threshold selection match the original estimators.
For the 2.0-point contribution you asked about (decoder scores 72.5% → full 74.5%), the difference is +2.03 percentage points [+1.14, +3.24], positive in all three local decoders, and +6.99 [+6.19, +7.67] on MOUS (Table 33). The other steps are in Table R2 and in the revised Table 33. This isolates the incremental benefit of retaining separate information.
In addition, the Discussion now states the conclusion for the fused-score summaries we test: "the other fused-score summaries we test, from entropy to the fused profile, reverse as well."
Q2 (margin versus composition changes in weight sweeps)
Yes. On stable subsets, where no window changes group, ordering still moves under both priors, so the movement does not come from changes in group membership (Tables 11 and 17).
A stable subset, as you propose, keeps only the windows that are repairs at both adjacent weights or residual errors at both. Its repair-separation AUROC can therefore change only through the margins.
Under history aggregation, stable-subset AUROC rises significantly in all 27 cells, with increases ranging from 0.096 to 0.392. Windows change group in one direction only: 47 to 1,269 residual errors per cell become repairs, against at most 1.3 repairs that become residual errors. These new repairs enter with small margins, so, interestingly, the change in membership by itself lowers AUROC, which further strengthens the causal test of the mechanism; thank you for your suggestion.
The same holds under the free-running LM prior. Up to α = 1, including the validation-selected α = 0.15, stable-subset AUROC rises in all 30 cells, with increases ranging from 0.014 to 0.314. Beyond α = 1, where the accuracy gain is already negative, repairs turn back into residual errors (435 against 28 at ranks 2–3 between α = 2 and 3), and the further rise of shallow-group AUROC comes from this change in membership, not from the margins.
Where the prior helps accuracy, the ordering therefore moves through the margins, as the account predicts. Section 4.3 reports this result, and Appendices G.1 and G.4 give the full analysis.
Reference. Jia, Y., & Van hamme, H. (2026). Leveraging beam search information for confidence estimation in E2E ASR. IEEE Open Journal of Signal Processing, 7.