acceptodds
Under review as a conference paper at ICLR 2027

Confidence-Ordering Reversal under Contextual Priors in Neural Decoding

Abstract

Contextual priors improve neural-to-language decoding by reshaping candidate scores. However, confidence is read from the same reshaped scores, so the errors a prior leaves behind can become more confident with no change in accuracy to reveal it. In this work, we ask how a prior shapes the confidence of these errors, studying speech retrieval on MEG-MASC and MOUS with local decoding scores and a contextual prior combined by additive shallow fusion, and the fused top-two margin as confidence. Among initially incorrect predictions, we find a **confidence-ordering reversal**: a larger margin makes a *repair*, an error that fusion corrects, more likely when the correct candidate starts near the top of the local ranking, but less likely when it starts lower. On MEG-MASC, pooled correctness AUROC is 0.87, yet the AUROC separating repairs from *residual errors*, which fusion leaves uncorrected, falls from 0.70 at initial ranks 2–3 to 0.39 at ranks 21–50. Errors starting beyond rank 20, inside the reversed region, make up 46.6% of all errors after fusion. We propose a score-level account: a repair must first close the correct candidate's initial deficit, which limits its final margin, whereas a residual error can build a large margin between two incorrect candidates. Through a causal intervention that changes only the fusion weight, we show that the reversal moves to deeper ranks, as the account predicts, and that under a word-level LM prior it keeps moving after accuracy gain peaks, so a weight chosen for accuracy does not settle confidence. Guided by this account, we read local and prior scores separately: read before fusion, the prior's own scores already separate repairs from residual errors where the fused margin reverses, and estimators built on local and prior scores let a selective decoder answer on 74.5% of windows instead of 56.7%, with 92% of its output sets still containing the correct candidate. Confidence after contextual fusion should retain the local and contextual evidence behind each prediction, not just the fused scores. [Project Page](https://confidencereversal.github.io/) [Code](https://anonymous.4open.science/r/Confidence_Reversal_Neural_Decoding/)

Primary area:
applications to neuroscience & cognitive science
Keywords:
neural decoding, confidence estimation, shallow fusion, contextual priors, selective prediction, MEG

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yunWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests whether higher accuracy from contextual fusion also improves confidence estimates on remaining errors. Using fixed-candidate MEG speech retrieval, the authors add contextual priors to local neural scores and measure confidence by the top-two score margin.

They uncover a confidence-ordering reversal based on the correct answer's initial rank: on MEG-MASC, overall correctness AUROC is 0.872, but the AUROC separating fixed errors from unfixed ones drops from 0.704 (initial ranks 2–3) to 0.392 (ranks 21–50), with MOUS showing the same pattern. This happens because a fixed answer must first overcome its initial score gap, keeping its final margin small, whereas an unfixed error can build a wide gap between two wrong choices.

The authors confirm this mechanism through fusion-weight and prior tests, then evaluate estimators using local, fused, and prior features. On MEG-MASC with history aggregation, their full estimator raises emission from 56.7% to 74.5% at a matched 92.24% coverage. Overall, the paper explains post-fusion confidence failures and shows which signals help estimate confidence reliably.

Critical strengths

The rank-conditioned analysis uncovers a critical issue masked by overall scores: while pooled AUROC reaches 0.872, confidence ordering drops below chance for lower-ranked candidates. Since errors starting beyond rank 20 make up 46.6% of post-fusion errors on MEG-MASC, this finding has clear practical impact. Results across two datasets, within-sentence checks, and repairability controls further back this up, and the authors rightly note sparse repairs and higher uncertainty in the lowest MOUS rank group.

The fusion-weight sweep keeps score streams, histories, and candidate sets fixed to cleanly isolate how prior weights shape confidence ordering. Score-deficit tests and matched feature comparisons also clarify the gap between accuracy, correction reach, and confidence ordering. Finally, the authors transparently report non-monotonic free-running trends, loose LM-prior bounds, and test-time violations of validation policies.

Critical weaknesses

The feature comparisons show that keeping local and prior signals helps—raising emission from 66.6% (fused summaries) to 72.5% (local-plus-fused) and 74.5% (all features)—but do not prove they are strictly necessary over richer fused-only features. Without stronger fused-score baselines (as seen in Li et al., 2021 and SR-CEM), advising against using fused scores alone is overreaching.

The proposed explanation and remedy also need clearer boundaries. Margin bounds are conditional and do not guarantee monotonic boundary shifts, while weight sweeps mix margin effects with group membership changes. Furthermore, gains on hard cases remain unclear: Table 23 lacks fused-only or prior-only baselines at matched operating points, Table 20 shows prior features beating the full model on deep-group top-five discrimination, and coverage stays at 7.3% beyond rank 50. Since learned tests only cover MEG-MASC with history aggregation, the Discussion should separate proven findings from broader claims and limit ordering-invariance to strictly increasing scalar recalibration.

Questions for authors
  1. Can you test a richer fused-only estimator using normalized top-candidate scores or probability profiles, and then measure the added gains from local and prior features? Please match the training data, model size, and validation policy selection, and report paired uncertainty for these gains (including the final 2.0-point gain from prior features).

  2. Can you separate margin shifts from group membership changes during weight sweeps? Tracking pairwise ordering for examples that keep the same repair or residual status—alongside shifts between groups—would make the mechanism clear. Does this ordering shift still happen within stable subsets under both history aggregation and a free-running LM prior?

  3. Can you compare group-specific reliability–emission tradeoffs for the full, decoder-score, fused-only, and prior-feature estimators at ranks 11–20, 21–50, and beyond 50? Please keep output sizes fixed (including K = 5), apply thresholds globally, and use initial rank only for evaluation. Does the prior-feature estimator's higher AUROC in deep groups lead to better selective output? Testing these features on MOUS or with an LM prior would also show how well the results generalize.

3 comments
AuthorsOpenReview ↗

Response to Reviewer yunW [1/2]

Dear Reviewer yunW,

Thank you for your detailed, specific review; for noting that the rank-conditioned analysis reveals "a meaningful failure hidden by pooled evaluation"; and for appreciating our transparent reporting of nonmonotonic free-running results and loose LM-prior bounds. We have run all three analyses you asked for. Because new tables were added, your original Tables 20 and 23 are Tables 23 and 26 in the revision.

W1 and Q1 (richer fused-only estimator and paired gains)

The richer fused-only estimator you suggested still falls short of the estimators that use the prior's information: the gains from retaining the local and prior scores are larger, and each is significant (General Response R2, Table R2).

Following your suggestion, the fused profile contains the 20 largest probabilities of the softmax over fused scores together with the fused entropy-based confidence, 21 features in total. Like SR-CEM (Jia & Van hamme, 2026), it reads only output scores and their ranks and learns confidence with a small network; here, the scores are the fused ones. Training data, model capacity, and validation-based threshold selection match the original estimators.

For the 2.0-point contribution you asked about (decoder scores 72.5% → full 74.5%), the difference is +2.03 percentage points [+1.14, +3.24], positive in all three local decoders, and +6.99 [+6.19, +7.67] on MOUS (Table 33). The other steps are in Table R2 and in the revised Table 33. This isolates the incremental benefit of retaining separate information.

In addition, the Discussion now states the conclusion for the fused-score summaries we test: "the other fused-score summaries we test, from entropy to the fused profile, reverse as well."

Q2 (margin versus composition changes in weight sweeps)

Yes. On stable subsets, where no window changes group, ordering still moves under both priors, so the movement does not come from changes in group membership (Tables 11 and 17).

A stable subset, as you propose, keeps only the windows that are repairs at both adjacent weights or residual errors at both. Its repair-separation AUROC can therefore change only through the margins.

Under history aggregation, stable-subset AUROC rises significantly in all 27 cells, with increases ranging from 0.096 to 0.392. Windows change group in one direction only: 47 to 1,269 residual errors per cell become repairs, against at most 1.3 repairs that become residual errors. These new repairs enter with small margins, so, interestingly, the change in membership by itself lowers AUROC, which further strengthens the causal test of the mechanism; thank you for your suggestion.

The same holds under the free-running LM prior. Up to α = 1, including the validation-selected α = 0.15, stable-subset AUROC rises in all 30 cells, with increases ranging from 0.014 to 0.314. Beyond α = 1, where the accuracy gain is already negative, repairs turn back into residual errors (435 against 28 at ranks 2–3 between α = 2 and 3), and the further rise of shallow-group AUROC comes from this change in membership, not from the margins.

Where the prior helps accuracy, the ordering therefore moves through the margins, as the account predicts. Section 4.3 reports this result, and Appendices G.1 and G.4 give the full analysis.

Reference. Jia, Y., & Van hamme, H. (2026). Leveraging beam search information for confidence estimation in E2E ASR. IEEE Open Journal of Signal Processing, 7.

AuthorsOpenReview ↗

Response to Reviewer yunW [2/2]

Q3 (group-wise selective output, K = 5, and generality)

Yes. In the deep groups, the prior-feature estimator gives the highest coverage at every emission level, so its stronger AUROC does translate into better selective output (General Response, Table R4).

The comparison follows the controls you specified: one global threshold rule, output size fixed at K = 5, and initial rank used only to group the results. For example, when 20% of the windows at ranks 11–20 are emitted, the prior-feature estimator covers 86.4%, the full estimator 72.8%, and the fused profile 43.3%, which is the group's base rate. Figure 7 shows the curves for all three groups, including beyond rank 50.

The same holds on MOUS (Figure 7D–F; the feature comparison on MOUS is in Tables 32 and 33) and, under an LM prior, in generative decoding (General Response, Table R1).

W2 (effectiveness in the difficult groups; original Tables 20 and 23)

The remedy is effective in the difficult groups, and following your suggestion, we have completed the comparison: all four estimators, at a fixed output size and at matched operating points, on both datasets (Tables 27, 29, and 30; Figure 7).

The prior features' advantage over the full estimator (original Table 20, now Table 23) carries into selective output (Q3): in the deep groups, the information is in the prior scores.

The prior clearly improves coverage. In the most difficult group (beyond rank 50), the correct candidate is rarely among the emitted candidates, and the estimator's role is not to pick the correct answer but to decide whether to emit. Our results show that retaining the prior still doubles coverage there: at matched overall emission, from 3.6% with decoder scores to 7.4% with the full estimator (+3.9 [+3.1, +4.6]), and from 0.7% to 4.4% on MOUS.

Original Table 23 (now Table 26) is unchanged; the comparison you asked for is in the new Tables 27 and 29.

W2 (limits of the account and the Discussion)

We have revised the text as you suggested.

The bound (Section 4.2). It is now stated as a sufficient condition for reversal that constrains ordering at each weight rather than fixing how the boundary moves between weights; the boundary's movement is tested directly by the stable-subset analysis in Q2 (General Response R3).

Recalibration (Discussion). "No calibration can undo the reversal" is now "Any strictly increasing recalibration of the margin preserves its ordering and so cannot undo the reversal".

Scope (Discussion). The conclusion about priors is restricted to "the priors and decoding settings we test".

Summary of revisions. For W1 and Q1, we added the fused profile and the paired gains (Table 2(b), Section 4.4, Appendices J.2 and J.4, Tables 23 and 31–33). For W2 and Q3, we revised the bound in Section 4.2 and two sentences in the Discussion, and we added selective output by rank group on both datasets (Appendix J.3, Figure 7, Tables 27, 29, 30 and 35). For Q2, we added the stable-subset analysis under both priors (Section 4.3, Appendices G.1 and G.4, Tables 11 and 17).

Thank you again for the time and effort you have given to the review. Your suggestions improved our paper. If our responses resolve your concerns, we would be grateful if you would consider raising your rating.

Thank you for your reply. I will keep my score. The paper still lacks a comparison with other fusion methods.

Reviewer_CVdbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how contextual priors affect confidence in neural-to-language decoding. The authors identify a “confidence-ordering reversal”: among initially incorrect predictions, a larger fused top-two margin is associated with a higher probability of repair when the correct candidate initially ranks near the top, but with a lower probability of repair when it starts deeper in the ranking. The phenomenon is demonstrated on MEG-MASC and MOUS under several contextual priors. The paper further provides a score-level explanation based on the initial deficit of the correct candidate and the contribution of the contextual prior. By varying only the fusion weight, the authors show that the reversal boundary changes systematically with prior strength. Finally, they demonstrate that retaining local and prior scores separately enables better confidence estimation and selective output than relying only on the final fused margin.

Critical strengths
  1. The paper identifies a non-obvious and practically relevant failure mode of confidence after contextual fusion. The distinction between pooled correctness confidence and confidence over repair versus residual-error outcomes is insightful, and the reversal is replicated across two MEG datasets rather than being demonstrated on a single experimental setting.
  2. The empirical analysis is unusually thorough. The paper combines a score-level account, controlled fusion-weight interventions, multiple contextual priors, participant-level bootstrap analyses, and selective-output experiments. The observation that local and prior scores retain useful information that is lost in the final fused margin provides a concrete practical implication rather than only a descriptive phenomenon.
Critical weaknesses
  1. The empirical scope remains relatively narrow. Both main datasets use offline MEG speech retrieval with fixed candidate pools, and the analysis benefits from separately accessible local and prior score streams. It is therefore unclear how strongly the reported confidence-ordering reversal generalizes to generative neural decoding, dynamically changing candidate sets, or systems in which contextual feedback affects subsequent predictions. The paper acknowledges this limitation, but the broader claims about contextual fusion should be interpreted accordingly.
  2. The proposed score-level explanation does not fully account for all observed reversals. The repair-margin bound explains much of the history-aggregation behavior, but the paper reports that the corresponding sufficient condition certifies only 1 of 29 negative cells for the word-level teacher-forced LM prior. Thus, the empirical reversal appears more general than the current theoretical account, and the conditions under which additive fusion should produce the reversal remain only partially characterized.
Questions for authors
  1. Can the authors clarify the scope of the theoretical claim for additive fusion? In particular, what additional conditions beyond the current repair-margin bound would be needed to predict the reversal observed with the word-level LM prior?
  2. The main selective-output improvement in Table 2 is reported at matched test-curve coverage, while validation-selected policies are reported separately. Could the authors clarify which result should be interpreted as the realistic deployment comparison, and whether the main conclusion about the benefit of retained local/prior information remains equally strong using only validation-selected operating points?
  3. Do the authors have evidence, even preliminary, that the same ordering reversal occurs in a setting with generated or changing candidate sets rather than fixed retrieval candidates?
1 comment
AuthorsOpenReview ↗

Response to Reviewer CVdb

Dear Reviewer CVdb,

Thank you for your careful, detailed review and for noting that the paper identifies "a non-obvious and practically relevant failure mode of confidence after contextual fusion" with an "unusually thorough" empirical analysis. Below, we address each point.

W1 and Q3 (generated or changing candidate sets)

Yes. The same ordering reversal occurs in generative decoding, with generated hypotheses and contextual feedback (revised Section 4.3, Appendix H, Table 20).

Each sentence is generated word by word by beam search rather than retrieved from a pool of candidate sentences; the hypotheses change with the decoding path, and the LM conditions on each hypothesis's own decoded history, so each output affects later predictions.

At the weight selected for WER, confidence still looks good overall (pooled correctness AUROC near 0.80), but the reversal still appears once errors are stratified by initial rank. Within the rank groups, estimating confidence with the prior's information is again better than estimating it from the fused scores alone (Table 34; General Response R1). The findings from retrieval are therefore confirmed in generative decoding, and the broader claims about contextual fusion now rest on both settings.

W2 and Q1 (scope of the theoretical account)

The account comes from additive fusion itself. It is a sufficient condition, so it cannot guarantee a complete prediction of where the reversal occurs. Under a language-model prior, we think the key additional ingredient is the prior's distribution, that is, how the prior scores the incorrect candidates (General Response R3).

The bound limits only the repairs' margins; whether the reversal occurs also depends on the residual errors' margins, which depend on each prior. Under the word-level LM, two steps close part of the gap. Replacing the prior's full score range with the correct candidate's prior advantage over the strongest local competitor raises the cells in which the bound proves the reversal from 1 to 11 of 29 (Section 4.2; Appendix F.3). For the remaining cells, one needs to know which incorrect candidate the prior moves into second place: when the strongest local competitor is still the runner-up after fusion, the bound gives the repair's margin exactly; otherwise the real margin is smaller than the bound.

The revised Section 4.2 states that the bound is a sufficient condition, and the Discussion restricts its empirical claims about confidence ordering to the priors and decoding settings we test.

Q2 (realistic deployment comparison)

The realistic deployment comparison is the one with validation-selected policies. We have added its key result to Section 4.4, and the full results are in Table 25.

Table Q2. Emission at matched coverage (Table 2), and emission and coverage on test under validation-selected policies with a coverage target of 90% (Table 25). MEG-MASC.

FeaturesEmission at matched coverageEmission, validation-selectedCoverage, validation-selected
Fused margin (margin-binned in Table 25)56.7%55.9%91.7%
Fused scores (6 features)66.6%73.2%89.8%
Fused profile (21 features)67.5%76.3%88.8%
Decoder scores (fused + local)72.5%73.6%91.8%
Full (fused + local + prior)74.5%74.5%92.2%

The two results are equally strong: confidence estimation needs to include the local and prior scores. Under validation-selected policies, the learned estimators that use only fused scores fall below the coverage target (89.8% and 88.8%), and adding the local and prior scores restores it (91.8% and 92.2%).

Summary of revisions. For W1 and Q3, we added word-sequence decoding (Section 4.3, Appendix H, Tables 20 and 34). For W2 and Q1, Section 4.2 now states the bound as a sufficient condition and reports the 11 of 29 cells. The Discussion restricts its empirical claims to the priors and decoding settings we test. For Q2, Section 4.4 now reports the comparison under validation-selected policies (Table 25), and Appendix J.3 adds the comparison by rank group (Figure 7, Tables 27, 29, 30 and 35).

Thank you again for recognising our work. Your comments helped us strengthen the scope of the theory and the deployment comparison. If our responses resolve your concerns, we would be grateful if you would consider raising your rating.

Reviewer_gnuBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The work addresses contextual priors in neural speech decoding. In particular, it shows that in some cases adding a prior can make incorrect predictions more confident. The authors identify the "reverse order" problem in which when the correct candidate is initially ranked low, a larger confidence margin can be associated with errors that remain uncorrected rather than successful corrections. This effect is analyzed using additive score fusion on two MEG datasets. To address the problem, the authors propose a small MLP using neural, contextual, and fused-score features to estimate prediction correctness.

Critical strengths

S1. The paper identifies and demonstrates an interesting limitation of confidence estimation with contextual priors. Results are supported by experiments on two MEG datasets.

S2. The analysis is systematic, examining different priors and fusion strengths, with an explanation for the observed behavior.

Critical weaknesses

W1. Novelty is limited. The "confidence reversal" follows naturally from additive score fusion. This is not surprising nor general beyond the specific setup.

W2. Solution is relatively standard. A small MLP is trained on score features to predict correctness. Most of the improvement comes from richer fused and neural features, while adding prior features provides only a modest additional gain.

W3. The experiments are limited to offline retrieval with a fixed candidate pool and known sentence structure. The strongest language-model results also rely on teacher-forced histories. It remains unclear whether the findings and proposed estimator generalize to realistic brain-to-text decoding.

Questions for authors

Q1. Why a separate confidence estimator is used rather than learning to combine neural and contextual scores directly? Would a learned fusion model or knowledge distillation provide a stronger baseline?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gnuB [1/2]

Dear Reviewer gnuB,

Thank you for your review, and for noting that our paper identifies "an interesting limitation of confidence estimation with contextual priors" through a "systematic" analysis. This is how we see the paper as well: it answers why brain-to-text decoding sometimes makes confident, fluent errors, with a finding and its explanation. Its focus is therefore on why this finding stays hidden, on the mechanism and the causal experiment that tests it, and on its practical value, rather than on presenting a better-performing model.

W1 (novelty and generality of the reversal)

Our novelty is that we are the first to stratify errors by the correct candidate's initial rank; only then can the confidence reversal, a widely hidden phenomenon, be seen and defined. Pooled over all windows, confidence looks sound (correctness AUROC 0.872 on MEG-MASC); stratified, repair-separation AUROC falls from 0.704 at ranks 2–3 to 0.392 at ranks 21–50 (Section 4.1). The reversal can explain the confident errors in brain decoding that earlier papers reported and left open, and it shows where the finding is useful: keep the prior when estimating confidence. To our knowledge, no earlier work reports this reversal; the closest work we know of is in speech recognition: it shows that shallow fusion can degrade confidence overall (Section 2), but it does not stratify errors by initial rank, so the reversal could not be observed. If you have such work in mind, we would be grateful for the reference and will discuss it in the paper.

A simple explanation does not make the finding obvious. The naturalness you mention was not easy to achieve: it seems so reasonable because we excluded nuisance variables, examined the causal relation between the prior's contribution and the confidence ordering alone, and gave it the simplest explanation we could. This is a strength of the presentation, not a limitation of novelty.

In addition, the reversal is not a necessary consequence of additive fusion: our mechanism provides a bound on a repair's margin (Section 4.2). Where the ordering reverses still depends on the type of prior: The sentence-level teacher-forced LM prior and the history-aggregation prior improve accuracy by nearly the same amount, 14.74 and 14.40 percentage points, yet keep positive ordering through ranks 11–20 and 4–5, respectively (Section 4.3 of the submitted version). This also shows why stratification matters: looking only at overall performance, as is common practice, one would not see that the reversal's position differs sharply under the same overall metric.

Finally, the finding holds broadly: In the submitted version, the reversal already appears in every setting we test (Sections 4.1 and 4.3). The mechanism applies to any additive fusion of this kind, and current decoders such as Brain2Qwerty (Lévy et al., 2026, Nature Neuroscience) bring in the contextual prior with the same shallow-fusion formula; the revision adds the same result for generative decoding (W3).

W2 (the estimator and the gain from prior features)

The estimator needs to be standard and as simple as possible, because it tests the practical value of the mechanism we propose: fusion discards information held in the local and prior scores, and the fused-score features we test, up to 21 of them, cannot recover the confidence information held in the prior.

The prior's contribution has to be read by comparison: the seven prior features, added last, add 2 and 7.5 times what 15 further fused-score features add (+1.76 and +6.60 points of emission, against +0.88 on each dataset). On MOUS, this is the largest step of all, above the local features (+6.17; General Response, Table R2).

Where the local errors are deep, the cases the paper studies specifically, the prior features add more than the fused and local scores together. Coverage (%) of selective output within each group, at 20% emission and K = 5:

Dataset, ranksBase rateFused scores onlyFused + localFused + local + prior
MEG-MASC, ranks 11–2043.843.354.272.8
MEG-MASC, ranks 21–5012.512.516.434.1
MOUS, ranks 11–2024.123.426.553.6
MOUS, ranks 21–503.63.74.014.1

The fused scores stay at the base rate, the local scores add 0.3 to 10.9 points, and the prior features then add 10.1 to 27.1 points (General Response, Table R4; Table 35 of the revised paper).

The gain falls exactly where the mechanism locates the problem, confirming that our analysis leads to an improvement that is both targeted and effective. Across all windows, the gain is spread over a larger total, since a large share of them (44% on MEG-MASC) are already correct before fusion and should not be changed.

AuthorsOpenReview ↗

Response to Reviewer gnuB [2/2]

W3 (retrieval setting and teacher-forced histories)

The same ordering reversal occurs in generative decoding. At the weight selected for WER, the fused margin ranks repairs below residual errors in every populated rank group (AUROC 0.235–0.481), while the prior's own margin ranks them correctly (0.523–0.789). Within the rank groups, retaining the local and prior scores again improves confidence over the fused scores alone (Table 34; General Response, Table R1; Section 4.3, Appendix H).

For realistic brain-to-text decoding, the setting follows the current state of the art (Willett et al., 2023; d'Ascoli et al., 2025; Lévy et al., 2026): the sentence structure is taken from the stimulus, each sentence is decoded word by word with beam search, without a pool of candidate sentences, and the LM prior conditions on the decoder's own output.

The reversal does not depend on teacher forcing. The submitted version already reports each LM prior with free-running history, where it conditions on predicted words, and the reversal appears there as well (Section 4.3). The estimator results do not use teacher forcing: they use the history-aggregation prior, which aggregates neural evidence from preceding windows and uses no text, and, in generative decoding, the decoder's own output as history. In this paper, teacher-forced history serves as a simulated upper bound for the LM prior (Section 3.3). Decoding systems differ in accuracy, so we use priors with different accuracy gains to cover this range; the gains we list describe the settings. Teacher forcing is the extreme case, where the history is entirely correct, and it shows that the reversal still exists. It supports the generality of our finding. The paper's strongest results are therefore the reversal, which appears in every setting we test, with or without teacher forcing, and its explanation.

Q1 (separate estimator versus learned fusion or distillation)

Accuracy and confidence are different targets. Accuracy is read from the peak of the output distribution; confidence is read from the distribution itself.

Current brain-to-text decoders learn to combine neural and contextual scores directly (Willett et al., 2023; d'Ascoli et al., 2025; Lévy et al., 2026): they optimise accuracy, which does not guarantee optimised confidence. This is why confidence estimation or calibration is needed.

Our estimator is itself a learned combination of the neural and contextual scores, trained for correctness: its inputs are the local, prior and fused scores, and the MLP learns how to combine them (W2).

Learned fusion: we study shallow fusion because it is currently the most widely used form, including in the most recent decoders (Lévy et al., 2026; Jayalath & Parker Jones, 2026), so the findings apply broadly. The reversal is also stable across changes in accuracy: the conclusion holds under priors whose accuracy gains range from 1.25 to 14.74 points (Sections 4.2 and 4.3).

Distillation: there is not yet an established generalised confidence estimator that we could distil from or learn from; to our knowledge, confidence estimation in BCI is still in its early stages.

Summary of revisions. For W1 and W3, we added word-sequence decoding (Section 4.3, Appendix H, Tables 20 and 34). For W2, we added the fused profile and the paired gains from the local and prior scores (Table 2(b), Section 4.4, Appendices J.2 and J.4, Tables 23 and 31–33). We also added selective output by rank group on both datasets (Appendix J.3, Figure 7, Tables 27, 29, 30 and 35).

Thank you again for the time and effort you have given to the review. We hope these responses make the paper's real contribution and value clear. If our responses resolve your concerns, we would be grateful if you would reconsider your rating.

Reference. Jayalath, D., & Parker Jones, O. (2026). Removing Timing Shortcuts Improves Non-Invasive Brain-to-Text. arXiv:2609.40359.

Reviewer_gnuBOpenReview ↗

Authors responses

Thank you to the authors for responding to to my review with further clarifications and answering questions. The rebuttal has clarified some effects of the approach, especially the response to W2 which clearly identifies where the approach of prior+fused and local comes into play. Overall my assessment remains similar to original one since I still think that while the work is interesting, the solution is somewhat specific and would be important to compare it and perhaps enhance with other powerful approaches of context+neural fusion.

Dear Reviewer gnuB,

Thank you for your feedback. We are glad our response clarified W2.

We disagree that the fusion setting studied in this paper is too specific. Combining neural predictions with language-model scores is central to speech neuroprostheses tested with patients (Willett et al., 2023, Nature), including a system used independently at home for nearly two years (Card et al., 2026, Nature Medicine). Lévy et al. (2026, Nature Neuroscience) use weighted fusion for non-invasive decoding of typed sentences. Jayalath and Parker Jones (arXiv, September 2026) also use additive fusion to reintroduce context after identifying a timing shortcut in the joint word-decoding approach of d'Ascoli et al. (2025, Nature Communications). We included and analysed these journal papers in our original submission, and we use the same additive fusion rule as these systems as our baseline.

Our contribution is to show that, under this widely used fusion rule, a larger confidence margin can rank a residual error above a successful repair. Pooled evaluation conceals this reversal. We identify where it occurs, explain how it arises, and show which scores help estimate confidence when it does.

You suggest comparing with "other powerful approaches of context+neural fusion." We believe the same fusion setting as in these papers is a clear, sufficient, and strong baseline. Your review and follow-up do not identify a specific method or explain how the comparison bears on the finding. Could you name the method you have in mind and clarify which conclusion you believe requires that comparison?

Thank you for your consideration.

Best,
The authors

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response to All Reviewers [1/3]

We thank all reviewers for their thoughtful and constructive feedback. We are encouraged that they find that our rank-conditioned analysis reveals "a meaningful failure hidden by pooled evaluation" (yunW) and "a non-obvious and practically relevant failure mode of confidence after contextual fusion" (CVdb), that the paper "identifies and demonstrates an interesting limitation of confidence estimation with contextual priors" (gnuB), and that the analysis is "systematic" (gnuB) and "unusually thorough" (CVdb). The paper asks a scientific question: why does brain-to-text decoding sometimes make confident, fluent errors? It answers with a finding, the confidence-ordering reversal, an explanation of it, and a consequence for design: confidence estimation should keep the prior. Below, we first address four recurring points (R1–R4) and then respond to reviewer-specific comments in each reply. Changes in the revised PDF are marked in blue.

Summary of new results.

R1. Generative decoding generalisation (gnuB: W3; CVdb: W1, Q3): The same ordering reversal occurs when sentences are generated word by word and the LM conditions on the decoder's own output.

R2. A richer fused-score estimator (gnuB: W2; yunW: W1, Q1): A richer fused-only estimator adds less than one percentage point; the gains come from retaining the local and prior scores, and each gain is significant.

R3. Scope of the theory (CVdb: W2, Q1; yunW: W2): The bound follows from additive fusion alone and is a sufficient condition; on stable subsets, the margins themselves move the boundary.

R4. Group-wise selective output (CVdb: Q2; yunW: W2, Q3): The prior features' stronger deep-group AUROC does translate into better selective output, and the benefit of retained information holds at validation-selected operating points.

R1. Generative decoding generalisation (gnuB: W3; CVdb: W1, Q3)

The same ordering reversal occurs in generative decoding, where sentences are generated word by word and the LM prior conditions on the decoder's own output (revised Section 4.3, Appendix H, Table 20).

To validate that our finding also holds in a generative decoding pipeline, we added a word-sequence decoding experiment. Following current brain-to-text decoding pipelines (Willett et al., 2023; d'Ascoli et al., 2025; Lévy et al., 2026), we decode each sentence presentation word by word over a closed vocabulary of the 50 or 250 most frequent words in the training texts (V50, V250). At each step, the local score of a word from the frozen local decoder is added to its Qwen3-4B-Base log probability with weight α, and beam search sums these scores along each hypothesis. The LM conditions on the hypothesis's own decoded history, so each output affects later predictions. The results are summarised below:

Table R1 (Table 20 and Appendix H of the revised paper). Word-sequence decoding on MEG-MASC (beam 5; means over 3 local decoders). Repair-separation AUROC of the fused margin by initial rank at the validation-selected weight, the range of the prior margin over the same groups, and pooled correctness AUROC; "–" marks groups with fewer than 5 repairs in any decoder.

VocabularyRanks 2–34–56–1011–2021–50Prior marginPooled: fused margin → learned estimator
V50 (α = 0.3)0.3140.2600.268––0.538–0.6620.807 → 0.835
V250 (α = 0.5)0.4810.4120.3740.3340.2350.523–0.7890.799 → 0.838

At the weight selected for WER, the prior lowers WER (47.8% → 46.0% with V50, 55.1% → 50.6% with V250), and confidence still looks good overall: pooled correctness AUROC is 0.807 and 0.799 (last column). Stratified by initial rank, the reversal appears: the fused margin is below 0.5 in every populated group (0.235–0.481), while the prior's own margin orders them correctly (0.523–0.789). Raising the fusion weight further corrects the ordering in deeper groups (with V50, from no positive group at α ≤ 0.75 to ranks 21–50 at α ≥ 1.5; with V250, from none at α ≤ 0.5 to ranks 51–100 at α ≥ 2), but only after WER has started to worsen (46.0% → 57.8% and 50.6% → 70.1%): the weight that is best for accuracy is not the one that is best for confidence, as in retrieval.

Overall, Table R1 shows that the reversal does not depend on a fixed candidate pool or on teacher-forced history: with generated hypotheses and contextual feedback, the fused margin misorders repairs and residual errors, and the prior's own scores still separate them. A learned estimator again improves pooled confidence (last column: 0.807 → 0.835 and 0.799 → 0.838). Within the rank groups, retaining the local and prior scores improves over the fused profile (Table 34). We have added this result to Section 4.3 and Appendix H.

AuthorsOpenReview ↗

General Response to All Reviewers [2/3]

R2. A richer fused-score estimator (gnuB: W2; yunW: W1, Q1)

The further gains come from retaining the local and prior scores, not from richer features.

Following yunW's suggestion, we built a fused profile: the 20 largest probabilities of the softmax over fused scores together with the fused entropy-based confidence, 21 features in total. It is similar to learned confidence estimators that read only output scores and their ranks, such as SR-CEM (Jia & Van hamme, 2026), and it uses the same training data, network, and threshold selection as the original estimators.

We then add the two local features and the seven prior features. Emission at the full estimator's matched coverage is summarised below:

Table R2 (Tables 31–33 of the revised paper). Emission (%) at the full estimator's matched coverage (MEG-MASC 92.24%, MOUS 90.30%), with paired increments over the previous row from the fused profile onward (95% participant-bootstrap intervals).

FeaturesDimMEG-MASCIncrementMOUSIncrement
Fused margin156.7–17.4–
Fused scores666.6–21.7–
Fused profile2167.5+0.88 [+0.53, +1.12]22.6+0.88 [+0.66, +1.20]
+ local2373.2+5.70 [+3.91, +8.05]28.7+6.17 [+5.35, +7.10]
+ local + prior3075.0+1.76 [+0.88, +2.89]35.3+6.60 [+5.80, +7.27]

Fifteen further fused-score features add +0.88 points on each dataset; the local features then add +5.70 and +6.17 points, and the prior features +1.76 and +6.60; all of these gains are significant. Where the fused margin reverses, the fused profile does no better than the margin (Table 23): at ranks 21–50, it separates repairs from residual errors with AUROC 0.360, against 0.392 for the fused margin, and adding local and then prior features raises this to 0.618 and 0.774 (on MOUS, 0.401, 0.559, and 0.929, against 0.411 for the fused margin).

Confidence after fusion should therefore keep the local and prior scores, not richer summaries of the fused scores alone. The fused profile is now in Table 2(b) and Section 4.4, and Appendices J.2 and J.4 (Tables 23 and 31–33) give the full comparison on both datasets.

R3. Scope of the theoretical account (CVdb: W2, Q1; yunW: W2)

Our account is based on how fusion works at the level of scores, so it holds regardless of what the candidates are and how they are produced. It gives a sufficient condition. We do not claim that it pins down where every reversal occurs; that also depends on the prior's preferences over the candidates, a complex, prior-specific system that cannot be derived in advance and must be measured (revised Section 4.2, Appendix F.3).

This is why the bound proves the reversal in more cells under history aggregation (16 of 21) than under the word-level LM, whose prior spreads its scores over 709 candidate words; there, using the correct candidate's actual prior advantage instead of the prior's full score range raises the number from 1 to 11 of 29.

How the boundary moves between weights is a separate question, which we test directly on stable subsets (see our reply to yunW, Q2; Tables 11 and 17): the ordering still moves in all 27 cells under history aggregation, and in all 30 cells up to α = 1 under the free-running prior.

AuthorsOpenReview ↗

General Response to All Reviewers [3/3]

R4. Group-wise selective output under global thresholds (CVdb: Q2; yunW: W2, Q3)

In the deep groups, the fused scores alone select no better than at random; the information that makes selective output work is in the prior's scores.

We compare four estimators with one threshold rule for all test windows, using initial rank only to group the results: the full estimator (fused, local, and prior features), decoder scores (fused and local features), the fused profile (fused scores only), and the prior features alone. With the output size fixed at K = 5, we trace coverage against the share of a group's windows that are emitted:

Table R4 (Table 35 of the revised paper). Coverage (%) at K = 5 within each initial-rank group, by the share of the group's windows emitted (MEG-MASC).

GroupEstimator10%20%50%100%
11–20Fused profile43.443.344.243.8
11–20Decoder scores59.954.247.343.8
11–20Full83.072.856.643.8
11–20Prior features93.786.460.743.8
21–50Fused profile12.612.512.312.5
21–50Decoder scores19.816.413.212.5
21–50Full45.634.120.312.5
21–50Prior features63.544.922.612.5

The fused profile stays at the group's base rate: it cannot tell which of these windows succeed. That information is in the prior scores: the prior features alone give the highest coverage at every emission level, above the full estimator by +13.6 [+11.3, +15.4] and +10.8 [+8.7, +12.9] points at 20% emission. The curves are reproduced on MOUS (Figure 7, Table 35).

At validation-selected thresholds, the operating points a deployed system would use, the full estimator has higher coverage than decoder scores and the fused profile in both groups and on both datasets, and every difference is significant (Tables 29 and 30).

The stronger discrimination that the prior scores provide therefore translates into better selective output in the deep groups, where errors are concentrated, and the fused margin reverses. Section 4.4 reports the matched-emission comparison, and Appendix J.3 gives the curves and the group-wise tables (Figure 7; Tables 27, 29, 30, and 35).

References. d'Ascoli, S., et al. (2025). Towards decoding individual words from non-invasive brain recordings. Nature Communications, 16, 10521. Jia, Y., & Van hamme, H. (2026). Leveraging beam search information for confidence estimation in E2E ASR. IEEE Open Journal of Signal Processing, 7. Lévy, J., et al. (2026). Noninvasive decoding of typed sentences from human brain activity. Nature Neuroscience. Willett, F. R., et al. (2023). A high-performance speech neuroprosthesis. Nature, 620, 1031–1036.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.