- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper is a pure theory work, and studies calibrated confidence in sequential predictors. Specifically, it shows when calibration scores at each timestamp are insufficient for decision making. The authors introduce the notion of ideal confidence, and show that for the final "abstain vs answer" decision, their introduced notion of ideal confidence is enough. In contrast, for the decision to continue, even ideal confidence is insufficient and only works under additional Markov assumptions. The paper concludes with an error bound under this Markov assumption.
Critical strengths
The strengths of the paper are mainly that it is a complete piece of theory:
- The question is timely and relevant.
- The paper is self-contained, introduces necessary notation and provides proofs in the appendix.
- While I did not check the proofs in detail, they appear to be sound.
Critical weaknesses
I have three major concerns:
-
The most obvious one is that there is no empirical validation in the main paper whatsoever. There is no problem with theory-heavy papers that have somewhat limited experimental evidence, but the paper moves all results to the appendix. This makes it, in my opinion, not suited for ICLR.
-
This connects to the missing experiments, and is, in my opinion, even more important: While I do think the formalization and the core findings are sound, and are possibly interesting for a mathematics outlet, there is no practical algorithm or deployable method. The paper is merely characterizing the problem (sometimes confidence is sufficient, sometimes it is not), but gives no guidance on i) how to estimate the ideal confidence, or ii) what to do in practice for the non-markovian case.
-
To be fair with the authors, I also checked the appendix for the (limited) experiments they attached. The real world dataset is basically a negative result of their theory. I appreciate the honesty of the authors that they attached this result, but there are basically no improvements in terms of decision risk.
So in summary, while the theory looks good, I have serious doubts that the work in its current stage has the potential for ICLR; sufficient experimental validation is missing, practical tools are missing, and the limited experiments in the appendix are more or less a negative report.
Questions for authors
-
How would you estimate ideal confidence in practice?
-
How do you deal with the non-markovian case?
-
Do you have explanations why your experiments do not show any serious improvements?
2 comments
Thank you for your rebuttal. Unfortunately, I do not think the additions address my main concerns.
Your revised Section 3.5 outlines finite-class empirical risk minimization and backward regression. They are standard procedures and not a new algorithmic contribution. The theoretical guarantees either involve an uncontrolled policy-class approximation term or depend on population-level estimation errors that are not directly observable / controlled. Finally, the question of how to obtain a sufficiently informative representation + reliable stopping decisions in realistic non-Markovian settings is still unclear.
The experiments are also insufficient. Figure 1b considers only four history-based versus two score-only policies in a constructed two-step example. Convergence to the corresponding oracle risks maybe confirms the theoretical construction but does not show practical usefulness.
More importantly, there is still no convincing real-world validation of improved stopping decisions. The real-data analysis is a calibration diagnostic, and not an evaluation of decision risk, and the authors themselves acknowledge that the observed increase in ECE is a mechanical consequence of bin refinement.
I appreciate your efforts to improve the paper. However, the fundamental limitations remain completely unchanged: the contribution is a theoretical characterization of information sufficiency, without a compelling new learning methodology or any convincing empirical evidence that the findings lead to improvements in practical applications. I therefore maintain my original concerns and rejection recommendation.
Thank you for your comments. The revision makes the theoretical objective explicit: identify the information needed for stopping decisions and quantify the cost of approximating it. We have added controlled numerical illustrations and concrete learning procedures to connect those results to policy construction.
How would we estimate ideal confidence? The target is the conditional mean of the observed correctness indicator, $r_t(h_t)=\mathbb{E}
1Y=Y^t∣Ht=ht$. Section 3.5 now describes fitting this quantity with a proper scoring rule on independent complete labeled trajectories from a frozen predictor. Splits are by trajectory, not by exit, and fitted targets and final evaluation data are separated. A score-only recalibrator estimates a conditional mean given the score; history features estimate a mean given those features. Neither automatically recovers the conditional probability given all accessible history.
The same section gives a finite-class alternative that learns stopping and terminal decisions directly from realized trajectory costs, without first estimating ideal confidence. Proposition 8 supplies its oracle excess-risk bound. We also give a pre-specified conditional-residual test with a finite-sample null bound, explain what rejection means, and explicitly exclude interpreting nonrejection as a sufficiency certificate.
What should be done when confidence is non-Markov? Retain information needed to predict continuation: prior scores, candidate identity and changes, margins, observed signal quality, and time remaining are possible features. Another option is to regress continuation directly on the observed prefix. We now state the backward recursion and its training protocol. Proposition 16 in Appendix A.22 bounds the executed policy's risk through probability error and continuation-prediction error without requiring a scalar Markov state. Section 3.5 includes its main inequality.
An example in Section 3.3 makes the remedy concrete. Under an asymmetric binary observation channel, two histories have MAP confidence 0.6 but different candidate labels. Their continuation errors are 0.22 and 0.28. At step cost 0.15, retaining the candidate identity restores the optimal decision; confidence alone loses 0.015 in total risk. This is an example-specific sufficient augmentation, not a guarantee that any larger feature vector is sufficient.
Purpose of the numerical examples. The paper's contribution is primarily theoretical. Its central comparisons concern the optimal risks attainable from different information sets under the same observation law, costs, and action space. Section 3.6 uses controlled constructions to make those comparisons explicit. Table 2 lists every policy in the finite-class example, including both fixed-time rules, so the information gap can be checked directly rather than inferred from a calibration metric.
The theory also explains why different continuation laws need not change the decision at a given cost. Proposition 2 identifies the relevant condition: histories sharing a score must favor opposing actions. A claim of general practical superiority would require an application-specific model, estimator, and evaluation beyond that information comparison. We have kept the paper's claims within its theoretical scope. The supplementary calibration plot is descriptive: refining score bins increases the empirical absolute-residual sum mechanically, so that comparison alone is not a test of history dependence or a policy-risk guarantee.
Main-text evidence and theorem scope. Figure 1 now includes exact construction checks and a finite-data learning experiment where history-based policies approach risk 0.275 while score-only policies approach 0.325. The comparison isolates the contribution of accessible history within a fully specified model. Finally, Theorem 8 does not assume Markov confidence: its true and surrogate optimizations use the same full filtration. The approximate-kernel result is separate. We have made that distinction explicit next to both statements.
We hope we addressed your comments in a satisfactory manner.