- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper targets the gap between standard LLM confidence calibration and how confidence is actually used in real decisions. Real-world deployments usually operate within a band of decision thresholds, but full Brier training wastes most signal on irrelevant thresholds, while clipped Brier cannot tell confidence values apart outside the band. The authors propose a weighted scoring rule that mixes band-focused clipped Brier with a small full-range Brier "reserve" term. They formally prove the score keeps the incentive to answer correctly and is strictly proper. Tested across three training setups (frozen feature reader, separate confidence reporter, full joint RL) on two QA datasets, the method roughly halves calibration error and lowers decision cost within the target threshold band.
Critical strengths
- Clearly motivated problem with direct real-world relevance
- Solid theoretical grounding with clean proofs of key properties
- Full reproducibility details included in the appendix
Critical weaknesses
- It is hard to tell how much of the gain comes from the proposed scoring rule itself. The idea is interesting, but I am not sure the current experiments are enough to show that the proposed scoring rule is the key factor behind the improvements. It would be useful to see how it compares with simpler alternatives, such as standard Brier score training or other threshold-aware objectives.
- The evaluation is quite narrow. All experiments use the same 4B model and only two QA datasets. I wonder if the results would hold for larger models or tasks beyond QA. At the moment, it is difficult to tell how general the method really is.
- The choice of (\eta) seems somewhat arbitrary. The paper uses (\eta=0.1), but does not make it clear why this value is chosen or how much the results depend on it. Does the method still work well with different values of (\eta), or does it require careful tuning for each setting? This matters for anyone trying to use the method in practice.
- The role of the reserve term is not fully clear. The main idea is to combine the clipped Brier score with a full-range Brier term, so that the model focuses on the target threshold band without losing calibration outside it. However, it is not clear how much the reserve term actually helps in practice, beyond ensuring strict propriety theoretically. Could a simpler alternative achieve a similar trade-off between focusing on the target band and maintaining calibration outside it?
Questions for authors
See weaknesses.