- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies conformal prediction for LLM question answering where the correct answer may remain unseen. The proposed BAYES-STATE method combines a Pitman–Yor sampling reference with a shared truth-tilting correction learned from labeled historical questions. The proposed method constructs scores over observed answer clusters and an everything-else category, followed by split conformal calibration. The authors establish finite-sample marginal coverage under exchangeability and bound excess mean squared conditional coverage error in terms of posterior approximation error under a margin condition. Experiments on short-answer QA benchmarks evaluate score quality and conformal performance under matched supervision, while a separate cold-start study examines uncertainty in future sampling counts.
Critical strengths
-
The paper considers several practical aspects of LLM question answering, including prompt-specific answer spaces, frequent but incorrect responses, and the possibility that the correct answer is absent from the sampled candidates. These considerations inform the local semantic alphabet, the everything-else category, and the learned truth correction. The resulting formulation captures difficulties that arise in the LLM question answering problem.
-
The experiments cover two QA benchmarks, two LLMs, and different query budgets. Comparisons under matched supervision and component ablations help assess the contributions of truth learning and the sampling reference.
Critical weaknesses
- My main concern is whether the reported improvements lead to more useful answer sets. At the primary coverage levels, all observed answer clusters are retained, and the full model produces the same sets as the gate-only version. The lower groupwise coverage error is also accompanied by more fallback use and larger displayed sets. Although these tradeoffs are acknowledged, the results leave unclear how much the full truth-tilting model improves answer selection beyond learning when to abstain.
Questions for authors
-
Could the authors clarify the finite-sample coverage guarantee given the complex score construction and prompt-specific semantic clustering? A brief explanation of which components are fixed after fitting and why the calibration and test scores remain exchangeable would help readers understand why the standard conformal result applies.
-
At the primary coverage levels, the full model and the gate-only version return the same prediction sets. When learned ranking changes the sets at lower coverage levels, does it reduce the number of displayed answers or improve coverage at similar fallback rates?
-
GT+Tilt and Direct-State perform comparably to BAYES-STATE under matched supervision. In what QA settings would the PYP reference improve the final answer sets? It would be helpful to connect its cold-start sampling benefits to answer coverage or fallback behavior.
Thank you for the detailed response. I'll keep my positive score.