- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates language models that produce multiple candidate answers per query. It leverages conformal methods to bound the risk of returning an answer set with no valid acceptable answers within a pre-specified budget, while minimizing the set size.
Critical strengths
The paper’s core contribution links set-size minimization in risk-controlled generation to selective prediction and Chow’s optimal rejection rule, yielding a concise regime law. The ACF-LV method achieves notable APSS reduction under a unified protocol, where the majority of improvements come from the straightforward and sensible single-stage calibration design.
Critical weaknesses
The theoretical assumptions are relatively strong, and ablation results demonstrate that the overall improvement mainly stems from single-stage calibration. In addition, several descriptions in the manuscript are unclear, and the writing, mathematical notations and tables require careful revision and verification.
Questions for authors
Beyond the aforementioned weaknesses, I wonder why ACF-LTT increases APSS on ScienceQA with Qwen2.5-7B/14B in the reported results.