- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how adapting confidence baselines affects the attribution of performance gains in selective question answering. The authors introduce a fixed-answer evaluation protocol that compares different confidence scoring methods under identical answer candidates and coverage levels, separating baseline adaptation from the benefits of additional comparison features. Experiments show that refitting native confidence improves reference EM by approximately 3.5 percentage points, outperforming the proposed fusion approach at 50% coverage. Additional controls and evaluations further examine the effects of training data, ranking changes, and comparison features. The findings highlight the importance of properly adapted baselines for fair performance attribution, while the independent benefits of comparison features remain uncertain.
Critical strengths
-
The paper addresses an important evaluation issue in selective QA, showing how comparisons against outdated confidence baselines may overestimate the benefits of more complex scoring methods.
-
The fixed-answer evaluation protocol effectively isolates confidence ranking from answer generation and candidate selection. The matched-count, matched-target, and intercept-based controls further strengthen the experimental design.
Critical weaknesses
-
Confidence calibration, domain adaptation, and selective prediction have been extensively studied. The main contribution lies in re-evaluating existing methods under stronger controls, with limited methodological novelty or theoretical advances.
-
The practical efficiency of the proposed scoring approaches remains insufficiently evaluated. Inference latency and computational overhead are not systematically compared, making it difficult to assess their cost-effectiveness.
Questions for authors
-
The discussion of related work is limited. The paper lacks a systematic comparison with existing studies on confidence calibration, selective QA, and evidence-based scoring, making it difficult to clearly assess its novelty and contributions beyond prior work.
-
Given the identified inconsistencies between reference EM and semantic correctness, can the authors further assess whether the main adaptation gains remain robust to potential reference-label errors, using the existing case analyses or sensitivity results?
-
The evaluation sets exclude exact normalized-question duplicates but may still contain semantic duplicates or overlapping retrieved documents. Can the authors provide further evidence or clarification regarding these potential dependencies and their impact on the reported results?