acceptodds
Withdrawn from ICLR 2027

Baseline Adaptation Changes Attribution in Fixed-Answer Selective QA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study how adapting a confidence baseline changes the attribution of gains in selective question answering. Each question's original answer is fixed, so scorers change only which answers are retained. At half coverage, native confidence refitting improves reference EM by about 3.5 percentage points in two cohorts with no overlapping normalized questions. The first is a frozen seven-scorer comparison; the second is a separately reported native-only evaluation with policies frozen before generation, with comparison-feature evaluation unfinished. Fusion loses its apparent advantage over historical confidence when compared with the adapted practical baseline, while its increment over a joint-target-matched control remains uncertain. Matched-count controls and intercept interventions separate sample count, score offsets and ranking changes. Direct comparison features show exploratory signals that depend on coverage. Native confidence already contains contextual information; the question is the value of additional comparison information. Decision accounting and case review connect measured gains to retained answers while distinguishing reference matching from factual correctness.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
selective question answering, confidence calibration, retrieval-augmented generation, baseline adaptation, uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3vHXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how adapting confidence baselines affects the attribution of performance gains in selective question answering. The authors introduce a fixed-answer evaluation protocol that compares different confidence scoring methods under identical answer candidates and coverage levels, separating baseline adaptation from the benefits of additional comparison features. Experiments show that refitting native confidence improves reference EM by approximately 3.5 percentage points, outperforming the proposed fusion approach at 50% coverage. Additional controls and evaluations further examine the effects of training data, ranking changes, and comparison features. The findings highlight the importance of properly adapted baselines for fair performance attribution, while the independent benefits of comparison features remain uncertain.

Critical strengths
  1. The paper addresses an important evaluation issue in selective QA, showing how comparisons against outdated confidence baselines may overestimate the benefits of more complex scoring methods.

  2. The fixed-answer evaluation protocol effectively isolates confidence ranking from answer generation and candidate selection. The matched-count, matched-target, and intercept-based controls further strengthen the experimental design.

Critical weaknesses
  1. Confidence calibration, domain adaptation, and selective prediction have been extensively studied. The main contribution lies in re-evaluating existing methods under stronger controls, with limited methodological novelty or theoretical advances.

  2. The practical efficiency of the proposed scoring approaches remains insufficiently evaluated. Inference latency and computational overhead are not systematically compared, making it difficult to assess their cost-effectiveness.

Questions for authors
  1. The discussion of related work is limited. The paper lacks a systematic comparison with existing studies on confidence calibration, selective QA, and evidence-based scoring, making it difficult to clearly assess its novelty and contributions beyond prior work.

  2. Given the identified inconsistencies between reference EM and semantic correctness, can the authors further assess whether the main adaptation gains remain robust to potential reference-label errors, using the existing case analyses or sensitivity results?

  3. The evaluation sets exclude exact normalized-question duplicates but may still contain semantic duplicates or overlapping retrieved documents. Can the authors provide further evidence or clarification regarding these potential dependencies and their impact on the reported results?

Reviewer_vyCwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how adapting a confidence baseline affects the attribution of performance gains in selective question answering. The authors propose a fixed-answer evaluation protocol in which different scoring methods rank the same generated answers at equal coverage, isolating answer selection from generation. Experiments on NQ-open show that refitting native confidence improves retained reference EM by 3.55 percentage points, outperforming the more complex Fusion method. However, Fusion's incremental benefit over a target-matched native baseline remains statistically inconclusive. A second cohort further supports the value of native confidence refitting, although its comparison-feature evaluation remains incomplete. Overall, the paper highlights the importance of strong adapted baselines when assessing additional scoring information.

Critical strengths

1. Well-controlled experimental design. The fixed-answer protocol holds generated answers and coverage constant, allowing performance differences to be attributed to answer ranking rather than generation or answer replacement. The matched-count and target-matched controls also help distinguish baseline adaptation from additional comparison information.

2. Careful statistical analysis and transparent reporting. The authors account for uncertainty in both fitting the scoring models and evaluating their performance, distinguish exploratory analyses from frozen evaluations, and explicitly acknowledge inconclusive or unfinished experiments. This strengthens the credibility of the reported findings.

Critical weaknesses

1. Limited novelty and research significance. Prior work has already established the importance of confidence calibration and adaptation in QA [1], while recent studies have further explored how calibration methods, contextual information, and evaluation protocols affect selective QA performance [2,3]. Although this paper offers a careful controlled analysis of how baseline adaptation changes the apparent benefits of additional comparison features, its incremental contribution beyond these existing findings remains insufficiently established, particularly given its limited validation across generators and datasets.

References:

[1] Kamath, A., Jia, R., & Liang, P. (2020). Selective Question Answering under Domain Shift. ACL2020

[2] Joren, H., Zhang, J., Ferng, C.-S., Juan, D.-C., Taly, A., & Rashtchian, C. (2025). Sufficient Context: A New Lens on Retrieval Augmented Generation Systems. ICLR 2025.

[3] Xia, Y., Luz De Araujo, P. H., Zaporojets, K., & Roth, B. (2025). Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles. ACL2025

2. Incomplete validation and limited generalizability. The central fixed-answer experiment is limited to one generator and data source, while the additional cohort only confirms the benefits of native-confidence refitting, leaving the comparison-feature evaluation unfinished. Furthermore, Fusion's incremental benefit over the target-matched joint-native baseline remains statistically inconclusive (+0.65 pp, 98.33% CI [−0.82, 3.47]). Consequently, the paper does not yet provide sufficient evidence that comparison features improve performance beyond a well-adapted native baseline, or that the observed findings extend to other generators and datasets.

Questions for authors

1. Generalizability of the findings. Do the main conclusions hold across different generators and QA datasets? Since the completed fixed-answer evaluation primarily uses Qwen3.5-9B on NQ-open, broader validation would help establish whether the observed effects reflect a general phenomenon.

Reviewer_uma1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper seems to propose an study on changing the baseline for selective answering and the impact of doing this.

The paper is hard to read let alone understand. I suggest the authors re-write it and resubmit, in this way it will be possible for reviewers to give any comments on it (goals, motivation, approach, etc.).

Critical strengths

N/A

Critical weaknesses

N/A

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.