acceptodds
Under review as a conference paper at ICLR 2027

Strong yet Controllable LLM Abstention without Learning to Abstain

10 views

Abstract

Large language models (LLMs) can reduce hallucination by abstaining from questions they would likely get wrong. Abstention can be implemented outside the model (selective answering) using a confidence threshold that controls the answer rate (coverage), or learned inside it (abstention training), which can perform better but fixes coverage once trained. To compare the two, we match their coverage and decompose risk (error rate) into answering ability and distinguishing ability, *i.e.*, how well the model answers questions and how accurately it abstains. The decomposition shows that abstention training performs better mainly by answering better: with strong confidence estimators, the two have nearly equal distinguishing ability. This motivates **T**rain for **A**ccuracy, then **P**ost-hoc **S**elect (**TAPS**), a modular approach that keeps coverage controllable, needs only standard post-training and an off-the-shelf confidence estimator, and benefits from advances in either component. Across three LLMs and three factual QA benchmarks, TAPS with a probing-based confidence estimator attains 1.5–9.4 pp lower mean risk than three representative abstention training methods at matched coverage. Within TAPS, accuracy training often weakens the model's own confidence signals, yet refitting the estimator largely preserves distinguishing ability. Together, these results establish TAPS as a strong yet controllable baseline and raise the bar for future abstention training: sacrificing control over coverage should yield gains beyond what can be achieved by optimizing answering and abstaining separately. The source code will be made publicly available upon acceptance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Abstention, Selective Prediction, Confidence Estimation, Hallucination Mitigation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_pXijOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to make LLMs say "I don't know." Existing methods either train the model to abstain directly (e.g., TruthRL) or rely on an external confidence score to filter out uncertain answers after generation. Noting the lack of a fair comparison between the two, the authors compare their error rates at a fixed answer rate. They also break down the overall error into the model's base accuracy and its ability to selectively withhold answers. They find that if a good confidence estimator is used, both approaches are equally effective at choosing what to withhold; the main benefit of abstention training actually comes from improved base accuracy. Following this finding, the authors propose TAPS, which trains the model purely for accuracy and leaves the abstention decision to an existing confidence method. This approach performs on par with abstention training and keeps the answer rate flexible.

Critical strengths
  1. The motivation is clear and the definitions are careful. The paper sets up a fair way to compare the two kinds of abstention: it fixes the answer rate and then compares error rates. It also splits the error rate into two parts, how well the model answers and how well it chooses what to withhold. I found this a useful way to think about abstention, and I think other papers in this area could use the same protocol.
Critical weaknesses
  1. The gain over TruthRL is small, and it doesn't hold up across models. TAPS with probing beats TruthRL by 1.49 pp on average, but almost all of that comes from Qwen3-8B (+5.06 pp). On Qwen2.5-7B, TruthRL is actually better on average (−0.70 pp), and on Llama the two are about the same (+0.12 pp). There are no multiple seeds or confidence intervals, so I can't tell how much of this is noise.
  2. TAPS itself is not new as a method, and I see it mainly as supporting evidence for the paper's analysis. But for an analysis paper, the set of compared methods is quite narrow: only three representative methods are included. Several newer methods have built on TruthRL since then, such as TIAR (Pan et al., 2026), and KARL (Gao et al., 2026). I would like to see these methods added to the comparison and the analysis, which would make the conclusions more convincing.
Questions for authors
  1. SE needs 10 extra samples per question, plus entailment clustering. Probing needs labeled data for every model and dataset, and access to hidden states. TruthRL needs only one generation. Could you report the actual inference and training cost?
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1 of 2)

Dear reviewer pXij,

Thank you for your detailed and constructive review. We are grateful that you acknowledge the clear motivation and careful definitions, the fair comparison protocol, and the useful decomposition that other papers in this area could adopt.

W1: Robustness across seeds

(1) Since TAPS retains the coverage control that abstention training lacks, a large margin over these methods is not our aim. Performing on par with the strongest of them, TruthRL, is already enough for TAPS to be the stronger deployment choice, and it does slightly better. Our evaluation is not limited to a single setting: it spans 9 independently trained model–dataset settings and three abstention rewards for TruthRL, so the conclusion does not hinge on any one run.

(2) To check that this is not seed noise, we repeat the full Qwen3-8B pipeline with 3 seeds and find TAPS's performance against TruthRL statistically significant on all three datasets (p≤0.033p \le 0.033, paired t-test). Each seed trains its own TruthRL (abstention reward set to 0) and TAPS checkpoints and refits the probe. TAPS is matched to the coverage of the same-seed TruthRL checkpoint.

DatasetTruthRL Cov. (%)TruthRL (0) Risk (%) ↓\downarrowTAPS (Probing) Risk (%) ↓\downarrowRisk reduction (pp) ↑\uparrow95% Confidence Interval
NQ48.36 ± 1.9136.67 ± 0.5233.54 ± 0.533.13 ± 0.88[0.94, 5.32]
PopQA20.50 ± 0.5032.55 ± 0.9726.16 ± 1.046.39 ± 0.13[6.06, 6.71]
TriviaQA78.24 ± 0.4015.93 ± 0.5914.67 ± 0.401.26 ± 0.41[0.26, 2.27]

W2: Comparison with recent abstention training methods

Our study covers SFT-based (R-Tuning), reward-based (TruthRL under three abstention rewards), and advantage-based (UCPO) abstention training. Following your suggestion, we further consider the two recent methods:

(1) TAPS achieves clearly lower risk than TIAR, and TIAR shows the same pattern as TruthRL in terms of decomposition. For a controlled comparison, we implement TIAR's published objective in our shared verl-based framework with its default method-specific hyperparameters, and train and evaluate it on Qwen3-8B under exactly the same setup as all other methods on all three datasets.

a. Comparison with TAPS. At TIAR's native coverage, TAPS (Probing) achieves 22.32% risk on average, compared with 28.87% for TIAR.

DatasetTIAR Coverage (%)TIAR Risk (%) ↓\downarrowTAPS (Probing) Risk (%) ↓\downarrowRisk reduction (pp) ↑\uparrow
NQ42.3235.3928.996.40
PopQA22.0437.2527.459.80
TriviaQA71.6313.9810.503.47
Average—28.8722.326.56

b. Decomposition. To see where TIAR's performance comes from, we repeat the decomposition analysis in Section 4, comparing TIAR with selective answering using semantic entropy (S.E.) on the starting model at TIAR's native coverage (%):

DatasetCoverageAnswering gain GansG_{ans} ↑\uparrow (TIAR / S.E.)Distinguishing gain GdisG_{dis} ↑\uparrow (TIAR / S.E.)Risk ↓\downarrow (TIAR / S.E.)
NQ42.321.66 / 082.60 / 82.7635.39 / 36.82
PopQA22.04−0.49 / 089.21 / 90.7037.25 / 31.51
TriviaQA71.631.51 / 076.95 / 79.0213.98 / 14.67
Average45.330.89 / 082.92 / 84.1628.87 / 27.67

TIAR nearly matches S.E. in distinguishing gain, so its standing depends on answering gain. On NQ and TriviaQA, its answering gain is positive and its risk is lower. On PopQA, its answering gain is close to zero and cannot offset even a small distinguishing gap, so its risk is higher.

(2) KARL has not released code or checkpoints, so we could not include it within the rebuttal period. As an RL-reward method like TruthRL and UCPO, it can be analyzed with the same protocol once available.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2 of 2)

Q1: Training and inference cost

TAPS (Probing) does not add much cost over TruthRL to train and run. But to change coverage, TAPS only moves a threshold, while TruthRL must retrain for another 64 hours.

We measure costs on Qwen3-8B with 4 A100 80GB GPUs for training and inference (a separate set of 4 GPUs serves the judge), using the hyperparameters in Appendix C.4. The reported times are wall-clock durations recorded during our experiments and should be read as approximate.

(1) Training: one TAPS run costs about as much as one TruthRL run, but TruthRL must retrain for every target coverage. Both methods are trained for 8 epochs (624 steps for NQ and TriviaQA, and 312 steps for PopQA). The probe is fitted once on the trained model's outputs, adding a one-time 0.44 hours on average, including answer generation, labeling and fitting.

Training cost (hours)NQPopQATriviaQAAvg.
TruthRL81.4740.3971.4164.42
TAPS (training)82.6839.8770.8964.48
TAPS (probe fitting)0.500.290.520.44

(2) Inference: TAPS (Probing) is as fast as TruthRL (19.4 vs. 19.6 min), while TAPS(Semantic Entropy) costs hours due to 10 additional samples per question and entailment clustering.

Inference time on the test set (min)NQPopQATriviaQAAvg.
TruthRL9.520.328.919.6
TAPS (Vanilla)9.123.029.520.5
TAPS (Probing)9.221.627.419.4
TAPS (Semantic Entropy)386.91043.51255.3895.2

Overall, TAPS (Probing) adds little cost over TruthRL in a single run, and saves a full retraining every time the target coverage changes.

These results have been added to Appendix K.


Thank you for your constructive suggestions. With seed-robust results, the comparison with TIAR, and the measured costs, we would be grateful if you would consider raising your score. We are happy to provide any additional details.

AuthorsOpenReview ↗

Rebuttal by Authors (Additional Results)

Additional experiments further confirm that our conclusions are robust across seeds on all three models.

In our earlier response, we reported multi-seed results on Qwen3-8B. We have now completed the same multi-seed experiment on the other two models: Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct. In every setting, the direction of the comparison matches the single-run result in our paper (Table 1), suggesting that TAPS consistently achieves comparable or better abstention performance than abstention training, while keeping coverage adjustable. Below are the detailed results.

For each of three seeds, we retrain TruthRL and TAPS, and match TAPS to TruthRL's native coverage to compare risk. We report mean ± std over seeds. Δ = TruthRL risk − TAPS risk (positive favors TAPS).

ModelDatasetCoverageTruthRLTAPS (Probing)Δ ↑\uparrow95% Confidence Interval of Δ
Qwen2.5-7B-InstructNQ49.34 ± 1.2440.65 ± 0.9637.93 ± 0.682.72[0.55, 4.90]
PopQA20.33 ± 0.5729.41 ± 0.3429.01 ± 0.270.40[−0.28, 1.07]
TriviaQA72.98 ± 0.3716.94 ± 0.6317.13 ± 0.60−0.19[−2.99, 2.60]
Qwen3-8BNQ48.36 ± 1.9136.67 ± 0.5233.54 ± 0.533.13[0.94, 5.32]
PopQA20.50 ± 0.5032.55 ± 0.9726.16 ± 1.046.39[6.06, 6.71]
TriviaQA78.24 ± 0.4015.93 ± 0.5914.67 ± 0.401.26[0.26, 2.27]
Llama-3.1-8B-InstructNQ76.78 ± 0.7333.23 ± 0.3934.51 ± 0.59−1.28[−1.87, −0.69]
PopQA43.67 ± 0.9827.86 ± 0.5329.31 ± 1.50−1.44[−4.60, 1.72]
TriviaQA90.46 ± 0.7911.23 ± 0.4610.96 ± 0.260.27[−0.25, 0.80]
Avg.27.1625.911.25[0.93, 1.57]

We would greatly appreciate it if you could let us know whether our responses have addressed your concerns. We are happy to discuss anything further if helpful.

Reviewer_SFzBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper compares abstention training (R-Tuning, TruthRL, UCPO) with selective answering (thresholding a confidence estimator) at matched coverage. It decomposes risk into answering ability and distinguishing ability, and concludes that abstention training’s advantage over strong post-hoc selectors comes mainly from better answers rather than better abstention decisions. Building on this, it proposes TAPS: train the model for accuracy with GRPO, then abstain post hoc using an off-the-shelf confidence estimator. Across three models and three closed-book QA datasets, TAPS with a probing estimator achieves lower mean risk than the abstention-training baselines at matched coverage while keeping coverage adjustable.

Critical strengths

The paper argues convincingly that common single-number abstention metrics (accuracy, hallucination rate, or their difference) implicitly assign a value to abstention and cannot fairly compare policies operating at different coverages. The findings are also insightful that we should difficientiate if the abstention improvement comes from more accurate answers or better abstention decision.

Critical weaknesses
  1. TAPS (Probing) improves on TruthRL by only 1.49 pp on average (17 of 27 comparisons). The gain is not consistent across models. Against TruthRL with its default reward (r_abs = 0), the average difference is about −3.9 pp on Qwen3-8B, −0.2 pp on Qwen2.5-7B, and +1.5 pp on Llama-3.1-8B, where TAPS is worse. Each method is trained once, and no seeds, confidence intervals, or significance tests are reported. Given typical RL seed variance and test risks estimated on a few thousand answered questions, a gap of this size could plausibly arise from experimental variance alone.

  2. A central concern for abstention methods is whether the learned abstention behavior generalizes beyond the training distribution. Here, every model is trained and evaluated on the same dataset, and the probing estimator. Prior work reports that such correctness probes transfer poorly across datasets, so in-distribution evaluation may particularly favor TAPS. A cross-dataset evaluation (train on one benchmark, test on the others, with the probe fit on the source dataset) is needed to show whether TAPS remains a strong baseline in the setting where abstention matters most.

Questions for authors
  1. Can you report results over multiple training seeds for the TAPS vs. TruthRL comparison? Why does TAPS underperform TruthRL on Llama-3.1-8B?
  2. How do TAPS and the abstention-training baselines compare under cross-dataset evaluation?
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1 of 2)

Dear reviewer SFzB,

We sincerely appreciate your thorough review and constructive feedback. We are grateful that you find our argument against single-number abstention metrics convincing and our decomposition insightful.

W1 / Q1: Robustness across seeds and results on Llama-3.1-8B

(1) Our goal is not a large margin over abstention training: TAPS also keeps coverage control, which these methods give up. Matching TruthRL, the strongest of them, is thus sufficient to make TAPS preferable in deployment, and TAPS slightly exceeds it. Our comparison already covers multiple models and datasets: 9 independently trained settings and three abstention rewards for TruthRL, so the result is not tied to one particular run or configuration.

(2) To address seed variance directly, we rerun the full Qwen3-8B pipeline with 3 seeds. On all three datasets, TAPS's performance against TruthRL is statistically significant (p≤0.033p \le 0.033, paired t-test). Every seed uses its own TruthRL (abstention reward set to 0) and TAPS checkpoints with a refitted probe, and TAPS is matched to the coverage of the TruthRL checkpoint from the same seed.

DatasetTruthRL Cov. (%)TruthRL Risk (%) ↓\downarrowTAPS (Probing) Risk (%) ↓\downarrowRisk reduction (pp) ↑\uparrow95% Confidence Interval
NQ48.36 ± 1.9136.67 ± 0.5233.54 ± 0.533.13 ± 0.88[0.94, 5.32]
PopQA20.50 ± 0.5032.55 ± 0.9726.16 ± 1.046.39 ± 0.13[6.06, 6.71]
TriviaQA78.24 ± 0.4015.93 ± 0.5914.67 ± 0.401.26 ± 0.41[0.26, 2.27]

(3) On Llama-3.1-8B, TAPS is on par with TruthRL overall, and our decomposition traces the remaining gap as below. The 1.5 pp gap you note is computed against TruthRL with abstention reward rabs=0r_{abs}=0. Averaged over all three TruthRL reward settings, TAPS has 0.11 pp lower risk. On the same model, it also outperforms R-Tuning and UCPO in all settings.

The gap at rabs=0r_{abs}=0 arises only on NQ and PopQA. To further address your concern, we apply our decomposition to TAPS at TruthRL's native coverage (all results are in %):

DatasetCoverageAnswering gain GansG_{ans} ↑\uparrow (TruthRL / TAPS)Distinguishing gain GdisG_{dis} ↑\uparrow (TruthRL / TAPS)Risk ↓\downarrow (TruthRL / TAPS)
NQ77.486.61 / 6.7578.17 / 72.3733.60 / 35.15
PopQA43.379.12 / 7.0086.89 / 86.3027.92 / 30.81
  • On NQ, TAPS matches TruthRL in answering gain, and the gap comes entirely from distinguishing gain. This is consistent with Figure 4: on Llama NQ, accuracy training causes the largest confidence estimation drop of the probe across all nine settings (−3.7 pp).
  • On PopQA, the gap instead comes mainly from answering gain: here TruthRL attains its largest answering gain among all nine settings (Table 5).
AuthorsOpenReview ↗

Rebuttal by Authors (Part 2 of 2)

W2 / Q2: Cross-dataset evaluation

To address your concern, we conduct cross-dataset experiments on Qwen3-8B and find that, both methods are affected by the distribution shift, while TAPS (Probing) still achieves lower risk than TruthRL in all 6 transfer directions (33.38% vs. 35.12% on average).

For each source dataset, we take the AccRL and TruthRL (0 abstention reward) checkpoints trained on it and evaluate them on the other two datasets. The probe is fitted on source-dataset outputs only, with no refitting or tuning on the target, and TAPS is thresholded to match TruthRL's coverage on the target. The table below reports risk on Qwen3-8B (%, lower is better), together with TruthRL's coverage on the target (%):

Train → TestTruthRL CoverageTruthRL ↓\downarrowTAPS (Probing) ↓\downarrowTAPS (Semantic Entropy) ↓\downarrow
NQ → PopQA24.2839.5836.9843.09
NQ → TriviaQA69.8712.5611.0716.25
PopQA → NQ48.2347.1143.8541.11
PopQA → TriviaQA70.5117.5515.9712.96
TriviaQA → NQ61.9743.9543.0645.62
TriviaQA → PopQA34.8550.0049.3444.74
Average—35.1233.3833.96

Moreover, because TAPS is modular, its selector can be chosen for the deployment setting. TAPS (Semantic Entropy), which needs no fitting at all, also beats TruthRL on average (33.96% vs. 35.12%) and is best in 3 of 6 directions. Under broader shifts, TAPS can switch to such estimators without retraining the answering model, whereas TruthRL's abstention is fixed in its weights.

These results have been added to Appendix J.


We appreciate your expert and rigorous review. Since the new seed and cross-dataset results speak directly to your concerns about variance and generalization, we hope they warrant a reconsideration of your score. We welcome any further feedback.

AuthorsOpenReview ↗

Rebuttal by Authors (Additional Results)

Additional experiments further confirm that our conclusions are robust across seeds on all three models.

In our earlier response, we reported multi-seed results on Qwen3-8B. We have now completed the same multi-seed experiment on the other two models: Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct. In every setting, the direction of the comparison matches the single-run result in our paper (Table 1), suggesting that TAPS consistently achieves comparable or better abstention performance than abstention training, while keeping coverage adjustable. Below are the detailed results.

For each of three seeds, we retrain TruthRL and TAPS, and match TAPS to TruthRL's native coverage to compare risk. We report mean ± std over seeds. Δ = TruthRL risk − TAPS risk (positive favors TAPS).

ModelDatasetCoverageTruthRLTAPS (Probing)Δ ↑\uparrow95% Confidence Interval of Δ
Qwen2.5-7B-InstructNQ49.34 ± 1.2440.65 ± 0.9637.93 ± 0.682.72[0.55, 4.90]
PopQA20.33 ± 0.5729.41 ± 0.3429.01 ± 0.270.40[−0.28, 1.07]
TriviaQA72.98 ± 0.3716.94 ± 0.6317.13 ± 0.60−0.19[−2.99, 2.60]
Qwen3-8BNQ48.36 ± 1.9136.67 ± 0.5233.54 ± 0.533.13[0.94, 5.32]
PopQA20.50 ± 0.5032.55 ± 0.9726.16 ± 1.046.39[6.06, 6.71]
TriviaQA78.24 ± 0.4015.93 ± 0.5914.67 ± 0.401.26[0.26, 2.27]
Llama-3.1-8B-InstructNQ76.78 ± 0.7333.23 ± 0.3934.51 ± 0.59−1.28[−1.87, −0.69]
PopQA43.67 ± 0.9827.86 ± 0.5329.31 ± 1.50−1.44[−4.60, 1.72]
TriviaQA90.46 ± 0.7911.23 ± 0.4610.96 ± 0.260.27[−0.25, 0.80]
Avg.27.1625.911.25[0.93, 1.57]

We would greatly appreciate it if you could let us know whether our responses have addressed your concerns. We are happy to discuss anything further if helpful.

Reviewer_bSDTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

Hallucination can be reduced through abstention. Abstention has two approaches: a) selective answering: generate the output, and then use a thresholded confidence estimator to decide whether to answer; b) abstention training: train the model to learn when to abstain, but here the "threshold" is fixed. The authors compare both methods under matched coverage and find that abstention training primarily improves answering ability (generating fewer incorrect answers), rather than distinguishing ability (knowing when to abstain).

Based on this finding, the authors propose TAPS, a method that first trains a model to optimize answering ability and then uses a separate confidence estimator with flexible thresholds for abstention. The authors show that TAPS outperforms several abstention training methods while retaining control over coverage.

Critical strengths
  1. The authors use the selective prediction evaluation framework for abstention, allowing for a fair comparison of models with different coverage rates. Further, the paper derives an exact decomposition of behavior at a specific coverage rate into answering ability (how often the model answers incorrectly) and distinguishing ability (how well it identifies when to abstain). Through this, the authors show that the core benefit of abstention training primarily comes from fewer incorrect answers rather than better abstention decisions. This is a helpful finding for work in the abstention community.

  2. Across various models and baselines, the authors show that TAPS outperforms several abstention training methods. TAPS also provides modularity (allowing one to choose their preferred training method and confidence estimator), allowing for further improvements to the method over time.

Critical weaknesses
  1. How is TAPS fundamentally different from selective answering? Both seem to involve optimizing a model for the downstream task, followed by adding a calibrated selector.

  2. The authors judge how good the abstention decisions are based on whether the original model would have gotten those questions wrong. But training also changes what the model knows. A question the original model got wrong might now be something the trained model can answer correctly. So abstaining on that question isn't necessarily a good decision anymore. I'm not sure this metric fully captures how well the trained model knows when to abstain.

Questions for authors
  1. Do the trends observed for TruthRL (in terms of answering ability and distinguishing ability) also hold for other abstention training methods?

  2. Since the empirical improvements of TAPS are relatively modest, could the authors repeat a small subset of the experiments across multiple random seeds to confirm whether the improvements are statistically significant?

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1 of 3)

Dear reviewer bSDT,

We sincerely thank you for your thoughtful review. We are grateful that you acknowledge the fair comparison framework, the exact decomposition that yields a helpful finding for the abstention community, and TAPS's modularity.

W1: How TAPS differs from selective answering

TAPS and selective answering share the same two-stage pipeline: generating an answer and then deciding whether to keep it. The difference is which stage is improved.

  • Selective answering typically focuses on improving the second stage, while keeping the answering model fixed.
  • TAPS also improves the first stage, because our analysis identifies it as where selective answering falls behind. We show that standard accuracy training alone turns selective answering's 3.9 pp risk deficit against TruthRL into a 1.5 pp advantage, with the same probing estimator.

This diagnosis, rather than TAPS itself, is our main novelty:

  • We bring the two abstention paradigms onto common ground. Selective answering and abstention training have developed in parallel and are evaluated differently. We compare them at matched coverage and measure how much abstention training actually gains over selective answering.
  • We build a lens for attributing these gains. We decompose risk into answering and distinguishing gains, and find that abstention training's advantage over strong selective answering comes mainly from answering gain.
  • We validate this finding with TAPS, which also serves as a strong, controllable baseline for future abstention methods.
AuthorsOpenReview ↗

Rebuttal by Authors (Part 2 of 3)

W2: Interpreting the decomposition for abstention-trained policies

We agree that abstention training also changes the model's answers, so judging its abstentions by the original model's answers does not fully reflect intrinsic abilities. Below we show that this does not change our main conclusion, explain our choice of reference, and describe our revisions.

(1) Our conclusion that TruthRL's advantage comes mainly from answering gain holds:

  • For selective answering, the PreRL reference is the method's own answer, so the decomposition is strict there.
  • For TruthRL, our conclusion could be weakened only if training had a negative answering gain on withheld questions, i.e., broke more correct would-be answers (the answers it would have given) than it fixed incorrect ones. This is because withholding an answer that training broke would then count as a good abstention, moving credit from answering gain to distinguishing gain. However, on answered questions, answering gain is positive in all nine settings (Table 5). Even if it turned negative on withheld questions, in the worst case where training fixes none of the incorrect would-be answers, overturning our conclusion would require breaking correct would-be answers at an implausible rate (4.5× the rate observed on answered questions).

Details of the analysis above. On withheld questions, let (a) be training breaking a correct would-be answer and (b) training fixing an incorrect one. Since both are observable only on answered questions, we estimate their rates there. Our conclusion flips if distinguishing gain becomes the main source of TruthRL's advantage over selective answering with semantic entropy. The rate of (a) needed for this lies between two bounds:

  • Lower bound: (b) never occurs, the worst case for us.
  • Upper bound: (b) occurs at its estimated rate.
Setting(a) needed to flip our conclusion (Lower bound)(a) needed to flip our conclusion (Upper bound)(a) estimated
Qwen2.5 / NQ29.0>100 (b = 32.0)16.2
Qwen2.5 / PopQA9.6>100 (b = 25.1)5.1
Qwen2.5 / TriviaQA61.4>100 (b = 42.3)4.3
Qwen3 / NQ17.4>100 (b = 23.2)8.6
Qwen3 / PopQA------
Qwen3 / TriviaQA35.7>100 (b = 33.4)3.7
Llama / NQ50.0>100 (b = 36.0)13.1
Llama / PopQA24.7>100 (b = 33.7)5.3
Llama / TriviaQA45.4>100 (b = 44.3)3.8

All numbers are in %. On Qwen3 / PopQA, TruthRL does not outperform selective answering, so there is no advantage to attribute.

  • At the upper bound, flipping would require breaking over 100% of withheld correct answers, which is impossible.
  • Even at the lower bound, it would require 34% on average versus 7.5% estimated (4.5×), which is implausibly large.

We believe this provides strong evidence that TruthRL's edge comes mainly from better answers, even when abstentions are judged strictly by its own would-be answers.

(2) We use PreRL as the reference because would-be answers cannot be reliably observed. Measuring intrinsic distinguishing ability would require knowing whether each would-be answer is correct. However, an abstention-trained policy gives no answer once it abstains, and forcing it to answer often still ends in abstention, consistent with [1].

(3) We have revised the paper (changes in blue):

  • The terms are now answering gain and distinguishing gain to avoid misinterpretation as the model's intrinsic ability.
  • Related claims now describe behavior relative to the initial model. For example, the introduction now refers to withholding answers that "were initially wrong" instead of "would be wrong", and our contribution now states that abstention training "improves both the answers it gives and the questions it declines" instead of "learns when to abstain".

[1] Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning. ICLR 2026 Workshop.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3 of 3)

Q1: Does the decomposition trend hold for other abstention training methods?

For TruthRL, we observed two trends:

(i) its distinguishing gain does not exceed that of strong selective answering

(ii) it has an advantage in answering gain.

Trend (i) holds for both R-Tuning and UCPO, while trend (ii) holds for UCPO but not for R-Tuning.

We apply the same decomposition, comparing each method with selective answering using semantic entropy (S.E.) at its native coverage (averages over 9 settings, %):

MethodDistinguishing gain GdisG_{dis} ↑\uparrow (method / S.E.)Answering gain GansG_{ans} ↑\uparrow (method / S.E.)Risk ↓\downarrow (method / S.E.)
R-Tuning73.24 / 76.97−1.15 / 028.98 / 23.61
UCPO67.42 / 75.525.32 / 031.96 / 24.63
  • Trend (i): both methods have lower distinguishing gain than S.E. in all 18 settings, by 3.7 pp for R-Tuning and 8.1 pp for UCPO.
  • Trend (ii): RL-based UCPO, like TruthRL, has positive answering gain in all 9 settings, while SFT-based R-Tuning's is negative on average. The latter is consistent with prior findings that SFT generalizes worse than RL [2] and that fine-tuning on facts unknown to the model increases hallucination [3].

These results consistently support our motivation. Since abstention training cannot beat strong selective answering in distinguishing gain, its advantage can only come from answering gain. UCPO's answering gain is too small to offset its distinguishing gap, and R-Tuning has none, so both have higher risk than strong selective answering.

[2] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training. ICML 2025.

[3] Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? EMNLP 2024.

Q2: Are the improvements robust across seeds?

(1) TAPS is not meant to beat abstention training by a wide margin, since it also provides the coverage control these methods lack. Being on par with the strongest of them, TruthRL, already makes TAPS the better choice for deployment, and it is in fact slightly ahead. This conclusion is already drawn from a broad set of experiments: 9 independently trained model–dataset settings, with TruthRL trained under three abstention rewards, so it does not depend on any single run or setting.

(2) Following your suggestion, we further repeat the full Qwen3-8B pipeline with 3 seeds. TAPS's performance against TruthRL is statistically significant on all three datasets (p≤0.033p \le 0.033, paired t-test). For each seed, we train separate TruthRL (abstention reward set to 0) and TAPS checkpoints, refit the probe, and match TAPS to the coverage of the same-seed TruthRL checkpoint.

DatasetTruthRL Cov. (%)TruthRL Risk (%) ↓\downarrowTAPS (Probing) Risk (%) ↓\downarrowRisk reduction (pp) ↑\uparrow95% Confidence Interval
NQ48.36 ± 1.9136.67 ± 0.5233.54 ± 0.533.13 ± 0.88[0.94, 5.32]
PopQA20.50 ± 0.5032.55 ± 0.9726.16 ± 1.046.39 ± 0.13[6.06, 6.71]
TriviaQA78.24 ± 0.4015.93 ± 0.5914.67 ± 0.401.26 ± 0.41[0.26, 2.27]

We are grateful for your thoughtful questions. With the clarified relation to selective answering, the decomposition extended to R-Tuning and UCPO, and seed-robust results, we hope you will consider updating your score. Please let us know if anything remains unclear.

Reviewer_XmDSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper compares abstention training and post-hoc selective answering in a common risk–coverage framework. It introduces an exact, baseline-relative decomposition of matched-coverage risk into answering and distinguishing terms, and uses this decomposition to motivate Train for Accuracy, then Post-hoc Select (TAPS). TAPS combines standard accuracy-only RL with an off-the-shelf confidence estimator, preserving threshold-based coverage adjustment.

Across three 7–8B LLMs and three factual QA benchmarks, TAPS with a supervised probe reports lower mean risk than R-Tuning, TruthRL, and UCPO at matched coverage.

Critical strengths
  1. A useful common evaluation framework. The decomposition makes it clear what changes between an initial model, an abstention-trained model, and the post-hoc selection.
  2. The experiments are strictly controlled, with comparison made using the same starting model and training data across methods.
Critical weaknesses
  1. Insufficient variability analysis. Only the last checkpoint is reported, with no training-seed experiment to get confidence intervals. Therefore, the 1.49pp advantage of TAPS is small enough to affect the final conclusion.

  2. The evaluation demonstrates test-set coverage matching more directly than deployable coverage control. Risk@Cov selects exactly the required number of top-confidence test answers, but does not show that a threshold chosen beforehand achieves a desired coverage on unseen data. Consider choose thresholds on validation data for several target coverages and report realized test coverage and risk.

  3. The decomposition is exact, but its interpretation as separating two abilities is too strong. For abstention-trained policies, GdisG_dis is defined using initial PreRL correctness rather than the trained policy’s own candidate-answer correctness on abstained questions. Because training jointly changes answering and abstention, this baseline needs not isolate intrinsic distinguishing ability. So, the claims should be more moderate.

Questions for authors

Most of my concerns and questions are written in the weaknesses. If the authors can address them in the rebuttal, I would consider raising my overall score.

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1 of 2)

Dear reviewer XmDS,

Thank you for your careful review and helpful suggestions. We are encouraged that you appreciate our useful common evaluation framework, exact decomposition, and strictly controlled experiments.

W1: Robustness across seeds

(1) We do not aim to surpass abstention training by a large margin, because TAPS additionally offers coverage control that these methods lack. Matching the strongest of them, TruthRL, is already enough to make TAPS a stronger deployment choice, and TAPS in fact slightly outperforms it. Our experiments already span multiple models and datasets, covering 9 independently trained model–dataset settings and three abstention rewards for TruthRL, so our conclusion does not rest on any single run or setting.

(2) To further rule out seed variance, we repeat the full Qwen3-8B pipeline with 3 seeds. The results show that TAPS's performance against TruthRL is statistically significant on all three datasets (p≤0.033p \le 0.033, paired t-test). Each seed has its own TruthRL (abstention reward set to 0) and TAPS checkpoints with the probe refitted. TAPS is matched to the coverage of the TruthRL checkpoint from the same seed.

DatasetTruthRL Cov. (%)TruthRL Risk (%) ↓\downarrowTAPS (Probing) Risk (%) ↓\downarrowRisk reduction (pp) ↑\uparrow95% Confidence Interval
NQ48.36 ± 1.9136.67 ± 0.5233.54 ± 0.533.13 ± 0.88[0.94, 5.32]
PopQA20.50 ± 0.5032.55 ± 0.9726.16 ± 1.046.39 ± 0.13[6.06, 6.71]
TriviaQA78.24 ± 0.4015.93 ± 0.5914.67 ± 0.401.26 ± 0.41[0.26, 2.27]

(3) We report the last checkpoint by design: all RL methods use the same training budget and are evaluated at their final step, by which training has converged. The small seed variance above further shows that this choice is stable.

W2: Coverage control on unseen data

Following your suggestion to test deployable coverage control, we find that a threshold chosen beforehand on validation data reliably reaches the target test coverage, with a mean deviation of 1.3–2.2 pp. For TAPS (Probing), we choose thresholds on the validation sets for target coverages of 25%, 50%, and 75%, and apply them unchanged to the test sets. The table reports realized test coverage (%) and corresponding risk (%) under each target coverage:

Model / DatasetCov. (Target Cov. 25%)Risk (Target Cov. 25%)Cov. (Target Cov. 50%)Risk (Target Cov. 50%)Cov. (Target Cov. 75%)Risk (Target Cov. 75%)
Qwen2.5 / NQ26.0428.8050.5038.5475.7151.09
Qwen2.5 / PopQA22.6931.1648.0653.0373.3566.70
Qwen2.5 / TriviaQA27.453.7951.078.2276.5419.92
Qwen3 / NQ24.4321.3949.5632.7476.1548.29
Qwen3 / PopQA24.4630.2944.0248.5672.1165.79
Qwen3 / TriviaQA25.551.6451.403.6474.6412.19
Llama / NQ25.4019.3650.9426.0874.7634.32
Llama / PopQA21.1318.7945.4731.9773.7747.53
Llama / TriviaQA25.111.5952.932.9276.866.59
Average24.7017.4249.3327.3074.8839.16
Mean abs. deviation of coverage (pp)1.32—2.19—1.29—

The largest deviations occur on PopQA, whose validation set is half the size of the others (256 vs. 512 questions), explaining this difference. On the other datasets, all deviations stay within 3 pp.

In contrast, abstention-trained models are hard to set to a target chosen in advance: their coverage stays fixed and barely moves even when retrained with different abstention rewards (Section 6.2, Appendix B).

These results have been added to Appendix I.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2 of 2)

W3: Interpreting the decomposition for abstention-trained policies

We agree that, for abstention-trained policies, the decomposition is a PreRL-relative accounting, not a measure of intrinsic abilities, and we have moderated our claims accordingly.

(1) Following your suggestion, we have revised the paper (changes in blue):

  • The terms are now answering gain and distinguishing gain to avoid misinterpretation as the model's intrinsic ability.
  • Related claims now describe behavior relative to the initial model. For example, the introduction now refers to withholding answers that "were initially wrong" instead of "would be wrong", and our contribution now states that abstention training "improves both the answers it gives and the questions it declines" instead of "learns when to abstain".

(2) We use PreRL as the reference because the policy's own would-be answers cannot be reliably observed. Measuring intrinsic distinguishing ability would require knowing whether the policy's own answer to each withheld question is correct, which we call would-be answers. However, an abstention-trained policy gives no answer once it abstains, and forcing it to answer often still ends in abstention, which is consistent with [1].

(3) This has limited impact on our conclusion that TruthRL's advantage comes mainly from answering:

  • For selective answering, the PreRL reference is the method's own answer, so the decomposition is strict there.
  • For TruthRL, our conclusion could be weakened only if training had a negative answering gain on withheld questions, i.e., broke more correct would-be answers than it fixed incorrect ones. This is because withholding an answer that training broke would then count as a good abstention, moving credit from answering gain to distinguishing gain. However, on answered questions, answering gain is observed to be positive in all nine settings (Table 5). Even if it turned negative on withheld questions, in the worst case where training fixes none of the incorrect would-be answers, overturning our conclusion would require breaking correct ones at an implausible rate (4.5× the rate observed on answered questions).

Together, the revised terminology keep our claims within what the decomposition measures, while its main conclusion remains supported.

[1] Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning. ICLR 2026 Workshop.


Thank you again for your careful review. We hope these additions address the concerns you listed and would be grateful if you would reconsider your score. We are happy to discuss any remaining questions.

AuthorsOpenReview ↗

Rebuttal by Authors (Additional Results)

Additional experiments further confirm that our conclusions are robust across seeds on all three models.

In our earlier response, we reported multi-seed results on Qwen3-8B. We have now completed the same multi-seed experiment on the other two models: Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct. In every setting, the direction of the comparison matches the single-run result in our paper (Table 1), suggesting that TAPS consistently achieves comparable or better abstention performance than abstention training, while keeping coverage adjustable. Below are the detailed results.

For each of three seeds, we retrain TruthRL and TAPS, and match TAPS to TruthRL's native coverage to compare risk. We report mean ± std over seeds. Δ = TruthRL risk − TAPS risk (positive favors TAPS).

ModelDatasetCoverageTruthRLTAPS (Probing)Δ ↑\uparrow95% Confidence Interval of Δ
Qwen2.5-7B-InstructNQ49.34 ± 1.2440.65 ± 0.9637.93 ± 0.682.72[0.55, 4.90]
PopQA20.33 ± 0.5729.41 ± 0.3429.01 ± 0.270.40[−0.28, 1.07]
TriviaQA72.98 ± 0.3716.94 ± 0.6317.13 ± 0.60−0.19[−2.99, 2.60]
Qwen3-8BNQ48.36 ± 1.9136.67 ± 0.5233.54 ± 0.533.13[0.94, 5.32]
PopQA20.50 ± 0.5032.55 ± 0.9726.16 ± 1.046.39[6.06, 6.71]
TriviaQA78.24 ± 0.4015.93 ± 0.5914.67 ± 0.401.26[0.26, 2.27]
Llama-3.1-8B-InstructNQ76.78 ± 0.7333.23 ± 0.3934.51 ± 0.59−1.28[−1.87, −0.69]
PopQA43.67 ± 0.9827.86 ± 0.5329.31 ± 1.50−1.44[−4.60, 1.72]
TriviaQA90.46 ± 0.7911.23 ± 0.4610.96 ± 0.260.27[−0.25, 0.80]
Avg.27.1625.911.25[0.93, 1.57]

We would greatly appreciate it if you could let us know whether our responses have addressed your concerns. We are happy to discuss anything further if helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.