acceptodds
Withdrawn from ICLR 2027

Factual Fine-Tuning Separates Abstention Frequency from Score Selectivity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selecting fine-tuning examples that a language model already answers correctly can reduce its exposure to unfamiliar facts, yet it remains unclear which aspects of reliable answering this selection preserves. Observed abstention, ranking of unreliable answers, and separation of initially known and unknown questions can change differently, making refusal rate an incomplete evaluation of factual adaptation. We examine this distinction in closed-book PopQA and TriviaQA using three instruction-tuned model families and 72 training runs across three adapter ranks. Repeated initial-model answers define historical known and unknown panels; known-fact, unknown-fact, and high-loss known-fact training share an answer-only objective, while held-out question trajectories distinguish lost correct answers from abstentions that become errors. Known-fact training converts many prior abstentions into incorrect answers, and unknown-fact training introduces additional costs whose origins depend on the model's initial behavior. Relative to known-fact training, it adds 14.53 and 12.17 percentage points of correct-to-error transitions in Llama and Qwen, averaged across datasets, prompts, and seeds, whereas OLMo's lower final accuracy primarily reflects a smaller improvement from a highly abstaining starting point. Qwen nearly stops abstaining under both arms, but its abstention score distinguishes them: the area under the receiver operating characteristic curve (AUROC) for current answered-response correctness after known/unknown training is 0.873/0.426 on PopQA under one prompt and 0.880/0.596 under another. This distinction extends beyond historical labels, although ranking alone does not establish a calibrated abstention policy. At rank 16, hidden-state classifiers kept fixed or refitted after training show that historical-label transfer depends on the elicitation prompt, limiting conclusions about internal knowledge from a single probe. These results identify what an abstention-frequency comparison misses: the origins of new errors and the ranking quality of an abstention score. Separate evaluation shows data curators both the benefits and the remaining failures of known-fact selection.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
factual fine-tuning, knowledge-aware abstention, epistemic uncertainty, knowledge-state representations, hallucination, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EQtoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether selecting fine-tuning examples that the model already answers correctly preserves reliable abstention. Its premise is that observed abstention, the ranking of unreliable answers, and the separation of known and unknown questions can change in different ways. Refusal rate is therefore an incomplete evaluation of factual adaptation.

The study uses closed-book PopQA and TriviaQA, three instruction-tuned model families, and 72 training runs across three adapter ranks. Repeated initial-model answers define historical known and unknown question panels. The known-fact, unknown-fact, and high-loss known-fact arms share an answer-only objective. Held-out question trajectories distinguish lost correct answers from abstentions that become errors.

Known-fact training converts many prior abstentions into incorrect answers: 57.72% of the unknown panel in Llama and 27.69% in Qwen. Unknown-fact training adds 14.53 and 12.17 points of correct-to-error transitions over known-fact training in Llama and Qwen, respectively. OLMo's lower final accuracy mainly reflects a smaller improvement from a highly abstaining start. Qwen nearly stops abstaining under both arms (0.44% and 0.03%). Yet its abstention score separates current answer correctness with AUROC values of 0.873 and 0.426 on PopQA under one prompt, and 0.880 and 0.596 under another. At rank 16, fixed and refit hidden-state classifiers show that historical-label transfer depends on the elicitation prompt.

The paper is diagnostic. It offers no new training objective or validated mitigation. It also states that its known/unknown contrast does not isolate unknownness from other properties of the selected facts.

Critical strengths
  1. The transition breakdown separates where new errors come from, which final accuracy cannot do.Final accuracy cannot distinguish losing a correct answer from switching from abstention to a wrong guess. For each question, the paper compares the base model response with the checkpoint response. It then counts transitions such as abstain-to-wrong and right-to-wrong. It uses the whole panel as the denominator, so the counts sum to the total change in errors. The authors verify this sum on all 768 rank-16 panels. The method only needs saved responses, so it is cheap to reuse.

  2. The paper attributes an accuracy gap to the model's starting behavior, not to the size of the drop.The same accuracy drop can mean very different things. In Llama and Qwen, it reflects real damage: previously correct answers turn wrong. This right-to-wrong movement is 14.53 and 12.17 points higher than in K-SFT. OLMo's base model abstains on 92.58% of its own K₀ questions. These questions had passed the correct-answer criterion. Thus OLMo's lower U-SFT accuracy mostly means it started low and climbed less. It has only 0.58 points of extra right-to-wrong movement. Reading a gap in this way generalizes beyond these three models.

Critical weaknesses

The paper identifies several metrics behind abstention frequency but does not say how to combine them or what they imply for subsequent training.

Questions for authors

Q1. If a practitioner has computed all three metrics and they conflict, as with the probe at 0.896 and s_IDK at 0.437 on the same checkpoint, which should take precedence when deciding whether to add explicit abstention training? Should this depend on model or dataset, and if so, how should that dependence be handled in practice?

Q2. Concretely, how should these three metrics be used to improve a subsequent training run? Should they guide data selection, determine when to stop, or decide between filtering alone and filtering plus abstention supervision? A worked example on one of the three model families would clarify the intended use.

Reviewer_zPJzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Past work has studied how training models on facts that they already know can reduce hallucinations compared to training on a full dataset of knowledge that may also contain new information. Inspired by this, the authors investigate how this form of knowledge-aware finetuning affects models' pre-existing abstention behavior, particularly comparing training on knowledge models already know versus questions to which they cannot produce correct answers. The authors find that when they train models to exclusively produce answers without abstention, the models' abstention rates decrease.

Critical strengths
  • The authors are very precise with experimental details.
  • Comparing how abstention behavior is affected based on finetuning on known/unknown facts is an interesting framing.
Critical weaknesses

There are two crucial issues with the paper, which together explain why I have given the scores I have given.

The first problem is the paper's motivation. The paper presents a false dilemma between performing knowledge-aware finetuning and models losing their abstention capabilities. This is not a real problem. There exist methods for finetuning models on new tasks while also training them to abstain. Most relevant to this particular paper are approaches such as "Alignment for Honesty" [1], which similarly presume prior knowledge of facts that models know and do not know (measured by assessing answer correctness over multiple samples) and construct datasets where models provide answers for facts they know and abstain for those they do not. Alignment for Honesty uses SFT, while alternative approaches such as that of Cheng et al [2] use DPO. As a result, I struggle to see how the paper's contribution is informative.

The second, and key, problem is the paper's writing. With all due respect to the authors, I fear this paper does not pass the bar for scientific communication. As some examples:

  • From the abstract, consider the sentence starting with "Repeated initial-model answers." Telegraphic compound phrases inventing new terminology such as "high-loss known-fact training," without any proper definition make the paper very difficult to parse on an initial pass. Other sentences in the abstract such as "unknown-fact training introduces additional costs whose origins depend on the model’s initial behavior" provide no understandable information.
  • The introduction is guilty of the same issues. The second paragraph, starting from the sentence "A hidden state probe..." is incomprehensible to me, despite having published papers on abstention.
  • There are some more egregious issues in Section 3.2. The paper, for instance, introduces terms such as CAL-FIT without proper definition. While some terms are not properly defined, there are excessive levels of detail on other parts, such as explicit specification of random seeds in the main text of the paper.
  • From what I can tell, the authors are even referring to the training code itself. Section 3.4 contains a passing reference to a "core-v2 pipeline" without any explanation of what this even is.
  • I have tended to not be very explicit about this in other reviews in ICLR this year, but all of these are hallmarks of LLM writing, which is particularly egregious in the case of this paper. Independent of whether the results or motivation are informative or sound, the paper's writing is very poor, does not meet the standards of a conference like ICLR and, in my opinion, should be rejected purely on that basis. For the paper to improved, it would need to be rewritten completely.

References:

  • Yang, Y., Chern, E., Qiu, X., Neubig, G., & Liu, P. (2024). Alignment for honesty. Advances in Neural Information Processing Systems, 37, 63565-63598.
  • Cheng, Q., Sun, T., Liu, X., Zhang, W., Yin, Z., Li, S., Li, L., He, Z., Chen, K., and Qiu, X. Can AI assistants know what they don’t know? In Forty-first International Conference on Machine Learning, 2024
Questions for authors

What does core-v2 represent? What do CAL-FIT and CAL-SELECT mean?

1 comment

I am maintaining my scores due to the lack of response from the authors.

Reviewer_kwcFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how SFT on known and unknown factual questions affects large language models’ factual reliability, abstention behavior, and uncertainty related score selectivity. The main contribution includes a systematic empirical analysis demonstrating that abstention frequency alone is insufficient to characterize factual reliability. The authors also examine hidden state probes and show that their ability to distinguish historically known and unknown questions depends on the evaluation prompt and whether probes are refitted after fine tuning.

Critical strengths
  • The paper is well motivated and clearly identifies the limitations of using abstention frequency alone to evaluate the factual reliability of large language models.

  • The paper provides a comprehensive empirical evaluation.

  • The proposed response transition analysis provides valuable insights into how fine tuning on known and unknown facts affects previously correct answers, abstention behavior, and newly introduced errors.

Critical weaknesses
  • In Section 3.3 and Table 2, the experiments only compare the base models, such as K-SFT, U-SFT, and HardK-SFT. Although Section 2 discusses representive approaches, such as R-Tuning, US-Tuning, SEAT and Seal, none of these methods is included in the experiments.

  • Table 7 reports substantial differences between K-SFT and U-SFT training sets. HardK-SFT uses only 600 training examples and 114 optimization steps, compared with 1000 examples and 189 steps for K-SFT and U-SFT. The observed performance differences cannot be attributed exclusively to whether the training facts are previously known or unknown.

  • The experimental evaluation relies heavily on LLM based correctness judgments without independent human validation.

Questions for authors
  • Could the authors include comparisons with R-Tuning, US-Tuning, SEAT, and Seal to better contextualize the findings and demonstrate how these training strategies compare with established abstention preserving methods?

  • Table 7 reports substantial differences in training data characteristics between K-SFT and U-SFT, and HardK-SFT uses fewer training examples and optimization steps. How can the authors ensure that the observed performance differences are primarily attributable to factual unknownness but not differences in example difficulty, data distributions, or optimization exposure? Could additional experiments with more closely matched training sets be provided?

  • The authors rely on GPT-5.6 for semantic correctness judgments but do not report independent human validation. Could the authors conduct a human evaluation on a representative subset of generated responses and report agreement statistics between human annotators and the LLM judge?

1 comment

I will maintain my rate, as there is no response from the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.