- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how RAG systems can choose between PK-only and CK-conditioned answers, or abstain when neither is reliable. It constructs a model-specific benchmark using the correctness of both answer paths. SABER combines question-only hidden states with representations from multiple reasoning traces and self-evaluation passes, and trains two lightweight correctness predictors while keeping the backbone frozen. The predictors select an existing candidate answer or optionally trigger abstention. Experiments on four 3B–8B models report improvements in conflict handling and selective answering over the evaluated inference-time baselines.
Critical strengths
-
The model-specific benchmark distinguishes PK-only and CK-conditioned answer correctness and explicitly includes cases where both answers are wrong. This enables a more informative evaluation of answer selection and abstention than overall accuracy alone.
-
SABER requires training only small prediction heads while keeping the backbone frozen. Across the four evaluated backbones, the results show improved conflict handling and more effective abstention than other baselines.
Critical weaknesses
- The self-evaluation prompt explicitly relies on the model's own knowledge. Could this lead the model to reject factually correct contextual information beyond its PK? How does the CK evaluation distinguish between the required knowledge being absent from the context and the model being misled by its own PK?
- The contribution beyond supervised correctness probing remains unclear. How does SABER compare with a simple classifier trained on the same labels using candidate-answer hidden states, without additional reasoning or self-evaluation? Besides, how does performance change when self-evaluation or either the PK or CK feature branch is removed?
- The evaluation is restricted to small (3B-8B) models, with no larger open-weight models, limiting the breadth of the empirical findings and making the framework's broader conclusions less convincing.
- The statement in lines 403–406 that reduced variants outperform or match SABER in both metrics is inconsistent with Figure 4: on Qwen-2.5-3B, C3 has slightly higher Acc (56.2% vs. 56.1%).
Questions for authors
Please refer to Weaknesses.
6 comments
Rebuttal by Authors (2/5)
W1 (contined). Whether the own-knowledge self-evaluation prompt leads SABER to reject correct context beyond its PK, and how SABER distinguishes knowledge missing from the context from being misled by its own PK
To test whether SABER rejects correct context beyond its PK, as you ask, we compare it with the two baselines in our paper that rely on the model's verbal self-judgment. Context-Eval asks the model whether the retrieved passage is reliable before answering, which is the context-side evaluation you describe. Internal-Eval asks the model whether its own PK answer is correct and chooses between this answer and the context accordingly.
Results. We report CF, the accuracy on the instances, averaged over the five data sources in Tables 1 and 2. The results are shown below.
| Backbone | Context-Eval (%) ↑ | Internal-Eval (%) ↑ | SABER (%) ↑ |
|---|---|---|---|
| Llama-3.1-8B | 49.5 | 57.5 | 85.3 |
| Qwen-2.5-7B | 69.4 | 75.3 | 93.3 |
| Llama-3.2-3B | 65.8 | 67.0 | 91.7 |
| Qwen-2.5-3B | 26.8 | 67.5 | 94.4 |
From the table, we observe that SABER's CF is 18.0–67.6% higher than that of both verbal baselines on every backbone. This is because a verbal judgment that consults the model's own knowledge tends to reject a context that contradicts this knowledge even when the context is correct. In contrast, SABER's CK belief is supervised by the actual correctness of the CK answer. Therefore, the risk you raise affects verbal self-evaluation, and SABER avoids it.
To test whether SABER distinguishes the three situations, as you ask, we examine SABER's decisions in each cell on the test set, under the abstention setting of Table 3 (). In 94.1–97.6% of the instances, the context contains the gold answer, so these instances are correct contexts beyond the model's PK.
Results. We report the share of each decision in each cell, as ranges over the four backbones. The results are shown below.
| Cell | Answer with CK (%) | Answer with PK (%) | Abstain (%) |
|---|---|---|---|
| 81.8–86.9 | 1.5–5.5 | 11.1–12.7 | |
| 9.8–14.4 | 46.1–62.0 | 28.2–40.5 | |
| 16.6–20.9 | 5.2–12.9 | 68.7–78.3 |
From the table, we have the following observations.
(1) SABER follows correct context beyond its PK. On , it answers with the context in 81.8–86.9% of cases and falls back to its own knowledge in only 1.5–5.5%. The rest are abstentions, which produce no wrong answer. This is because the CK belief is supervised by the correctness of the CK answer. Therefore, the self-evaluation does not lead SABER to reject such context.
(2) SABER separates a context that lacks the knowledge from a correct context that conflicts with wrong PK. The PK answer is wrong in both and , yet SABER answers with the context in 81.8–86.9% of the instances and in only 16.6–20.9% of the instances, where it mostly abstains (68.7–78.3%). On , it keeps the PK answer in 46.1–62.0% of cases and follows the misleading context in only 9.8–14.4%. This is because the CK belief estimates whether the context supports a correct answer, while the PK belief separately estimates whether the model's own knowledge does. Therefore, the three situations you describe receive three different decisions.
Rebuttal by Authors (3/5)
W2. SABER's contribution beyond supervised correctness probing, a comparison with a simple candidate-answer classifier, and ablations of self-evaluation and the feature branches
We thank you for these insightful questions. Below we first clarify SABER's contribution beyond supervised correctness probing, then compare SABER with the simple classifier you describe, and finally report the ablations you request.
Supervised probing is only the tool SABER uses to read hidden states. SABER's contribution lies in what it estimates and how it decides. Standard correctness probing estimates whether one given answer is correct. SABER instead formulates knowledge conflict as two separate correctness questions, one for the PK path and one for the CK path, and labels both paths for each backbone in our model-specific benchmark. It reads the reliability of each path from the states after the model reasons under that source, and it combines the two beliefs in a 4-cell decision that answers with PK, answers with CK, or abstains. We verify this contribution below.
To test whether the two-path design adds to probing a single answer, as you ask, we keep SABER's own predictions behind Table 3 and change only the decision rule. A single-path rule corresponds to standard correctness probing. It answers from one path when its belief reaches and abstains otherwise.
Results. We report the Score and abstention F1 of Table 3 on the test set. The results are shown below.
| Backbone | Decision rule | Score (%) ↑ | F1 (%) ↑ |
|---|---|---|---|
| Llama-3.1-8B | PK path only | 13.0 | 59.9 |
| CK path only | 39.3 | 71.5 | |
| Two paths (SABER) | 43.9 | 72.5 | |
| Qwen-2.5-7B | PK path only | 10.8 | 62.7 |
| CK path only | 38.3 | 75.3 | |
| Two paths (SABER) | 43.4 | 78.0 | |
| Llama-3.2-3B | PK path only | 7.9 | 63.8 |
| CK path only | 34.9 | 75.9 | |
| Two paths (SABER) | 38.5 | 76.8 | |
| Qwen-2.5-3B | PK path only | 5.7 | 64.9 |
| CK path only | 37.5 | 75.9 | |
| Two paths (SABER) | 39.6 | 76.6 |
From the table, we observe that with identical predictions, the two-path rule outperforms the better single-path rule on every backbone, by 2.1–5.1% in Score and 0.7–2.7% in F1. This is because a single-path rule can only accept or reject one answer. When that answer is wrong, it cannot fall back to the other path, and it cannot recognize the cases where both paths fail. Therefore, the gain comes from the two-path formulation rather than from probing itself.
SABER outperforms a simple classifier trained on our labels, because such a classifier judges only one answer. To compare SABER with the simple classifier you describe, we train an MLP on the hidden state of the model as it generates its CK answer, without reasoning or self-evaluation, to predict whether this answer is correct. As in standard correctness probing, it keeps the CK answer when the answer is judged correct and otherwise falls back to the PK answer, or abstains in the setting of Table 3.
Results. We report Acc, MFS, and the Score of Table 3 on the test set. The results are shown below.
| Backbone | Method | Acc (%) ↑ | MFS (%) ↑ | Score (%) ↑ |
|---|---|---|---|---|
| Llama-3.1-8B | Simple classifier | 63.1 | 81.6 | 39.4 |
| SABER | 65.9 | 83.6 | 43.9 | |
| Qwen-2.5-7B | Simple classifier | 57.2 | 85.6 | 36.7 |
| SABER | 61.0 | 88.1 | 43.4 | |
| Llama-3.2-3B | Simple classifier | 57.2 | 83.0 | 35.2 |
| SABER | 60.8 | 85.9 | 38.5 | |
| Qwen-2.5-3B | Simple classifier | 53.8 | 84.3 | 32.7 |
| SABER | 57.0 | 82.9 | 39.6 |
From the table, we observe that SABER achieves higher Acc and Score than the simple classifier on every backbone, by 2.8–3.8% and 3.3–6.9% respectively, as well as a higher average MFS. This is because the classifier estimates only whether the CK answer is correct. When it rejects this answer, it cannot tell whether the PK answer is reliable, so it either switches to the PK answer blindly or abstains. SABER instead estimates both paths and chooses between them. Therefore, SABER's contribution beyond correctness probing lies in its two-path estimation and decision.
Rebuttal by Authors (1/5)
We sincerely thank you for your valuable comments and insightful suggestions, and for recognizing our model-specific benchmark, the lightweight design of SABER on a frozen backbone, and its improved conflict handling and abstention. Below we address each of your questions in turn.
We thank you for this insightful question. Below we first explain how SABER's design addresses each of your two questions, and then verify both with experiments.
SABER does not reject correct context beyond its PK, because its CK belief is learned from whether the CK answer is actually correct. SABER does not act on the model's verbal verdict. It reads hidden states over the reasoning span, and the CK-side trace is generated under a prompt that instructs the model to assume the document is reliable and reason only from it. These states therefore reflect how the model reasons with the document. The CK predictor maps them to a belief trained on yCK, the observed correctness of the CK answer. Correct contexts beyond the model's PK are common in our benchmark, where the instances with only the CK answer correct make up 35.4–37.3% of the test set on each backbone. A predictor that rejected such contexts would be penalized on all of them. Therefore, the supervision itself teaches SABER to trust them.
SABER separates missing knowledge in the context from being misled by its own PK, because the CK predictor also sees how reliable the model's own knowledge is. The CK predictor takes the self-prior together with the CK-side representation, and the self-prior reflects how much the model knows about the question, and thus how much it needs external evidence. The CK predictor therefore knows when the model's own knowledge is weak, and it can discount a conflict between the document and such knowledge. Together with the PK belief, the 4-cell decision separates the situations you describe.
(1) The context supplies knowledge beyond the model's PK (C01). The PK answer is wrong and the CK answer is right, so SABER should answer with the context. This is where being misled by its own PK would cause harm.
(2) The context misleads while the model's PK is right (C10). SABER should keep the PK answer.
(3) The context lacks the knowledge and the model's PK is also wrong (C00). SABER should abstain.
In both (1) and (3), the model's PK is wrong, so a judgment driven by the model's own knowledge cannot tell them apart. The only difference is whether the context contains the needed knowledge, which is exactly what the CK belief estimates.