- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies context-grounded question answering through a paired evaluation of answering and abstention. It introduces PFaithBench, which associates each question with sufficient and insufficient contexts, and defines joint metrics that measure whether a model answers correctly in the former condition and abstains in the latter. The benchmark contains 3,344 paired examples across multi-hop, single-hop, original factual, and counterfactual subsets, with human verification of the evaluation data. The authors evaluate 39 model configurations from seven families and report frequent over-answering under insufficient context. They further conduct GRPO-based training experiments to examine source–sufficiency correlations and different proportions of answering and abstention examples, including asymmetric and symmetric reward settings. The reported results highlight source-dependent shortcuts and the sensitivity of answering and abstention behavior to training data composition.
Critical strengths
-
The paired evaluation design provides a useful diagnostic of context-sensitive behavior. PFaithBench holds the question fixed while varying whether the context contains sufficient evidence. Its strict pairwise accuracy requires both a correct answer under the sufficient context and abstention under the insufficient context. This captures whether both behaviors succeed for the same question, which cannot be determined from aggregate answering and abstention accuracies alone. I consider this a meaningful contribution to the evaluation of context-grounded QA.
-
The empirical study has substantial breadth and provides useful observations about training data composition. The evaluation covers 39 model configurations across seven families and includes multi-hop, single-hop, original factual, and counterfactual contexts. The training analyses demonstrate pronounced source-dependent answering/abstention behavior under mismatched data construction and examine different proportions of answering and abstention examples. Appendix C.2 also evaluates a symmetric reward, providing additional support that the qualitative data-composition effect is not confined to the asymmetric reward used in the main experiments.
Critical weaknesses
-
The training comparison does not isolate the contribution of preserving question-level context pairs. In Section 4.2, MHSuff-SHInsuff and SHSuff-MHInsuff make contextual sufficiency directly correlated with the QA source. PairContext simultaneously removes this source–sufficiency association and exposes each question under both sufficient and insufficient contexts. The results in Table 4 therefore support the concern that source-correlated training can induce shortcuts, but do not establish that question-level pairing itself is responsible for the improvement. A source-balanced, non-paired baseline, with answering and abstention examples drawn from different questions within each source and matched training budgets, is needed to distinguish these explanations. This matters because the paper explicitly uses the comparison to argue for the importance of preserving context pairs during training.
-
The interpretation of the Leakage metric is insufficiently justified, particularly for counterfactual examples. Equation (3) defines Leakage as the probability of producing the reference answer under insufficient context. This is a behavioral proxy rather than direct evidence that the model relied on memorized parametric knowledge. More specifically, under the stated definition, the reference answer for PFB-ConCF is the counterfactual answer. In Appendix C.6, the model answers “Spanish” after the supporting evidence for “Catalan” is removed, and the paper interprets this as internal knowledge leakage. However, if Equation (3) is applied using the counterfactual reference, this response is classified as W rather than C and does not contribute to Leakage. The operational metric would therefore miss the very type of fallback illustrated by this case study. The reference used for this metric needs clarification, and fallback to the original factual answer should be distinguished from reproduction of the counterfactual answer. This concern affects the conclusions about internal knowledge leakage, although it does not by itself invalidate the OverAns or SFAcc metrics.
Overall, I find the benchmark perspective useful, but the evidence for the specific benefit of paired training and the interpretation of internal knowledge leakage remains insufficient. These concerns lead me to recommend weak rejection in the current form.
Questions for authors
-
Controlled comparison for the pairing claim: Do you have results for a non-paired training condition in which each QA source contains a balanced mixture of sufficient and insufficient examples, but the two conditions use different questions? With training example counts, source proportions, and optimization budgets matched to PairContext, does question-level pairing still provide a substantial improvement? This comparison would materially affect my assessment of the training contribution.
-
Clarification of counterfactual Leakage: For PFB-ConCF, is the C label in Equation (3) computed against the counterfactual answer or the original factual answer? How is the “Spanish” response in Appendix C.6 counted in the reported Leakage results? Please clarify the implementation and, if available, report the rates of fallback to the original factual answer separately from reproduction of the counterfactual answer.
-
Validation of response classification: Appendix A.5 uses Qwen3-30B-A3B-Instruct-2507 to classify responses as IDK, C, or W. Was this classification validated against human judgments, particularly for counterfactual answers and responses that combine uncertainty with a concrete answer? Agreement rates and an analysis of classification errors would help establish the reliability of the benchmark results.
11 comments
On PFB-SingleHop
| Model | Setting | DFAcc ↑ | SFAcc ↑ | AnsAcc ↑ | AbsAcc ↑ | OverAns ↓ | OverAbs ↓ |
|---|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | MH+/SH- | 37.14 | 31.50 | 32.28 | 98.56 | 1.44 | 61.68 |
| SH+/MH- | 16.01 | 14.70 | 91.60 | 16.80 | 83.20 | 1.05 | |
| NonPair | 76.25 | 72.31 | 85.17 | 83.99 | 16.01 | 8.66 | |
| Pair | 79.13 | 75.33 | 86.22 | 86.35 | 13.65 | 7.74 | |
| Llama-3.2-3B-Instruct | MH+/SH- | 26.90 | 23.10 | 24.93 | 97.51 | 2.49 | 70.87 |
| SH+/MH- | 36.35 | 33.07 | 82.55 | 39.90 | 60.10 | 4.20 | |
| NonPair | 64.96 | 58.79 | 75.72 | 78.74 | 21.26 | 14.04 | |
| Pair | 67.45 | 62.86 | 79.27 | 79.40 | 20.60 | 12.47 | |
| Qwen3-4B (NoThink) | MH+/SH- | 67.72 | 56.30 | 61.02 | 93.44 | 6.56 | 26.51 |
| SH+/MH- | 69.42 | 63.78 | 86.48 | 73.62 | 26.38 | 5.25 | |
| NonPair | 73.75 | 66.27 | 80.31 | 82.02 | 17.98 | 8.92 | |
| Pair | 77.95 | 74.15 | 81.63 | 90.29 | 9.71 | 12.60 | |
| Qwen3-1.7B (NoThink) | MH+/SH- | 43.57 | 19.16 | 21.26 | 93.96 | 6.04 | 51.18 |
| SH+/MH- | 38.58 | 31.50 | 74.54 | 42.65 | 57.35 | 5.12 | |
| NonPair | 56.04 | 41.34 | 59.19 | 73.88 | 26.12 | 19.69 | |
| Pair | 58.01 | 50.52 | 65.88 | 79.53 | 20.47 | 23.62 |
On PFB-ConOri
| Model | Setting | DFAcc ↑ | SFAcc ↑ | AnsAcc ↑ | AbsAcc ↑ | OverAns ↓ | OverAbs ↓ |
|---|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | MH+/SH- | 66.75 | 61.44 | 66.62 | 94.06 | 5.94 | 27.69 |
| SH+/MH- | 15.80 | 14.29 | 91.40 | 16.81 | 83.19 | 1.14 | |
| NonPair | 78.38 | 72.95 | 85.34 | 86.73 | 13.27 | 8.34 | |
| Pair | 80.40 | 75.98 | 85.84 | 89.00 | 11.00 | 9.10 | |
| Llama-3.2-3B-Instruct | MH+/SH- | 65.36 | 52.72 | 57.77 | 93.30 | 6.70 | 28.45 |
| SH+/MH- | 52.84 | 49.18 | 88.24 | 57.14 | 42.86 | 4.80 | |
| NonPair | 76.61 | 70.92 | 84.20 | 85.21 | 14.79 | 8.72 | |
| Pair | 78.00 | 73.32 | 85.71 | 86.22 | 13.78 | 8.22 | |
| Qwen3-4B (NoThink) | MH+/SH- | 76.36 | 65.99 | 76.36 | 86.73 | 13.27 | 11.38 |
| SH+/MH- | 70.80 | 65.87 | 86.09 | 77.75 | 22.25 | 6.95 | |
| NonPair | 75.47 | 68.02 | 81.67 | 85.08 | 14.92 | 9.61 | |
| Pair | 80.66 | 75.60 | 81.29 | 94.18 | 5.82 | 13.65 | |
| Qwen3-1.7B (NoThink) | MH+/SH- | 67.89 | 44.75 | 49.94 | 91.40 | 8.60 | 23.89 |
| SH+/MH- | 50.82 | 46.90 | 84.20 | 57.27 | 42.73 | 7.21 | |
| NonPair | 69.66 | 60.68 | 77.37 | 79.77 | 20.23 | 11.38 | |
| Pair | 68.39 | 62.83 | 79.39 | 81.42 | 18.58 | 13.40 |
On PFB-ConCF
| Model | Setting | DFAcc ↑ | SFAcc ↑ | AnsAcc ↑ | AbsAcc ↑ | OverAns ↓ | OverAbs ↓ |
|---|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | MH+/SH- | 42.98 | 24.78 | 25.92 | 94.94 | 5.06 | 53.86 |
| SH+/MH- | 12.39 | 7.59 | 52.72 | 14.79 | 85.21 | 3.79 | |
| NonPair | 67.64 | 37.67 | 42.10 | 85.21 | 14.79 | 20.99 | |
| Pair | 70.16 | 42.35 | 47.03 | 87.74 | 12.26 | 19.22 | |
| Llama-3.2-3B-Instruct | MH+/SH- | 48.17 | 19.97 | 20.73 | 95.07 | 4.93 | 49.30 |
| SH+/MH- | 39.19 | 23.14 | 38.56 | 53.35 | 46.65 | 21.74 | |
| NonPair | 60.30 | 36.41 | 42.10 | 81.92 | 18.08 | 25.16 | |
| Pair | 63.46 | 40.83 | 46.40 | 84.70 | 15.30 | 24.40 | |
| Qwen3-4B (NoThink) | MH+/SH- | 63.08 | 32.49 | 35.90 | 87.61 | 12.39 | 27.18 |
| SH+/MH- | 63.97 | 34.39 | 41.09 | 77.37 | 22.63 | 15.68 | |
| NonPair | 65.87 | 36.54 | 41.47 | 82.05 | 17.95 | 18.84 | |
| Pair | 68.65 | 39.70 | 40.71 | 94.56 | 5.44 | 28.07 | |
| Qwen3-1.7B (NoThink) | MH+/SH- | 61.44 | 20.86 | 22.63 | 91.15 | 8.85 | 30.47 |
| SH+/MH- | 50.06 | 27.69 | 47.16 | 56.01 | 43.99 | 8.98 | |
| NonPair | 63.59 | 30.97 | 38.94 | 77.62 | 22.38 | 15.93 | |
| Pair | 61.69 | 34.26 | 41.72 | 80.66 | 19.34 | 20.99 |
The additional experiments provide two important findings that directly address the reviewer's concern.
First, source balancing substantially mitigates source-conditioned behavioral biases. Compared with mismatched-source training, NonPairContext improves average DFAcc and SFAcc and substantially reduces the imbalance between over-answering and over-abstention. This suggests that eliminating source–sufficiency correlations helps prevent models from relying on source-specific cues to determine whether to answer or abstain.
Second, preserving question-level context pairs provides additional benefits beyond source balancing. Despite using sufficient and insufficient examples from both QA sources, NonPairContext consistently underperforms PairContext in strict pairwise faithfulness. Across the four models and four evaluation subsets, PairContext improves SFAcc in all 16 comparisons, with an average improvement of 6.16 percentage points. DFAcc also improves in 14 of 16 comparisons, with an average gain of 4.33 percentage points. Furthermore, the mean absolute OverAns–OverAbs difference decreases from 9.85 under NonPairContext to 6.32 under PairContext. These results indicate that question-level pairing contributes additional improvements in both joint faithfulness and behavioral balance, beyond merely eliminating source–sufficiency correlations.
Overall, the new experiments distinguish two complementary factors underlying effective faithfulness training: source balancing mitigates source-conditioned biases, while preserving question-level context pairs further improves joint answering–abstaining behavior. We will incorporate these additional experiments and revise Section 4.2 to clarify the respective contributions of both factors.
Dear Reviewer zLKQ,
Thank you for your valuable comments. In response to your questions, we will address them point by point, and our replies are as follows:
A: We thank the reviewer for pointing out that the original experiments in Section 4.2 do not fully disentangle the effects of source–sufficiency correlation and question-level context pairing. To address this concern, we conducted additional GRPO experiments with a source-balanced, non-paired baseline ∗∗NonPairContext∗∗, allowing us to separately investigate these two factors.
We constructed NonPairContext by dividing the questions within each of the SingleHop and MultiHop datasets into two disjoint halves. One half was used with sufficient contexts for answering supervision, while the other half was used with insufficient contexts for abstention supervision. Thus, both QA sources contribute equally to answering and abstention supervision, while each question appears under only one contextual condition. This eliminates the source–sufficiency association without introducing question-level pairing, enabling a more controlled comparison with PairContext.
We evaluated NonPairContext on four models and compared it with both mismatched-source settings and PairContext across all four PFaithBench subsets. The complete results are presented below.
On PFB-MultiHop