- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CoffeeQA, a few-shot knowledge distillation framework for open-book question answering. The paper first generates counterfactual contexts supporting alternative entity answers for factual samples, and then filters them using Counterfactual Feasibility Difficulty (CFD). This paper demonstrates that counterfactual pairing can benefit low-margin hard samples.
Critical strengths
- The paper discusses an important problem. Data quality and data selection are important to both knowledge distillation and LLM post-training, especially when the training budget is limited. Identifying a small set of informative examples could have substantial practical value.
- Using counterfactual examples to expose changes in the teacher’s behavior is intuitively meaningful.
Critical weaknesses
- The independent contribution of CoffeeQA is unclear. All reported results compare KD/LWD/SeqKD with “+CoffeeQA”. The paper does not separate the effects of CFD selection, counterfactual pairing, and additional supervision. Also, the average gains are modest.
- The experiments only compare CoffeeQA against baselines trained on a selected low-margin subset. It does not show that CoffeeQA is preferable to simpler data-selection strategies, such as random sampling or selecting medium-margin or high-margin factual examples. What if a simple sampling strategy match or outperform CoffeeQA without any counterfactual generation?
- The model family is outdated. The experiments only use T5 and Qwen2.5 in 2026. Since the paper propose a solution for efficient training and distillation, it should demonstrate that the method is useful on recent small-model families, such as Qwen3 or Llama 3.x. This is an essential experiment in the current era of rapid LLM development. The current experiments do not establish that CoffeeQA remains effective for contemporary models with stronger instruction-following and context-use abilities.
- The GPT-4o-mini counterfactual generation is not validated. The method directly asks GPT-4o-mini to rewrite contexts. No experiments or studies verify whether GPT-4o-mini reliably follows the generation instructions to produce valid and minimally modified counterfactual contexts.
Questions for authors
- Why is CoffeeQA evaluated only on open-book QA? Is the framework inherently limited to QA tasks, or is this only a limitation of the current experimental setup?
- How sensitive is CoffeeQA to the counterfactual generator? GPT-4o-mini may produce inconsistent or low-quality context rewrites. Have the authors compared it with a stronger model?
9 comments
(continuing ..)
- Overall contribution and statistical significance: COFFEQA achieves a statistically significant improvement of 4.1 EM points (95% CI: [3.4, 4.8], p < 0.001) and 2.8 F1 points (95% CI: [2.5, 3.2], p < 0.001) over factual-only KD. Importantly, COFFEQA outperforms KD across all five random seeds and achieves higher EM in 24 of the 25 dataset and k-shot budget combinations, with no statistically significant decreases. (See Appendix A.2 of the revised manuscript. Also in Reviewer 1, Part 2 response).
These results provide additional evidence that counterfactual supervision contributes beyond the original factual selection strategy. We have incorporated the new ablations and clarified the contributions and limitations of each component in Appendix A.2, A.3, A.6 of the revised manuscript.
Part 2 (W2): Effect of Factual Sampling Baselines
COFFEQA provides additional performance gains across alternative factual sampling baselines. To investigate whether the improvements of COFFEQA arise from counterfactual supervision or factual sample selection, we conducted additional experiments on T5 using three factual sampling baselines: random, medium-margin, and large-margin sampling. We compared factual-only KD against COFFEQA under each baseline across five datasets and five few-shot budgets, using five random seeds. As shown in Table 3, COFFEQA improves average EM over factual-only KD across all five datasets under random and large-margin sampling. Under the medium-margin baseline, COFFEQA improves performance on four of the five datasets.
| Method | NQ | SQuAD | NewsQA | TriviaQA | HotpotQA | Avg |
|---|---|---|---|---|---|---|
| KD (paper) | ||||||
| COFFEQA (paper) | ||||||
| Random KD | ||||||
| Random COFFEQA | ||||||
| Medium margin KD | ||||||
| Medium margin COFFEQA | ||||||
| Large-margin KD | ||||||
| Large-margin COFFEQA |
Table 3: Comparison of factual sampling baselines for KD and COFFEQA. In-domain Exact Match (EM) scores are reported across five datasets, averaged over different few-shot budgets in the T5 setting. The baselines include low-margin, random, medium-margin, and high-margin sampling strategies. Standard deviations are shown as subscripts. Bold values indicate the better-performing method within each KD–COFFEQA pair. The Avg column reports the mean across the five datasets.
We also observe that the medium-margin factual-only KD baseline outperforms the original low-margin COFFEQA configuration on all five datasets, highlighting the importance of factual sample selection. Nevertheless, the improvements observed under matched factual sampling baselines demonstrate that counterfactual supervision provides additional benefits beyond factual selection alone. We have included these comparisons in Appendix A.3 of the revised manuscript.
(continued in the next comment.)
(continuing ..)
Part 3 (W3): Generalization of COFFEQA to Recent Model Families (Qwen3)
COFFEQA consistently improves distillation performance with the newer Qwen3 model family. To evaluate whether COFFEQA remains effective with a more recent model family, we extended our experiments to Qwen3, using Qwen3-8B as the teacher and Qwen3-1.7B as the student. Both KD and COFFEQA use the same experimental settings, with few-shot budgets ranging from 16 to 256 across five random seeds. As shown in Table 4, COFFEQA achieves higher average EM than KD across all five datasets. The largest improvements are observed on TriviaQA (42.91 to 52.80), HotpotQA (54.37 to 62.88), and NewsQA (40.83 to 47.35), with additional gains on NQ (61.11 to 63.83) and SQuAD (62.44 to 66.29). These results demonstrate that the benefits of counterfactual supervision extend to newer model families, further supporting the generalizability of COFFEQA. We have included these results in Appendix A.4 of the revised manuscript.
| Dataset | Method | 16 | 32 | 64 | 128 | 256 | Avg |
|---|---|---|---|---|---|---|---|
| NQ | KD | ||||||
| COFFEQA | |||||||
| SQuAD | KD | ||||||
| COFFEQA | |||||||
| HotpotQA | KD | ||||||
| COFFEQA | |||||||
| NewsQA | KD | ||||||
| COFFEQA | |||||||
| TriviaQA | KD | ||||||
| COFFEQA |
Table 4: Qwen3 In-Domain Exact Match (EM) Comparison of KD and COFFEQA. Results are reported across five datasets and five few-shot budgets (k = 16–256), using Qwen3-8B as the teacher and Qwen3-1.7B as the student. Values represent mean EM over five random seeds, with standard deviations shown as subscripts. Bold values indicate the better-performing method.
Part 4 (Q1): Generalizability Beyond Open-Book QA
COFFEQA is not inherently limited to open-book QA, but our current formulation focuses on this setting because counterfactuals can be generated by modifying the supporting context while keeping the question fixed. We also focus on named-entity answers, which makes it easier to construct and validate alternative answers.
Extending this approach to other generative tasks is challenging because small input changes may affect multiple parts of the output, and there may not be a single correct alternative response. Nevertheless, the framework could be adapted to multiple-choice QA by modifying the evidence to support a different answer option, mathematical reasoning by changing numerical values or premises and updating the solution, or summarization by modifying key facts and their corresponding summaries. Such extensions would require task-specific counterfactual generation and validation methods. We have clarified in Appendix B of the revised manuscript that this is a limitation of our current experimental setup rather than the general idea of counterfactual-guided distillation.
(continued in the next comment.)
We thank the reviewer for recognizing the contributions of our work and for providing constructive feedback and insightful questions. We address the reviewer's comments and questions below.
Part 1 (W1): Ablation Studies and Independent Contributions of COFFEQA
We had included several ablation studies in Appendix G of the original manuscript, examining the effects of CFD validation, CFD threshold selection. To further clarify the independent contributions of COFFEQA, we have now conducted additional experiments on T5, including component-wise CFD ablations, comparisons with simple entity replacement, alternative factual sampling strategies, and statistical significance analysis.
These existing and newly added experiments provide the following insights:
Table 1: Ablation study of CFD components. Exact Match (EM) scores for standard KD, AA-only, CLR-only, and full CFD filtering across five datasets, averaged over five few-shot budgets and five random seeds. Values in parentheses indicate EM differences relative to full CFD (COFFEQA). Bold indicates the best-performing method for each dataset.
Table 2: Comparison of COFFEQA with simple entity replacement and standard KD. Factual Exact Match (EM) scores across five datasets using T5, averaged over different few-shot budgets. Standard deviations are shown as subscripts. Bold values indicate the best-performing method among the three approaches. The Avg column reports the mean across the five datasets.
(continued in the next comment.)